Android

After the one-time model download, the only network traffic is usage reporting. Why on the device.

Before you start

Speech in (16 kHz mono), picture frames out. Each model is one Gradle dependency from maven.bithuman.ai. The SDK is pre-1.0 (0.x); 1.0 is planned.

Known issue in 0.10.x: an idle avatar keeps rendering at full rate, so long idle sessions can warm the device. A fix is in progress.

• Android SDK platform 35 and build-tools 34 (sdkmanager "platforms;android-35" "build-tools;34.0.0" platform-tools), found through ANDROID_HOME or sdk.dir in local.properties. To check the toolchain without a phone, run ./gradlew :app:assembleDebug in bithuman-examples/android/essence2-hello; it writes app/build/outputs/apk/debug/app-debug.apk.
• JDK 17, Gradle 8.11+ and Android Gradle Plugin 8.7+. The example accepts JDK 17–23 (17 or 21 recommended). Gradle 8.11 cannot run on a newer default JDK such as 25; the error is only a Java version number. Set export JAVA_HOME=/path/to/jdk-17, or org.gradle.java.home=/path/to/jdk-17 in ~/.gradle/gradle.properties.
• Frame rate depends on the phone. Essence 2 plays at 25 frames a second, so live speech needs at least that render rate. Recent flagship chips exceed it; older chips, such as the Snapdragon 8 Gen 2 (Galaxy Z Flip5), can fall below real time, more so when warm. Test your target phones; measured rates.
• Essence 1 is not available on phones (First generation). The deprecated first-generation ai.bithuman:sdk: troubleshooting.

Install

Add bitHuman’s Maven repository for ai.bithuman, restrict the build to arm64-v8a, and turn on legacy packaging (native libraries extracted to disk). Merge the lines into your app module, which keeps its namespace, compileSdk and Java 17 targets. The secret reaches your code through BuildConfig.

// settings.gradle.kts
dependencyResolutionManagement {
    repositories {
        google()
        mavenCentral()
        exclusiveContent {   // ai.bithuman resolves from bitHuman's repository only
            forRepository { maven { url = uri("https://maven.bithuman.ai") } }
            filter { includeGroup("ai.bithuman") }
        }
    }
}

// app/build.gradle.kts
// bithumanApiSecret=… in ~/.gradle/gradle.properties, else BITHUMAN_API_SECRET in the environment,
// else bithuman.apiSecret=… in local.properties (git-ignored)
val bithumanApiSecret: String =
    providers.gradleProperty("bithumanApiSecret").orNull?.takeIf { it.isNotBlank() }
        ?: providers.environmentVariable("BITHUMAN_API_SECRET").orNull?.takeIf { it.isNotBlank() }
        ?: run {
            val props = java.util.Properties()   // fully qualified, so these lines can go anywhere in the file
            val f = rootProject.file("local.properties")
            if (f.isFile) f.inputStream().use { props.load(it) }
            props.getProperty("bithuman.apiSecret")
        }
        ?: ""

android {
    buildFeatures { buildConfig = true }
    defaultConfig {
        minSdk = 26   // 29 for Essence 2
        ndk { abiFilters += "arm64-v8a" }
        buildConfigField("String", "BITHUMAN_API_SECRET", "\"$bithumanApiSecret\"")
    }
    packaging { jniLibs { useLegacyPackaging = true } }   // required
}
dependencies {
    implementation("ai.bithuman:expression2-android:0.6.2")
    // or: implementation("ai.bithuman:essence2-android:0.10.3")
}

Keep the secret outside your source tree, in ~/.gradle/gradle.properties: the Android Expression 2 example reads the three places in the order the comment above lists (Android Studio keeps local.properties out of git). Never commit the secret or put it on a command line. An empty secret makes the session refuse to start.

Without an NDK, the build prints Unable to strip the following libraries, packaging them as they are: libLiteRt.so, libQnn…; that is expected.

It also works in a Groovy settings.gradle; the engines’ own dependencies come from google() and mavenCentral().

expression2-android brings the Qualcomm accelerator runtime (com.qualcomm.qti:qnn-litert-delegate:2.49.0 and com.qualcomm.qti:qnn-runtime:2.49.0). To keep the APK small and render on the CPU, exclude it:

implementation("ai.bithuman:expression2-android:0.6.2") {
    exclude(group = "com.qualcomm.qti")
}

Authenticate

Before you download or create an avatar, pass your API secret (create one) in code: Expression2Credential.set(secret) or Essence2Credential.set(secret). One call covers download and session (Your API secret).

Warning: a buildConfigField compiles the secret into the APK, where anyone with the file can read it. Use it for local builds only.

A shipped app fetches its secret from your backend (What a shipped app holds).

Run your first avatar

Expression 2, with the published wise-pup avatar (agent code A23WJF0199). Call render off the main thread.

import ai.bithuman.expression2.Expression2Avatar
import ai.bithuman.expression2.Expression2Credential
import ai.bithuman.expression2.Expression2ModelStore
import ai.bithuman.expression2.Expression2Options
import android.content.Context
import android.graphics.Bitmap

/** [pcm16k] is 16 kHz mono float32 in [-1, 1]. */
fun render(context: Context, pcm16k: FloatArray, show: (Bitmap) -> Unit) {
    Expression2Credential.set(BuildConfig.BITHUMAN_API_SECRET)        // before fetch() and create()
    val model = Expression2ModelStore(context).fetch("A23WJF0199")     // ~160 MB, first run only

    Expression2Avatar.create(context, model, Expression2Options()).use { avatar ->
        val frame = avatar.newFrameBitmap()   // 416 x 720, allocate once
        avatar.feed(pcm16k)
        avatar.flushTail()                    // end of the utterance
        while (true) {
            if (avatar.pull(frame) != null) { show(frame); continue }
            if (!avatar.hasPendingTail && avatar.queuedFrames == 0) break
            Thread.sleep(10)                  // null means "not ready yet"
        }
    }
}

Expected: show receives 20 frames per second of audio, lips following the speech. The first create() after install is slower (the accelerator prepares once). Create once, at app start, on a background thread.

The install line and the create … use { … } block alone (show and pcm16k are yours):

implementation("ai.bithuman:expression2-android:0.6.2")

Expression2Avatar.create(context, model, Expression2Options()).use { avatar ->
    val frame = avatar.newFrameBitmap()
    avatar.feed(pcm16k)
    avatar.flushTail()
    while (true) {
        if (avatar.pull(frame) != null) { show(frame); continue }
        if (!avatar.hasPendingTail && avatar.queuedFrames == 0) break
        Thread.sleep(10)
    }
}

You bring the replies. The hello example lip-syncs a bundled clip; a character that answers needs your own speech to text, language model and voice feeding feed().

Essence 2 takes 16-bit little-endian PCM bytes, as a 16 kHz mono WAV stores them, and fills an RGBA ByteBuffer sized from the identity:

import ai.bithuman.essence2.Essence2Avatar
import ai.bithuman.essence2.Essence2Credential
import ai.bithuman.essence2.Essence2ModelStore
import android.content.Context
import java.nio.ByteBuffer

fun render(context: Context, pcm16le: ByteArray, show: (ByteBuffer, Int, Int) -> Unit) {
    Essence2Credential.set(BuildConfig.BITHUMAN_API_SECRET)                   // before fetch() and create()
    val identity = Essence2ModelStore(context).fetch("A52DHS2219")            // sofia-ramirez; 226–281 MB, first run only

    Essence2Avatar.create(identity.dir).use { avatar ->
        val frame = avatar.newFrameBuffer()   // width * height * 4, RGBA
        avatar.feed(pcm16le)
        avatar.endOfAudio()
        var quietMs = 0
        while (quietMs < 5_000) {             // 5 s with no frame at all = finished
            if (avatar.pull(frame)) { show(frame, avatar.width, avatar.height); quietMs = 0; continue }
            Thread.sleep(10)                  // false means "not ready yet"; the first frame can take seconds
            quietMs += 10
        }
        avatar.checkRender()                  // throws Essence2RenderFailed if the engine stopped
        // a refused session throws Essence2MeteringRefused from pull() or idle()
    }
}

The identity sets the frame size (portrait 1080×1920, landscape 1920×1080 or 1280×720), the same on both delivery paths: read avatar.width and avatar.height rather than hard-coding them.

Play with sound (8 seconds, captions)

Essence 2 on the new engine

E2UAvatar runs Essence 2 on the new engine, Essence2Avatar (above) on the existing one. E2UAvatar and its downloader E2UModelStore ship in the ai.bithuman.elevate package of ai.bithuman:essence2-android 0.10.0+. The rest of the 0.9.6 API is unchanged; an app that never opens an E2UAvatar never loads the new engine.

• Modes: E2UAvatar.Mode.FULL is the full body at 1920×1080; Mode.FACE is 512×512. setMode switches within 4 frames, no reload, no audio drop. An avatar without a full body opens in FACE (hasFullBody false).
• Audio in, frames out: 16 kHz mono float samples in; RGBA frames out at 25 fps of audio, pulled in order.

val store = E2UModelStore(context, apiSecret)          // downloads and caches; sha256-verified, resumable
val fetched = try {
    store.fetch(agentCode, E2UAvatar.Mode.FULL) { done, total -> /* one progress total */ }
} catch (p: E2UModelStore.Preparing) {
    // still being prepared for this device: retry after p.etaSeconds, or after 30 s, 60 s, 120 s, then every 5 min
    return
}
val avatar = E2UAvatar.create(fetched.shared, fetched.avatar, agentCode, mode = fetched.defaultMode)
val buf = avatar.newFrameBuffer()
avatar.feed(floatSamples)            // 16 kHz mono float
avatar.endOfAudio()                  // at the end of an utterance, so its last frames render
while (avatar.pull(buf)) { /* RGBA, avatar.lastWidth x avatar.lastHeight */ }
avatar.setMode(E2UAvatar.Mode.FACE)  // switch at any time
avatar.close()                       // Closeable; billing stops here

E2UModelStore.fetch throws Preparing until the agent’s Android set is published; retry later.

1. Metering. Metered like Essence 2 (How billing works) until close(), so always call it. A missing or refused API secret fails at create with Essence2MeteringRefused.
2. First download is large (the device set plus shared networks), cached once.
3. NPU. The AAR does not bundle the Qualcomm QNN runtime. For a Qualcomm NPU, add com.qualcomm.qti:qnn-runtime (2.49.0 verified) and packaging { jniLibs { useLegacyPackaging = true } }; Android 12+ may also need <uses-native-library android:name="libcdsprpc.so" android:required="false"/> inside <application>. Without QNN, and on non-Qualcomm phones, the networks run on the CPU; the full body renders on the GPU (OpenGL ES 3.1 compute). avatar.placement reports where each network runs.
4. First open on an NPU phone is slower: it compiles the networks (cached in npuCache).
5. pull returns false when no frame is ready yet, or in the background past the interim rule.
6. create defaults: threads = 2, workers = defaultWorkers() (cores minus 2, between 1 and 6).

Performance