Android

Both models render on the phone: you feed 16 kHz mono speech in and pull picture frames out. After the one-time model download, the only network traffic is usage reporting. Each model is one Maven Central dependency.

Play with sound (8 seconds, captions)

Before you start

• JDK 17, Gradle 8.11 or newer and Android Gradle Plugin 8.7 or newer.
• A physical arm64 phone. Emulators cannot load the engines.
• Essence 1 is not available on phones: use Essence 2 or Expression 2 on devices (First generation).

Install

Add Maven Central, restrict the build to arm64-v8a, and turn on legacy packaging so the engines’ native libraries are extracted to disk. The API secret reaches your code through BuildConfig.

// settings.gradle.kts
dependencyResolutionManagement {
    repositories {
        google()
        mavenCentral()
    }
}

// app/build.gradle.kts
android {
    buildFeatures { buildConfig = true }
    defaultConfig {
        minSdk = 26   // 29 for Essence 2
        ndk { abiFilters += "arm64-v8a" }
        buildConfigField(
            "String", "BITHUMAN_API_SECRET",
            "\"${providers.gradleProperty("bithumanApiSecret").getOrElse("")}\"",
        )
    }
    packaging { jniLibs { useLegacyPackaging = true } }   // required
}
dependencies {
    implementation("ai.bithuman:expression2-android:0.5.2")
    // or: implementation("ai.bithuman:essence2-android:0.8.1")
}

Put the secret in ~/.gradle/gradle.properties as bithumanApiSecret=…, outside your source tree.

expression2-android brings the Qualcomm accelerator runtime with it (com.qualcomm.qti:qnn-litert-delegate:2.49.0 and com.qualcomm.qti:qnn-runtime:2.49.0). To keep the APK small and render on the CPU instead, exclude it:

implementation("ai.bithuman:expression2-android:0.5.2") {
    exclude(group = "com.qualcomm.qti")
}

Authenticate

Pass your API secret (create one) in code before you download or create an avatar: Expression2Credential.set(secret) for Expression 2, Essence2Credential.set(secret) for Essence 2. That one call covers the download and the session. Expression2Metering.apiSecret and Essence2Metering.apiSecret still work but are deprecated. See Your API secret.

Credits pay for active session time, talking or idle, billed to the second (pricing).

Warning: a buildConfigField compiles the secret into the APK, where anyone with the file can read it. Use it for local builds only.

Every copy of a shipped app carries its secret, so treat it as exposed: fetch it from your backend at startup, give each app its own secret, and rotate it if usage looks wrong (What a shipped app holds).

First frame

Expression 2, with the published wise-pup avatar (agent code A23WJF0199). Call render off the main thread.

import ai.bithuman.expression2.Expression2Avatar
import ai.bithuman.expression2.Expression2Credential
import ai.bithuman.expression2.Expression2ModelStore
import ai.bithuman.expression2.Expression2Options
import android.content.Context
import android.graphics.Bitmap

/** [pcm16k] is 16 kHz mono float32 in [-1, 1]. */
fun render(context: Context, pcm16k: FloatArray, show: (Bitmap) -> Unit) {
    Expression2Credential.set(BuildConfig.BITHUMAN_API_SECRET)        // before fetch() and create()
    val model = Expression2ModelStore(context).fetch("A23WJF0199")     // ~160 MB, first run only

    Expression2Avatar.create(context, model, Expression2Options()).use { avatar ->
        val frame = avatar.newFrameBitmap()   // 416 x 720, allocate once
        avatar.feed(pcm16k)
        avatar.flushTail()                    // end of the utterance
        while (true) {
            if (avatar.pull(frame) != null) { show(frame); continue }
            if (!avatar.hasPendingTail && avatar.queuedFrames == 0) break
            Thread.sleep(10)                  // null means "not ready yet"
        }
    }
}

Expected: show receives 20 frames for each second of audio, and the avatar’s lips follow the speech. The first create() after install prepares the accelerator once and takes noticeably longer than later launches, which reuse it. Create once, at app start, on a background thread.

Essence 2 takes 16-bit little-endian PCM bytes, as a 16 kHz mono WAV stores them, and fills an RGBA ByteBuffer sized from the identity:

import ai.bithuman.essence2.Essence2Avatar
import ai.bithuman.essence2.Essence2Credential
import ai.bithuman.essence2.Essence2ModelStore
import android.content.Context
import java.nio.ByteBuffer

fun render(context: Context, pcm16le: ByteArray, show: (ByteBuffer, Int, Int) -> Unit) {
    Essence2Credential.set(BuildConfig.BITHUMAN_API_SECRET)                   // before fetch() and create()
    val identity = Essence2ModelStore(context).fetch("A52DHS2219")            // sofia-ramirez; 226–281 MB, first run only

    Essence2Avatar.create(identity.dir).use { avatar ->
        val frame = avatar.newFrameBuffer()   // width * height * 4, RGBA
        avatar.feed(pcm16le)
        avatar.endOfAudio()
        var idle = 0
        while (idle < 100) {                  // 1 s with no frame = drained
            if (avatar.pull(frame)) { show(frame, avatar.width, avatar.height); idle = 0; continue }
            idle++
            Thread.sleep(10)
        }
        avatar.checkRender()                  // throws Essence2RenderFailed if the engine stopped
        // a refused session throws Essence2MeteringRefused from pull() or idle()
    }
}

Frame size belongs to the identity (portrait 1080×1920, landscape 1920×1080 or 1280×720). Read avatar.width and avatar.height; do not hard-code them.

Complete example

Two apps you can clone and run on a phone, each with idle motion between replies:

• Android Expression 2: the wise-pup sample avatar.
• Android Essence 2: the sofia-ramirez sample avatar at full resolution.

Integrate into your app

In a live conversation, keep one avatar open and stream into it.

Resample 24 kHz speech (OpenAI Realtime’s) to 16 kHz, and close the avatar when the app leaves the screen: Companion app.

After your API secret is accepted, a network loss does not stop the session for 5 minutes of rendered video. After that, render calls throw a retryable exception until the connection returns. Usage is reported to your account when it does.

For a Flutter app, the Flutter plugin wraps these engines.

Platform notes

• Release builds: isMinifyEnabled = true needs nothing extra. Both AARs ship their own keep rules.

• Two models in one app: essence2-android needs minSdk 29. Raise the app to 29, or put each model in its own module.

• Threads: download and create() on a background thread. The first download is the size shown above, into app-private storage.

• Check the version you resolved: Gradle keeps an exact version, so read it back when behaviour differs from this page.

  ./gradlew :app:dependencies --configuration releaseRuntimeClasspath | grep ai.bithuman

• Private avatars: an avatar you created downloads with the secret you set with Expression2Credential.set or Essence2Credential.set; there is nothing else to pass.

Performance

Troubleshooting

Reference

• Android API reference: every public class in both AARs.
• Examples: Expression 2 · Essence 2, complete apps you can clone.
• Flutter: the Flutter plugin, built on these engines.
• Changelog and Downloads & versions.
• FFmpeg in essence2-android is LGPL: relink materials.