Android Essence 2

Play with sound (8 seconds, captions)

The app downloads the sofia-ramirez sample avatar once, renders every frame of a speech clip on the phone, then plays the audio with the frames in sync. Tap the screen to replay.

Requirements

Get the code

git clone https://github.com/bithuman-product/bithuman-examples.git
cd bithuman-examples/android/essence2-hello

Set your API secret

Write local.properties next to settings.gradle.kts. The file is git-ignored.

sdk.dir=/path/to/your/Android/sdk
bithuman.apiSecret=<your API secret>

BITHUMAN_API_SECRET in the environment works instead of the second line.

Run it

./gradlew :app:assembleDebug && adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n com.example.e2hello/.MainActivity

The first launch renders the 16 kHz clip bundled in the app. To use your own audio, push any 16-bit PCM WAV (the app converts it to 16 kHz mono) and restart:

adb push my.wav /storage/emulated/0/Android/data/com.example.e2hello/files/speech.wav && adb shell am start -S -n com.example.e2hello/.MainActivity

Expected output

adb logcat -s E2HELLO shows the download, then:

engine: 1080x1920 targetFrames=251
rendered 347 frames in 13 s — playing…
347 frames, 13.87 s — tap to replay

347 frames for 13.87 seconds of audio is 25 fps at the avatar’s own 1080×1920. The first run downloads about 240 MB; later runs start at the engine.

How it works

MainActivity.kt does four things, all from ai.bithuman:essence2-android:

1. sets your API secret once with Essence2Credential.set(BuildConfig.BITHUMAN_API_SECRET), which covers the download and the session;
2. downloads the avatar with Essence2ModelStore(this).fetch(agentCode) (A52DHS2219 by default);
3. opens it with Essence2Avatar.create(identity.dir) on a background thread;
4. calls feed(pcm) and endOfAudio(), then pull(frame) until every frame is out: one frame per 40 ms of audio.

The calls and the live-streaming loop are on Android.

The code that matters

The render path of MainActivity.kt, as it is in the repository: the credential, the download, then feed and pull until the utterance is out.

// excerpt: android/essence2-hello/app/src/main/java/com/example/e2hello/MainActivity.kt
// 1. The CREDENTIAL, once, before anything downloads or opens an engine.
//    One setter covers the store's download and the engine's meter;
//    create() refuses without it.
Essence2Credential.set(secret)
// …
// 2. The STORE downloads with the credential set above.
//    Blocks on the network the first time; that is why this is a worker thread.
val store = Essence2ModelStore(this)
val identity = store.fetch(agentCode, progress = { member, done, total ->
    if (done == total) Log.i(TAG, "fetched $member ($total B)")
})
// …
Essence2Avatar.create(identity.dir).use { avatar ->
    // …
    val frame = avatar.newFrameBuffer()   // direct, width * height * 4, RGBA
    // …
    avatar.feed(pcm)                      // 16-bit little-endian PCM bytes, as read
    avatar.endOfAudio()                   // "that is the whole utterance"
    // …
    // pull() returns false until frames are ready, so poll. Stop after 5 s
    // with no new frame — but give the FIRST frame longer (30 s): a cold
    // first render on a slower phone can take more than a few seconds, and
    // giving up before it arrives is a blank screen, not a result.
    var quietMs = 0
    while (quietMs < (if (out.isEmpty()) 30_000 else 5_000)) {
        frame.clear()
        if (avatar.pull(frame)) {         // true = a frame was written
    // …
            quietMs = 0
            frame.rewind()
    // …
    // A render that failed must not reach you as silence: checkRender()
    // turns a zero-frame render into a throw that names the reason.
    avatar.checkRender()

The complete file is on GitHub.

Make it your own

• Your own avatar: create one with the Agents API with "model": "essence-2", then pass its agent code to fetch; the secret you set with Essence2Credential.set downloads it.

• Live speech: feed 16 kHz mono 16-bit audio as it arrives and pull frames at 25 fps; call endOfAudio() at the end of each reply and idle(buffer) between replies. For the microphone, declare and request RECORD_AUDIO first (Build an Android app).

• Ship it: the secret in BuildConfig can be read out of the APK. A real app fetches it from your backend at startup and passes it to Essence2Credential.set.

• Zero-copy frames (essence2-android 0.7.0 and newer): useHardwareBuffers() switches delivery to zero-copy: pullHardwareBuffer() and idleHardwareBuffer() return an Essence2HardwareFrame whose RGBA HardwareBuffer your renderer samples directly. Close each frame after presenting it. pull(ByteBuffer) is unchanged, and nothing changes until you call useHardwareBuffers() (Android API).

  val avatar = Essence2Avatar.create(bundleDir)
  avatar.useHardwareBuffers()            // once, before the first frame
  avatar.feed(pcm)
  avatar.pullHardwareBuffer()?.use { frame ->
      renderer.draw(frame.buffer)        // e.g. an EGLImage / Vulkan import, or Bitmap.wrapHardwareBuffer
  }

Troubleshooting

More on Android: Troubleshooting.

Next

• Android example: Expression 2 · Android SDK · Android API reference · source on GitHub