Build an Android app
Stream audio into an Android avatar and show its frames in your app.
Integrate into your app
In a live conversation, keep one avatar open and stream into it.
| Job | Expression 2 | Essence 2 |
|---|---|---|
| Audio in | 16 kHz mono FloatArray, −1 to 1 | 16 kHz mono 16-bit little-endian PCM ByteArray |
| Stream audio as it arrives | feed(chunk) per chunk | feed(chunk) per chunk |
| Show frames | pull(bitmap), 20 a second | pull(buffer), 25 a second |
| End of a reply | flushTail() | endOfAudio() |
| Idle between replies | avatar. | idle(buffer) |
| Interrupt the reply | resetState(true) | resetAudio() |
| Check the session | Expression2Exception from create or pull | Essence2MeteringRefused from pull/idle; checkRender() throws Essence2RenderFailed if the engine stopped |
| Close it | close() | close() |
To feed the microphone, declare <uses-permission android:name="android.permission.RECORD_AUDIO"/> in your manifest and ask for it at runtime (ActivityResultContracts.RequestPermission) before you start AudioRecord; without it the buffers are silent. The SDKs need only INTERNET, which their AARs merge in for you.
Resample 24 kHz speech (OpenAI Realtime’s) to 16 kHz, and close the avatar when the app leaves the screen: Companion app.
After your API secret is accepted, a network loss does not stop the session for 5 minutes of rendered video. After that, render calls throw a retryable exception until the connection returns. Usage is reported to your account when it does.
For a Flutter app, the Flutter plugin wraps these engines.
Complete example
Two apps you can clone and run on a phone, each with idle motion between replies:
- Android Expression 2: the
wise-pupsample avatar. - Android Essence 2: the
sofia-ramirezsample avatar at full resolution.
Platform notes
-
Release builds:
isMinifyEnabled = trueneeds nothing extra. Both AARs ship their own keep rules. -
Two models in one app:
essence2-androidneedsminSdk 29. Raise the app to 29, or put each model in its own module. -
Threads: download and
create()on a background thread. The first download is the size shown above, into app-private storage. -
Check the version you resolved: Gradle keeps an exact version, so read it back when behaviour differs from this page.
./gradlew :app:dependencies --configuration releaseRuntimeClasspath | grep ai.bithuman -
Private avatars: an avatar you created downloads with the secret you set with
Expression2Credential.setorEssence2Credential.set; there is nothing else to pass. -
Known issue, heat while idle (Essence 2,
essence2-android0.10.x): an app that feeds silence to keep the avatar moving between replies heats the phone during long idle periods. With a full-body avatar on a Galaxy Z Flip5 (0.10.3), the thermal status reached critical after about 8 minutes. Frames and audio stay correct. Stop feeding while the avatar is silent, and loop a short idle clip you rendered once:// excerpt: idle without feeding silence (Essence 2) // Once, after create(): render about 3 s of idle frames (25 a second) and keep them. val idleFrames = ArrayList<ByteBuffer>() while (idleFrames.size < 75) { val buf = avatar.newFrameBuffer() if (avatar.idle(buf)) idleFrames += buf } // Every frame tick: feed() and pull() only while there is speech to play. // While the avatar is silent, feed nothing and draw the cached clip in a loop. var idleIndex = 0 fun frameToDraw(speaking: Boolean, live: ByteBuffer): ByteBuffer? = if (speaking) live.takeIf { avatar.pull(it) } else idleFrames[idleIndex++ % idleFrames.size]A fix is coming in a future release.
Reference
- Android API reference: every public class in both AARs.
- Examples: Expression 2 · Essence 2, complete apps you can clone.
- Flutter: the Flutter plugin, built on these engines.
- Changelog and Downloads & versions.
- FFmpeg in
essence2-androidis LGPL: relink materials.