Give an app's voice assistant a face

6 steps

The code in these steps is from Companion app and Android: Run your first avatar.

1. Choose where the face renders

   Where	Use	The conversation
   In your app, on the device	the Swift package (iPhone, iPad, Mac) or the Android SDK (arm64)	stays with your app and the services it uses
   In your own Python code, on a Mac or a Linux machine	the Python SDK: render takes 16 kHz mono audio	stays with your code and the services it uses
   In the bitHuman cloud or a browser tab	the web embed or a LiveKit agent with a bitHuman cloud avatar	runs on bitHuman’s servers with the web embed

   This page follows the first row. Android, and Essence 2 on iPhone and iPad, need a physical device, not an emulator or the Simulator.

   Expected

   One SDK: the Swift package or the Android SDK.

   Done: step 1

2. Pick the avatar

   Use the wise-pup sample (Expression 2, agent code A23WJF0199) for a character, or the sofia-ramirez sample (Essence 2, agent code A52DHS2219) for a photoreal person, while you build. Or create your own from one portrait.

   Expected

   An agent code.

   Done: step 2

3. Add the SDK and set your API secret

   • iPhone and iPad: the Swift package, product Expression2 or Essence2Kit (install).
   • Android: the Android SDK, expression2-android or essence2-android (install).

   One call covers the avatar’s download and the session. Set it before anything downloads:

   SwiftKotlin

   Swift

   Expression2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"] ?? "")

   Kotlin

   Expression2Credential.set(BuildConfig.BITHUMAN_API_SECRET)        // before fetch() and create()

   On Android, BuildConfig.BITHUMAN_API_SECRET comes from your app’s build.gradle.kts (set BITHUMAN_API_SECRET in the environment before you build, or read local.properties as on the Android page):

   android {
       buildFeatures { buildConfig = true }
       defaultConfig {
           buildConfigField("String", "BITHUMAN_API_SECRET", "\"${System.getenv("BITHUMAN_API_SECRET") ?: ""}\"")
       }
   }

   That field compiles the secret into the APK, so use it for local builds only. Keep the secret out of the app bundle you ship: fetch it from your backend at runtime (What a shipped app holds).

   Expected

   No refusal when the avatar opens. Without a secret, opening it fails and says why.

   Done: step 3

4. Resample speech to 16 kHz

   The engines take 16 kHz mono speech. A voice service that returns another rate needs one conversion first; a service that returns 24 kHz 16-bit PCM, for example, needs a 24 kHz to 16 kHz converter. Keep one converter for the whole reply and pass each chunk through it before you feed the avatar:

   SwiftKotlin

   Swift

   // Add to your app: 24 kHz 16-bit mono PCM in, the 16 kHz [Float] that feed(_:) takes out.
   import AVFoundation
   
   final class To16k {
       private let from = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 24_000, channels: 1, interleaved: false)!
       private let to = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 16_000, channels: 1, interleaved: false)!
       private lazy var converter = AVAudioConverter(from: from, to: to)!
   
       func convert(_ pcm24k: Data) -> [Float] {
           let n = AVAudioFrameCount(pcm24k.count / 2)
           guard n > 0, let input = AVAudioPCMBuffer(pcmFormat: from, frameCapacity: n),
                 let output = AVAudioPCMBuffer(pcmFormat: to, frameCapacity: n) else { return [] }
           input.frameLength = n
           pcm24k.withUnsafeBytes { input.int16ChannelData![0].update(from: $0.bindMemory(to: Int16.self).baseAddress!, count: Int(n)) }
           var given = false
           _ = converter.convert(to: output, error: nil) { _, status in
               if given { status.pointee = .noDataNow; return nil }
               given = true
               status.pointee = .haveData
               return input
           }
           return Array(UnsafeBufferPointer(start: output.floatChannelData![0], count: Int(output.frameLength)))
       }
   }

   Kotlin

   // Add to your app: 24 kHz 16-bit mono PCM in; floats() for Expression 2, pcm16() for Essence 2.
   import java.nio.ByteBuffer
   import java.nio.ByteOrder
   
   class To16k {
       private var rest = FloatArray(0)   // input not used yet
       private var pos = 0.0              // read position in the input, in samples
   
       fun floats(pcm24k: ByteArray): FloatArray {
           val s = ByteBuffer.wrap(pcm24k).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer()
           val input = rest + FloatArray(s.remaining()) { s.get() / 32768f }
           val out = ArrayList<Float>()
           while (pos + 1 < input.size) {
               val i = pos.toInt()
               val f = (pos - i).toFloat()
               out.add(input[i] * (1 - f) + input[i + 1] * f)
               pos += 1.5                   // 24 000 / 16 000
           }
           rest = input.copyOfRange(pos.toInt(), input.size)
           pos -= pos.toInt()
           return out.toFloatArray()
       }
   
       fun pcm16(pcm24k: ByteArray): ByteArray {
           val f = floats(pcm24k)
           val b = ByteBuffer.allocate(f.size * 2).order(ByteOrder.LITTLE_ENDIAN)
           for (x in f) b.putShort((x.coerceIn(-1f, 1f) * 32767).toInt().toShort())
           return b.array()
       }
   }

   Expected

   Each 24 kHz chunk comes out as two thirds as many 16 kHz samples, and the lips keep pace with the voice. Speech fed at the wrong rate makes the mouth run slow and long.

   Done: step 4

5. Feed the reply and draw the frames

   Your speech recognition hears the user, your language model writes the reply, and your voice turns it into audio. Feed that audio as 16 kHz mono, say where the reply ends, and draw the frames the avatar hands back. On Android, with Expression 2 and the wise-pup sample (call render off the main thread):

   import ai.bithuman.expression2.Expression2Avatar
   import ai.bithuman.expression2.Expression2Credential
   import ai.bithuman.expression2.Expression2ModelStore
   import ai.bithuman.expression2.Expression2Options
   import android.content.Context
   import android.graphics.Bitmap
   
   /** [pcm16k] is 16 kHz mono float32 in [-1, 1]. */
   fun render(context: Context, pcm16k: FloatArray, show: (Bitmap) -> Unit) {
       Expression2Credential.set(BuildConfig.BITHUMAN_API_SECRET)        // before fetch() and create()
       val model = Expression2ModelStore(context).fetch("A23WJF0199")     // ~160 MB, first run only
   
       Expression2Avatar.create(context, model, Expression2Options()).use { avatar ->
           val frame = avatar.newFrameBitmap()   // 416 x 720, allocate once
           avatar.feed(pcm16k)
           avatar.flushTail()                    // end of the utterance
           while (true) {
               if (avatar.pull(frame) != null) { show(frame); continue }
               if (!avatar.hasPendingTail && avatar.queuedFrames == 0) break
               Thread.sleep(10)                  // null means "not ready yet"
           }
       }
   }

   Pass To16k().floats(chunk) as pcm16k. In Swift, feed(samples) and flushTail() do the same, and the reply’s audio starts on its first frame (audioTime == 0): the complete loop is in iOS & iPadOS: First frame, and an Essence 2 version is in Companion app: Speak a reply.

   Expected

   The lips follow the reply's audio from its first word, and idle motion returns when the reply ends.

   Done: step 5

6. Let the user interrupt

   When your speech recognition hears the user start talking over the avatar, stop your audio player and drop the rest of the reply: interrupt() in Swift; in Kotlin, resetState(true) for Expression 2 or resetAudio() for Essence 2 (Android: Integrate into your app). Idle motion continues from the current frame.

   Expected

   The mouth stops with the voice, and the next reply starts cleanly.

   Done: step 6

All steps done. Next: make it your own.