Put an animated character in an iPhone app

7 steps

1. Run the example first

   Clone the example and download the wise-pup avatar, the shared engine and a 16 kHz speech clip. The download is anonymous:

   git clone https://github.com/bithuman-product/bithuman-examples.git
   cd bithuman-examples/swift/ios-expression2
   ./setup.sh

   Add BITHUMAN_API_SECRET under Product → Scheme → Edit Scheme → Run → Environment Variables, then open the project, pick your team under Signing & Capabilities and press Run:

   open IOSExpression2.xcodeproj

   Expected

   The avatar appears and idles. Tap Speak: it says the sample line with its lips in sync. The first launch prepares the engine on the device and takes a few seconds longer than later launches.

   Done: step 1

2. Add the Swift package to your app

   In Xcode choose File → Add Package Dependencies…, paste https://github.com/bithuman-product/homebrew-bithuman.git and attach the Expression2 product to your target. The current version pin and the Package.swift line are on iOS & iPadOS: Install.

   Expected

   Your app builds with import Expression2.

   Done: step 2

3. Set your API secret

   The engine checks an API secret when a session starts. Set it before you create the engine:

   Expression2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"] ?? "")

   Set BITHUMAN_API_SECRET in the scheme’s environment while you build. Keep the secret out of the app bundle you ship (What a shipped app holds).

   Expected

   No refusal when the engine opens in the next step.

   Done: step 3

4. Add the character’s files

   Download the wise-pup sample avatar, the shared Expression 2 engine and a 16 kHz speech clip, and add them to your app. No account is needed for these downloads:

   curl -fL -o A23WJF0199.imx "https://api.bithuman.ai/v1/agent/A23WJF0199/model/download?model=expression-2"
   curl -fLO "https://github.com/bithuman-product/homebrew-bithuman/releases/download/expression2-engine-mac-arm64-1.0.0/mac-arm64-1.0.0.engine"
   curl -fL -o speech16k.wav "https://api.bithuman.ai/v1/agent/A23WJF0199/model/download?model=expression-2&member=demo_speech_16k.wav"

   The mac engine file is the right one for iPhone apps too.

   Expected

   Three files in your app: the avatar (A23WJF0199.imx), the shared engine and the speech clip.

   Done: step 4

5. Feed it speech and draw the frames

   The example keeps the engine in an actor. create opens the two files as they are, feed takes 16 kHz mono float audio, flushTail() ends a reply, and pull() returns the next frame, or nil until a chunk of frames is ready:

   // excerpt: swift/ios-expression2/Sources/App.swift
   actor Renderer {
       private var engine: Expression2Engine?
           // …
           let e = try Expression2Engine.create(avatarContainer: avatar,
                                                sharedEngineContainer: sharedEngine,
                                                stagingDir: staging)
           engine = e
       // …
       func idleFrame() -> [UInt8]? { engine?.idle }
       func feed(_ samples: [Float]) { engine?.feed(samples) }
       func flushTail() { engine?.flushTail() }
       func reset() { engine?.resetState(clearFrames: true) }
       // …
       func pullOne() -> [UInt8]? { engine?.pull()?.frame }
       func queued() -> Int { engine?.queuedFrames ?? 0 }
   }

   Feed the speech in chunks and take frames out as they appear, so the app feeds and drains at the same time:

   // excerpt: swift/ios-expression2/Sources/App.swift
   var i = 0
   while i < pcm.count {
       let j = min(i + chunk, pcm.count)
       await renderer.feed(Array(pcm[i..<j]))
       i = j
       // Generation is ASYNCHRONOUS — pull() returns nil until a chunk of
       // frames lands, so this usually takes nothing on the first passes and
       // then keeps up. It is not a busy-wait: it only removes what is ready.
       while let f = await renderer.pullOne() {
           if let cg = makeCGImage(f, w, h) { frames.append(cg) }
       }
   }
   await renderer.flushTail()

   To play the reply’s audio in step with the lips, start it on the reply’s first frame (audioTime == 0): the full loop is in iOS & iPadOS: First frame.

   Expected

   The character idles, then says the clip with its lips in sync, rendered in your app.

   Done: step 5

6. Use your own character

   Create an Expression 2 avatar from one portrait (Turn a drawing, mascot or pet photo into a talking character), then download it with your agent code in place of A23WJF0199. In the example, run BITHUMAN_API_SECRET=… ./setup.sh <AGENT_CODE>.

   Expected

   Your character in place of wise-pup, with the same calls.

   Done: step 6

7. Run it on a device

   Choose your iPhone or iPad as the run destination. Expression 2 also runs in the iOS Simulator, which is fine while you build; Essence 2, the photoreal model, needs a physical device. How fast each device renders is on Performance.

   Expected

   The character idles and speaks on the device.

   Done: step 7

All steps done. Next: make it your own.