iOS Essence 2

A SwiftUI app that opens an Essence 2 avatar, shows its idle motion, and speaks a line with the lips in sync, all rendered on the phone at the avatar’s own resolution (up to 1080p). Speak plays the line again.

Sources/App.swift uses Essence2Kit (Apple): Essence2Engine.create(identity:resourcesDirectory:) opens the bundled avatar, and frames(following:) paces the picture to the audio player.

Requirements

Get the code

git clone https://github.com/bithuman-product/bithuman-examples.git
cd bithuman-examples/swift/ios-essence2
./setup.sh

setup.sh downloads the default avatar into Sources/Model/ and the engine resources into Sources/EngineResources/ (both git-ignored), checks both, and makes a speech clip. ./setup.sh <AGENT_CODE> fetches another avatar.

Set your API secret

In Xcode: Product → Scheme → Edit Scheme → Run → Environment Variables, add BITHUMAN_API_SECRET. The app reads it at launch and passes it to Essence2Credential.set(_:).

Run it

open IOSEssence2.xcodeproj

Pick your team under Signing & Capabilities, choose your iPhone as the run destination, and press Run.

Expected output

The avatar appears in its idle motion, then says the bundled line once. The console prints:

[ios-essence2] engine ready: 1080x1920, ready in <n> s

The first launch unpacks the avatar and prepares the engine, so it is slower than later launches.

How it works

1. Credential first: Essence2Credential.set(_:) is called before the engine is created.
2. Open the avatar: Essence2Engine.create(identity:resourcesDirectory:) opens the bundled .imx with the engine’s runtime files from Sources/EngineResources.
3. One draw loop: frames(following: player) hands out 25 frames a second for the life of the app: idle motion between replies, and a reply’s frames as the player plays their audio.
4. Speak: feed(_:) the reply’s 16 kHz samples, then flushTail(); the draw loop starts the reply’s audio with its first speech frame, so the lips stay on the voice.
5. Stream, don’t collect: one 1080×1920 frame is 6.2 MB, so the app draws each frame as it arrives.

Sources/App.swift is the whole app. The full API is on Apple and Apple API reference.

The code that matters

Sources/App.swift sets the secret, opens the avatar, and draws the frames frames(following:) hands out, starting the reply’s audio with its first speech frame:

// excerpt: swift/ios-essence2/Sources/App.swift
Essence2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"])
// …
let e = try await Essence2Engine.create(identity: imx, resourcesDirectory: Payload.resources)

// excerpt: swift/ios-essence2/Sources/App.swift
/// 5a. The draw loop, for the life of the app. `frames(following:)` paces itself
/// (25 per second): idle motion between replies, and a reply's frames as the player
/// plays their audio. Start the reply's audio with its first speech frame.
private func startLoop(_ e: Essence2Engine) {
    loop?.cancel()
    let frames = e.frames(following: player)
    loop = Task { [weak self] in
        for await f in frames {
            guard let self else { return }
            if f.audioTime == 0, let r = self.reply {
                self.player.stop()
                self.player.scheduleBuffer(r)
                self.player.play()
            }
            if let cg = makeCGImage(bgr: f.bgr, f.width, f.height) {
                self.sink.show(cg)
                self.hasFrame = true
            }
    // …
    e.feed(samples)
    e.flushTail()

The complete file is on GitHub.

Make it your own

• Another sample avatar: ./setup.sh <AGENT_CODE> with any of these; the app reads the frame size back from the engine.

  Avatar	Agent code	Frame
  warm-clear-professional-presenter (default)	A21SKT4314	1080×1920
  sofia-ramirez	A52DHS2219	1080×1920
  kwame-warm-museum-guide	A62SJB3901	1080×1920
  afro-latina-astrophysics-mentor	A23KSG5258	1920×1080
  calm-product-specialist-advisor	A24EKJ8433	1280×720

• Your own avatar: create one with the Agents API ("model": "essence-2"), then run BITHUMAN_API_SECRET=… ./setup.sh <AGENT_CODE>.

• Ship it: fetch the secret from your backend or the Keychain at launch and pass it to Essence2Credential.set(_:); never put it in the app bundle.

Troubleshooting

More on Apple: Troubleshooting.

Next

• iOS example: Expression 2 · Android example: Essence 2 · iOS & iPadOS · source on GitHub