iOS & iPadOS

The avatar renders inside your app, with no render server. Why on the device.

Quick startOpen the sample app

Before you start

You feed 16 kHz mono speech in and take lip-synced frames out. The same package builds Mac apps.

• Xcode 26 or newer. An Apple Developer team is needed for device builds only; the Simulator needs none.
• A paid plan (Creator or higher): billing: How billing works; pricing.
• A physical iPhone or iPad for device builds. Essence 2 does not run in the Simulator: use a physical device. Expression 2 does run in the Simulator.
• Essence 1 is not available on phones or in the Swift package: use Essence 2 or Expression 2 on devices (First generation).

Install

One bitHuman SDK per iOS app, for now. The current Flutter plugin and Swift package are built on different engine versions, so an iOS app that includes both may fail to link. Use only one until their versions match.

.package(url: "https://gitlab.com/bithuman/sdk/homebrew-bithuman", from: "3.0.0")
// then attach the product your target uses:
//   .product(name: "Expression2", package: "homebrew-bithuman")
//   .product(name: "Essence2Kit", package: "homebrew-bithuman")
//   .product(name: "Essence2", package: "homebrew-bithuman")

The package lives in the homebrew-bithuman repository, so its package identity is homebrew-bithuman; the name is expected. The Simulator slices are arm64 only: for a generic/platform=iOS Simulator or other command-line build, set EXCLUDED_ARCHS[sdk=iphonesimulator*] = x86_64 in your target’s build settings.

Authenticate

Run your first avatar

The quickest way to see it work is the example app: clone bithuman-examples, open the iOS Expression 2 example in Xcode, set BITHUMAN_API_SECRET in the scheme and run it on your iPhone. The code below is the core of that app.

Download the wise-pup sample avatar (agent code A23WJF0199), the shared Expression 2 engine and a 16 kHz speech clip. No account is needed for these downloads:

curl -fL -o wise-pup.imx "https://api.bithuman.ai/v1/agent/A23WJF0199/model/download?model=expression-2"
curl -fLO "https://downloads.bithuman.ai/homebrew-bithuman/expression2-engine-mac-arm64-1.0.0/mac-arm64-1.0.0.engine"
curl -fL -o speech16k.wav "https://api.bithuman.ai/v1/agent/A23WJF0199/model/download?model=expression-2&member=demo_speech_16k.wav"

Add them to your app, then render. The mac engine file is the right one for iPhone apps too. For Essence 2, download the avatar in the app with Essence2Download (Swift: Integrate into your app).

// excerpt: inside your app. samples is the clip as [Float], 16 kHz mono;
// show(_:_:_:) draws B, G, R bytes; the URLs point at the files above.
import Expression2
import AVFoundation

Expression2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"] ?? "")
let engine = try Expression2Engine.create(
    avatarContainer: avatarURL,              // wise-pup.imx
    sharedEngineContainer: sharedEngineURL,  // mac-arm64-1.0.0.engine
    stagingDir: stagingURL)                  // any writable directory; keep it between launches

let audio = AVAudioEngine(), player = AVAudioPlayerNode()         // your app's audio output
let format = AVAudioFormat(standardFormatWithSampleRate: 16000, channels: 1)!
audio.attach(player); audio.connect(player, to: audio.mainMixerNode, format: format); try audio.start()
let reply = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(samples.count))!
reply.frameLength = reply.frameCapacity
samples.withUnsafeBufferPointer { reply.floatChannelData![0].update(from: $0.baseAddress!, count: samples.count) }

// seconds of the current reply your player has played (nil before it starts); frames() reads it
// off the main actor, so the player goes in a Sendable box
final class PlayedSeconds: @unchecked Sendable {
    let node: AVAudioPlayerNode
    init(_ node: AVAudioPlayerNode) { self.node = node }
    func callAsFunction() -> Double? {
        guard let t = node.lastRenderTime, let pt = node.playerTime(forNodeTime: t) else { return nil }
        return Double(pt.sampleTime) / pt.sampleRate
    }
}
let played = PlayedSeconds(player)

engine.feed(samples)   // [Float], 16 kHz mono
engine.flushTail()     // end of the reply
for await frame in engine.frames(audioClock: { played() }) {
    show(frame.bgr, frame.width, frame.height)                        // B, G, R bytes
    if frame.audioTime == 0 { player.scheduleBuffer(reply, completionHandler: nil); player.play() }   // the reply's first frame: start its audio
    if frame.endsReply { break }                                      // the reply is over; idle frames follow
}

// excerpt: inside your app. samples is the clip as [Float], 16 kHz mono;
// show(_:_:_:) draws B, G, R bytes.
import Essence2Kit
import AVFoundation

Essence2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"] ?? "")
let imxURL = try await Essence2Download.identity(agentCode: "A52DHS2219")   // sofia-ramirez
let engine = try await Essence2Engine.create(identity: imxURL)            // waits until the engine is ready

let audio = AVAudioEngine(), player = AVAudioPlayerNode()         // your app's audio output
let format = AVAudioFormat(standardFormatWithSampleRate: 16000, channels: 1)!
audio.attach(player); audio.connect(player, to: audio.mainMixerNode, format: format); try audio.start()
let reply = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(samples.count))!
reply.frameLength = reply.frameCapacity
samples.withUnsafeBufferPointer { reply.floatChannelData![0].update(from: $0.baseAddress!, count: samples.count) }

engine.feed(samples)                                              // [Float], 16 kHz mono
engine.flushTail()                                                // that is the whole reply
for await frame in engine.frames(following: player) {
    show(frame.bgr, frame.width, frame.height)                    // B, G, R bytes, width * height * 3
    if frame.audioTime == 0 {                                     // the reply's first speech frame:
        player.stop(); player.scheduleBuffer(reply, completionHandler: nil); player.play()  // start its audio now
    }
    if frame.endsReply { break }                                  // the reply is over; idle frames follow
}
engine.shutdown()

The loop on its own, without the audio setup, as it reads in your app. show, play and played are your app’s own: show draws the B, G, R frame, play(reply) starts the reply’s audio (player.scheduleBuffer(reply, completionHandler: nil) then player.play() in the example above) and played() is the clock from PlayedSeconds. You bring the conversation.

.package(url: "https://gitlab.com/bithuman/sdk/homebrew-bithuman",
         from: "3.0.0")

let engine = try Expression2Engine.create(
    avatarContainer: avatarURL,
    sharedEngineContainer: sharedEngineURL,
    stagingDir: stagingURL)

engine.feed(samples)
engine.flushTail()
for await frame in engine.frames(audioClock: { played() }) {
    show(frame.bgr, frame.width, frame.height)
    if frame.audioTime == 0 { play(reply) }
    if frame.endsReply { break }
}

Build for the Simulator from the command line. A Simulator build is unsigned, so switch signing off. From the example’s folder:

xcodebuild -project IOSExpression2.xcodeproj -scheme IOSExpression2 -destination 'generic/platform=iOS Simulator' ARCHS=arm64 CODE_SIGNING_ALLOWED=NO build

In your own project, keep ARCHS=arm64 (or the EXCLUDED_ARCHS setting from Install) for the Simulator, and pass CODE_SIGNING_ALLOWED=NO the same way.

Give the example your API secret without Xcode. Debug builds of the iOS Expression 2 example read the secret from a file instead of the scheme, so it never appears in a command line: put it in the app’s Documents/bithuman.secret.

On a device: xcrun devicectl device copy to --device <udid> --domain-type appDataContainer --domain-identifier ai.bithuman.example.ios-expression2 --source <your secret file> --destination Documents/bithuman.secret. In the Simulator, copy the file into $(xcrun simctl get_app_container booted ai.bithuman.example.ios-expression2 data)/Documents/.

The app reads the file once at launch, passes it to Expression2Credential.set(_:) and deletes it; Release builds ignore it. Never commit the file or paste the secret into a command line; a shipped app gets its secret from your backend (your API secret).

You bring the replies. The Swift package renders the speech you give it; it does not write a reply. A character that answers needs your own conversation backend (speech to text, a language model and a voice) or a key for one: you bring the conversation.

The C interface for C, C++ and plugins (Essence2) is on the Swift reference.

Performance