iOS & iPadOS

One Swift package carries both models. The avatar renders inside your app on the iPhone or iPad: you feed 16 kHz mono speech in and take lip-synced frames out, with no render server. The same package builds Mac apps.

Before you start

• Xcode 26 or newer and an Apple Developer team.
• A physical iPhone or iPad for device builds. Essence 2 does not run in the Simulator; Expression 2 does.
• Essence 1 is not available on phones or in the Swift package: use Essence 2 or Expression 2 on devices (First generation).

Install

.package(url: "https://github.com/bithuman-product/homebrew-bithuman.git", from: "2.18.0")
// then attach the product your target uses:
//   .product(name: "Expression2", package: "homebrew-bithuman")
//   .product(name: "Essence2Kit", package: "homebrew-bithuman")
//   .product(name: "Essence2", package: "homebrew-bithuman")

Authenticate

First frame

Download the wise-pup sample avatar, the shared Expression 2 engine and a 16 kHz speech clip. No account is needed for these downloads:

curl -fL -o A23WJF0199.imx "https://api.bithuman.ai/v1/agent/A23WJF0199/model/download?model=expression-2"
curl -fLO "https://github.com/bithuman-product/homebrew-bithuman/releases/download/expression2-engine-mac-arm64-1.0.0/mac-arm64-1.0.0.engine"
curl -fL -o speech16k.wav "https://api.bithuman.ai/v1/agent/A23WJF0199/model/download?model=expression-2&member=demo_speech_16k.wav"

Add them to your app, then render. The mac engine file is the right one for iPhone apps too. For Essence 2, download the avatar in the app with Essence2Download (below).

// excerpt: inside your app. samples is the clip as [Float], 16 kHz mono;
// show(_:_:_:) draws B, G, R bytes; the URLs point at the files above.
import Expression2
import AVFoundation

Expression2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"] ?? "")
let engine = try Expression2Engine.create(
    avatarContainer: avatarURL,              // A23WJF0199.imx
    sharedEngineContainer: sharedEngineURL,  // mac-arm64-1.0.0.engine
    stagingDir: stagingURL)                  // any writable directory; keep it between launches

let audio = AVAudioEngine(), player = AVAudioPlayerNode()         // your app's audio output
let format = AVAudioFormat(standardFormatWithSampleRate: 16000, channels: 1)!
audio.attach(player); audio.connect(player, to: audio.mainMixerNode, format: format); try audio.start()
let reply = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(samples.count))!
reply.frameLength = reply.frameCapacity
samples.withUnsafeBufferPointer { reply.floatChannelData![0].update(from: $0.baseAddress!, count: samples.count) }

// seconds of the current reply your player has played (nil before it starts); frames() reads it
// off the main actor, so the player goes in a Sendable box
final class PlayedSeconds: @unchecked Sendable {
    let node: AVAudioPlayerNode
    init(_ node: AVAudioPlayerNode) { self.node = node }
    func callAsFunction() -> Double? {
        guard let t = node.lastRenderTime, let pt = node.playerTime(forNodeTime: t) else { return nil }
        return Double(pt.sampleTime) / pt.sampleRate
    }
}
let played = PlayedSeconds(player)

engine.feed(samples)   // [Float], 16 kHz mono
engine.flushTail()     // end of the reply
for await frame in engine.frames(audioClock: { played() }) {
    show(frame.bgr, frame.width, frame.height)                        // B, G, R bytes
    if frame.audioTime == 0 { player.scheduleBuffer(reply); player.play() }   // the reply's first frame: start its audio
    if frame.endsReply { break }                                      // the reply is over; idle frames follow
}

// excerpt: inside your app. samples is the clip as [Float], 16 kHz mono;
// show(_:_:_:) draws B, G, R bytes.
import Essence2Kit
import AVFoundation

Essence2Credential.set(ProcessInfo.processInfo.environment["BITHUMAN_API_SECRET"] ?? "")
let imxURL = try await Essence2Download.identity(agentCode: "A52DHS2219")   // sofia-ramirez
let engine = try await Essence2Engine.create(identity: imxURL)            // waits until the engine is ready

let audio = AVAudioEngine(), player = AVAudioPlayerNode()         // your app's audio output
let format = AVAudioFormat(standardFormatWithSampleRate: 16000, channels: 1)!
audio.attach(player); audio.connect(player, to: audio.mainMixerNode, format: format); try audio.start()
let reply = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(samples.count))!
reply.frameLength = reply.frameCapacity
samples.withUnsafeBufferPointer { reply.floatChannelData![0].update(from: $0.baseAddress!, count: samples.count) }

engine.feed(samples)                                              // [Float], 16 kHz mono
engine.flushTail()                                                // that is the whole reply
for await frame in engine.frames(following: player) {
    show(frame.bgr, frame.width, frame.height)                    // B, G, R bytes, width * height * 3
    if frame.audioTime == 0 {                                     // the reply's first speech frame:
        player.stop(); player.scheduleBuffer(reply); player.play()  // start its audio now
    }
    if frame.endsReply { break }                                  // the reply is over; idle frames follow
}
engine.shutdown()

The C interface for C, C++ and plugins (Essence2) is on the Swift reference.

Complete example

Two SwiftUI apps you can clone and run on an iPhone or iPad, each with a microphone button, idle motion and interruption:

• iOS Expression 2: the wise-pup sample avatar.
• iOS Essence 2: a photoreal Essence 2 avatar at full resolution.

Integrate into your app

For file rendering, set engine.pacing = .unpaced: Essence 2 then hands out frames as fast as it renders them. If your audio does not go through an AVAudioPlayerNode, pass your own clock: frames(audioClock: { secondsOfThisReplyPlayed }).

Resample 24 kHz speech (OpenAI Realtime’s) to 16 kHz, and close the avatar when the app leaves the screen: Companion app.

Download an avatar in the app

Your app can download an avatar file itself, with the secret you set in Authenticate:

let avatarURL = try await Expression2Download.avatar(agentCode: "A23WJF0199")   // Expression 2
let imxURL = try await Essence2Download.identity(agentCode: "A52DHS2219")      // Essence 2

Both return a local file to pass to create. They download the Apple build of the avatar, which is smaller than the full file, and refuse a file whose sha256 does not match. Files are kept in the app’s Caches directory under their sha256, so a second call for the same avatar downloads nothing. Pass directory: to keep them somewhere else. The shared Expression 2 engine file is not an avatar; download it from the release as in First frame.

Platform notes

• Your own MLX: Essence 2 contains no MLX. Link your own mlx-swift (MLX, MLXNN) in the same target, also with -ObjC or -all_load; nothing to embed.

• Simulator: simulator slices are arm64 only; pass ARCHS=arm64. Essence 2 does not run in the Simulator (be_essence2_create returns -2); Expression 2 does.

• Privacy strings: add NSMicrophoneUsageDescription to hear the user.

• Check the version you resolved. SwiftPM keeps what Package.resolved holds, so run swift package update after you raise from:, then read it back:

  grep -A3 homebrew-bithuman Package.resolved   # "version" must be the one on Downloads & versions

Performance

Troubleshooting

Reference

• Swift reference: every Swift and C entry point.
• Examples: iOS Expression 2 · iOS Essence 2 · macOS Expression 2.
• Sample avatars: Ready-made avatars. Your own agent’s model: GET /v1/agent/{code}/model/download with your API secret.
• Changelog and Downloads & versions.