Swift SDK

On-device, real-time, lip-synced avatars for iOS, iPadOS, and macOS. Apple Silicon only. Preview maturity.

Overview

On Apple platforms, bitHuman ships as bitHumanKit — a single SwiftPM package that drops a real-time voice agent, with an optional lip-synced avatar, into your Mac, iPad, or iPhone app. The umbrella framework re-exports both on-device engines:

Expression — animates any portrait image at runtime (speech encoder → animator → face decoder on the GPU + Apple Neural Engine). Home of VoiceChat / VoiceChatConfig / AvatarConfig.
Essence — the portable libessence C++ runtime that renders a pre-built .imx avatar (audio in, composed BGR frames out). Reached via Bithuman.create(modelPath:).

Audio in (16 kHz mono PCM), CGImage / BGR frames out at 25 FPS. All inference runs on-device; a once-per-minute billing heartbeat meters avatar mode (audio-only is unmetered).

Maturity This rail is preview, not GA. The published package vends two products: bitHumanKit (import bitHumanKit), the binary umbrella that bundles everything, and BithumanEngineProtocol, a source-only Layer-0 engine interface. The standalone Layer-1 engine products (Expression, Bithuman) are not published — naming one fails at resolve time with product 'Expression' ... not found in package 'homebrew-bithuman'. Attach the umbrella.

Second-generation models are not on this rail. The published package carries no expression-2 or essence-2 engine: neither is a SwiftPM product, and neither is bundled inside bitHumanKit (checked against the bitHumanKit.xcframework attached to release v2.4.0 — the binary this package resolves to). essence-2-max is cloud-only by design. To reach any second-generation model from an Apple app today, call the REST API or join a LiveKit session; the on-device rail in this package is first-generation only.

Install

In Xcode: File → Add Package Dependencies… → paste the package URL:

https://github.com/bithuman-product/homebrew-bithuman.git

Pick 2.4.0 (“Up to Next Major Version” from 2.4.0) and attach the bitHumanKit product to your target. Or in Package.swift:

.package(url: "https://github.com/bithuman-product/homebrew-bithuman.git",
         from: "2.4.0")   // 2.4.0 is the only tag that carries a Package.swift

Do not pin 0.8.x here. This repo has no 0.8.2 tag, and no v0.x tag carries a Package.swift — those tags hold Homebrew formula files. Resolving from: "0.8.1" fails with error: the package manifest at '/Package.swift' cannot be accessed. The 0.8.x numbers belong to the retired bithuman-sdk-public repo, archived when the SwiftPM distribution moved here; v2.4.0 is the only release carrying bitHumanKit.xcframework.zip.

The package wraps a pre-compiled bitHumanKit.xcframework; every third-party dependency (MLX, HuggingFace, Tokenizers, …) is statically linked, so consumers have zero transitive Swift Package dependencies. Just import bitHumanKit.

Auth: export BITHUMAN_API_KEY or set VoiceChatConfig.apiKey before starting avatar mode. Get a key at Developer → API Keys. Audio-only voice runs keyless and unmetered.

Note The Swift SDK reads BITHUMAN_API_KEY; every other surface (Python, CLI, REST API) reads BITHUMAN_API_SECRET. Same value, two names — export both if you move between rails.

Quick start: voice agent

The highest-level surface is VoiceChat — STT, LLM, and TTS all on-device. No API key needed without an avatar:

import bitHumanKit

var config = VoiceChatConfig()
config.localeIdentifier = "en-US"
config.systemPrompt = "You are a helpful assistant. One sentence per turn."

let chat = VoiceChat(config: config)
try await chat.start()
// Speak into the mic. The agent listens, thinks, and replies aloud.

Add the lip-synced avatar by pointing the config at the Expression weights and a portrait, and supplying your key:

import bitHumanKit

let weights = try await ExpressionWeights.ensureAvailable()  // ~1.6 GB, cached

var config = VoiceChatConfig()
config.avatar = AvatarConfig(modelPath: weights, portraitPath: portraitURL)
config.apiKey = ProcessInfo.processInfo.environment["BITHUMAN_API_KEY"]

let chat = VoiceChat(config: config)
try await chat.start()   // throws .missingAPIKey / .authenticationFailed

The Essence runtime

For a pre-built .imx avatar (branded characters, 720p+, lowest credit rate), drive the runtime directly — push PCM in, drain frames out:

import bitHumanKit
import CoreGraphics

let result = try Bithuman.create(modelPath: modelURL)
let runtime = result.bithuman        // result.staticIdleImage is the rest pose
try await runtime.start()

// Push audio as it arrives — 24 kHz for playback, 16 kHz for the encoder.
try await runtime.pushAudio(audio24k: samples24, audio16k: samples16)

// Drain rendered chunks; each carries its frames and the audio they pair with.
while let chunk = runtime.tryDequeueChunk() {
    let frames: [CGImage] = chunk.frames   // 25 FPS
    // hand the frames to your view layer
}

await runtime.interrupt()            // at end-of-utterance
await runtime.shutdown()

This is the Apple expression of the audio-streaming push/drain loop. The entry point is Bithuman.create — there is no createRuntime on the published module. Verified to compile against bitHumanKit 2.4.0 with Xcode 26.5.

Permissions + entitlements

Info.plist (all platforms):

<key>NSMicrophoneUsageDescription</key><string>Talk to your assistant.</string>
<key>NSSpeechRecognitionUsageDescription</key><string>Recognise what you say.</string>

Without these, mic / speech start fails silently (the OS denies and remembers). Sandboxed Mac apps also need com.apple.security.device.audio-input in .entitlements.

Warning The iOS increased-memory entitlement is mandatory. Without it, iOS kills your app mid-conversation (~30 s into a turn) when memory exceeds the default ~3 GB ceiling. Request approval before development — Apple takes 1–3 business days.
<key>com.apple.developer.kernel.increased-memory-limit</key><true/>
<key>com.apple.developer.kernel.extended-virtual-addressing</key><true/>
Request at developer.apple.com → Account → Membership → Request Additional Capabilities.

Audio-only keyless mode

On-device voice chat (no lip-synced avatar) needs no API key — STT, LLM, and TTS all run locally and audio-only mode is unmetered. You only need a key (and the billing heartbeat fires) once you add the lip-synced avatar.

Hardware floor

Gate this at runtime — on under-spec devices, guide people to a friendly fallback rather than a half-loaded engine. Use HardwareCheck.evaluate() to branch your SwiftUI root and show your own UnsupportedDeviceView for .unsupported(reason).

	Essence	Expression
macOS	M3+, macOS 26	M3+, macOS 26
iPadOS	iPad Pro M4+, iPadOS 26	iPad Pro M4+, 16 GB, iPadOS 26
iPhone	iPhone 16 Pro+ (A18 Pro)	iPhone 16 Pro+ (A18 Pro) — preview, on-device validation in progress

Requires Xcode 26+ (older Xcodes reject the Swift 6 concurrency syntax). Expression on Apple Silicon auto-spawns a bithuman-expression-daemon subprocess; on unsupported hardware it raises ExpressionModelNotSupported — not a crash. See models.

Performance

Measured on an M5 MacBook Pro (libessence 1.19.1, single conversation):

Metric	Value
Per-tick mean	1.43 ms
Per-tick p99	1.51 ms
Sustained (tight loop)	698 FPS
Cold start	~290 ms
Peak RSS	~84 MB
Wrapper overhead vs raw libessence	+1.7 %

Comfortable headroom over the 25 FPS / 40 ms tick budget.

Troubleshooting

Mic / speech start fails silently

Missing Info.plist privacy strings — the OS denies mic / speech and caches the denial for the session.

App killed ~30 s into a conversation (iOS)

Missing the increased-memory-limit entitlement. See the warning above — it must be approved by Apple before it takes effect.

Avatar disappears on re-render

When hosting AvatarRendererView in SwiftUI, return the same renderer view instance from both makeXxxView and updateXxxView. SwiftUI rebuilds the parent constantly; a fresh renderer each time means a vanishing avatar.

Under-spec device shows a friendly fallback

Working as intended. Branch on HardwareCheck.evaluate().