Swift reference

Covers the Swift package at the version on Downloads & versions. How to use these calls in an app is on iOS & iPadOS and macOS.

Products

All products ship ios-arm64, ios-arm64-simulator (arm64 only) and macos-arm64.

Expression 2 (Swift)

import Expression2

Expression2Credential.set(_ secret: String?)   // before create; else BITHUMAN_API_SECRET; nil clears it

static func Expression2Engine.create(
    avatarContainer: URL,             // the .imx you downloaded
    sharedEngineContainer: URL? = nil, // the shared .engine file
    sharedEngineDir: URL? = nil,
    stagingDir: URL,
    warmSpeech: [Float]? = nil) throws -> Expression2Engine

static func Expression2Engine.create(   // for containers you already unpacked
    modelPath: URL,
    sharedEngineDir: URL? = nil,
    warmSpeech: [Float]? = nil) throws -> Expression2Engine

func feed(_ samples: [Float])                   // 16 kHz mono PCM
func flushTail()                                // end of an utterance
func pull() -> (frame: [UInt8], speech: Bool)?  // BGR, width * height * 3; nil until a frame is ready
var idle: [UInt8]?                              // the next idle frame
func idle(into buffer: inout [UInt8]) -> Int    // the next idle frame into your buffer; bytes written
func resetState(clearFrames: Bool = true)       // interrupt: drop queued audio and frames
func shutdown()                                 // waits for the last usage report

// Since Swift package 2.18.0: one stream of frames to show, idle between replies, 20 fps.
func frames(audioClock: (@Sendable () -> Double?)? = nil) -> AsyncStream<Expression2Frame>  // your clock: seconds of the reply played; ends at shutdown or cancellation
func nextFrame(audioClock: (@Sendable () -> Double?)? = nil) async -> Expression2Frame?     // the next frame when it is due; nil after shutdown
func events() -> AsyncStream<Expression2Event>  // .replyStarted, .replyEnded (once per reply); ends at shutdown
func interrupt()                                // drop queued audio and frames (resetState)
var droppedFrames: Int                          // frames skipped to keep a reply on its audio's timeline
static let framesPerSecond: Double              // 20

struct Expression2Frame {
    let bgr: [UInt8]; let width: Int; let height: Int   // B, G, R bytes, width * height * 3
    let isSpeech: Bool                                   // false: idle motion between replies
    let endsReply: Bool                                  // the first frame after the reply's audio ran out
    let index: Int
    let audioTime: Double?                               // seconds into the reply's audio; 0 = its first frame, start the audio
}
var isReady: Bool
var hasPendingTail: Bool
var queuedFrames: Int
var meteringRefusal: String?
let width: Int, height: Int

static func Expression2Download.avatar(   // download an avatar file; sha256-checked, cached
    agentCode: String,
    directory: URL? = nil) async throws -> URL   // nil: Caches/bitHuman/expression2/avatars
static var Expression2Download.revalidateInBackground: Bool   // default true

create throws Expression2LoadError.meteringRefused(reason:) when the API secret is missing or rejected, or when the service cannot be reached at the start; meteringRefusal carries the message. pull() never blocks; poll it. Expression2Download.avatar throws Expression2Download.Failure when the download is refused, fails, or does not match its sha256.

Essence 2 (Swift)

import Essence2Kit

Essence2Credential.set(_ secret: String?)   // before create; else BITHUMAN_API_SECRET; nil clears it

static func Essence2Engine.create(
    identity: URL,                  // the Essence 2 .imx you downloaded
    resourcesDirectory: URL? = nil, // nil: fetch the runtime files once into Application Support
    readyTimeout: Double = 300) async throws -> Essence2Engine

func feed(_ samples: [Float])               // 16 kHz mono PCM; never blocks
func flushTail()                            // the reply's audio is complete; it ends now, not after 0.6 s of silence
func frames(following player: AVAudioPlayerNode) -> AsyncStream<Essence2Frame>  // speech frames on the player's audio
func frames(audioClock: (@Sendable () -> Double?)? = nil) -> AsyncStream<Essence2Frame>  // your clock: seconds of the reply played
                                            // every frame, 25 per second, until shutdown or cancellation
func events() -> AsyncStream<Essence2Event> // .replyStarted, .replyEnded (once per reply)
func nextFrame(audioClock:) async -> Essence2Frame?   // waits for the next frame; nil after shutdown
func pullFrame(audioClock:) -> Essence2Frame?         // the next frame if one is due; nil means keep the current one
func pull() -> (frame: [UInt8], speech: Bool)?   // the same, as bytes + speech flag
func idle(into out: inout [UInt8]) -> Int   // the next frame into your buffer; 0 means keep the current one
var pacing: Essence2Pacing                  // .realtime (default, 25 fps) or .unpaced (offline rendering)
var droppedFrames: Int                      // frames skipped to keep a reply on its audio's timeline
static let framesPerSecond: Double          // 25
func interrupt()                            // drop queued audio and frames
func shutdown()                             // flush the last usage report, release the engine
static func quiesceAll(timeoutMs: Int32 = 5000)   // at app exit
var width: Int
var height: Int
var isReady: Bool
var meteringRefusal: String?                // the service's refusal while this session is refused
var runtimeFailure: String?                 // set when the engine stopped
var pendingSamples: Int                     // fed audio the engine has not taken yet

struct Essence2Frame {
    let bgr: [UInt8]                        // B, G, R, width * height * 3
    let width: Int, height: Int
    let isSpeech: Bool                      // the mouth is driven by your audio
    let endsReply: Bool                     // the first frame after a reply (once per reply)
    let index: Int                          // frames handed out before this one
    let audioTime: Double?                  // speech frames: seconds into the reply's audio (0, 0.04, …)
}

Essence2Resources.ensure() async throws -> URL   // the runtime files, fetched and sha256-checked
Essence2Resources.releaseTag                     // the release they come from

static func Essence2Download.identity(   // download an avatar file; sha256-checked, cached
    agentCode: String,
    directory: URL? = nil) async throws -> URL   // nil: Caches/bitHuman/essence2/avatars
static var Essence2Download.revalidateInBackground: Bool   // default true

Essence2Download.identity and Expression2Download.avatar return an avatar already on the device at once, after checking its sha256, and look for a newer file in the background; the next call returns it. With a copy on the device, a call succeeds even when the download service cannot be reached; without one it fails. Set revalidateInBackground = false to check before returning.

Every way of taking frames (frames, nextFrame, pullFrame, pull, idle(into:)) draws from the same engine and hands out at most 25 frames a second. A reply’s first speech frame anchors its timeline: frame k is due k/25 s later, and a frame that would be shown a full frame late is skipped, so a reply never drifts from its audio.

create throws Essence2KitError.meteringRefused(reason:) when the API secret is missing or rejected, or when the service cannot be reached at the start. It throws .identityUnreadable for a file the engine cannot open, .resourcesUnavailable when the runtime files cannot be fetched or fail their checksum, and .notReady after readyTimeout. Essence2Download.identity throws .resourcesUnavailable when the download is refused, fails, or does not match its sha256.

create throws .identityOutdated(path:agentCode:reason:) for an avatar file published before the renderer its engine carries. It downloads a file that Essence2Download fetched once more by itself; download any other file again.

Essence 2 (C)

Audio is 16 kHz mono int16. Frames are packed height * width * 3 bytes in B, G, R order. Nothing blocks except be_essence2_quiesce_all.

The C library does not download its runtime files. Before be_essence2_create, put w2v_ess_fp16_v1.onnx, audio_encoder_fp16_window_trunk.onnx and audio_encoder_fp16_window_head.onnx from the essence2-v1.15.5 release at the root of your app bundle (in Xcode, add them as a group, not a folder reference), or next to the .imx. Without them the engine never becomes ready. In Swift, Essence2Kit fetches and checks these files for you.

A minimal loop

be_essence2_handle h;
be_essence2_set_api_secret(secret);                    // or Essence2Credential.set in Swift
if (be_essence2_create(imx_path, NULL, 0, &h) != 0) { /* -3: no secret, rejected, or no network */ }
while (!be_essence2_is_ready(h)) { /* show be_essence2_idle_frame() meanwhile */ }
int32_t w, hgt; be_essence2_get_info(h, &w, &hgt);     // frame is w * hgt * 3 BGR bytes
int32_t fed = 0, spoke = 0;
for (;;) {                                             // once per display tick, 25 a second
    while (fed < count) {                              // push 0.2 s at a time
        int32_t n = count - fed < 3200 ? count - fed : 3200;
        if (be_essence2_push_audio(h, pcm16k_int16 + fed, n) != 0) break;   // -2: ring full, retry next tick
        fed += n;
        if (fed == count) be_essence2_end_utterance(h);   // the reply's audio is complete
    }
    int32_t got = be_essence2_pull_frame(h, buf, w * hgt * 3);   // bytes; 0 none yet; -3 refused
    if (got < 0) break;
    if (got > 0) {
        show(buf, got);
        int32_t kind = be_essence2_last_frame_kind(h);
        if (kind == BE_ESSENCE2_FRAME_SPEECH) spoke = 1;
        else if (spoke && kind == BE_ESSENCE2_FRAME_IDLE) break;   // idle after speech: the reply is over
    }
    usleep(40000);
}
be_essence2_destroy(h);

Recommended: gate each frame’s audio on its display (lowest latency). If your presenter is voice-clocked, open the voice ~200 ms after the first frame to avoid gaps after a barge-in. On a device that renders below real time, also call be_essence2_set_playout_position with the samples you have played, so late frames are skipped instead of freezing the face.

Credentials

Lifecycle

Audio and frames

Display and status

Storage and privacy

Downloaded avatars stay in Caches/bitHuman/… unless you pass a directory, and Essence 2’s runtime files in Application Support/bitHuman/essence2/<release>. Everything the SDK downloads (Essence2Download, Expression2Download and Essence 2’s runtime files) is excluded from iCloud and computer backups (isExcludedFromBackup), because it is downloaded again when needed. A directory you pass in is not flagged as a whole; only the files the SDK writes into it are.

The package ships an App Store privacy manifest (PrivacyInfo.xcprivacy) in the resource bundles of Essence2Kit and of every Expression 2 and Essence 2 product, so Xcode’s privacy report includes it. It declares:

• Required-reason APIs: file timestamps (C617.1, the SDK’s own files in your app’s container) and system boot time (35F9.1, timing inside your app).
• Data the SDK sends: an install identifier the SDK creates and the session’s duration, sent to bitHuman to bill your account. Used for app functionality only, not linked to the user, and not used for tracking.

A cached avatar is handed only to an API secret the download service has allowed to open it. A different secret on the same device is checked with the service first, so its first open needs the network.

Sessions and billing

A session checks the API secret when it starts and reports its session time, talking or idle. If the network drops after the secret is accepted, frames continue for 5 minutes of rendered video, then pull_frame and idle_frame return -3 (Essence 2) or pull() returns nil (Expression 2) until the connection returns. A secret rejected mid-session is final: destroy the engine. Prices are on pricing.

The secret check names the avatar. If the secret’s account may not use that avatar, the session refuses to start with an error naming AGENT_NOT_ENTITLED, and nothing is billed. Use the API secret of the account that owns the avatar, or of a workspace it is shared with.

A session nobody watches ends. If your app takes no frames for 60 seconds (for example, while it is in the background), the session closes and is billed up to that point. The next frame you ask for starts a new session. Coming back within 60 seconds keeps the session.