Swift reference

Covers the Swift package at the version on Downloads & versions. How to use these calls in an app is on iOS & iPadOS and macOS.

Products

All products ship ios-arm64, ios-arm64-simulator (arm64 only) and macos-arm64.

Expression 2 (Swift)

import Expression2

Expression2Credential.set(_ secret: String?)   // before create; else BITHUMAN_API_SECRET; nil clears it

static func Expression2Engine.create(
    avatarContainer: URL,             // the .imx you downloaded
    sharedEngineContainer: URL? = nil, // the shared .engine file
    sharedEngineDir: URL? = nil,
    stagingDir: URL,
    warmSpeech: [Float]? = nil) throws -> Expression2Engine

static func Expression2Engine.create(   // for containers you already unpacked
    modelPath: URL,
    sharedEngineDir: URL? = nil,
    warmSpeech: [Float]? = nil) throws -> Expression2Engine

func feed(_ samples: [Float])                   // 16 kHz mono PCM
func flushTail()                                // end of an utterance
func pull() -> (frame: [UInt8], speech: Bool)?  // BGR, width * height * 3; nil until a frame is ready
var idle: [UInt8]?                              // the next idle frame
func idle(into buffer: inout [UInt8]) -> Int    // the next idle frame into your buffer; bytes written
func resetState(clearFrames: Bool = true)       // interrupt: drop queued audio and frames
func shutdown()                                 // waits for the last usage report

// Since Swift package 2.18.0: one stream of frames to show, idle between replies, 20 fps.
func frames(audioClock: (@Sendable () -> Double?)? = nil) -> AsyncStream<Expression2Frame>  // your clock: seconds of the reply played
func nextFrame(audioClock: (@Sendable () -> Double?)? = nil) async -> Expression2Frame?     // the next frame when it is due; nil after shutdown
func events() -> AsyncStream<Expression2Event>  // .replyStarted, .replyEnded (once per reply)
func interrupt()                                // drop queued audio and frames (resetState)
var droppedFrames: Int                          // frames skipped to keep a reply on its audio's timeline
static let framesPerSecond: Double              // 20

struct Expression2Frame {
    let bgr: [UInt8]; let width: Int; let height: Int   // B, G, R bytes, width * height * 3
    let isSpeech: Bool                                   // false: idle motion between replies
    let endsReply: Bool                                  // the first frame after the reply's audio ran out
    let index: Int
    let audioTime: Double?                               // seconds into the reply's audio; 0 = its first frame, start the audio
}
var isReady: Bool
var hasPendingTail: Bool
var queuedFrames: Int
var meteringRefusal: String?
let width: Int, height: Int

static func Expression2Download.avatar(   // download an avatar file; sha256-checked, cached
    agentCode: String,
    directory: URL? = nil) async throws -> URL   // nil: Caches/bitHuman/expression2/avatars

create throws Expression2LoadError.meteringRefused(reason:) when the API secret is missing or rejected, or when the service cannot be reached at the start; meteringRefusal carries the message. pull() never blocks; poll it. Expression2Download.avatar throws Expression2Download.Failure when the download is refused, fails, or does not match its sha256.

Essence 2 (Swift)

import Essence2Kit

Essence2Credential.set(_ secret: String?)   // before create; else BITHUMAN_API_SECRET; nil clears it

static func Essence2Engine.create(
    identity: URL,                  // the Essence 2 .imx you downloaded
    resourcesDirectory: URL? = nil, // nil: fetch the runtime files once into Application Support
    readyTimeout: Double = 300) async throws -> Essence2Engine

func feed(_ samples: [Float])               // 16 kHz mono PCM; never blocks
func flushTail()                            // the reply's audio is complete; it ends now, not after 0.6 s of silence
func frames(following player: AVAudioPlayerNode) -> AsyncStream<Essence2Frame>  // speech frames on the player's audio
func frames(audioClock: (@Sendable () -> Double?)? = nil) -> AsyncStream<Essence2Frame>  // your clock: seconds of the reply played
                                            // every frame, 25 per second, until shutdown or cancellation
func events() -> AsyncStream<Essence2Event> // .replyStarted, .replyEnded (once per reply)
func nextFrame(audioClock:) async -> Essence2Frame?   // waits for the next frame; nil after shutdown
func pullFrame(audioClock:) -> Essence2Frame?         // the next frame if one is due; nil means keep the current one
func pull() -> (frame: [UInt8], speech: Bool)?   // the same, as bytes + speech flag
func idle(into out: inout [UInt8]) -> Int   // the next frame into your buffer; 0 means keep the current one
var pacing: Essence2Pacing                  // .realtime (default, 25 fps) or .unpaced (offline rendering)
var droppedFrames: Int                      // frames skipped to keep a reply on its audio's timeline
static let framesPerSecond: Double          // 25
func interrupt()                            // drop queued audio and frames
func shutdown()                             // flush the last usage report, release the engine
static func quiesceAll(timeoutMs: Int32 = 5000)   // at app exit
var width: Int
var height: Int
var isReady: Bool
var meteringRefusal: String?                // the service's refusal while this session is refused
var runtimeFailure: String?                 // set when the engine stopped
var pendingSamples: Int                     // fed audio the engine has not taken yet

struct Essence2Frame {
    let bgr: [UInt8]                        // B, G, R, width * height * 3
    let width: Int, height: Int
    let isSpeech: Bool                      // the mouth is driven by your audio
    let endsReply: Bool                     // the first frame after a reply (once per reply)
    let index: Int                          // frames handed out before this one
    let audioTime: Double?                  // speech frames: seconds into the reply's audio (0, 0.04, …)
}

Essence2Resources.ensure() async throws -> URL   // the runtime files, fetched and sha256-checked
Essence2Resources.releaseTag                     // the release they come from

static func Essence2Download.identity(   // download an avatar file; sha256-checked, cached
    agentCode: String,
    directory: URL? = nil) async throws -> URL   // nil: Caches/bitHuman/essence2/avatars

Every way of taking frames (frames, nextFrame, pullFrame, pull, idle(into:)) draws from the same engine and hands out at most 25 frames a second. A reply’s first speech frame anchors its timeline: frame k is due k/25 s later, and a frame that would be shown a full frame late is skipped, so a reply never drifts from its audio.

create throws Essence2KitError.meteringRefused(reason:) when the API secret is missing or rejected, or when the service cannot be reached at the start. It throws .identityUnreadable for a file the engine cannot open, .resourcesUnavailable when the runtime files cannot be fetched or fail their checksum, and .notReady after readyTimeout. Essence2Download.identity throws .resourcesUnavailable when the download is refused, fails, or does not match its sha256.

Essence 2 (C)

Audio is 16 kHz mono int16. Frames are packed height * width * 3 bytes in B, G, R order. Nothing blocks except be_essence2_quiesce_all.

The C library does not download its runtime files. Before be_essence2_create, put w2v_ess_fp16_v1.onnx, audio_encoder_fp16_window_trunk.onnx and audio_encoder_fp16_window_head.onnx from the essence2-v1.15.0 release at the root of your app bundle (in Xcode, add them as a group, not a folder reference), or next to the .imx. Without them the engine never becomes ready. In Swift, Essence2Kit fetches and checks these files for you.

A minimal loop

be_essence2_handle h;
be_essence2_set_api_secret(secret);                    // or Essence2Credential.set in Swift
if (be_essence2_create(imx_path, NULL, 0, &h) != 0) { /* -3: no secret, rejected, or no network */ }
while (!be_essence2_is_ready(h)) { /* show be_essence2_idle_frame() meanwhile */ }
int32_t w, hgt; be_essence2_get_info(h, &w, &hgt);     // frame is w * hgt * 3 BGR bytes
int32_t fed = 0, spoke = 0;
for (;;) {                                             // once per display tick, 25 a second
    while (fed < count) {                              // push 0.2 s at a time
        int32_t n = count - fed < 3200 ? count - fed : 3200;
        if (be_essence2_push_audio(h, pcm16k_int16 + fed, n) != 0) break;   // -2: ring full, retry next tick
        fed += n;
        if (fed == count) be_essence2_end_utterance(h);   // the reply's audio is complete
    }
    int32_t got = be_essence2_pull_frame(h, buf, w * hgt * 3);   // bytes; 0 none yet; -3 refused
    if (got < 0) break;
    if (got > 0) {
        show(buf, got);
        int32_t kind = be_essence2_last_frame_kind(h);
        if (kind == BE_ESSENCE2_FRAME_SPEECH) spoke = 1;
        else if (spoke && kind == BE_ESSENCE2_FRAME_IDLE) break;   // idle after speech: the reply is over
    }
    usleep(40000);
}
be_essence2_destroy(h);

Credentials

Lifecycle

Audio and frames

Display and status

Sessions and billing

A session checks the API secret when it starts and reports its session time, talking or idle. If the network drops after the secret is accepted, frames continue for 5 minutes of rendered video, then pull_frame and idle_frame return -3 (Essence 2) or pull() returns nil (Expression 2) until the connection returns. A secret rejected mid-session is final: destroy the engine. Prices are on pricing.