# Swift reference

URL: https://docs.bithuman.ai/platforms/swift/reference

> Every entry point in the Swift package: the Expression2 and Essence2Kit Swift APIs, the Essence2 C interface, credentials and return codes.

Covers the Swift package at the version on [Downloads & versions](https://docs.bithuman.ai/downloads). How to use these calls in an app is on [iOS & iPadOS](https://docs.bithuman.ai/platforms/ios) and [macOS](https://docs.bithuman.ai/platforms/macos).

## Products

| Product | Import | What it is |
|---|---|---|
| `Expression2` | `Expression2` | the Expression 2 engine, Swift API |
| `Essence2Kit` | `Essence2Kit` | the Essence 2 engine, Swift API; includes `Essence2` |
| `Essence2` | `Essence2` (or `CLibEssence2`) | the Essence 2 engine, C interface |
| `BithumanEngineProtocol` | `BithumanEngineProtocol` | the shared engine protocol. `Expression2` already contains it; do not add both |

All products ship `ios-arm64`, `ios-arm64-simulator` (arm64 only) and `macos-arm64`.

## Expression 2 (Swift)

```swift
import Expression2

Expression2Credential.set(_ secret: String?)   // before create; else BITHUMAN_API_SECRET; nil clears it

static func Expression2Engine.create(
    avatarContainer: URL,             // the .imx you downloaded
    sharedEngineContainer: URL? = nil, // the shared .engine file
    sharedEngineDir: URL? = nil,
    stagingDir: URL,
    warmSpeech: [Float]? = nil) throws -> Expression2Engine

static func Expression2Engine.create(   // for containers you already unpacked
    modelPath: URL,
    sharedEngineDir: URL? = nil,
    warmSpeech: [Float]? = nil) throws -> Expression2Engine

func feed(_ samples: [Float])                   // 16 kHz mono PCM
func flushTail()                                // end of an utterance
func pull() -> (frame: [UInt8], speech: Bool)?  // BGR, width * height * 3; nil until a frame is ready
var idle: [UInt8]?                              // the next idle frame
func idle(into buffer: inout [UInt8]) -> Int    // the next idle frame into your buffer; bytes written
func resetState(clearFrames: Bool = true)       // interrupt: drop queued audio and frames
func shutdown()                                 // waits for the last usage report

// Since Swift package 2.18.0: one stream of frames to show, idle between replies, 20 fps.
func frames(audioClock: (@Sendable () -> Double?)? = nil) -> AsyncStream<Expression2Frame>  // your clock: seconds of the reply played
func nextFrame(audioClock: (@Sendable () -> Double?)? = nil) async -> Expression2Frame?     // the next frame when it is due; nil after shutdown
func events() -> AsyncStream<Expression2Event>  // .replyStarted, .replyEnded (once per reply)
func interrupt()                                // drop queued audio and frames (resetState)
var droppedFrames: Int                          // frames skipped to keep a reply on its audio's timeline
static let framesPerSecond: Double              // 20

struct Expression2Frame {
    let bgr: [UInt8]; let width: Int; let height: Int   // B, G, R bytes, width * height * 3
    let isSpeech: Bool                                   // false: idle motion between replies
    let endsReply: Bool                                  // the first frame after the reply's audio ran out
    let index: Int
    let audioTime: Double?                               // seconds into the reply's audio; 0 = its first frame, start the audio
}
var isReady: Bool
var hasPendingTail: Bool
var queuedFrames: Int
var meteringRefusal: String?
let width: Int, height: Int

static func Expression2Download.avatar(   // download an avatar file; sha256-checked, cached
    agentCode: String,
    directory: URL? = nil) async throws -> URL   // nil: Caches/bitHuman/expression2/avatars
```

`create` throws `Expression2LoadError.meteringRefused(reason:)` when the API secret is missing or rejected, or when the service cannot be reached at the start; `meteringRefusal` carries the message. `pull()` never blocks; poll it. `Expression2Download.avatar` throws `Expression2Download.Failure` when the download is refused, fails, or does not match its sha256.

## Essence 2 (Swift)

```swift
import Essence2Kit

Essence2Credential.set(_ secret: String?)   // before create; else BITHUMAN_API_SECRET; nil clears it

static func Essence2Engine.create(
    identity: URL,                  // the Essence 2 .imx you downloaded
    resourcesDirectory: URL? = nil, // nil: fetch the runtime files once into Application Support
    readyTimeout: Double = 300) async throws -> Essence2Engine

func feed(_ samples: [Float])               // 16 kHz mono PCM; never blocks
func flushTail()                            // the reply's audio is complete; it ends now, not after 0.6 s of silence
func frames(following player: AVAudioPlayerNode) -> AsyncStream<Essence2Frame>  // speech frames on the player's audio
func frames(audioClock: (@Sendable () -> Double?)? = nil) -> AsyncStream<Essence2Frame>  // your clock: seconds of the reply played
                                            // every frame, 25 per second, until shutdown or cancellation
func events() -> AsyncStream<Essence2Event> // .replyStarted, .replyEnded (once per reply)
func nextFrame(audioClock:) async -> Essence2Frame?   // waits for the next frame; nil after shutdown
func pullFrame(audioClock:) -> Essence2Frame?         // the next frame if one is due; nil means keep the current one
func pull() -> (frame: [UInt8], speech: Bool)?   // the same, as bytes + speech flag
func idle(into out: inout [UInt8]) -> Int   // the next frame into your buffer; 0 means keep the current one
var pacing: Essence2Pacing                  // .realtime (default, 25 fps) or .unpaced (offline rendering)
var droppedFrames: Int                      // frames skipped to keep a reply on its audio's timeline
static let framesPerSecond: Double          // 25
func interrupt()                            // drop queued audio and frames
func shutdown()                             // flush the last usage report, release the engine
static func quiesceAll(timeoutMs: Int32 = 5000)   // at app exit
var width: Int
var height: Int
var isReady: Bool
var meteringRefusal: String?                // the service's refusal while this session is refused
var runtimeFailure: String?                 // set when the engine stopped
var pendingSamples: Int                     // fed audio the engine has not taken yet

struct Essence2Frame {
    let bgr: [UInt8]                        // B, G, R, width * height * 3
    let width: Int, height: Int
    let isSpeech: Bool                      // the mouth is driven by your audio
    let endsReply: Bool                     // the first frame after a reply (once per reply)
    let index: Int                          // frames handed out before this one
    let audioTime: Double?                  // speech frames: seconds into the reply's audio (0, 0.04, …)
}

Essence2Resources.ensure() async throws -> URL   // the runtime files, fetched and sha256-checked
Essence2Resources.releaseTag                     // the release they come from

static func Essence2Download.identity(   // download an avatar file; sha256-checked, cached
    agentCode: String,
    directory: URL? = nil) async throws -> URL   // nil: Caches/bitHuman/essence2/avatars
```

Every way of taking frames (`frames`, `nextFrame`, `pullFrame`, `pull`, `idle(into:)`) draws from the same engine and hands out at most 25 frames a second. A reply's first speech frame anchors its timeline: frame *k* is due *k*/25 s later, and a frame that would be shown a full frame late is skipped, so a reply never drifts from its audio.

`create` throws `Essence2KitError.meteringRefused(reason:)` when the API secret is missing or rejected, or when the service cannot be reached at the start. It throws `.identityUnreadable` for a file the engine cannot open, `.resourcesUnavailable` when the runtime files cannot be fetched or fail their checksum, and `.notReady` after `readyTimeout`. `Essence2Download.identity` throws `.resourcesUnavailable` when the download is refused, fails, or does not match its sha256.

## Essence 2 (C)

Audio is 16 kHz mono `int16`. Frames are packed `height * width * 3` bytes in B, G, R order. Nothing blocks except `be_essence2_quiesce_all`.

The C library does not download its runtime files. Before `be_essence2_create`, put `w2v_ess_fp16_v1.onnx`, `audio_encoder_fp16_window_trunk.onnx` and `audio_encoder_fp16_window_head.onnx` from the [essence2-v1.15.0 release](https://github.com/bithuman-product/homebrew-bithuman/releases/tag/essence2-v1.15.0) at the root of your app bundle (in Xcode, add them as a group, not a folder reference), or next to the `.imx`. Without them the engine never becomes ready. In Swift, `Essence2Kit` fetches and checks these files for you.

### A minimal loop

```c
be_essence2_handle h;
be_essence2_set_api_secret(secret);                    // or Essence2Credential.set in Swift
if (be_essence2_create(imx_path, NULL, 0, &h) != 0) { /* -3: no secret, rejected, or no network */ }
while (!be_essence2_is_ready(h)) { /* show be_essence2_idle_frame() meanwhile */ }
int32_t w, hgt; be_essence2_get_info(h, &w, &hgt);     // frame is w * hgt * 3 BGR bytes
int32_t fed = 0, spoke = 0;
for (;;) {                                             // once per display tick, 25 a second
    while (fed < count) {                              // push 0.2 s at a time
        int32_t n = count - fed < 3200 ? count - fed : 3200;
        if (be_essence2_push_audio(h, pcm16k_int16 + fed, n) != 0) break;   // -2: ring full, retry next tick
        fed += n;
        if (fed == count) be_essence2_end_utterance(h);   // the reply's audio is complete
    }
    int32_t got = be_essence2_pull_frame(h, buf, w * hgt * 3);   // bytes; 0 none yet; -3 refused
    if (got < 0) break;
    if (got > 0) {
        show(buf, got);
        int32_t kind = be_essence2_last_frame_kind(h);
        if (kind == BE_ESSENCE2_FRAME_SPEECH) spoke = 1;
        else if (spoke && kind == BE_ESSENCE2_FRAME_IDLE) break;   // idle after speech: the reply is over
    }
    usleep(40000);
}
be_essence2_destroy(h);
```

### Credentials

| Function | Purpose |
|---|---|
| `int32_t be_essence2_set_api_secret(const char* secret)` | Sets the API secret for later sessions (`NULL` clears it). Without it the engine reads `BITHUMAN_API_SECRET`. Swift: `Essence2Credential.set(_:)` |

### Lifecycle

| Function | Purpose | Returns |
|---|---|---|
| `be_essence2_create(const char* imx_path, const char* motion_dir, int32_t chunk, be_essence2_handle* out)` | Opens an avatar. Pass `NULL` for `motion_dir` and `0` for `chunk` | `0`; `-1` bad argument; `-2` the file could not be opened; `-3` the session was refused (no secret, rejected secret, or no network at the start) |
| `be_essence2_is_ready(h)` | Warm-up finished; speech frames can flow | `1` or `0` |
| `be_essence2_destroy(h)` | Releases one engine; background work drains on its own | — |
| `be_essence2_quiesce_all(int32_t timeout_ms)` | Stops every engine and waits for GPU work. Call from `applicationWillTerminate` | engines stopped |
| `be_essence2_last_refusal(char* out, int32_t capacity)` | The sentence behind the latest `-3` (no secret, rejected secret, or cannot reach bitHuman) | its full byte length; `0` if nothing was refused |

### Audio and frames

| Function | Purpose | Returns |
|---|---|---|
| `be_essence2_push_audio(h, const int16_t* samples, int32_t count)` | Queues speech | `0`; `-2` nothing queued (pull frames, then push again) |
| `be_essence2_frames_available(h)` | Frames ready to pull | count |
| `be_essence2_pull_frame(h, uint8_t* out, int32_t capacity)` | Takes the oldest frame | bytes written; `0` none ready; `-3` the session was refused (destroy the engine) |
| `be_essence2_get_info(h, int32_t* width, int32_t* height)` | Frame size for the current mode | — |
| `be_essence2_idle_frame(h, uint8_t* out, int32_t capacity)` | Next idle frame | bytes written; `0` keep the current frame; `-3` refused |
| `be_essence2_reset(h)` | Interrupt: drops queued audio and frames | — |
| `be_essence2_end_utterance(h)` | The reply's audio is complete: the rest renders and eases to rest now, not after 0.6 s without audio | `0`; `-1` bad handle |
| `be_essence2_last_frame_kind(h)` | What the last pulled frame shows: `BE_ESSENCE2_FRAME_IDLE` (0), `_SPEECH` (1) or `_RAMP` (2, easing back to rest). A reply is over when an idle frame follows its speech | kind; `-1` before any frame |

### Display and status

| Function | Purpose | Returns |
|---|---|---|
| `be_essence2_set_mode(h, int32_t mode)` | `BE_ESSENCE2_MODE_FULL` (0, default) or `BE_ESSENCE2_MODE_HEAD` (1, the square head frame) | `0`; `-1` bad handle; `-2` bad mode |
| `be_essence2_render_status(h, char* reason, int32_t len, int64_t* failures)` | Whether the engine's runtime failed | `0` healthy; `-1` failed, with `reason` |
| `be_essence2_pulled_speech_frames(h)` | Speech frames returned so far (tells speech from idle) | count |
| `be_essence2_set_playout_anchor(h, int32_t on)` | Start each reply at the live audio position. Leave off if your app holds audio until the first frame | `0`; `-1` bad handle |
| `be_essence2_reanchor_slots(h)`, `be_essence2_anchor_slots(h)` | Counters for frames skipped to keep sync under load | count |

## Sessions and billing

A session checks the API secret when it starts and reports its session time, talking or idle. If the network drops after the secret is accepted, frames continue for 5 minutes of rendered video, then `pull_frame` and `idle_frame` return `-3` (Essence 2) or `pull()` returns `nil` (Expression 2) until the connection returns. A secret rejected mid-session is final: destroy the engine. Prices are on [pricing](https://docs.bithuman.ai/pricing).