Chatterbox Multilingual — all-CoreML (ANE-optimized)
On-device Apple Silicon build of
ResembleAI's multilingual Chatterbox
(t3_mtl23ls_v3, 23 languages) for iOS 18+ / macOS 15+, arm64. The whole
pipeline is CoreML; T3 prefill + decode run on the Apple Neural Engine and
the S3Gen synth on ANE + GPU. v1 retains full multilingual capability and is
validated/tuned on Russian (on-device, ear-checked, iPhone 17 Pro Max /
iOS 26.5.1).
All 23 languages generate speech with upstream's own per-language demo prompts (checked for plausible output length, not ear-checked outside Russian).
Turbo (English, GPT-2) sibling: iliasaz/chatterbox-turbo-coreml.
Links
- Upstream model — ResembleAI/chatterbox by Resemble AI.
- Runtime — iliasaz/chatterbox-coreml, the Swift package + demo app that loads this repo.
- Watermarking — iliasaz/perth-coreml (Swift) / weights. Every utterance the runtime generates is watermarked with Perth, as upstream chatterbox does.
- Russian stress — iliasaz/ruaccent-coreml (Swift) / weights, a CoreML port of RUAccent.
How it differs from turbo
The multilingual T3 is a different transformer, not a re-export:
- LLaMA_520M, 30 layers, RoPE + RMSNorm + SwiGLU (SiLU), split q/k/v/o, bias-free
(turbo = GPT-2-medium, 24 layers, learned
wpe, LayerNorm, GELU). - Classifier-free guidance at batch=2 — the uncond lane is the same prefix
with the text embedding zeroed; per step
logits = cond + cfg·(cond − uncond). - Cond prefix = speaker(1) + Perceiver(32) + emotion(1) = 34 rows; the Perceiver resampler and emotion vector are computed host-side.
- Grapheme BPE tokenizer (text vocab 2454); speech side (start 6561 / stop
- unchanged. Russian stress comes from
iliasaz/ruaccent-coreml
(neural), with a manual
U+0301override / dictionary fallback.
- unchanged. Russian stress comes from
iliasaz/ruaccent-coreml
(neural), with a manual
Artifacts
| File | What | Compute |
|---|---|---|
T3LM.mlpackage |
T3 prefill + decode, multifunction, shared MLState KV cache, 8-bit palettized (per_grouped_channel) |
CoreML — ANE (prefill+decode) |
S3Encoder.mlpackage |
flow encoder: speech tokens → mel mu (re-exported for the multilingual flow) |
CoreML — ANE |
S3CFM.mlpackage |
conditional flow-matching solver — non-meanflow, 10-step cosine CFG (two predicts/step) | CoreML — GPU |
S3Vocoder.mlpackage |
mel → 24 kHz waveform | CoreML — GPU |
CAMPPlus, MatchaMel, VEMel, VELSTM, S3Tokenizer .mlpackage |
on-device voice-cloning conditioning encoders | CoreML |
perceiver_*.npy |
Perceiver-resampler weights (host-side cond block) | — |
speech_emb.npy, speech_pos_emb.npy, text_emb.npy, text_pos_emb.npy |
embedding + learned position tables | — |
spkr_enc_*.npy, emotion_adv_fc_weight.npy |
speaker + emotion conditioning | — |
default-conds.safetensors, tokenizer.json, tokenizer_config.json |
default voice + grapheme tokenizer | — |
The presence of perceiver_query.npy is how the consumer app auto-detects the
multilingual variant.
Pipeline
text (+ optional Russian stress) → grapheme BPE → host-assemble inputs_embeds
(speaker + Perceiver(32) + emotion + text) + position_ids + masks
→ T3LM "prefill" (CoreML, ANE) — batch-2 CFG, writes shared KV state
→ T3LM "decode" (CoreML, ANE) — autoregressive, host CFG combine per step
→ S3Encoder (ANE) → S3CFM 10-step CFG (GPU) → S3Vocoder (GPU) — 24 kHz waveform
On-device numbers (iPhone 17 Pro Max, iOS 26.5.1, warm)
| stage | latency | engine |
|---|---|---|
| T3LM cold load (first launch after install) | ~52 s | ANE AOT compile |
| T3LM warm load (subsequent launches) | ~0.4–1.1 s | persisted compiled model (cache hit) |
| decode predict | ~58 ms / token | ANE (8-bit, batch-2 CFG) |
| synth (S3Encoder + 10-step CFG S3CFM + S3Vocoder) | ~1–2 s | ANE + GPU |
The fp16 T3LM fails the iPhone ANE plan build (error -14); the shipped T3LM is
8-bit palettized, which loads both functions on the ANE (parity vs fp16:
cos 0.9972, 9/9). The runtime persists the compiled model across launches and overlaps chunk N+1
decode (ANE) with chunk N synth (GPU).
Russian defaults
Mirrors the upstream multilingual server: temperature 0.8, exaggeration 1.3, cfg_weight 0.5, top_p 0.95, repetition_penalty 1.2 (min-p sampling, no top-k).
License
MIT. These weights are a CoreML format conversion of ResembleAI/chatterbox, which Resemble AI publishes under the MIT licence; the conversion inherits those terms, and copyright in the weights stays with Resemble AI — no claim of ownership is made over them here. Redistribute with attribution to Resemble AI.
The Swift runtime that loads them is separately MIT-licensed; see the LICENSE and
NOTICE files in iliasaz/chatterbox-coreml.
- Downloads last month
- 74
Model tree for iliasaz/chatterbox-multilingual-coreml
Base model
ResembleAI/chatterbox