wegotomars/OmniVoice-CoreAI-ANE

OmniVoice on Apple Neural Engine

Core AI conversion of k2-fsa/OmniVoice for multilingual speech synthesis, voice cloning, and voice design, using tinkerbox-ane.

The FP16 Qwen3 diffusion transformer and audio projection run fully on the ANE. Text processing, embeddings, scheduling, sampling, and the bundled 24 kHz audio codec run on the CPU while your GPU stays idle and cool 😎❄️.

Requires macOS 27, Apple Silicon M3 or newer, and Python 3.12 or 3.13. M1 and M2 are not supported: their ANE compiler rejects the long-context attention.

Usage

Install tinkerbox-ane[speech]>=2.2.0. The downloader fetches only this Mac's bundle: its checkpoint folder carries the tokenizer, configuration, embeddings, and audio codec the CPU uses.

Generate speech and save it as WAV:

import soundfile as sf
from tinkerbox_ane import omnivoice
from tinkerbox_ane.bundle import download_bundle

bundle = download_bundle("omnivoice", "wegotomars/OmniVoice-CoreAI-ANE", "v2.2.0")
with omnivoice.load(bundle) as model:
    audio = model.generate("Hello! Welcome to Tinkerbox.", language="en", seed=42)[0]
    sf.write("speech.wav", audio, model.sampling_rate)

Clone a voice from a recording and its transcript, reusing the downloaded bundle:

with omnivoice.load(bundle) as model:
    voice = model.create_voice_clone_prompt(
        "reference.m4a", ref_text="The words spoken in the reference recording."
    )
    audio = model.generate("A new sentence in the same voice.", voice_clone_prompt=voice, seed=42)[0]
    sf.write("cloned.wav", audio, model.sampling_rate)

For voice design, pass an instruction such as instruct="female, low pitch, British accent" to generate. Voice prompts can be saved with voice.save("voice.pt") and reused with model.load_voice_clone_prompt("voice.pt").

Text, reference, and generated audio share the original model's full 40,960-position context per diffusion call. Up to 4,096 positions run as one fused ANE call, longer sequences use blockwise attention. Long-form speech uses upstream chunking above its default 30-second threshold. OmniVoice refines every audio frame together until the last diffusion step, so it does not stream partial audio.

Supported architectures

Architecture	SoC name
h15g	Apple M3
h15s	Apple M3 Pro
h15m	Apple M3 Max (14-core CPU)
h15c	Apple M3 Max (16-core CPU)
h15d	Apple M3 Ultra
h16g	Apple M4
h16s	Apple M4 Pro
h16c	Apple M4 Max
h17g	Apple M5
h17s	Apple M5 Pro
h17c	Apple M5 Max
h17d	Apple M5 Ultra
h18g	Apple M6

Benchmarks

Measured October 26, 2026 on Apple M5 Pro (h17s, 24 GiB), macOS 27.0.1, warm. Real-time multiplier is audio duration divided by generation time (higher is faster); RTF is its inverse (lower is faster).

Request	Audio	Generation time	Real-time multiplier	RTF	ANE energy
Automatic voice	4.4 s	1.11 s	4.0×	0.25	4.2 J
Saved clone	2.9 s	1.03 s	2.8×	0.35	3.8 J
Passage, default chunking	18.5 s	5.74 s	3.2×	0.31	21.2 J

Memory	Peak
Wired, including ANE weights	1.35 GiB
Python process	1.49 GiB

Accuracy

Compared with the original omnivoice==0.2.1 in CPU FP16 on identical inputs:

Check	Result
Full-transformer logits (96–3,000 positions), cosine	≥ 0.999985
40,960-key attention kernel, cosine	≥ 0.999998
60-second passage, mean transcription error	0.187 (original 0.193)

Original checkpoint: k2-fsa/OmniVoice@c5fdb5ccb189668d56333f77ba2629f4cd7535f4. Weights and converted assets retain CC-BY-NC-4.0. Upstream code is Apache-2.0. The bundled Higgs audio codec retains the Boson Higgs Audio 2 Community License, which incorporates the Meta Llama 3 Community License. See NOTICE.

Built with Higgs Materials licensed from Boson AI USA, Inc., Copyright Boson AI USA, Inc., All Rights Reserved and Meta Llama 3 licensed under the Meta Llama 3 Community License, Copyright Meta Platforms, Inc., All Rights Reserved.