itsmechico/omnivoice-coreml-ane

OmniVoice for Core ML on the Neural Engine

A Core ML conversion of OmniVoice by k2-fsa. It runs with MLComputeUnits.cpuAndNeuralEngine, without the GPU, so it keeps working in an iOS app playing in the background.

Built with Higgs Materials licensed from Boson AI USA, Inc., Copyright Boson AI USA, Inc., All Rights Reserved and Meta Llama 3 licensed under the Meta Llama 3 Community License, Copyright Meta Platforms, Inc., All Rights Reserved.

Licenses

• step is converted from the OmniVoice weights, which the OmniVoice authors license under CC BY-NC: non-commercial use only, with attribution. Changes: re-exported to Core ML with the rewrites listed below.
• HiggsAudio2Decoder is converted from the decoder of the audio tokenizer shipped with OmniVoice, Boson AI's Higgs Audio 2 tokenizer, under the Boson Higgs Audio 2 Community License (LICENSE-Boson-Higgs-Audio-2), which incorporates the Meta Llama 3 Community License (LICENSE-Meta-Llama-3). See NOTICE.
• Voices f1, f2, m1, m2 are outputs of OmniVoice's voice design; rogerio is a LibriVox recording in the public domain. Each voices/<name>.json records its source.

Files

Model	Functions	Size
step	c{L}_t{T}: one unmasking step, 12 buckets, classifier-free guidance inside the graph	1.2 GB
HiggsAudio2Decoder	f{F}: codes → 24 kHz audio, F = 32…384 frames	53 MB

Each model is present as .mlpackage and as .mlmodelc (compiled for iOS 26). omnivoice.json holds the buckets, special tokens, duration rules and quality levels; tokenizer.json is the upstream Qwen2 tokenizer.

Changes for the Neural Engine

• RMSNorm divides by max|x| first: the residual stream reaches about 1.3e4, whose square overflows fp16.
• log_softmax is written as log-sum-exp; the ANE's version returns -45440 below about -17, which guidance turned into winning tokens.
• Attention is block-diagonal per segment in a channels-first layout.
• In the decoder, transposed convolutions become polyphase convolutions, Snake is rewritten fp16-safe, and the waveform is folded into rows of 16 frames.

Measurements (Apple M5 Max, macOS 27, .cpuAndNeuralEngine)

• Neural Engine share of cost: 93.9% for step (the embedding lookups run on the CPU), 99.8% for the decoder.
• Real-time factor on a 7.3 s Portuguese sentence: 0.076 / 0.112 / 0.148 / 0.292 at 8 / 12 / 16 / 32 steps.
• Whisper large-v3-turbo: English with no errors; Portuguese with one word error, which a run of the fp32 PyTorch step with the same sampler also makes.