seanll95/sidon-coreml

Sidon — Core ML

Core ML conversions of sarulab-speech/sidon-v0.1, a speech restoration model, rebuilt to run on an iPhone's Neural Engine. Used by Excerpt, an iOS audiobook player, to lift noise and muffling out of narration on the device.

16 kHz in, 48 kHz out, duration preserved. The weights are upstream's, unchanged — nothing here is fine-tuned. What changed is the graph around them, so that it compiles for the Neural Engine.

Contents

File	Precision	Size	Fixed input
sidon_encoder_ane.mlpackage	fp16	468 MB	feats [1, 300, 160] -> hidden [1, 300, 1024]
sidon_decoder_ane.mlpackage	fp16	101 MB	feat [1, 1024, 68] -> [1, 1, 65280]
sidon_fbank.bin	float32	84 KB	Front-end constants: 400-tap Povey window and 257 x 80 mel filters

.mlpackage files are uncompiled. Compile them on the device with MLModel.compileModel(at:) and cache the result; the first compile can take a minute or more. Load both with MLComputeUnits.cpuAndNeuralEngine.

How this differs from upstream

The feature extractor is not in the model. Upstream feeds raw audio through SeamlessM4TFeatureExtractor (from facebook/w2v-bert-2.0) in Python. Here the encoder takes the 160-dimensional stacked fbank features directly, and sidon_fbank.bin carries that extractor's exact window and mel filters so the features can be computed natively. Recomputing them from a formula instead shifts every frame. The extractor normalises per utterance, so compute features once over a long stretch of audio and slice the features, never the audio.

The encoder runs 300 frames (6 s) per call. Upstream has no fixed length. The Neural Engine compiler fails at 600 frames — it runs out of memory expanding the relative position tables, and on a phone it is killed. 300 frames compiles and runs 99.3 % on the Neural Engine. The conformer uses long-range context, so shorter windows cost accuracy: against 12 s windows this measures about 2 dB lower SNR, which was inaudible in blind listening tests. The encoder's dictionary output is also rewired to a plain tensor, since Core ML has no dictionary op; that surgery is checked against the original to 1e-5.

The decoder is rebuilt, and its transposed convolutions are rewritten. Upstream's DAC decoder is a frozen TorchScript module whose scripted Snake activations cannot be converted. It is reconstructed in Python from the repo config with the weights transplanted in, checked bit-exact against the original. Its ConvTranspose1d layers — which the Neural Engine computes wrongly, dropping SNR by 53 dB at a single layer — are replaced with an equivalent sub-pixel form: two convolutions whose outputs interleave. Because every layer has kernel == 2 * stride, this is exact, not an approximation (max|d| 0.0 in PyTorch). It is also 1.65x faster on the iPhone GPU.

The decoder runs 68 frames per call. The Neural Engine caps every tensor dimension at 65536, and the decoder's output is 960 samples per frame, so 68 frames is the most that fits. Run it in overlapping windows with a 12-frame halo on each side and keep only the middle; that reconstructs a single long call to 120 dB. Every window must hold real frames — slide the last one back rather than zero-padding it.

What it costs

Against the same models on the GPU, the Neural Engine output measures 1.7 dB lower, from fp16 hidden states and the shorter encoder window. The difference sits above 16 kHz, 52 dB down; across the speech band it is within 0.4 dB. In a blind, shuffled listening test, PyTorch, all-GPU, GPU + Neural Engine and all-Neural Engine renders could not be told apart.

On an iPhone, warm, this runs at 11.2x realtime, against 4.8x on the CPU alone. A backgrounded iOS app cannot use the GPU, so for long audio that is the comparison that matters.

What it does

Sidon re-synthesises speech from w2v-BERT semantic features rather than repairing the waveform. Running it on clean audio changes it about as much as running it on damaged audio. Expect cleaner, clearer narration in the same voice, not a sample-accurate restoration of the original recording.

Licence and attribution

MIT, inherited from the upstream model — see LICENSE. The model, its training and its weights are the work of the Sidon authors (Wataru Nakata, Yuki Saito); this repository contributes only the Core ML conversion.