rolandnsharp/oscillator-asr

Oscillator streaming speech recogniser (10.4 M parameters, one C file)

A small English speech recogniser that transcribes while you speak, built for a handheld ARM computer (ClockworkPi uConsole). Text appears every 320 ms instead of after you stop, and the whole model runs faster than real time on one slow ARM core, from a single dependency-free C file.

It is a research model with honest limits: accurate enough to dictate with on clean speech, well behind Whisper and production systems on accuracy. The point is the architecture and its cost.

How it works

• Front end: a bank of 64 damped complex oscillators running at the 16 kHz sample rate, z_t = c z_(t-1) + x_t with c = exp(-dt/tau + 2 pi i f dt). Frequencies and decay times are learned. No FFT, no convolution. Every 20 ms the bank's state (real, imaginary, energy) is the feature vector.
• Backbone: six recurrent blocks of complex poles at 50 Hz (learned decay times from under one frame to about 8 s), each followed by a gated MLP. No attention: the state is a fixed size, so cost and memory per second of audio are constant, and output is available as the audio arrives.
• CTC over characters, decoded greedily as you speak, and re-decoded at each pause with a word 3-gram (prefix beam search, also in the C file).

Every experiment that led here, including the failures, is written up with predictions made before each run (https://experiments.blah.dev/EXP-0049).

Results

LibriSpeech, word error rate (lower is better), all 2,620 / 2,939 test utterances. "LM" = beam search (width 64) with LibriSpeech's own pruned 3-gram (OpenSLR 11), its two weights chosen on dev-clean, run by the int8 C program. Greedy columns are the PyTorch model; the int8 C program scores 16.9 % greedy on test-clean.

model	params	test-clean greedy	test-other greedy	test-clean + LM
this model	10.4 M	16.8 %	37.9 %	9.1 %
round 3 (previous)	5.8 M	38.6 %	62.1 %	19.4 %

Speed on the uConsole (ClockworkPi A06 core, Rockchip RK3399), one core, real-time factor = compute seconds per second of audio (lower is faster), 87 s of test audio, model and LM loading included:

recogniser	streaming	A72 1.8 GHz	A53 1.4 GHz	WER on these 12 utterances
this model, int8 + 3-gram	yes, 320 ms	0.16	0.35	7.7 %
sherpa-onnx streaming Zipformer 20M, int8	yes	0.21	0.48	8.6 % (no LM)
whisper.cpp tiny.en	no (30 s window)	2.26 (1 thread)	4.49 (1 thread)	5.0 %

Use

cc -O3 -ffast-math -fno-finite-math-only -o zara_stt zara_stt.c -lm
./zara_stt model.bin speech.wav --int8 --lm librispeech-3gram.zlm --alpha 0.7 --beta 1.0   # 16 kHz mono WAV
arecord -q -f S16_LE -r 16000 -c 1 -t raw | ./zara_stt model.bin --stream --int8 --lm librispeech-3gram.zlm --alpha 0.7 --beta 1.0

--stream prints letters live and a cleaned-up line after each pause. On the uConsole headset use dictate.sh (the codec delivers the mic as left minus right).

Training

LibriSpeech 960 h (train-clean-100, -360, -other-500), 16 kHz mu-law, 150,000 updates of 240 s batches (about 11.3 passes), one NVIDIA RTX 3090, about 28 hours, about $7. Shape: osc front (64 poles), 6 blocks of width 448, gated MLP 672. Optimiser: Muon on the blocks' matrices (Nesterov momentum, Newton-Schulz orthogonalised updates), AdamW for everything else, peak learning rate 3e-3, 200 warmup updates, cosine to 10 %. The language-model weights used by default (alpha 0.7, beta 1.0, passed by dictate.sh) were chosen on dev-clean.

Limitations

• English read speech only; accents and microphones outside LibriSpeech are weaker.
• No punctuation or capitals.
• The language model is built from public-domain books: modern and technical words are often misrecognised.
• Accuracy far below Whisper and production recognisers.

Files

zara_stt.c (inference: oscillator bank, blocks, int8 NEON kernels, beam search, streaming), model.bin, params.pt (PyTorch), librispeech-3gram.zlm (the 3-gram in the C file's format), zara_stt.py and zara_block.py (training: model, block, optimizer), mulaw.py, pack_libri.py, export_c.py, export_lm.py, score_lines.py, dictate.sh, SHA256SUMS.

Credits

Roland Sharp, with Claude (Anthropic). Muon: Jordan et al. (2024); Liu et al. (2025), "Muon is Scalable for LLM Training". Data: LibriSpeech audio (Panayotov et al., 2015), CC BY 4.0 (OpenSLR 12); the 3-gram and vocabulary are from OpenSLR 11 (public domain).