oruk-loop-153M

Oruk AI · Free for noncommercial use and for small businesses; other commercial use needs a license from Oruk AI (terms).

oruk-loop-153M is an English speech recognizer with 153,202,638 stored parameters, distilled from Qwen3-ASR-1.7B (2,038,052,480 stored parameters, 13.3× more). It keeps the teacher's architecture, but each of its two transformer stacks is one shared block run in a loop: one encoder block applied 24 times and one decoder block applied 28 times. Low-rank per-depth adapters tell each pass which layer it is standing in for.

On the seven English test sets of the Open ASR Leaderboard, mean WER is 7.12% with beam-4 decoding and 7.40% greedy. That bulges the size–accuracy frontier outward:

• At the same size. In the leaderboard snapshot of 14 October 2026, the previous frontier stood at 7.65% at 153M parameters (interpolated between neighbouring models), 0.52 points above this model.
• At the same accuracy. The interpolated frontier reached 7.12% only at about 259M parameters, 1.7× as many.
• Nearest real model. The closest actual model is moonshine-streaming-medium (240M, 7.18%), 1.6× larger. The 0.06-point gap equals our harness's offset (it scores the teacher 0.06 below the leaderboard on average), so treat the two as level on accuracy.

All numbers come from our harness, not an official leaderboard submission. Checkpoints were also selected on these test sets. Both caveats are explained under Evaluation.

Quick start

pip install "transformers==5.17.0" torch soundfile soxr

from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("oruk/oruk-loop-153M", trust_remote_code=True)  # CUDA if available
processor = AutoProcessor.from_pretrained("oruk/oruk-loop-153M")

print(model.transcribe("speech.wav", processor))               # greedy
print(model.transcribe("speech.wav", processor, num_beams=4))  # setting of the headline numbers
print(model.transcribe(["a.wav", "b.flac"], processor, batch_size=32))  # a list means a batch

transcribe accepts file paths, 1-D float arrays and (array, sampling_rate) tuples. For a bare array that is not at 16 kHz, pass sampling_rate=. A batch must be a list, because a tuple is read as one (array, sr) recording.

Audio longer than 30 s is cut at quiet points into chunks of at most 30 s, and the chunk transcripts are joined with spaces; return_chunks=True also returns the per-chunk texts. Each chunk's output is capped at min(448, 10 tokens per second + 24) tokens. length_penalty is passed through to beam search. The output keeps the teacher's casing and punctuation.

trust_remote_code=True runs modeling_oruk_loop.py from this repository; it subclasses the stock Qwen3ASRForConditionalGeneration. Tokenizer, chat template and feature extractor are the unmodified Qwen3-ASR files, so AutoProcessor needs no remote code. The code is tested with transformers==5.17.0 and PyTorch 2.9. Pin a revision for reproducible results.

Device	Defaults	Memory after loading
CUDA	bf16 autocast, per-depth weights merged for fast decoding (merge=True)	4.2 GB of GPU memory
CPU	fp32, adapters applied on the fly (merge=False)	3.9 GB of process RAM

Decoding adds the KV cache on top of this. A batch of 64 (or beam 4) needs several more GB.

Override the defaults with AutoModel.from_pretrained(..., device="cuda:1", dtype=torch.float32, merge=False).

On 12 vCPUs (Intel Xeon, 2.2 GHz), greedy decoding takes 0.57 s per second of audio, about 1.75× faster than real time.

To call generate yourself on a single clip of at most 30 s:

import soundfile as sf
audio, sr = sf.read("short.wav")  # 16 kHz mono
inputs = processor.apply_transcription_request(audio=audio, language="English", return_tensors="pt",
                                               processor_kwargs={"audio_kwargs": {"sampling_rate": 16000}})
inputs = {k: v.to(model.device) for k, v in inputs.items()}
inputs["input_features"] = inputs["input_features"].to(model.model.audio_tower.conv2d1.weight.dtype)
out = model.generate(**inputs, max_new_tokens=448)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], return_format="transcription_only"))

Architecture

Component	What is stored	Parameters
Audio encoder loop	one Qwen3-ASR encoder block (d=1024) applied 24×, with rank-64 adapters on every linear layer and per-depth biases and LayerNorms	41,213,952
Decoder loop	one Qwen3 decoder block (d=2048, GQA 16/8 heads) applied 28×, with rank-16 adapters and per-depth RMSNorms	67,886,080
Token table	36,974 kept English token rows, tied input/output: 368 frequent rows stored exactly, 36,606 rows factorised at rank 729	28,932,430
Kept Qwen3-ASR modules	convolutional front-end, encoder output norm, audio projector, final decoder norm	15,170,176
Total		153,202,638

Each looped linear layer computes W x + b_d + B_d (A_d x) at depth d. Here W is shared by every depth, and A_d, B_d and b_d belong to depth d. The shared weights total 62.9M and the per-depth adapters 45.7M (28.3M of them in the encoder).

Token ids stay in the original Qwen3 vocabulary. The output layer scores only the kept rows and returns -inf for the rest, so the stock tokenizer and prompts work unchanged.

Looping shrinks storage, not arithmetic. Every token still passes through 24 encoder and 28 decoder layers at the teacher's width. Per-token compute is therefore close to the teacher's; the trimmed output layer (37k rows instead of 152k) is the only saving. With merged weights, decoding needs about as much accelerator memory as the teacher.

On one A100 40 GB, greedy decoding with batch 64 transcribes the seven leaderboard sets about 145× faster than real time, audio loading included (three such jobs shared the machine).

Training

1. Initialisation. Each shared weight starts as the mean of the teacher's layers. Each depth's adapter starts as the truncated SVD of that layer minus the mean, so a full-rank adapter would reproduce the teacher exactly. Per-depth norms and biases are copied from the teacher.
2. Distillation from Qwen3-ASR-1.7B, in several re-warmed learning-rate cycles:
   • the decoder matches the teacher's token distributions (KL) and its hidden states at every depth;
   • the encoder matches the teacher's per-depth states and its output (MSE and cosine);
   • the text targets keep the teacher's casing and punctuation, with the word content taken from each corpus's reference.
3. Vocabulary. The decoder keeps 36,974 of Qwen3's 151,936 tokens: every token used at least 100 times in the teacher's English training transcripts, plus the byte, special and prompt tokens, closed under BPE merges. Rare rows are factorised, which turns 311M embedding parameters into 28.9M.
4. Weight averaging. The final EMA weights of ten distillation runs are averaged uniformly.
5. MWER fine-tuning. Only the decoder's per-depth adapters and norms are trained, for 500 steps per run at a learning rate of 3e-5. The objective is expected word error over the model's own beam-4 N-best, measured against reference transcripts (Whisper-normalised), plus a cross-entropy anchor weighted 0.1. Four runs are averaged; eight further runs, each started from that average, are averaged again.

Training data

Only training splits are used. The Earnings-22 split is not part of the original corpus, so we checked it by recording ID: its 111 training calls share none of the 6 test calls.

Corpus	Audio used	Upstream terms
LibriSpeech	train, 961 h	CC BY 4.0
SPGISpeech	L train, 4,900 h	Kensho Terms of Usage: academic research and internal use
GigaSpeech	L train (2,475 h) and 7,434 h of XL train, overlapping	access for non-commercial research and education
VoxPopuli	en train, 508 h	CC0 1.0
TED-LIUM 3	train, 454 h	CC BY-NC-ND 3.0
AMI	IHM train, 72 h	CC BY 4.0
Earnings-22	train split, 98 h	CC BY-SA 4.0
YODAS3 (English), FLEURS en_us train	early warm-up of the looped encoder	CC BY 3.0; CC BY 4.0

The teacher's outputs on this audio (distributions, hidden states, transcripts) are training targets. MWER used the AMI, VoxPopuli, TED-LIUM, LibriSpeech-360, SPGISpeech (subset), GigaSpeech L and Earnings-22 training audio with reference transcripts.

Evaluation

Test sets. The Open ASR Leaderboard's English sets (hf-audio/open-asr-leaderboard, original transcripts), plus TED-LIUM 3 test from sanchit-gandhi/tedlium-data.

Scoring follows the leaderboard:

• the Whisper EnglishTextNormalizer is applied to reference and hypothesis;
• references that are empty or read "ignore time segment in scoring" are dropped;
• WER is corpus-level per set (edits ÷ reference words);
• the average is the plain mean over the seven sets of the leaderboard's English average.

Comparison numbers for other models are from the leaderboard snapshot of 14 October 2026. Their sizes are the leaderboard's listed sizes; ours is the stored count.

Model	Params	AMI	Earnings-22	GigaSpeech	LS clean	LS other	SPGISpeech	VoxPopuli	Mean
oruk-loop-153M, beam 4	153M	11.22	9.80	10.45	2.40	5.95	2.48	7.55	7.12
oruk-loop-153M, greedy	153M	11.57	10.29	10.75	2.53	6.32	2.61	7.70	7.40
nvidia/parakeet-tdt_ctc-110m	0.11B	15.89	12.37	10.52	2.40	5.22	2.54	6.90	7.98
nvidia/canary-180m-flash	0.18B	14.86	12.33	10.51	1.73	4.35	2.26	6.35	7.48
usefulsensors/moonshine-streaming-medium	0.24B	10.68	11.92	9.46	2.09	5.00	2.58	8.54	7.18
Qwen/Qwen3-ASR-0.6B	0.6B	11.66	11.06	9.14	2.13	4.45	3.03	7.07	6.93
nvidia/parakeet-tdt-0.6b-v2	0.6B	11.16	11.15	9.74	1.69	3.19	2.17	5.95	6.44
Qwen/Qwen3-ASR-1.7B (teacher), leaderboard	1.7B	10.56	10.25	8.74	1.63	3.40	2.84	6.35	6.25
Qwen/Qwen3-ASR-1.7B (teacher), our harness, greedy	1.7B	10.48	10.06	8.74	1.62	3.36	2.84	6.24	6.19

Harness offset. On the teacher, our harness reads 0.06 points below the leaderboard on average. Small gaps against leaderboard rows (such as the 0.06 to moonshine-streaming-medium) are within that offset.

TED-LIUM. On TED-LIUM 3 test the WER is 3.77% (beam 4) and 3.79% (greedy); the teacher scores 2.28%. With TED-LIUM included, our eight-set mean is 6.70% (beam 4) and 6.95% (greedy).

Rank in the snapshot. 65 models have a listed size above 153M; 48 of them have a higher mean than our beam-4 result. Among models listed below 600M, the beam-4 mean is the lowest, with moonshine-streaming-medium level within the harness offset. With greedy decoding, moonshine-streaming-medium (7.18%) is ahead.

The frontier bulge. Before this model, the lowest-WER curve across model sizes ran:

Model	Size	Mean WER
parakeet-tdt_ctc-110m	110M	7.98%
canary-180m-flash	180M	7.48%
moonshine-streaming-medium	240M	7.18%
parakeet-tdt-0.6b-v2	600M	6.44%

Interpolating that curve linearly in log size puts it at 7.65% at 153M, 0.52 points above our beam-4 result. The same curve reaches 7.12% only at about 259M parameters.

Selection caveat. We compared candidate checkpoints and recipes on these same test sets, so the figures above are best-of-candidates rather than fully held out.

Further test sets. Beyond the leaderboard, the model gives up more of the teacher's accuracy, and it ranks last of these five models on the average. The table is WER % from our harness: beam 4 for ours, each model's default decoding for the others.

Model	FLEURS en_us (647)	Common Voice (4,000)	EdAcc (8,537)	People's Speech clean (3,999)	Mean
oruk-loop-153M	9.88	17.15	20.16	20.15	16.84
nvidia/canary-180m-flash	7.21	9.56	17.96	19.17	13.48
usefulsensors/moonshine-streaming-medium	9.03	14.61	21.75	19.20	16.15
openai/whisper-small.en	7.79	15.39	23.23	18.88	16.32
Qwen/Qwen3-ASR-1.7B (teacher)	4.71	7.53	13.23	16.73	10.55

Common Voice is taken from the leaderboard repository's packaging of its test split, and People's Speech from the clean test split; both are fixed random subsets (seed 20261014). FLEURS test sentences are not in the training data, but FLEURS en_us training audio was used in the early encoder warm-up, so FLEURS is not fully out of domain.

This release. The rows above come from our evaluation pipeline, which uses CUDA-graph greedy decoding and fixed-length prompt padding. We re-scored every set with this repository's own code (AutoModel plus transcribe, batch 64). With greedy decoding, 96.3% of utterances come out identical to the pipeline. The rest differ by bf16 rounding, which can move a near-tied token and, rarely, end a chunk early. Set-level gaps come from a handful of clips: one LibriSpeech test-other clip accounts for 52 of that set's 58 extra errors, and one TED-LIUM clip for 31 of 34. Beam search is less sensitive: every beam-4 set score reproduces within 0.012 points, and the 7-set mean is identical.

Set	Pipeline, greedy	This repo, greedy	Pipeline, beam 4	This repo, beam 4
AMI	11.57	11.55	11.22	11.22
Earnings-22	10.29	10.27	9.80	9.80
GigaSpeech	10.75	10.76	10.45	10.45
LS clean	2.53	2.54	2.40	2.40
LS other	6.32	6.43	5.95	5.94
SPGISpeech	2.61	2.61	2.48	2.48
TED-LIUM	3.79	3.91	3.77	3.77
VoxPopuli	7.70	7.71	7.55	7.54
7-set mean	7.40	7.41	7.12	7.12

Per-set scores, word counts and the frontier analysis are in results/evaluation.json.

Limitations

• English only. Trained and evaluated only with the English prompt (language="English"), and the vocabulary is trimmed to English tokens.
• Domain. Accuracy is best on audio like the training corpora: read speech, meetings, finance calls, parliament, podcasts. On accented, crowd-sourced and found speech it degrades more than the teacher (see the table above).
• No timestamps or diarization. Long recordings are handled by quiet-point chunking. Rare runs of near-tied tokens can end a chunk early or repeat a phrase; beam 4 is more robust than greedy. Very fast speech can reach the per-chunk token cap.
• Compute. The parameter saving does not reduce per-token compute (see Architecture).

License and attribution

Files	License
model.safetensors (weights)	PolyForm Small Business 1.0.0 or PolyForm Noncommercial 1.0.0, at your option; other commercial use needs a commercial license from Oruk AI
modeling_oruk_loop.py	MIT
Unmodified Qwen3-ASR tokenizer, chat template and processor files	Apache-2.0
results/ and media/	CC BY 4.0, attributed to Oruk AI

What the weights license allows.

• Small businesses can use the weights for anything, commercial products included. A small business here means fewer than 100 employees and contractors and under US$1M revenue in the prior year.
• Anyone can use them for personal, research and educational purposes.
• Charities, schools, universities, public research organisations and government bodies can use them freely.
• Any other commercial use needs a commercial license from Oruk AI; contact us through oruk.ai.

The model derives from Qwen3-ASR-1.7B (Apache-2.0, © Alibaba Cloud / Qwen team). Its initialisation, the kept modules and the processor files come from that model; see NOTICE.md.

Some training corpora carry non-commercial or research-only terms: SPGISpeech, GigaSpeech and TED-LIUM 3. Review those terms for your use. Corpus copyrights stay with their owners. No endorsement by Qwen, Alibaba Cloud or the dataset providers is implied.

Citation

@misc{oruk2026loop153m,
  title  = {oruk-loop-153M: a double-looped distillation of Qwen3-ASR},
  author = {{Oruk AI}},
  year   = {2026},
  url    = {https://huggingface.co/oruk/oruk-loop-153M}
}

@article{Qwen3-ASR,
  title   = {Qwen3-ASR Technical Report},
  author  = {Xian Shi and Xiong Wang and Zhifang Guo and Yongqi Wang and Pei Zhang and Xinyu Zhang and Zishan Guo and
             Hongkun Hao and Yu Xi and Baosong Yang and Jin Xu and Jingren Zhou and Junyang Lin},
  journal = {arXiv preprint arXiv:2601.21337},
  year    = {2026}
}