naklitechie/steno-speech-models

Steno speech models (int8 ONNX)

Two int8 ONNX models used by the Steno meeting recorder, loaded through parakeet-rs and ONNX Runtime on the CPU:

Folder	Model	Upstream	Size	Licence
parakeet-ultra-int8/	Speech recognition, 25 languages	moondream/parakeet-ultra (post-trained nvidia/parakeet-tdt-0.6b-v3)	935 MB	CC-BY-4.0
nemotron-3-diarization-int8/	Speaker diarization, up to 8 speakers	nvidia/Nemotron-3-Diarization	107 MB	OpenMDW-1.1 (LICENSE in that folder)

These are quantised copies, not official releases. The models, their training and their evaluations are Moondream's and NVIDIA's; see their model cards for intended use, limits, bias and privacy notes.

How they were made

Source: the fp32 ONNX exports in altunenes/parakeet-rs at revision b53b9101fa87271041773b5036aeb8db9d810d40 (parakeet-ultra/, nemotron-3-diarization/). Each graph was quantised with ONNX Runtime dynamic quantisation, weights only, MatMul and Gemm nodes only (attention left in fp32 so the graph loads in ONNX Runtime 1.24):

from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic(src, dst, weight_type=QuantType.QInt8, op_types_to_quantize=["MatMul", "Gemm"])

vocab.txt is unchanged from the source.

Evaluation (int8 against the fp32 source)

Run on 29 Oct 2026 on an Apple Silicon Mac, CPU only, with the same pipeline Steno uses (loudness to -20 dBFS, ~10 s chunks cut at quiet points, offline diarization profile). AMI recordings are the headset mix of six AMI test meetings, scored against AMI's manual words (WER, OpenAI English normaliser) and manual speaker segments (frame-level DER, 10 ms, no collar, overlap scored, optimal speaker mapping). "Steno" rows are two real 70- and 97-minute meetings in Indian English with some Hindi; they have no reference, so only int8-vs-fp32 disagreement is shown. Speeds are rough (a shared machine).

ASR: Parakeet Ultra fp32 (2.55 GB) against int8 (935 MB)

Recording	Minutes	fp32 WER %	int8 WER %	Δ	int8 vs fp32 %	fp32 speed	int8 speed
AMI ES2004a	17	16.0	15.9	-0.1	1.7	67×	71×
AMI ES2004b	39	14.5	14.6	+0.1	1.9	59×	65×
AMI IS1009a	14	19.0	18.9	-0.1	1.2	60×	70×
AMI IS1009b	34	13.8	13.5	-0.3	1.3	58×	72×
AMI TS3003a	25	13.4	13.4	+0.0	1.8	61×	73×
AMI TS3003b	37	13.3	13.4	+0.1	2.4	60×	72×
Steno 2026-06-02-122536	70	n/a	n/a	n/a	5.2	59×	65×
Steno 2026-06-01-134022	97	n/a	n/a	n/a	7.4	58×	63×
AMI pooled		14.5	14.5	-0.1

Diarization: Nemotron-3 fp32 (402 MB) against int8 (107 MB)

Recording	True speakers	fp32 found	int8 found	fp32 DER %	int8 DER %	int8 vs fp32 %	fp32 speed	int8 speed
AMI ES2004a	4	4	4	34.3	34.4	1.3	152×	164×
AMI ES2004b	4	5	5	25.9	26.2	1.2	137×	159×
AMI IS1009a	4	4	4	31.7	32.0	1.8	147×	162×
AMI IS1009b	4	4	4	16.3	16.4	0.9	146×	147×
AMI TS3003a	4	4	4	43.9	44.0	1.1	130×	160×
AMI TS3003b	4	4	4	32.0	32.2	0.7	145×	160×
Steno 2026-06-02-122536	?	8	8	n/a	n/a	5.9	139×	150×
Steno 2026-06-01-134022	?	4	4	n/a	n/a	3.4	144×	151×

Summary: on 2.8 hours of AMI meetings the int8 recogniser matches fp32 (14.5% pooled WER for both) and runs 10-20% faster with about 40% less peak RAM; the int8 diarizer finds the same speakers and stays within 0.3 points of fp32 DER. On a Windows laptop (Intel i5-9300H), a 70-minute meeting took 466 s with int8 against 643 s with fp32, peak RAM 1.95 GB against 2.96 GB.

Usage (Rust)

use parakeet_rs::{ExecutionConfig, ParakeetTDT, TimestampMode, Transcriber};
let cfg = ExecutionConfig::new().with_environment(parakeet_rs::ort::init().build()?);
let mut asr = ParakeetTDT::from_pretrained("parakeet-ultra-int8", Some(cfg))?;
let text = asr.transcribe_samples(pcm_16k_mono, 16_000, 1, Some(TimestampMode::Sentences))?.text;

Keep each call to about 10 seconds of audio: longer pieces drop words.

Attribution

• Parakeet Ultra by Moondream, derived from NVIDIA Parakeet TDT 0.6B v3, licensed CC-BY-4.0. Quantised to int8 by NakliTechie; no other changes.
• Nemotron-3 Diarization by NVIDIA, licensed OpenMDW-1.1 (copy in nemotron-3-diarization-int8/LICENSE). Quantised to int8 by NakliTechie; no other changes.