Parakeet-EOU-120M-ONNX-INT8

Streaming, multilingual speech-to-text with end-of-utterance detection — a 120M-parameter cache-aware FastConformer RNN-T, exported to ONNX for on-device inference on Linux, Windows, and Android. INT8 encoder, FP32 decoder + joint.

The model emits <EOU> (end-of-utterance) and <EOB> (end-of-boundary) tokens inline with the transcript, so a voice agent can detect turn boundaries from the ASR stream itself instead of a separate VAD-based turn detector.

Model

Property	Value
Parameters	~120M
Architecture	Cache-aware FastConformer encoder (17 layers, 512 hidden) + 1-layer LSTM decoder + RNN-T joint
Quantization	Encoder INT8 (weight-only); decoder + joint FP32
Format	ONNX (opset 17)
Sample rate	16 kHz, 128-bin mel
Streaming	320 ms chunks (64 mel frames), left context 70, conv cache 8
Vocab	1024 BPE + <EOU> + <EOB> + blank (1027 logits)
Languages	25 European

Files

File	Size	Description
parakeet-eou-encoder.onnx	~132 MB	Cache-aware FastConformer encoder (INT8), mel + cache state → encoded frames + new cache
parakeet-eou-decoder.onnx	~16 MB	LSTM prediction network (FP32), token + (h, c) → hidden + (h', c')
parakeet-eou-joint.onnx	~6 MB	RNN-T joint (FP32), encoder frame + decoder hidden → logits
vocab.json	~17 KB	Token id → BPE piece, incl. <EOU> / <EOB>
config.json	<1 KB	Mel frontend, encoder/decoder dims, streaming chunk params

Performance

Measured with onnxruntime (CPU, INT8 encoder), streaming decode with cache handoff:

Metric	Value	Notes
Peak RSS	~420 MB	macOS arm64, onnxruntime CPU (incl. Python/runtime overhead; native C++ is lower)
Encoder RTF	~67×	Faster than real time; per-chunk compute well under the 320 ms window
Bundle size	~153 MB	Encoder INT8 + decoder/joint FP32

Roughly 3× lighter than the 0.6B Parakeet (~1.1–1.3 GB peak RSS) while adding streaming + EOU, at similar footprint to a small streaming Zipformer but with multilingual coverage.

Usage

Python (onnxruntime) — streaming greedy decode with cache handoff:

import onnxruntime as ort, numpy as np, json

enc = ort.InferenceSession("parakeet-eou-encoder.onnx", providers=["CPUExecutionProvider"])
dec = ort.InferenceSession("parakeet-eou-decoder.onnx", providers=["CPUExecutionProvider"])
jnt = ort.InferenceSession("parakeet-eou-joint.onnx", providers=["CPUExecutionProvider"])
cfg = json.load(open("config.json"))

# Feed 128-bin log-mel in 64-frame chunks; carry cache_last_channel / cache_last_time /
# pre_cache between chunks. For each encoded frame run decoder→joint→argmax; a `blank`
# advances the frame, `<EOU>` (id 1024) marks the end of the utterance.

CLI (via the C++ runtime): see the speech-core docs linked below.

Source

Exported from nvidia/parakeet_realtime_eou_120m-v1.

Links

• speech-core — C++ runtime
• Docs — runtime docs
• soniqo.audio — website
• blog — blog