ooobo/titanet-large-onnx-int8

TitaNet-large (ONNX, int8)

Static int8 quantization of NVIDIA TitaNet-large speaker verification, for sherpa-onnx. Drop-in replacement for nemo_en_titanet_large.onnx: same inputs, same 192-d output, same metadata, same embedding space.

	fp32	int8
file size	101.4 MB	28.1 MB (3.6×)
resident (model + ORT buffers)	301 MB	241 MB
200 s of audio, 1 thread	9.57 s	5.06 s (1.9×)
200 s of audio, 4 threads	3.87 s wall / 15.3 s CPU	2.24 s / 8.9 s (1.7×)

Quantized from the sherpa-onnx fp32 export (nemo_en_titanet_large.onnx), which is itself an export of nvidia/speakerverification_en_titanet_large.

Accuracy

Measured on 32 identity-labelled voice clips — 4 recurring ABC Radio Melbourne presenters over 7 days, each cropped to its dominant speaker by Sortformer diarization. Genuine/impostor cosine pairs; "trusted" is the 27-clip subset with clean, verified labels.

	fp32	int8
EER (trusted, 27 clips)	0.00%	0.00%
cross-day EER	0.00%	0.00%
worst genuine / best impostor	0.690 / 0.386	0.685 / 0.376
margin	+0.304	+0.309
same / different mean cosine	0.905 / 0.203	0.901 / 0.193
EER (full 32-clip set, noisy labels)	19.59%	19.59%
worst-case cosine to the fp32 embedding	—	0.986

Two properties matter as much as EER, and both hold:

• The score scale is preserved (impostor mean 0.203 → 0.193), so a threshold calibrated against fp32 still means what it meant. A quantization can keep EER perfect while compressing the score distribution upward — EER is rank-based and cannot see it.
• The embedding space is unchanged. Worst-case cosine between the two builds is 0.986, against typical operating thresholds around 0.4. Enrolled speaker profiles transfer between builds without re-enrolling.

Caveat: this is a small, domain-specific set (4 well-separated voices, one station, compressed radio audio). Read it as "int8 is not the bottleneck", not as a general accuracy claim — and re-measure on your own data before relying on it.

Usage

import sherpa_onnx
from huggingface_hub import hf_hub_download

model = hf_hub_download("ooobo/titanet-large-onnx-int8",
                        "nemo_en_titanet_large.int8.onnx")
ext = sherpa_onnx.SpeakerEmbeddingExtractor(
    sherpa_onnx.SpeakerEmbeddingExtractorConfig(model=model, num_threads=4))

s = ext.create_stream()
s.accept_waveform(16000, samples)   # float32 mono, 16 kHz
s.input_finished()
embedding = ext.compute(s)          # 192-d

How it was quantized

quantize_static via ONNX Runtime, QOperator format, uint8 activations, int8 per-channel weights, percentile (99.999) calibration over 48 × 4 s clips of in-domain radio audio. Conv and MatMul only; the first encoder block and the attention-pooling score conv stay fp32.

Every one of those choices is load-bearing — several plausible alternatives produce a model that still reports 0.00% EER while being badly degraded (quantizing Softmax/Where collapses the embedding to near-constant; per-tensor weight scales give 0.64 cosine; MinMax calibration 0.65; quantize_dynamic is 7× slower and shifts the impostor mean enough to break a calibrated threshold).

Reproduce, or rebuild calibrated on your own audio:

python scripts/quantize_sv_model.py --audio your_audio.aac

Script, measurements and the full list of failure modes: RadioWhisper — scripts/quantize_sv_model.py, docs/titanet-int8-quantization.md.

License

CC-BY-4.0, inherited from the upstream NVIDIA model. Attribution: TitaNet-large by NVIDIA (NeMo); ONNX export by the sherpa-onnx project (k2-fsa).