soniqo/Canary-180M-Flash-ONNX

Canary 180M Flash — ONNX

NVIDIA Canary 180M Flash exported to ONNX for on-device speech recognition. FastConformer encoder with an autoregressive Transformer decoder: the encoder consumes a whole utterance, then tokens are decoded one at a time against a cached decoder state. Offline per utterance — there is no streaming mode.

English, German, Spanish and French, with punctuation and capitalisation. Source and target language are separate prompt tokens, so translation between those four is reachable from the same graphs.

Model

Parameters	182M (17-layer FastConformer encoder, 4-layer Transformer decoder)
Format	ONNX (opset 18)
Quantization	FP32, and dynamic INT8 over MatMul/Gemm (per-channel, reduce_range)
Sample rate	16 kHz mono
Features	128-bin log-mel, 512 FFT / 160 hop / 400 window
Vocabulary	5248 SentencePiece pieces
Languages	en, de, es, fr
Tasks	transcription, translation

Files

File	Size	Description
canary-encoder.onnx	461 MB	FP32 encoder + decoder projection
canary-encoder-int8.onnx	174 MB	INT8 encoder
canary-decoder.onnx	316 MB	FP32 cached autoregressive decoder
canary-decoder-int8.onnx	99 MB	INT8 decoder
vocab.json	85 KB	Token id → SentencePiece piece
config.json	4 KB	Decode contract: prompt ids, cache dimensions, feature front end

Both INT8 graphs together are 273 MB; either can be paired with its FP32 counterpart.

Graph contract

encoder: audio_signal [B, 128, T] float32, length [B] int64
      -> encoder_embeddings [B, T', 1024] float32, encoder_mask [B, T'] float32

decoder: input_ids [B, L] int64, encoder_embeddings, encoder_mask,
         decoder_mems [6, B, C, 1024] float32
      -> logits [B, 1, 5248] float32, decoder_hidden_states [6, B, C+L, 1024] float32

The encoder graph includes the projection into decoder width and builds the encoder mask, so a caller passes mel features and gets exactly what the decoder wants.

The decoder emits one position — the newest token — so logits[0, 0] is the distribution to sample. Values are log probabilities (the export keeps the model's log-softmax head), which makes exp(mean log p) over the emitted tokens a usable confidence.

decoder_mems is the decoder cache: pass an empty [6, B, 0, 1024] tensor on the first step with the full prompt in input_ids, then feed decoder_hidden_states straight back with only the newest token. The graph reads its own position offset from the cache length.

Prompt

Canary's decode prompt is nine tokens and is published in config.json as promptIds, with the per-language token ids in languageTokenIds:

<|startofcontext|> <|startoftranscript|> <|emo:undefined|> <src> <tgt>
<|pnc|> <|noitn|> <|notimestamp|> <|nodiarize|>

Swap <src>/<tgt> for another language token to change language, or make them differ to request translation. Read the ids from config.json rather than looking tokens up by string: this vocabulary has no bare-space token (word boundaries are SentencePiece's ▁, U+2581), and a prompt token that silently resolves to -1 produces fluent text that stops after a couple of words or repeats a fragment — a failure that reads like a cache bug.

Front end

Features must follow the NeMo AudioToMelSpectrogramPreprocessor contract the model was trained with; config.json spells it out:

• pre-emphasis 0.97
• torch.stft(n_fft=512, hop=160, win=400, center=True, pad_mode="constant")
• symmetric Hann window (periodic=False)
• Slaney-normalised mel bank, 128 bins
• log(x + 2^-24)
• per-feature normalisation using the sample (N−1) variance, epsilon 1e-5

Performance

FLEURS, greedy decoding, INT8 graphs, ONNX Runtime on CPU with 4 threads (Apple silicon). 100 utterances per language; the Spanish slice had 11 available.

Language	WER	RTF	Latency
English	9.2%	0.024	232 ms/utterance
German	7.2%	0.028	416 ms/utterance
Spanish	2.0%	0.019	130 ms/utterance
French	10.4%	0.027	301 ms/utterance

Quantization, measured over 20 English utterances in a process holding nothing but ONNX Runtime and the graphs. All three produce identical transcripts on that set.

Variant	Graphs	Peak RSS	Latency	Throughput
INT8 encoder + INT8 decoder	273 MB	920 MB	186 ms	50.6× real time
INT8 encoder + FP32 decoder	490 MB	1136 MB	253 ms	37.2× real time
FP32	778 MB	1395 MB	252 ms	37.4× real time

Greedy decode of the fully INT8 pair reproduces the source checkpoint's own transcripts verbatim on the validation clips, and the encoder matches its PyTorch output to 1.3e-6.

Usage

import json
import numpy as np
import onnxruntime as ort

cfg = json.load(open("config.json"))
vocab = {int(k): v for k, v in json.load(open("vocab.json")).items()}

encoder = ort.InferenceSession("canary-encoder-int8.onnx")
decoder = ort.InferenceSession("canary-decoder-int8.onnx")

# features: [1, 128, T] float32 log-mel, see the front-end contract above
enc_emb, enc_mask = encoder.run(
    ["encoder_embeddings", "encoder_mask"],
    {"audio_signal": features, "length": np.array([features.shape[2]], np.int64)},
)

mems = np.zeros((cfg["decoderMemLayers"], 1, 0, cfg["decoderHidden"]), np.float32)
ids = np.array([cfg["promptIds"]["en-en"]], np.int64)
tokens = []
for _ in range(256):
    logits, mems = decoder.run(
        ["logits", "decoder_hidden_states"],
        {"input_ids": ids, "encoder_embeddings": enc_emb,
         "encoder_mask": enc_mask, "decoder_mems": mems},
    )
    best = int(np.argmax(logits[0, -1]))
    if best == cfg["specialTokenIds"]["eos"]:
        break
    tokens.append(best)
    ids = np.array([[best]], np.int64)

text = "".join(vocab[t] for t in tokens if not vocab[t].startswith("<|"))
print(text.replace("▁", " ").strip())

Source

Exported from nvidia/canary-180m-flash (CC-BY-4.0). Prompt ids, cache dimensions and the feature contract are read off the checkpoint at export time rather than hand-written.

Links

• speech-core — C++ runtime
• Docs — runtime docs
• soniqo.audio — website
• blog — blog