luweigen/canary-1b-v2-fi-en-android

Canary-1B-v2 — ONNX, Qualcomm NPU and LiteRT GPU files for on-device Finnish ↔ English speech translation (Android)

Source

• Original model: nvidia/canary-1b-v2 (CC-BY-4.0), revision d455706339a6b32e1aa40f82c713a482a0c938e2, NVIDIA's multilingual speech recognition and speech translation model (FastConformer encoder + Transformer decoder, 25 European languages ↔ English). Weights were taken from its Hugging Face Transformers format (CanaryForConditionalGeneration, fp32) and exported to ONNX by the transvoice-android project. The LiteRT (.tflite) encoder chunks were converted from the same PyTorch weights with litert-torch 0.10.0 (fp16 weights).
• nemo128.onnx (log-mel front end) is an unchanged copy of the file shipped with onnx-asr 0.12.0 (also used by istupakov/canary-1b-v2-onnx).
• This repository is used by the native Android app of the transvoice-android project. The app downloads the files and checks each one against a sha256 value compiled into the APK. See the original model card for intended use, limitations and the license terms. Only Finnish → English and English → Finnish were evaluated for this conversion.

Files

File	What
nemo128.onnx	16 kHz audio → 128-bin log-mel features (NeMo preprocessing)
tokens.txt	Vocabulary for decoding: id, piece, 1 if special (16384 lines). Text = non-special pieces concatenated, ▁ → space
encoder.15s.full.ns_ctx.onnx + encoder.15s.full.ns_ctx_qnn.bin	Encoder with a fixed 15 s window (features[1,1501,128] → encoder_out[1,188,1024] and the cross-attention keys / values of all 8 decoder layers cross_k, cross_v [8,1,8,188,128]). The attention mask is folded into constants (the window is always full; shorter speech is padded with −80 dBFS noise), and every LayerNorm input is pre-scaled by a power of two so that x² stays inside the fp16 range; compiled offline to a Qualcomm QNN HTP context (fp16, Snapdragon 8 Elite / SM8750, HTP v79) with onnxruntime-qnn 2.7.0 / QNN 2.51
encoder.15s.full.w8a16.pc_ctx.onnx + encoder.15s.full.w8a16.pc_ctx_qnn.bin	The same encoder (without the LayerNorm pre-scaling) statically quantized for the HTP: uint8 weights per output channel, uint16 activations (QDQ, MinMax calibration on 16 FLEURS Finnish windows), compiled to a QNN HTP context. Half the size of the fp16 context, same translation quality in our tests (FLEURS fi→en BLEU 26.5 vs 26.8), about 40 ms slower per window. Used by the app's default htp and htp-cpu profiles; the fp16 context is the optional htp-fast profile
encoder.15s.full.int8.onnx	The same encoder for the CPU: MatMul weights dynamically quantized to int8 (per tensor)
decoder.step.L256.onnx	Decoder, one token per call, with the self-attention KV cache as inputs / outputs (fixed length 256): input_id, position, self_k, self_v [8,1,8,256,128], self_mask, cross_k, cross_v, encoder_mask → logits[1,16384], new_k, new_v [8,1,8,1,128]. fp32
decoder.step.L128.te188_ctx.onnx + decoder.step.L128.te188_ctx_qnn.bin	The same decoder with KV cache length 128 and the encoder length fixed to 188 (15 s window), compiled to a QNN HTP context (fp16, SM8750); 27% faster per step than the L256 version with identical output. Used by the app's htp profile
encoder.15s.c0of4.fp16.tflite, encoder.15s.c1of4.fp16.tflite, encoder.15s.c2of4.fp16.tflite, encoder.15s.c3of4.row.fp16.tflite, encoder.15s.cross.fp16.tflite	The same encoder for mobile GPUs with LiteRT 2.3.2 (CompiledModel API, OpenCL GPU accelerator), split into 4 chunks of 8 layers each because compiling the whole encoder for the GPU needs more memory than a phone with 8 GB has. Chunk 0: features [1,1501,128] → hidden [1,188,1024]; chunks 1–3: hidden → hidden (the output of chunk 3 is encoder_out); cross: hidden → cross_k, cross_v [8,8,188,128] (same memory layout as the ONNX [8,1,8,188,128]). fp16 weights; no attention mask input (the window is always full; a mask made the GPU compilation fail); LayerNorm inputs pre-scaled by powers of two as in the HTP version. In chunk 3 (layers 24–31) a fixed scale made the variance of low-energy frames underflow in fp16 (NaN on the GPU), so its LayerNorms instead divide each frame by its own max|x| at run time (.row, mathematically equivalent); everything runs in fp16. Measured on a OnePlus 6 (Snapdragon 845, Adreno 630): 1.11 s per 15 s window (int8 encoder on the CPU: 3.9 s); encoder_out matches the fp32 ONNX encoder to cosine 0.9998. Used by the app's gpu profile (with the CPU decoder decoder.step.L256.onnx)
decoder.step.L256.te188_ctx.onnx + decoder.step.L256.te188_ctx_qnn.bin	Earlier HTP version with KV cache length 256 (used by app builds before 2026-10-20 evening)

The prompt is ▁ <|startofcontext|> <|startoftranscript|> <|emo:undefined|> <|src|> <|tgt|> <|pnc|> <|noitn|> <|notimestamp|> <|nodiarize|> (ids 16053 7 4 16 <src> <tgt> 5 9 11 13, Finnish 70, English 64; use src = tgt for speech recognition). Note the leading ▁ (16053): the Transformers chat template of this revision leaves it out, and without it English → Finnish quality drops and outputs repeat.

Measured quality and speed (Finnish → English, FLEURS test, 100 sentences as one 25-minute stream, energy-VAD segments)

Device	Configuration	Corpus BLEU / chrF	Real-time factor
Host, fp32	reference	26.9 / 68.4	—
realme RMX5090 (Snapdragon 8 Elite)	encoder + decoder on the HTP	26.8 / 68.3	0.05
OnePlus 6 (Snapdragon 845)	int8 encoder + fp32 decoder on the CPU	27.1 / 68.5	0.6
OnePlus 6 (Snapdragon 845)	LiteRT GPU encoder (fp16) + fp32 decoder on the CPU	26.6 / 68.3	0.32

The HTP encoder matches the fp32 encoder to cosine 0.98–0.99 per window; the int8 encoder 0.97–0.98. Translations differ in single words for about a third (HTP) to a half (int8) of the segments, with no change in the corpus scores.