sujitvasanth/Parakeet-110M-MNN-INT8

Parakeet 110M MNN INT8

Experimental MNN conversion of NVIDIA Parakeet TDT/CTC 110M. Includes separate encoder, recurrent decoder and TDT joiner with INT8 compression of supported weights. This is a fixed-length prototype, not a drop-in sherpa-mnn bundle or a finished Android integration.

Model files

File	Bytes	Inputs → outputs
encoder-int8.mnn	110145604	float32 mel [1,80,400] → encoded [1,50,512]
decoder-int8.mnn	5971544	int32 token [1], float32 hidden/cell [1,640] → new hidden/cell [1,640]
joiner-int8.mnn	1430456	float32 encoder [1,512], decoder [1,640] → logits [1,1030]

Total: 117,547,604 bytes (112.1 MiB). INT8 refers to compressed supported weights, not guaranteed INT8 arithmetic or all-weight quantization. Embedding and other operations may remain floating point.

The encoder accepts exactly 400 mel frames, approximately four seconds, for one unpadded utterance. Variable-length input, masking and streaming are not implemented. Padding shorter recordings or processing unrelated chunks does not preserve full-context behavior.

Inference contract

Audio: 16 kHz mono. NVIDIA configuration is included in original_config.yaml: 80-band log-mel features, 25 ms Hann window, 10 ms hop, FFT 512, pre-emphasis and per-feature normalization. Conversion scripts do not include an audio frontend in the MNN encoder.

Decoder is an explicitly expanded LSTM step. Initialize hidden/cell to zero, start with blank token 1024. Joiner logits 0–1024 are tokens (blank 1024); logits 1025–1029 represent durations [0,1,2,3,4]. Use TDT decoding, not a Zipformer RNNT loop. The original checkpoint also contains a CTC head, but that head is not shipped as an MNN file here.

Validation and CUDA patch

All three MNN models executed on Windows CPU. Float and INT8 MNN TDT transcription matched an independent Parakeet reference on one 3.99-second speech sample:

Well, I don't wish to see it any more, observed Phoebe, turning away her

CUDA testing used RTX 3060 and CUDA 12.4. The local MNN CUDA low-memory implementation had a pointwise-convolution row-count bug. Apply cuda-low-memory-fix.patch to the matching MNN source and rebuild before using this tested configuration: MNN_FORWARD_CUDA, Precision_High, Memory_Low, MNN_LOW_MEMORY=ON.

With the patch, the INT8 encoder produced the reference sentence via a separately evaluated CTC head. All three components executed on CUDA, but a complete CUDA TDT transcription loop was not tested. Observed backend assignments were CUDA; these are not instruction-level hardware traces.

Measured encoder MNN session memory was 116.512 MiB; device free-memory delta after context initialization was 128 MiB, compared with 428 MiB in normal-memory mode. Device-wide free-memory changes are not exact process peak-resident measurements. No speed benchmark is claimed. Other precision modes have not been qualified after the fix.

Android, Mali and Adreno/OpenCL inference remain untested. The CUDA patch does not apply to OpenCL. One speech sample is not a WER evaluation or evidence of general accuracy.

Provenance and reproduction

Original architecture and training: NVIDIA parakeet-tdt_ctc-110m, CC-BY-4.0.

Source weights: AbijahKaj/parakeet-tdt-110m-web, revision c42e5aff702f0129eff7c081c0d0db5ee6db83bc, FP16 safetensors derived from NVIDIA. Credit to NVIDIA and AbijahKaj; this repository performs format conversion and supported-weight compression, not fine-tuning.

Graphs were reconstructed with PyTorch and checked against NVIDIA's architecture configuration. ONNX serves only as an intermediate graph format. MNN converter version 3.6.1 with --weightQuantBits 8. Download the pinned source model.safetensors beside export_graphs.py, run that script, then run convert_mnn.py in an environment with MNN's _tools extension. Converter optimization choices and runtime source changes can affect reproducibility.

Independent reference: csukuangfj/sherpa-onnx-nemo-parakeet_tdt_ctc_110m-en-36000, CTC model and test_wavs/0.wav. See included validation JSON files for the saved sample results.