whisper-edge

Whisper encoders compiled for edge accelerators, plus the CPU decoders they pair with.

Consumed by openvoicestream's voxedge.backends.whisper backend; server/core/model_downloader.py fetches from this repo. Measurements across five boards, and the reasoning behind each choice, are in docs/perf/whisper-cross-device-20260827.md there.

Layout

Path	For
encoder/hailo/tiny-whisper-encoder-10s_15dB.hef	Hailo-8, tiny, 10 s window
encoder/hailo/base-whisper-encoder-5s.hef	Hailo-8, base, 5 s window
encoder/rk/whisper_encoder_base_10s.rknn	RK3588 / RK3576, base, 10 s window
encoder/rk/whisper_encoder_base_20s.rknn	RK3588, base, 20 s window
encoder/jetson/enc_base_30s.onnx	Jetson — input for an on-device TensorRT build
decoder/base/	Shared by every base encoder above
decoder/tiny/	Hailo tiny only
mel_80_filters.txt, vocab_en.txt, vocab_zh.txt	Shared front end and vocabulary

Notes

The decoder runs on the CPU deliberately. Neither vendor's NPU decoder has a KV cache — Hailo compiles a fixed 32-token sequence, Rockchip a 12-slot sliding window — so both recompute the whole sequence every autoregressive step. Running the encoder on the accelerator and the decoder as a CPU ONNX graph measured both faster and more accurate on every board: RK3588 English long-form went 10.44% → 7.58% WER while RTF went 0.149 → 0.061.

tiny and base decoders are not interchangeable (4 layers / d384 against 6 / d512). Crossing them produces fluent nonsense rather than an error.

Build the Jetson engine with --bf16, not --fp16. The fp16 build of that encoder graph scores cosine 0.826 against onnxruntime and fails silently — the decoder emits fluent text that drifts off-topic. bf16 scores 0.9996 and matches fp32's error rates. Passing --fp16 --bf16 together yields an engine bit-identical to the pure fp16 one.

The window is fixed at compile time. It must match the seconds in the filename; rknn-lite does not validate it, and a mismatch reinterprets the buffer and returns plausible nonsense rather than an error.

Whisper here is for English. Chinese CER measured 35–56% across every board, which is Whisper base/tiny's own ceiling rather than anything a faster accelerator fixes. Use Paraformer, SenseVoice or Qwen3-ASR for Chinese.

Provenance

The Hailo HEFs come from Hailo's own edge_whisper example assets and the RKNN encoders from Rockchip's model zoo conversion flow; the ONNX decoders are optimum exports of OpenAI Whisper. Original licences apply to each.