openEuler/sherpa-kws

Model Card for Sherpa-ONNX KWS Zipformer zh-en 3M (IB-Robot)

A streaming Zipformer2 transducer keyword spotter (3M parameters, Chinese + English) packaged for the IB-Robot framework and compiled for Ascend 310P1. This is the always-on stage: it runs on the idle robot and decides when to wake the rest of the pipeline (VAD → STT → dialogue → TTS).

Keywords are configured at run time from a text file of token sequences — no retraining is needed to change the wake word.

Deployments

deployment	backend	artifacts
ascend_310p	Ascend ACL (Ascend310P1)	encoder.om, decoder.om, joiner.om
ascend_310b	Ascend ACL (Ascend310B1)	encoder.om, decoder.om, joiner.om
torch_cpu	ONNX Runtime CPU	assets/onnx/{encoder,decoder,joiner}.onnx

Shared contract: tensor_model/zipformer2_transducer_kws/keyword_spotting (host.kws.features [1,45,80] -> host.kws.logit [1,263]), preprocessing kaldi-fbank80-16khz-povey-snipedgesfalse-v1, output keyword-label-on-trigger-v1, logical revision sherpa-onnx-kws-zipformer-zh-en-3M@2025-12-20-chunk16-left64.

Both deployments are stateful: the encoder carries 38 cache tensors across chunks (state_scope: stream, execution_structure: iterative, cancellation_granularity: checkpoint). The manifest declares them as a single state bank encoder.bank, linking each *_in to the matching *_out.

The contract is orchestration_visibility: session, not executor: the caller owns the loop. Each call runs one encoder → decoder → joiner pass over a single chunk; advancing the caches, shifting the feature buffer and running the beam search are the session's job — see assets/reference_impl/live_kws.py.

Tensor contract

encoder — 1 feature input + 38 state inputs, 1 output + 38 state outputs:

x            float32 [1, 45, 80]    45 fbank frames, 80 mel bins
encoder_out  float32 [1,  8, 320]   8 output frames per chunk

The chunk is 45 frames in / 8 frames out and advances by 32 frames (decode_chunk_len = 32), i.e. each step consumes 320 ms of new audio while seeing 130 ms of right context. This is the chunk-16-left-64 streaming variant.

State tensors (all zeros at stream start, fed back verbatim), 6 encoder layers with left_context = [64,32,16,8,16,32], key_dim = [128,128,128,256,128,128], value_dim = [48,48,48,96,48,48]:

cached_key_{i}          [left_context[i], 1, key_dim[i]]
cached_nonlin_attn_{i}  [1, 1, left_context[i], 96]
cached_val1_{i}         [left_context[i], 1, value_dim[i]]
cached_val2_{i}         [left_context[i], 1, value_dim[i]]
cached_conv1_{i}        [1, 128, 7]
cached_conv2_{i}        [1, 128, 7]
embed_states            [1, 128, 3, 19]
processed_lens          int64 [1]        frames consumed so far, +32 per chunk

decoder — y int64 [1, 2] (context size 2) → decoder_out float32 [1, 320].

joiner — encoder_out [1, 320] + decoder_out [1, 320] → logit [1, 263].

Feature extraction

Kaldi-compatible fbank: 16 kHz mono, 80 mel bins, 25 ms window / 10 ms shift, povey window, snip_edges = false, no dithering, no CMVN.

snip_edges = false defines frame centres over the whole signal with mirrored padding, so features cannot be computed per audio block independently — doing so produces wrong frames at every block boundary. The reference implementation keeps a rolling PCM buffer, recomputes fbank over it, and emits only newly-stable frames, dropping the last 3 (their right context has not arrived yet). See assets/reference_impl/live_kws.py.

Repository Structure

• inference_manifest.json — deployment routing (schema v3)
• assets/adapter.json — adapter identity (zipformer2_transducer_kws / keyword_spotting)
• assets/tokens.txt — 263-entry token table, English ARPAbet phones plus pinyin syllables with tone marks (needed to encode keywords)
• assets/keywords.txt, assets/keywords_sensitive.txt — example keyword sets (default and higher-sensitivity thresholds)
• assets/en.phone — English lexicon required to encode new keywords
• assets/onnx/*.onnx — upstream ONNX models (the torch_cpu deployment and the ATC compilation source)
• assets/pretrained-epoch-13-avg-2.pt — the upstream PyTorch training checkpoint the ONNX graphs were exported from (13 MB, averaged weights)
• assets/reference_impl/np_fbank.py — dependency-free numpy fbank
• assets/reference_impl/kws_decode.py — context graph + transducer keyword beam search (max_active_paths=4, num_trailing_blanks=1, unk_id=2)
• assets/reference_impl/live_kws.py — end-to-end streaming driver
• artifacts/ascend_310p/*.om
• artifacts/ascend_310b/*.om — 310B compiled trio

Source Model

The origin is the icefall training repository pkufool/icefall-kws-zipformer-zh-en-3M-2025-12-20 on ModelScope (branch master, declared license Apache-2.0), checkpoint epoch-13-avg-2, streaming variant chunk-16-left-64, logical revision sherpa-onnx-kws-zipformer-zh-en-3M@2025-12-20-chunk16-left64.

The widely-cited download is the sherpa-onnx GitHub release tarball, but that tarball redistributes this repository's onnx/ directory — all three ONNX graphs we ship are byte-identical to the files there, which is how the origin was identified rather than assumed:

shipped as	ModelScope path	bytes	sha256
assets/onnx/encoder.onnx	onnx/encoder-epoch-13-avg-2-chunk-16-left-64.onnx	11,976,272	540ff509ed89bd22afe04bf7049a54bb1c95c6d8a18742ea9691910cdb5f859e
assets/onnx/decoder.onnx	onnx/decoder-epoch-13-avg-2-chunk-16-left-64.onnx	759,829	63a22dd60f40fff082ac3e09afa507f6787da36df76ded2fbe145fa233e22c21
assets/onnx/joiner.onnx	onnx/joiner-epoch-13-avg-2-chunk-16-left-64.onnx	338,154	76f7a24ed0c08633af14b2ee377f747af880d3b65eeba2cd3f31f3380fb73e8d
assets/tokens.txt	data/lang_phone/tokens.txt	1,928	2d3f32311f9b692b964da3c90e830258d3e78e013cb0c992dbfb15cd5a1a71b0
assets/pretrained-epoch-13-avg-2.pt	checkpoint/pretrained-epoch-13-avg-2.pt	13,057,683	62188d073aaf7ec33f85a2c1e1f2a75751658743daeb203f17241ec3bf4ff6eb

Only the encoder distinguishes the streaming variants — decoder and joiner are the same bytes for chunk-8-left-64, chunk-16-left-64 and chunk-16-left-128 in that repository. The variant is therefore pinned by the encoder hash above, and independently confirmed by matching the ONNX metadata (T=45, decode_chunk_len=32) against the streaming driver's constants rather than trusting a file name. chunk-16 is the 320 ms-latency variant; the 160 ms chunk-8 variant exists upstream and is not used here.

The training checkpoint is shipped (assets/pretrained-epoch-13-avg-2.pt, the averaged weights the ONNX export was produced from), matching the convention in the rest of this organisation, where the upstream checkpoint lives under assets/. The repository also holds checkpoint/epoch-13.pt (51,959,636 bytes, full training state including optimizer); that is not needed for inference or re-export and is not shipped.

assets/en.phone is not in the ModelScope repository — it exists only in the sherpa-onnx release tarball, so that remains the source for it:

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/\
sherpa-onnx-kws-zipformer-zh-en-3M-2025-12-20.tar.bz2
# 32,885,699 bytes
# sha256 68447f4fbc67e70eee3a93961f36e81e98f47aef73ce7e7ca00885c6cd3616a6

shipped as	upstream name in archive	bytes	sha256
assets/en.phone	en.phone	3,330,061	f7000ec3a90544c0c7c16090d8951779c2b322e14dad5006290f498567d439ea

The five files taken from that archive (three ONNX, tokens.txt, en.phone) were each verified byte-identical against a fresh download as well, so the two sources agree.

The ONNX export is not byte-reproducible from the checkpoint here: the icefall export script is not part of this bundle. What is pinned is the exact checkpoint, the exact exported graphs, and the fact that the two come from the same upstream commit.

Compiled on the board with ATC (--soc_version=Ascend310P1):

file	bytes	sha256
artifacts/ascend_310p/encoder.om	62,669,509	dc95d61a66677b1e83b566bfc274d2092eba8fc8d4cfbf7c51beddfc0381c20d
artifacts/ascend_310p/decoder.om	476,554	8b4e5a7b986427541ee1c8ced274ddd1d6141b9585be3717973d9db116e1c2eb
artifacts/ascend_310p/joiner.om	216,152	76a21d460a27e7b6424722067365d5697a8c7168f47ab11a62a54939190e20e5

The upstream repository declares Apache-2.0 but publishes no training-corpus description or dataset licence for this release.

Validation

Measured on a real Ascend 310P1 with the streaming driver, 8.5 s of 16 kHz audio containing two wake-word utterances:

chunks              26
mean compute        91 ms / chunk   (OM inference + keyword decode)
audio per chunk    320 ms
RTF                0.29
detections         2.37 s, 5.25 s   (both expected utterances, no false alarms)
backlog            30–50 ms         (stable — the pipeline keeps up)

RTF 0.29 leaves ~3× headroom, and the backlog stays flat rather than growing, which is the real test for a streaming model: it is consuming audio at least as fast as it arrives.

OM vs ONNX, tensor level

Continuous streaming comparison on the same 310P1: both backends run the full 45-frame-in / 32-frame-shift loop over the upstream test_wavs, each keeping its own 38 cache tensors, so any error accumulates exactly as it would in production. ONNX Runtime 1.23.2 CPU fp32 is the reference.

wav	chunks	encoder_out cosine (min)	encoder_out rel_l2 (mean)	worst state cosine	state cosine, last chunk
zh_1.wav	15	0.991565	0.021	0.989623	0.999934
en_0.wav	20	0.998495	0.013	0.997338	0.999178
en_1.wav	51	0.986783	0.017	0.956989	0.999128

decoder cosine 0.99999991, joiner cosine 0.99999992, joiner argmax agreement 1.00 — the OM and ONNX joiners pick the same token every time. processed_lens (the int64 frame counter carried through the state) is exactly equal on every chunk of every file.

The state does not drift. The worst state cosine is reached mid-stream (chunk 1, 6 and 14 respectively) and the value recovers — after 51 chunks of continuous streaming, 16.7 s of audio, en_1 is back at 0.9991. The error is per-chunk numerical noise, not accumulation, which is the failure mode that would actually matter for a model whose caches never reset.

OM vs ONNX, end to end

The number that decides whether the OM is usable is not a cosine, it is whether detections change. Same audio, same decoder, same context graph — only the backend differs:

wav	chunks	OM hits	ONNX hits	identical
en_0.wav	20	LIGHT_UP @ 3.65 s	LIGHT_UP @ 3.65 s	yes
en_1.wav	51	LOVELY_CHILD @ 6.21 s	LOVELY_CHILD @ 6.21 s	yes
zh_0/1/2.wav	17/15/13	none	none	yes
zh_3.wav	24	文森特卡索 @ 2.37 s, 法国 @ 5.25 s	same	yes
zh_4.wav	13	蒋友伯 @ 1.73 s, 女儿 @ 3.65 s	same	yes
zh_5.wav	12	周望军 @ 1.41 s, 落实 @ 2.69 s	same	yes
zh_6.wav	10	朱丽楠 @ 1.41 s, 见面会 @ 3.01 s	same	yes

9 files, 10 detections, zero disagreement — same keyword, same chunk. The three files with no listed keyword produce no hits on either backend, so this covers false alarms as well as misses. The detections also match the upstream ground truth (test_wavs/en_trans.txt, keywords_raw.txt), so both backends are right, not merely consistent.

Usage

from inference_manifest import load_inference_manifest

validated = load_inference_manifest("models/sherpa-kws", "ascend_310p")

Per chunk: run the encoder on 45 fbank frames plus the carried state, feed the 8 output frames to the transducer keyword decoder, keep the returned state, and slide the feature buffer forward by 32 frames.

A standalone run, reading raw 16 kHz mono s16le PCM from stdin:

parec --device=<src> --rate=16000 --channels=1 --format=s16le \
  | python3 live_kws.py --model-dir . --tokens tokens.txt --keywords keywords.txt

Defining your own keywords

The keyword file holds token sequences, not plain text, so a new wake word has to be encoded first. This model is bilingual and needs both the pinyin and the English lexicon:

# keywords_raw.txt: one keyword per line, spaces replaced by underscores,
# and the original text after an "@", e.g.:  n i3 h ao3 @你好
sherpa-onnx-cli text2token \
  --tokens tokens.txt \
  --tokens-type phone+ppinyin \
  --lexicon en.phone \
  keywords_raw.txt keywords.txt

Each line then accepts optional per-keyword overrides: :score boosts the context-graph bonus, #threshold sets the detection threshold for that keyword alone. keywords_sensitive.txt is the same keyword set at a lower threshold — more hits, more false alarms. Tune per keyword rather than globally: a short wake word needs a higher threshold than a long one.

No retraining is involved; changing the wake word is a text-file change.

License

Code and packaging: Apache-2.0. Model weights follow the upstream sherpa-onnx / k2-fsa license (Apache-2.0).

Citation

@article{zipformer2023,
  title   = {Zipformer: A faster and better encoder for automatic speech recognition},
  author  = {Yao, Zengwei and Guo, Liyong and Yang, Xiaoyu and Kang, Wei and Kuang, Fangjun and Yang, Yifan and Jin, Zengrui and Lin, Long and Povey, Daniel},
  journal = {arXiv preprint arXiv:2310.11230},
  year    = {2023}
}
@software{sherpa_onnx,
  title   = {sherpa-onnx: Real-time speech recognition and keyword spotting},
  author  = {{Next-gen Kaldi (k2-fsa)}},
  url     = {https://github.com/k2-fsa/sherpa-onnx},
  license = {Apache-2.0}
}
@software{ib_robot,
  title   = {IB-Robot: Intelligence Boom Robot},
  url     = {https://atomgit.com/openeuler/IB_Robot},
  license = {Apache-2.0}
}