TingMix-Large

Speech recognition for Taiwanese Mandarin, English and intra-sentence Mandarin-English code-switching, built on Embed2ASR. Code: github.com/tc3oliver/tingmix.

• Encoder: google/embeddinggemma-2 audio tower (not included; downloaded from Google under its own terms) adapted with LoRA (rank 16, alpha 32, all 12 layers; encoder_delta.safetensors).
• Reader: 22.2M-parameter GRU-CTC head (model.safetensors; CJK characters + 400 English BPE pieces).
• Decoder: CTC prefix beam search with the character 5-gram LM in lm/lmCN_cv_parl_ascend_ntuml.charlm.json.gz (alpha 0.35, beta 1.5, beam 16); the 8 best hypotheses are rescored by a 30.8M-parameter Transformer attention decoder over the Reader states (attention_decoder.safetensors, weight 1.0, on by default; --att-weight 0 = plain CTC + LM). The LM is an n-gram count table built from training transcripts (no audio).
• Streaming: buffered re-decoding of the open speech segment (energy VAD): partial results are revised, final results are fixed.

Use

pip install torch --index-url https://download.pytorch.org/whl/rocm7.1   # or the CUDA / CPU wheel
pip install "tingmix[large,server] @ git+https://github.com/tc3oliver/tingmix.git"
tingmix transcribe audio.wav --model tc3oliver/TingMix-Large --lm lm/lmCN_cv_parl_ascend_ntuml.charlm.json.gz --lm-alpha 0.35 --lm-beta 1.5
tingmix listen --model tc3oliver/TingMix-Large --lm lm/lmCN_cv_parl_ascend_ntuml.charlm.json.gz --lm-alpha 0.35 --lm-beta 1.5   # microphone
tingmix serve  --model tc3oliver/TingMix-Large --lm lm/lmCN_cv_parl_ascend_ntuml.charlm.json.gz --lm-alpha 0.35 --lm-beta 1.5   # WebSocket

Previous version (CTC + LM, no attention decoder): revision v2, e.g. load_model("tc3oliver/TingMix-Large", revision="v2").

Test results (error rate %, single run of this checkpoint, LM + attention rescoring)

Test set	Metric	Embed2ASR-v1 (greedy)	TingMix-Large v2 (CTC + LM)	TingMix-Large v3 (+ attention rescoring)
Common Voice zh-TW 27.0	CER	15.62	12.26	12.02
Taiwan Parliament A	CER	19.17	17.41	17.10
Taiwan Parliament B	CER	19.31	17.86	17.57
ASCEND, code-switched sentences	MER	23.90	20.76	20.02
NTUML2021, code-switched sentences (in-domain: NTUML train used)	MER	29.48	15.15	13.39
FLEURS en_us	WER	24.52	19.88	19.00
LibriSpeech test-clean	WER	8.99	7.89	7.05
LibriSpeech test-other	WER	19.72	17.28	16.11
VoxPopuli en	WER	15.31	13.61	12.78
AMI IHM	WER	23.59	20.90	18.76

Model, LM and decoding weights (incl. the rescoring weight) were chosen on development sets only. Streaming: decode p50 108 ms, final p50 0.76 s after speech end (dev). CER for Mandarin, WER for English, MER (CJK characters + English words) for code-switched sentences; rows whose reference contains digits are dropped, as in the Embed2ASR benchmark. ASCEND speakers are from Hong Kong; Parliament references are not verbatim.

Training data

Common Voice zh-TW and English, Taiwan Parliament speech, LibriSpeech, VoxPopuli, AMI, FLEURS, People's Speech, ASCEND, NTUML2021 (train split). Training audio, transcripts and manifests are not distributed.

Limitations

English remains well behind Whisper large-v3; numbers are spoken-form only (no digits in the output); the streaming mode re-decodes the current segment (GPU recommended); speaker/accent coverage beyond the data above is untested. CPU inference works but is not real time (RTF about 7 on a server CPU; measured in a clean CPU-only install), use a GPU. The default energy VAD (-50 dBFS, adaptive noise floor) misses very quiet speech (below about -55 dBFS) and steady noise at the very start of a stream can produce a short spurious segment.