embeddinggemma2asr-cantonese: fast Cantonese ASR from a frozen EmbeddingGemma 2 audio encoder

A 20.8M-parameter CTC reader on top of the frozen audio encoder of google/embeddinggemma-2. The encoder is never fine-tuned. On a desktop GPU it transcribes Cantonese at roughly 750–850× real time, about 3× faster than SenseVoice-Small on the same GPU, and it beats SenseVoice-Small on read speech (MDCC).

Inspired by Embedding Gemma Transcribe, which does the same for English.

How it works

16 kHz audio
  → EmbeddingGemma 2 audio_tower (≈300M, frozen, 40 ms frames)
  → take 4 layers: conformer layer 6, layer 9, layer 12 (after output_proj), embed_audio output
  → per-layer PCA to 256 dims (stored in the checkpoint as fixed buffers)
  → learned softmax weighting of the 4 layers (SUPERB-style weighted layer sum)
  → reader: 6-layer Transformer (d=384) + depthwise conv
  → CTC over 12,368 units (Chinese characters + English words)

• Output units: one Chinese character or one English word per unit. Text is Traditional Chinese (Hong Kong) with colloquial Cantonese characters (係、喺、嘅、咗…). No punctuation.
• Training-only auxiliary loss: a Jyutping (syllable + tone) CTC head after reader layer 3, weight 0.3. It is removed from this checkpoint and costs nothing at inference.
• Decoding: greedy CTC, or CTC prefix beam search with an optional character-level 4-gram LM (not included). With the LM weight re-tuned for this model (α=0.3, β=2.5), the LM lowers CER by about 1 point on every test set; greedy is faster.

Results

Character error rate (CER, lower is better), greedy decoding / with the 4-gram LM (α=0.3, β=2.5). SenseVoice-Small was run on the same GPU and the same audio.

Test set	This model (greedy / +LM)	SenseVoice-Small
Held-out mix of read + code-switched speech (1,030 utt.)	10.8% / 9.6%	9.7%
MDCC test (read audiobooks, 2,000 utt., 108 min)	5.3% / 4.5%	7.9%
SpiCE interviews (spontaneous, Vancouver bilinguals, VAD segments)	34.1% / 33.3%	28.0%

Caveats:

• α/β were picked on one half of these test sets; on the other half alone the LM gain is the same (about 1 point).
• The model was trained on MDCC train. MDCC test is in-domain for this model but zero-shot for SenseVoice.
• 1,037 hours of the training data are radio speech labelled by SenseVoice (see below). The model is therefore partly a distillation of SenseVoice and is unlikely to beat it on spontaneous speech.
• SpiCE is the hardest set: spontaneous conversation, heavy code-switching and overseas accents.

Speed. On an RTX 5070 Ti with batch 16, bf16 and greedy decoding, 108 minutes of audio take 7.5–8.8 s (≈750–850× real time), including the encoder. Beam search with the pure-Python LM is much slower, roughly 300× real time.

How we got here

Version	Change	Training data	MDCC (+LM)¹	SpiCE (greedy)
r2	TTS-style read speech	143 h	15.8%	n/a
r3	+ real read speech (MDCC)	≈200 h	5.6%	47.8%
r4	+ teacher-labelled radio speech	720 h	5.7%	38.0%
r5	+ Jyutping auxiliary CTC	720 h	5.2%	37.1%
r6	+ 4 encoder layers (PCA 256 each, fp8 storage)	720 h	4.9%	36.3%
r6b (this model)	+ 518 h more radio speech	1,238 h	4.5%	34.1%

¹ r2–r6 with α=0.7, β=4.5; r6b with α=0.3, β=2.5 (with the old weights r6b gets 4.7%).

Findings along the way:

• Data matters most. Every increase in real or conversational speech gave the biggest gains. Adding 518 hours on top of r6 lowered the validation CER from 7.9% to 6.4%. A bigger reader (33M) did not help at 143 hours.
• Intermediate encoder layers help. The final 512-dim output, which we used originally, received the lowest learned weight; conformer layer 6 received the highest.
• Per-layer PCA to 256 dims plus fp8 storage caused no measurable loss and keeps the features as small as the original 512-dim fp16 ones.
• Re-tune the LM weight when the acoustic model improves. With the old α=0.7 the 4-gram LM made r6b worse on two of three test sets; α=0.3 made it help on all of them.
• Did not help (without fine-tuning): LLM post-correction (Qwen3.5 0.8B/2B, Gemma 4 E4B), and Gemma 4 E2B/E4B as a direct transcriber (42–84% CER). N-best rescoring with Qwen3.5-0.8B gave only 0.4–0.6 points at +45 ms per utterance.

Usage

This checkpoint needs the code in the embeddinggemma2asr-cantonese repository (g2asr package) and the base model google/embeddinggemma-2.

import soundfile as sf
from g2asr.encoder import Encoder
from g2asr.infer import load_reader, transcribe_batch
from g2asr.text import join_units

enc = Encoder()                              # loads google/embeddinggemma-2 (audio part only)
model, units = load_reader("path/to/embeddinggemma2asr-cantonese")   # folder with model.safetensors + config.json
wav, sr = sf.read("speech_16k.wav", dtype="float32")  # mono, 16 kHz, ≤ 30 s per segment
print(join_units(transcribe_batch(enc, model, units, [wav])[0]))

For long recordings, split them first with VAD (scripts/transcribe.py uses silero-vad). For live microphone input, use scripts/live.py.

Training

• Hardware: one Colab A100. The 1,238 h of features (fp8, 107 GB) did not fit in RAM, so training read them from disk (--mmap); 12 epochs (21,660 steps) took about 3.8 h. Batches of up to 60,000 frames, AdamW, lr 1e-3, cosine schedule, SpecAugment-style masking on the features.
• Best validation CER: 6.38%.

Training data (≈1,238 h)

Dataset	Hours	License stated on the dataset page
AlienKevin/wordshk_cantonese_speech	104	CC BY 4.0
AlienKevin/mixed_cantonese_and_english_speech	34	not stated
ziyou-li/cantonese_daily	4	CC BY-NC-ND 4.0
ming030890/mdcc (train)	57	not stated on the HF page; see the MDCC paper
alvanlii/cantonese-radio (gated)	1,037	not stated; radio broadcasts

Radio utterances were kept only when the Whisper and SenseVoice transcripts in the dataset agreed (CER < 25%), the SNR was ≥ 15 dB and the length was 1–30 s. The SenseVoice transcript was used as the label. About 170 of the dataset's 1,015 parquet files were used (every 12th file at offsets 0, 2, 4, 6, 8 and 10).

License and intended use

EmbeddingGemma 2 is Apache-2.0, but some training data is non-commercial (CC BY-NC-ND) or has no stated license (including radio broadcast audio). These weights are therefore released for research and non-commercial use only. Check the terms of each dataset before any other use. This is not legal advice.

Limitations

• Cantonese is the target. English works only for words seen in code-switched training data; Mandarin and other languages were not trained.
• Spontaneous conversation is still weak (≈34% CER on SpiCE).
• No punctuation, no timestamps and no speaker labels.
• Input segments should be ≤ 30 s.