kyle/vietnamese-lalm-asr-lora

Vietnamese LALM ASR — LoRA adapter (r16)

A Large Audio Language Model (LALM) for Vietnamese speech-to-text, built the cheapest way possible: bolt a frozen audio encoder onto a small frozen LLM through a thin trainable projector, then unlock the LLM a little at a time with LoRA.

This repository releases the LoRA + projector adapter (~31 MB) plus a detailed write-up of the training journey, the problems hit, and the measured results. It is primarily a documentation / archival release — see Intended use & limitations below.

Tóm tắt (VI): Model ghép audio encoder (FastConformer, đóng băng) + LLM nhỏ (Sailor 0.5B, đóng băng) qua một projector tuyến tính, rồi mở khóa LLM bằng LoRA. Repo này chia sẻ **adapter

• kinh nghiệm + kết quả** của quá trình fine-tune ba nấc. Chi tiết số liệu ở phần Results.

---

Model summary

Task	Vietnamese automatic speech recognition (with English code-switch and spoken-digit robustness)
Architecture	3-block SALM: audio encoder → projector → decoder LLM
Audio encoder	FastConformer, 108.8M params, frozen (in-house Vietnamese fine-tune)
Decoder LLM	Sailor-0.5B-Chat (Qwen2, 24 layers, hidden 1024, vocab 151936), frozen
Projector	Linear 512 → 1024, 0.53M params, trained from scratch
Trained params	LoRA (r=16) on 7 projections × 24 layers + projector = 8.1M / 736.5M ≈ 1.1%
Framework	NVIDIA NeMo speechlm2 (SALM)

The only trainable weights are the projector and the LoRA matrices. The encoder and the LLM (including its embedding and lm_head) stay frozen — the LLM's 151936-token vocabulary already covers both Vietnamese and English, which is exactly what a closed-vocabulary CTC/transducer decoder cannot do.

---

Intended use & limitations

• What this repo is: the trained adapter (adapter.pt) + a full engineering write-up. It is meant for people studying how to stand up an X-LM (audio/vision/video encoder + LLM) on a small compute budget, and for archiving the exact weights behind the numbers below.
• ⚠️ Not runnable stand-alone. The adapter contains only the LoRA deltas and the projector. To run inference you also need the frozen SALM base: the in-house FastConformer encoder and the Sailor-0.5B-Chat LLM, wired together with NeMo speechlm2. The frozen encoder checkpoint is not released here, so this release does not reproduce end-to-end inference on its own.
• Domain: tuned for Vietnamese, including conversational/telephony-style speech; English technical terms are handled via the LLM's open vocabulary.
• Not for: safety-critical transcription, languages other than Vietnamese, or as a general chat model.

---

Architecture & data flow

Audio 16 kHz mono
   → Preprocessor (mel-80, 25 ms window / 10 ms hop)
   → FastConformer encoder (frozen)  — subsampling ×8, 1 frame ≈ 80 ms, 512-dim
   → Projector 512 → 1024 (trainable)
   → Sailor LLM (frozen + LoRA)  — audio vectors spliced into a fixed text prompt
   → Transcript tokens

Input is audio plus a fixed text prompt containing an audio-locator tag; the projected audio vectors replace that tag inside the prompt, so the task can be changed by editing the prompt rather than the architecture. A 10-second utterance becomes ~126 audio frames fed to the LLM.

---

Training recipe — three-stage curriculum

The guiding principle: unlock parameters in order of return on compute. Align the projector before letting gradients touch the LLM; open the LLM with ~1% LoRA before considering anything deeper.

1. Stage 1 — projector only. Freeze encoder + LLM, align the 0.53M-param projector. Proves the pipeline runs and gives the next stage a warm projector. Ceiling is low (WER 27–71%) because a frozen chat LLM prefers to paraphrase rather than transcribe.
2. Stage 2 — unlock the LLM with LoRA. Insert LoRA (r=16, α=32, dropout 0.05) into all 7 projections per layer (q/k/v/o_proj, gate/up/down_proj). WER drops 2–4× on every set; English code-switch recall jumps thanks to the LLM's open vocab. Two weaknesses surface: spoken-digit recall regresses, and the model ignores context-biasing prompts.
3. Stage 3 — retrain on data that patches the weak spots. Warm-start from the stage-2 adapter; add synthetic spoken-digit data, widen the broad-domain sets, and interleave prompts carrying a term list to teach the context-biasing channel. WER improves on all 12 evaluation sets and the biasing channel starts working.

Optimizer: AdamW, LoRA lr 1e-4, projector lr 2e-4, weight decay 1e-3, cosine schedule with warmup, batch 16, warm-started between rounds. Because the streaming (lhotse iterable) data has no declared length, all control is by step, not epoch — stop by max_steps, validate by integer val_check_interval, and checkpoint the adapter every N steps.

This adapter is the v3 round checkpoint (a full run of the stage-3 curriculum with the enlarged data mix).

---

Results

Word Error Rate (%, lower is better) on public Vietnamese benchmarks. LALM and external baselines are measured on the first 300 utterances per set; the in-house transducer (s7) is measured on the full suite, so gaps under ~1 point should not be read as real differences.

model	vivos	bud500	cv	fleurs	fosd	lsvsc	vlsp	vietmed	vss	digit_eval
Stage 1 (projector only)	27.56	26.12	44.36	54.07	41.74	52.23	71.25	—	—	—
Stage 2 (LoRA)	9.89	10.36	21.64	18.42	20.41	14.51	30.52	26.10	16.52	22.51
Stage 3 (patched)	7.90	8.01	18.70	15.35	17.59	12.04	26.54	23.89	13.31	12.23
This adapter (v3)	6.78	5.51	17.18	14.58	15.73	10.89	24.79	20.91	11.97	9.00
s7 transducer 114M (ref)	9.23	6.92	18.03	15.81	17.95	12.25	24.83	—	—	—
Qwen3-ASR-0.6B	8.48	6.83	16.55	9.14	13.27	11.35	18.60	22.24	33.07	12.85
parakeet-ctc-0.6b-vi	6.59	7.95	10.73	12.15	8.91	10.24	16.06	23.28	26.72	7.66
chunkformer-large-vie	2.78	4.82	8.39	10.42	5.21	5.98	9.98	15.73	12.42	8.23

Specialized recall (%, higher is better) — English code-switch terms and spoken digits:

model	CS-recall (seen terms)	CS-recall (unseen terms)	digit-recall
This adapter (v3)	52.10	15.38	90.68
— with oracle term hint	—	38.46	—
s7 transducer (closed vocab)	26.74	12.53	—
Qwen3-ASR-0.6B	30.17	31.66	86.74
chunkformer-large-vie (closed vocab)	11.38	6.59	93.99

Reading the numbers:

• With ~1.1% trainable parameters, the LALM matches and on 4 sets beats the same-encoder transducer (s7), and reaches strong WER on the in-house conversational set (vss 11.97).
• Open vocabulary is the real differentiator. Code-switch recall (52.10) is nearly double the closed-vocab transducer (26.74) — chunkformer wins broad-domain WER but its CTC decoder is stuck at 11.38 on code-switch. This capability cannot be reached by fine-tuning a closed-vocab model further; it needs the LLM decoder.
• Broad-domain Vietnamese still has clear public champions (chunkformer leads 8/12 sets) — the gap is data and training scale, not architecture.
• The context-biasing channel works when trained: feeding the correct term list at inference lifts unseen-term recall from 15.38 to 38.46 (oracle upper bound).

---

Key lessons (the "experience" part)

• The projector is only a coupling. To change the LLM's behaviour you must let gradients reach it — a 0.5M-param linear layer alone cannot teach a chat LLM to transcribe verbatim.
• Changing the decoder changes the error distribution. Re-measure every specialized skill (e.g. digit recall) after swapping architecture; the aggregate WER hides regressions.
• Measure the oracle before training a control channel. If injecting the answer into the prompt moves the metric, the channel is worth training; if the oracle does not move, stop early.
• Patch weak skills with targeted synthetic data — far cheaper than collecting more real audio. Synthetic digit data halved WER around digit strings, though lifting exact per-digit recall past a strong closed-vocab baseline needs more/diverse data.
• Control training by step, not epoch, for any length-less streaming data pipeline.
• Checkpoint the trained params periodically — the cheapest insurance against infra failures; here it is a 31 MB file, and it let a hung run resume by warm-starting from the last checkpoint.
• Standardize a results registry (one JSON schema) from day one so in-house and external models live in one comparison table; a new measurement campaign is just adding a row.
• The whole recipe is modality-agnostic — swap the audio encoder for a vision/video encoder and the three-block + curriculum recipe stays the same.

---

Files in this repo

• adapter.pt — the trained adapter (~31 MB): projector weights + LoRA matrices for the frozen Sailor-0.5B-Chat LLM. Load into a NeMo speechlm2 SALM whose encoder is the frozen FastConformer and whose LLM is sail/Sailor-0.5B-Chat.
• README.md — this card.

---

Acknowledgements & license

• Decoder LLM: Sailor-0.5B-Chat (Apache-2.0).
• Framework: NVIDIA NeMo speechlm2.
• Audio encoder derived from the NeMo FastConformer family, fine-tuned in-house on Vietnamese.
• Public evaluation benchmarks: VIVOS, VietBud500, Common Voice, FLEURS, FOSD, LSVSC, VLSP2020, VietMed, VietSuperSpeech.

Released under Apache-2.0 (matching the Sailor base and NeMo framework). The adapter weights are the authors' own contribution.