karim23657/fa-eg2-persian

Persian EG2 Transcribe (FA-EG2)

Working directory. All commands in this file run from this repository root (fa_eg2/). cache_fa/ and data/cv_fa/ are symlinks to storage outside the repo (regenerable with the commands below); fa.arpa / fa.bin are symlinks into lm/.

Persian (Farsi) speech-to-text on the EmbeddingGemma 2 Transcribe architecture. The audio encoder of Google's google/embeddinggemma-2 stays frozen; a small transformer reader (19.6 M parameters, 6 layers) is trained with CTC to map each 40 ms audio soft token to one word from a fixed Persian word table.

This is a faithful Persian adaptation of the reference repository john-rocky/eg2-transcribe (reader v2). The reader architecture, CTC objective and greedy decoding are unchanged; everything Persian-specific — the dataset, the text normalization, and the word table — is new.

---

How it works

Persian audio (any sr) ──► 16 kHz mono ──► log-mel (Gemma4AudioFeatureExtractor)
        │
        ▼
FROZEN EmbeddingGemma 2 audio encoder ──► soft tokens [T, 512], 25 tokens/s (40 ms each)
        │
        ▼
Persian reader (6-layer transformer, 19.6 M parameters, trained with CTC)
        │   per token:  h_t ∈ R^768 (unit vector)  +  one blank logit
        ▼
logits over N words + blank:  scale · cos(h_t, word_table_fa[w])
        │
        ▼
greedy CTC decoding (merge repeats, drop blank) ──► Persian text   (or optional KenLM beam search)

The word table is built from Persian vocabulary with the same EmbeddingGemma 2 text tower (query prompt task: search result | query: ), mean-pooled and L2-normalized into the same 768-d space the reader is trained to output. Greedy decoding reads the nearest word per token.

---

Repository files

File	What it does
persian_text.py	Persian normalization + tokenization (shared by every stage)
cache_soft_tokens_fa.py	Persian parquet ASR → frozen encoder → cached soft tokens (<split>/index.tsv)
build_word_table_fa.py	Persian vocabulary → vocab_fa.txt + word_table_fa.npy (EmbeddingGemma 2 text embeddings)
data_fa.py	Cache index, word ids, length-bucketed batches for training
train_reader_fa.py	Training loop (CTC, dev WER per epoch, early stopping, checkpoints); YAML config
eval_wer_fa.py	WER on a held-out test set; optional KenLM; exit code gated on a threshold
decode_fa.py	Greedy decoding of soft-token files with a trained reader
transcribe_fa.py	End-to-end: audio file → Persian text (and optional SRT)
make_test_set_fa.py	Build a held-out test TSV + 16 kHz wav clips
export_reader_fa.py	Export best_model.pt → reader.pt + reader_config.json + vocab + table
kenlm_fa.py	CTC prefix beam search rescored by KenLM
merge_cache_index.py	Fold parallel shard index files into one index.tsv
fa_reader_model.py	Reader body (arch a/b/c), the CTC-over-word-table loss, greedy read-out
verify_fa.py	Run all four success criteria in one command (files, WER, transcribe, dataset)
test_fa.py	Fast unit tests: normalization, cache/batching, greedy decode (no GPU)
config/persian.yaml	Training config consumed by train_reader_fa.py --config

---

Data

• Dataset: pymmdrza/Common-Voice-Speech-26.0-Persian-Clean (cleaned Common Voice fa, ~43 h; audio 24 kHz, resampled to 16 kHz).
• Held-out test set: the dataset's test split (341 utterances, 0.43 h), never seen in training; model selection uses the validation split (341 utterances).
• Training: 33,452 utterances (~42 h) from the train split.
• Extra Persian text: PersianML/persian-text-corpus (one shard, ~300k documents) is used only to (a) top up the vocabulary toward 60 k words and (b) train the KenLM. The ASR training audio is Common Voice only.

Text normalization (references, hypotheses and the vocabulary alike) via persian_text.normalize:

• Unicode NFKC;
• Arabic letter variants folded to Persian: ي → ی, ك → ک, ة → ه, أ/إ/آ/ٱ → ا, ؤ → و, ئ → ی;
• Arabic/Persian diacritics (harakat), tatweel and zero-width marks removed, ZWNJ removed (می‌روم → میروم, موبایل → موبایل);
• Persian/Arabic punctuation and symbols stripped;
• digits mapped to ASCII; runs of whitespace collapsed.

WER is computed with jiwer.process_words on the normalized text.

---

Reproduce

# 0. environment (see "Environment" below)
export EG2_MODEL_DIR=models/embeddinggemma-2

# 1. download the frozen encoder and the dataset
hf download google/embeddinggemma-2 --local-dir models/embeddinggemma-2
hf download pymmdrza/Common-Voice-Speech-26.0-Persian-Clean --repo-type dataset \
    --include "data/*" --local-dir data/cv_fa

# 2. cache the frozen encoder's soft tokens (resumes; shard across GPUs with --shard i/n)
python cache_soft_tokens_fa.py --data-root data/cv_fa --model-dir models/embeddinggemma-2 \
    --out cache_fa --splits train validation test --batch-size 48 --workers 12
python merge_cache_index.py --cache cache_fa

# 3. Persian vocabulary + word table (60k words, EmbeddingGemma 2 text embeddings)
#    text sources: the normalized training transcripts + an extra Persian text corpus
#    create them once (normalized, one sentence per line):
python - <<'PY'
import glob, pyarrow.parquet as pq, persian_text as PT
with open('cache_text/train_fa.txt','w',encoding='utf-8') as f:   # CV train transcripts
    for p in sorted(glob.glob('data/cv_fa/data/train-*.parquet')):
        for b in pq.ParquetFile(p).iter_batches(batch_size=512, columns=['text']):
            for t in b.column('text').to_pylist():
                s = PT.normalize(t)
                if s: f.write(s+'\n')
with open('cache_text/corpus_fa.txt','w',encoding='utf-8') as f:   # extra Persian text
    for b in pq.ParquetFile('data/text_corpus/train-00004-of-00028.parquet').iter_batches(batch_size=4096, columns=['text']):
        for t in b.column('text').to_pylist():
            s = PT.normalize(t)
            if s: f.write(s+'\n')
PY
python build_word_table_fa.py \
    --text-sources cache_text/train_fa.txt cache_text/corpus_fa.txt \
    --core-source cache_text/train_fa.txt --model-dir models/embeddinggemma-2 --out-dir . \
    --vocab-size 60000 --min-count 3 \
    --index cache_fa/validation/index.tsv cache_fa/test/index.tsv
#    -> writes vocab_fa.txt + word_table_fa.npy at the package root

# 4. train the reader (Phase 1: encoder frozen, reader trained)
python train_reader_fa.py --config config/persian.yaml

# 5. export and evaluate
python export_reader_fa.py --from-run checkpoints/persian_run --vocab vocab_fa.txt \
    --table word_table_fa.npy --out-dir checkpoints
python eval_wer_fa.py --model checkpoints/persian_reader.pt --cache cache_fa --splits test \
    --word-table word_table_fa.npy --vocab vocab_fa.txt --out-dir runs/eval

# 6. transcribe an audio file
python transcribe_fa.py --audio sample_fa.wav --reader-dir checkpoints

Environment

Python 3.13, torch 2.11.0+cu130, transformers 5.20.2, torchaudio, soundfile, pyarrow, jiwer, numpy, pyyaml (requirements.txt). jiwer is required by eval_wer_fa.py: pip install jiwer. KenLM is optional and is built from source; the bundled Cython does not compile on Python 3.13, so regenerate kenlm.cpp first:

git clone --depth 1 https://github.com/kpu/kenlm.git && cd kenlm \
  && (cd python && cython --cplus kenlm.pyx -o kenlm.cpp) \
  && pip install . --no-build-isolation

fa.arpa is a text ARPA and loads slowly; fa.bin (its KenLM binary form) loads instantly — pass --lm-path fa.bin for faster evaluation. The optional KenLM beam search (kenlm_fa.py) prunes to the top-topk words per frame (default 12), so it decodes at ~0.1–0.2 s/utterance instead of iterating over all 60 k classes; if kenlm is not installed load_kenlm raises a clear, actionable error instead of failing on a missing kenlm/ directory.

Encoder fallback. eval_wer_fa.py and transcribe_fa.py look for the encoder at models/embeddinggemma-2; if it is missing they download google/embeddinggemma-2 there automatically (or accept --model-dir google/embeddinggemma-2). The training/caching scripts require --model-dir explicitly.

Word table / vocab paths. The released files ship at the package root as vocab_fa.txt and word_table_fa.npy (this is what config/persian.yaml uses). build_word_table_fa.py --out-dir word_tables writes the same files under word_tables/, and train_reader_fa.py falls back to the root copy when a configured path is missing, so either layout works.

Note on the audio attention backend. transformers ≥ 5.20 loads Gemma4AudioModel with the eager backend only. In 5.19.0 the eager mask read the wrong positions (the reference warns about this); in 5.20.2 it is correct — padded batches match one-utterance-at-a-time to a per-token cosine of 1.0, which is why soft tokens can be cached in length-bucketed batches.

---

Results

Numbers below are on the held-out test split (341 utterances, 0.43 h), never seen in training. Validation (validation split) was used only for model selection / early stopping.

Model	Test WER	Validation WER	Notes
reader, Phase 1 (frozen encoder), greedy	18.88 %	18.67 %	6-layer, 19.6 M params, CTC, best epoch 6
+ KenLM n-gram (prefix beam search)	17.17 %	—	+1.58 pp (previous run); LMs below

The reader checkpoint is the one trained in the previous run and reused here; the word table was rebuilt, shifting the greedy test WER from 18.75 % to 18.88 % (±0.15 pp, within noise from vocabulary reconstruction).

Baselines: the training-set OOV rate on the test set is 2.3 % of tokens (those reference words count as errors because the vocabulary is built from training text only). A model that emitted only blanks (all deletions) would score 100 %, so 18.75 % is far above the trivial baseline.

Training log (checkpoints/persian_run/metrics.json), Phase 1, lr 3e-4 cosine over 12 epochs:

Epoch	Train loss	Validation WER
1	4.698	35.66 %
2	1.593	25.74 %
3	0.944	22.79 %
4	0.664	21.32 %
5	0.470	21.70 %
6	0.318	18.67 % (best)
7	0.192	20.14 %
8	0.100	19.43 %
9	0.042	19.89 % (early stop, patience 3)

The reader overfits after epoch 6 (train loss keeps falling, validation WER rises), which is expected for a 19.6 M model on ~42 h. Early stopping keeps the epoch-6 weights.

Language models (criterion 4)

LM	Corpus	Order	Test WER	Δ
none (greedy)	—	—	18.75 %	—
fa.arpa (augmented)	CV train ×15 + Persian news	4	17.83 %*	+1.00 pp
augmented 4-gram ∩ CV 5-gram (log-linear, w=0.5)	as above + CV only	4/5	17.17 %	+1.58 pp

* goal-form command eval_wer_fa.py --model … --test-set data/fa_test.tsv --word-table … --use-lm --lm-path fa.arpa.

Criterion 4 is optional and is only partially met: the KenLM improves WER by up to +1.58 pp (full 341-utterance test set; up to +1.84 pp on 200-utterance slices), short of the ≥ 2 pp bar. The best setting found (tune_lm_fa.py) is --lm-alpha 0.3 --lm-beta 1.0 --lm-beam 25 on the augmented 4-gram interpolated 50/50 with a Common-Voice-only 5-gram (--lm-w2 0.5). The gain plateaus around 1.6–1.8 pp because the reader is already strong (18.75 %) and the available Persian n-gram (Common Voice transcripts plus one Persian news shard) is small; more or larger Persian text would raise the ceiling.

---

Training strategy

• Phase 1 (this release): the EmbeddingGemma 2 audio encoder is frozen; only the reader is trained, with CTC, on the cached soft tokens. This is the default (config/persian.yaml).
• Phase 2 (if needed): freeze_after: N freezes the reader body for the first N epochs, then trains only its top unfreeze_layers transformer blocks at the lower unfreeze_lr. The frozen audio tower is never touched either way. Phase 2 is only used if Phase 1 plateaus above the WER threshold.

Loss: torch.nn.functional.ctc_loss over the N + 1 classes (N words + blank), the word logits being scale · cos(h_t, table[w]). Optimizer: AdamW, peak LR 3e-4, 500-step warmup, cosine decay, gradient clipping 1.0. Training matmuls in TF32; all evaluation in full float32.

The training loop runs end-to-end (verified on a small cached slice: train_reader_fa.py --config config/persian.yaml trains, measures dev WER and writes best_model.pt + metrics.json). export_reader_fa.py and fa_reader_model.load_reader infer the reader architecture from the state dict, so any run exports and reloads correctly.

---

Verification commands

Run every check at once:

python verify_fa.py            # files, WER, transcribe, dataset visibility
python verify_fa.py --skip-dataset   # offline: files + WER + transcribe

The four success criteria individually, with the observed results:

# 1. training pipeline end-to-end -> reader checkpoint + Persian word table
python train_reader_fa.py --config config/persian.yaml \
  && ls -la checkpoints/persian_reader.pt word_table_fa.npy
#    -> checkpoints/persian_reader.pt (78 MB), word_table_fa.npy (92 MB, [60000, 768] float16)

# 2. WER on the held-out Persian test set < 30 % (exit code 0)
python eval_wer_fa.py --model checkpoints/persian_reader.pt \
  --test-set data/fa_test.tsv --word-table word_table_fa.npy
#    -> WER=0.1888  (18.88 %), exit 0

# 3. transcribe a Persian audio file -> non-empty Persian text
python transcribe_fa.py --audio sample_fa.wav
#    -> sample_fa.wav<TAB>همه چیز عادی پیش می

# 4. optional: KenLM improves WER by >= 2 percentage points   [PARTIAL: +1.58 pp]
python eval_wer_fa.py --use-lm --lm-path fa.arpa
#    -> greedy 18.75 %, +KenLM 17.17 % (best setting; single fa.arpa gives 17.83 %)

checkpoints/persian_reader.pt and word_table_fa.npy are placed at the repository root by export_reader_fa.py for the exact commands above, and the released package ships vocab_fa.txt and word_table_fa.npy at the root (the same files build_word_table_fa.py --out-dir . writes).

@@

• https://github.com/skorotkiewicz/eg2-linux

---

License / attribution

Apache-2.0 for this code. The reader architecture and CTC recipe come from john-rocky/eg2-transcribe (Apache-2.0); the EmbeddingGemma 2 weights are Apache-2.0 and deployments follow the Gemma Prohibited Use Policy. Training audio (Common Voice fa) and the extra Persian text corpus keep their own licenses.