epoch-discovery/ep-hindi-asr

Hugging Face | Base model | Vaani Benchmark | Vaani toolkit
License: Apache 2.0 | Authors: Epoch Discovery

ep-hindi-asr is a Hindi speech recognition model built on the audio encoder of EmbeddingGemma 2. A reader reads a learned mix of the encoder's layers. Two CTC heads predict Gemma tokenizer units:

• the native head writes Devanagari;
• the Mixed head writes Hindi in Devanagari and English words in Latin script.

Romanized (Hinglish) text and digits come from rule-based post-processing. The model decodes in one pass, with no autoregressive decoder. An optional word 4-gram KenLM is included for beam search.

This release is stage 3. It continues from stage 2 with a larger and more varied data mix, the Mixed head and a larger KenLM.

Model Overview

Parameters	Total	366,199,152 (366.2M)
	Native mode	354.0M: encoder, layer mix, reader and native head
	Mixed head	12.2M (used only in Mixed and Romanized modes)
	Trained in stage 3	111.7M: top 2 encoder layers, layer mix, reader, both heads
Architecture	Encoder	EmbeddingGemma 2 audio encoder, 12-layer Conformer, 25 frames/s (40 ms)
	Layer mix	Softmax-weighted sum of 13 hidden states (1024-d)
	Reader	Linear 1024→768, depthwise conv (k=15), 6 Transformer blocks (768-d, 12 heads)
	Native head	10,539 units (10,537 Gemma tokens + blank + unknown), Devanagari
	Mixed head	23,096 units: the 10,537 native units, 12,557 added Gemma tokens from the Mixed transcripts (English words in Latin script), blank, unknown
	Loss / decoding	CTC; greedy, or beam search with a word 4-gram KenLM (native head)
Input/Output	Input	16 kHz mono audio (other sample rates are resampled)
	Output: Native	Hindi in Devanagari, no punctuation
	Output: Mixed	Hindi in Devanagari, English words in Latin script
	Output: Romanized	Mixed output with Devanagari words romanized (post-processing, romanize.py)
	Digits	Optional inverse text normalization (--digits, itn.py): number words → digits
	Files	model.safetensors (bf16, 0.73 GB, both heads) + config.json; hi_big_4gram.bin (KenLM, 333 MB) + hi_big_unigrams.txt

Benchmark Results

All results use the released checkpoint in bf16 on an NVIDIA A10. The checkpoint and the KenLM settings (alpha 0.3, beta 0.5, beam 32) were selected on dev sets only. "Stage 2" is the previous release of this repository.

Benchmark	Utterances	Metric	Stage 2, greedy	Greedy	+ KenLM
CommonVoice Hindi test	1,727	WER ¹	26.28	23.96	18.05
FLEURS Hindi test	418	WER ¹	21.74	20.57	17.09
IndicTTS Hindi (100-utterance sample)	100	WER ¹	20.89	19.46	16.60
MUCS 2021 Hindi test	3,897	WER ¹	21.69	16.48	13.28
Kathbath Hindi test-unknown ³	1,929	WER ¹	18.52	17.29	12.74
Gram Vaani test	1,032	WER ¹	33.41	32.44	32.22
IndicVoices Hindi valid	5,504	WER ²	18.45	18.63	17.17
Lahaja	6,152	WER ²	18.11	18.24	15.47
Vaani-Benchmark-V1.0, Hindi test	5,050	WER, 3 references, approach 3 ⁴	14.27	14.66	13.10

¹ Vistaar normalization recipe (the six Vistaar test sets). ² Single reference, normalize.py (strict) on reference and hypothesis. ³ Kathbath speakers read sentences from IndicCorp. The KenLM text includes IndicCorp v2, so the Kathbath + KenLM number benefits from the IndicCorp prompts. ⁴ Official Vaani toolkit: clean_text, then the best of 3 human references per segment (approach 3).

Published numbers of other models use other normalizers. Compare numbers only when the normalization is the same. On Gram Vaani (rural telephone speech) the KenLM does not reduce WER, and CER increases (19.93 → 25.68).

Inference

1. Get access and download

The repository and the base model are gated. Accept the terms on both pages first: epoch-discovery/ep-hindi-asr and google/embeddinggemma-2.

pip install -U "huggingface_hub[cli]"
hf auth login
hf download epoch-discovery/ep-hindi-asr --local-dir ep-hindi-asr
cd ep-hindi-asr
pip install -r requirements.txt

transcribe.py loads the weights from model.safetensors and config.json. From google/embeddinggemma-2 it downloads only the configuration files and the tokenizer. Use a GPU in bf16 or a CPU in fp32 (automatic).

2. Command line

# Native (Devanagari), greedy decoding
python transcribe.py --ckpt model.safetensors audio.wav

# Mixed: English words in Latin script
python transcribe.py --ckpt model.safetensors --mode mixed audio.wav

# Romanized (Hinglish): Mixed output, Devanagari words romanized
python transcribe.py --ckpt model.safetensors --mode roman audio.wav

# Numbers as digits (works with every mode)
python transcribe.py --ckpt model.safetensors --mode mixed --digits audio.wav

# Several files in one call (the model loads once)
python transcribe.py --ckpt model.safetensors a.wav b.flac c.mp3

# Native, beam search with the 4-gram KenLM (lower WER; the search runs on the CPU)
python transcribe.py --ckpt model.safetensors --decode lm audio.wav

# CPU only
python transcribe.py --ckpt model.safetensors --device cpu audio.wav

--decode lm uses hi_big_4gram.bin and hi_big_unigrams.txt next to the checkpoint and the settings in config.json (alpha 0.3, beta 0.5, beam 32). --lm, --unigrams, --alpha, --beta and --beam override them. The KenLM is for the native head only. Add --normalize to apply the strict scoring normalizer (normalize.py).

3. Output format

Each line of output is <file>\t<text>\t(RTF <real-time factor>). Example for one file in the three modes:

--mode native   टेबल पे बगल में हमें यहां पे डस्टबिन देखने को मिल रहा है डस्टबिन के साइड में हमें बेड देखने को मिल रहा है
--mode mixed    table पे बगल में हमें यहाँ पे dustbin देखने को मिल रहा है dustbin के side में हमें bed देखने को मिल रहा है
--mode roman    table pe bagal mein hamein yahan pe dustbin dekhne ko mil raha hai dustbin ke side mein hamein bed dekhne ko mil raha hai

4. Python

import torch
from transcribe import load, read_audio, log_probs, greedy, postprocess

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, units, ck = load("model.safetensors", device)    # units: native head; ck["units_mixed"]: Mixed head

wav = read_audio("audio.wav")                           # mono float32, resampled to 16 kHz

lp = log_probs(model, wav, device)                      # native head, CTC log-probabilities, 25 frames/s
print(greedy(lp, units))                                # Devanagari

lp_m = log_probs(model, wav, device, head="mixed")      # Mixed head
mixed = greedy(lp_m, ck["units_mixed"])
print(mixed)                                            # Devanagari + Latin English words
print(postprocess(mixed, mode="roman"))                 # Romanized
print(postprocess(mixed, mode="mixed", digits=True))    # numbers as digits

5. Long recordings

The model was trained on segments of 30 s or less, and transcribe.py decodes each file in one pass. Split longer audio into segments of up to 30 s before transcription. Split at pauses (for example with a voice activity detector), not at fixed times, so that words are not cut.

6. Speed and hardware

One file at a time (batch 1), measured on about 10 minutes of Vaani speech (CPU) and 100 Vaani segments (GPU). Speed includes audio reading and decoding. Model loading takes about 35 s and is not included.

Hardware	Precision	Mode and decoding	Speed	Peak RAM
Intel Xeon Platinum 8358 (A10 host), 8 threads	fp32 (default on CPU)	Native, greedy	20.8x real time	3.2 GB
Intel Xeon Platinum 8358 (A10 host), 8 threads	fp32 (default on CPU)	Mixed, greedy	19.7x real time	3.2 GB
Intel Xeon Platinum 8358 (A10 host), 8 threads	fp32 (default on CPU)	Romanized (--mode roman), greedy	19.7x real time	3.3 GB
Intel Xeon Platinum 8358 (A10 host), 8 threads	fp32 (default on CPU)	Mixed + digits (--digits), greedy	19.7x real time	3.3 GB
Intel Xeon Platinum 8358 (A10 host), 8 threads	fp32 (default on CPU)	Native, KenLM beam 16	18.9x real time	3.8 GB
Intel Xeon Platinum 8358 (A10 host), 8 threads	fp32 (default on CPU)	Native, KenLM beam 32 (default)	17.6x real time	3.9 GB
NVIDIA A10 24 GB	bf16	Native, greedy	227x real time	–
NVIDIA A10 24 GB	bf16	Native, KenLM beam 32 (search on CPU)	61x real time	–

transcribe.py uses fp32 on CPU. bf16 is slower on CPUs without bf16 support: on this CPU, bf16 autocast ran at about 9x real time (about 2x slower than fp32).

7. Notes and limitations

• Precision: run in bf16 (GPU) or fp32 (CPU). fp16 is not supported.
• Attention: use SDPA attention (the default in eg2_model.py). Eager attention in transformers 5.19.0 gives wrong outputs.
• Romanized output: the romanization is rule-based (romanize.py, conventions in CONVENTIONS.md). Hindi has no single standard romanized spelling, so one word can have other valid spellings.
• English words: brand names and rare English words can come out in Devanagari, or with a wrong Latin spelling. The Mixed-head training targets come from the English glosses in Vaani transcripts and from MUCS 2021 Hindi-English; Mixed-mode accuracy is lower on other code-switched speech.
• Numbers: the training transcripts spell numbers as words. Use --digits to get digits.
• Formatting: no punctuation or capitalization.
• Domain: rural telephone speech (Gram Vaani) is the hardest domain.
• Streaming: not supported. Each segment uses its full context.

License and Acknowledgements

The fine-tuned weights and code are released under Apache 2.0. Use of the base model is subject to the terms of google/embeddinggemma-2.

Data note: Training data includes academic-use (Gram Vaani) and MSR open-data (MUCS) sets; check their terms before commercial use.

Training data, each under its own terms: IndicVoices, Kathbath and Shrutilipi (AI4Bharat); Vaani (ARTPARK, IISc); Gram Vaani (OpenSLR 118); MUCS 2021 Hindi (OpenSLR 103) and Hindi-English (OpenSLR 104) (Microsoft Research); FLEURS (Google). Noise augmentation uses MUSAN and simulated room impulse responses (OpenSLR 28). The KenLM text comes from the training transcripts and IndicCorp v2.

The recipe follows QUAN-Transcribe and the idea of transcribing with the EmbeddingGemma 2 audio encoder from Embedding Gemma Transcribe.