mukherjee78/abc-voice-slm-asr

From-scratch Zipformer-Transducer ASR (English + Bengali + Hindi)

A speech recognizer implemented from first principles — a Zipformer-style encoder and a stateless RNN-Transducer predictor/joiner, hand-written in PyTorch (only the RNN-Transducer loss itself uses an established library call, torchaudio.functional.rnnt_loss — see the source repository for exactly why). Pretrained from scratch on English (LibriSpeech), then fine-tuned onto Bengali and Hindi via cross-lingual vocabulary extension.

This is a research/teaching artifact, not a production ASR system. See Limitations before using it for anything that matters.

• Code: not yet public — the GitHub push is pending resolution of two release-housekeeping items on the maintainer's end. This model card will be updated with the repository link as soon as it's live.
• License: Apache 2.0 (code and English checkpoint). See Training data & license for the Bengali and Hindi checkpoints' CC BY-SA 4.0 data provenance.

Model details

Architecture	Zipformer-style encoder (multi-resolution, Conformer-style blocks) + stateless RNN-Transducer predictor/joiner
Parameters	21.4M
Feature front-end	80-bin log-mel filterbank, hand-computed (no torchaudio.transforms)
Tokenizer	Character-level, extensible across languages
Loss	RNN-Transducer (torchaudio.functional.rnnt_loss)
Framework	PyTorch (custom architecture — not a transformers-library model; see Loading)

Checkpoints in this repository

File	Language	Fine-tuned from	Training data	Test WER
model_english_100h.pt	English	— (from scratch)	100.6h, LibriSpeech train-clean-100	21.9%
model_bengali_full.pt	Bengali	model_english_100h.pt	37.4h, OpenSLR SLR53	47.1%
model_hindi_full.pt	Hindi	model_english_100h.pt	37.4h, OpenSLR SLR103 (subset, capped to match Bengali's volume)	36.8%

All WER numbers are measured with greedy decoding (no beam search, no external language model) on a held-out test split in the same language, never seen during training. For scale: Whisper tiny.en (680k hours of pretraining) scores roughly 5-8% WER on the same English test set. These are honest from-scratch numbers at a laptop training budget, not claimed to be competitive with production systems. Bengali and Hindi were fine-tuned on identical training-data volume (37.4h) and recipe from the same English base, specifically so the two numbers are a controlled per-language comparison rather than confounded by different amounts of data — Hindi scoring 10.3 points better than Bengali under those matched conditions is a real, currently unexplained asymmetry (see the source repository's README for candidate explanations not yet confirmed).

Fairness and bias

Measured, not assumed — from evaluate_fairness.py in the source repository, run against model_english_100h.pt. The single largest finding is domain shift: evaluated out-of-domain on Common Voice English (crowd-sourced short prompts, n=4,862) rather than this model's own LibriSpeech (read audiobook) test set, WER rises from 21.9% to 81.3%. Every subgroup number below shares that same domain shift, so treat the absolute WERs as inflated but the gaps between groups as meaningful.

Axis	Result	Gap
Gender	male 85.2% (n=2,041) vs female 81.1% (n=441)	4.1 pts
Accent	Canadian English 62.5% (n=79) best vs India/South Asia 96.5% (n=454) worst	34.0 pts — largest measured
Utterance duration	short 74.7% vs long 87.7% (n≈1,620 each)	13.0 pts
Speaking rate (proxy)	fast 78.3% vs slow 84.1% (n≈1,620 each)	5.8 pts (direction counterintuitive; rate correlates with duration in this data, treat as directional)
Language (each on its own native test set)	English 21.9%, Bengali 47.1%, Hindi 36.8%	25.2 pts abs / 115% relative (Bengali worse than English)

The accent gap tracks training data composition directly: LibriSpeech's readers are entirely North American/British-accented; South Asian English accents have zero representation in training. Groups below n=30 are directional only and excluded from the disparity figures above. No audio-conditions/noise axis is reported — Common Voice has no reliable label for it, and deriving one from the waveform would be a fabricated signal. Full breakdown (180+ granular accent self-descriptions, per-group counts): checkpoints/fairness_results.json in the source repository.

Intended use

Research and education: studying transducer-based ASR architectures, cross-lingual transfer via vocabulary extension, and what a from-scratch model can and can't achieve on modest compute and data. Not intended for production transcription, safety-critical use, or any deployment where the WER numbers above would cause harm if wrong.

Limitations

• Greedy decoding only — no beam search, no external language model, no streaming/low-latency inference path.
• Bengali and Hindi are early fine-tunes on 37.4 hours of data each from an English base, not dedicated from-scratch Indic models — the WER gap to dedicated Indic ASR systems (and to this same project's own measurement of a pretrained streaming Zipformer at 21.6% WER on Bengali) is real. The Hindi checkpoint used only a subset of the 95.1h OpenSLR SLR103 corpus available, deliberately capped to match Bengali's data volume for a controlled comparison — not a data-availability limit.
• A handful of disclosed architectural simplifications relative to the published Zipformer paper this is based on (fewer encoder stacks, absolute rather than relative positional attention, standard AdamW rather than the paper's custom optimizer) — see the source repository's model.py docstring.
• Trained and evaluated on read/prompted speech (audiobooks, crowdsourced prompts) — expect substantially worse performance on spontaneous speech, accented speech outside the training distribution, noisy audio, or overlapping speakers, none of which were in the training or test data.
• Measured fairness gaps, not hypothetical ones — see Fairness and bias above. Out-of-domain WER on Common Voice (81.3%) is nearly 4x the in-domain LibriSpeech number (21.9%), and accent alone accounts for a 34-point gap between the best- and worst-performing groups measured. Do not treat this model as equally reliable across accents, genders, or languages.

Training data & license

Language	Source	License
English	LibriSpeech train-clean-100 (Panayotov et al., 2015)	CC BY 4.0
Bengali	OpenSLR SLR53 (Kjartansson et al., 2018)	CC BY-SA 4.0
Hindi	OpenSLR SLR103 (MUCS 2021, Microsoft Research India)	CC BY-SA 4.0

The Bengali and Hindi training data are both CC BY-SA 4.0 (ShareAlike). This model card attributes both sources plainly; if you redistribute model_bengali_full.pt or model_hindi_full.pt specifically, the conservative approach is to retain that attribution and a ShareAlike- compatible license for the checkpoint, even though whether trained weights are a "derivative work" of their training data is an unresolved question under copyright law that this project takes no position on.

Loading and inference

This is a custom PyTorch architecture, not wrapped in transformers. Clone the source repository (link above) and:

from model import load_checkpoint

model, tokenizer, ckpt = load_checkpoint("model_english_100h.pt", map_location="cpu")
model.eval()

# waveform: (1, num_samples) float32 tensor at 16kHz
# num_samples: (1,) tensor with the true (unpadded) sample count
token_ids = model.greedy_decode(waveform, num_samples, blank_id=0)
text = tokenizer.decode(token_ids[0])

Citation

If you use this model, please cite the source repository (link above) and the underlying datasets:

LibriSpeech: Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015).
  Librispeech: an ASR corpus based on public domain audio books. ICASSP 2015.

OpenSLR SLR53 (Bengali): Kjartansson, O., et al. (2018). Crowd-Sourced
  Speech Corpora for Javanese, Sundanese, Sinhala, Nepali, and Bangladeshi
  Bengali. SLTU 2018.

OpenSLR SLR103 (Hindi): Multilingual and code-switching ASR challenges
  for low resource Indian languages (MUCS 2021), Microsoft Research India.
  https://www.openslr.org/103/

Zipformer (architecture this project's encoder is based on):
  Yao, Z., et al. (2023). Zipformer: A faster and better encoder for
  automatic speech recognition. arXiv:2310.11230.