asr — a from-scratch ASR model

A 30.5M-parameter speech recognition model trained from scratch on Apple Silicon (MPS backend) on LibriSpeech train-clean-100. No pretrained weights, no fine-tuning — the encoder, decoder, training loop, and tokenizer are all in this repo.

Code also at github.com/shubhexists/asr.

What's here

File	Size	What it is
checkpoints/model_weights.pt	112 MB	Weights only — use this for inference
checkpoints/latest.pt	335 MB	Full checkpoint: weights + optimizer + scheduler, for resuming training
configs/tokenizer.model	313 KB	SentencePiece BPE model (vocab 5000) — required for both
configs/zipformer_s.yaml		Model + training config
src/, scripts/		Model, training, evaluation, and transcription code

Both checkpoints are epoch 40 / step 71320 and contain identical weights.

Architecture

Encoder (src/zipformer.py) — Zipformer-inspired (arXiv:2310.11230) 5-stage U-Net over time: full-res → half-res → quarter-res (bottleneck) → half-res → full-res, with additive skip connections between matching resolutions. Downsampling the middle stages fits more layers into the same compute budget. Built from standard PyTorch primitives — not a byte-exact port of icefall's custom ops.

Frontend (src/model.py) — log-mel fbank (80 mels) → per-utterance mean/var normalization → SpecAugment, computed on the fly.

Loss (src/model.py) — hybrid CR-CTC + attention:

• Two SpecAugmented views of each utterance are stacked on the batch dimension for a single encoder pass, then split. Each gets its own CTC loss, plus a symmetric KL consistency loss between the two views (CR-CTC, arXiv:2410.05101).
• An attention decoder (src/decoder.py, standard Transformer decoder) consumes one view and is trained with cross-entropy.
• Total loss weights CTC (0.3), attention (0.7), and consistency (0.2) terms.

Tokenizer (src/tokenizer.py) — SentencePiece BPE, vocab 5000, fixed special ids: <pad>=0 (also the CTC blank), <unk>=1, <bos>=2, <eos>=3, <spk_change>=4, regular pieces from 5.

Shape — d_model=256, 14 encoder layers across the 5 U-Net stages ([2, 3, 4, 3, 2]), 4 attention heads, 4-layer attention decoder.

Training

Data	LibriSpeech train-clean-100 (100 h), dev-clean for validation
Hardware	Apple Silicon, PyTorch MPS backend
Steps	71,320 (40 epochs)
Batch size	16
Optimizer	AdamW, lr 3e-4, weight decay 0.01, 2500 warmup steps, grad clip 5.0
Final val loss	0.789 on dev-clean

WER has not been benchmarked. Only losses were logged during training. To measure it yourself, see the evaluation command below — src/evaluate.py reports WER for both the greedy CTC and greedy attention-decoder paths.

Usage

git clone https://huggingface.co/shubhexists/asr
cd asr
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Transcribe a single file:

python -m scripts.transcribe --config configs/zipformer_s.yaml \
    --checkpoint checkpoints/model_weights.pt --audio /path/to/clip.wav

Evaluate WER on a LibriSpeech split (needs the data extracted under data/, see below):

python -m src.evaluate --config configs/zipformer_s.yaml \
    --checkpoint checkpoints/model_weights.pt --split test-clean

Resume training from the full checkpoint:

python -m src.train --config configs/zipformer_s.yaml --resume checkpoints/latest.pt

Training and evaluation need LibriSpeech from openslr.org/resources/12, extracted into data/ in torchaudio's layout:

data/LibriSpeech/<split>/<speaker>/<chapter>/<speaker>-<chapter>-<utterance>.flac
data/LibriSpeech/<split>/<speaker>/<chapter>/<speaker>-<chapter>.trans.txt

torch.nn.functional.ctc_loss has no MPS kernel, so the training and eval scripts set PYTORCH_ENABLE_MPS_FALLBACK=1 to run that one op on CPU.

Limitations

• Trained on 100 hours of read audiobook English only. Expect it to degrade on conversational speech, accents outside LibriSpeech's distribution, noisy audio, and any non-English input.
• Expects 16 kHz mono audio.
• No language model, no beam search — decoding is greedy.
• WER is unmeasured, so treat this as a working reference implementation rather than a model with a known quality bar.