w2v-bert-2.0-luganda-main-best-2

A Luganda automatic speech recognition (ASR) model, fine-tuned from facebook/w2v-bert-2.0 on a combined Luganda speech corpus. This is a variant of keystats/w2v-bert-2.0-luganda-main-best with a different train/validation split of the WAXAL data — see Difference from -main-best below.

Model description

facebook/w2v-bert-2.0 — a large-scale, multilingual self-supervised speech encoder pretrained with a BERT-style masked prediction objective — is used as the backbone, with a from-scratch character-level CTC (Connectionist Temporal Classification) head fine-tuned specifically for Luganda.

Text casing note: like -main-best, this model's vocabulary was built with NFKC-only normalization — case, punctuation, and diacritics from the source transcriptions are preserved as-is. This matches how the WAXAL competition (Zindi) actually scores submissions: raw, unnormalized WER/CER, where case and punctuation mismatches count as errors.

Difference from -main-best

This checkpoint uses a different split of the WAXAL data than -main-best:

	-main-best	-main-best-2 (this model)
WAXAL train split	training	training
WAXAL validation split	held out, used only for evaluation	pooled into training
WAXAL test split	held out, used only for evaluation	used as the validation split during training (early stopping / checkpoint selection)

Everything else — base model, architecture, processor/tokenizer construction, hyperparameters, data filtering, and seed — is unchanged from -main-best. See below for the full details.

Training data

Training pool (train split only from each source, deduplicated by audio hash and by transcription-within-source, unless noted otherwise):

Source	Role
google/WaxalNLP (lug_asr config)	train and validation splits pooled into training; test split used as the validation split for training (early stopping / checkpoint selection)
FarmerlineML/luganda_dataset_2.0	All splits pooled into training
Bateesa/luganda-tts-toby	All splits pooled into training

Note: keystats/luganda_asr_dataset (~230k rows, used in the -main sibling's training) was deliberately excluded, for the same reason as in -main-best — dropping it makes it easier to train more epochs.

None of the additional sources overlap with WAXAL's own train/validation/test split boundaries, so pooling every split from them carries no evaluation leakage risk.

Because the WAXAL validation split is now part of the training pool, it is not a valid held-out benchmark for this checkpoint. The WAXAL test split was used as the training-time validation set (for early stopping / best-checkpoint selection), so it is also not a clean independent benchmark for this checkpoint. Evaluate this model on a genuinely held-out set if you need an unbiased score.

Training procedure

• Base model: facebook/w2v-bert-2.0
• Architecture: Wav2Vec2BertForCTC, add_adapter=True
• Processor: Wav2Vec2BertProcessor — SeamlessM4TFeatureExtractor for audio features + a Wav2Vec2CTCTokenizer built from scratch on the combined training + validation transcriptions (character-level vocabulary, case and punctuation preserved, NFKC Unicode normalization only, | as the word delimiter, [PAD] doubling as the CTC blank token)
• Sample rate: 16 kHz mono
• Hardware: single RTX PRO 6000
• Epochs: 10 (with early stopping, patience 5, on validation WER)

Hyperparameter	Value	Rationale
Learning rate	3e-5	A much higher rate (e.g. 1e-3) is too aggressive for full fine-tuning of a model this size
Effective batch size	32 (per-device 4 × grad-accum 8)	Batch size 1 gives very noisy gradients at this model scale
Checkpoint selection	best-by-WER	load_best_model_at_end + early stopping
Dropout	0.05 (attention / hidden / feature-projection)	Non-zero regularization appropriate for this dataset size
Weight decay	0.01	Standard AdamW regularization
LR schedule	cosine, 10% warmup	Gentler decay than linear, avoids an abrupt ramp-down
Precision	fp16, gradient checkpointing	Memory efficiency

• Data filtering: clips whose transcript is too long for CTC to align within the available encoder output length ("CTC-impossible" clips, roughly output_steps < 2 * label_length) are dropped from both train and validation before training
• Seed: 42 (deterministic — same seed for Python/NumPy/PyTorch/CUDA)

Evaluation results

Evaluation numbers for this checkpoint are not available yet in this README — they'll be added once a proper held-out evaluation has been run (see the note in Training data on why WAXAL's validation/test splits aren't clean benchmarks for this model).

How to use

Two ways to use this model, depending on your needs:

• Option 1 — model alone (greedy decoding): faster, no extra dependencies, slightly lower accuracy.
• Option 2 — model + KLM (recommended): requires pyctcdecode + KenLM, noticeably higher accuracy via beam-search decoding with a matching language model.

Option 1 — model alone (greedy decoding)

import torch
import librosa
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor

MODEL_ID = "keystats/w2v-bert-2.0-luganda-main-best-2"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

processor = Wav2Vec2BertProcessor.from_pretrained(MODEL_ID)
model = Wav2Vec2BertForCTC.from_pretrained(MODEL_ID).to(DEVICE).eval()

audio_array, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(input_features=inputs.input_features.to(DEVICE)).logits

predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)[0]

print(transcription)  # cased, punctuated Luganda text (to whatever extent seen in training)

Option 2 — model + KLM (recommended, higher accuracy)

A companion n-gram KenLM language model, trained on the same raw (cased, punctuated) text convention as this ASR model, is available at keystats/waxal-kenlm-models-best (luganda/luganda_5gram_correct-best.arpa).

Important: use the matching KLM variant for whichever ASR checkpoint you're using — this model pairs with keystats/waxal-kenlm-models-best (raw/cased text), while the -main checkpoint pairs with the separate keystats/waxal-kenlm-models repo (normalized/lowercase text). Mixing a raw-text ASR model with a normalized-text KLM (or vice versa) will cause a vocabulary mismatch during decoding.

# pip install pyctcdecode
# pip install https://github.com/kpu/kenlm/archive/master.zip

import torch
import librosa
from huggingface_hub import hf_hub_download
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor
from pyctcdecode import build_ctcdecoder

MODEL_ID = "keystats/w2v-bert-2.0-luganda-main-best-2"
KLM_REPO_ID = "keystats/waxal-kenlm-models-best"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

processor = Wav2Vec2BertProcessor.from_pretrained(MODEL_ID)
model = Wav2Vec2BertForCTC.from_pretrained(MODEL_ID).to(DEVICE).eval()

klm_path = hf_hub_download(repo_id=KLM_REPO_ID, repo_type="dataset",
                            filename="luganda/luganda_5gram_correct-best.arpa")

def build_vocab_list(tokenizer, vocab_size):
    vocab_dict = tokenizer.get_vocab()
    vocab_list = [None] * vocab_size
    for tok, idx in sorted(vocab_dict.items(), key=lambda kv: kv[1]):
        if idx < vocab_size:
            vocab_list[idx] = tok
    pad_id = tokenizer.pad_token_id
    if pad_id is not None and pad_id < len(vocab_list):
        vocab_list[pad_id] = ""
    word_delim = getattr(tokenizer, "word_delimiter_token", None)
    if word_delim:
        delim_id = vocab_dict.get(word_delim)
        if delim_id is not None:
            vocab_list[delim_id] = " "
    return vocab_list

vocab_list = build_vocab_list(processor.tokenizer, model.config.vocab_size)
decoder = build_ctcdecoder(
    vocab_list,
    kenlm_model_path=klm_path,
    alpha=0.5,   # LM weight -- tune against your own validation set
    beta=0.7,    # word insertion bonus -- tune against your own validation set
)

audio_array, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(input_features=inputs.input_features.to(DEVICE)).logits

transcription = decoder.decode(logits.cpu().numpy()[0], beam_width=100)
print(transcription)

Note on alpha/beta: the values above are starting points, not universal defaults — grid-search them against your own labeled validation set, since optimal weights depend on your specific audio domain.

Intended uses & limitations

• Intended for transcribing spoken Luganda audio into cased, punctuated text (to whatever extent case/punctuation exist in the training transcriptions).
• As a CTC-based model, it assumes single-speaker, forward-only audio and has no mechanism for overlapping speech from multiple speakers.
• Trained on a mix of WAXAL and two community-contributed Luganda datasets; acoustic conditions, recording quality, and dialectal coverage reflect that combined pool, not any single controlled source.
• Because WAXAL's validation split was folded into training and its test split was used for training-time model selection, neither split gives an unbiased read on this model's real-world accuracy — treat any score computed on them with that caveat in mind.
• Raw-text WER/CER (with case and punctuation counted as errors) will read higher than a normalized-text comparison of the same underlying transcription quality — this is expected and matches how the source competition (Zindi/WAXAL) actually scores submissions.

Related checkpoints

• keystats/w2v-bert-2.0-luganda-main-best — same setup, but with WAXAL's validation and test splits kept fully held out for evaluation instead of used in/for training
• keystats/w2v-bert-2.0-luganda-main — same base model and general training procedure, trained on lowercased text instead, and additionally includes keystats/luganda_asr_dataset in its training pool

Citation

If you use this model, please cite the training/fine-tuning work and the underlying datasets:

@misc{keystats_wav2vec2bert_luganda_best2,
  title={w2v-bert-2.0-luganda-main-best-2: A Luganda ASR model fine-tuned from facebook/w2v-bert-2.0 on raw text},
  author={keystats},
  year={2026},
  howpublished={\url{https://huggingface.co/keystats/w2v-bert-2.0-luganda-main-best-2}}
}

@misc{waxal,
  title={WAXAL: A Multilingual African Speech Dataset},
  author={Google},
  howpublished={\url{https://huggingface.co/datasets/google/WaxalNLP}}
}

@misc{farmerline_luganda,
  title={luganda\_dataset\_2.0},
  author={FarmerlineML},
  howpublished={\url{https://huggingface.co/datasets/FarmerlineML/luganda_dataset_2.0}}
}

@misc{bateesa_luganda_tts,
  title={luganda-tts-toby},
  author={Bateesa},
  howpublished={\url{https://huggingface.co/datasets/Bateesa/luganda-tts-toby}}
}

@inproceedings{w2vbert2,
  title={Seamless: Multilingual Expressive and Streaming Speech Translation},
  author={Seamless Communication and others},
  year={2023},
  howpublished={\url{https://huggingface.co/facebook/w2v-bert-2.0}}
}