ghananlpcommunity/kasa42-asr

KASA-42 — one speech model for 42 Ghanaian languages

Author: Prince Nasamu Alhassan

Overview

A single CTC checkpoint that transcribes 42 Ghanaian language subsets and identifies which of them is being spoken, without being told. Fine-tuned from DONDO (KhayaAI/w2v-bert-…, Apache-2.0), which covers 11 Northern Ghanaian languages, and extended to the full ghana-speech corpus with a jointly trained language-ID head.

The 42 configs map to 41 ISO 639-3 codes: Akuapem Twi and Asante Twi share twi but are separate recording projects and separate rows below.

Results

Book-disjoint test set — no book appears in both training and test. 8,400 utterances, 200 per language.

Metric	Micro	Macro
WER	30.2%	32.5%
CER	10.5%	11.8%
Language ID accuracy	96.8% mean	91.5% min, 100% max

Language ID is the capability DONDO does not have: it conditions on a one-hot language prefix, so the caller must already know the language. Here it is inferred, at 96.8% mean accuracy over 42 classes (chance is 2.4%).

Per-language WER ranges from 13.0% (Gonja) to 69.8% (Kabiye). Read those to the nearest few points, not to one decimal — see the variance note below.

Full per-language results (n=200 each, computed from records_honest.jsonl with the released scorer)

Language	WER	CER	Language ID
Gonja (gjn)	13.0%	6.0%	96.5%
Birifor Southern (biv)	15.2%	4.0%	100.0%
Sisaala Tumulung (sil)	15.5%	4.4%	99.0%
Bimoba (bim)	15.6%	5.0%	99.5%
Paasaal (sig)	17.0%	3.6%	99.5%
Mampruli (maw)	17.5%	5.5%	100.0%
Dagaare (dga)	21.9%	9.2%	95.0%
Selee (snw)	22.2%	3.8%	99.5%
Kasem (xsm)	22.8%	11.1%	97.0%
Nawuri (naw)	22.8%	10.1%	98.0%
Tampulma (tpm)	23.4%	5.6%	98.5%
Lelemi (lef)	24.1%	8.8%	95.5%
Dagbani (dag)	24.5%	9.1%	97.0%
Konni (kma)	24.7%	5.9%	98.5%
Fante (fat)	25.1%	5.9%	93.0%
Buli (bwu)	26.2%	12.0%	95.5%
Gikyode (acd)	26.2%	5.6%	99.0%
Dangme (ada)	27.7%	7.8%	99.5%
Sekpele (lip)	28.0%	5.7%	99.5%
Anyin (any)	28.9%	8.2%	99.5%
Vagla (vag)	29.2%	10.8%	95.0%
Deg (mzw)	29.4%	11.7%	95.5%
Asante Twi (twi)	29.5%	8.6%	93.5%
Kusaal (kus)	30.2%	16.3%	95.5%
Fulfulde Maasina (ffm)	31.2%	9.7%	98.5%
Chumburung (ncu)	31.8%	10.9%	100.0%
Konkomba (xon)	31.9%	16.6%	97.5%
Ewe (ewe)	33.1%	10.5%	96.5%
Tuwuli (bov)	37.6%	14.9%	93.0%
Avatime (avn)	40.0%	14.2%	93.0%
Bissa (bib)	40.9%	9.5%	99.0%
Tem (kdh)	41.5%	8.7%	98.0%
Akuapem Twi (twi)	41.6%	16.5%	92.0%
Bassar Ntcham (bud)	42.6%	18.6%	97.5%
Sehwi (sfw)	45.8%	22.2%	93.0%
Ntrubo (ntr)	46.9%	13.3%	97.0%
Hausa (hau)	47.5%	31.7%	98.0%
Siwu (akp)	47.7%	14.3%	94.5%
Nzema (nzi)	55.4%	18.9%	94.0%
Nkonya (nko)	58.0%	14.2%	91.5%
Ninkare (gur)	62.5%	42.1%	96.0%
Kabiye (kbp)	69.8%	25.9%	95.5%

Against other systems

Every system scored on the same book-disjoint test set, with the same text normalisation applied to references and hypotheses alike. Crucially, each row recomputes both sides over only the languages that system covers — comparing an average over 34 covered languages against an average over all 42 would flatter whichever system declined the hardest ones.

System	Languages	Their WER	Ours	Their CER	Ours
Whisper large-v3	42	102.3%	30.2%	57.3%	10.5%
MMS-1B-all	34	25.7%	29.2%	12.1%	9.7%
DONDO (unconditioned)	8	75.9%	27.6%	40.8%	14.1%

MMS-1B-all beats us on WER by 3.5pp. We beat it on CER by 2.4pp, and cover eight languages it has no adapter for (both Twis, Dagbani, Dangme, Fante, Fulfulde and two others). It is also roughly 1.7× our parameter count and does not do language identification.

That WER/CER split is the informative part. MMS's ratio is 2.1×, ours 3.0× — we get more characters right and more words wrong, which is a word-segmentation weakness rather than an acoustic one. The same signature appears in our per-language table (Bissa 40.9% WER against 9.5% CER). Space prediction is the obvious place to look next, and we have not looked yet.

Whisper exceeds 100% WER because WER counts insertions: it transcribes these languages into something else entirely and emits more words than the reference contains. It was never trained on them.

The DONDO row is not a fair comparison and should not be cited as one. DONDO conditions on a one-hot language prefix; asr/baselines.py supplies none and runs it as a plain CTC model, so 75.9% is a lower bound on its ability, not a measurement of it. Its published figure is ~10.3% average WER on 11 languages. We report the number for transparency about what we ran, not as evidence about DONDO. Fixing this needs the conditioning implemented, which the window did not allow.

Cross-corpus: does it know Kusaal, or one ministry's recordings?

Every number above comes from ghana-speech. A model trained on one recording project and tested on the same one cannot distinguish having learned a language from having learned a studio. So we scored it on an independent Kusaal corpus — Kusaal ASR Dataset, 81.7 h from Faith Comes By Hearing and Global Recordings Network, forced-aligned into 9.5 s clips against GILLBT text.

Restricted to the 11 books KASA-42 held out of training, because scripture corpora share a translation: on any other book the model would already have trained on the sentences, and only the audio would be new.

	ghana-speech (version 3752)	Kaggle corpus (FCBH / GRN)
Clips	200	2,895 (8.35 h)
WER	30.2%	37.9%
CER	16.3%	21.9%
Language ID	95.5%	99.8%

Transcription degrades about 25% relative on unfamiliar recordings of the same language and domain. That is the honest cost of changing recording source, and it is the number to quote when asking what this model would do on audio from anywhere else.

Language identification improves, to 99.8%. The LID head evidently keys on the language rather than the channel — it is more robust to a change of studio than the transcription is. On 2,895 clips that is a tighter estimate than any per-language figure elsewhere in this card.

Three caveats we state rather than leave implied:

• Both corpora are scripture. This measures robustness to recording conditions, not to conversational speech, which remains untested.
• The two may use different Kusaal translations — GILLBT here against whatever version 3752 uses — so some of the WER gap is vocabulary and phrasing rather than acoustics.
• The Kaggle clips average 10.4 s against ghana-speech's ~7.9 s, and longer clips usually score lower WER because errors amortise over more words. If anything that understates the degradation.

The leakage experiment, and its negative result

ghana-speech is Bible audio: source_file parses as BOOK.CHAPTER.VERSION. A random segment split therefore puts adjacent verses — same narrator, same session — on both sides of the boundary, and in six languages the same book appears under multiple recording versions, so a random split can place a byte-identical reference transcript in train and test.

We split by book for that reason, and then tested whether it mattered. Identical weights were scored on two sets: segments from held-out books, and segments from seen books that the model was not trained on. Only the second carries book-level overlap.

Construction	honest − leaked	95% CI
Unmatched	−1.1pp	[−1.9, −0.3]
Book-diversity matched	−0.7pp	[−1.4, +0.1]

The interval spans zero. On these weights, book-level overlap gives no measurable advantage. We report this as a negative result rather than omitting it.

Two caveats we state rather than leave to the reader:

• This says the model is not memorising books at this scale — one pass over 700 h at ~30% WER — not that leakage cannot occur. A longer-trained or higher-capacity model may behave differently.
• We did not test chapter-adjacent leakage, which is the strongest form: a naive split puts consecutive verses from one recording session on both sides. Our leaked set samples across whole books, which dilutes that.

Book-disjoint splitting remains the conservative default. Our result says it cost us nothing.

The first construction was confounded — the honest set spanned 4.2 books per language against the leaked set's 27.2, with 4.9% shorter references — so the comparison varied content diversity alongside book-seen-ness. The matched row above removes that.

Sampling variance

Three independent draws of the same test design gave micro WER of 30.3%, 30.0% and 30.2%, while individual languages moved by up to 7.3pp (Ninkare 57.2% → 64.5%). At n=200 per language, the micro-average is stable to a few tenths and per-language figures are not. Quote them accordingly.

Training data

ghananlpcommunity/ghana-speech, CC BY-NC 4.0. 1,411,467 segments, 2,334.9 h across 42 configs.

Training used a temperature-sampled mixture (α=0.5, 40 h per-language cap) over train-split books only: 700 h from 393,160 segments, which reduces the largest language's share from 8.5% to 4.9%.

Three things we found in the corpus

1. id is not unique in 13 of 42 configs, by factors up to 1.52×. Anything selecting segments by id — the obvious design — silently over-samples those languages, and the hours accounting reconciles either way, so nothing looks wrong.
2. One version code does not imply one narrator. Kusaal is 100% version 3752, yet ECAPA-TDNN speaker embeddings split its books cleanly into Old Testament (28 books, 39.7 h) and New Testament readers.
3. 21 Hebrew characters appear in Ewe and Hausa — Psalm 119's acrostic stanza headings, 67 occurrences total. They look like an encoding fault and are not.

Intended use and limitations

Non-commercial use only (CC BY-NC 4.0, inherited from the training data).

This is trained entirely on read scripture: formal, archaic, proper-noun dense, studio-recorded, one or two narrators per language. It will transcribe spontaneous conversational speech considerably worse than these numbers suggest. Every model trained on this corpus has that limitation; ours is not exempt.

How to use

import torch, soundfile as sf
from transformers import SeamlessM4TFeatureExtractor
from kasa42.asr.model import Kasa42ForCTC, model_state
from kasa42.asr.dataset import CharTokenizer

tok = CharTokenizer.from_json("vocab.json")
model = Kasa42ForCTC("KhayaAI/w2v-bert-gjn_maw_gur_dag_dga_kus_lxn_wlx_xon_xsm_en",
                     vocab_size=len(tok), n_languages=42, blank_id=tok.blank)
model.load_state_dict(model_state("final.pt"))
model.eval()

fe = SeamlessM4TFeatureExtractor.from_pretrained(model.encoder.name_or_path)
wav, sr = sf.read("clip.wav")                       # 16 kHz mono
feats = fe([wav], sampling_rate=16000, return_tensors="pt",
           padding=True, return_attention_mask=True)
with torch.no_grad():
    out = model(**feats)
print(tok.decode(out["logits"].argmax(-1)[0][: out["input_lengths"][0]]))
print("language:", languages[out["lid_logits"].argmax(-1).item()])

An ONNX export is included and verified against the torch model at 150/300/500/900 frames, so it is safe for variable-length audio and runs on CPU.

Training details

Base	DONDO KhayaAI/w2v-bert-gjn_maw_gur_dag_dga_kus_lxn_wlx_xon_xsm_en (Apache-2.0)
Architecture	w2v-BERT 2.0 encoder + CTC head + mean-pool linear LID head
Vocabulary	96 chars — DONDO's 49 keep their indices and trained head rows, 47 added
Blank	[PAD] = 33, not 0 (DONDO's position)
Steps	24,000, batch budget 160 s of padded audio, gradient checkpointing
Optimiser	AdamW, lr 5e-5, 500 warmup, cosine decay, encoder frozen 300 steps
Precision	bf16 (Hopper)
Hardware	1 × NVIDIA H200 (shared), ~6 h

Claims we do not make

facebook/mms-tts-kus exists, MMS-1B-all covers Kusaal ASR, and DONDO covers Kusaal at 13.3% WER. This is not the first Kusaal speech model, and not the first multilingual Ghanaian ASR. The claims are narrower: broader language coverage in one checkpoint, language ID that DONDO does not have, and results on a split that does not leak, with the leakage question actually tested.

See the comparison below for how this sits against MMS, Whisper and DONDO. It is not first on every metric, and the section says so.

Acknowledgements

Compute provided by AI Skills and Compute Africa (AISCA), trained on the Ghana NLP H200. Corpus by the GhanaNLP community. Base checkpoint by Paul Azunre (DONDO).

Please keep derivatives non-commercial and share improvements back with ghananlpcommunity.