xlsr-twi-codeswitch-ipa

A phoneme recogniser for Ghanaian English–Twi code-switched speech.

KoelLabs/xlsr-english-01 fine-tuned on ghananlpcommunity/Ghana_English-Twi_Code-switching_Speech-ipa, which is KasaSpeech with an ipa column produced by ghanag2p-uni.

The model outputs phonemes in the same 79-token inventory as its base model. Nothing was added to the vocabulary and the CTC head was not resized: the training targets were mapped into that inventory first, so every pretrained output row kept its meaning.

Results

Phoneme error rate (PER), lower is better. The test split was held out completely during training — it was read once, by this evaluation.

	PER
Base model, before fine-tuning (2159 validation rows)	0.687
Fine-tuned, validation (2159 rows)	0.303
Fine-tuned, test (1731 rows)	0.280

Test errors break down as 17,783 substitutions, 22,139 deletions, 3,678 insertions over 155,838 reference phonemes.

PER by utterance length on test:

length	n	PER
0-5s	900	0.227
5-10s	281	0.248
10-20s	221	0.302
20-40s	323	0.294
40-999s	6	0.294

Usage

import json, torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import Wav2Vec2ForCTC, Wav2Vec2FeatureExtractor

model = Wav2Vec2ForCTC.from_pretrained("ghananlpcommunity/xlsr-twi-codeswitch-ipa").eval()
fe = Wav2Vec2FeatureExtractor.from_pretrained("ghananlpcommunity/xlsr-twi-codeswitch-ipa")
vocab = json.load(open(hf_hub_download("ghananlpcommunity/xlsr-twi-codeswitch-ipa", "vocab.json")))
inv = {i: u for u, i in vocab.items()}

speech, sr = sf.read("utterance.wav")      # must be 16 kHz mono
x = fe(speech, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    ids = model(x.input_values).logits.argmax(-1)[0].tolist()

# CTC decode: collapse runs, drop the blank (id 0)
out, prev = [], None
for i in ids:
    if i != prev and i != 0:
        out.append(inv[i])
    prev = i
print(" ".join(out))       # e.g. "ɛ j ɛ d e n o b u t ɪ ʔ l l t ɾ j"

Training

Base	KoelLabs/xlsr-english-01 (Wav2Vec2ForCTC, XLS-R large)
Data	69.1 h of code-switched speech, 49,033 utterances
Epochs	2
Effective batch	32
LR	5e-5 (10% warmup)
Precision	bf16
Frozen	convolutional feature encoder
Selection	best validation PER

Clips over 20 s were excluded from training only; the validation and test splits were evaluated in full, long utterances included.

Caveats

• Targets come from a grapheme-to-phoneme rule set, not human phonetic annotation. The model learns to predict what ghanag2p-uni would produce for the reference transcript, so it inherits that system's conventions — and its errors.
• Everything is phonemised under Twi rules, English stretches included, so English words are rendered as a Twi speaker's phonology rather than with an English G2P.
• The target inventory is lossy against real Twi: tʰ collapses to t, vowel length is dropped, ç becomes ʃ and ɲ becomes n j.

Credit

Speech and transcripts are from KasaSpeech (Kenneth Dotse). Licence Apache-2.0, as the base model and source corpus.