rebreda/moonshine-base-welsh

Moonshine Base, Welsh

A Welsh speech recogniser fine-tuned from Moonshine Base. It is a small model: it runs on a CPU and is about 29 times smaller than the Welsh Whisper models from Bangor University.

Versions in this repository

Each version is a branch. Pick one with revision=.

branch	what it is	parameters	download	Welsh mean WER	English WER
main (recommended)	compact: five heads off, smaller token table	53.2M	106 MB	40.1	21.0
full-vocab	five heads off, full token table	61.5M	123 MB	39.9	16.4
v1	first release (checkpoint 5,500)	61.5M	246 MB	43.5	16.6

Welsh mean = mean WER over Common Voice, FLEURS Welsh, Lleisiau Arfor and Siarad. Use main for Welsh. Use full-vocab if you also need plain English. v1 is kept so earlier results can be reproduced.

model = MoonshineForConditionalGeneration.from_pretrained(
    "rebreda/moonshine-base-welsh", revision="full-vocab")

main and full-vocab were made from v1's training run with no further training. The steps were:

1. Checkpoint averaging. The LoRA adapter is the average of ten checkpoints of the continuation run (steps 4,000 to 8,500), not a single checkpoint.
2. Five attention heads switched off. Tracing the model's repetition loops (the same word over and over) showed five decoder self-attention heads that copy the previous words and outbid the end-of-text token. They are heads 0, 1 and 3 of layer 4, head 1 of layer 2 and head 1 of layer 7 (counting from 0). Their output weights are set to zero (0.18% of the weights). The heads were found using development data and Banc validation clips only, never test data. On Siarad, the held-out conversation test, this cut WER from 87.0 to 76.4.
3. A smaller token table (main only). The model kept the 32,768-token table of the original English model, and most of it was never used. main keeps 12,735 tokens: those used at least 3 times in the training transcripts, all 256 byte tokens (so any word can still be spelled out), and the pieces the tokenizer needs to build them. The tokens were chosen from training text only. This removes 8.3M parameters.

The write-up is on the project page: https://quickthoughts.ca/projects/welsh-asr/.

• Parameters: 53.2M (main), 61.5M (full-vocab)
• Languages: Welsh, and English words as spoken inside Welsh speech. Plain English is worse on main (see below).
• Input: 16 kHz mono audio, short utterances
• Output: text with casing and punctuation
• Licence: MIT (inherited from Moonshine)

How to use

The model works with plain transformers (tested with 5.20). Audio must be 16 kHz mono. The weights are stored in fp16 and load as fp32 by default; fp16 storage changed none of 600 test decodes.

import torch
import soundfile as sf
from transformers import AutoProcessor, MoonshineForConditionalGeneration

model_id = "rebreda/moonshine-base-welsh"
processor = AutoProcessor.from_pretrained(model_id)
model = MoonshineForConditionalGeneration.from_pretrained(model_id).eval()

audio, sr = sf.read("clip.wav", dtype="float32")  # 16 kHz mono
assert sr == 16000

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
seconds = len(audio) / 16000
# Cap the output at about 25 tokens per second of audio (see below).
max_length = min(194, int(25 * seconds) + 10)

with torch.no_grad():
    ids = model.generate(**inputs, num_beams=5, max_length=max_length)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Or with the pipeline:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="rebreda/moonshine-base-welsh")
print(asr("clip.wav", generate_kwargs={"num_beams": 5})["text"])

For another version, add revision="full-vocab" to both from_pretrained calls, or to pipeline(...).

Decoding settings

• Beam search, 5 beams. All the scores below were measured this way. The transformers speech pipeline adds num_beams=5 by itself when the generation config does not set it, and this model's generation config does not. If you call generate() yourself, pass num_beams=5.
• Cap the output length at about 25 tokens per second of audio. The switched-off heads remove most repetition loops (on Siarad, 4 of 1,500 clips still loop), but a cap of 25 tokens per second plus 10 is a cheap safety net. Fast Welsh conversation reaches about 21 tokens per second at the extreme. The scores below were measured without the cap.
• Long recordings should be cut into short segments (for example with a voice activity detector) before transcription. The model was trained and tested on single utterances.

Results

Word error rate (WER, %, lower is better; CER in brackets). Text was normalised with Whisper's basic normaliser, keeping diacritics. Beam 5, no length cap, no language model.

test set	what it is	clips	main	full-vocab	v1
Common Voice 27.0 Welsh, test	read prompts	5,390 (whole split)	25.8 (9.0)	25.8 (9.0)	26.1 (9.0)
FLEURS Welsh (cy_gb), test	read news and Wikipedia	1,020 (whole split)	37.6 (15.8)	37.5 (15.8)	38.2 (15.8)
Lleisiau Arfor, test (Welsh)	conversation	1,500	20.1 (9.4)	19.9 (9.3)	21.5 (11.4)
Lleisiau Arfor, unseen sentences only	conversation	403	33.8	33.7	37.6
Siarad (held out)	everyday conversation, code-switching	1,500	76.8 (41.1)	76.4 (40.9)	88.3 (48.8)
Banc Trawsgrifiadau Bangor, test	podcasts	1,500	32.7 (13.8)	32.6 (13.7)	33.3 (14.1)
FLEURS English (en_us), test	read English	500	21.0 (10.7)	16.4 (8.5)	16.6 (8.7)
mean of CV, FLEURS Welsh, Arfor, Siarad			40.1	39.9	43.5

Where the gains come from, measured on the same test sets with paired bootstrap confidence intervals: checkpoint averaging improved Arfor by 2.1 points (95% CI 0.6 to 4.3); switching off the five heads improved Siarad by 10.6 points (5.7 to 16.3) and changed no other set by more than 0.5; the smaller token table changed every Welsh set by 0.4 points or less and cost 4.5 points (3.9 to 5.2) on English.

Please read these caveats before quoting the numbers:

• English got worse. The training text had only about 2,600 English sentences, so most English word tokens were removed. English words that are common inside Welsh speech were kept, and the Welsh sets did not get worse, but plain English went from 16.4 to 21.0 WER. If you need English, use the full-vocab branch.
• Arfor: most Lleisiau Arfor test sentences also appear as prompts in Arfor's training split (different speakers, same words). The fair figure is the 33.8 on the 403 test clips whose sentence is not in Arfor's training data.
• Siarad is hard. It is natural conversation with frequent switching between Welsh and English, and it was never used for training or for choosing the edits. Even at 76.8, this model is not good enough for transcribing free conversation.
• Common Voice: the test set is Common Voice 27.0's Welsh test split. Training used other Common Voice 27.0 clips whose audio, speaker and sentence stay out of its dev and test splits.
• Bangor's Whisper is more accurate, but much larger. techiaith/whisper-large-ft-cy-en (Whisper large-v2, 1.5B parameters, about 29 times this model) scored 27.6 on unseen Arfor against 33.8 for main. On the Banc podcast test, which both models trained on, it scored 34.7 and main 32.7. On Siarad it scored 103.8. Its Common Voice score (10.3) may be inflated: it trained on an older Common Voice release whose clips may overlap the CV 27.0 test.

With an external language model

Not part of this repository, but for context: rescoring 8-beam candidate lists of full-vocab with a Welsh word n-gram model and Bangor's Welsh Llama 3.2 1B gave Common Voice 20.5, FLEURS 34.6, Arfor 18.8, Siarad 71.7 and Banc 31.8 (mean 36.4).

Training

Training used listenr on one AMD Radeon 8060S (ROCm, bf16), in two stages:

1. From moonshine-base: a rank-128 LoRA (alpha 256, learning rate 3e-4) on Common Voice 27.0 Welsh, Lleisiau Arfor's Welsh train rows and FLEURS English train (about 60 hours), then merged into the weights.
2. Continuation: a fresh rank-128 LoRA (alpha 256, learning rate 1e-4, warm-up 200 steps, batch 32) on the same data plus Bangor University's Banc Trawsgrifiadau Bangor and Corpws CLlLC: 129,938 clips, 175.1 hours. The adapter weights of the ten checkpoints from steps 4,000 to 8,500 were averaged and then merged into the model. The average was chosen over single checkpoints on 3,000 dev clips.

Then the three edits above were made directly to the weights. Training text was cleaned: Bangor's markup for non-speech sounds and English words was removed, and {spoken|written} pairs were reduced to the spoken form. Lleisiau Arfor rows were filtered to Welsh.

Training audio (175.1 h): Common Voice 23.9 h, Lleisiau Arfor 28.4 h, FLEURS English 7.5 h, Banc 63.0 h, CLlLC 52.4 h.

Training data and licences

dataset	used	licence
Common Voice Scripted Speech 27.0, Welsh (via Mozilla Data Collective)	train	CC0-1.0
Lleisiau Arfor, train_clean, Welsh rows	train	CC0-1.0
FLEURS en_us train	train	CC-BY 4.0
Banc Trawsgrifiadau Bangor 26.10, train (validation used to choose the head edit)	train	CC0-1.0 (gated)
Corpws CLlLC	train	CC0-1.0 (gated)

FLEURS is by Conneau et al., Google (CC-BY 4.0): "FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech" (2022). Banc Trawsgrifiadau Bangor and Corpws CLlLC are published by the Language Technologies Unit (Uned Technolegau Iaith), Bangor University. Banc's terms include an agreement not to try to identify its speakers; please respect it. Common Voice's terms also forbid trying to identify speakers.

Intended use and limitations

• Good for: short, fairly clear Welsh utterances, such as dictation, voice commands, read speech and podcast-style speech.
• Not good for: free conversation with a lot of code-switching (see Siarad), plain English on main, noisy audio, or anywhere errors have consequences. About one word in four is still wrong on read speech.
• Like the base model, please do not use it to transcribe recordings of people made without their consent.

Credits and licence

Based on Moonshine by Useful Sensors / Moonshine AI, released under the MIT licence:

MIT License

Copyright (c) 2024 Useful Sensors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Please cite Moonshine if you use this model:

@misc{jeffries2024moonshinespeechrecognitionlive,
      title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
      author={Nat Jeffries and Evan King and Manjunath Kudlur and Guy Nicholson and James Wang and Pete Warden},
      year={2024},
      eprint={2410.15608},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2410.15608},
}

The fine-tuned weights in this repository are released under the same MIT licence. Thanks to Mozilla Common Voice and its Welsh contributors, the Lleisiau Arfor project, Google (FLEURS) and Bangor University's Language Technologies Unit for the data.