Towards large-scale speech foundation models for a low-resource minority language (2025)

Abstract: Modern ASR systems require massive amounts of training data. While ASR training data for most languages are scarce and expensive to transcribe, a practical solution is to collect huge amounts of raw untranscribed speech and pre-train the ASR model in a self-supervised manner. Unfortunately, for many low-resource minority languages, even untranscribed speech data are scarce. In this paper, we propose a solution for the Northern Sámi language with 22,400 hours of speech extracted from the Finnish radio and television archives. We evaluated the model performance with different decoding algorithms and examined the models’ internal behavior with interpretation-based techniques.

## introduction self-supervised learning (ssl) has caused a paradigm shift in automatic speech recognition (asr), enabling the development of highly accurate end-to-end models even with a limited amount of data. low-resource languages also benefited from this advancement, as models pretrained on other languages proved to be a good foundation for the development of asr models using small supervised corpora @xcite . northern sámi, a language spoken by only about 20,000 people has also seen rapid advancements in speech technology (hiovain-asikainen and de la rosa, 2023; @xcite . while fine-tuning speech foundation models such as wav2vec 2.0 @xcite ) can now be considered standard procedure, choosing the right pre-trained system is still very critical. several works have reported that monolingual pre-training tends to produce the best foundation @xcite @xcite , which could be impossible without access to large speech-only corpora. alternatively, continuing the pre-training of an existing model could adapt it to new languages @xcite . in this work, we build speech foundation models for northern sámi with about 22,400 hours of speech from radio broadcasts, which puts them on par with most publicly available monolingual speech foundation models for high-resource languages @xcite @xcite @xcite @xcite . in the past, various training methods have been explored for wav2vec 2.0. still, its inference is most commonly done via a greedy decoding algorithm. here, we explore whether a more advanced technique called prefix beam search @xcite could lead to better results. the main issue with the standard greedy algorithm stems from the blank symbol, which usually receives a considerable portion of the probability mass @xcite , thus leading to spiky outputs and many deletion errors. to avoid this unwanted effect, prefix beam search merges multiple paths that would result in the same output, lessening the suppression effect of the blank output. while this technique was originally proposed to be used with recurrent models, its variants have been successfully utilized with large ssl models @xcite and encoder-decoder-based architectures @xcite too, albeit those works also employ an external lm during the decoding procedure. in contrast, we only utilize prefix beam search to decode the wav2vec 2.0 model without any lm parts, as low-resource languages often lack in terms of text data too, which prevents the development of a good lm. besides the training and decoding algorithms, we also take a closer look at our models' mistakes and propose a new interpretation-based solution to learn more about the reasons for the misrecognition. one of our main observations revealed systematic, repeating mistakes, which we hypothesized were due to the dominance of the internal lm developed by the model during the finetuning phase @xcite . to validate this hypothesis, we utilized the integrated gradients (ig) technique @xcite to investigate whether the model behaves differently when it predicts various characters. our experiments revealed that several characters which caused the problems were predominantly outputted by using mainly the long-term information embeddings while ignoring the current acoustic information. furthermore, we have found that the model dedicated considerably more neurons towards detecting the rare sámispecific characters compared to the common latin characters. in summary, in this paper, we made the following contributions: ## data for pre-training the sámi models, we extracted 35,614 hours of radio broadcasts of yle saamen radio. the broadcasts have been originally recorded by the radio and television archive (rtva) since 2009 and provided for research by the finnish national audiovisual institute (kavi). since the raw dataset also contained a considerable amount of non-speech events, including music and silence, we pre-processed it with a neural voice activity detector (vad) @xcite . after that, continuous speech segments longer than 30 seconds were split into shorter utterances. the final size of pre-training data was 22,415 hours, meaning that nearly 37% of the audio was recognized as non-speech. for the asr fine-tuning, we used the sámi parliament data foot_0 featuring about 20 hours of transcribed speech. for testing the asr models, 1 hour of out-of-domain read-aloud and spontaneous speech of varying audio quality was used. ## experiments we pre-trained the foundation models with the fairseq toolkit @xcite . pre-training was done on 512 gpus of the lumi supercomputer foot_1 for 125,000 steps (approx. 115 epochs) for the base models (95m parameters) and 167,000 steps (approx. 100 epochs) for the large ones (317m parameters). the models were then fine-tuned on the sámi parliament data for 60 epochs with huggingface transformers @xcite . in continued pre-training, we adapted models originally pre-trained on european parliament plenary session recordings @xcite . the base model was a monolingual finnish foundation, while the large one also included speech from two other uralic languages (hungarian and estonian). we evaluated the models with the standard asr performance metrics such as word and character error rate (wer and cer) and compared them to existing asr solutions, including whisper (radford et al., 2023) fine-tuned on 34 hours of spontaneous northern sámi (hiovain-asikainen and de la rosa, 2023) and xls-r @xcite first fine-tuned on high-resource finnish data and then adapted to northern sámi with the sámi parliament data @xcite . next, we performed statistical significance tests on both the word and character levels using the matched pair sentence segment approach. to run the tests, we employed the sctk toolkit foot_2 . looking at the models with continued pre-training, models of both sizes gave significantly (p ď 0.001) lower cer compared to pre-training from scratch, but only the large one significantly (p ď 0.05) outperformed its counterpart pre-trained from scratch on the word level. switching from greedy decoding to prefix beam search further improved the cer. on the word level, however, a significant improvement can be observed only for the large model pre-trained from scratch, while it insignificantly changed the error rate in either direction for the rest of the models. a more detailed analysis of the results revealed that the prefix beam search always increased the number of substitutions and insertions but decreased the number of deletions compared to greedy decoding. overall, the best results were obtained by continued pre-training of the large model. it gave a noticeable improvement on a character level over pre-training on the same data from scratch (14% relative cer reduction), which may suggest that continued pre-training allowed the model to benefit from acoustic patterns learned from other languages and combine them with the newly learned acoustic information of the target language. on the other hand, minor changes in the wer and the distribution of error rates in figure 1 may indicate that the gained language knowledge was still not sufficient enough to properly recognize complete words. ## analysis of the results to better understand how our best model (large-22k cpt + prefix beam search) works, and why it makes certain mistakes, we first inspected the character-level confusion matrix on the test data, see figure 2 . overall, most characters could be recognized with relatively good accuracy, and only a few rare characters like å, ä, x, ö have extremely low recognition rates. while these mistakes can be explained by the lack of training data, we also noticed other systematic problems on the word level. one such issue was related to the word "na" (in english: "well"), which was quite common in the training data. interestingly, in the test set other similar words, like "ni" and "no" were almost al-system wer, % cer, % xls-r eft @xcite 47.70 15.15 whisper (hiovain-asikainen and de la rosa, 2023) 43.15 14.05 base-22k 43.07 16.50 base-22k + prefix beam search 43.12 16.20 ˚˚b ase-22k cpt 43.04 15.76 base-22k cpt + prefix beam search 42.74 15.51 ˚˚l arge-22k 33.32 12.76 large-22k + prefix beam search 32.94 ˚˚12.51 ˚˚l arge-22k cpt 32.28 10.83 large-22k cpt + prefix beam search 32.29 10.76 ways replaced by the word "na", which implied that the model developed a strong internal lm, which forced it to predict the character "a" after the letter "n", especially at the beginning of the sentence, when the model has limited context. to validate this hypothesis about the internal lm, we employed our proposed solution to better understand why our best wav2vec 2.0 made certain mistakes. our first observation was that the overall attribution ratio on the whole set was above 1.3, proving that the temporal information introduced by the positional embedding layer was indeed utilized by the model, but acoustic features were valued considerably more. furthermore, we saw that the blank and the word boundary symbols have the lowest ratios, signaling that they were predicted mostly based on acoustic information. looking at the actual characters, we saw that the average ratio was approximately 2, suggesting that, on average, the temporal information introduced in the positional embedding layer was as useful as the acoustic features extracted by the cnn component. after a closer look at the ratios per character (see figure 3a ), we identified two groups; in the first one, the ratio was below 2, suggesting that these were predicted mainly using acoustic features. this group includes characters such as "h", "i", "n", etc. on the other hand, we can see several characters, including "a", which were primarily predicted by the influence of the internal lm. these results imply that for some characters the acoustic component of the model was not good enough, and it would benefit from seeing additional training material with more diverse textual content in order to force the model to rely more on the acoustic information. next, we investigated the counterpart of the best model, trained from scratch (large-22k), see figure 3b . this model demonstrated a quite different behavior: all tokens except the blank la-bel had an attribution ratio above 2, meaning that the system's output was determined mostly by the temporal information added by the positional embeddings. the average attribution ratio for nonblank characters was 2.7, signaling that the acoustic component had a considerably smaller attribution towards the output than the internal lm. considering that the model was pre-trained only with a relatively small dataset, we can conclude that the acoustic component produced by the continued pre-training is more appropriate and extracts more relevant information. the purely northern sámi model's overreliance on temporal information indicates that it most probably obtained most of its knowledge by simply memorizing parts of the training transcripts during the fine-tuning phase, as large models are prone to do so @xcite . validating this theory is out of the scope of this paper, but remains an important future task. lastly, we also investigated individual neurons in the two selected layers. here, we aimed to find out which character needed the most actively contributing neurons. we looked at each neuron's attribution values per character. first, we calculated the average and standard deviation of the attributions in each layer. our first observation at this stage was that the majority of the neurons had an attribution close to the mean (which was approximately 0 in all cases), and only a few neurons displayed large attributions similar to the findings of @xcite . based on these observations, we decided to separate the neurons into two groups; the highly contributing ones, whose accumulated attribution was farther than one standard deviation from the mean, and the rest categorized as low-contributing. figure 4 illustrates the amount of highly attributing neurons in each investigated layer of the best model. the first observation is that common characters like "r", "b" and "k" required only a few dedicated neurons, while special sámi characters like "•" and "ae" were predicted based on a large number of neurons. in general, many latin characters required less than a 100 highly contributing neurons, while many sámi charaters needed more units. this implies that the acoustic features of the cpt model were quite good for most latin characters that were well represented in the original pre-training corpus, while some ("d", "c" and "t") required more units, perhaps due to non-standard pronunciation. additionally, we can see that the model dedicated a larger portion of neurons to the sámi specific outputs, implying that despite the language adaptation via cpt and finetuning, it still has difficulties recognizing them well. ## limitations while experimental results suggest that prefix beam search is beneficial on the character level, its wers proved to be quite similar to the greedy decoding algorithm's. as the lower cer suggests better quality output, testing its readability by humans and comparing it to the greedy alternative remains an important future task. additionally, we should mention that here, we utilized the prefix search without any modifications, but it might benefit from adjustments in terms of hyperparameters and vocabulary usage of wav2vec 2.0, especially regarding the word separator symbol. while our model interpretation experiments have revealed interesting facts about the internal functions of the models, they should be rigorously tested and validated. on the one hand, interpretation techniques are known to be fragile @xcite . thus, our experiments should be repeated with other attribution estimation methods to ensure that our observations hold. furthermore, we made several simplifications in this work, including the decision to accumulate the attributions over time, thus ignoring their changes in different contexts. in the future, we intend to investigate how the attributions' trajectories change over time and in different contexts to gain a deeper understanding of when temporal information is valued more than acoustic information. lastly, all of our findings should be validated by the use of a reliable ilm estimation method. unfortunately, currently, no such technique is available for nonautoregressive models.