Fine-Tuning ASR models for Very Low-Resource Languages: A Study on Mvskoke (2024)

Abstract: Recent advancements in multilingual models for automatic speech recognition (ASR) have been able to achieve a high accuracy for languages with extremely limited resources. This study examines ASR modeling for the Mvskoke language, an indigenous language of America. The parameter efficiency of adapter training is contrasted with training entire models, and it is demonstrated how performance varies with different amounts of data. Additionally, the models are evaluated with trigram language model decoding, and the outputs are compared across different types of speech recordings. Results show that training an adapter is both parameter efficient and gives higher accuracy for a relatively small amount of data.

## introduction endangered languages are often overlooked in research on speech technology and other nlp applications. research obstacles include data scarcity and the effort it takes to collect new data, as well as funding and a perceived limited impact on small speech communities. however, these technologies can be hugely beneficial to assisting communityled language revitalization efforts and are worthy of the effort it takes, if it is done with consideration and care for the speech community. automatic speech recognition (asr) technology can help speed up transcription and documentation work, as well as be a stepping stone to other applications such as spoken term detection, which can help in identifying certain topics or key information contained in recordings. other useful applications for the speech community are speech-to-text input and automatic subtitling. these applications can be helpful in encouraging use of the language and promoting language education. asr is a relatively mature technology when applied to high-resource languages @xcite . but it is only more recent advancements such as model size and multilinguality that have enabled comparable accuracy for resourceconstrained situations @xcite . this work focuses on the evaluation and analysis of two highly multilingual speech models when trained for mvskoke, a language indigenous to the southeastern united states @xcite . ## data the texts and recordings used in these experiments primarily come from language documentation work conducted over the last few decades. two documentation books, by @xcite and @xcite are collections of stories, historical letters, and other cultural documents. a portion of these texts were recorded in a studio setting by two female speakers. in order to incorporate male speakers and spontaneous speech, a small segment of the new testament was selected, as well as a few short sections of recorded interviews. splits. train and development sets are split 90/10 at run-time. two evaluation sets are kept separate from the training set. "eval (clean)" is read speech from the same documentation sources as the training set, and "eval (other)" is noisier speech, consisting of one overlapping male speaker and one held-out female speaker. in the transcripts for all the audio data, there are a total of 6,840 utterances and 19,154 words, for an average of 2.8 words per utterance. the train and "eval (other)" sets include both read and spontaneous speech, while the "eval (clean)" set is only read speech. other features of the datasets are shown in language model. the text data for the language model (lm) includes the two books above as well as the transcriptions from a series of interviews conducted by the pumvhakv school in 2015. for these experiments, the interview recordings are not used for training due to noise including nature sounds, speech errors, and singing, but the transcriptions provide valuable vocabulary. the texts and transcriptions of the evaluation set were excluded from the text training data. the text corpus used for lm training has 118,021 words and 27,795 unique words. at this time, the dataset will not be publicly released due to copyright constraints of the source material. currently, the muscogee (creek) nation is working to consolidate data and establish language resource policies. however, much of the source of the data can be viewed on the muskogee documentation project website foot_1 . ## methodology the goal of this work is to evaluate the effectiveness of fine-tuning an adapter for a large multilingual model. this is one state-of-the-art path for asr that requires less manual work than other methods such as an hmm, and generally requires less data due to the existing pre-trained acoustic knowledge of the multilingual models. additionally, other aspects that are evaluated are how much data is required and whether or not a language model can improve results. ## results the mms-1b-l1107 performed best overall, with best results of 37% word error rate (wer) and 5% character error rate (cer). results are shown in figure 1 . ## conclusion and future work this study shows that fine-tuning multilingual transformer models is an effective method for train-79 ing asr systems in low-resource language contexts. fine-tuning the adapter for a 1 billion parameter model, mms-1b-l1107, yields better results when compared to training entire models such as xslr-53 and mms-300m. however, the performance of such systems depends highly on the recording quality and type of speech. although language modeling improves overall accuracy measures such as wer and cer, it can also degrade the output in some cases. alternatives like subword or character-level modeling could offer a more effective approach, particularly for applications where fidelity to the original speech signal is preferred. a future direction would be to incorporate the asr model into a keyword-spotting or sparse transcription system. the high error rates for noisy recordings in this study mean that manual transcription may still be faster than correcting asr output. sparse transcription can be helpful in situations where high asr error rates lead to lowquality transcriptions @xcite . transcribing only high-confidence words can be useful for indexing recordings and providing an overview of recorded content that can then be used for knowledge gathering.