Improving on the Limitations of the ASR Model in Low-Resourced Environments Using Parameter-Efficient Fine-Tuning (2024)

Abstract: Modern general-purpose speech recognition systems are more robust in languages with high resources. In contrast, achieving state-of-the-art accuracy for low-resource languages is still challenging. The fine-tuning of the pre-trained model is one of the highly popular practices which utilizes the existing information while efficiently learning from a small amount of data to enhance the precision and robustness of speech recognition tasks. This work attempts to diagnose the performance of a pre-trained model when transcribing the audio from the low-resource language. In this work, we apply an adapter-based iterative parameter-efficient fine-tuning strategy on a limited dataset aiming to improve the quality of the transcription of a previously fine-tuned model. For the experiment we used Whisper’s multilingual pre-trained speech model and Nepali as a test language. Using this approach we achieved Word Error Rate of 27.9%,which is more than 19% improvement over pre-trained Whisper Large − V2.

## introduction automatic speech recognition (asr) is a subset of speech technology that uses machine learning techniques and neural networks to analyze and transcribe audio recordings or convert real-time speech into text. with the emergence of deep learning methods in recent years, speech-based technology has made significant advancements. machine learning-based asr systems can be trained using supervised, semi-supervised, or unsupervised techniques. supervised asr systems acquire knowledge via precise alignment of spoken and transcribed text, requiring a substantial amount of highly curated data from manual alignment. a lot of time and work has to go into manually aligning spoken words with their written versions to make sure they match up correctly. the initial proposal for unsupervised asr implementation was presented by @xcite . following this, unsupervised methods also have gained widespread popularity. a recent study by @xcite revealed that the performance of the unsupervised model is equally comparable to that of supervised models. the availability of the larger pre-trained speech models are increasing in number with access of the computing resources and advancement in deep learning techniques. large pre-trained speech models are typically trained on thousands of hours of diverse speech datasets. recently released w av2v ec2 -bert 2.0 @xcite was pre-trained on 4.5m hours of unlabeled audio data covering more than 143 languages. in line with this, the whisper -large model @xcite ) is trained on 680, 000 hours of labeled audio data and has 1550m parameters. these models excel at capturing complex acoustic and linguistic patterns, enabling them to generalize effectively across multiple languages, accents, and noises. having said that their size also poses challenges, such as the need for significant computational resources and the risk of over-fitting particularly in low-resource environments. the large speech model can be fine-tuned for any speech-related downstream tasks. the full parameter fine-tuning paradigm requires multiple graphical processing unit (gpu) working in parallel which is very inefficient and non-sustainable. parameter-efficient fine-tuning (peft) based approaches has gained popularity due to its efficiency and effectiveness in adapting pre-trained models to specific tasks, especially in resource-constrained environments. unlike full parameter fine-tuning, the peft-based methods introduce the concept of a small adapter that can be trained leaving the majority of the pre-trained parameters untouched. this approach significantly reduces the number of trainable parameters, making it computationally efficient and reducing the risk of over-fitting, particularly in low-resource settings. @xcite proposed the fine-tuning technique by freezing the pre-trained model weights and injecting the trainable rank decomposition matrix to each transformer layer called low-rank adaptation (lora). this approach is effective in the context of the large language model (llm). for example in gp t -3175b model, lora reduced the trainable parameters by 10, 000 times and gpu requirements by 3 times @xcite . in this paper, we propose the fine-tuning architecture for the low-resourced language, nepali. we first introduce the language to the model by fullweight fine-tuning. the limitations of the resulting module are identified and later adapter-based fine-tuning is applied to enhance the overall quality of the asr. additionally, we investigated the use of lora @xcite for fine-tuning the w hisper -large model @xcite . the rest of the paper is organized as follows: in section 2 the related works are explained followed by the methodology in section 3. section 4 and 5 presents the conducted experiments and discussion of results. finally, the paper concludes with section 6 where summary of findings, future plans, and potential extensions to the work are explained. ## related works pre-trained large speech models have revolutionized the speech-related downstream task such as asr. there are various types of the pre-trained model that we can use for the fine-tuning task. multilingual supervised, semi-supervised, and unsupervised -all type of the model can be effectively fine-tuned. wav2vec2-conformer @xcite , whisper @xcite , mms-1b (pratap et al., 2024), hubert @xcite , wav2vec2-bert 2.0 @xcite , wav2vec2-phoneme @xcite , wav2vec2.0 @xcite , w av2v ec @xcite ) are some of the example of pre-trained speech models trained on massive amounts of multilingual speech datasets. w hisper, for example, is a powerful encoderdecoder model that can transcribe speech into text in multiple languages. w av2v ec and its successors @xcite @xcite use contrastive learning to learn robust speech representations. these pre-trained models have significantly improved the accuracy and robustness of asr systems, making them more accessible and useful in a variety of applications. various published research @xcite @xcite @xcite shows that the accuracy of the asr in low-resource languages including nepali can be improved by fine-tuning the pre-trained models @xcite . as per these researches, the fine-tuning approach requires less computing and also reduces the model training time significantly compared to full parameter training. however due to higher number of the parameters involved in the network the full parameter fine-tuning is still challenging. the lightweight adapter tuning for speechrelated task has been addressed by @xcite . as per the author this type of the adapter tuning can be used to -(1) fine-tune the large and generic model for downstream task, and (2) glue the two adapters to find solutions to new downstream tasks. this type of the adapter can be merged into the pretrained model either in serial or parallel fashion to produce the output. use of the parameter-efficient fine-tuning approach lora and its variants are becoming very common on speech model fine-tuning. @xcite used lora for fine-tuning the w hisper model on child speech dataset. they found that lora-based techniques are very effective in fine-tuning the w hisper model on a lowresourced child dataset. @xcite proposed the parameter-efficient and extensible model for w hisper fine-tuned with lora called lora-whisper. the lora-whisper yields a relative gain of 18.5% over baseline system for multilingual asr model. the lora based model adaptation mechanism naturally allows multiple lora modules can be formed and merged. loading multiple lora and merging them with main model for inference purpose still requires higher memory capacity. to address this situation, @xcite has proposed a method in which we can serve thousands of concurrent lora adapters called s-lora. @xcite demonstrate that fine-tuning using lora is much more effective on the pre-trained models for low-resource asr. the nepali asr is still in its early stages of research and development. however, there are some promising results as reported in previous works @xcite @xcite @xcite . among them, the work reported by @xcite is the only work related to fine-tuning for building nepali asr system. the author proposed semi-supervised fine tuning of the pre-trained model using an active learning approach. this research uses the slr54 @xcite dataset for the full parameter fine-tuning resulting 6.77% cer on the massively multilingual speech (mms)-1b (mms1b) (pratap et al., 2024) model. the use of lora and similar parameter-efficient fine-tuning approaches for speech-related downstream tasks is increasing, even in the case of lowresourced languages. these approaches are designed to solve downstream tasks. unfortunately, it should be noted that the use of multiple lora adapters to enhance the output is the least explored. we can train and adopt multiple adapters, which reduces the limitations of each other. this is the primary motivation for this work and we have experimented with this approach in low-resource nepali asr. ## experiments we used the slr54 dataset @xcite which has a nepali speech corpus of 157k utterances and has 165 hours of speech recording featuring 527 unique speakers. among these 4 hours of speech dataset that are used as ds train , 20 minutes are used as the validation dataset ds val and the remaining are used for the pool or stock dataset that can be used for further adapter training (ds stock ). the w hisper @xcite model family is used as the pre-trained speech model. they have tiny, base, small, medium, and large models ranging from 39m parameters to 1550m parameters. we used a pre-trained w hisper -large -v 2 model. this pre-trained model also receives nepali language training. so, there is no need to explicitly introduce the vocabulary. for this experiment, we used the hugging face transformer library 4 . to make our experiment comparable, we trained each adapter with a total of five (5) epochs with the following parameters: parameter-efficient fine-tuning is implemented using the peft library @xcite . the best combination of lora configurations is estimated by using a hyperparameter tuning technique. the best estimated parameters are as follows: to check the effectiveness of the module, a total of 3 adapters are fine-tuned, namely adapter 1 , adapter 2 , and adapter 3 . all of these adapters are linearly merged to form adapter merged with the weights β as in equation( 1 ). the value of β obtained from the hyperparamerter tuning is as follows: fine-tuning may be performed by training any number of adapters until the model starts to overfit. we observed overfitting after three iterations of fine-tuning because of the limited size of the dataset employed in our trials. this discovery highlights the necessity of carefully overseeing model performance during fine-tuning to avoid overfitting and guarantee effective generalization to unseen data. the newly produced adapter, termed adapter merged is later integrated with the fine-tuned basis model (f t base ). upon completion of the integration, an assessment is performed to evaluate the performance of the merged model. ## results and discussions we used w hisper -large -v 2 pre-trained model. this is a multilingual large speech model also trained in nepali speech as well. the reported word error rate (wer) of this model is 47.1%. while inspecting the output, we saw that the transcript of the module is more toward hindi language. hence, we performed full-parameter fine-tuning using our dataset and ended up with a wer of 36.2%. this language (re)introduction by full-parameter fine-tuning improved the performance of the base model. now, our base model is f t base with 36.2% wer. all experiments are listed in three different low-rank adaptation (lora) based adapters are trained as per section 3.3 and algorithm 1. the individual wers of the adapter range from 33.1% to 30.1%. the best wer is achieved by combining the three adapters together, which is 27.9% foot_0 . this is more than 8% improvement over full-parameter fine-tuned base model (f t base ) and more than 20% improvement over the pre-trained model w hisper -large -v 2. our work can also be compared with existing fully supervised end-to-end nepali asr models. the hidden markov mode (hmm) based model proposed by @xcite has achieved 29.45% wer. the cnn-gru-based model proposed by @xcite has reported a wer of 37. 50%. in all of these cases, our model performs better. the fine-tuned model based on active learning proposed by @xcite has reported the 6.77% cer. because of the different evaluation matrices, we could not compare the result. the results of our investigation indicate that combining numerous lora adapters, each trained on different subsets of data, allows them to complement each other's strengths and minimize shortcomings. an adaptor may capture only specific patterns or features, resulting in a higher wer. however, when integrated, these adapters can offer a more thorough adaptation by addressing the shortcomings of the others. this collective effect improves the model's capacity to generalize and reliably recognize speech, resulting in a lower overall wer. ## conclusion fine-tuning of large language models for lowresource languages is a prevalent and growing practice, particularly in the context of speech-related tasks. due to nepali being a low-resourced language, the fine-tuning task has been comparatively less investigated. our study focused on evaluating the usefulness of adapter-based fine-tuning through experiments conducted with lora and whisper. our strategy involves initially (re)introducing the language into the bigger model through full parameter tuning. subsequently, the shortcomings of the model are recognized, and by addressing those drawbacks, we refine certain aspects of the model using lora, thereby improving the overall quality of the model. using this methodology, we achieved a word error rate (wer) of 27.9%, which is an improvement over 19.2% as reported in previous work. although we observed a significant improvement in implementing the suggested strategy, there is still plenty of space to improve the precision of the model. the algorithm for generating ∆ds uses the cer compared to some threshold. however, to achieve further improvements, this area can be analyzed using language-specific similarity metrics. additionally, at this stage, our study focused only on whisper. the study could be extended to other larger models to compare the corresponding results.