Active Learning Approach for Fine-Tuning Pre-Trained ASR Model for a Low-Resourced Language: A Case Study of N epali (2023)

Abstract: Fine tuning of the pre-trained language model is a technique which can be used to enhance the technologies of low-resourced languages. The unsupervised approach can fine-tune any pre-trained model with minimum or even no language-specific resources. It is highly advantageous, particularly for languages that possess limited computational resources. We present a novel approach for fine-tuning a pre-trained Automatic Speech Recognition (ASR) model that is suitable for low resource languages. Our methods involves iterative fine-tuning of pre-trained ASR model. mms-1b is selected as the pretrained seed model for fine-tuning. We take the Nepali language as a case study for this research work. Our approach achieved a CER of 6.77%, outperforming all previously recorded CER values for the Nepali ASR Systems.

## introduction automatic speech recognition (asr) systems use machine learning algorithms and neural networks to analyze and transcribe audio recordings or live speech into text form. asr has a wide range of applications, including -voice assistants, transcription services, accessibility tools, language translation services, voice search etc. asr technology has advanced significantly in recent years following the development of deep learning techniques, and it continues to play a crucial role in the development of natural language processing and human-computer interaction systems. the supervised asr systems learn from aligned speech and transcribed text pairings, requiring lots of well-prepared data. speech and transcribed text must be perfectly aligned and the alignment basically is done manually in the supervised approach. the manual alignment demands extensive time and effort as spoken words must be precisely matched with their written counterparts. unsupervised asr implementation was first proposed by @xcite . since then unsupervised approaches have been in popular use. recently, the research conducted by @xcite demonstrated that the performance of unsupervised asr is equivalent to supervised asr models performance. however, developing an asr model for lowresourced languages is a challenging task. the pre-trained model could be fine-tuned with some limited available labeled speech corpus. this fine-tuned model is then used to label the unlabeled corpus. the labeling achieved can be augmented to the manually labeled speech corpus. the whole process can be described as a semi-supervised learning approach which is a combination of supervised and unsupervised approaches. also, there is yet another approach called transfer learning which is also becoming predominantly popular in such contexts. transfer learning as the name suggests is about applying a model trained and developed for one purpose or task to a somewhat related but quite different task and purpose. similarly, active learning is focused on selecting the most informative examples for labeling to improve the model's performance since manual labeling is resource-intensive. while they both aim to enhance machine learning models, they address different aspects of the learning process: transfer learning focuses on the use of other supportive models, while active learning concentrates on data selection and annotation. the concept of the active learning are being popular in the asr domain after popularity of the larger asr pre-trained model such as hu-bert @xcite , wav2vec @xcite , wav2vec2.0 @xcite @xcite , mockingjay @xcite . these models are trained using larger and multilingual speech corpus. many research @xcite @xcite @xcite show that the accuracy of the asr in low-resourced languages can be improved by fine-tuning the pre-trained models. the finetuning approach also saves the model training time significantly. in this paper, we propose an active learningbased self-training approach to improve the quality of the asr model. we used language score as a active learning dataset selection criteria. wav2vec2.0 based massively multilingual speech (mms) -1b (mms-1b) pre-trained asr model is used as a seed model for fine tuning. the rest of the paper is organized as follows the related works and the methodology used are explained in section 2 and 3 respectively. section 4 explains the dataset used for training the model. section 5 presents the conducted experiments and their results. finally, the paper concludes with section 6 where summary of findings, future plans, and potential extensions to the work are explained. ## related works self-supervised approaches have been popular for the past few years, especially in the context of low-resourced languages. the goal of this approach is to optimistically utilize the large unlabeled speech data to improve the overall accuracy of any given task in low-resourced languages. there are various large models available for asr task. @xcite has proposed a hidden-unit bert (hubert), a self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a bert-like prediction loss. @xcite combined various speech dataset of 40 indian languages (17,000 hours of raw speech data) including nepali and trained wav2vec pre-trained model called indicwav2vec. @xcite has proposed a framework for the semi-supervised learning of speech representation. the authors trained unsupervised model on 53k hours of unlabeled data to produce pre-trained seed model. that model is further fined tuned using just 10 minutes of labeled data. they achieved 4.8/8.2% wer on clear/other test sets. this demonstrates the feasibility of speech recognition with limited amounts of labeled data. the data augmentation-based approach is another way to take benefit of the larger resources. @xcite utilized the transliteration based augmentation to address the labeled data scarcity. high-resource graphemes of english are transliterated to lowresourced languages graphemes( experimented on : hindi, telugu, gujarati,bengali, korean, amharic). those transliterated dataset are used for extended pre-training of the seed model. the fine-tuning of the pre-trained model on 10 hours of gujarati dataset significantly reduced wer from 55.8% to 32.4%. @xcite has proposed ensemble model to combine the hubert and wav2vec2.0 to fine-tune asr models. the performance of ensembled approach is compared with model generated from fine-tuning of indivisual hubert and wav2vec2.0 model. on the librispeech dataset the ensemble model performed better. @xcite further explored the data selection approach for active learning. the authors extracted different level of granularity (k-means id, phoneme, word) from unlabeled data using using hubert, wav2vec-u2.0 and wfst. they computed perplexity (ppl) and used ppl-based contrastive data selection approach. the proposed data selection framework effectively improved the word error rate (wer) by more than 11%. the utilization of pseudo-labeling for unsupervised datasets has gained popularity as a means to mitigate the substantial effort and cost associated with generating labeled speech datasets. @xcite implemented iterative pseudo-labeling(ipl) technique for fine tuning the pre-trained model.this approach improved the word error rate on the lib-rispeech test dataset in both standard and low-resourced settings. @xcite also experimented the pseudo-labels based selftraining approach for low-resourced speech recognition. they fine tuned wav2vec2.0 based cross-lingual xlsr-53 model @xcite on the punjavi speech dataset. the unlabeled dataset are labeled with pseudolabels and marked with confidence score. those data that pass the threshold confidence score are selected for the next level of finetuning. the process is repeated until the model stops improving. @xcite has fine-tuned their proposed indicwav2vec model using nepali openslr dataset. when larger language model (lm large ), augmented lexicon, and rescoring is used they improved the performance of model for nepali speech input. as per @xcite the pseudo-labeling and selftraining approach on wav2vec2.0 can significantly improve the output in the low-resourced setting. the noisy student training with specaugment using giant conformer models pre-trained using wav2vec2.0 is experimented by @xcite . by doing this, they are able to achieve word-error-rates 1.4% on the librispeech. ## dataset the validation of proposed model is conducted in a low-resourced environment. the nepali language is used as the test language. @xcite has developed a crowdsourced speech corpus, known as oslr53, specifically designed for nepali and other languages that have limited resources available. the dataset can be accessed by the public through the open slr website 2 . the dataset consists of a total of 165 hours of recorded speech, which has been contributed by 527 distinct individuals. the dataset has a total of 157, 905 recorded phrases. for the validation of our proposed model we splited oslr53 dataset into three parts -1) labeled train dataset (ds_t rain labeled ), 2) labeled test dataset(ds_t est labeled ), and 3) unlabeled dataset (ds u nlabeled ). the dataset details are summarized into the un-labeled dataset unlabeled dataset preparation task is comparatively easier than labeled dataset but still requires significant time to collect, curate, and pre-process. unfortunately, the speech corpus for the unsupervised asr task is not available. we are working on the unlabeled dataset preparation task. but at the moment we are not in a position to use them. so for the validating our proposed model we have decided to use some portion of the data from oslr53 as a unlabeled dataset after seprating train/test dataset. ## experiment, result and discussion there are couple of pre-trained seed models (as explained in section 2) available for asr task. these pre-trained models are trained on the larger speech corpus of single or multi-lingual speakers. for the experiment we used massively multilingual speech (mms) -1b (mms-1b) as a pre-trained seed model. the character based tokenizer is used to tokenize the label. the devanagari characters that were used in the nepali language are listed in ## conclusion and future work the active learning-based self-supervised approach for developing asr for low-resourced languages particularly nepali is presented in this paper. the proposed method utilizes an iterative training approach, which has proven to be effective, especially in the context of languages with limited available resources. this iterative training process involves repeated cycles of learning and refining, allowing the system to adapt and improve its performance over time. by employing the suggested method, it is possible to eliminate the extensive labeling efforts in creating speech datasets for fully supervised asr models. the experimental analysis demonstrates the effectiveness of our algorithms in terms of improving the performance of the asr model in low-resourced settings of nepali language. this research can be further extended for other languages which has limited language resources. there are several areas for improvement in this research. we have considered only wav2vec2.0 based pre-trained seed model. we can test the algorithms on other pre-trained asr models such as hubert. language model is not being used yet for the decoding purpose. we can use the appropriate language model which is trained from larger text to improve the performance of the overall system. the language model threshold is fixed throughout the training. some adaptive approach for the threshold calculation can be done.