Speech Personalization using Parameter Efficient Fine-Tuning for N epali Speakers (2025)

Abstract: The performance of Automatic Speech Recognition (ASR) systems has improved significantly, driven by advancements in large-scale pre-trained models. However, adapting such models to low-resource languages such as Nepali is challenging due to the lack of labeled data and computational resources. Additionally, adapting the unique speech parameters of the speaker to a model is also a challenging task. Personalization helps to target the model to fit the particular speaker. This work investigates parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) and Decomposed Weight Low-Rank Adaptation (DoRA) to improve the performance of fine-tuned Whisper ASR models for Nepali ASR tasks by Personalization. These experiments demonstrate that the PEFT methods obtain competitive results while significantly reducing the number of trainable parameters compared to full fine-tuning. LoRA and DoRA show a relative WER to FT_{Base} increment of 34.93% and 36.79%, respectively, and a relative CER to FT_{Base} increment of 49.50% and 50.03%, respectively. Furthermore, the results highlight a 99.74% reduction in total training parameters.

## introduction automatic speech recognition (asr) systems like voice assistants are widely used with the rapid development of deep learning models @xcite . however, the system's performance depends on the diversity of the speech data. the model performs poorly on a different speaker with different speech characteristics that were not initially trained on. personalization of the speaker helps fill that gap by making the asr model work with the unique characteristics of the individual speaker. training an asr model requires highquality speech data @xcite . collecting and training user-specific speech data for the model is challenging due to factors that consider user privacy. most asr applications are used on lightweight handheld devices with limited processing power. customization of the model by fine-tuning the model based on user data is very difficult and inefficient due to the significant training parameters. techniques such as residual adapter for fine-tuning @xcite , federated learning on devices by adopting the subset of weights @xcite , and finetuning the model's attention and bias independently @xcite . asr tasks for low-resource languages and indo-aryan languages like nepali differ due to the language's structure and nature @xcite . currently, parameter efficient finetuning (peft) is often used for large language models (llms) because it lowers the computation power required to tune the model. the peft strategies, like lora and dora, are used to adapt the models with large trainable parameters. due to the low use of resources by the adapted and merged model, it allows the large model to be inference using a consumer-grade gpu @xcite . approaches like lora and dora are implemented for improving the efficiency of some state-of-theart asr models @xcite . here, an efficient speaker personalization approach using peft, two approaches, lora and dora, is proposed. the approach uses the low-rank approximation to efficiently adopt the asr model for the targeted speaker with the limited weight addition, reducing the computation and memory restrictions @xcite . ## related works traditional automatic speech recognition (asr) architectures employed hidden markov models (hmms) together with gaussian mixture models (gmm-hmm) to efficiently capture temporal dy-namics and phonetic transitions @xcite . however, hmms suffered in terms of speaker variability, strict temporal assumptions, and scalability limitations @xcite . the development of hybrid architectures with deep neural network -hidden markov model (hmm-dnn) combinations improved robustness but at the expense of heavy computational resources @xcite . major architectures used for speech recognition are @xcite , lstm @xcite , bilstm @xcite , rnn @xcite , gru @xcite . traditional and recent asr model uses a combination of the above architectures to perform speech transcription tasks in the nepali asr domain @xcite . the transformer @xcite based models work great for nepali asr tasks @xcite . for speech personalization, different approaches like the deep neural network (dnn) based acoustic modeling @xcite , (singular value decomposition) svd based compression scheme @xcite , user feedback (mahesh krishnamoorthy, 2016), controllable speech synthesis approach @xcite , enhancing quantized model @xcite were used. before that, the data sparsity issue was solved by using speaker dependence with condensed vectors, reducing parameters during model adaptation using some regularization approach @xcite @xcite . the peft strategies used with llms includes methods like adapt-former @xcite , visual prompt tuning (vpt) @xcite , low rank adaptation (lora) @xcite , weight-decomposed low-rank adaptation (dora) @xcite , and scaling & shifting your features (ssf) ( @xcite , among those the lora and the dora are implemented for improving the efficiency of the llms like gpt-2/3 @xcite , roberta @xcite , and in some stateof-the-art asr models @xcite . currently, the model is biased per speaker by fine-tuning the model's attention based on methods such as implementing a residual adapter for fine-tuning, federated learning on edge devices, and adopting a subset of weights. there is also a way of targeting a speaker for speech synthesis @xcite , which uses the cleaned version of speaker data for asr models to process @xcite . speech data is prepro-cessed to improve speech parameters fed to the asr model, thus enhancing the performance of the asr model in noisy backgrounds @xcite . experiments have also shown that a small volume of disordered speech of an individual's training data can benefit from personalized asr @xcite . fine-tuning the base model has also shown an improvement in the performance of the asr for the nepali language using active learning @xcite and @xcite . the transformer @xcite model has attention mechanism components, which are used by low-rank adaptation as the q, k, and v target modules for speech personalization, which can be used to personalize the asr system without compromising the model's performance @xcite . many commercial products like google home foot_0 , amazon alexa foot_1 , apple siri foot_2 , microsoft copilot foot_3 , and different voice assistants also perform speech personalization to make interaction with users easier @xcite . the use of lora and dora is increasing in low-resource languages as well, and this motivated us to perform speaker personalization on the nepali language to improve the overall interaction with the asr system. ## dataset a portion of the commonvoice17 @xcite dataset with slice/split of ne-np/validated was taken for four speakers, and two speakers audio data was taken from the nepds @xcite dataset by taking the speaker having a commutative speech duration of more than 4 minutes. the data from both datasets is compiled and merged to form a compiled dataset (commonvoice17 and ilprl, 2025). the speakerspecific utterances are ordered reversely based on the number of utterances. first, six speakers were selected based on training data ranging from 4 to 18 minutes.