JUNLP @ LT - EDI -2025: Efficient Low-Rank Adaptation of Whisper for Inclusive T amil Speech Recognition Targeting Vulnerable Populations (2025)

Abstract: Speech recognition has received extensive research attention in recent years. It becomes much more challenging when the speaker’s age, gender and other factors introduce variations in the speech. In this work, we propose a fine-tuned automatic speech recognition model derived from OpenAI’s whisperlarge-v2. Though we experimented with both Whisper-large and Wav2vec2-XLSR-large, the reduced WER of whisper-large proved to be a superior model. We secured 4th rank in the LT-EDI-2025 shared task. Our implementation details and code are available at our GitHub repository1.

## introduction automatic speech recognition (asr) has transformed the way humans interact with machines by enabling devices to understand spoken language. it plays a crucial role in enhancing accessibility for individuals with disabilities, such as the elderly and those with hearing or speech impairments @xcite . by allowing voice-based interaction, asr improves ease of communication and overall quality of life for these groups. while asr systems have achieved impressive accuracy in languages like english, low-resource languages such as tamil still face challenges @xcite . tamil, spoken by millions across tamil nadu, sri lanka, and singapore, is linguistically rich and features numerous regional dialects, making speech recognition particularly complex. these challenges are amplified when recognizing speech from vulnerable populations, such as those with dysarthria or slurring @xcite . in this work, we focus on building an inclusive tamil asr system by fine-tuning the whisper model (vasista22/whisper-tamil-large-v2), known for its strong multilingual performance @xcite . to make the fine-tuning process efficient, we use low-rank adaptation (lora), which reduces the computational burden while maintaining high accuracy @xcite . our training dataset includes tamil speech samples from diverse dialects and speakers with impairments. the fine-tuned model achieves a word error rate (wer) of 38.42%, demonstrating significant improvement and the potential of whisper models in developing accessible asr systems for underrepresented languages. ## related work automatic speech recognition (asr) has evolved from hybrid hidden markov model-gaussian mixture model (hmm-gmm) @xcite frameworks to end-to-end deep learning systems. early systems leveraged hmms for temporal modeling and dnns for acoustic feature extraction, achieving significant accuracy improvements over traditional methods. transitioning to architectures like lstms and transformers enabled better sequential context capture, with models like conformer and contextnet integrating convolutional and self-attention mechanisms for spectral and global dependencies @xcite . self-supervised learning paradigms, such as wav2vec 2.0, further advanced low-resource asr by leveraging unlabeled data for robust feature learning @xcite . recent efforts focus on domain-specific challenges, including elderly and vulnerable populations as well as low-resource speech recognition. @xcite presented findings from a shared task on tamil asr for vulnerable individuals, emphasizing the difficulty of recognizing atypical speech patterns in elderly and impaired speakers. their work demonstrated the utility of hmm-dnn hybrid systems @xcite and end-to-end models alongside data augmentation and transfer learning to improve robustness. in a follow-up shared task, @xcite expanded the dataset and evaluated multilingual models (e.g., xls-r, whisper), showing that fine-tuning, domain adaptation, and acoustic normalization techniques effectively addressed speech variations and noise in low-resource settings. similar advances include acoustic model adaptation using age-specific corpora like ears and vote400, which reduce word error rates (wer) by 25% for elderly speech by mitigating spectral and prosodic variations. for lowresource languages, techniques like self-training and text-to-speech augmentation improve wer by 20-25%, as demonstrated for gronings and mvskoke. transformer-based streaming architectures, employing time-restricted attention, balance latency and accuracy, while hybrid hmm-dnn systems remain relevant for stable frame-level processing. despite progress, challenges persist in dataset diversity, real-time adaptation, and computational efficiency for edge deployment. ## dataset description and analysis the dataset focuses on addressing the challenges faced by vulnerable groups, specifically elderly individuals and transgender people in tamilspeaking communities, where elderly individuals often encounter difficulties using digital tools in essential locations like banks, hospitals, and administrative offices, where speech-based systems could significantly ease their interactions @xcite . similarly, transgender individuals, frequently deprived of primary education due to societal prejudice, rely heavily on speech as their primary mode of communication @xcite . by capturing the spontaneous speech patterns of these groups, the dataset aims to facilitate the development of inclusive and accessible asr systems that cater to their unique linguistic needs and daily life challenges @xcite . the dataset contains 908 samples totaling nearly 5 hours of speech. we have split the entire corpus into training (894 samples, 4.87 hours), validation (9 samples, 0.05 hours), and test sets (5 samples, 0.03 hours) for tracking the performance metrics at different stages of model development. in addition to this, we were provided with 2 hours of highquality audio speech data, which will be used for testing purposes after successfully training our best model and following best practices. we conduct spectrogram analysis @xcite on the speech dataset to characterize the time-varying frequency properties of the audio signals. in figure-1 the analysis confirms that all samples exhibit dominant speech energy below 4 khz, with clearly observable formant structures. the overall spectral clarity and low background noise across all samples suggest high-quality recordings. these observations not only confirm the suitability of the data for further speech processing tasks-such as automatic speech recognition or speaker profiling @xcite , but also highlight the diversity in speaking styles and potential demographic differences among the speakers (narayanan and georgiou; ?). such variability is crucial for developing robust and inclusive speech systems that generalize well across different populations. in figure-2 the pitch distribution(deruty et al., 2025) graph reveals a clear multimodal pattern, with a dominant peak near 200 hz and secondary peaks around 100 hz and at higher frequencies, indicating demographic diversity. the use of a logarithmic x-axis reflects the perceptual nature of pitch. variations in peak heights highlight gender imbalance, which may introduce bias in asr performance toward dominant voice types. the dataset exhibits a bell-shaped silence distribution (jin shim et al., 2024) (mean 27.6%, median 26.4%) with a right skew, where most samples contain 10-50% silence (peaking at 25-30%) under a 30 db/16 khz detection threshold (refer figure 3 ). this aligns with natural speech patterns, where pauses constitute approximately one-quarter of spoken content @xcite , informing asr design for effective endpoint detection and robustness @xcite . the balanced silence distribution facilitates training on realistic speech rhythms and timing structures (jurafsky and martin, 2000), improving temporal generalization in deployment scenarios. from the analysis of temporal features (figure 4 ), we found the audio dataset exhibits highquality temporal features with segmented speech (@xmath0.5 units) and precise silence intervals, evidenced by rms energy drops to zero and spectral rolloff between 500-3500 hz. stable rms levels (∼0.4-0.5) during speech segments indicate consistent articulation, while rolloff variations (1000-3000 hz) reflect phonetic diversity, demonstrating complementary temporal-spectral features (waveform, energy, rolloff) that reveal controlled recording conditions ideal for training robust speech models requiring precise acoustic characterization @xcite @xcite . the log-scaled fft analysis of the training dataset reveals concentrated spectral energy (10 1 -10 foot_1 magnitude) in lower frequencies (0-4000 bins) with a sharp roll-off at 4000-5000 bins across samples, indicating bandwidth-limited audio rich in harmonic content (refer figure 5 ). consistent noise floors ( 10 -1 -10 0 magnitude) and spectral homogeneity suggest uniform recording/post-processing conditions, while the preserved harmonic structures and logarithmic energy distribution (aligning with auditory perception) highlight key perceptual features of speech signals @xcite @xcite . per and xlsr. both models were trained on the tamil corpus, and the best results were submitted for the competition. the whisper model @xcite ) is a pre-trained automatic speech recognition (asr) model trained on 680,000 hours of multilingual and multitask supervised data sourced from the web. in our work, we have utilized vasista22/whisper-tamil-large-v2 2 , which is a fine-tuned version of openai/whisper-large-v2 3 on the tamil data available from multiple publicly available asr corpora. this transformer-based encoderdecoder model processes log-mel spectrograms through convolutional layers in the encoder and generates text autoregressively in the decoder. the model was further fine-tuned on a tamil corpus of the given training dataset, providing a robust baseline for tamil speech recognition. to adapt the 1.59-billion-parameter whisper model efficiently, we utilize low-rank adaptation (lora) @xcite and dynamic rank adaptation (dora) @xcite . these techniques freeze pre-trained weights and inject trainable low-rank matrices into specific transformer submodules, reducing computational overhead while preserving model performance @xcite . lora decomposes weight updates (∆w ) into two low-rank matrices a and b, where ∆@xmath1. for a weight matrix @xmath2, the adapted 2 https://huggingface.co/vasista22/whisper-tamil-large-v2 3 https://huggingface.co/openai/whisper-large-v2 weights become: we apply lora to the query, key, value, and output projection layers of each transformer block. to ensure stable training, weight scaling is used: training employed mixed-precision arithmetic (fp16) with the adamw optimizer (β 1 = 0.9, β 2 = 0.98, ϵ = 10 -6 ), a learning rate of 10 -5 with 50 warmup steps, and gradient accumulation over 2 steps. only 2.99% of parameters (47.5m out of 1.59b) were trainable through selective application of lora to the query, key, and value projection layers. during implementation, a comprehensive data preprocessing pipeline was constructed using whisperprocessor components, which extract audio features with a sampling rate of 16khz and prepare corresponding text transcriptions for supervised training. we have used a custom datacollatorspeechseq2seqwithpadding that effectively handles variable-length audio inputs and properly masks padding tokens in labels with -100 to be ignored during loss calculation. the combined use of 8-bit quantization, lora, and dora reduced memory requirements by 4 times compared to full-precision fine-tuning and achieved a 97% reduction in trainable parameters without significant accuracy degradation, demonstrating the efficacy of parameter-efficient methods @xcite for large-scale asr @xcite adaptation. on the other hand, we fine-tuned the pretrained anuragshas/wav2vec2-xlsr-53-tamil 4 checkpoint with the hugging face trainer api. the model is a wav2vec2forctc type model @xcite and was fine-tuned with full-scale finetuning, without layer freezing or modifications. connectionist temporal classification (ctc) loss was used during training and performance was tracked with word error rate (wer) and character error rate (cer). mixed precision training was activated with @xmath3, and the best model was chosen based on the minimum wer on the evaluation set. gradient accumulation with an accumulation step of 2 was used to stabilize training and mimic larger batch sizes. ## result and discussion submissions to the shared task on speech recognition for vulnerable individuals in tamil were evaluated using the word error rate (wer) between the asr hypotheses and the reference human transcriptions for the evaluation set @xcite . @xmath4: s is the number of substitutions, d is the number of deletions, i is the number of insertions, and n is the number of words in the reference transcriptions. during the fine-tuning phase, a close watch was kept on the wer and character error rate (cer) of both models, which were trained for the same number of epochs @xcite . we compared both the models' wer and cer. since the whisper-tamil-large-v2 model demonstrated significantly lower wer and cer than the wav2vec2-large-xlsr-53-tamil model, we selected it for generating transcriptions for the test dataset and submitted those results for final evaluation. we achieved a wer of 38.42 on the test dataset, which helped us secure the 4 th rank in the shared task. this performance demonstrates the robustness of parameter-efficient fine-tuning strategies for multilingual asr tasks on low-resource and demographically sensitive datasets @xcite . ## limitations despite its contributions, this work has several limitations. the dataset's limited size and dialectal diversity may hinder generalization, particularly for underrepresented tamil accents @xcite . computational constraints restricted the exploration of more complex architectures and large-scale training @xcite . evaluation primarily relied on wer, which may not fully reflect real-world intelligibility or user-centric performance, especially for vulnerable populations @xcite . the model's performance varied across regional pronunciations, suggesting a need for more balanced data. additionally, the absence of human-centered evaluations, such as user studies or error analysis on critical phrases, limits insights into practical usability @xcite . resource limitations also prevented extensive hyperparameter tuning and ablation studies. broader metrics, including semantic accuracy and user satisfaction, could better assess assistive utility @xcite . finally, ethical considerations, such as bias mitigation and inclusivity in data collection, were not thoroughly examined @xcite . addressing these gaps in future work could enhance robustness and fairness in tamil speech recognition. ## conclusion this work presents junlp's efficient approach to building an inclusive tamil automatic speech recognition (asr) system for vulnerable populations, including elderly and transgender speakers. using parameter-efficient fine-tuning (peft) methods low-rank adaptation (lora) and dynamic rank adaptation (dora), we adapted the multilingual whisper-large-v2 model for low-resource tamil speech with demographic variation. our model achieved a word error rate (wer) of 38.42% on the lt-edi-2025 evaluation set, securing 4th place. by freezing whisper's 1.59b pretrained weights and injecting low-rank matrices, we reduced trainable parameters by 97% (47.5m) and memory usage by 4 times, enabling fine-tuning on limited hardware. dora's decomposition improved expressiveness, and 8-bit quantization with mixed-precision training stabilized optimization. trained on 908 speech samples (5 hours) reflecting dialectal diversity, the model showed promise in inclusive asr. limitations include dataset size, regional bias, and reliance on wer. future directions include expanding diverse corpora and integrating user-centered evaluations. this study affirms peft-enhanced whisper models as viable for equitable asr in tamil.