C rew X @ LT - EDI -2025: Transformer-Based T amil ASR Fine-Tuning with AVMD Denoising and GRU - VAD for Enhanced Transcription Accuracy (2025)

Abstract: This research presents an improved Tamil Automatic Speech Recognition (ASR) system designed to enhance accessibility for elderly and transgender populations by addressing unique language challenges. We address the challenges of Tamil ASR—including limited high-quality curated datasets, unique phonetic characteristics, and word-merging tendencies—through a comprehensive pipeline. Our methodology integrates Adaptive Variational Mode Decomposition (AVMD) for selective noise reduction based on signal characteristics, Silero Voice Activity Detection (VAD) with GRU architecture to eliminate non-speech segments, and fine-tuning of OpenAI’s Whisper model optimized for Tamil transcription. The system employs beam search decoding during inference to further improve accuracy. Our approach achieved state-of-the-art performance with a Word Error Rate (WER) of 31.9,winning first place in the LT-EDI 2025 shared task.

## introduction communication has been the cornerstone of human evolution, enabling individuals to share thoughts, emotions, and information. human communication evolved naturally through verbal expression, with speech emerging as one of our earliest and most intuitive forms of interaction. with recent advancements in technology, a more natural way for human-computer interaction is necessary, which is satisfied with the help of speech processing techniques. this has led to the invention of automatic speech recognition (asr) @xcite systems. the goal of an asr system is to convert spoken language into text and it plays a crucial role in various applications such as virtual assistants, transcription services, and accessibility tools. despite advancements in technology, asr still remains a complex task due to several challenges. the high variability in speech data makes it difficult for the models to generalize across different speakers. additionally, while asr systems for languages like english, spanish and mandarin benefit from extensive datasets and pretrained models, dravidian languages like tamil and malayalam suffer from the scarcity of extensive & good quality annotated datasets, making it hard to build robust asr systems. tamil, in particular, has unique phonetic characteristics and word-merging tendencies that further complicate transcription accuracy @xcite . this work addresses these challenges by finetuning a transformer-based asr system for tamil speech, specifically for vulnerable old-aged and transgender individuals, who often face difficulties in accessing essential services due to lack of literacy and familiarity with technology. to enhance transcription accuracy, our method integrates adaptive variational mode decomposition (avmd) @xcite for noise reduction and silero voice activity detection (vad) @xcite to eliminate non-speech segments. we then, finetune openai's whisper @xcite model for automatic speech recognition, leading to promising results. this approach achieved stateof-the-art (sota) performance, ranking first in the shared task with a word error rate (wer) of 31.9. this demonstrates the effectiveness of this pipeline in handling the complexities of tamil speech for vulnerable populations @xcite . ## related work early asr systems relied on rule-based phonetic models and statistical methods like hidden markov models (hmms) @xcite , gaussian mix-ture models (gmms) @xcite and dynamic time warping (dtw) @xcite . however, these methods struggled with high variability in speech, such as speaker, accents, age, gender, background noise, and spontaneous speech patterns. this led to the emergence of traditional deep learning based models such as recurrent neural networks (rnns) @xcite and long short-term memory (lstm) @xcite networks. @xcite used deep neural network and hmm based hybrid model which further improved the quality of the transcriptions generated. but even they struggled with capturing long range dependencies. the major breakthrough in processing temporal was made with the invention of the transformer architecture @xcite in 2017. it introduced the multi head self attention (mhsa) mechanism which excelled in capturing both long term and short term dependencies. hence, the transformer-based models significantly improved transcription accuracy by learning complex patterns from large datasets. self supervised models like wav2vec 2.0 @xcite reduced reliance on labeled data by learning speech representations from raw audio, significantly improving asr for low-resource languages. openai's whisper further advanced asr with large-scale weak supervision, enabling robust transcription, translation, and language identification across diverse datasets @xcite . ## dataset elderly individuals frequently visit essential service locations such as banks, hospitals, and administrative offices but struggle with technological tools designed to assist them. similarly, transgender individuals often face educational barriers due to societal prejudices, making speech a primary mode of communication for accessing essential services. hence, the dataset provided for this shared task specifically targets vulnerable elderly and transgender individuals @xcite . the dataset consists of 7.5 hours of spontaneous tamil speech, which is split into 5.5 hours of transcribed speech for training and 2 hours of unlabeled speech for testing. the train data provided was further made into a 80-20 split for training and development. the dataset distribution is provided in ## methodology the speech signal was initially denoised using adaptive variational mode decomposition (avmd), which decomposes it into variational modes and reconstructs the relevant components to remove noise while preserving speech clarity. next, silero vad (gru based) was applied to eliminate silence and non-speech segments, reducing computational load due to unnecessary processing and at the same time improving transcription accuracy.the processed audio is then passed through the whisper processor, which converts it into log-mel spectrogram @xcite features using a standardized pipeline. finally, the extracted features are passed to the whisper-tamil-medium model for generating transcriptions, with beam search decoding (lowerre, 1976) during testing to enhance the accuracy and reduce the wer. the overall workflow is depicted in figure 1 . ## experimentation for training, the model was optimized using adamw @xcite with a learning rate of 10 -5 and weight decay of 10 -2 to prevent overfitting. the loss function used was crossen-tropyloss, which measures the difference between predicted and actual token distributions. we employed reducelronplateau to automatically decrease the learning rate by half whenever validation loss stopped improving for two consecutive epochs, which helped maintain smooth training progress. the models were trained on a 4080 super gpu with 4 as the batch size. during testing, beam search decoding was employed, thus increasing transcription accuracy. the trends observed during the training and validation process are given in figure 2 . 6 result and analysis ## conclusion in this paper, we presented a robust asr pipeline tailored for noisy audio conditions. in conclusion, our methodology showcases the potential of combining noise-aware preprocessing and advanced decoding strategies to deliver accurate and reliable transcription in real-world, low-resource scenarios. this also provides a strong foundation for future enhancements in speech recognition under challenging conditions. the entire implementation can be found here: https://github.com/ ganesh2609/vulnerablespeechasr