CEN _ A mrita@ LT - EDI 2024: A Transformer based Speech Recognition System for Vulnerable Individuals in T amil (2024)

Abstract: Speech recognition is known to be a specialized application of speech processing. Automatic speech recognition (ASR) systems are designed to perform the speech-to-text task. Although ASR systems have been the subject of extensive research, they still encounter certain challenges when speech variations arise. The speaker’s age, gender, vulnerability, and other factors are the main causes of the variations in speech. In this work, we propose a fine-tuned speech recognition model for recognising the spoken words of vulnerable individuals in Tamil. This research utilizes a dataset sourced from the LT-EDI@EACL2024 shared task. We trained and tested pre-trained ASR models, including XLS-R and Whisper. The findings highlight that the fine-tuned Whisper ASR model surpasses the XLSR, achieving a word error rate (WER) of 24.452, signifying its superior performance in recognizing speech from diverse individuals.

## introduction speech is the most prevalent, clear, and frequently used form of worldwide communication. speech processing involves obtaining valuable information from voice signals, such as automatic speech recognition (asr) @xcite . the goal of any asr system is to teach computers to understand human speech and carry out user-defined tasks. the method of recognizing spoken words from audio input by applying auditory features is known as speech recognition. the majority of voice recognition algorithms have been trained on languages with abundant resources, like english, german, spanish, and so on. with languages with few resources, such as tamil, kannada, malayalam, etc., this is not the case. despite being one of the most investigated fields among researchers, speech recognition systems still face issues when it comes to the conventional learning paradigm @xcite , which can be utilized for both resourcerich and resource-poor languages. this is still one of the most unresolved challenges among the researchers @xcite . the fundamental reason for this is the various natures of speech, often known as speech variants. speech recognition systems face a challenge when it comes to recognizing variances in speech. these variations are caused by factors such as the speaker's age, gender, and vulnerability @xcite and etc. there have been a number of studies @xcite @xcite @xcite that have studied various methods to address these issues. these methods include the development of corpora and the fine-tuning of pre-trained models, particularly for languages that have limited resources. in response, the lt-edi team gathered a tagged tamil speech corpus from elderly and transgender vulnerable individuals who had everyday conversations in administrative offices, banks, and hospitals. some of these individuals were also vulnerable. therefore, the vulnerability of the speaker is the primary focus. we propose modifying the whisper automatic speech recognition (asr) model @xcite in order to improve speech recognition for those who are vulnerable. in preparation for the work that the lt-edi team is doing on voice recognition for the tamillanguage shared initiative, we fine-tuned the pre-trained whisper model by using tamil datasets. with a word error rate (wer) of 24.452, the 'cen_amrita' team was ranked top in the classification criteria. this achievement shows that the proposed strategy may address speech differences, especially in vulnerable populations. the following sections describe the paper's contribution: section 2 covers relevant works; section 3 materials and technique; section 4 results; and section 5 conclusion. ## related works advancements in deep learning have significantly impacted speech processing, notably in the domain of automatic speech recognition (asr). transformer-based architectures like @xcite @xcite , initially tailored for text interpretation, have been extended to capture speech sequences, leveraging contextual information to enhance accuracy. deepspeech @xcite , a flexible open-source program employing recurrent neural networks (rnns), stands out for its adaptability across various languages and effective training methods. self-supervised learning models such as wav2vec @xcite excel at speech pattern recognition by extracting pertinent features directly from unlabeled audio data. attention-based models, such as listen, attend, and spell (las) @xcite , change how much weight is given to inputs during decoding. this helps with accurate transcription after a lot of training on big datasets. lightweight architectures like quartznet (kriman et al., 2020) emphasize high performance while maintaining low computational demands. hybrid models, like espnet @xcite , combine convolutional and recurrent networks, showing that they are good at a number of different asr benchmarks. recent improvements, like hubert @xcite , build upon wav2vec by integrating hierarchical transformations and elevating representation learning and asr accuracy. these improvements have made significant advancements in the field of asr and have achieved impressive results. however, when it comes to resource-poor languages like tamil, existing models underperform due to the variability in speech among native speakers. to address this, recent research has focused on fine-tuning pre-trained asr models for specific languages. models like xlsr-wav2vec2 @xcite have been customized for tamil speech recognition, showcasing promising results with a significantly reduced word error rate (wer) of 39.65% @xcite and 37.71% @xcite . this customized approach aims to enhance performance in understanding and transcribing speech for languages with limited available resources. ## conclusion in this work, we utilized a pre-existing, pretrained model such as whisper and xls-r to improve the efficiency of an automatic speech recognition (asr) system for understanding conversational speech from elderly and transgender tamil speakers. this work is carried out as part of participation in the shared task of speech recognition for vulnerable individuals in tamil. on the tamil conversational speech corpus, the pre-trained models, such as whisper and the xls-r model, have been fine-tuned and compared to one another. the results of all the experiments indicate that the fine-tuned version of whisper asr models performs better than the xls-r model, which has a word error rate (wer) of 24.452.