SSNCSE _ NLP @ LT - EDI - ACL 2022: Speech Recognition for Vulnerable Individuals in T amil using pre-trained XLSR models (2022)

Abstract: Automatic speech recognition is a tool used to transform human speech into a written form. It is used in a variety of avenues, such as in voice commands, customer, service and more. It has emerged as an essential tool in the digitisation of daily life. It has been known to be of vital importance in making the lives of elderly and disabled people much easier. In this paper we describe an automatic speech recognition model, determined by using three pre-trained models, fine-tuned from the Facebook XLSR Wav2Vec2 model, which was trained using the Common Voice Dataset. The best model for speech recognition in Tamil is determined by finding the word error rate of the data. This work explains the submission made by SSNCSE_NLP in the shared task organized by LT-EDI at ACL 2022. A word error rate of 39.4512 is achieved.

## introduction speech recognition (also known as speech-totext or automatic speech recognition) is a technique used to convert human speech into a written format. it is an important tool, and has many applications, such as in mobile phones (voice commands for call routing and voice searching), customer service, emotion recognition, and more importantly, in helping disabled people. it can not only help convert words to text to assist hearing impaired people, but also aids physically impaired people in performing activities such as typing and browsing using voice commands, instead of having to manually operate a computer. tamil is the official language of tamil nadu, puducherry, sri lanka and singapore. @xcite was the first language to be classified as a classical language in india, out of over 22 scheduled languages in india. it is also one of the oldest languages in the world, seemingly originating over 2000 years ago. the speech recognition is achieved by considering the linguistic features of the tamil language. the natural language processing approach is used for the speech recognition task. the team of ssncse_nlp has participated in the speech recognition for vulnerable individuals in tamil shared task, obtaining the first position with a word error rate of 39.4512. in our paper, we have used pre-trained models designed for the tamil language, to transcript the speech audios into tokens, eventually decoding them into text. we have made use of three pre-trained models, namely amrrs/wav2vec2-large-xlsr-53-tamil foot_0 , akashsivanandan/wav2vec2-large-xls-r-300mtamil-colab-final foot_1 and nikhil6041/wav2vec2large-xlsr-tamil-commonvoice foot_2 . the issue with automatic speech recognition in older adults as well as physically or mentally impaired people is that they tend to display mild dysarthric speech, or slurred speech, causing erroneous transcription of the data. furthermore, in tamil speaking places, people from different regions speak in non-identical dialects, accents, and speeds, and hence, the transcription of the data differs from person to person. when trained with audios from a single region, there is incapacity to accurately predict what a person from a different region is saying. the rest of this paper is arranged as follows. section 2 discusses the related work on speech recognition tasks. the dataset about the shared task is described in section 3. section 4 outlines the features and machine learning algorithms used for this task. results are presented in section 5. section 6 concludes the paper. ## related work researchers have experimented with a few approaches to deal with speech recognition in minority languages such as tamil recently. the writers of voice and speech recognition in tamil language @xcite ,propose the use of a hidden markov model(or hmm). it is a statistical pattern matching approach which can generate speech using a number of states for each model. since the hmm model uses positive data, it scales well and it reduces the time and complexity of the recognition process. in design and development of a large vocabulary, continuous speech recognition system for tamil, the authors (madhavaraj and ramakrishnan, 2017) build two independent recognition systems for phone recognition (pr) and for continuous speech recognition (csr) using deep neural networks (dnn). the dnn based triphone acoustic model is proven to yield significantly better results in csr and pr. the authors of speech rate control for improving elderly speech recognition of smart devices @xcite , take the help of a convolutional neural network (cnn) to generate feature vectors to be fed into a fully connected network (fc) for frame by frame syllable transition boundary classification. thus the syllable transition probability is calculated and the syllables are segmented. they take the help of a synchronized overlap-add (sola) algorithm to adjust the speech rate according to the measured ratio on a time-scale. in transformer-transducer: end-to-end speech recognition with self-attention @xcite , the authors attempt to build a model for end-to-end speech recognition using transformer networks in neural transducer. they propose two methods, namely using vggnet with causal convolution to incorporate positional information and reduce frame rate for efficient inference and using truncated self-attention to enable streaming for transformer and reduce compu- ## dataset analysis and preprocessing the data set given by the shared task organizers consists of a training set and a testing set, each consisting of 909 and 239 instances respectively @xcite . the training set contains the audio files and transcriptions of the audios in the tamil language, whereas the testing set contains only the audio files. the audios in both the training set and the testing set contain audio recordings, each having an average running time of 20 seconds. ## experimental setup and features for feature extraction, the n-gram model is experimented upon. the three pre-trained models are each used to extract the features. all three pre-trained models are fine-tuned versions of the facebook xlsr wav2vec2 model, trained using the common voice dataset containing 9283 hours of audios of different languages. these models have been trained using the tamil speech corpus in the same. the pre-trained xlsr model maps the speech signal to a sequence of context representations. for the model to map the latter to its corresponding transcriptions, a linear layer used to classify each context representation to a token class has to be added on top of the transformer block. the output size of this layer corresponds to the number of tokens in the vocabulary, which does not depend on xlsr's pre-training task, but only on the labeled dataset used for fine-tuning. the training data is run and the transcriptions are obtained. punctuation marks and other characters without meaning are removed from the transcriptions and all distinct letters of the training data are used to build the vocabulary (an enumerated dictionary). the each training audio file is loaded as a floating point time series at a sampling rate of 16000 samples per second. a data collator is defined to pad the training data to the longest sample in the batch. the pre-trained checkpoint of wav2vec2-xlsr is loaded and all parameters related to training are defined. the model is then trained and the word error rate is found. the three models are trained in the above manner and used to generate input values using tokenizer and the logits are found out using the model. the tokens for the logits are predicted and decoded to find the transcriptions of the audio. ## observations at a first glance at the transcriptions, it can be seen that the amrrs model is unable to not only differentiate between when a particular word ends and another begins, but is also unable to apply the stressed consonants in many places, such as in the words எù≈ள× and ஒùொ≈ாõ . this can be justified by the stress being applied for uncertain periods of time in different words. it is also ineffective at identifying vowels before stressed consonants which are mainly used only as stressed consonants and not simply as consonants, such as ó and í, because their sounds are almost always preceded by a vowel, hence making it indiscernible. looking at the transcriptions generated by the akashsivanandan model, it can be said that it is unable to distinguish between consonants of the same pronunciation sets, such as the harshly pronounced letters, feebly pronounced letters and the medially pronounced letters. this can be seen in words such as ãýú which are transcripted as ãñî due to the stressed consonant ý being misinterpreted, subsequently leading to the next letter to also be misunderstood. the occasional english word in the audios is pronounced differently compared to its tamil transliteration, causing it to be wrongly interpreted. the nikhil6041 model is found to produce the most accurate transcriptions out of all three models tested. it occasionally mislabels similar consonants(such as ள and ∆) and vowels (such as அ and எ) and is sometimes unable to mark the vacant spaces between two words, but for the most part, it generates transcriptions which are in the vicinity of how the words are actually pronounced in the audio. however, it does not always correspond to the actual transcription of the audio as the pronunciations differ when they are spoken or written. ## conclusion the need for automatic speech recognition for vulnerable individuals is growing to be increasingly important every day. more and more of people's daily lives are made easier by technology, regardless of whether they have disabilities or not. speech recognition technology, though popular and well refined for prominent western languages such as english, is not available easily to minorities who do not speak that language. our motive has been to make automatic speech recognition software more easily accessible to the dravidian population, more importantly, the tamil speaking population. in this paper we propose to use pretrained speech recognition models created for the tamil language and use it to transcribe the testing audios provided by the organizers. it is noted that the nikhil6041 model yields the best results out of all the three used models. the above model can be finetuned further by obtaining and using a more extensive dataset, and training the model against a more sizable range of accents and dialects. this will lead to an overall more accurate transcription of the audios.