Automatic Speech Recognition System for M alasar Language using Multilingual Transfer Learning (2023)

Abstract: This study pioneers the development of an automatic speech recognition (ASR) system for the Malasar language, an extremely low-resource ethnic language spoken by a tribal community in the Western Ghats of South India. Malasar is primarily an oral language which does not have a native script. Therefore, Malasar is often transcribed in Tamil script, a closely related major language. This work presents the first ever effort of leveraging the capabilities of multilingual transfer learning for recognising malasar speech. We fine-tune a pre-trained multilingual transformer model with Malasar speech data. In our endeavour to fine-tune this model using a Malasar speech corpus, we could successfully bring down the WER to 48.00% from 99.08% (zero shot baseline). This work demonstrates the efficacy of multilingual transfer learning in addressing the challenges of ASR for extremely low-resource languages, contributing to the preservation of their linguistic and cultural heritage.

## introduction this work introduces an automatic speech recognition (asr) system for the malasar language, spoken by the indigenous malasar community in the western ghats region of southern india 1 . the malasar language is at risk due to its limited number of speakers, with just 7,760 reported in the 2001 census of india @xcite . this asr system, developed with the goal of preserving the malasar culture and language, is based on fine tuning the whisper transformer model, which is pretrained on 680,000 hours of multilingual audio data, using transfer learning approach @xcite . the malasar language shares lexical similarities with multiple other languages, including tamil, 1 https://www.ethnologue.com/language/ymr/ malayalam, muduga, eravallan, and certain irula dialects @xcite . the different language influences in malasar will result in frequent code-switching. the malasar language is not welldocumented in written form, and there is severe scarcity of quality audio data for training asr models. the diversity in phonetic and linguistic characteristics along with the data scarcity contribute towards difficulty in building an asr system for malasar. the objectives of this work include addressing data scarcity by collecting and curating malasar speech data, developing an asr model capable of handling code-switching and multilingualism, and contributing to the preservation of the malasar linguistic heritage. this work represents a significant step towards safeguarding linguistic diversity in the face of endangered languages. ## literature review low-resource languages (lrls) are languages with limited resources, such as data, tools, and expertise. working with lrls presents a number of challenges, including data scarcity, tool scarcity, and expertise scarcity. these challenges were addressed by using techniques such as data augmentation, transfer learning, and domain adaptation @xcite . the introduction of transformer models in asr has indeed brought about a significant revolution in the field, especially for low-resource languages. asr performance has improved drastically in recent years, mainly enabled by self-supervised learning (ssl) based acoustic models such as wav2vec2 @xcite and large-scale multi-lingual transformer models like whisper @xcite . wav2vec2 is an encoder only transformer model. using self supervised learning, this neural speech representation model is trained with a lot of unlabelled data @xcite , in a process referred to as pre-training. for example the xsl-r model, that preceded the wav2vec2 model, was trained on 436k hours of publicly available unlabelled speech data from 128 languages @xcite . the pretrained model could be adapted with comparably less amount of labelled data for a specific task like asr, making it suitable for low-resource languages. the process of fine-tuning the pre-trained encoder model involves adding a linear classification layer on top of the transformer and training the entire model by minimising the connectionist temporal classification (ctc) loss function @xcite . whisper transformer model, on the other hand is a sequence-to-sequence model which consists of an encoder and a decoder linked via a crossattention mechanism @xcite . unlike wav2vec model, whisper is trained on labelled speech data that amounts to 680k hours, of which 117k hours is non-english speech data in 99 different languages. there has been efforts in the past to fine-tune both xls-r and whisper to make them transcribe seen and unseen languages, both high and low resource ones @xcite . attempts to fine-tune asr systems for dravidian languages from x-lsr models have shown effective improvements in accuracy @xcite @xcite . in this work, we aim to undertake a groundbreaking initiative by developing an asr system for the malasar language using whisper transformer models. ## malasar speech corpus the malasar language is primarily oral and does not possess a standardized script for written documen-tation. given the linguistic similarities between malasar and tamil, transcription efforts often rely on using the tamil script as a means of representing the spoken malasar language in a written form. a corpus of malasar speech to fine tune the transformer model was created from a spoken bible database foot_0 . the original datasource contained audio recordings of malasar speech, along with their transcriptions in tamil script. the audio recordings are a total of 5 hours in length. the speakers consist of one male and one female in the age group of 30-35. the details about the dataset is described in ## proposed malasar asr our proposed malasar language asr model is based on whisper transformer architecture @xcite . transformers offer a powerful architecture for asr due to their ability to model long-term dependencies, efficient parallelization, self-attention mechanism, multilingual capabilities, and contextual understanding @xcite . pretrained transformers offer added advantages for ultra-low resource languages as they can learn from a small corpora. these advantages have made transformers a popular choice for various natural language processing tasks. ## experimental setup five hours of malasar speech corpus and its corresponding tamil transcript is utilised for the experiment described in this work. 80% of the corpus is used for fine-tuning the pretrained model, while the remaining 20% was allocated for validation and testing purposes. the data partition used for fine tuning, validation and testing is listed in table . 2. the fine tuned model designed by optimizing several hyperparameters to achieve the highest performance possible such as training batch size, learning rate, warmup steps and evaluation batch size. we adopted a trial-and-error approach, where experimentation conducted with various values are assessed for their impact on the performance of the model. the best combination of hyperparameters based on wer is considered as the optimum parameter set for this problem. in ## result and discussion we evaluate the performance of the first ever malasar language asr on the held out test dataset. being a dravidian language, malasar has a morphologically complex word structure. it results in unreasonably high wer, even if a single space was introduced in the decoded text when compared to the ground truth transcript (anoop and ramakrishnan, 2023). a better indicator of performance hence would be character error rate (cer), where the error rates are calculated at character level according to eq. 2. the comparative analysis of wer and cer between the zero shot baseline (whisper small model) and the fine-tuned models illustrates significant improvements in the accuracy of the asr system. the baseline model exhibits high error rates in both wer (99.08%) and cer (34.74%), as indicated in table . 4. in contrast, the fine-tuned model yields notably reduced error rates with a wer of 48.00% and a cer of 13.46%. these results were additionally verified by linguistic experts at wycliff india. these improvements underscore the effectiveness of the fine-tuning process, enhancing the asr system's ability to accurately transcribe spoken language into written text, a valuable development in speech recognition tasks. ## conclusion this work presents the preliminary results of applying transfer learning on a transformer-based asr system tailored for the endangered malasar language. our experimental results show a substantial reduction in wer as well as cer on the fine-tuned model, in comparison with the zero shot baseline. the wer and cer on the unseen test dataset are respectively 48.00% and 13.46%. looking forward we plan to collaborate with the lingustic experts to expand the domain of training dataset and improve the transcription accuracy further. moreover, the asr system should be integrated into language learning and preservation efforts to maximise its impact on the malasar community.