K onkani ASR (2023)

Abstract: Konkani is a resource-scarce language, mainly spoken on the west coast of India. The lack of resources directly impacts the development of language technology tools and services. Therefore, the development of digital resources is required to aid in the improvement of this situation. This paper describes the work on the Automatic Speech Recognition (ASR) System for Konkani language. We have created the ASR by fine-tuning the whisper-small ASR model with 100 hours of Konkani speech corpus data. The baseline model showed a word error rate (WER) of 17, which serves as evidence for the efficacy of the fine-tuning procedure in establishing ASR accuracy for Konkani language.

## introduction konkani belongs to the southern indo-aryan language group spoken on the western coast of india. it is the official language of goa and one of india's 22 officially recognized languages. some people also speak konkani in the coastal regions of @xcite it has approximately 2.3 million speakers @xcite . devanagari is the official script for writing the language, and the antruzi dialect spoken in ponda taluk is considered the standard dialect for official communication. language is resource-scarce and lacks a lot of digital resources and tools. in this digital era, there is a need for digital resources to be made available in all local and regional languages so that their identity is maintained. it may also help in preserving language diversities and identities. some digital solutions require common fundamental components. the automatic speech recognition (asr) system can be considered one of the most important digital resources for any language to survive in this digital era. speech is the most natural form of communication. if asr is developed for all the native languages, it has the potential to break the barriers between technology and people. asr has the power to make technology available to all. some immediate beneficiaries can be a person with a vision disability, an illiterate person, or a person with reading problems. asr solutions are well explored, and we can see many commercial solutions for resource-rich languages like english. ironically, it is also true that asr is not available for lowresourced languages. the main reason could be a lack of language resources like large annotated speech corpora, text corpora, etc. here, we have attempted to develop a quick asr for konkani language by banking on the resources of resource-rich languages. the paper is organized as follows: a brief introduction to konkani language is provided in section 1 above. a quick literature review of the asr development is provided in section 2. the motivation behind the current work is provided in section 3. properties of the dataset used in the development of asr are provided in section 4. the methodology used for the design of the asr is provided in section 5. results and discussions based on the asr development are provided in section 6. finally, the conclusion and future work are discussed in section 7. ## literature review asr systems have evolved from simple limited command and control word setups to today's fluent natural language processing systems @xcite and fluent end-to-end systems @xcite . initial work on asr has employed statistical modeling. the most famous was the gmm-hmm-based model @xcite @xcite . this model re-quires vast amounts of data and takes a big team to develop the model. later, ml models started replacing these statistical models. still, this model takes a large amount of annotated corpus, and a good acoustic and language model. acoustic model creation is similar to feature extraction, like gmm models. nowadays, the large pre-trained model can be used to develop asr solutions for low-resourced languages. this pre-trained model can be later fine-tuned to get asr in target languages @xcite . this method, however, required annotated speech data for fine-tuning. if data for fine-tuning is not present, then it needs to be generated for a specific language. ## motivation of the work every language holds and represents its own unique linguistic and cultural heritage. konkani is a resource-scarce language that deserves complete preservation in this digital age. one logical step towards conservation would be to build the linguistic tools required for survival in this digital age. here, we have attempted to develop one such tool. through this work, we have developed an asr system using tools and resources from resource-rich languages and available konkani data. ## dataset properties we have used ciil konkani annotated speech corpus @xcite . data consists of more than 100 hours of annotated speech data. the dataset consists of ten subcategories: ## methodology data pre-processing: the ciil speech data is already transcribed and annotated with corresponding text labels for a total of 72937 records. resampling was conducted on all audio files, reducing the sampling rate from 48khz to 16 khz. all phrases that exceeded 448-byte pairs were removed through the process of filtering, as the model supports a maximum of 448-byte pairs for the text input. the data was split into a train-evaluation-test split in 80/10/10 ratio. whisper-small @xcite , a pretrained asr model, is selected as the base model for fine-tuning. the model has 12 layers with 12 heads, resulting in 244 million parameters. the model is pre-trained on a vast quantity of labeled audio-transcription multilingual datasets, which include data from various languages, enabling it to learn general speech representations. konkani language was not included in the set of languages where the whisper-small model was initially pretrained. the whisper tokenizer received pre-training using transcriptions from 96 languages that were used for pre-training. as a result, it possesses a comprehensive byte pair suitable for nearly all multilingual asr applications. the fine-tuning process of the whisper model involves loading the tokenizer specific to the language and utilizing it for fine-tuning without making any additional adjustments. due to the absence of a konkani tokenizer, the marathi tokenizer is used. the system design can be referred to in figure 1 . we fine-tuned the whisper-small model using the ciil speech dataset. during fine-tuning, the model's weights are adjusted by utilizing konkanispeech and text pairs while preserving the previously acquired representations from the multilingual pre-training stage. transfer learning @xcite enables the model to effectively adjust to the distinctive phonetic and linguistic attributes inherent in konkani language. the system was trained for three epochs, with an early stopping criterion for evaluation loss, using a v100 16 gb gpu. hyperparameter tuning was performed using the validation set. the average training time for the system was around 5-6 days. evaluation metric: the evaluation metric employed to assess the performance of the asr sys-tem on the konkani test set is the word error rate (wer). the wer is a metric used to determine the percentage of errors at the word level in the asr output when compared to the ground-truth transcription. this value indicates the average number of errors per reference word. the lower the value, the better the performance of the asr system, with a wer of 0 being a perfect score. ## results and discussion the fine-tuned whisper-small asr model is evaluated on the test set. the obtained wer of 17 serves as evidence for the efficacy of the finetuning procedure in enhancing asr performance for konkani language. additionally, we conducted experiments using the hindi tokenizer. however, our findings indicated that utilizing the marathi tokenizer yielded better scores than hindi. to check the model's performance in a real-world case, we conducted manual testing using 70 newly curated sentences. two native speakers were tasked with recording and noting down the output of the model. the number of space-separated tokens in sentence collection ranged from 3 to 14. following are some examples from the dataset used to test the final model. the words 'ना खं य' are merged into a single word, 'नाकां य' and 'नाका'. during manual testing of the asr, we observed that errors are more likely to occur in the output if there is background noise. this can be improved by pre-processing voice signals for noise removal, and it will be considered in our future work. we also observed that asr does not recognize some words. this could be because of the marathi tokenizer used in training the model. this was done because the unavailability of the konkani tokenizer and other tokenizers, like the hindi tokenizer, was not helping in the improvement of asr performance. the creation of the konkani tokenizer will be attempted in our future work. ## conclusion and future work through this work, we have created an asr system for konkani language spoken on the western coast of india. it has been demonstrated that a pre-trained model can be successfully used to build asr by fine-tuning the model with annotated speech data in a less-resourced target language. asr showed a baseline wer of 17 on the test data. in the future, we want to use better modeling architectures to improve the asr model.