Improve S inhala Speech Recognition Through e2e LF - MMI Model (2021)

Abstract: Automatic speech recognition (ASR) has experienced several paradigm shifts over the years from template-based approaches and statistical modeling to the popular GMM-HMM approach and then to deep learning hybrid model DNN-HMM. The latest shift is to end-to-end (e2e) DNN architecture. We present a study to build an e2e ASR system using state-of-the-art deep learning models to verify the applicability of e2e ASR models for the highly inflected and yet low-resource Sinhala language. We experimented on e2e Lattice-Free Maximum Mutual Information (e2e LF-MMI) model with the baseline statistical models with 40 hours of training data to evaluate. We used the same corpus for creating language models and lexicon in our previous study, which resulted in the best accuracy for the Sinhala language. We were able to achieve a Word-error-rate (WER) of 28.55% for Sinhala, only slightly worse than the existing best hybrid model. Our model, however, is more context-independent and faster for Sinhala speech recognition and so more suitable for general purpose speech-to-text translation.

## introduction there are two main architectures in training the asr system. they are statistical asr architecture and end-to-end(e2e) deep neural architecture. statistical asr was the state of the art for many years, however after the year 2015, researchers tend to move towards e2e asr systems due to the higher results. the main difference between these two architectures is the number of models needed to create, in training the asr system. in statistical asr, it needs 3 types of models and they are acoustic models, pronunciation models and the language model. but in e2e asr, it will compress those 3 models into a single deep-neural-network (dnn) @xcite . so many research have been done using e2e architecture for creating asr systems for different languages since this is a new trending area of natural-language-processing (nlp) and speech recognition. and there are previous researches conducted for english speech recognition that show better results when using e2e architecture than the traditional statistical approach @xcite . currently, there are previous and ongoing researches on building asr systems for sinhala language using statistical asr architecture, gaussian mixture model with hidden markov model (gmm-hmm) based models and hybrid deep-neural-network with hidden markov model (dnn-hmm) based models @xcite . e2e architecture would be a new approach for sinhala asr and it will help to improve the available resources as well. especially, e2e architecture opens the doorway to transfer learning, which is the new trending for low resource speech recognition, to improve the accuracy. these domains for speech recognition were popular in later 2019 among the researchers and it is essential to have models trained in e2e architecture in sinhala language to get a generic idea when accessing these domains @xcite . in this paper, we present a study on e2e dnn architecture based asr systems for sinhala speech recognition using e2e lf-mmi model. the performances of the e2e model will be evaluated and compared with the statistical models such as gmm-hmm, dnn-hmm and combinational models of sgmm-dnn. the paper is organized as follows. section 2 presents the related studies, section 3 describes the methodology, data preparation and implementation in greater detail. section 4 describes the results and evaluation. section 5 presents the conclusions and the future work. ## literature review more than 30 years later also this methodology still predominates in asr. nowadays, most practical speech recognition systems are based on the statistical approach @xcite . with the improvement of deep learning, dnn is introduced for creating acoustic models. the role of dnn is to calculate the posterior probability of the hmm state, by replacing the traditional gmm observation probability. so dnn-hmm models become the state-of-the-art asr model by achieving better results than gmm-hmm models @xcite . the training process and decoding process of the hmm-based model determines whether it faces the following difficulties in actual use. due to the above-mentioned shortcomings or anomalies in the hmm-based models, more research was carried out in the e2e architecture with the trending of deep learning. the end-toend model is a system that directly maps input audio sequence to sequence of words or other graphemes @xcite . so direct mapping of utterances to character sequence is conducted where no intermediate states like calculating posteriors in the output @xcite . data alignment is the major problem in both hmm based and e2e models but e2e models require soft alignments where hmm based models use forced alignments. however, main problem in e2e architecture is that it requires a large amount of speech data to achieve higher accuracy in recognition @xcite . till year 2018 low resource speech recognition systems are never used e2e architecture because this architecture is used in large vocabulary continuous speech recognitio (lvcsr). @xcite paper shows that, abundance of training data make the system lagged comparable to hybrid dnn systems when trained on smaller training sets. interspeech 2018 low resource automatic speech recognition challenge for indian languages research conducted by microsoft corporation in india created a challenge by giving 50 hours of transcribed speech in each tamil, telugu and gujarati which are three main indian languages and asked participants to build asr systems restricted to this dataset. evaluation carried out on a blind test set @xcite . the isi-billa system presented to address the above challenge @xcite was an eesen based end-to-end multilingual lstm network trained using the connectionist temporal classification (ctc) training criterion . this is the first time to use an e2e model for south asian languages. both monolingual and multilingual systems trained using this model and it outperformed the baseline models in all three languages @xcite . after year 2019, transfer learning and unsupervised learning techniques have become famous in the speech recognition domain. one of the major improvements was to tackle the low resource problems researchers are introducing in these domains @xcite . in the investigation of transfer learning for asr using lf-mmi trained neural networks paper they used weight transfer and multitask learning transfer learning techniques to address the low resource problem @xcite . meta learning for endto-end low-resource speech recognition paper shows that using multilingual ctc models can be used to improve the accuracy of the asr using meta learning techniques @xcite . in wav2vec: unsupervised pretraining for speech recognition paper used a model is trained using the auto segmentation criterion. untranscribed web audio for low resource speech recognition paper has introduced semi-supervised training is done by using lattice-free maximum mutual information (lf-mmi) to untranscribed data @xcite . ## results and evaluation the server used for training of all deep neural architectures and the decoding of the models contains 4 gpus -geforce rtx 2080 ti of 10.8gb each. all 4 gpus have been used when training, thereby accelerating the deep learning training process by leveraging cuda. the performance of e2e sinhala asr systems are evaluated in terms of accuracy on the recordings taken with the noise environment. this accuracy can be obtained by calculating either word error rate (wer) or sentence error rate (ser). wer is the number of words that are wrongly identified out of the total number of words in the audio sample used for recognition. ser is the number of sentences that are improperly identified out of the total number of sentences @xcite .the wer is widely used to discrete and compare speech recognition systems and we also used the wer throughout the study. ## conclusion and future work even though there is a slightly better accuracy in statistical approach, using the e2e approach we created a less context dependent and faster model for sinhala speech recognition for using general purpose speech-to-text transcription. we found out that using only statistical models (gmm+hmm, sgmm, sgmm+mmi) is not useful in further research conducted for sinhala speech recognition that even hybrid system where dnn uses to calculate the posterior probabilities for the hmm perform far better than those traditional approaches. currently, we were able to achieve 28.55% wer for sinhala e2e speech recognition using e2e lf-mmi implemented on kaldi toolkit. this model also can be improved with fine tuning but this study is not going to fine tune and has used the basic implementations and recipes available for the wsj dataset which have 80 hours of training data. current domain of speech recognition moves toward addressing the low resource problem. there are large datasets available for english and france like languages with the state-of-the-art results. common solution for addressing the low resource problem is to transfer learning from high resource language to a low resource lan-guage. in e2e lf-mmi technique transfer learning can be done by using weight transfer and multi-task training @xcite . so from the results of this study it will be useful to do the necessary data augmentations and choosing parameters for transfer learning techniques mentioned above.