End-to-End Automatic Speech Recognition for G ujarati (2020)

Abstract: We present a novel approach for improving the performance of an End-to-End speech recognition system for the Gujarati language. We follow a deep learning based approach which includes Convolutional Neural Network (CNN), Bi-directional Long Short Term Memory (BiLSTM) layers, Dense layers, and Connectionist Temporal Classification (CTC) as a loss function. In order to improve the performance of the system with the limited size of the dataset, we present a combined language model (WLM and CLM) based prefix decoding technique and Bidirectional Encoder Representations from Transformers (BERT) based post-processing technique. To gain key insights from our Automatic Speech Recognition (ASR) system, we proposed different analysis methods. These insights help to understand our ASR system based on a particular language (Gujarati) as well as can govern ASR systems’ to improve the performance for low resource languages. We have trained the model on the Microsoft Speech Corpus, and we observe a 5.11% decrease in Word Error Rate (WER) with respect to base-model WER.

## introduction asr is the process of deriving the transcription (word sequence) of an utterance, given the speech waveform. speech recognition has been an active area of research for many decades. initial work in asr was based on statistical modeling techniques like hidden markov model (hmm) @xcite and used phonemes to represent distinct sounds that make up the word. with the rise of deep learning based techniques and the increasing availability of data, the end-to-end speech recognition systems started showing competitive results. initial deep learning based asr models, based on recurrent neural network (rnn) and ctc @xcite , overcame the issues of statistical systems and provided the mapping of variable length input to output. with the further advancements in algorithms and resources, various complex deep learning architectures have been introduced for an effective end-to-end speech recognition system. end-to-end speech recognition for low resource languages has not gained significantly from the advancements in deep learning due to lack of training data compared to other high resource languages. linguistic diversities foot_0 also makes it difficult to adopt models across languages. in this paper, we present a speech recognition system for the gujarati language. gujarati is a rich language consisting of 34 consonants and 13 vowels. while the more number of vowels may reduce the homophones, more number of consonants may increase the ambiguity. the proposed system reduced the wer to 65.54% from the initial 70.65%. we analyzed the system using the testing hypothesis and derived many useful insights on the performance of the system as well as the cause of the errors in the hypothesis. we analyzed that the errors produced in the gujarati language are mainly because of interchanging/mismatching diacritics ('િ◌', '◌ી', etc.), consonants ('ધ', 'જ', 'ય', etc.), independents ('આ', 'અ', 'ઇ', 'ઈ', etc.) and some homophones. the remaining of the paper is organized as follows, section 2 describes the literature survey of asr system architectures. the proposed approach is described in section 3. section 4 constitutes the experiments conducted and its observations are followed in section 5. section 6 provides the conclusion of our work. ## related work since the first asr circuit developed by bell laboratories @xcite in the 1950s, asr has remained an active area of research. in early 1960's @xcite presented a phoneme based speech recognition which involved the first use of speech segmenter in different portions of the input utterances. @xcite introduced the concept of the nonuniform time scale for alignment of speech patterns (dynamic wrapping). both of these works lead to the viterbi algorithm @xcite which had been an indispensable technique in asr for decades. by the mid-1970s, the basic ideas of applying fundamental pattern recognition technology to speech recognition, based on linear predictive coding (lpc) (atal and hanauer, 1971) methods, were proposed by itakura @xcite ). cmu's harpy system @xcite , was the first ever system to use the finite state network (fsn) to reduce computation for matching in speech recognition. however, methods which optimized the resulting fsn did not come about until the early 1990's @xcite , which were limited to small to medium vocabulary electronic based solutions for asr. the earlier approaches of electronics based asr were eventually replaced by statistical approaches with the introduction of hmm based speech recognition. the basic implementation of hmm based speech recognition model was first published in 1975 by baker @xcite at cmu. further work on hmm continued with the introduction of first ever use of hmm for continuous speech recognition in 1976 @xcite . as the research continued, the hmm model was tried with various machine learning techniques including the hmm/ann architecture in 1990 @xcite , hmm/gmm architecture in 1997 @xcite and hmm/svm architecture in 1998 @xcite . this wave for hmm continued till the introduction of rnn based approaches in early 2005. the above approaches had major drawbacks such as • it requires high task-specific knowledge, e.g. to design the state models for hmms. as we saw end-to-end asr gives great results but at the cost of large data required to train it, which is not feasible for low resource languages. according to interspeech 2018, low resource automatic speech recognition challenge foot_1 , tdnn-based systems (peddinti et al., 2015) are efficient in modelling long temporal context and performed well even in the low-resource setting @xcite . even with the smaller amount of data, with some enhancements, end-to-end systems showed promising results @xcite . the work similar to our approach are presented in speech recognition primer foot_2 , and gujarati automatic speech recognition foot_3 . the first approach is based on a combination of cnn @xcite and bil-stm @xcite the latter approach uses a combination of 3 gated recurrent units (grus). the first approach is designed for english, while the second is for gujarati. bert is a neural network-based technique for natural language processing pre-training. the pre-trained bert model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications @xcite . multilingual-bert uses a representation that is able to incorporate information from multiple languages @xcite . ## proposed approach figure 1 describes the end-to-end speech recognition system proposed in the paper. this section describes the processing involved in the various stages. ## conclusion in this paper, we have presented an end-to-end speech recognition system for gujarati. we propose a prefix decoding technique that uses two language models to improve the performance of the system. we have also used a bert based spelling corrector model in a post-processing step to further improve the performance. we observe that the proposed approach reduces the overall wer by 5.11%. while deep learning models require a lot of training data for better results, in this paper we showed that without increasing the training data we can improve the performance. this is particularly important for a resourceconstrained language like gujarati. we are optimistic that with an increase in data our optimizations would perform even better. we explored and analyzed the inferences from our asr system to gain key insights which consist of checking the correctness of the word error due to consonants, diacritics, independents, half-conjugates, and homophones. these insights can help to understand our asr system based on a particular language (gujarati) as well as can govern asr systems' to improve the performance for low resource languages.