An End-to-End Speech Recognition for the N epali Language (2021)

Abstract: In this era of AI and Deep Learning, Speech Recognition has achieved fairly good levels of accuracy and is bound to change the way humans interact with computers, which happens mostly through texts today. Most of the speech recognition systems for the Nepali language to date use conventional approaches which involve separately trained acoustic, pronunciation and language model components. Creating a pronunciation lexicon from scratch and defining phoneme sets for the language requires expert knowledge, and at the same time is time-consuming. In this work, we present an End-to-End ASR approach, which uses a joint CTC- attention-based encoder-decoder and a Recurrent Neural Network based language modeling which eliminates the need of creating a pronunciation lexicon from scratch. ESPnet toolkit which uses Kaldi Style of data preparation is the framework used for this work. The speech and transcription data used for this research is freely available on the Open Speech and Language Resources (OpenSLR). We use about 159k transcribed speech data to train the speech recognition model which currently recognizes speech input with the CER of 10.3%.

## introduction there has been increasing use of automatic speech recognition (asr) technologies in many application domains like hospital information systems to iot devices, industrial robotics, forensic, defense and aviation, etc. to name a few. according to @xcite , a conventional speech recognition system consists of several modules that comprise the acoustic, lexical, and language models supported by a probabilistic model for noisy channels. to build an acoustic model, we first need to build a hidden markov model (hmm) and a gaussian mixture model (gmm). in addition, the system requires linguistic knowledge based on a lexical model, usually based on a handmade pronunciation dictionary that does not have an explicit word limit. the lexical model requires language-specific tokenization modules for language modeling to develop asr for new languages. finally, decoding must be done with the synergistic action of all the modules, resulting in a complex decoding process @xcite . nevertheless, today's systems rely heavily on the end-to-end architectures that have emerged around traditional techniques @xcite . unlike in the traditional methods like hidden markov model (hmm) based model, the end-to-end approach addresses a single network architecture within a deep learning framework that directly maps language features to words or characters @xcite . there are two main architectures for end-to-end asr. connectionist temporal classification (ctc) allows the training of acoustic models without frame-level alignment between transcripts and acoustic frames, while attention models perform alignment between acoustic frames and identifiers @xcite . @xcite present a ctc/attention-based collaborative end-to-end asr that uses the ctc objective from loss function during the attention model training. the joint ctc-attention-based encoder-decoder utilizes both the benefits of ctc and attention during training. the ctc predictions are also used in the decoding process. ctc can be interpreted as just a loss function used for training neural networks such as cross-entropy models. it is used in a difficult situation where the availability of aligned data is an issue, like in asr. the capability to model temporal correlations with appropriate context information can be found in convolutional neural network (cnn) @xcite . also, the language model (lm) integration is widely used for the hmm-based systems, something that is still applicable and effective for end-to-end asr as well @xcite . the attention-based approach is used with ctc and recurrent neural network-based language model (rnn-lm) as a joint decoder and a cnn-based shared encoder to achieve a state-of-the-art accuracy @xcite . most of the works on speech recognition for nepali are based on the traditional methods. nepali is written in the devanagari script, which is essentially phonetic, so its pronunciation is very similar to how it is written. in nepali, there are a total of 11 vowels and 33 consonants @xcite . we use an end-to-end speech recognition architecture that is based on a joint ctc/attention-based encoder-decoder with a recurrent neural network-based language modeling developed by @xcite . we use the espnet framework @xcite @xcite , which is an end-to-end speech processing toolkit. it sits on top of the kaldi speech recognition toolkit @xcite and the deep learning frameworks based on pytorch @xcite . this paper is divided into five sections. section i gives an introduction about the problem, section ii gives an overview of the related works in nepali speech recognition, section iii describes the methodology for developing rnn-lm and ctc/attention-based models, section iv discusses the model and presents the experimental results, and finally section v presents the conclusion and future works. ## related works some prior works on nepali asr in the character, word, and sentence level have been conducted. google, for example, provides cloud-based speech recognition for more than 80 languages including nepali. unfortunately, there is not any publicly available documentation on the underlying methods and techniques used for the project. @xcite analyzed existing models for speech recognition and upon finding the shortcomings of dynamic time wrapping (dtw's) approach, they proposed a new model called the ear model. they report to have obtained better accuracy than existing methods, but only for single alphabets. the classical, most commonly used model for speech recognition is hidden markov's model which is used by @xcite where the authors have obtained a fairly good accuracy of 75% for isolated words. similarly,, @xcite used the rnn-ctc model and obtained a cer of nearly about 34% accuracy making use of the language model. @xcite proposed a model comprising cnn, gru, and ctc networks. the dataset used by the authors is provided by open speech and language resources. their build model recognizes speech with the wer of 11%. @xcite present a comparative study of popular speech recognition methods for the nepali language where they built a phonetic dictionary from scratch and presented the findings on 50k vocabulary for dnn and gmm based techniques with speaker adaptation. the experiments were carried out in the kaldi toolkit. the lowest wer for different gmm-hmm models were 29.45% and similarly, the lowest wer for different dnn models using dnn-tdnn-lstm was 11.55%. our study reveals that the researches till now, in nepali speech recognition, have used the traditional methods except for bhatta et al (2020) and @xcite who have used the ctc based end-to-end approach for small vocabulary tasks. the main difference between the ctc and attention method is that the conditional independence assumptions are not employed in the attention-based methods whereas ctc requires several conditional independence assumptions to get the probability of a label sequence. from this perspective, the attention mechanism, on the other hand, is surprisingly flexible as it allows extremely non-sequential alignments. however, for speech recognition tasks, the alignment is usually monotonic. as a regularization method, @xcite uses a ctc objective from loss function to an attention-based encoder network which encourages the alignments' monotonicity. this method improves the accuracy of asr compared to ctc or attention-based methods alone. ## architecture for joint ctc/attention model there are a series of steps involved in a speech recognition system requiring different components like data collection and preparation, data preprocessing, feature extraction, model building, training and testing, and decoding. for a machine learning system, clean and processed data are a basic prerequisite and thus represents a very crucial resource. figure 1 : joint ctc-attention asr architecture with deep cnn and rnn-lm. source: @xcite audio data should basically be collected and recorded in a controlled environment, without background noise such as coughing, wheezing, and throat cleansing, or environmental sounds such as beeps, phone rings, and door slams. in the nepali language, there are some speech corpora provided by the open speech and language resource for research purposes. fortunately, the speech data that we use from the aforementioned source is noisefree and thus ideal for our research. in this research, a ctc-attention based joint encoder-decoder is used that takes advantage of both ctc and attention during training. for the language modeling part, recurrent neural network is used. the rnn-lm network shows good improvement over the hybrid/hmm model and is merged in parallel with the attention decoder, which can be trained individually or jointly. the training is done with character sequences without word-level knowledge. figure 1 shows the architecture of a ctc-attention based joint encoder-decoder @xcite . this approach's performance is superior to the different state-of-the-art hybrid asr systems. in the given architecture, at first analog electrical signals are converted to digital signals. this is done in the feature extraction part where mel frequency cepstral coefficients (mfcc) are used to extract audio features to distinguish different sounds or letters of a language. after that, the features are passed to a deep cnn-based encoder that uses the visual geometry group (vgg) network. the input for the cnn network is the mel-scale feature from the raw speech features. the initial layers of the vgg net architecture (k. @xcite with 6-layer cnn architecture are used followed by blstm layers in the encoder network @xcite . the output is used by the ctc and the attention decoder as a shared encoder. the joint ctc-attention-based encoderdecoder utilizes both the benefits of ctc and attention during training. the ctc predictions are also used in the decoding process. the model outputs the l-length character sequence as a set of individual characters u. '१' , '२' , '३' , '४' , '५' , '६' , '७' , '८' , '९' , ' '} out of 129 unicode characters, 71 characters are used which are extracted from the text transcription of the speech data used in this research. the character set is indexed from 0 to 70. the input of ctc is the last hidden layer of the bidirectional long-short term memory (blstm). joint decoding is performed with a one-pass beam search algorithm that combines both attentionbased scores and ctc scores to further eliminate irregular alignments. use one tuning parameter to linearly interpolate both objectives, the multitasking learning (mtl) rate, which is typically set to 0.3. for the language modeling part, recurrent neural network is used. the rnn-lm probabilities of output label prediction are used in conjunction with the decoder network because they assign a probability to each clause in such a way that the more probable strings (in a sense) get a higher probability and we tend to choose one. similarly, an additional rescoring step is not needed if we combine the lm probabilities while decoding @xcite . thus, this model can be viewed as a single gigantic neural network, even though its parts are pretrained independently. ## experiments and results the audio data and the text transcription are collected from the openslr.org website which contains transcribed audio data for nepali language @xcite . the dataset contains nepali speech data containing ~157k utterances and a text file that contains the utterance id and the text of the respective speech utterances. all speech utterance sums up to speech of duration of 9,278 minutes and 11 seconds. this corpus contains 86,062 unique utterances. nepali speech corpus from open speech and language resource named "multispeaker tts data for nepali (ne-np)" (slr43) has been also used to the train models. it contains about 2064 long sentences-based utterance spoken by 18 different female speakers. all speech utterances sum up to speech of duration of 167 minutes and 45 seconds. this corpus contains 2064 unique utterances. altogether, ~159k utterances are used. for conducting the experiments, rtx 2060 gpu is used for training the model and ryzen 9 cpu with 16 cores is used for the decoding process. the dataset was split in the ratio of 8:2 for the train and test dataset and from the remaining 80% of the training data, again it was split for train and validation set in a ratio of 8:2. the cnn blstm encoder uses 80 mel-scale filter banks with the delta and delta-delta features as input features. a 4-layer blstm with 320 units per layer and direction is used. to extract the convolutional features, 10 centered convolutional filters with a width of 100 were used. a 1-layer lstm with 320 cells units are used as a decoder network. a singlelayer lstms for rnn-lms are individually trained using transcription, combined with a decoder network, and optionally retrained in collaboration with encoders, ctc networks, and decoders. the training is done for 20 epochs with the patience of value 2. here, no extra text data were used but the use of additional un-transcribed data can further improve the results. the adadelta algorithm with gradient clipping was used for the optimization @xcite . the beam width and ctc-weight were set to 20 and 0.3 in decoding process. the cer is used as the evaluation metric which is defined as the sum of characters that is substituted, inserted, and deleted in particular