Improving End-to-End B angla Speech Recognition with Semi-supervised Training (2020)

Abstract: Automatic speech recognition systems usually require large annotated speech corpus for training. The manual annotation of a large corpus is very difficult. It can be very helpful to use unsupervised and semi-supervised learning methods in addition to supervised learning. In this work, we focus on using a semi-supervised training approach for Bangla Speech Recognition that can exploit large unpaired audio and text data. We encode speech and text data in an intermediate domain and propose a novel loss function based on the global encoding distance between encoded data to guide the semi-supervised training. Our proposed method reduces the Word Error Rate (WER) of the system from 37% to 31.9%.

## introduction in this work, we focus on improving an endto-end speech recognition system for bangladeshi 1 * authors contributed equally bangla using semi-supervised training. there are very few publicly available large speech corpora for bangladeshi bangla. google released 229 hours of speech corpus for bangladeshi bangla @xcite . but there are huge amounts of publicly available news audio files, audiobooks, recordings in youtube and other media sources. there are a lot of text sources too like news websites, blogs, e-books, etc. considering the abundance of unpaired audio and text data for bangla language, a semi-supervised training method that can exploit both unpaired audio and text is very useful. proper use of the unpaired data along with existing paired speech corpus can boost the performance of the bangla asr system. different researchers have tried different ways of incorporating this unlabelled, unannotated data for speech recognition. our approach is similar to the approach used by @xcite . we utilize an intermediate representation of speech and text data using a shared encoder network for semi-supervised training of the asr system. our contributions in this work are as follows: the paper is organized in the following manner. we discuss the related works in section 2, the system architecture in section 3, details of our inter-domain loss in section 4, corpus description in section 5, experimental results in section 6, and conclusion in section 7. ## related works researchers have explored different methods of semi-supervised training for speech recognition. @xcite investigate large-scale semisupervised training to improve acoustic models for automatic speech recognition. they provide an empirical analysis of semi-supervised training with respect to transcription quality, data quality, filtering, etc. @xcite pre-train the encoder-decoder network with unpaired speech and text. they use a large amount of unpaired audio to pre-train the encoder and synthesized audio from the unpaired text to pre-train the decoder. @xcite , @xcite integrate active learning jointly with semi-supervised training in speech recognition system. @xcite use transcribed multilingual data and semi-supervised training to circumvent the lack of sufficient training data for acoustic modeling. they train deep neural networks as data-driven feature front ends. @xcite use utterance-level and frame-level confidences for data selection during self-training. they find it beneficial to reduce the disproportion in amounts of paired and unpaired data by including the paired data several times in semi-supervised training. @xcite describe the combination of deep neural networks and graph-based semi-supervised learning for acoustic modeling in speech recognition. dhaka and salvi (2017) use a sparse auto-encoder to take advantage of both unlabelled and labeled data simultaneously through mini-batch stochastic gradient descent. @xcite try to improve the performance of a code-switching speech recognition system for mandarin-english using semi-supervised training. they apply semi-supervised learning for lexicon learning as well as acoustic modeling. similarly, @xcite use untranscribed data for luxembourgish & lithuanian asr respectively. šmídl et al. (2018 ) use a twostep training method to generalize the air traffic control speech recognizer. first, a baseline speech recognition system is trained using a paired speech corpus and it is used to transcribe publicly available unlabeled data. the transcribed data is then filtered based on confidence scores and is used to retrain the acoustic model. recently, semi-supervised training has been proposed in the context of end-to-end asr. @xcite propose a shared encoder architecture for speech and text inputs that can encode both data from their respective domain to a common intermediate domain. they combine speech-to-text and text-to-text mapping by using the shared network to improve speech-to-text mapping. they propose an inter-domain loss function based on gaussian kl-divergence which represents the dissimilarity between the encoded features of speech and text data. they later proposed an inter-domain loss function based on maximum mean discrepancy @xcite . in both cases, they assume that the encoded speech features in the current minibatch are sampled from one distribution and encoded text features in the current minibatch are sampled from a second distribution. the interdomain loss is calculated based on the discrepancy of these two distributions. this approach has some weaknesses. the performance of this system varies based on the chosen minibatch size. moreover, this approach does not take into account the variance of the current encoded features in the global context. we solve both problems by introducing a new inter-domain loss function based on global encoding distance. ## our system in this section, we describe our baseline end-to-end architecture as well as semi-supervised architecture. 3.1 baseline system our baseline system is an end-to-end asr system based on the work of watanabe et al. (2017). the architecture is shown in figure 1. ctc and attention networks are combined in this architecture. both networks share an encoder network. the shared encoder network has 6 layers of bidirectional long short term memory (blstm) units. each layer has 320 blstm units. a linear projection layer is connected to each blstm layer. the linear projection layer consists of 320 units. the decoder has 1 layer of unidirectional lstm units. the number of lstm units in this layer is 300. the scores from the attention network and the ctc network are combined during decoding. let p(c t ) be the probability of output label c t at position t, given previous output labels and let w be the ctc weight. as for the audio feature, we use 40 melfrequency cepstral coefficients (mfcc) per audio frame. we also consider their first and second-order temporal derivatives. so, we have 120 speech features per audio frame. these features are fed to the shared encoder and the attention decoder generates the character sequence. we use a recurrent neural network (rnn), based language model, in shallow fusion @xcite with the baseline end-to-end architecture. we use both character level and word level rnn in our experiments. the character level rnn has 2 layers of lstm, with each layer having 650 lstm units. the word-level rnn has 1 hidden layer and this layer has 1000 lstm units. for the word level rnn, we use most frequently used 65000 bangla words as our vocabulary set. text input speech input decoder encoder pyramid blstm text output inter domain loss embedding figure 2: semi-supervised system ## the inter-domain loss in this section, we describe our proposed interdomain loss function. ## corpus description in this section, we describe the corpus used for our experiments. ## evaluations in this section, we describe the experimental results. ## conclusions in this paper, we present a semi-supervised approach for the incorporation of unpaired audio data to boost the performance of a bangla asr system. our proposed inter-domain loss function based on global encoding distance performs better than the gaussian kl divergence and mmd loss proposed previously. we exploit 1000 hours worth of unpaired audio and a similar amount of text data in our semi-supervised training to optimize our speech recognition system. our asr which is trained on publicly available paired speech corpus and unpaired data resources outperforms the asr trained only on the paired speech corpus with language models. in the future, we will try to improve the performance of the semi-supervised system further by fusing an additional language model during decoding.