An Investigation of Hybrid architectures for Low Resource Multilingual Speech Recognition system in I ndian context (2021)

Abstract: India is a land of language diversity. There are approximately 2000 languages spoken around, and among which officially registered are 23. In those, there are very few with Automatic Speech Recognition (ASR) capability. The reason for this is the fact that building an ASR system requires thousands of hours of annotated speech data, a vast amount of text, and a lexicon that can span all the words in the language. At the same time, it is observed that Indian languages share a common phonetic base. In this work, we build a multilingual speech recognition system for low-resource languages by leveraging the shared phonetic space. Deep Neural architectures play a vital role in improving the performance of low-resource ASR systems. The typical strategy used to train the multilingual acoustic model is merging various languages as a unified group. In this paper, the speech recognition system is built using six Indian languages, namely Gujarati, Hindi, Marathi, Odia, Tamil, and Telugu. Various state-of-the-art experiments were performed using different acoustic modeling and language modeling techniques.

## introduction according to the 2011 census, india has 23 constitutionally recognized official languages and 1600 other languages @xcite . in this era of digitization, speech technologies for indian languages play a pivotal role across various business domains. building automatic speech recognition (asr) @xcite ) and text-to-speech (tts) @xcite based interfaces for indian markets is big challenge as majority of languages are "low resourced" @xcite . in general, a language is referred to as low resource when there is: (i) lack of availability of speech, text, transcribed data, (ii) lack of linguistic expertise in a particular language, or (iii) lack of pronunciation dictionary @xcite . in order to build state-of-the-art asr systems for indian languages, tons of training data is required for achieving human parity. in the training asr system, the model expects audio data and corresponding transcripts as an input. though there is a lot of raw speech freely available for indian languages, it is a complex and costly process to get the corresponding transcription. hence, very few efforts were made in these directions over the past decade. as indian languages are syllabic, efforts were put in generating the pronunciation dictionary from a simple rule-based parser @xcite @xcite . indian languages have a peculiar attribute of sharing the same phonetic space. however, they differ phonotactically foot_0 @xcite . so these attributes could be exploited to build an asr system for achieving better performance. with the introduction of digital india mission in 2015, there have been efforts towards handling these low resource languages for building speech recognition technologies. different approaches have been proposed in acoustic modeling over the recent years to address the low resource speech recognition problem. in @xcite , the attempt has been made to use cross-lingual acoustic data to initialize deep neural network (dnn) based acoustic models through unsupervised restricted boltzmann machine (rbm) pre-training. it showed that unsupervised pre-training remains vital for the hybrid setups, especially with limited amounts of transcribed training data. the idea of transfer learning approach was introduced for handling low resource languages in @xcite @xcite . data augmentation approaches proposed in @xcite @xcite were effective in low resource settings. a different line of work @xcite @xcite , where attempts have been made to extract multilingual features that help in improving low resource asr systems started gaining prominence. in practice, a language identification (lid) block is used a front-end wherein it tries to predict the language first, and later it is mapped to the corresponding monolingual system or the best possible monolingual system. in this type of approach, the lid block, which acts as a front-end, should be as accurate as possible and robust enough to operate in a multi-thread environment. so to circumvent this issue, acoustic models where it handles multiple languages without any prior knowledge of the language have been proposed @xcite . in (shetty and umesh, 2021), authors have explored the benefits of phonetic sound principles and treated each character unit across the languages as a separate entities with the help of common label set (cls) approach. in this paper, the authors compare and analyze different time delay neural networks (tdnn) variants @xcite as they seem to be best suited for such kind of low resource speech recognition tasks. this paper investigates the effectiveness of different acoustic models for low resource multilingual speech recognition for six indian languages (namely hindi (hi), marathi (mr), odia (od), tamil (ta), telugu (te), and gujarati (gu)). among these languages, hindi, marathi, odia and gujarati are indo-aryan languages, while tamil and telugu fall under the category of dravidian languages. as mentioned above, the authors considered six indian languages ie., hi, mr, od, ta, te and gu, among these languages except hi and mr all others have different orthography (grapheme style). moreover, every languages has its own training set they are (x hi ,l hi ), (x m r ,l m r ), (x od ,l od ), (x t a ,l t a ), (x t e ,l t e ), and (x gu ,l gu ), where x corresponds to the input acoustic sequence and l represents the label for target acoustic sequence. initially, the monolingual systems are built for each language and trained with corresponding pairs which is acoustic sequence and its labels. later, the authors attempt to show that the developed multilingual systems offer improved performance. as a part of multilingual system a joint acoustic model is trained, for which training dataset is constructed by pooling data from all the six languages, i.e., (x the joint acoustic model is a single acoustic where the parameters are shared across all the six languages. this keeps the authors on similar lines with other research done on the development of asr systems for low resource languages. in this work, a joint acoustic model-based asr has been developed using different acoustic models. a joint acoustic model (jam), which recognizes multiple languages with a single acoustic model, is widely appreciated. many research groups have been actively working on this for the past few years @xcite . the usage of the joint acoustic model leverages the crosslingual knowledge transfer. it reduces the complexity in the asr pipeline significantly compared with the monolingual model as it has to maintain one model per language. the results show that the multilingual tdnn system result in lower word error rates (wer). we use kenlm @xcite and recurrent neural network (rnn) based language models @xcite for decoding and rescoring respectively. the organization of the remainder of the paper is as follows: section 2 details the proposed lexicon along with the rationale behind the chosen mappings and database analysis. section 3 explains the experimental setup. section 4 discuss experimental results and few of the analysis which we have carried on. finally, the study is concluded in section 5, with possible future research directions to explore. ## dataset & lexicon details interspeech 2018 has organized asr challenge as a special session @xcite . as part of the challenge, 40 hours of speech data has been released for three languages of gujarati, telugu, and tamil. in continuation, inter-speech 2021 has extended the challenge to six languages, i.e., hindi (hin), marathi (mar), odia (odi), tamil (tam), telugu (tel) and gujarati (guj) respectively @xcite . the speech data statistics for the six languages are tabulated in the ## speech recognition experimental setup in this section, the authors describe the experimental setup for both the monolingual and multilingual asr systems. the word error rate (wer) is the metric used to evaluate the performance of asr systems throughout this paper. ## experimental results & analysis the experiment results for the six languages are reported in the hmm do not share the common distribution among the six languages. this has led to the poor performance when the jam is trained using sgmm. jam trained using low rank tdnn has yielded better performance than tdnn and sgmm based systems. unlike sgmms, low rank tdnn and tdnn acoustic models are trained to model longterm temporal variations. the variabilities caused due to the presence of six languages have been effectively handled using low rank tdnns due to the attributes of the architecture like skip connection, bottleneck linear transformation layer and batch normalization. from the last row of the ## conclusion & future work in this paper, a tdnn based multilingual asr system for six indian languages, i.e., hindi, marathi, odia, tamil, telugu, and gujarati was explored. our experiment results show that the multilingual models achieve comparable results to the monolingual models when the parameters are in a comparable range. in few cases, asr performance improved by including external text data while building languages model. cls is investigated for studying the effectiveness of jam for building a multilingual asr system for six indian languages. it is observed that low rank tdnn has shown superior performance over conventional tdnns. since most indian languages are syllabic in nature and share a common phonetic space, the authors believe that the cls approach can be further extended to more indian languages in future. the feasibility of adapting lm along with the am can also be explored for improving the performance of low resource multilingual asr system. as india is a multilingual society, it is common occurrence for code-switching to be observed. in general, the authors would like to focus on two types of code-switching; (i) intra-sentential and (ii) intersentential. in the regular conversation, people try to switch, that is, the language exchange takes place at the sentence boundaries, and in the latter case, the languages switch into sentences, thus, creating a more complex problem. the six language multilingual model which we have built can deal with the inter-sentential cases. however, in hindi and marathi both intra and inter-sentential cases works as these two languages share same grapheme structure. therefore, our future work will focus on recognizing intra-sentential code-switching utterances by exploring different architecture and utilizing monolingual data. the experimental findings from this paper will benefit researchers planning