Exploring the Effect of Dialect Mismatched Language Models in T elugu Automatic Speech Recognition (2022)

Abstract: Previous research has found that Acoustic Models (AM) of an Automatic Speech Recognition (ASR) system are susceptible to dialect variations within a language, thereby adversely affecting the ASR. To counter this, researchers have proposed to build a dialect-specific AM while keeping the Language Model (LM) constant for all the dialects. This study explores the effect of dialect mismatched LM by considering three different Telugu regional dialects: Telangana, Coastal Andhra, and Rayalaseema. We show that dialect variations that surface in the form of a different lexicon, grammar, and occasionally semantics can significantly degrade the performance of the LM under mismatched conditions. Therefore, this degradation has an adverse effect on the ASR even when dialect-specific AM is used. We show a degradation of up to 13.13 perplexity points when LM is used under mismatched conditions. Furthermore, we show a degradation of over 9% and over 15% in Character Error Rate (CER) and Word Error Rate (WER), respectively, in the ASR systems when using mismatched LMs over matched LMs.

## introduction automatic speech recognition (asr) systems are rapidly becoming part of our everyday lives through voice assistants such as siri, alexa, and google assistant. since these voice assistants can now perform various day-to-day tasks exceedingly well, they have now become an integral part of many devices such as phones, televisions, music players, and smartwatches. accurate and reliable asr systems for indian languages would have a significant impact due to two reasons: firstly, india is home to many languages and dialects. many of these languages and dialects do not have a written form. secondly, a considerable amount of the population in india cannot read or write, as evidenced by the low literacy rates. 1 this leaves such people with only one mode of communication -the spoken form. despite the advances made by spoken technology research in recent years, dialect or accent variation proves to be a huge challenge. 2 @xcite show that accent variation contributes most to speech variability after gender. @xcite ; @xcite show the amount of degradation in asr performance when it is not trained on dialect-specific data. therefore, currently, state-of-the-art systems, including those of google and microsoft, use dialect-specific asr systems. however, multi-dialect asr is an attractive solution in scenarios where sufficient dialect-specific data or information is not available. therefore, @xcite ; @xcite ; @xcite al. (2018 @xcite use auxiliary accent trees to model chinese accent variation. these are decision trees that model accent-specific triphone units and have a similar function as the decision trees that are used for state-tying of standard triphones. @xcite show that graphemebased recurrent neural network-connectionist temporal classification (rnn-ctc) asr models outperform their phoneme-based counterparts when trained and used in multi-dialect english conditions. furthermore, they study modelling phoneme recognition as an auxiliary task to im- @xcite encoder to simultaneously detect the dialect and transcribe an audio sample. more recently, with increased interest in self-supervised learning, @xcite explored self-supervised learning techniques to predict the accent from speech and use the predicted information to train an accentspecific self-supervised asr. they report that such a model significantly outperforms an accentindependent asr system. many researchers have previously studied the effects of dialect mismatched acoustic models in asr systems. however, to the best of our knowledge, we are the first to explore the effects of a dialect mismatched language model (lm) in asr systems. our language of interest in this paper is telugu. telugu is a south central dravidian language primarily spoken in two states of india: telangana, and andhra pradesh. as previously mentioned, low literacy states in these states has motivated researchers to build telugu asr systems @xcite @xcite @xcite . however, they largely concentrate on building asr systems for "standardised" telugu. while mirishkar et al. (2021b) collect dialect-specific telugu data, they do not conduct any asr experiments on individual telugu dialects. we conduct our experiments on three regional telugu dialects, i.e., telangana, rayalaseema, and coastal andhra. a considerable portion of dialect variation in telugu can be seen in the lexicon, grammar, and occasionally semantics. additionally, since indian languages are considered to be low-resource in nature, adding external text to the lm is a solution that has gained interest @xcite @xcite . while such a method has shown significant benefits in their asr systems, we argue that if proper care is not taken in matching the dialect of the external text with that of the asr, it could lead to degradation in performance. these are the primary motivations for us to conduct this study. to this effect, the following are the major contributions of the paper: the rest of the paper is organised as follows. in section 2, a brief description of the three dialects used in this study is given. in section 3, we describe the dataset used in the study. in section 4 and section 5, we describe our experimental setup and discuss results under matched and mismatched settings. we conclude the paper with section 6 and discuss possible future directions. ## telugu dialects all telugu dialects can be broadly classified into three regional dialects: telangana, coastal andhra, and rayalaseema. the formation of these dialects is primarily due to the influence of neighboring states, and the regional culture @xcite . the nizams ruled the telangana region, whose official languages were persian and urdu. thus, one can see the influence of urdu with many nativised urdu words present in telangana @xcite . here are some such examples: కాకా, జాగా, దవాఖానా . foot_2 there is also some influence of the neighboring states' languages like kannada on telangana. the coastal andhra dialect is largely influenced by sanskrit as well as tamil due to historical and geographical reasons @xcite . finally, the rayalaseema dialect is influenced by neighboring states' languages, i.e., tamil and kannada (shivaprasad and sadanandam, 2020). interested readers are referred to ## dataset we conduct our experiments on a corpus of three telugu dialects collected by @xcite . it is a crowd-sourced read speech corpus collected from the native speakers of the regional dialects of telugu. in all audio used in this study is mono channel, sampled at 16khz with 16-bit encoding. the prompt given to the speakers is hand-curated. therefore, we were able to ensure that the datasets across dialects have no domain mismatch. this allows us to study dialect mismatch better, which is our primary interest in this study. ## experimental setup all of the asr experiments are conducted using espnet @xcite . the input acoustic features are 80-dimensional log mel features extracted on the fly. we choose to use the conformer model @xcite as it was able to achieve state-of-the-art performance on many standard datasets. the encoder of the asr uses 12 conformer @xcite blocks with 8 attention heads while the decoder uses 6 transformer @xcite blocks with 4 attention heads. we train both the encoder and decoder with a dropout rate of 0.1. all the models are trained based on the hybrid ctc/attention architecture @xcite . the training is done within the multi-objective learning (mol) framework. the ctc loss term helps the attention model converge faster. the training objective (l mol ) is as follows: here, λ is the multitask coefficient which should satisfy the following condition: 0 ≤ λ ≤ 1. we found λ set to 0.3 while training and 0.4 while decoding gave us the best results for our datasets. c is the output unit. this could be characters, subword units, or words. using words as output units could lead to two major issues: out of vocabulary (oov%) cannot be handled well. the number of output units could be very high, especially in an agglutinative language like telugu, which could lead to data sparsity. @xcite show that using subwords over characters leads to better performance of end-to-end asr systems. thus we opted to use subwords as the output units. we used sentencepiece @xcite to tokenise the words into subwords. 6 we found a vocabulary of around 500 tokens to give us the best performance on all the three datasets. we refer readers interested in how vocabulary size affects the performance of asr of different telugu dialects to appendix a. finally, x, in the above equation, is the input acoustic features. 6 we used no external text to train the tokeniser. we take mucs21_subtask1 foot_5 recipe in espnet since it is tuned to perform well on a similar sized indian dataset and make the following modifications: change the initial learning rate to 5 × 10 -4 , and use early stopping with a criterion to stop training the model if its performance does not improve for 5 consecutive epochs on the validation set. we train an independent 16 block transformer lm with an embedding size of 128 and a hidden encoder size of 512 for a maximum of 25 epochs. finally, the decoder uses an lm weight of 0.6 to predict a sequence of subwords. 8 this method of integrating lm into the end-to-end asr is known as shallow fusion @xcite and it is shown to give better results than other forms of integrating lm into the end-to-end asr @xcite . to decode, we use beam search of size 10 to predict the sequence. ## results & discussion in this section, the results of the experiments conducted are reported, and briefly analysed. @xcite experiment the effectiveness of cross-dialect asr in arabic by experimenting with cross-dialect acoustic model (am) and training the lm on target dialect data. in this paper, we take the exact opposite approach, i.e., train the am (in this case, end-to-end asr before the independent lm is fused) on the target dialect data and experiment by using a cross-dialect lm. we do this to test the effectiveness of the lm and thereby the asr in cross-dialect conditions. 9 no external text was used to train lms as it is difficult to obtain dialect information of external text. 10 we report the performance of the lm both in terms of extrinsic metric, i.e., cer and wer of the asr which uses the lm in question as well as an intrinsic metric, i.e., perplexity. from as expected, the perplexity of the lm trained on all the dialects is the least. lm's perplexity under matched settings much better in all the three dialects compared to mismatched conditions. as discussed before, telangana lm is highly sensitive to dialect mismatch, with perplexity increasing by over 13 points. lm trained on coastal andhra and telangana and tested on rayalaseema leads to the highest increase in perplexity, i.e., 5.82 and 13.13 points, respectively. on the other hand, rayalaseema lm is most robust to any dialect mismatch. ## conclusion & future work this paper studies how lms perform under dialect mismatched conditions. our experiments reveal that lms perform poorly, with the perplexity score increasing sharply in dialect mismatched conditions. we use the mismatched lms in asr systems to study how they are affected. similar to what we have observed with perplexity scores of the lm, we notice a significant degradation in the performance of the asr with over 15% difference in wer in dialect mismatched conditions when compared to its matched counterpart. furthermore, through our study, we show that mismatched lms can actively hinder the performance of asr by comparing it to asr systems with no lm. these findings show the importance of careful curation of external text when training a dialectspecific asr system. these experiments have also led to an interesting finding: rayalaseema dialect is more robust under dialect mismatched conditions as it shares a lot of similarities with both coastal andhra and telangana. in the future, we plan to improve the lm and thereby the asr in dialect mismatched conditions using various adaptation techniques available in the literature. we hope that our future work would lead to lms that are more robust to dialect mismatched conditions, thereby leading to improved asr systems.