IE - CPS Lexicon: An Automatic Speech Recognition Oriented I ndian- E nglish Pronunciation Dictionary (2021)

Abstract: Indian English (IE), on the surface, seems quite similar to standard English. However, closer observation shows that it has actually been influenced by the surrounding vernacular languages at several levels from phonology to vocabulary and syntax. Due to this, automatic speech recognition (ASR) systems developed for American or British varieties of English result in poor performance on Indian English data. The most prominent feature of Indian English is the characteristic pronunciation of the speakers. The systems are unable to learn these acoustic variations while modelling and cannot parse the non-standard articulation of non-native speakers. For this purpose, we propose a new phone dictionary developed based on the Indian language Common Phone Set (CPS). The dictionary maps the phone set of American English to existing Indian phones based on perceptual similarity. This dictionary is named Indian English Common Phone Set (IE-CPS). Using this, we build an Indian English ASR system and compare its performance with an American English ASR system on speech data of both varieties of English. Our experiments on the IE-CPS show that it is quite effective at modelling the pronunciation of the average speaker of Indian English. ASR systems trained on Indian English data perform much better when modelled using IE-CPS, achieving a reduction in the word error rate (WER) of upto 3.95% when used in place of CMUdict. This shows the need for a different lexicon for Indian English.

## introduction today, speech processing technology is gaining an undeniable importance. automatic speech recogni- * these authors contributed equally to the paper tion (asr) systems in particular are being sought after, as smart assistants like siri, alexa and google assistant grow in popularity at remarkable rates. effective and accurate speech recognition is also a major concern in india for two major reasonsfirstly, india is home to thousands of spoken languages, not all of which possess written forms; secondly, india has a low literacy rate, so a considerable portion of the population can communicate only via speech. unfortunately, there is a dearth of available speech data on many languages spoken in india, so most asr systems tend to be ineffective at modelling these. in order to both overcome the drawbacks caused by lack of available speech data, and leverage the similarities between the various indian languages, linguistically motivated approaches are gaining appreciation among researchers in the field. many research groups have started working towards having a good speech recognition system for all the major indian languages. to achieve this, it is necessary to have a good pronunciation modelling block, consisting of a grapheme-to-phoneme (g2p) system. a g2p is needed by text-to-speech (tts) systems as well. @xcite work on building a rule-based g2p on the low-resource malayalam language. @xcite work on building a g2p for major indian languages such as hindi, tamil and telugu and test it on tts systems. @xcite build a multilingual g2p that works for 61 languages out of the box, in which 7 indian languages are supported. they also provide a guideline to integrate any language into their system. @xcite work on statistical g2p that predicts schwa deletion in hindi more accurately and show improvements over their baseline models. @xcite work on sinhala g2p, a language that is closely related to other indian languages, and further, propose rules to handle schwa epenthesis. english is also a language with widespread use in india. given india's rich language diversity, english has largely been adopted in the spheres of the educated public as a lingua franca @xcite like in other parts of the world. the english spoken in india is influenced by the other languages with which it exists in constant contact, affecting its structure and vocabulary as well as its pronunciation. this means that existing asr systems for general american english (gae) or british english (bre) @xcite are markedly ineffective when dealing with indian english (ie). new asr systems need to be created which are capable of processing the variety of english spoken in india, catering to the accents and language patterns @xcite of the people speaking it. considering the limited work in this direction and the impact it may have, we study the major phonetic variation between indian english and standard english, and propose an indian english lexicon which can be used in an ie asr system. to show its effectiveness, the performance of the two different asr systems is compared on utterances of indian english -an american english asr system trained on librispeech @xcite and an indian english asr system trained on national programme on technology enhanced learning (nptel) speech corpus (described further in section 4.1). the organisation of the remainder of the paper is as follows. section 2 describes prior work. section 3 details the proposed lexicon and the rationale behind the chosen mappings. sections 4 and 5 explain the experimental setup and results when using the new dictionary for asr. further discussion on the need for this lexicon is done in section 6. finally, possible future ventures are briefly mentioned in section 7, and the paper is concluded in section 8. ## related work arpabet is a phonetic transcription code used to represent the phonemes and allophones of gae using distinct ascii sequences. the cmu pronouncing dictionary (cmudict) @xcite is an open-source pronunciation dictionary which makes use of a modified form of arpabet. currently, there has been very limited work done to replicate such efforts for other languages, or even other varieties of english. our proposed lexicon attempts to fill this gap in the field of language phonology. the work by @xcite is also closely related to our line of research in this paper. they collected a hindi-english code-switched speech corpus. in addition to releasing that corpus, they proposed a few measures that could help model such speech better. one of the measures they proposed was a way to handle the pronunciation model block for this code-switched corpus that is most often used in asr and text-to-speech (tts) systems. they proposed a cmudict-based phone mapping with some of the english phones mapped to a common indian language phone set @xcite based on perceptual similarity. the paper, however, did not present any experimental results for asr with their proposed pronunciation model. we explain how we build upon this in greater detail, in section 3. another study by @xcite compared the phonology of gae and the phonology of (hindi-based) ie and developed a phoneme set for ie in x-sampa @xcite similar to that of gae. however, the dataset on which they built the asr system has its own limitations. it has speech data from just 10 native hindi speakers with longterm exposure to only the english spoken in new delhi. as such, the language modelled was a very specific variety of ie, which is not representative of a large-scale indian population. @xcite collected a corpus of indian english, consisting of 240 hours of recorded speech. this is called indic timit. they extensively analysed indian english accent and compared it to standard english. they further came up with various kinds of rules to improve the pronunciation model for indian english speech. however, such rules needed another complex module -a letter to phoneme aligner. the letter to phoneme aligner is used to align a word's letters to its individual sound units. our work eliminates the need for this module, by applying only those rules which were exhibited by native speakers of a majority of indian languages. in addition to that, @xcite also did not use the now largely standardised phone codes for six major indian languages -hindi, marathi, bengali, tamil, malayalam and telugu -proposed by @xcite , the common phone set (cps). this makes @xcite 's work difficult to use in a multilingual or a code-switched setting where one of the languages is an indian language. they also did not compare the performance of their proposed-lexicon-based asr system and cmudict-based asr system. therefore, it is unclear what benefits their proposed lexicon might have to an asr system. in addition to the above works, the studies by @xcite ; @xcite ; @xcite were helpful in providing the general perspective on indian english. they described some specific differences made in the pronunciation of english by indians, at both the sound and word level. we incorporate some of the differences they point out at the sound level in our lexicon. in addition to the differences, they also provided insight into the question of whether indian english is better treated as a variety of standard english or as a completely separate language. this is discussed further in section 6. ## indian english-common phone set (ie-cps) given the limited availability of annotated ie speech data, there is need for a mechanism to provide the pronunciations for this kind of speech. our lexicon, indian english common phone set (ie-cps), is able to provide a simple yet effective solution to this by leveraging the existing cmudict, avoiding use of large amounts of data and complex architectures to model ie speech. all of the english phones in cmudict have been mapped to the existing common phone set (cps) phones proposed by @xcite . cps is a language-independent set of the phones of six major indian languages (hindi, marathi, bengali, tamil, malayalam and telugu) which maps parallel phones to the same code. reusing cps was favoured over creating a new encoding because it already provides a phone dictionary for a large set of indian languages, and is continuously extended to include a greater number. since english, too, is becoming more widespread in india, extending cps to include the proposed indian english phone dictionary is the logical consequence. while some of the mappings in this paper are inspired by @xcite , we did make changes to further reflect the accent that native indian language speakers would have when speaking english. these changes are elaborated on in section 3.2. ## experimental setup in this subsection, we elaborate on the gae and ie speech corpora upon which we ran our experiments. we also briefly describe two components of the asr system -the acoustic model and the language model. these components have not been changed in the study, and are only used to highlight the effectiveness of the ie-cps as a lexicon for indian english asr. ## results this section presents our results and analysis on ie-cps for acoustic model training. further, we compare and analyse the scenarios in which it outperforms the existing cmudict. all the experiments were conducted on librispeech and nptel data as mentioned in section 4.1. for evaluating ie-cps on a speech recognition task, two sets of experiments were performed with varying conditions, i.e., read and spontaneous condition. ## discussion & analysis in this section we investigate the behaviour of the ie-cps in more detail by asking the questions detailed below. if indian english varies so much with region, then how can this common lexicon be useful? it has been mentioned earlier that the regional lan-guages have a significant effect on spoken english. this results in several different accents and thus phoneme inventories. however, ie-cps is based on the general speech characteristics observed in speakers of a range of languages. the rules applied to map the phones from american to indian english are generated from the most prominent and widespread approximations made by indians speaking english, such as the transformation from alveolar to retroflex plosives. the notable improvement demonstrated by experimental results performed on the nptel speech data (consisting of a broad range of regional indian accents) also proves the overall effectiveness of the lexicon despite the variation in indian english. consequently, it may be extrapolated that such a lexicon can be applied to the english in regions where academic effort to model the speech is still deficient -such as north-east india, with languages bodo, khasi, assamiya, meitei, mizowithout any extra linguistic expertise on the matter. any required region specific tuning also becomes easier as there are fewer changes to make, which can be done using the already mostly standardised encoding of cps. furthermore, if there is the need for a lexicon for multilingual speech (also containing english), such a pronunciation dictionary with the phones common to most varieties of indian english would serve to better represent the multiple possible accents. end-to-end models do not need a lexicon. what is the need for such a lexicon? with sufficiently large amounts of data, it is possible to discard pronunciation models entirely and build end-to-end (e2e) asr systems. however, as it lexicon sentence ie-cps its disappointing when you are convinced that there is nothing cmudict this appointing when you are convinced that there is nothing ground truth its disappointing when you are convinced that there is nothing in cases where even that is not available, rule-based parsers are the only choice for researchers wanting to model a language or an accent. additionally, even with large training data, seq2seq models the long tail in the distribution of the data causes a deterioration in the overall performance. rulebased parsers avoid this issue as they do not rely on learning the data. in such scenarios, the rulebased ie-cps can be used to obtain considerable improvements as shown in ## future work the current ie-cps generalises several characteristics of spoken indian english. however, our preliminary analysis reveals that fine-tuning the proposed lexicon to a particular regional accent of indian english, such as bengali or malayalam, results in a better asr performance when testing on the same regional accent. for example, a bengali accent would cause [a] to be pronounced as @xcite , and aspiration of consonants would be largely absent for a malayalam speaker. such specific changes can further be adapted in ie-cps to reflect regional varieties of indian english. therefore, we plan to propose different lexicons for different regional accents of indian english in the future. there is a growing amount of interest in modelling code-switched speech, especially in the indian community @xcite . this can be seen by how successful the code-switching subtask was in the 2021 multilingual and codeswitching (mucs) asr challenges for low resource indian languages foot_8 @xcite . we also noticed that there is an increased interest in having a specialised pronunciation block for such settings. in this direction, we plan to use ie-cps to model the indian english part of the code-switched speech. this is especially useful because the other indian language can be modelled with the same set of phone codes -cps. therefore, such a lexicon would be easy to integrate into existing frameworks in such settings. ## conclusions in this paper, we propose a lexicon designed for use in asr -the indian english common phone set (ie-cps). the ie-cps is a lexicon that can be easily integrated into the existing, (largely) standardised phonetic code for indian languages -the common phone set (cps). we show the improvements in an indian english asr system when ie-cps is used as the pronunciation model. this proves that fine-tuning the pronunciation model to indian english when the asr system is deployed to work on indian english speech is the correct way going forward. this is especially true when the user is limited by either the available data or the computational resources to utilise the data.