Improving Code-switched ASR with Linguistic Information (2022)

Abstract: This paper seeks to improve the performance of automatic speech recognition (ASR) systems operating on code-switched speech. Code-switching refers to the alternation of languages within a conversation, a phenomenon that is of increasing importance considering the rapid rise in the number of bilingual speakers in the world. It is particularly challenging for ASR owing to the relative scarcity of code-switching speech and text data, even when the individual languages are themselves well-resourced. This paper proposes to overcome this challenge by applying linguistic theories in order to generate more realistic code-switching text, necessary for language modelling in ASR. Working with English-Spanish code-switching, we find that Equivalence Constraint theory and part-of-speech labelling are particularly helpful for text generation, and bring 2% improvement to ASR performance.

## introduction although accurate statistics on the number of worldwide bilingual speakers are hard to determine, it has been generally believed that more than half of the population can communicate in more than one language @xcite @xcite . with the rising popularity of voice assistant and translation applications, automatic speech recognition (asr) systems have been increasingly integrated into people's lives. considering the abundance of bilingual countries foot_0 , and code-switching foot_1 is common in everyday conversations, there has been great interest into developing such system in the corresponding setting. the most widely accepted definition of codeswitching is the phenomenon whereby a speaker shifts from one language to another within a single utterance, especially in an environment in which both languages are being used @xcite . some previous work make a distinction between code-switching and code-mixing, where the former occurs at sentence-level while the latter occurs at word-level @xcite . however, in recent years, this distinction has becomes unclear @xcite . to avoid confusion, we will only use the term code-switching in this work, and denote the differences in switching position as sub-types of it @xcite @xcite . although different language pairs may present varying extents or types of code-switching, they can generally be categorised as inter-sentential, intra-sentential, and tag switching, where respectively the language switches at sentence or clause boundary; within the sentence or clause; or by inserting a tag phrase foot_2 . in this paper, we focus only intra-sentential switching, which is a much harder task compared with other types, because the acoustic variance of mixed languages within the sentence can be larger than across sentence @xcite . the challenge of developing a code-switched asr system comes from both linguistic and computational perspectives. on the one hand, codeswitching can be driven by multiple factors, which makes it flexible but hard to predict. linguists have studied the phenomenon for decades, and the main views held are that people tend to codeswitch to compensate for lack of language proficiency, express solidarity or certain feelings, discuss some particular topic, and distinguish themselves from other classes to imply a certain social status @xcite @xcite . asr systems generally require a large amount of transcribed data in a monolingual setting. however, a relatively small number of the approximately 7000 existing languages have large readily corpora, and the data scarcity is more severe for code-switching problems, which -to make matters worse -usually involve one or more lowresourced languages @xcite . motivated by the these considerations, we propose a novel code-switched asr framework with the aid of established linguistic theories. we demonstrate the effectiveness of the approach on spanish-english conversational code-switching data from the bangor-miami corpus @xcite . in doing so, we prove both phonological and syntactic information can improve the performance of language modelling and asr. ## related work there have been many attempts to approach the problem of modelling code-switched speech from acoustic, pronunciation and language modeling perspectives for conventional hybrid asr systems. in early work a language identification (lid) system was combined to determine which hypothesis from multiple monolingual decoders should be chosen @xcite @xcite . however, the drawback of this approach is a strong assumption that the speech segments are independent from each other and a heavy dependency on the accuracy of lid module @xcite . the choice of phone inventories is important, and many studies have been conducted to merge phone sets of different language pairs manually or automatically, which improves the performance of asr systems by effectively yielding more training data for each phone, and enabling the pronunciation influence between languages to be learnt @xcite @xcite . considering that there is a much larger amount of monolingual text than code-switched text, code-switched text generation by imposing language theories to parallel monolingual texts has been a popular research direction @xcite @xcite . from this, language models can be improved by training them on the generated text or can also be achieved by apply the theories directly to restrict the search paths in a wfst framework @xcite . in recent years, end-to-end (e2e) models have also been explored to handle the problem. to ad-dress the issue that e2e models usually require a large amount of data, transfer learning from monolingual models has been used @xcite @xcite . however, when the models are finetuned on code-switched data, the performance on monolingual speech is degraded. to improve the robustness of the model, techniques such as learning without forgetting and adversarial training have been proposed @xcite . ## results and discussion our baseline model uses the union of english and spanish phoneme sets while mapping base maps the spanish phoneme set to english phoneme set. their language models are identical which are trained only on the transcripts of bangor-miami corpus. it can be observed from the result that phoneme mapping can improve the performance by 0.3 absolute wer, so we fix it as our acoustic model and the only difference among the experiments on the bottom block is that they have different synthetic texts for language modelling. translation model denotes that the language model is trained on the transcripts as well as the translation of them and + symbol describes with what techniques, code-switched text have been generated and added to the training text. we interpolate the lm with models trained on external text data foot_5 to show that simply using larger but out-ofdomain text data doesn't help improve the perfor-mance. after obtaining the word alignments of parallel sentences, we compare the results of generating the code-switched text by simply random replacement of the aligned words, or ranking the possible replacements with the pos tags of the preceding/current/following words or the acceptance under ec theory. we can find that pos and ec have similar improvement on wer, while the combination of them shows their advantages cannot directly add up. one possible explanation can be that the implementation of ec is based on the constituency parse, which is also heavily related to the pos tags. therefore, the linguistic information implied by them are overlapped with each other and only little improvement can be achieved when combined. our model with the best performance uses all of linguistic information we discussed before, with approximately 2% improvement on both wer and ppl. ## conclusions in this paper, we present a framework for codeswitched asr task. by using phonological features for phoneme mapping, and pos tags and ec theory for more more natural code-switched text generation, we eventually achieve 2% improvement on ppl as well as wer. it should be noted that although the experiments are conducted on bangor-miami corpus, there are no language specific constraints with this approach, which shows a potential to be extended to cover more language pairs. as a future work, we would like to compare the performances of different linguistic theories in our proposed framework, which can serve as an indirect validation of their influence on different language pairs. also, motivated by natural distributions of linguistic structures, exploring different sampling techniques can also be promising.