Text-Derived Language Identity Incorporation for End-to-End Code-Switching Speech Recognition (2023)

Abstract: Recognizing code-switching (CS) speech often presents challenges for an automatic speech recognition system (ASR) due to limited linguistic context in short monolingual segments, resulting in language confusion. To mitigate this issue, language identity (LID) is often integrated into the speech recognition system to provide additional linguistic context. However, previous works predominately focus on extracting language identity from speech signals. We introduce a novel approach to learn language identity from pure text data via a dedicated language identity-language model. Besides, we explore two strategies: LID state fusion and language posterior biasing, to integrate the text-derived language identities into the end-to-end ASR system. By incorporating hypothesized language identities, our ASR system gains crucial contextual cues, effectively capturing language transitions and patterns within code-switched utterances. We conduct speech recognition experiments on the SEAME corpus and demonstrate the effectiveness of our proposed methods. Our results reveal significantly improved transcriptions in code-switching scenarios, underscoring the potential of text-derived LID in enhancing code-switching speech recognition.

## introduction automatic speech recognition (asr) systems have long grappled with the complex task of accurately transcribing multilingual speech, especially when it involves the phenomenon known as code-switching (cs). code-switching, or code-mixing, refers to the practice of alternating between two or more languages or dialects within a single conversation. within these intricate language mixtures, the presence of very short monolingual segments further compounds the challenge. the limited contextual information within these segments often leads to confusion in recognizing phonetically similar words from different languages, substantially affecting the overall performance of multilingual asr systems @xcite @xcite . traditional approaches to code-switching asr have resorted to language identification or code-switching detection systems as separate pre-processing or post-processing steps. while these methods have been effective in providing additional language context @xcite @xcite , they introduce additional complexity to the asr pipeline and increase processing time. the advancement of deep learning has brought forth a remarkable paradigm shift in multilingual and code-switching asr systems. end-to-end (e2e) approaches have gained substantial attention, harnessing advanced neural network architectures, such as the transformer @xcite , to directly transcribe code-switched speech @xcite . these e2e cs asr systems automatically capture both the acoustic and linguistic characteristics of code-switched speech, eliminating the need for explicit acoustic and language modeling. to reduce language confusion in e2e cs asr systems, researchers have studied methods for integrating language identity (lid) information learned from both paired speechtext data @xcite @xcite and unpaired speech data @xcite @xcite . however, these prior approaches have predominantly focused on learning language identities from speech data, overlooking the untapped potential of pure text data. compared to annotated speech data, text data offers a more accessible and readily available resource. this paper explores the possibility of learning language-switching patterns from pure text data without relying on speech data. text-derived features, such as part-of-speech tags and syntactic features, have been shown to improve the performance of code-switching language models @xcite @xcite . inspired by these achievements, we propose to infer language identities from text data and integrate these text-derived language identities into the end-to-end cs asr system. our objective is twofold: to acquire language-switching syntax and patterns from text data in the form of language identities, and to leverage these language identities to mitigate the language confusion in code-switching speech recognition. we believe the paired in-domain text inherently encompasses a wealth of linguistic information, which can be effectively harnessed to advance code-switching speech recognition. to acquire code-switching patterns sorely from text data, we propose a novel language modeling scheme, called language identity-language model, to predict the language identity of the next text token based on a combined history of previous text and language identity tokens. the proposed language identity-language model is jointly trained with a transformer-based asr model. furthermore, we explore two strategies for integrating the predicted language identities into an e2e asr system: lid state fusion and language posterior biasing. in the lid state fusion strategy, we combine token-level language identity hidden states with asr hidden states using learned weights. on the other hand, the language posterior biasing strategy directly adjusts the asr posterior probabilities based on the hypothesized language identities and language posteriors. we evaluate the effectiveness of these proposed methods on the seame corpus, a mandarin-english speech dataset. the results underscore the efficacy of our approaches in reducing language confusion during code-switching speech recognition, leading to more accurate transcriptions of code-switched speech. the remainder of this paper is structured as follows: section 2 reviews the background of the transformer-based asr with parallel speechtext decoder. section 3 presents related work for leveraging language identification in multilingual and code-switching speech recognition. section 4 presents the proposed lid-lm for generating language identities from pure text data and lid integration strategies. section 5 describes the datasets, models, and evaluation metrics used to assess the performance of the proposed method. section 6 presents the results and analysis of the experiments. finally, section 7 summarizes the contributions and outlines directions for future research. ## background in this section, we briefly review the transformerbased asr architecture with parallel speech-text decoder that we used to implement our method. ## related work language identification plays a vital role in various multilingual speech processing applications, particularly in multilingual or code-switching speech recognition systems. it provides valuable contextual information that regulates speech recognizers and reduces language confusion. early multilingual asr systems adopted a two-stage approach, where a language identification component was incorporated at the front-end to distinguish speech from different languages, followed by the use of monolingual recognizers to transcribe speech in specific languages at the back-end @xcite . subsequently, frame-level language identities predicted by a dedicated language identification module were integrated into the asr decoding process to handle rapid language changes @xcite . with the advancements of deep learning, there has been a shift towards incorporating language identification as an auxiliary task, jointly learned with end-to-end multilingual and code-switching speech recognition systems @xcite @xcite @xcite . additionally, seki et al. @xcite dynamically track the language identity in code-switching utterances by adding language identity tokens before code-switching points in speech transcriptions, eliminating the need for an external language identification module. this lid token augmentation method is also used in @xcite , where different language embeddings are concatenated with the word embedding of text tokens to further enhance the distinguishability between word embeddings of different languages. however, existing research has predominantly concentrated on extracting language identity from speech signals. in contrast, our work takes an innovative approach, delving into the prospect of learning language identity directly from raw text data. by harnessing the rich linguistic information within textual data, we aim to reduce language confusion in end-to-end code-switching speech recognition systems, leading to more accurate transcriptions. ## conclusion in this paper, we presented a novel language identity-language model scheme to predict language identity from pure text data, eliminating the need for reliance on speech data. we also explored two innovative methods to effectively incorporate text-derived language identity cues into asr models. our code-switching speech recognition experimental evaluations on the seame corpus demonstrated the effectiveness of our methods. by incorporating language identity information, our asr system exhibits significantly reduced language confusion in transcribing code-switching utterances, yielding more precise transcriptions for both monolingual and code-switched utterances. future work includes further enhancements to the lid-lm ar-chitecture and investigating additional integration strategies to better leverage language identity information in asr systems. additionally, exploring pre-training and fine-tuning techniques for lid-lm would open up more possibilities for efficiently utilizing text data to predict language identity.