Unified Model for Code-Switching Speech Recognition and Language Identification Based on Concatenated Tokenizer (2023)

Abstract: Code-Switching (CS) multilingual Automatic Speech Recognition (ASR) models can transcribe speech containing two or more alternating languages during a conversation. This paper proposes (1) a new method for creating code-switching ASR datasets from purely monolingual data sources, and (2) a novel Concatenated Tokenizer that enables ASR models to generate language ID for each emitted text token while reusing existing monolingual tokenizers. The efficacy of these approaches for building CS ASR models is demonstrated for two language pairs, English-Hindi and English-Spanish, where we achieve new state-of-the-art results on the Miami Bangor CS evaluation corpus. In addition to competitive ASR performance, the proposed Concatenated Tokenizer models are highly effective for spoken language identification, achieving 98%+ accuracy on the out-of-distribution FLEURS dataset.

## introduction automatic speech recognition (asr) systems are moving from specialized monolingual models to asr architectures capable of handling multiple languages simultaneously @xcite @xcite @xcite . code-switching (cs) is a special category of multilingual speech in which two or more languages or varieties of languages are used in the same utterance. it can further be divided into two categories: inter-sentential codeswitching where the switching between languages happens predominantly at the sentence boundaries and intra-sentential code-switching, which happens within the sentence @xcite . most of the work in code-switching asr is dependent on the availability of a good quality codeswitching speech corpus @xcite . one of the questions that we explore in this paper is: how to better utilize the readily available monolingual speech corpora and train cs asr systems that can perform well in real-world code switching scenarios. text post-processing after asr, e.g. punctuation and capitalization @xcite and inverse text normalization @xcite , is another important problem for multilingual and cs speech systems. such post-processing is harder than the monolingual scenario as it requires accurate language identification in addition to transcript generation. traditionally, separate language identification (lid) and asr models have been trained for the task, usually with a common acoustic encoder. li et al. @xcite was one of the first few works to propose an end-to-end architecture for intra-sentential cs asr. they trained two separate monolingual asr systems and a frame-level lid model. the posteriors of asr models were adjusted with the lid scores and greedy decoding was used without any language model rescoring. in @xcite , the authors proposed to use multigraph weighted finite state transducers, which was shown to be more effective than transformer-based systems for the intra-sentential cs. recent works @xcite , @xcite approach this problem differently by introducing special lid symbols such as @xcite , that are added to the vocabulary for language identification. these symbols are predicted either at the start of the utterance to identify which language the decoded text belongs to @xcite , or during the utterance to mark spans of decoded tokens belonging to each language @xcite . in this paper, we propose a streamlined technique for learning token level language id, which we term concatenated tokenizer. unlike previous approaches of "aggregating" tokenizers that take the union of the per-language tokenizer vocabularies @xcite or create a shared sub-word token set across languages @xcite , in the concatenated tokenizer method we reuse monolingual tokenizers and map them to mutually ex-clusive label spaces for each language. this helps provide explicit language information to the asr model while training and leads to inexpensive prediction of token level lid at decoding time. the main contributions of the paper are as follows: ## multilingual and code-switching asr modern natural language processing (nlp) and asr models use tokenizers to represent text @xcite . the traditional approach requires that a new tokenizer is learned for each language and domain. in asr, this tokenizer is also used to reduce the target sequence length to satisfy ctc requirements under aggressive downsampling with respect to original audio length @xcite . in this section, we discuss the proposed concatenated tokenizer and the synthetic code-switching data generation pipeline. ## results and discussion in this section, we present results for the experiments outlined in section 3.2. the results for english-spanish cs experiments are highlighted in when multilingual models are used to decode codeswitched speech, we observe that they tend to stick with the language in which the utterance started and are not able to switch between languages as they occur within the utterance. this is evidenced by the correspondingly high wers in tables 2a and 2b , and is consistent with the fact that the models did not encounter code-switched data during training. to illustrate, here are sample transcripts from one english-spanish code-switched audio: ml model output: con qué departamento puedo dejar fitbax sobre mi experiencia de compra en la tienda que estaba ubicada en one tú threforme ave cs model output: con qué departamento puedo dejar feedbacks sobre mi experiencia de compra en la tienda que estaba ubicada en one two three fourth avenue we can see that the ml model is not able to switch from spanish (majority language) to english (underlined for easier visual comparison). on the other hand, the output of the cs model is 100% accurate. finally, the results of the language identifica-tion experiments for all the three languages are presented in ## conclusion in this paper we investigate training of bilingual and code-switching models using purely monolingual datasets. we propose two novel techniques: