Meta-Transfer Learning for Code-Switched Speech Recognition (2020)

Abstract: An increasing number of people in the world today speak a mixed-language as a result of being multilingual. However, building a speech recognition system for code-switching remains difficult due to the availability of limited resources and the expense and significant effort required to collect mixed-language data. We therefore propose a new learning method, meta-transfer learning, to transfer learn on a code-switched speech recognition system in a low-resource setting by judiciously extracting information from high-resource monolingual datasets. Our model learns to recognize individual languages, and transfer them so as to better recognize mixed-language speech by conditioning the optimization on the code-switching data. Based on experimental results, our model outperforms existing baselines on speech recognition and language modeling tasks, and is faster to converge.

## introduction in bilingual or multilingual communities, speakers can easily switch between different languages within a conversation @xcite . people who know how to code-switch will mix languages in response to social factors as a way of communicating in a multicultural society. generally, code-switching speakers switch languages by taking words or phrases from the embedded language to the matrix language. this can occur within a sentence, which is known as intrasentential code-switching or between two matrix language sentences, which is called inter-sentential code-switching @xcite . learning a code-switching automatic speech recognition (asr) model has been a challenging task for decades due to data scarcity and difficulty in capturing similar phonemes in different * these two authors contributed equally. languages. several approaches have focused on generating synthetic speech data from monolingual resources @xcite . however, these methods are not guaranteed to generate natural code-switching speech or text. another line of work explores the feasibility of leveraging large monolingual speech data in the pre-training and applying fine-tuning on the model using a limited source of code-switching data, which has been found useful to improve the performance @xcite . however, the transferability of these pretraining approaches is not optimized on extracting useful knowledge from each individual languages in the context of code-switching, and even after the finetuning step, the model forgets about the previously learned monolingual tasks. in this paper, we introduce a new method, metatransfer learning 1 , to learn to transfer knowledge from source monolingual resources to a codeswitching model. our approach extends the model-agnostic meta learning (maml) @xcite to not only train with monolingual source language resources but also optimize the update on the codeswitching data. this allows the model to leverage monolingual resources that are optimized to detect code-switching speech. figure 1 illustrates the optimization flow of the model. different from joint training, meta-transfer learning computes the firstorder optimization using the gradients from monolingual resources constrained to the code-switching validation set. thus, instead of learning one model that is able to generalize to all tasks, we focus on judiciously extracting useful information from the monolingual resources. the main contribution is to propose a novel method to transfer learn information efficiently from monolingual resources to the code-switched speech recognition system. we show the effectiveness of our approach in terms of error rate, and that our approach is also faster to converge. we also show that our approach is also applicable to other natural language tasks, such as code-switching language modeling tasks. code-switching asr @xcite introduces a statistical method to incorporate a linguistic theory into a code-switching speech recognition system, and @xcite explore syntactic and semantic features on recurrent neural networks (rnns). @xcite adapts effective curriculum learning by training a network algorithm 1 meta-transfer learning require: d src , d tgt require: α, β: step size hyperparameters 1: randomly initialize θ 2: while not done do 3: sample batch data d tra ∼ (d src , d tgt ), d val ∼ d tgt 4: for all d tra t @xmath0: evaluate ∇ θ l d tra t i (f θ ) using d tra t i 6: compute adapted parameters with gradient descent: ## meta-transfer learning we aim to effectively transfer knowledge from source domains to a specific target domain. we denote our model by f θ with parameters θ. our model accepts a set of speech inputs @xmath1, . . . , x n } and generates a set of utterances @xmath2, . . . , y m }. the training involves a set of speech datasets in which each dataset is treated as a task t i . each task is distinguished as either a source d src or target task d tgt . for each training iteration, we randomly sample a set of data as training d tra , and a set of data as validation d val . in this section, we present and formalize the method. ## conclusion we propose a novel method, meta-transfer learning, to transfer learn on a code-switched speech recognition system in a low-resource setting by judiciously extracting information from highresource monolingual datasets. our model recognizes individual languages and transfers them so as to better recognize mixed-language speech by conditioning the optimization objective to the code-switching domain. based on experimental results, our training strategy outperforms joint training even without adding a fine-tuning step, and it requires less iterations to converge. in this paper, we have shown that our approach can be effectively applied to both speech processing and language modeling tasks. finally, we will explore further the generability of our meta-transfer learning approach to more downstream multilingual tasks in our future work.