The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding (2021)

Abstract: While recent benchmarks have spurred a lot of new work on improving the generalization of pretrained multilingual language models on multilingual tasks, techniques to improve code-switched natural language understanding tasks have been far less explored. In this work, we propose the use of bilingual intermediate pretraining as a reliable technique to derive large and consistent performance gains using code-switched text on three different NLP tasks: Natural Language Inference (NLI), Question Answering (QA) and Sentiment Analysis (SA). We show consistent performance gains on four different code-switched language-pairs (Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA and on Hindi-English for NLI and QA. We also present a code-switched masked language modeling (MLM) pretraining technique that consistently benefits SA compared to standard MLM pretraining using real code-switched text.

## introduction code-switching is a widely-occurring linguistic phenomenon in which multiple languages are used within the span of a single utterance or conversation. while large pretrained multilingual models like mbert @xcite and xlm-r (conneau et al., 2020) have been successfully used for low-resource languages and effective zero-shot cross-lingual transfer @xcite @xcite , techniques to help these models generalize to code-switched text have not been sufficiently explored. intermediate-task training @xcite was recently proposed as an effective training strategy for transfer learning. this scheme involves fine-tuning a pretrained model on data from one or more intermediate tasks, followed by fine-tuning on the target task. the intermediate task could differ from the target task and it could also be in a different language. this technique was shown to help with both task-based and language-based transfer; it benefited target tasks in english @xcite and helped improve zero-shot crosslingual transfer @xcite . in this work, we introduce bilingual intermediatetask training as a reliable training strategy to improve performance on three code-switched natural language understanding tasks: natural language inference (nli), factoid-based question answering (qa) and sentiment analysis (sa). bilingual training for a language pair x-en involves pretraining with an english intermediate task along with its translations in x. the nli, qa and sa tasks require deeper linguistic reasoning (as opposed to sequence labeling tasks like part-of-speech tagging) and exhibit high potential for improvement via transfer learning. (the fact that nli, qa and sa have more room for improvement compared to pos and ner tagging is evident from the leaderboard statistics in @xcite .) we present sa results for four different language pairs: hindi-english (hi-en), spanish-english (es-en), tamil-english (ta-en) and malayalam-english (ml-en), and nli/qa results for hi-en. 1 our main findings can be summarized as follows: ## methodology intermediate-task training. this scheme starts with a publicly-available multilingual model that has been pretrained on large volumes of multilingual text using mlm-based training objectives. this model is subsequently fine-tuned using data from one or more intermediate tasks before finally fine-tuning on code-switched data from the target tasks. single intermediate-task training makes use of existing monolingual nli, sa and qa datasets as intermediate-tasks before fine-tuning on the respective code-switched target tasks. for a language pair x-en, where @xmath0, hi, ta, ml}, we explored the use of three different intermediate tasks: multi intermediate-task training involves two intermediate-tasks (t 1 and t 2 ) simultaneously. this training is done using two different task heads (one per task) with the pretrained models. each batch is randomly populated with instances from tasks t 1 or t 2 . we follow @xcite to sample batches from task t 1 with probability p t 1 = min(e t 1 ,k) min(e t 1 ,k)+min(e t 2 ,k) where e t 1 and e t 2 are the number of training examples in task t 1 and t 2 , respectively; p t 2 is similarly computed. the constant @xmath1. we experiment with nli and qa as the two intermediate-tasks t 1 or t 2 and refer to this system as hi-en/nli-qa multi-task. we use the merged en and hi datasets from hi-en sing-task for each task. we also explored mlm training on real code-switched text as one of the tasks, in addition to the merged x-en task-specific intermediate-tasks (referred to as x-en/mlm multi-task). code-switched mlm. a common approach to training models for code-switched tasks is to perform additional mlm on real (or synthetic) code-switched text. however, randomly masking from the pool of all tokens in a sentence may not be the most effective use of real code-switched text and differentiating it from monolingual text, especially if one has access to word-level language tags. given word-level language labels for each token in the code-switched sentences, we aim to emphasize switching via the mlm training objective by masking tokens from words that lie on the switching boundaries. we refer to this training strategy as code-switched mlm. for example, consider the following sentence where tokens that can be masked are enclosed within boxes for both the standard mlm and code-switched mlm strategies, respectively: in the first sentence, tokens from all the words can be masked, as in standard mlm pretraining. in the second sentence that uses code-switched mlm pretraining, only tokens from words at the boundary of a language switch can be masked. to implement this, we need access to annotated language tags for each sentence or a highly accurate language identity detection system. as the evaluation metric, we use accuracies for nli and sa over two (entailment/contradiction) and three labels (positive/negative/neutral), respectively, and f1 scores for the qa task. ## related work while pretrained multilingual models are being increasingly used for cross-lingual natural language understanding tasks, their effectiveness for codeswitched tasks has not been thoroughly explored. @xcite show that embeddings from pretrained multilingual models are not very ef-fective for code-switched tasks and more work is needed to effectively adapt them. intermediate task-training has proven to be effective for many nlp target tasks @xcite , as well as crosslingual zero-shot transfer from english tasks on multilingual models such as xlm-r @xcite and mbert @xcite . ours is the first work to show improved intermediate tasktraining strategies for code-switched target tasks. @xcite and @xcite showed that mbert is effective for hi-en part-of-speech tagging and a reading comprehension task on synthetic code-switched data, respectively. this was extended for a variety of code-switched tasks by @xcite , where they showed improvements on several tasks using mlm pretraining on real and synthetic code-switched text. @xcite further improved the nli performance of mbert by including large amounts of in-domain code-switched text during mlm pretraining. @xcite empirically demonstrate that pretraining is most beneficial when the domains of the intermediate and target tasks are similar, which we observe as well. differing from their recommendation of domain adaptive pretraining using mlm on large quantities of real code-switched data, we find intermediate-task training using significantly smaller amounts of labeled data to be more consistently beneficial across tasks and languages. in contrast to very recent work @xcite that reports results using a robertabased model trained exclusively for sentiment analysis and pretrained on 60m english tweets, we present a bilingual training technique that is consistently effective across tasks and languages while requiring significantly smaller amounts of data. instead of using mbert and xlm-r that are very broad in their coverage of languages, it would be interesting to examine whether our observed trends hold when using pretrained models specifically trained for the chosen target languages. we could consider using very recent models like indicbert @xcite and muril @xcite that are trained exclusively on indian languages and have been shown to outperform mbert on cross-lingual tasks (e.g., xtreme) and tasks like indicglue, respectively. we leave this investigation for future work. ## conclusion this is the first work to demonstrate the effectiveness of intermediate-task training for codeswitched nli, qa and sa on different language pairs, and present code-switched mlm that consistently benefits sa more than standard mlm. we also carry out ablations of transliteration systems and compare their performance across the same corpora translated using different techniques. we observe that high-quality translations and transliterations are important to derive performance improvements on downstream tasks. for future work, we plan to continue exploring pretraining strategies, based on more informed masking objectives and task-adaptive techniques. one key limitation of the newly introduced codeswitched mlm approach is the requirement of lid systems for the languages under consideration. future work can focus on mitigating this requirement.