Improving Pretraining Techniques for Code-Switched NLP (2023)

Abstract: Pretrained models are a mainstay in modern NLP applications. Pretraining requires access to large volumes of unlabeled text. While monolingual text is readily available for many of the world’s languages, access to large quantities of code-switched text (i.e., text with tokens of multiple languages interspersed within a sentence) is much more scarce. Given this resource constraint, the question of how pretraining using limited amounts of code-switched text could be altered to improve performance for code-switched NLP becomes important to tackle. In this paper, we explore different masked language modeling (MLM) pretraining techniques for code-switched text that are cognizant of language boundaries prior to masking. The language identity of the tokens can either come from human annotators, trained language classifiers, or simple relative frequency-based estimates. We also present an MLM variant by introducing a residual connection from an earlier layer in the pretrained model that uniformly boosts performance on downstream tasks. Experiments on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), involving four code-switched language pairs (Hindi-English, Spanish-English, Tamil-English, Malayalam-English) yield relative improvements of up to 5.8 and 2.7 F1 scores on QA (Hindi-English) and SA (Tamil-English), respectively, compared to standard pretraining techniques. To understand our task improvements better, we use a series of probes to study what additional information is encoded by our pretraining techniques and also introduce an auxiliary loss function that explicitly models language identification to further aid the residual MLM variants.

## introduction multilingual speakers commonly switch between languages within the confines of a conversation or a sentence. this linguistic process is known as codeswitching or code-mixing. building computational models for code-switched inputs is very important in order to cater to multilingual speakers across the world @xcite . multilingual pretrained models such as mbert @xcite and xlm-r @xcite appear to be a natural choice to handle code-switched inputs. however, prior work demonstrated that representations directly extracted from pretrained multilingual models are not very effective for code-switched tasks @xcite . pretraining multilingual models using code-switched text as an intermediate task, prior to task-specific finetuning, was found to improve performance on various downstream code-switched tasks @xcite . such an intermediate pretraining step relies on access to unlabeled code-switched text, which is not easily available in large quantities for different language pairs. this prompts the question of how pretraining could be made more effective for code-switching within the constraints of limited amounts of code-switched text. 1 in this work, we propose new pretraining techniques for code-switched text by focusing on two fronts: a) modified pretraining objectives that explicitly incorporate information about codeswitching (detailed in section 2.1) and b) architectural changes that make pretraining with codeswitched text more effective (detailed in section 2.2). pretraining objectives. the predominant objective function used during pretraining is masked language modeling (mlm) that aims to reconstruct randomly masked tokens in a sentence. we will henceforth refer to this standard mlm objective as stdmlm. instead of randomly masking tokens, we propose masking the tokens straddling language boundaries in a code-switched sentence; language boundaries in a sentence are characterized by two words of different languages. we refer to this objective as switchmlm. a limitation of this technique is that it requires language identification (lid) of the tokens in a code-switched sentence. lid tags are not easily obtained, especially when dealing with transliterated (romanized) forms of tokens in other languages. we propose a surrogate for switchmlm called freqmlm that infers lid tags using relative counts from large monolingual corpora in the component languages. architectural changes. inspired by prior work that showed how different layers of models like mbert specifically encode lexical, syntactic and semantic information @xcite , we introduce a regularized residual connection from an intermediate layer that feeds as input into the mlm head during pretraining. we hypothesize that creating a direct connection from a lower layer would allow for more language information to be encoded within the learned representations. to more explicitly encourage lid information to be encoded, we also introduce an auxiliary lid-based loss using representations from the intermediate layer where the residual connection is drawn. we empirically verify that our proposed architectural changes lead to representations that are more language-aware by using a set of probing techniques that measure the switching accuracy in a code-switched sentence. with our proposed mlm variants, we achieve consistent performance improvements on two natural language understanding tasks, factoidbased question answering (qa) in hindi-english and sentiment analysis (sa) in four different language pairs, hindi-english, spanish-english, tamil-english and malayalam-english. sections 3 and 4 elaborate on datasets, experimental setup and our main results, along with accompanying analyses including probing experiments. our code and relevant datasets are available at the following link: https://github.com/ csalt-research/code-switched-mlm . ## related work while not related to code-switching, there has been prior work on alternatives or modifications to pretraining objectives like mlm. @xcite is one of the first works to identify the lack of linguistically intuitive pretraining objectives. they propose new pretraining objectives which perform similarly to mlm given a similar pretrain duration. in contrast, @xcite sticks to the standard mlm objective, but questions whether masking only 15% of tokens in a sequence is sufficient to learn meaningful representations. @xcite maintains that higher masking up to even 80% can preserve model performance on downstream tasks. all of the aforementioned methods are static and do not exploit a partially trained model to devise better masking strategies on the fly. and content in different training stages. ours is the first work to offer both mlm modifications and architectural changes aimed specifically at codeswitched pretraining. prior work on improving code-switched nlp has focused on generative models of code-switched text to use as augmentation @xcite @xcite , merging real and synthetic code-switched text for pretraining @xcite , intermediate task pretraining including mlm-style objectives @xcite . however, no prior work has provided an in-depth investigation into how pretraining using code-switched text can be altered to encode information about language transitions within a code-switched sentence. we show that switch-point information is more accurately preserved in models pretrained with our proposed techniques and this eventually leads to improved performance on code-switched downstream tasks. ## conclusion pretraining multilingual models with codeswitched text prior to finetuning on task-specific data has been found to be very effective for code-switched nlp tasks. in this work, we focus on developing new pretraining techniques that are more language-aware and make effective use of limited amounts of real code-switched text to derive performance improvements on two downstream tasks across multiple language pairs. we design new pretraining objectives for code-switched text and suggest new architectural modifications that further boost performance with the new objectives in place. in future work, we will investigate how to make effective use of pretraining with synthetically generated code-switched text.