BERT ologi C o M ix: How does Code-Mixing interact with Multilingual BERT ? (2021)

Abstract: Models such as mBERT and XLMR have shown success in solving Code-Mixed NLP tasks even though they were not exposed to such text during pretraining. Code-Mixed NLP models have relied on using synthetically generated data along with naturally occurring data to improve their performance. Finetuning mBERT on such data improves it’s code-mixed performance, but the benefits of using the different types of Code-Mixed data aren’t clear. In this paper, we study the impact of finetuning with different types of code-mixed data and outline the changes that occur to the model during such finetuning. Our findings suggest that using naturally occurring code-mixed data brings in the best performance improvement after finetuning and that finetuning with any type of code-mixed text improves the responsivity of it’s attention heads to code-mixed text inputs.

## introduction massive multilingual models such as mbert @xcite and xlmr @xcite have recently become very popular as they cover over 100 languages and are capable of zero-shot transfer of performance in downstream tasks across languages. as these models serve as good multilingual representations of sentences @xcite , there have been attempts at using these representations for encoding code-mixed sentences @xcite @xcite . code-mixing (cm) is the mixing of words belonging two or more languages within a single sentence and is a commonly observed phenomenon in societies with multiple spoken languages. these multilingual models have shown promise for solving cm tasks having surpassed the previously achieved performances @xcite . this is an impressive feat considering that these models have never been exposed to any form of code-mixing during their pre-training stage. traditionally, cm has been a spoken phenomenon though it is slowly penetrating into written form of communication @xcite . however, they mostly occur in an informal setting and hence such cm data is not publicly available in large quantities. such scarcity of data would mean that building independent cm models can be unfeasible. with the onset of pre-trained multilingual models, further training with cm data can help in adapting these models for cm processing. however, even for further training, there is a requirement for a significant amount of data albeit lesser than starting from scratch. the amount of data available even for their monolingual counterparts is very less @xcite ) let alone the amount of real-world cm data. this can prove to be a bottleneck. rightly so, there have been previous works exploring synthesis of cm data for the purpose of data augmentation @xcite . synthesis of cm mostly rely on certain linguistic theories @xcite to construct grammatically plausible sentences. these works have shown that using the synthetic and real cm data in a curriculum setting while fine-tuning can help with achieving better performances on the downstream cm tasks. though this is analogous to adapting models to new domains, cm differs in that the adaptation is not purely at vocabulary or style level but rather at a grammatical level. although it is known such adaptation techniques can bring an improvement, it is not well understood how exactly fine-tuning helps in the cm domain. through this paper, we seek to answer these lingering questions which exist in area of cm processing. we first study the impact of finetuning multilingual models with different forms of cm data on downstream task performance. for this purpose, we rely on three forms of cm varying in their complexity of mixing, naturalness and obtainability -(i) randomly ordered code-mixing (l-cm), (ii) grammatically appropriate code-mixing (g-cm) both of which are synthetically generated and (iii) real-world code-mixing (r-cm). we perform this comparative analysis in a controlled setting where we finetune models with the same quantity of cm text belonging to different forms and then evaluate these finetuned models on 11 downstream tasks. we find that on average the r-cm performs better on all tasks, whereas the synthetic forms of cm (l-cm, g-cm) tend to diminish the performance as compared to the stock/non-finetuned models. however, these synthetic forms of data can be used in conjuction to r-cm in a curriculum setting which allows to alleviate the data scarcity issue. in order to understand the difference in the behavior of these models, we analyze their self-attention heads using a novel visualization technique and show how finetuning with cm causes the model to respond more effectively to cm texts. we notice that using r-cm for finetuning makes the model more robust and the representations more distributed leading to better and stable overall performances on the downstream tasks. the rest of the paper is organized as follows. section 2 surveys prior work done in domain adaptation of transformer-based lms, code-mixing and interpretability and analysis techniques. section 3 introduces the different types of code-mixing and the models that we build with them. section 4 and 5 respectively presents the task-based and attention-head based probing experiments along with the findings. section 6 concludes the paper by summarizing the work and laying out future directions. ## models in this section, we describe the mbert models, the modifications we make to them, and the types of cm data that we use for training. ## task-based probing in this section, we describe layer-wise task-based probing of the different models. ## structural probing as observed earlier, exposing mbert to r-cm help boost its overall and layer-wise performance on cm tasks. in this section, we describe three structural probing experiments, through which we will try to visualize the structural changes in the network, if any, induced by continued pre-training with cm data that are responsible for performance gains. we will first look at whether there are any changes in the behaviour of attention heads at a global level by checking the inter-head distances within a model. further, we want to localize and identify the heads whose behaviours have changed. finally, we take a look at how the attention heads respond to code-mixed stimulus. ## conclusion & future work in this work, we develop different methods of finetuning bert-like models for cm processing. we then compare the downstream task performances of these models through absolute performance, their stability as well as the layer-wise solvability of certain tasks. to further understand the varied performances between the three types of cm, we perform structural probing. we adopted an existing approach and introduced a couple of new techniques for the visualization of the attention heads as a response to probes. the most important finding from these probing experiments is that there are discernable changes introduced in the models due to exposure to cm data, of which a particularly interesting observation is that this exposure increases the overall responsivity of the attention heads to cm. as of now, these experiments are purely analytical in nature where we observed how the attention heads behave on a cm stimuli. one future direction is to expand the analysis to a wider range of domains and fine-tuning experiments to understand how generalizable are our findings of distributed information in bert-like models. we use a fairly simple and easily replicable method for testing this through the responsivity metric that we propose. this method can be further improved to rigorously verify our observations.