Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching (2021)

Abstract: Code-switching (CS), a ubiquitous phenomenon due to the ease of communication it offers in multilingual communities still remains an understudied problem in language processing. The primary reasons behind this are: (1) minimal efforts in leveraging large pretrained multilingual models, and (2) the lack of annotated data. The distinguishing case of low performance of multilingual models in CS is the intra-sentence mixing of languages leading to switch points. We first benchmark two sequence labeling tasks – POS and NER on 4 different language pairs with a suite of pretrained models to identify the problems and select the best performing char-BERT model among them (addressing (1)). We then propose a self training method to repurpose the existing pretrained models using a switch-point bias by leveraging unannotated data (addressing (2)). We finally demonstrate that our approach performs well on both tasks by reducing the gap between the switch point performance while retaining the overall performance on two distinct language pairs in both the tasks. We plan to release our models and the code for all our experiments.

## introduction code-switching (cs) is a phenomenon of switching back and forth between multiple languages and is very common in multilingual communities such as india, singapore, etc. understanding mixed language texts has several applications in an increasingly online world like hateful content detection, maintaining engagement with virtual assistants. despite this pervasive prevalence, cs is often overlooked in language processing research and current models still cannot effectively handle cs. we believe that the reasons behind this are (1) the lack of efforts in leveraging existing large scale multilingual resources or pretrained models and (2) dearth of annotated resources in switching scenar-ios. in this paper, we present solutions to address these two problems specifically. the advent of pretraining techniques marshalled the celebrated successes of several language understanding and generation tasks in english @xcite and multilingual tasks @xcite . however, the same level of commendatory results are not translated to cs scenarios; as studied by @xcite ; @xcite presenting a preliminary evaluation of multi-lingual pretrained models for cs scenarios. it is still largely unclear if the inadequacies are resulting due to dearth of data or ineptitude of quick adoption of multilingual models. we study precisely this problem of identifying the artifacts that hinder the competent performance of pretrained models on cs with a case study on sequence labeling tasks including part-of-speech (pos) tagging and named entity recognition (ner). our contributions from this work are as follows: (1) we first conduct a comprehensive benchmarking of different pretrained models for two sequence labeling tasks across 4 different language pairs. specifically we evaluate datasets in hinglish, tenglish, benglish and spanglish cs for the tasks ner and pos. (2) to broaden understanding towards the usefulness of different fine-tuning strategies, we investigate multitasking, character modeling uncovering the problematic switch point cases in §4. (3) we propose a novel switch-point bias based self training approach built upon on observations from the benchmarks and demonstrate improved results on both tasks. pretrained models for cs: before the advent of pretrained multingual models, pretrained monolingual models were combined in different ways to derive word embeddings @xcite , pos tagging (bhattu et al., 2020), sentiment analysis (singh and lefever, 2020) etc., similarly, pretrained multilingual models have been explored on various cs tasks like language identification, pos tagging, ner, question answering and natural language inference @xcite . however, @xcite show that these pretrained models do not assure high quality representations on cs. we examine prospective reasons for this and present a data augmentation technique to mitigate this. motivation for our work -gaps in cs adaptation: building off the prior work, we will briefly discuss primarily three techniques that demonstrated usefulness in adapting models to cs. first, non-standardization of cross-scripting (i.e, transliteration of words to another language) is identified as one of the major reasons behind the noisiness of cs datasets @xcite . prior literature on noisy texts proved the superiority of character level modeling to combat this problem @xcite ; @xcite . secondly, the domains of most of these noisy datasets are still vastly scattered. in order to improve generalization in cs patterns, prior studies have shown the potency of multitasking with an auxiliary task of language tag prediction @xcite . thirdly, the dearth of annotated cs data has been a dramatic problem across tasks. @xcite compare pretrained models with fined-tuned models augmented with unlabeled twitter text to corpus notation task # sentences twitter @xcite enhi-tw-p pos 1489 ud @xcite enhi-ud-p pos 1311 icon @xcite enhi-i-p pos 2630 icon @xcite enbn-i-p pos 625 icon @xcite ente-i-p pos 1979 miami ( alghamdi et al., 2019) enes-m-p pos 27893 twitter @xcite enhi-tw-n ner 1243 calcs @xcite enes-tw-n ner 50757 to this end, we adapt these techniques in conjunction with the pretraining strategies and propose a novel bias-based data iterative augmentation technique to get more bang for the buck in terms of the performance to augmented dataset size ratio. ## switch-point biased self training as observed in the previous section, performance of the models deteriorates at the switching points @xcite in cs. this motivates our approach to tackle this problem which can be stated concretely as: the pre-trained model favors embedded-to-matrix over matrix-to-embedded language switching points despite majority of training data in the former pattern. we demonstrate this by comparing figure 1 present the counts and val accuracy with the increasing percentage of train set on the x-axis to demonstrate the consistency of this pattern. as we can observe, the number of samples with switch points from @xmath0. however, the performance on switch points from @xmath1. algorithm 1 switch-point biased self training (selftr) input: annotator model a(θ), labeled data d l , unlabeled data d u output: trained end-task model e(φ ) 1. fine-tune e(φ) on d l 2. s ← identify the low-performing switch-point 3. d u s ← sub-sample data from d u with higher ratio of s 4. d wl s ← annotate d u s with a(θ ) 5. a(θ ) ← fine-tune a(θ) on d wl s + d l 6. e(φ ) ← train e(φ) on d wl s + d l 7. repeat steps 2 to 6 by updating a(θ ) and e(φ ) we posit that a switch point specific fine-tuning is required to combat this imbalance. our proposed approach is depicted in algorithm 1. the baseline for each task is the char-bert model fine-tuned on the task-specific data, which is referred as end-task ## conclusions cs, despite being a natural and prevalent form of communication is still vastly understudied in empirical research. this mainly stems from the (1) lack of efforts in re-purposing the celebrated pretrained models to cs scenarios and (2) lack of annotated resources. we tackle precisely these 2 problems with the main focus on evaluating and improving how these models fare at switch points between languages. first, we benchmark a suite of monolingual and multilingual pretrained models on cs and identify that particular switch points fare poorly. we propose a novel switch point bias based self training method to strategically use unlabeled data to enhance performance at switch points. while improving or retaining the overall performance compared to finetuning char-bert and multitasking, we show that our approach improves the performance of underperforming switch points as well. we believe that this bias based augmentation technique particularly helps in scenarios with less annotated data.