Adapting Multilingual Models for Code-Mixed Translation (2022)

Abstract: The scarcity of gold standard code-mixed to pure language parallel data makes it difficult to train translation models reliably.Prior work has addressed the paucity of parallel data with data augmentation techniques.Such methods rely heavily on external resources making systems difficult to train and scale effectively for multiple languages.We present a simple yet highly effective two-stage back-translation based training scheme for adapting multilingual models to the task of code-mixed translation which eliminates dependence on external resources.We show a substantial improvement in translation quality (measured through BLEU), beating existing prior work by up to +3.8 BLEU on code-mixed Hi \rightarrow En, Mr \rightarrow En, and Bn \rightarrow En tasks. On the LinCE Machine Translation leader board, we achieve the highest score for code-mixed Es \rightarrow En, beating existing best baseline by +6.5 BLEU, and our own stronger baseline by +1.1 BLEU.

## introduction as code-mixing @xcite @xcite becomes widespread in an increasingly digitized bilingual community, it becomes important to extend translation systems to handle code-mixed input. a major challenge for training code-mixed translation models is the lack of parallel data. recent work on generating synthetic parallel data using available non-code-mixed parallel data depend on language specific tools for transliteration, wordalignment, and language identification @xcite . this makes the approach difficult to scale to new languages and increases software complexity. back-translation (bt) is another effective and popular strategy to handle non-availability of parallel data @xcite . however, for the code-mixed to english translation task, simple bt is not an option since we cannot assume the presence of an english to code-mixed translation model. meanwhile the mainstream translation community is converging on frameworks based on multilingual models for translation between multiple language pairs @xcite @xcite @xcite . going forward, code-mixed translation needs to be integrated within these frameworks to impact practical systems. we propose a novel two stage back-translation methodology called back-to-back translation (b2bt) targeted for adapting multilingual models to code-mixed translation. our approach is simple and integrates easily with existing multilingual translation models without any need for special models or language specific tools. we compare b2bt with six other baselines on both standalone and mbart-based models across four benchmarks and show significant gains. for example, on codemixed hindi to english translation b2bt improves state-of-art accuracy by +3.8 and by +6.3 over default back-translation. we analyze the reasons for the gains via both human evaluation and impact on downstream models. we release a new dataset and will publicly release our code. ## our approach our objective is to train a model that can translate a sentence from the code-mixed language c, which contains words from english and an additional language s, to monolingual english e. following (myers-scotton, 1997) we refer to s as the matrix language as it lends its grammar in a code-mixed utterance, and english as the embedded language since it lends only its words. we are given parallel s to english corpus (s, e) ⊂ (s, e) and a non-parallel code-mixed corpus c ⊂ c. since code-mixing appears more in domains like social media, which differ from formal domains like news in which parallel data (s, e) is available, we addi- train multilingual model train model c e b e @xmath0: inp: <2en> arrays डे टा का एक सं ग्रह होता है । out: arrays is a collection of data. fine-tune for @xmath1: inp: <2en> maintenance पै नल में , customer branches लं क पर िक्लक करें । out: in the maintenance panel, click on the customer branches link. model e md c b back-translate fine-tune ~ẽ xample of back-translation: inp: <2cm> here, the person class is an abstract class. out: यहाँ , person क्लास एक abstract क्लास है । tionally use a domain-specific monolingual english corpora e m d ⊂ e. optionally, we can also exploit monolingual data s m ⊂ s and e m ⊂ e. our method called b2bt of training a @xmath2. ## related work code-mixing is receiving increasing interest in the nlp community @xcite al. (2020) the biggest challenge in translation of codemixed sentences is the lack of large parallel training data @xcite @xcite . @xcite propose to create synthetic parallel cm data via these two steps: (1) train an mbert model to identify word set w to switch in a sentence from s to e, effectively creating a sentence from c (2) align parallel sentences from (s, e) and replace words in w to their aligned english words. we call this the mbertaln method in this paper. this pipeline for a new language s requires the following four external tools: (1) mbert pre-trained on s, (2) a language identifier tool to spot english tokens in a cm sentence, (3) a word alignment model, and (4) a translator @xmath3. for low-resource languages such tools may not exist. in contrast b2bt is totally standalone. even when external tools exist, we show empirically that the synthetic sentences thus generated tend to be of lower quality than ours because of errors in any of the two steps. the calcs 2021 workshop @xcite also released a shared task for cm translation but the submissions so far are straight-forward application of bart multilingual models, with which we also compare our method. b2bt is reminiscent of dual learning nmt methods @xcite @xcite but these methods were designed for two generic languages whereas b2bt for code-mixed translation handles three languages related in specific asymmetric ways. we exploit that asymmetry to design our training schedule. for example, since @xmath4. ## experiments we use the notation @xmath5, to indicate translation from a code-mixed matrix language with code 'so' to english. we evaluate on four codemixed datasets: hindi (@xmath6. (2021 ), spanish (@xmath7, bengali (@xmath8. (2021) but augmented with the newly released samanantar data to create a stronger baseline (evaluation is done on the splits released by the authors), and a new marathi (@xmath9://ritual.uh.edu/lince/leaderboard lang pair method st-test st-oov st-hard @xmath10.9 33.9 2.1 @xmath11.9 41.4 18.6 mbertaln 46.4 44.6 23.4 multilingual 38.0 37.7 17.5 multilingual + @xmath12.0 40.9 22.6 multilingual + @xmath13.7 35.8 20.6 b2bt 50.2 49.9 30.7 @xmath14.8 31.1 14.1 @xmath15.9 41.2 21.2 mbertaln 41.4 41.9 22.3 multilingual 30.9 31.4 13.8 multilingual + @xmath16.7 42.0 22.0 b2bt 44.2 43.4 23.4 @xmath17.6 25.7 0.9 @xmath18.3 39.2 16.5 mbertaln 40.6 40.5 17.8 multilingual 29.1 29.7 9.0 multilingual + @xmath19.4 41.5 18.9 b2bt 41.2 41.3 18.7 our approach can also complement existing multilingual pre-trained models such as mbart. finally, we compare code-mixing statistics between the synthetic data generated by b2bt and mbert in varying degree of code-mixing following @xcite , we also evaluate the effectiveness of our model across different splits of the test set with varying code-mixing index (gambäck and das, 2016) (cmi). figure 3 presents the improvements from our model on the three splits of the test set. we see improvements across all splits, but the largest improvements are on the split with the highest degree of code-mixing. on the high cmi split, we see about +8.7 bleu point improvement over the mbert approach, and +14.5 bleu point improvement over the baseline. ## conclusion we present a simple two-stage back-translation approach (b2bt) for adapting multilingual models for code-switched translation. b2bt shows remarkable improvements on four datasets compared to recent methods, and default back-translation baselines. our approach fits naturally with existing multilingual translation frameworks, which is crucial in expanding coverage to low resource lan-guages without building per-language pair models. we demonstrate with ablation studies and human evaluations that the synthetic data created through the two step process in b2bt is objectively higher quality than the one used by existing work.