Exploring Enhanced Code-Switched Noising for Pretraining in Neural Machine Translation (2023)

Abstract: Multilingual pretraining approaches in Neural Machine Translation (NMT) have shown that training models to denoise synthetic code-switched data can yield impressive performance gains — owing to better multilingual semantic representations and transfer learning. However, they generated the synthetic code-switched data using non-contextual, one-to-one word translations obtained from lexicons - which can lead to significant noise in a variety of cases, including the poor handling of polysemes and multi-word expressions, violation of linguistic agreement and inability to scale to agglutinative languages. To overcome these limitations, we propose an approach called Contextual Code-Switching (CCS), where contextual, many-to-many word translations are generated using a ‘base’ NMT model. We conduct experiments on 3 different language families - Romance, Uralic, and Indo-Aryan - and show significant improvements (by up to 5.5 spBLEU points) over the previous lexicon-based SOTA approaches. We also observe that small CCS models can perform comparably or better than massive models like mBART50 and mRASP2, depending on the size of data provided. We empirically analyse several key factors responsible for these - including context, many-to-many substitutions, code-switching language count etc. - and prove that they all contribute to enhanced pretraining of multilingual NMT models.

## introduction recent research in neural machine translation (nmt) has focused on the pretraining of massively multilingual models @xcite @xcite ) -due to their high scalability, easy deployability and state-of-theart (sota) performances @xcite . one of the most common pretraining approaches trains a model to reconstruct (or ``denoise") a sentence noised using one or more mechanisms. following masking @xcite (a) example sentences @xcite figure 1 : example of the errors induced by aligned augmentation (aa). gt refers to `ground truth' translations, as provided by native speakers. @xcite , synthetic code-switching foot_0 has emerged as a more effective noising mechanism @xcite . with the motive of moving the denoising task from language modeling to machine translation, these works propose to randomly code-switch input sentences, and then to train the mt model to denoise these sentences to the original monolingual ones. the most noteworthy system in this line of research is the massively multilingual model mrasp2 @xcite . it was pretrained using an algorithm called aligned augmentation (aa)that constructs a synthetic code-switched sentence using dictionary-based word-level translations, followed by contrastive learning to semantically align this sentence with the reference sentence. @xcite showed that aa enables mrasp2 to achieve sota results across varied high, medium, and lowresource language pairs, and verified that this was due to improved multilingual semantic representations and enhanced cross-lingual transfer. despite being highly effective, a common weakness of this family of approaches is that they synthesize code-switched sentences using word lexicons -opening up several potential quality issues in the pretraining data. for instance, the sota algorithm aa as well as other related works @xcite use non-contextual, one-to-one word translations obtained from muse lexicons @xcite --which can be problematic in a variety of cases. firstly, failing to factor sentence-level context can cause violations in linguistic agreement, such as gender, case, tense and verb agreement. secondly, aa cannot adequately handle contextual synonyms or polysemes (often estimated to constitute up to 80% of english words @xcite ), with @xcite and @xcite assigning random word senses (and thus, translations) for polysemes --regardless of context. thirdly, one-to-one word translations create further issues, including the handling of multi-word expressions (eg. ``get out") and multi-word entities (eg. ``new york") --with these problems aggravating for agglutinative languages. finally, the bilingual muse lexicons themselves have been shown to be of dubious quality across a variety of languages @xcite , and using these for multilingual codeswitching can propagate such errors manifold. this is illustrated in figure 1 , which shows an example of aa noising taken directly from @xcite . with 6 errors in a sentence of 14 words (refer appendix a.1 for a detailed examination of these errors), we contend that these limitations could cause significant corruption in the pretraining corpus. we, thus, hypothesize that although aa has been effective in a variety of scenarios, the raised issues could lead mrasp2 to underperform. for this reason, we propose contextual code-switching (ccs) -a novel approach for extracting contextual, many-to-many word translations, leveraging massive foot_1 nmt models, and then using these for noising the pretraining corpus. we conduct experiments on 3 different language families: romance, uralic, and indo-aryan, and report significant average improvements across the board, with gains of up to +5.5 spbleu. we also find that ccs models narrow the gap with or outperform massively multilingual models like mbart50 @xcite and mrasp2, despite using a tiny fraction of the data and compute. lastly, we conduct ablation studies to analyze some of the most important factors to consider when synthesizing code-switched text for multilingual nmt pretraining -which constitutes another key, novel contribution of this work. our major contributions are, thus, as follows: ## experiments in this section, we seek to answer the following research questions: as part of 3 case studies, we evaluate ccs on 3 different language families -namely, the highresourced romance, the agglutinative uralic, and the low-resourced indo-aryan -in order to test its efficacy under different scenarios. for each family, we train from scratch small multilingual aa and ccs baselines on its languages. small models have the benefit of minimizing negative interference, while also satisfying our resource constraints. ## conclusion we explore a noising mechanism called contextual code-switching (ccs) that extracts contextual, many-to-many word translations for code-switched pretraining in multilingual nmt. our experiments, conducted on 3 different language families, show that ccs consistently beats the previous sota approach, aligned augmentation and also performs comparably or better than mbart50 and mrasp2, based on the quantity of training data provided. we analyse the impact of some major factors responsible for enhancing code-switched pretraining through examples and ablation studies. we hope the findings of this work will be useful to researchers studying nmt pretraining, as well as to academic and industry peers who may be looking for a way to fruitfully leverage massive nmt models, or conversely, to jump-start the training of even larger and better-performing ones.