Domain Curricula for Code-Switched MT at M ix MT 2022 (2022)

Abstract: In multilingual colloquial settings, it is a habitual occurrence to compose expressions of text or speech containing tokens or phrases of different languages, a phenomenon popularly known as code-switching or code-mixing (CMX). We present our approach and results for the Code-mixed Machine Translation (MixMT) shared task at WMT 2022: the task consists of two subtasks, monolingual to code-mixed machine translation (Subtask-1) and code-mixed to monolingual machine translation (Subtask-2). Most non-synthetic code-mixed data are from social media but gathering a significant amount of this kind of data would be laborious and this form of data has more writing variation than other domains, so for both subtasks, we experimented with data schedules for out-of-domain data. We jointly learn multiple domains of text by pretraining and fine-tuning, combined with a sentence alignment objective. We found that switching between domains caused improved performance in the domains seen earliest during training, but depleted the performance on the remaining domains. A continuous training run with strategically dispensed data of different domains showed a significantly improved performance over fine-tuning.

## introduction code-mixing (cmx) denotes the alternation of two languages within a single utterance @xcite . code-mixing occurs mostly in unofficial groups in multilingual environments. more than 77% of asians are multilingual @xcite , and other statistics estimate that 64.5% of europeans speak more than two languages, with more than 80% of adults in the region being bilingual @xcite . codemixing happens far more often in conversations than in writing, and mostly in unofficial settings, hence it rarely occurs in documented settings. this makes substantial data gathering for computational approaches to translations of code-mixed language difficult. parallel corpora for code-switched data is very scarce @xcite , this is because code-mixing mostly occurs in unofficial conversations like social media interactions. contemporary neural machine translation (nmt) mostly makes use of parametric sequenceto-sequence models @xcite , where an encoder receives a source sentence and outputs a set of hidden states, the decoder then scrutinizes these hidden states at each step, and outputs a sequence of softmax distribution over the target vocabulary space. considering that we would need vast quantities of data to train an adequate nmt for this task, we leverage large-scale synthetic and available small data and notably rank data on domain relevance, by finetuning with it, initiating training with the relevant domain and strategically placing it at the premier batches of the training data. essentially, the characteristics of the data an nmt model is trained on are paramount to its translation quality, in particular in terms of size and domain. it is quintessential to train nmt models based on the domain relevance of corpora. since most code-mixing occurs in unofficial communication, it is costly to find a lot of labeled data for every domain we are interested in. hence we attempt to find less expensive exigencies to supplement training data, pretrain on largely available data of different domains, strategically construct synthetic data, and apportion data to make up for missing domains. in these wmt subtasks -monolingual to code-mixed machine translation (subtask-1) and code-mixed to monolingual machine translation (subtask-2), we also fine-tune on different domains, align representations of data and find the best combination of approaches to solving the subtasks. the main intuition behind our proposed solution is that nmt models exhibit a significant translation correlation when trained on data from the same or similar domains. with different data domain re-1 1162 quirements, it performs better when trained with data of the most relevant domain as preliminary batches compared to finetuning. as most natural code-mixed data source is social media and it is difficult to gather a good amount to train a model, it is incumbent to find a strategy that makes the model prioritize this form of data above others. accordingly, we attempt to find less expensive techniques to supplement training data, pretrain on largely available data of different domains, strategically construct synthetic data, and apportion data to make up for missing domains. our result showed improved performance on innate code-mixed data (and non-synthetic wmt test set samples) when this was prioritized and performed strongly in a test with a mix of several other data sources. we observed a better performance with domain-specific evaluation upon finetuning but this intensely plummeted performance on other 'pretraining domains', and more balanced performance on passing the interesting domain in the preliminary batches in a single 'all domain' training. ## related work it is laborious to obtain 'one-fits-all' training data for nmt. most publicly available parallel corpora like tanzil, opus, unpc are sourced from documented communication, and these are often domain-specific. in nmt, data selection e.g. @xcite has remained as an underlying and important research concern. choosing training examples that are relevant to the target domain, or by choosing high-quality examples for data cleaning (also known as denoising), has been essential in domain adaptation. building a large-scale multidomain nmt model that excels on several domains simultaneously becomes both technically difficult and practically back-breaking. addressing research problems such as catastrophic forgetting @xcite , data balancing @xcite , adapters @xcite have shown improvement. unfortunately, several domains are difficult to handle with the single-domain data selection techniques currently in use. for instance, improving translation quality of one domain will often hurt that of another @xcite . @xcite replaced phrases with prespecified translation to perform "soft" constraint decoding. @xcite generated codemixed data from regular parallel texts and showed this training strategy yields mt systems that surpass multilingual systems for code-mixed texts. considering that code-mixed text belongs in less documented domains than most, there may be a need for domain adaptation used on sufficiently available data domains. our work is inspired by the following approaches: @xcite executed simultaneous data selection across several domains by gradually focusing on multi-domain relevant and noise-reduced data batches while carefully introducing instance-level domain-relevance features and automatically constructing a training curriculum. @xcite demonstrated that instance-level features are better able to distinguish between different domains compared to corpuslevel attributes. @xcite proposed modeling the difference between domains instead of smoothing over domains for machine translation. @xcite showed that an encoder alignment objective is beneficial for code-mixed translation, in addition to @xcite that proposed auxiliary losses on the nmt encoder that imposed representational invariance across languages for multilingual machine translation. ## data in since we need to augment provided data for a reasonable quantity to train a nmt model, we generated synthetic code-mixed data from the iitb english code-mixed (cmx) overhead charge is a percentage of the direct costs of providing the services under the contract. overhead charge, anubandh ke anusaar pradatt sevaon kee pratyaksh laagat ka ek pratishat hota hai. a strategy of ignoring potential problems on the basis that they may be exceedingly rare. us aadhaar par sambhaavit problems ko anadekha karane kee ek yukti, jahaan ki ve ati dushpraapy ho sakate hain. a standard of measurement, or a unit that can be studied separately / independently. koee maapadand athava koee a unit that svatantr roop se/alag se adhyayan kiya ja sakata ho. for the pretraining-finetuning setup, we pretrain with synthetic code-switched data generated from iitb corpus and fine-tune on the wmt data provided for each task. for both pretraining and finetuning, we coordinate the data similar to @xcite -for subtask-1, monolingual to codemixed machine translation subtask, we use the hindi sentence (devanagari script) as source sequence and the corresponding code-switched sentence (roman script) as target, then alternated the english sequence (roman script) as source sentence and the same corresponding code-switched sentence as the target sequence. the above two source-target parallel data are set after each other. for subtask-2, code-mixed to monolingual machine translation subtask, we have a similar arrangement as in subtask-1, but with the source sequences of hindi and code-mixing (hinglish) in roman script and as the target the corresponding english sequence. we removed sequences shorter than 2 tokens, and those longer than 250 tokens, and a target-to-source token ratio of more than 1.5. after cleaning the pretraining data, for subtask-1, we have about 2.5m parallel sentences and 2.3m parallel sentences for subtask-2. for the finetuning process, we made use of the wmt training data provided for each subtask and organized like the pretraining data as described above. after cleaning, for subtask-1 (synthetic + human-generated), we have a total of over 11k parallel sentences. for subtask-2, over 12k parallel sentences remain after cleaning. since the iitb corpus encompasses multiple sources and domains where code-mixing infrequently occurs, we decided to configure our model in a way it first learns from natural code-mixed data provided by wmt. we experiment with a hand-designed curriculum of the synthetic codeswitched data generated from the iitb corpus and the wmt provided data. we supply the model the non-synthetic wmt data only in the first few batches in the hope that this would faintly familiarize the model with domain-specific features before it learns from the synthetic code-switched data we generated from other domains. we compare the results of this approach to the above described pretraining-finetuning setup. all data is tokenized and normalized using sentencepiece foot_0 . ## training objective considering the effectiveness of clean finetuning @xcite , and pre-training @xcite , we attempt a combined pipeline of pretrain-ing+finetuning experiment and also a single training but with tactical positioning of the most important domain. in the finetuning process and training with specially ordered data, as recommended by @xcite , we add an alignment loss to the encoder to encourage source and target representations to be close in representation space minimizing the max-pooled cosine distance of the encoder representation as shown in equation 1: ## experiments and results in all of our experiments, we used transformer-base @xcite configuration with the fairseq @xcite framework. all models were trained on four tesla t400 gpus using iitb and wmt-'22 mixmt data for training as described in section 3, with a shared vocabulary of 77k bpe @xcite sub-words to create a joint vocabulary for both tasks and all models. the model's hyperparameters can be found in appendix a. ## conclusion we present a data domain sorting method that improves translation performance based on a target domain for the wmt 2022 code-switching shared tasks. we compared our result to a pretraining and fine-tuning pipeline, and demonstrated that the finetuning method improves on specified domain but upsets on previously learned data domain. an aspect we intend to delve further into is efficient domain adaptation strategies that may help lowresource domains such as code-mixing, and have little or no effect on high-resource domains, we are currently looking into domain adaptation learning curve @xcite , extraction of domainspecific parameters @xcite for better data augmentation strategies, better acquisition of code-mixed data, and the use of adapters @xcite .