Overview and Results of M ix MT Shared-Task at WMT 2022 (2022)

Abstract: In this paper, we present an overview of the WMT 2022 shared task on code-mixed machine translation (MixMT). In this shared task, we hosted two code-mixed machine translation subtasks in the following settings: (i) monolingual to code-mixed translation and (ii) code-mixed to monolingual translation. In both the subtasks, we received registration and participation from teams across the globe showing an interest and need to immediately address the challenges with machine translation involving code-mixed and low-resource languages.

## introduction code-mixing (or code-switching) is an interesting manifestation of multilingualism in communities across the globe. lately, we observe an uptick in the interest and efforts of the computational linguistic community to solve a multitude of challenges with code-mixed languages. several interesting resources and computational models have been proposed for problems such as language identification @xcite ), text generation @xcite , and sentiment analysis @xcite . machine translation which is an active area of research and development for monolingual languages is at the outset for code-mixed languages @xcite . in this shared task, we aim to explore the machine translation task involving a popular code-mixed language i.e., hinglish (code-mixing of hindi and english). through both subtasks, we aim to address the challenges in building a realworld multilingual translation system involving code-mixed language as the source/target. similar to the recent events on code-mixed languages @xcite @xcite , the mixmt shared task has received participation and engagement with teams from across the globe. in total, we received registration from 38 teams. throughout the competition, seven teams actively participated and submitted their system for the development phase, test phase, and human evaluation phase. ## submissions we received the submissions from seven teams (listed alphabetically): 1. cnlp-nits-pp (laskar et al., 2022): they leverage the external parallel corpus (samanantar @xcite ) to train their translation model which is built using opennmtpy framework @xcite with the default setting. to generate the synthetic dataset, they transliterate and align the words in parallel sentences. finally, they augment the provided dataset with the synthetic dataset to train their model. 2. domain curricula for code-switched mt (dc) @xcite : they ex-periment with different combinations of pretraining fine-tuning setups. they leverage the synthetic code-mixed dataset generated using the iit-b parallel corpus @xcite and matrix language theory (myers-scotton). further, the mixed data pretraining with synthetic and task-specific data shows the best result on the evaluation dataset. to build the translation model, they use transformers @xcite and fairseq toolkit @xcite . 3. gui @xcite : they leverage the multilingual pre-trained models to build their translation system. for m2cm task, they finetune multilingual-bart @xcite on the task-specific data with reduced vocabulary. they reduce the vocabulary using the tokens present in the task dataset, iit-b parallel corpus @xcite , and the dakshina dataset @xcite . they also perform the post-processing on the output generated from the fine-tuned model. for cm2m task, they finetune salesken.ai's pre-trained model provided on huggingface transformers which is a finetuned helsinki's opus-mt model on ai4bharat's samanantar dataset @xcite . 4. mucs @xcite : their translation model for both the task is built around transliteration @xcite and fine-tuning the indictrans pre-trained model @xcite . they generate synthetic parallel data using the samanantar corpus @xcite . they further fine-tune the indictrans model jointly with the synthetic and task-specific datasets. 5. nict @xcite : they propose a synthetic code-mixed data based pre-training and a multi-way fine-tuning strategy. to generate the synthetic dataset, they leverage the samanantar corpus @xcite , the transliteration toolkit foot_0 , and a min-max based approach for word alignment @xcite . they pre-train a multilingual model on the synthetic hinglish-english and english-synthetic hinglish dataset. to perform the multi-way fine-tuning, they fine-tune the pre-trained model on hinglish to english and english to hinglish jointly using a small subset of the english side of the synthetic data and the entire parallel corpus (phinc & hinge) together. they use a denoising strategy similar to bart @xcite by randomly masking english words in the source sentence. they use the yanmtt toolkit @xcite to their translation model. 6. sit-nmt (khan et al., 2022): they experiment with a variety of multilingual pre-trained models such as multilingual bart @xcite and multilingual t5 @xcite . they fine-tune these pre-trained models on external datasets. for m2cm task, they use kaggle hi-en @xcite and muse hi-en dictionary @xcite . for cm2m task, they use cmu movie reviews data @xcite and calcs'21 dataset @xcite . they also use selected wmt'14 news hi-en sentences @xcite and the mtnt fr-en and ja-en data @xcite . in addition, they also increase the size of the dataset by back-translating samples of the english side of tatoeba spanish dataset to the english (project, 2022) and sentiment140 dataset @xcite into hinglish using google translate. further, to enhance the model's performance, they perform the validation tuning on the task-specific validation dataset and use a multi-run ensemble @xcite to combine multiple model's best checkpoints. 7. uedin @xcite : their submission focuses on data generation using backtranslation from monolingual resources. for m2cm subtask, they explore the impact of constrained and unconstrained decoding strategies. they use the samanantar corpus @xcite as an external resource for backtranslation. for cm2m subtask, they explore several pretraining techniques, ranging from simple initialization from existing machine translation models to aligned augmentation @xcite which is a denoising-based pretraining technique. ## results and analysis in this section, we present the results from automatic and human-based evaluation of the submissions from the seven teams. as discussed in section 2.3, we use rouge-l f1 score (r-l) and word error rate (wer) for automatic evaluation. r-l score can vary from 0 to 1 whereas wer can take a value greater than or equal to 0. a high r-l score and a low wer score are preferred. on both subtasks followed by uedin. gui stood at the third position for m2cm task whereas mucs is ranked third for cm2m subtask. interestingly, mucs and nict get a consistent one score showing poor quality output consisting of lexical and grammatical mistakes. it further highlights the inefficacy of evaluation metrics for code-mixed natural language generation tasks as pointed out in several previous works @xcite . further, we analyze the submissions based on the dataset and the models used in the experiment. in section 2.4, we have highlighted the two criteria for the submission to be considered as constrained. in we observe that almost all the teams have used at least one external dataset for both subtasks with gui's submission for cm2m subtask being the only exception. we attribute this behavior to the fact we designed both subtasks in a low-resource setting. the submissions by four teams (i.e., cnlp, dc, nict, and uedin) are completely unconstrained for both subtasks as they are using an external dataset and training their own system from scratch. ## discussion in this paper, we present the findings from the mixmt shared task. we hosted two subtasks involving a code-mixed language i.e. hinglish. given the low-resource nature of the code-mixed languages (and the subtasks), the majority of the submissions rely on data augmentation either synthetically or from other external sources. the lack of dedicated pre-trained models for code-mixed languages pushed the teams to explore the available alternatives along with bold attempts to train the models from scratch. we posit several open challenges with code-mixed machine translation such as creating large-scale parallel data, efficient data augmentation strategies, and robust evaluation measures. the insights and findings from this task will be useful to future works on machine translation involving code-mixed and low-resource languages. they will broaden the horizon for works on multilingual machine translation.