SIT at M ix MT 2022: Fluent Translation Built on Giant Pre-trained Models (2022)

Abstract: This paper describes the Stevens Institute of Technology’s submission for the WMT 2022 Shared Task: Code-mixed Machine Translation (MixMT). The task consisted of two subtasks, subtask 1 Hindi/English to Hinglish and subtask 2 Hinglish to English translation. Our findings lie in the improvements made through the use of large pre-trained multilingual NMT models and in-domain datasets, as well as back-translation and ensemble techniques. The translation output is automatically evaluated against the reference translations using ROUGE-L and WER. Our system achieves the 1st position on subtask 2 according to ROUGE-L, WER, and human evaluation, 1st position on subtask 1 according to WER and human evaluation, and 3rd position on subtask 1 with respect to ROUGE-L metric.

## introduction code-mixing (or code-switching) is the phenomenon when another language like hindi is interleaved with english words in the same sentence. this code-mixed language is mostly used in social media text and is colloquially referred to as hinglish. despite hindi being the fourth most widely spoken language in the world @xcite , research in hinglish translation has been a relatively unexplored task. a major challenge in creating a translation system for code-mixed text is the limited amount of parallel data @xcite . typical methods use standard back-translation techniques @xcite for generating synthetic parallel data for training. massive multilingual neural machine translation (nmt) models have recently been shown to improve the translation performances for low-resource and even zero-shot settings. we propose using such large multilin-gual nmt models for our code-mixed translation tasks. previous work has only used smaller multilingual architectures @xcite . we use pre-trained multilingual models trained in up to 200 language directions. we finetune these models for the hindi to hinglish and hinglish to english tasks. one major challenge when using these massive models is the gpu memory constraint. another issue is the ratio of english and hinglish words interleaved for each translation output. we use multiple state-of-the-art gpus with model parallelization to overcome the memory issue. for the amount of english in the outputs, we tune the model parameters including learning rate, dropout, and the number of epochs to get the optimal translations. along with these pre-trained multilingual nmt models, we also use additional indomain data, back-translation to generate additional parallel data, and using multi-run ensemble to improve the final performance. all these methods gave us an improvement of +24.4 bleu for hindi to hinglish translation (subtask 1) and +28.1 bleu points for hinglish to english translation (subtask 2) compared to using only the organizer provided data and the baseline experiment. in this paper, we discuss our submission for the wmt 2022 mixmt shared task. we participate in both the subtasks and our submission system includes the following: multilingual neural machine translation (mnmt) word and subword-level tokenizations are widely used in natural language processing, including nmt/mnmt. morishita et al. (2018) propose to incorporate hierarchical subword features to improve neural machine translation. massively multilingual nmt models are proposed by aharoni et al. (2019) and arivazhagan et al. (2019). they are trained on a large number of language pairs and show a strong and positive impact on low-resource languages. however, these models tend to have representation bottlenecks (dabre et al., 2020), due to the large vocabulary size and the large diversity of training languages. two mnmt systems (tan et al., 2019; xiong et al., 2021) are proposed to solve this problem by modifying the model architectures, adding special constraints on training, or designing more complicated preprocessing methods. xiong et al. (2021) adopt the contrastive learning scheme in many-to-many mnmt. tan et al. (2019) propose a distillation-based approach to boost the accuracy of mnmt systems. however, these word/subword-based models still need complex preprocessing steps such as data augmentation or special model architecture design. code-mixed nmt the majority of research for code-mixed translation focuses on generating additional data using backtranslation methods. winata et al. (2019) used the sequence to sequence models to generate such data and garg et al. (2018) used a recurrent neural network along with a sequence generative adversarial network. pratapa et al. (2018) generated linguistically-motivated sequences. additionally, there have been several code-mixed workshops (bhat et al., 2017; aguilar et al., 2018) to advance the field of code-mixed data. hindi or hinglish nmt researchers have worked on machine translation from hindi to english (laskar et al., 2019; goyal and sharma, 2019), however, there has been far less work for hinglish. a major issue is the lack of parallel hinglish-english data. additional parallel data generated by back-translation is used to improve the performance (gau @xcite . the calcs'21 competition @xcite had a shared task for english to hinglish for movie review data. ## methods for the initial set of experiments, we use the baseline transformer model @xcite . for all the other experiments, we use pre-trained multilingual nmt models and finetuned them for the specific datasets. we can divide these into three groups based on the number of parameters. (1) smaller models including mbart-50 @xcite and facebook m2m-100 medium model @xcite (m2m-100), (2) the medium models include the facebook @xcite and google mt5 xl @xcite (mt5-xl), and (3) for large model we use the google mt5 xxl model @xcite (mt5-xxl). the parameter count for each of the models and the training time per epoch for baseline datasets are mentioned in for both subtasks, we use hindi as the source language tag and english as the target language tag. ## datasets the competition provided one dataset for each of the subtasks, hinge hi-hg @xcite for subtask 1 and phinc hg-en (srivastava and singh, 2020) for subtask 2. the competition also provided the validation data. in addition to these, we also use additional in-domain and out-of-domain datasets. due to a large overlap of english and hinglish vocabulary, we use hindi-english (hi-en) and hindi-hinglish (hi-hg) datasets for subtask 1. for subtask 2, we use various hinglish-english datasets. all the competition provided datasets, the additional in-domain datasets, and the additional out-of-domain datasets used for both the subtasks are listed in to preprocess the data, we tokenize using the moses tokenizer @xcite or the model-specific tokenizer provided by huggingface. additionally, we apply either byte pair encoding (bpe) @xcite for the baseline transformer model and sentence piece @xcite for all other models including mbart-50, m2m-100, nllb-200, mt5-xl and mt5-xxl to split words into subwords tokens. ## experiments this section describes the experimental details, including the toolkits, the parameter settings for the model training and decoding, and the results. ## results we evaluate the models with respect to the bleu score using sacrebleu. ## analysis we analyze the translation outputs of nllb, mt5-xl, and mt5-xxl models. for subtask 1, the issues we faced were that the sentences were translated entirely to english and did not contain any hinglish words. some words were translated partially to hinglish, and a portion of the words remained in the hindi language. for subtask 2, the issues we faced were that the names of animal species were not translated correctly. and idioms lose their meaning in translation. examples of these issues are shown in and significantly enhance our translation quality from 1.2 to 25.6 and 4.5 to 32.6 for subtasks 1 and 2 respectively. additionally, we also apply data-augmentation techniques including back-translation, tuning on in-domain data, and checkpoint ensemble. our system got the 1 st position in subtask 2 for both rouge-l and wer metrics, the 1 st position in subtask 1 for wer, and 3 rd position in subtask 1 for rouge-l.