Transfer Learning for Related Languages: Submissions to the WMT 20 Similar Language Translation Task (2020)

Abstract: In this paper, we describe IIT Delhi’s submissions to the WMT 2020 task on Similar Language Translation for four language directions: Hindi <-> Marathi and Spanish <-> Portuguese. We try out three different model settings for the translation task and select our primary and contrastive submissions on the basis of performance of these three models. For our best submissions, we fine-tune the mBART model on the parallel data provided for the task. The pre-training is done using self-supervised objectives on a large amount of monolingual data for many languages. Overall, our models are ranked in the top four of all systems for the submitted language pairs, with first rank in Spanish -> Portuguese.

## introduction machine translation (mt) is currently tackled using rule-based methods (rbmt) @xcite , phrase-based statistical methods (smt) @xcite and neural methods (nmt) @xcite @xcite . nmt has achieved high translation quality for several language pairs @xcite , but this level of performance usually requires large amounts of aligned data in the order of millions of sentence pairs. for low and medium resource languages, smt performs better than nmt @xcite . smt also shows better performance when there is a domain mismatch between the train and test datasets, which is typical of low and medium resource language pairs. in these settings, nmt performance can be boosted by leveraging additional monolingual data to enforce various types of constraints or increasing the training data using back-translation. these methods can be particularly helpful if the source and target languages in mt are closely related and share language structure and alphabet. recently, pre-training methods for sequence-to-sequence (seq2seq) models have been introduced like mass @xcite , xlm @xcite , bart @xcite , and mbart @xcite . these methods show significant gains in downstream tasks like nmt, summarization, natural language inference (nli), etc. in this paper, we focus on the transfer learning capabilities in nmt for the task of translation between related languages where parallel data is scarce. iit delhi participated in the wmt 2020 shared task on similar language translation for four language directions: hindi (hi) ↔ marathi (mr) and spanish (es) ↔ portuguese (pt). the first language pair is low resource and second is medium resource in terms of the parallel data available for the task. refer to we fine-tuned the pre-trained mbart model @xcite on the parallel data provided for the task. mbart gives better performance than smt models even when the parallel data is very limited. mbart is pre-trained on 25 languages, which contain hindi and spanish, but not marathi and portuguese. mbart is able to leverage transfer learning capabilities even for those languages that are originally not present during the pre-training phase. the fine-tuned mbart architecture forms our best submissions for both language pairs: hi ↔ mr and es ↔ pt. the rankings obtained by us in each of the language directions are listed in the rest of the paper is organized as follows: section 2 provides the background and related work for low/medium resource nmt. section 3 gives an direction bleu rank @xmath0.14 4 @xmath1.53 2 @xmath2.69 1 @xmath3.84 2 the results and analysis are detailed in section 5. we finally conclude in section 6. ## background smt is tackled by building a phrase table from the aligned parallel data. the target side translation is then generated by matching the most appropriate phrases in the source sentence conditioned on the target side language model along with a reordering model @xcite . nmt is modeled using encoder-decoder models @xcite @xcite , with the transformer model @xcite achieving state-of-the-art on many mt problems. but these models' reliance on large aligned parallel data for the source and target languages makes them unsuitable for low/medium resource language pairs @xcite . some of the previous works in these settings to improve nmt performance are described below: ## system overview we experimented with three different settings for hi ↔ mr as listed below. smt this phrase-based system leverages both monolingual and parallel data provided for the task. we use moses @xcite for training the smt systems. nmt (transformer) for this, we used the standard transformer large architecture from @xcite for training on the parallel data provided for the task. nmt (mbart) mbart @xcite is a large transformer pre-trained on monolingual data for 25 languages. the pre-training objective for mbart is seq2seq de-noising for natural text as in bart @xcite . mbart provides a general-purpose pre-trained transformer for any downstream task. it has been shown to give significant improvements over the random initialization for nmt and is the current state-of-the-art for many low resource language pairs. implementation details mbart uses a shared subword vocabulary of 250k tokens for all the 25 languages present in the pre-training. we use the same vocabulary for marathi and portuguese also, even though they were not used during the pre- additionally, the mbart architecture requires language specific token at the end of each input sequence to provide the language specific context for the decoder. since marathi and portuguese were not present during the pre-training phase, we use the token corresponding to the second most related language present in mbart pre-training for specifying the context at the time of decoding in each case. for marathi, we used the nepali language token and for portuguese, we used the italian language token. we could not use spanish language token for portuguese because we are doing translations to and from spanish. train valid test hi ↔ mr 43,274 1,411 1,941 es ↔ pt 3,472,860 1,283 1,495 ## experiments we use hi ↔ mr and es ↔ pt language pairs for our experiments. ## results and analysis results analysis even though marathi and portuguese are not present during the pre-training phase of mbart, fine-tuning on these languages provides significant boosts over smt and standard transformer. this shows that some level of language independent multilingual embeddings are present in the pre-trained model weights which can be exploited for the transfer task. ## discussion and conclusion we have participated in the similar language translation task on four language directions. we have shown that pre-trained models can help in low and medium resource nmt. our best system uses the pre-trained mbart model @xcite and fine-tunes on the parallel data provided for the specific translation task. our results demonstrate that pre-training can help even when the language used for fine-tuning is not present during pre-training. one direction of future work is to add linguistic information during the pre-training phase to get more fluent translations. when this information is not available directly (especially for low resource languages), pre-training on a related high resource language with syntax information can help low resource languages also.