NICT -5’s Submission To WAT 2021: MBART Pre-training And In-Domain Fine Tuning For Indic Languages (2021)

Abstract: In this paper we describe our submission to the multilingual Indic language translation wtask “MultiIndicMT” under the team name “NICT-5”. This task involves translation from 10 Indic languages into English and vice-versa. The objective of the task was to explore the utility of multilingual approaches using a variety of in-domain and out-of-domain parallel and monolingual corpora. Given the recent success of multilingual NMT pre-training we decided to explore pre-training an MBART model on a large monolingual corpus collection covering all languages in this task followed by multilingual fine-tuning on small in-domain corpora. Firstly, we observed that a small amount of pre-training followed by fine-tuning on small bilingual corpora can yield large gains over when pre-training is not used. Furthermore, multilingual fine-tuning leads to further gains in translation quality which significantly outperforms a very strong multilingual baseline that does not rely on any pre-training.

## introduction neural machine translation (nmt) @xcite is known to give state-of-the-art translations for a variety of language pairs. nmt is known to perform poorly for language pairs for which parallel corpora are scarce. this happens due to lack of translation knowledge as well as due to overfitting which is inevitable in a low-resource setting. fortunately, transfer learning via cross-lingual transfer @xcite , multilingualism @xcite , back-translation @xcite or monolingual pre-training @xcite @xcite can significantly improve translation quality in a low-resource situation. cross-lingual transfer learning involves pretraining a model using a parallel corpus for a resource-rich language pair xx -y y and then fine-tuning on a parallel corpus for a resource-poor language pair aa -bb. naturally the improvements in translation quality will be impacted by if @xmath0. cross-lingual transfer despite its simplicity and effectiveness relies on shared source or target languages for effective transfer and thus depending on methods that use monolingual corpora are preferable. this also applies to vanilla multilingual training which does not rely on monolingual corpora. another reason for focusing on utilizing monolingual corpora is that they are extremely abundant when compared to parallel corpora and they contain a large amount of language modeling information. in this regard, back-translation and multilingual pre-training are two of the most reliable methods. while back-translation is easy to use, it involves the translation of millions of monolingual sentences and quite often it is necessary to perform multiple iterations of the back-translation process to yield the best results @xcite which means that it is quite resource intensive. this leaves us with multilingual pre-training using methods such as bart/mbart @xcite which we use for developing our translation system. the advantage of bart/mbart is that we need to pre-train these models once and then fine-tune not only for machine translation but also for any natural language generation task such as summarization @xcite . these models can be upgraded to include additional language pairs in the future by simply resuming pre-training @xcite . in this paper, we describe our simple approach involving mbart pre-training and fine-tuning. first, we use the official monolingual corpora to train an mbart model spanning all 11 languages in the shared task. following this we fine-tune the mbart model using the officially provided indomain corpora in two different ways: bilingual fine-tuning and multilingual fine-tuning. additionally we also train multilingual models without any pre-training. the multilingual models are one-tomany (english to indic) and many-to-one (indic to english) in nature. the bilingual fine-tuning and non pre-trained multilingual model serve as strong baselines which significantly outperform the organizers weak bilingual baselines. our multilingual fine-tuning models exhibit the best translation quality out of all our models which shows the power of effectively combining monolingual corpora with multilingualism. we refer readers to the workshop overview paper @xcite for a better understanding of the task and the comparison of our results with those of other participants. ## related work the techniques used in this paper revolve around multilingualism, sequence-to-sequence pretraining and transfer learning. @xcite proposed multilingual neural translation using multiple encoders and decoders which was then simplified by @xcite to require a single encoder and decoder to be shared among multiple language pairs. due to the simplicity of the latter approach, most modern multilingual models are based on it and in this paper we also use the same approach. multilingualism involves implicit transfer learning but a more explicit way to do the same is to use fine-tuning @xcite . however all these aforementioned approaches rely on bilingual data which is not always readily available. this can be remedied by the use of monolingual corpora for backtranslation @xcite or for pre-training @xcite @xcite . as backtranslation is resource intensive, given that it involves translation of a large amount of monolingual corpora, pre-training is more attractive as a pre-trained model can be used for a variety of natural language generation tasks. in this paper we combine sequence-to-sequence pre-training followed by multilingual fine-tuning. for an overview of multilingual nmt we refer readers to a survey paper on multilingualism and low-resource nmt in general @xcite . ## our approaches for our submissions we focused on combining multilingual denoising pre-training (mbart) and multilingual fine tuning. ## experimental setup our goal was to study how far the translation quality can be pushed via mbart pre-training and multilingual fine-tuning. to do so, we describe the datasets, implementation details, evaluation metrics and the models trained. ## results and observations ## conclusion in this paper we have described our nmt systems and results for the multiindicmt task in wat 2021. we worked on mbart pre-training and multilingual fine-tuning which we found to significantly outperform unidirectional models with and without pre-training and multilingual models without pre-training. we did not train our mbart models for more than 1 epoch and used only the pmi data for fine-tuning instead of the whole parallel corpus. we did not try any additional methods such as back-translation either. despite this, our results are competitive and despite the simplicity of our methods our results do not lag far behind those of the best systems that use advanced methods such as data selection, domain adaptation, back-translation etc. this also means that we have a lot of room for improvement in the future.