Rakuten’s Participation in WAT 2021: Examining the Effectiveness of Pre-trained Models for Multilingual and Multimodal Machine Translation (2021)

Abstract: This paper introduces our neural machine translation systems’ participation in the WAT 2021 shared translation tasks (team ID: sakura). We participated in the (i) NICT-SAP, (ii) Japanese-English multimodal translation, (iii) Multilingual Indic, and (iv) Myanmar-English translation tasks. Multilingual approaches such as mBART (Liu et al., 2020) are capable of pre-training a complete, multilingual sequence-to-sequence model through denoising objectives, making it a great starting point for building multilingual translation systems. Our main focus in this work is to investigate the effectiveness of multilingual finetuning on such a multilingual language model on various translation tasks, including low-resource, multimodal, and mixed-domain translation. We further explore a multimodal approach based on universal visual representation (Zhang et al., 2019) and compare its performance against a unimodal approach based on mBART alone.

## introduction this paper introduces our neural machine translation (nmt) systems' participation in the 8th workshop on asian translation (wat-2021) shared translation tasks @xcite . we participated in the (i) nict-sap's it and wikinews, (ii) japanese-english multimodal translation, (iii) multilingual indic, and (iv) myanmar-english translation tasks. recent advances in language model pre-training have been successful in advancing the state-of-theart in various natural language processing tasks. multilingual approaches such as mbart @xcite are capable of pre-training a full sequenceto-sequence model through multilingual denoising objectives, which leads to significant gains in downstream tasks, such as machine translation. building upon our success with utilizing mbart25 in the 2020 edition of wat @xcite , we put more focus on multilingual and multimodal translation this year. in particular, instead of performing bilingual finetuning on mbart for each language pair, we train a single, multilingual nmt model that is capable of translating multiple languages at once. as first proposed by @xcite , we apply multilingual finetuning to mbart50 for the nict-sap task (involving 4 asian languages) and multilingual indic task (involving 10 indic languages). our findings show the remarkable effectiveness of mbart pre-training on these tasks. on the japanese-english multimodal translation task, we compare a unimodal text-based model, which is initialized based on mbart, with a multimodal approach based on universal visual representation (uvr) @xcite . last, we continue our work on myanmar-english translation by experimenting with more extensive data augmentation approaches. our main findings for each task are summarized in the following: ## conclusion we presented our submissions (team id: sakura) to the wat 2021 shared translation tasks in this paper. we showed the remarkable effectiveness of pre-trained models in improving multilingual and multimodal neural machine translation. on multilingual translation, models initialized with mbart50 achieved substantial performance gains on both nict-sap and multilingual indic tasks. on multimodal translation, a text-only model with mbart25 pre-training improves upon an mnmt model based on uvr. finally, we extended our data augmentation approaches on the myanmar-english translation tasks and obtained further improvements.