NICT ‘s Submission To WAT 2020: How Effective Are Simple Many-To-Many Neural Machine Translation Models? (2020)

Abstract: In this paper we describe our team‘s (NICT-5) Neural Machine Translation (NMT) models whose translations were submitted to shared tasks of the 7th Workshop on Asian Translation. We participated in the Indic language multilingual sub-task as well as the NICT-SAP multilingual multi-domain sub-task. We focused on naive many-to-many NMT models which gave reasonable translation quality despite their simplicity. Our observations are twofold: (a.) Many-to-many models suffer from a lack of consistency where the translation quality for some language pairs is very good but for some others it is terrible when compared against one-to-many and many-to-one baselines. (b.) Oversampling smaller corpora does not necessarily give the best translation quality for the language pair associated with that pair.

## introduction neural machine translation (nmt) @xcite is an end-to-end machine translation (mt) modeling approach that is known to give state-of-the-art translations for a variety of language pairs. although, it is known to work particularly well for language pairs with an abundance of parallel corpora it tends to perform rather poorly for language pairs that lack large parallel corpora. fortunately, multilingual neural machine translation (mnmt) @xcite methods can be applied in order to significantly improve the translation quality for such language pairs. the underlying reason for improvement is that sharing parameters among several language pairs enables transfer learning which is proven to improve translation quality regardless of the language pair. for the 7 th workshop on asian translation (wat, 2020), our team (nict-5) decided to focus on the indic languages task and the nict-sap task. both tasks showcase resource poor indic and south-east asian languages and thus multilingual nmt solutions can be applied to great effect in these tasks. it is common to train one-to-many or many-to-one nmt models for multilingual tasks but many-to-many models are often not showcased. this sparked out curiosity and we decided to investigate how well a many-to-many model would perform in the case of the two sub-tasks we chose. the many-to-many models we trained used the transformer @xcite architecture using the simple black-box token-prepending technique @xcite . in essence we simply concatenated the individual parallel corpora while prepending an artificial token such as 2xx where xx indicates the target language. typically, the smallest corpus in the multilingual dataset is oversampled to match the size of the largest one but we tried settings with and without oversampling. furthermore, following @xcite we additionally prepended source sentences with tokens such as 2dom where dom indicates the domain of the corpus. we only did this when we knew that the test (and train) sets would involve multiple domains. an evaluation of our models showed that their performance is not consistent because they sometimes outperform the one-to-many and manyto-one models and sometimes underperform them. furthermore the performance also depends on the language pair. as a secondary observation, we noticed that when parallel corpora sizes are not too different, oversampling smaller corpora negatively affects the final translation quality. we hope that our many-to-many models will serve as baselines which can be significantly improved upon in the future. although there are many-to-one and oneto-many models that may be better, many-to-many models have zero-shot translation @xcite capabilities and thus should be focused on in the interest of a one-for-all nmt model. neural machine translation (nmt) @xcite ) is now the defacto machine translation paradigm that is used in research as well as engineering applications. while the initial architectures used recurrent neural networks, the more recent architectures use self-attention and feed-forward networks @xcite which enable faster training and decoding. the main advantage of nmt is that the translation models are small (can fit on low-memory and low-computation devices) and the training approach is end-to-end (rather than modular). large translation models and modular (multiple components) design were the key features of the predecessor of nmt aka statistical machine translation (smt) @xcite which involved error compounding as the input sentences were processed by multiple components that were prone to making mistakes. another advantage of nmt is that its inner working is non-symbolic which enables it to incorporate multiple languages without any need to modify the basic architecture. while we use the multilingual nmt approach proposed by @xcite for multilingualism and the derivative multidomain nmt approach by @xcite we refer readers to @xcite and @xcite for overviews on multilingualism and domain-adaptation, respectively. we do not describe the multilingual or multi-domain nmt modeling techniques in this paper as they are the same as described in @xcite and @xcite . specific to wat, multilingual multi-domain approaches have been shown to improve translation quality for low-resource languages @xcite @xcite . ## experiments in this section we describe the tasks, datasets, implementation details, evaluation methodology and actual models trained. ## results tables 3 and 4 contain results for the nict-sap and indic tasks for translation to and from english. we primarily report bleu scores for our translations and mark scores that are either better than or not better than (check captions) the organizers translations. we used the same data as the organizers did. the organizers trained one-to-many or many-to-one models whereas we only trained many-to-many-models. for other scores such as ribes, jpo adequacy and am-fm scores kindly check the workshop overview paper @xcite or the evaluation page foot_7 as applicable. ## conclusion in this paper we have described our submissions to the nict-sap and indic multilingual tasks in wat 2020. in general our many-to-many models have mixed performance where they sometimes outperform one-to-many or many-to-one models (organzier's models) and sometimes do not. furthermore we observed that our models trained without oversampling smaller corpora tended to perform better than their counterparts that used oversampling of smaller corpora. this is especially true when parallel corpora for those pairs/domains contain approximately 100,000 sentences or fewer. this shows that investigation into improved data balancing methods might be necessary rather than relying on naive approaches we used in this paper. in the future we hope to improve upon our results by leveraging sophisticated methods involving better corpora balancing and monolingual corpora through back-translation.