Combination of Neural Machine Translation Systems at WMT 20 (2020)

Abstract: This paper presents neural machine translation systems and their combination built for the WMT20 English-Polish and Japanese->English translation tasks. We show that using a Transformer Big architecture, additional training data synthesized from monolingual data, and combining many NMT systems through n-best list reranking improve translation quality. However, while we observed such improvements on the validation data, we did not observed similar improvements on the test data. Our analysis reveals that the presence of translationese texts in the validation data led us to take decisions in building NMT systems that were not optimal to obtain the best results on the test data.

## introduction this paper describes the neural machine translation (nmt) systems and their combination built for a participation of the national institute of information and communications technology (nict) in the wmt20 shared news translation task. 1 we participated in three translation directions: @xmath0, @xmath1, and @xmath2. all our systems are constrained, i.e., we used only the parallel and monolingual data provided by the organizers to train and tune them, and validated/selected our best systems exclusively using the official validation data provided by the organizers. we trained nmt systems with several different frameworks and architectures, and combined them, for each translation direction, through n-best list reranking using informative features as proposed by @xcite . this simple combination method, associated with the exploitation of large tagged back-translated monolingual data, improved bleu scores on the official 1 the team id of our participation is "nict kyoto". validation data provided by the organizers. however, we did not observe these improvements on the test data for which our baseline systems remained the best. while we have rigorously selected our systems according to their performance on the validation data, the analysis of our results reveal how easily we would have been able to achieve bleu scores among the best submissions by choosing/selecting our best systems according to their performance on the test data, as encouraged by the wmt submission process (section 2). the remainder of this paper is organized as follows. in section 2, we briefly describe the wmt20 translation task. in section 3, we introduce the data pre-processing and cleaning. in section 4, we describe the details of our nmt systems' architectures and frameworks. in section 5, we describe two different strategies that we used to augment the training parallel data of our systems: parallel data extraction from monolingual data and backward/forward translations. then, the combination of our nmt systems is described in section 6. empirical results produced with our systems on the validation and test data are presented in section 7. we propose an analysis in section 8 to better understand why our best systems on the validation data are significantly worse on the test data. section 9 concludes this paper. ## description of the task the task is to translate texts in the news domain. for this purpose, news articles were sampled from online newspapers from september-november 2019. the sources of the test data are original texts whereas the targets are humanproduced translations, i.e., participants are not asked to translate translationese texts unlike past wmt translation tasks. although organizers also mentioned that the provided validation data were created in the same way as the test data, they were actually made half of translationese texts and half of original texts. for the @xmath3, source texts to translate in the test data were only translationese texts. training parallel and monolingual data were provided for all language pairs. participants were asked to mention whether they used additional external data. we chose to participate in the constrained settings using only the provided data to train our mt systems. validation data were also provided for each language pair. we used the entire data to keep it sufficiently large for validation purposes, even though half of it was made up of translationese texts. for collecting submissions, organizers relied on a new framework: ocelot. foot_0 each participant was allowed to submit up to 8 submissions per account but was not limited in the number of accounts. upon submission, ocelot shows the corresponding chrf and bleu scores computed using reference translations that were not released during the competition. participants could then rely on these scores to select and validate their best system on the test data. 3 we chose to ignore these scores obtained on the test data to remain in a much more realistic scenario where we do not have access to reference translations, i.e., we relied only on the validation data to select our primary submission. primary submissions selected by the participants were then evaluated by humans which is the official evaluation for wmt translation tasks. ## combination of nmt systems our primary submissions for the tasks were the result of a simple combination of all our nmt systems through n-best list reranking. as demonstrated by @xcite , it can significantly improve translation quality, even when there is a large difference in translation quality between the combined systems. following marie and fujita (2018), our system combination works as follows. ## results our main results are presented in the ## analysis one obvious observation from these results is that the bleu scores on the test data are all very close to the score of the validation data in original language (orig.). on the other hand, we also observe that the ranking of the systems given the bleu score on the entire validation data does not correlate well with the ranking of the systems given the bleu score on the validation data in the original language. it means that the translationese texts in the validation data had a negative impact on all our decisions for selecting the best framework, architecture, additional parallel sentences, and so forth, and that we could potentially had better results by taking our decisions by using only the original texts in the validation data. translationese texts are particularly harmful for training a reranker, as we can observe for @xmath5. using them as training data for the reranker leads to significantly lower bleu scores (#14) while training it only on the original texts of the validation data leads to our best bleu score (#15). for this translation direction, we also observe large improvements of bleu thanks to back-translations on the validation data that comes mainly from the translationese texts while translation quality drops when translating the original texts in the validation and test data, as expected. this was compensated by using tagged back-translations (#10-12) as suggested by @xcite . our observations are very different for the reverse translation direction. for @xmath6, training reranker on the entire validation data leads to the best bleu score, and it drops only slightly when training only on the translationese texts. even more surprisingly, using back-translations improves bleu scores for both original and translationese texts while using tagged backtranslations (#12) leads to bleu scores identical to those obtained by using back-translations (#9) for the original texts. these peculiarities observed for @xmath7, associated with our observations in section 5.2 that forward translations improves bleu, are in contradiction with the findings in previous work as follows. a possible explanation is that the texts denoted as "original" in the validation data and the test data, ## conclusion we participated in three translation directions and for all of them we did experiments with several frameworks and architectures, also exploiting additional synthetic parallel data made from monolingual data. combining all our systems led to significantly better bleu scores on the validation data. however, our analysis revealed that the presence of translationese texts in the validation data led us to take sub-optimal choices that prevented us from obtaining significantly better bleu scores on the test data. selecting/validating systems on the test data should not be possible, or at least not an option. we thus suggest organizers to provide validation data that better matches the characteristics of the test data, e.g., removing translationese texts if none are in the test data.