Tagged Back-translation Revisited: Why Does It Really Work? (2020)

Abstract: In this paper, we show that neural machine translation (NMT) systems trained on large back-translated data overfit some of the characteristics of machine-translated texts. Such NMT systems better translate human-produced translations, i.e., translationese, but may largely worsen the translation quality of original texts. Our analysis reveals that adding a simple tag to back-translations prevents this quality degradation and improves on average the overall translation quality by helping the NMT system to distinguish back-translated data from original parallel data during training. We also show that, in contrast to high-resource configurations, NMT systems trained in low-resource settings are much less vulnerable to overfit back-translations. We conclude that the back-translations in the training data should always be tagged especially when the origin of the text to be translated is unknown.

## introduction during training, neural machine translation (nmt) can leverage a large amount of monolingual data in the target language. among existing ways of exploiting monolingual data in nmt, the so-called back-translation of monolingual data @xcite ) is undoubtedly the most prevalent one, as it remains widely used in state-of-the-art nmt systems @xcite . nmt systems trained on back-translated data can generate more fluent translations @xcite thanks to the use of much larger data in the target language to better train the decoder, especially for low-resource conditions where only a small quantity of parallel training data is available. however, the impact of the noisiness of the synthetic source sentences generated by nmt largely remains unclear and understudied. @xcite even showed that introducing synthetic noise in back-translations actually improves translation quality and enables the use of a much larger quantity of back-translated data for further improvements in translation quality. more recently, @xcite empirically demonstrated that adding a unique token at the beginning of each back-translation acts as a tag that helps the system during training to differentiate back-translated data from the original parallel training data and is as effective as introducing synthetic noise for improving translation quality. it is also much simpler since it requires only one editing operation, adding the tag, and non-parametric. however, it is not fully understood why adding a tag has such a significant impact and to what extent it helps to distinguish back-translated data from the original parallel data. in this paper, we report on the impact of tagging back-translations in nmt, focusing on the following research questions (see section 2 for our motivation). q1. do nmt systems trained on large backtranslated data capture some of the characteristics of human-produced translations, i.e., translationese? q2. does a tag for back-translations really help differentiate translationese from original texts? ## motivation during the training with back-translated data @xcite , we can expect the nmt system to learn the characteristics of back-translations, i.e., translations generated by nmt, and such characteristics will be consequently exhibited at test time. however, translating translations is a rather artificial task, whereas users usually want to perform translation of original texts. nonetheless, many of the test sets used by the research community for evaluating mt systems actually contain a large portion of texts that are translations produced by humans, i.e., translationese. translationese texts are known to be much simpler, with a lower mean sentence length and more standardized than original texts @xcite . these characteristics overlap with those of translations generated by nmt systems that have been shown simpler, shorter, and to exhibit a less diverse vocabulary than original texts (burlot and yvon, 2018). these similarities raise q1. caswell et al. (2019) hypothesized that tagging back-translations helps the nmt system during training to make some distinction between the backtranslated data and the original parallel data. even though the effectiveness of a tag has been empirically demonstrated, the nature of this distinction remains unclear. thus, we pose q2. the initial motivation for back-translation is to improve nmt for low-resource language pairs by augmenting the training data. therefore, we verify whether our answers to q1 and q2 for highresource conditions are also valid in low-resource conditions, answering q3. ## discussions we empirically demonstrated that training nmt on back-translated data overfits some of its characteristics that are partly similar to those of translationese. using back-translation improves translation quality for translationese texts but worsens it for original texts. previous work @xcite showed that stateof-the-art nmt systems are better in translating translationese than original texts. our results show that this is partly due to the use of back-translations which is also confirmed by concurrent and independent work @xcite . adding a tag to back-translations prevents a large drop of translation quality on original texts while improvements of translation quality for translationese texts remain and may be further boosted by tagging test sentences at decoding time. moreover, in low-resource conditions, we show that the overall tendency is significantly different from the high-resource conditions: backtranslation improves translation quality for both translationese and original texts while adding a tag to back-translations has only a little impact. we conclude from this study that training nmt on back-translated data, in high-resource conditions, remains reasonable when the user knows in advance that the system will be used to translate translationese texts. if the user does not know it a priori, a tag should be added to back-translations during training to prevent a possible large drop of translation quality. for future work, following the work on automatic identification of translationese @xcite , we plan to investigate the impact of tagging translationese texts inside parallel training data, such as parallel sentences collected from the web.