On The Evaluation of Machine Translation Systems Trained With Back-Translation (2020)

Abstract: Back-translation is a widely used data augmentation technique which leverages target monolingual data. However, its effectiveness has been challenged since automatic metrics such as BLEU only show significant improvements for test examples where the source itself is a translation, or translationese. This is believed to be due to translationese inputs better matching the back-translated training data. In this work, we show that this conjecture is not empirically supported and that back-translation improves translation quality of both naturally occurring text as well as translationese according to professional human translators. We provide empirical evidence to support the view that back-translation is preferred by humans because it produces more fluent outputs. BLEU cannot capture human preferences because references are translationese when source sentences are natural text. We recommend complementing BLEU with a language model score to measure fluency.

## introduction back-translation (bt; @xcite @xcite ) is a data augmentation method that is a key ingredient for improving translation quality of neural machine translation systems (nmt; @xcite @xcite . nmt systems using largescale bt have been ranked top at recent wmt evaluation campaigns @xcite @xcite . the idea is to train a target-to-source model to generate additional synthetic parallel data from monolingual target data. the resulting sentence pairs have synthetic sources and natural targets which are then added to the original bitext in order to train the desired sourceto-target model. bt improves generalization and can be used to adapt models to the test domain by adding appropriate monolingual data. parallel corpora are usually comprised of two types of sentence-pairs: sentences which originate in the source language and have been translated by humans into the target language, or sentences which originate from the target language and have been translated into the source language. we refer to the former as the direct portion and the latter as the reverse portion. the setup we are ultimately interested in is models that translate direct sentences. translations produced by human translators, or translationese tend to be simpler and more standardized compared to naturally occurring text @xcite @xcite . several recent studies found that such reverse test sentences are easier to translate than direct sentences @xcite , and human judges consistently assign higher ratings to translations of target original sentences than to source original sentences. these studies therefore recommend to restrict test sets to source original sentences, a methodology which has been adopted by the 2019 edition of the wmt news translation shared task. unfortunately, automatic evaluation with bleu @xcite only weakly correlates with human judgements @xcite . furthermore, recent wmt submissions relying heavily on back-translation mostly improved bleu on the reverse direction with little gains on the direct portion @xcite ; barry haddow's personal communication and see also @xcite . this finding is concerning for two reasons. first, back-translation may not be effective after all since gains are limited to the reverse portion. improvements on reverse sentences may only be due to a better match with the back-translated training sentences in this case. second, it may further reduce our confidence in automatic evaluation, if human judges disagree with bleu for systems trained with back-translation. indeed, human evaluations of top performing systems at wmt'18 @xcite and wmt'19 @xcite did not agree with bleu to the extent that correlation is even negative for the top entries @xcite . in this paper, we shed light on the following questions. first, do bt systems only work better in the reverse direction? second, does bleu reflect human assessment for bt models? and if that is not the case, why not and how can we alleviate the weaknesses of bleu? our contribution is an extensive empirical evaluation of top-performing nmt systems to validate or disproof some of the above conjectures. first, we show that translationese sources are indeed easier to translate, but this is true for both nmt systems trained with and without back-translated data. second, we confirm that human assessment of bt systems poorly correlates with bleu. third, bleu cannot capture the higher quality of backtranslation systems because the outputs of both back-translation and non back-translation models are equally close to the translationese references. fourth, we show that bt system outputs are significanlty more fluent than the output of a system only trained on parallel data, and this may explain the human preference towards bt generations. finally, we recommend to improve automatic evaluation by complementing bleu with a language model score which can better assess fluency in the target language while avoiding the artifacts of translationese references. ## related work back-translation has been originally introduced for phrase-based machine translation @xcite . for back-translation with neural machine translation, there is a large body of literature building upon the seminal work of @xcite , from large-scale extensions with sampling @xcite or tagging @xcite to its use for unsupervised machine translation @xcite as well as analysis @xcite and iterative versions @xcite . more similar to our work, toral et al. (2018) analyzed performance of trained state-of-the-art nmt systems in direct and reverse mode. they observe that translationese is simpler to translate and claimed that gains for such systems mostly come from improvements in the reverse direction. concurrent to our work, @xcite find that automatic evaluation with bleu does not align with the hypothesis that reverse sentences are easier to translate instead. unfortunately, their findings are not very conclusive because they do not control for the change of actual content, as sentences in one direction may be extracted from documents which are just harder to translate. in this work we correct for this effect by comparing translations of source original sentences with their double translations. @xcite al. (2019) also observe that bleu does not reliably correlate with human judgements. while they consider a large variety of systems trained in various ways, we instead focus on the comparison between the same nmt system trained with and without back-translated data. earlier work on statistical machine translation models argued in favor of using source original data only to train translation models @xcite , language models for translation @xcite , and to tune translation models @xcite . all these studies base most of their conclusions on automatic evaluation with bleu, which is problematic since bleu is not reliable and this procedure may overly optimize towards translationese references. @xcite proposed a post-editing method to turn translationese system outputs into more natural text. as part of their evaluation, they also observed that human assessments poorly correlate with bleu. while we confirm some of these observations, our goal is an in-depth analysis of the evaluation of nmt systems trained with backtranslated data. we provide empirical evidence corroborating the hypothesis that the discrepancy between bleu and human assessment is due to the use of translationese references, and we provide a constructive suggestion on how to better automatically evaluate models trained with bt. ## experimental setup in the next sections we first discuss the datasets and models used. then, we report bleu evaluations showing a big discrepancy between the gains obtained by a bt system in forward versus reverse direction compared to a baseline trained only on parallel data. this is followed by a series of hypotheses about the reasons for this discrepancy, and empirical studies in support or to disprove these hypotheses. we conclude with a recommendation for how to better evaluate nmt systems trained with bt. ## improving bt evaluation in the previous sections, we gathered mounting evidence that bleu fails at capturing the improved fluency of bt in direct mode. next, we propose to use a language model to assess fluency as an additional measure to complement bleu. different to the setup above ( §4.5, 4.6), where we used a separate lm for each system, we propose to use a single lm for all systems in order to simplify the evaluation. the language model is trained on a large monolingual dataset disjoint from the monolingual dataset used for generating back-translated data for bt training. this restriction is critical, otherwise the language model is likely to assign higher probably to bt generations simply because training and evaluation sets overlap. to train these language models we sample 315m, 284m and 120m com-bt ppl op ppl de-en 74.8 78.7 en-de 48.6 52.6 ru-en 57.6 68.6 en-ru 61.7 72.4 the language model is used to score the outputs of bt and op on the direct portion of the test set. if two systems have similar bleu scores, then a lower perplexity with the lm indicates higher fluency in the target natural language. this fluency assessment is complementary to bleu which in turn is more sensitive to adequacy. in future work, we intend to further investigate how to best combine bleu and language model scoring in order to maximize correlation with human judgements, particularly when evaluating bt in direct mode. meantime, practitioners can use this additional metric in their evaluation to break ties in bleu scoring. ## conclusions according to our findings, back-translation improves translation accuracy, for both source and target original sentences. however, automatic metrics like bleu fail to capture human preference for source original sentences (direct mode). we find that bt produces outputs that are closer to natural text than the output of op, which may explain human preference for bt. we recommend distinguishing between direct and reverse translations for automatic evaluation, and to make final judgements based on human evaluation. if human evaluation is not feasible, complementing standard metrics like bleu with a language model ( §5) may help assessing the overall translation quality. in the future, we plan to investigate more thor-oughly the use of language models for evaluating fluency, the effect of domain mismatch in the choice of monolingual data, and ways to generalize this study to other applications beyond mt.