Human-Paraphrased References Improve Neural Machine Translation (2020)

Abstract: Automatic evaluation comparing candidate translations to human-generated paraphrases of reference translations has recently been proposed by freitag2020bleu. When used in place of original references, the paraphrased versions produce metric scores that correlate better with human judgment. This effect holds for a variety of different automatic metrics, and tends to favor natural formulations over more literal (translationese) ones. In this paper we compare the results of performing end-to-end system development using standard and paraphrased references. With state-of-the-art English-German NMT components, we show that tuning to paraphrased references produces a system that is ignificantly better according to human judgment, but 5 BLEU points worse when tested on standard references. Our work confirms the finding that paraphrased references yield metric scores that correlate better with human judgment, and demonstrates for the first time that using these scores for system development can lead to significant improvements.

## introduction machine translation (mt) has shown impressive progress in recent years. neural architectures @xcite @xcite have greatly contributed to this improvement, especially for languages with abundant training data @xcite @xcite . this progress creates novel challenges for the evaluation of machine translation, both for human @xcite and automated evaluation protocols @xcite . both types of evaluation play an important role in machine translation @xcite . while human evaluations provide a gold standard evaluation, they involve a fair amount of careful and hence expensive work by human assessors. cost therefore limits the scale of their application. on the other hand, automated evaluations are much less expensive. they typically only involve human labor when collecting human reference translations and can hence be run at scale to compare a wide range of systems or validate design decisions. the value of automatic evaluations therefore resides in their capacity to be used as a proxy for human evaluations for large scale comparisons and system development. the recent progress in mt has raised concerns about whether automated evaluation methodologies reliably reflect human ratings in high accuracy ranges. in particular, it has been observed that the best systems according to humans might fare less well with automated metrics @xcite . most metrics such as bleu @xcite and ter @xcite measure overlap between a system output and a human reference translation. more refined ways to compute such overlap have consequently been proposed @xcite @xcite . orthogonal to the work of building improved metrics, @xcite hypothesized that human references are also an important factor in the reliability of automated evaluations. in particular, they observed that standard references exhibit simple, monotonic language due to human 'translationese' effects. these standard references might favor systems which excel at reproducing these effects, independent of the underlying translation quality. they showed that better correlation between human and automated evaluations could be obtained when replacing standard references with paraphrased references, even when still using surface overlap metrics such as bleu @xcite . the novel references, collected by asking linguists to paraphrase standard references, were shown to steer evaluation away from rewarding translation artifacts. this improves the assessment of alternative, but equally good translations. our work builds on the success of paraphrased translations for evaluating existing systems, and asks if different design choices could have been made when designing a system with such an evaluation protocol in mind. this examination has several potential benefits: it can help identify choices which improve bleu on standard references but have limited impact on final human evaluations; or those that result in better translations for the human reader, but worse in terms of standard reference bleu. conversely, it might turn out that paraphrased references are not robust enough to support system development due to the presence of 'metric honeypots': settings that produce poor translations, but which are nevertheless assigned high bleu scores. to address these points, we revisit the major design choices of the best @xmath0, and measure their impact on standard reference bleu as well as on paraphrased bleu. this allows us to measure the extent to which steps such as data cleaning, back-translation, fine-tuning, ensemble decoding and reranking benefit standard reference bleu more than paraphrase bleu. revisiting these development choices with the two metrics results in two systems with quite different behaviors. we conduct a human evaluation for adequacy and fluency to assess the overall impact of designing a system using paraphrased bleu. our main findings show that optimizing for paraphrased bleu is advantageous for human evaluation when compared to an identical system optimized for standard bleu. the system optimized for paraphrased bleu significantly improves wmt newstest19 adequacy ratings (4.72 vs 4.27 on a six-point scale) and fluency ratings (63.8% vs 27.2% on side-by-side preference) despite scoring 5 bleu points lower on standard references. ## related work collecting human paraphrases of existing references has recently been shown to be useful for system evaluation @xcite . our work considers applying the same methodology for system tuning. there is some earlier work relying on automated paraphrases for system tuning, especially for statistical machine translation (smt). @xcite introduced an automatic paraphrasing technique based on english-to-english translation of full sentences using a statistical mt system, and showed that this permitted reliable system tuning using half as much data. similar automatic paraphrasing has also been used to augment training data, e.g. @xcite , but relying on standard references for evaluation. in contrast to human paraphrases, the quality of current machine generated paraphrases degrades significantly as overlap with the input decreases @xcite . this makes their use difficult for evaluation since @xcite suggests that substantial paraphrasing -'paraphrase as much as possible' -is necessary for evaluation. our work can be seen as replacing the regular bleu metric with a new paraphrase bleu metric for system tuning. different alternative automatic evaluation metric have also been considered for system tuning @xcite with minimum error rate training, @xcite . this work showed some specific cases where translation error rate (ter) was superior to bleu. our work is also related to the bias that the human translation process introduces in the references, including source language artifacts-translationese @xcite -as well as source-independent artifacts-translation universals @xcite . the professional translation community studies both systematic biases inherent to translated texts @xcite , as well as biases resulting specifically from interference from the source text @xcite . for mt, @xcite point at translationese as a source of mismatch between bleu and human evaluation, raising concerns that overlap-based metrics might reward hypotheses with translationese language more than hypotheses using more natural language. the impact of translationese on human evaluation of mt has recently received attention as well @xcite @xcite . more generally, the question of bias to a specific reference has also been raised, in the case of monolingual manual evaluation @xcite . different from the impact of translationese on evaluation, the impact of translationese in the training data has also been studied @xcite @xcite . finally, our work is also related to studies measuring the importance of the test data quality, looking specifically at the test set translation direction. for smt evaluation, @xcite and @xcite explored how the translation direction affects translation results. @xcite noted that the original language of the test sentences influences the bleu score of translations. they showed that the bleu scores for targetoriginal sentences are on average higher than sentences that have their original source in a different language. recently, a similar study was conducted for neural mt @xcite . ## experimental setup we first describe data and models, then present our human evaluation protocol. ## experimental results this section first presents our main result comparing the same system tuned with bleu on standard versus paraphrased references. we then break down how system design choices impact each metric differently. throughout, we refer to scores computed with standard references as bleu, and those computed with paraphrased references as bleup. ## analysis this section confirms the results from the previous section with additional references for newstest2019 and illustrates the behaviour of our systems on individual sentences. ## conclusions prior work has shown that bleu measured on paraphrased references (bleup) has better correlation with human evaluation than bleu measured on regular references (bleu) for the comparison of existing systems @xcite . motivated by this finding, we collected a development set of paraphrased references and assessed bleup for system development. this allowed us to evaluate if the design choices of a modern neural mt system impact bleu and bleup differently, including tuning a re-ranking noisy channel model to these metrics. our experiments followed the setup from the winning newstest19 @xmath1. for design choices, we observe that bleup seems to emphasize the importance of backtranslation even when test sets are source original. on the other end, bleup seems to de-emphasize the importance of ensembles, as the reliable prediction of common language by ensembles is less rewarded by this metric. our tuning experiments led to positive results. in human evaluation, the system tuned on bleup showed significant improvements in terms of adequacy and even greater gains in terms of fluency compared to the system tuned on bleu. example translations indicate that the model tuned on bleup produces noticeably less literal translations. our experiments also highlight a disconnect between regular bleu and human evaluation: the system tuned on bleup degrades standard bleu scores by over 5 points, while faring significantly better in human evaluation. paraphrased automatic evaluation therefore seems to be a promising proxy this research opens the question of whether these results can be confirmed over a wide range of language pairs. we also hope to achieve further improvements by refining the paraphrased evaluation protocol.