BLEU might be Guilty but References are not Innocent (2020)

Abstract: The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems. This paper demonstrates that, while choice of metric is important, the nature of the references is also critical. We study different methods to collect references and compare their value in automated evaluation by reporting correlation with human evaluation for a variety of systems and metrics. Motivated by the finding that typical references exhibit poor diversity, concentrating around translationese language, we develop a paraphrasing task for linguists to perform on existing reference translations, which counteracts this bias. Our method yields higher correlation with human judgment not only for the submissions of WMT 2019 English to German, but also for Back-translation and APE augmented MT output, which have been shown to have low correlation with automatic metrics using standard references. We demonstrate that our methodology improves correlation with all modern evaluation metrics we look at, including embedding-based methods. To complete this picture, we reveal that multi-reference BLEU does not improve the correlation for high quality output, and present an alternative multi-reference formulation that is more effective.

## introduction machine translation (mt) quality has greatly improved in recent years @xcite @xcite . this progress has cast doubt on the reliability of automated metrics, especially in the high accuracy regime. for instance, the wmt @xmath0. such discrepancies have also been observed in the past, especially when comparing rule-based and statistical systems @xcite @xcite . automated evaluations are however of crucial importance, especially for system development. most decisions for architecture selection, hyperparameter search and data filtering rely on automated evaluation at a pace and scale that would not be sustainable with human evaluations. automated evaluation @xcite typically relies on two crucial ingredients: a metric and a reference translation. metrics generally measure the quality of a translation by assessing the overlap between the system output and the reference translation. different overlap metrics have been proposed, aiming to improve correlation between human and automated evaluations. such metrics range from n-gram matching, e.g. bleu @xcite , to accounting for synonyms, e.g. meteor @xcite , to considering distributed word representation, e.g. bertscore @xcite . orthogonal to metric quality @xcite , reference quality is also essential in improving correlation between human and automated evaluation. this work studies how different reference collection methods impact the reliability of automatic evaluation. it also highlights that the reference sentences typically collected with current (human) translation methodology are biased to assign higher automatic scores to mt output that share a similar style as the reference. human translators tend to generate translation which exhibit translationese language, i.e. sentences with source artifacts @xcite . this is problematic because collecting only a single style of references fails to reward systems that might produce alternative but equally accurate translations @xcite . because of this lack of diversity, multi-reference evaluations like multi-reference bleu are also bi-ased to prefer that specific style of translation. as a better solution, we show that paraphrasing translations, when done carefully, can improve the quality of automated evaluations more broadly. paraphrased translations increase diversity and steer evaluation away from rewarding translation artifacts. experiments with the official submissions of wmt 2019 @xmath1. further, we run additional experiments for mt systems that are known to have low correlation with automatic metrics calculated with standard references. in particular, we investigated mt systems augmented with either back-translation or automatic post-editing (ape). we show that paraphrased references overcome the problems of automatic metrics and generate the same order as human ratings. our contributions are four-fold: (i) we collect different types of references on the same test set and show that it is possible to report strong correlation between automated evaluation with human metrics, even for high accuracy systems. (ii) we gather more natural and diverse valid translations by collecting human paraphrases of reference translations. we show that (human) paraphrases correlate well with human judgments when used as reference in automatic evaluations. (iii) we present an alternative multi-reference formulation that is more effective than multi reference bleu for high quality output. (iv) we release foot_0 a rich set of diverse references to encourage research in systems producing other types of translations, and reward a wider range of generated language. ## related work evaluation of machine translation is of crucial importance for system development and deployment decisions @xcite . human evaluation typically reports adequacy of translations, often complemented with fluency scores @xcite . evaluation by human raters can be conducted through system comparisons, rankings @xcite , or absolute judgments, direct assessments @xcite . absolute judgments allow one to efficiently compare a large number of systems. the evaluation of translations as isolated sentences, full paragraphs or documents is also an important factor in the cost/quality trade-offs @xcite . isolated sentence evaluation is generally more efficient but fails to penalize contextual mistakes @xcite . automatic evaluation typically collects human reference translations and relies on an automatic metric to compare human references to system outputs. automatic metrics typically measure the overlap between references and system outputs. a wide variety of metrics has been proposed, and automated metrics is still an active area of research. bleu @xcite is the most common metric. it measures the geometric average of the precision over hypothesis n-grams with an additional penalty to discourage short translations. nist @xcite is similar but considers up-weighting rare, informative n-grams. ter @xcite ) measures an edit distance, as a way to estimate the amount of work to post-edit the hypothesis into the reference. me-teor @xcite suggested rewarding n-gram beyond exact matches, considering synonyms. others are proposing to use contextualized word embeddings, like bertscore @xcite . rewarding multiple alternative formulations is also the primary motivation behind multiple-reference based evaluation @xcite . dreyer and marcu (2012) introduced an annotation tool and process that can be used to create meaning-equivalent networks that encode an exponential number of translations for a given sentence. orthogonal to the number of references, the quality of the reference translations is also essential to the reliability of automated evaluation @xcite . this topic itself raises the question of human translation assessment, which is beyond the scope of this paper @xcite . meta-evaluation studies the correlation between human assessments and automatic evaluations @xcite @xcite . indeed, automatic evaluation is useful only if it rewards hypotheses perceived as fluent and adequate by a human. interestingly, previous work @xcite has shown that a higher correlation can be achieved when comparing similar systems than when comparing different types of systems, e.g. phrase-based vs neural vs rulebased. in particular, rule-based systems can be penalized as they produce less common translations, even when such translations are fluent and adequate. similarly, recent benchmark results comparing neu-ral systems on high resource languages @xcite have shown mismatches between the systems with highest bleu score and the systems faring the best in human evaluations. freitag et al. (2019); edunov et al. (2019 ) study this mismatch in the context of systems trained with back-translation @xcite and noisy back-translation @xcite . they observe that systems training with or without backtranslation (bt) can reach a similar level of overlap (bleu) with the reference, but hypotheses from bt systems are more fluent, both measured by humans and by a language model (lm). they suggest considering lm scores in addition to bleu. freitag et al. (2019); edunov et al. (2019) point at translationese as a major source of mismatch between bleu and human evaluation. translationese refers to artifacts from the source language present in the translations, i.e. human translations are often less fluent than natural target sentences due to word order and lexical choices influenced by the source language @xcite . the impact of translationese on evaluation has recently received attention @xcite @xcite . in the present work, we are specifically concerned that the presence of translationese in the references might cause overlap-based metrics to reward hypotheses with translationese language more than hypotheses using more natural language. the question of bias to a specific reference has also been raised in the case of monolingual human evaluation @xcite . the impact of translationese in test sets is related to but different from the impact of translationese in the training data @xcite @xcite . in this work, we explore collecting a single reference translation, using human paraphrases to steer away as much as possible from biases in the reference translation that affect the automatic metrics to prefer mt output with the same style (e.g. translationese). automatic methods to extract paraphrase n-grams @xcite or full sentence paraphrases @xcite @xcite have been used to consider multiple references. in contrast, we generate a single unbiased reference translation generated by humans instead of trying to cover a wider space of possible translations. in contrast to human paraphrasing (our instructions asked for most diverse paraphrases), automatic paraphrasing are still far from perfect @xcite and mostly generate local changes that do not steer away from biases as e.g. introducing different sentence structures. ## collecting high quality and diverse references we acquired two types of new reference translations: first, we asked a professional translation service to provide an additional reference translation. second, we used the same service to paraphrase existing references, asking a different set of linguists. ## experimental set-up 4.1 data and models we use the official submissions of the wmt 2019 @xmath2. we then report correlations with the wmt human ratings from the same evaluation campaign. we chose @xmath3. ## experiments we generate three additional references for the wmt 2019 @xmath4. in addition to acquiring an additional reference (ar), we also asked linguists to paraphrase the existing wmt reference and the ar reference (see section 3 for details). we refer to these paraphrases as wmt.p and ar.p. ## conclusions this work presents a study on the impact of reference quality on the reliability of automated evaluation of machine translation. we consider collecting additional human translations as well as generating more diverse and natural references through paraphrasing. we observe that the paraphrased references result in more reliable automated evaluations, i.e. stronger correlation with human evaluation for the submissions of the wmt 2019 @xmath5. these findings are confirmed across a wide range of automated metrics, including bleu, chrf, meteor, bertscore and yisi. we further demonstrate that the paraphrased references correlate especially well for the top submissions of wmt, and additionally are able to correctly distinguish baselines from systems known to produce more natural output (those augmented with either bt or ape), whose quality tends to be underestimated by references with translationese artifacts. we explore two different approaches to multireference evaluation: (a) standard multi-reference bleu, and (b) selecting the best-rated references for each sentence. contrary to conventional wisdom, we find that multi-reference bleu does not exhibit better correlation with human judgments than single-reference bleu. combining two standard reference translations by selecting the best rated reference, on the other hand, did increase correlation for the standard reference translations. nevertheless, the combined paraphrasing references are of higher quality for all techniques when compared to the standard reference counter part. we suggest using a single paraphrased reference for more reliable automatic evaluation going forward. although a combined paraphrased reference shows slightly higher correlation for embedding based metrics, it is over twice as expensive to construct such a reference set. to drive this point home, our experiments suggest that standard reference translations may systematically bias against modelling techniques known to improve human-judged quality, raising the question of whether previous research has incorrectly discarded approaches that actually improved the quality of mt. releasing all reference translations gives the community a chance to revisit some of their decisions and measure quality differences for high quality systems and modelling techniques that produce more natural or fluent output. as a closing note, we would like to emphasize that it is more difficult for a human rater to rate a paraphrased translation than a translationese sentence, because the latter may share a similar structure and lexical choice to the source. we suspect that human evaluation is also less reliable for complex translations. future work, can investigate whether finer ratings could correct the bias in favor of lower effort ratings, and how this may interact with document-level evaluation.