Trained MT Metrics Learn to Cope with Machine-translated References (2023)

Abstract: Neural metrics trained on human evaluations of MT tend to correlate well with human judgments, but their behavior is not fully understood. In this paper, we perform a controlled experiment and compare a baseline metric that has not been trained on human evaluations (Prism) to a trained version of the same metric (Prism+FT). Surprisingly, we find that Prism+FT becomes more robust to machine-translated references, which are a notorious problem in MT evaluation. This suggests that the effects of metric training go beyond the intended effect of improving overall correlation with human judgments.

## introduction while trained evaluation metrics for machine translation (mt) tend to have a high correlation with human judgments @xcite , they remain black boxes, sometimes behaving in unexpected ways @xcite . this calls into question whether a metric's utility can be measured solely by its correlation with human judgments. in this paper, we intentionally provide mt metrics with machine-translated reference translations, as opposed to human-created references, and investigate how this factor influences the behavior of a metric. in mt evaluation research, the human translators who create reference translations are usually asked to produce them from scratch, in order to avoid references that are machine-translated or post-edited @xcite . nevertheless, traces of mt have been detected in some reference sets @xcite @xcite . it is therefore important to understand how metrics behave under such references. in our experiments, we use a surrogate for real post-edited references in the form of error-free output by various systems from the wmt 2021 news translation task @xcite . our results show that there is a stark difference between trained and non-trained metrics: while trained metrics maintain most of their accuracy when provided with such mt-derived references, non-trained metrics exhibit a substantial drop in accuracy. to corroborate this observation, we perform a controlled experiment involving prism @xcite , a metric that is based on a multilingual mt system. the original version of prism can be considered non-trained, since it learns from parallel sentences without human judgments. we then fine-tune prism on a dataset of human judgments, using a bidirectional pairwise ranking approach. as expected, the segment-level correlation of prism increases during fine-tuning, indicating that the metric learns to better predict human judgments (figure 1 ). moreover, we find that finetuning narrows the gap in performance between human-created and machine-translated references. our experiment thus indicates that training a metric on human evaluation data can influence its behavior in a way that is not captured by global correlation with human judgments. code to reproduce our findings will be made available. 1 to summarize, the paper makes the following contributions: ## fine-tuning the prism metric prism @xcite ) is a referencebased evaluation metric that relies on the paraphrasing probability between a system translation and a reference. the probability is estimated by a multilingual nmt model as a zero-shot translation direction. the model is expected to prefer mere copies of the source sequence to more creative paraphrases, which is especially useful for referencebased evaluation. the nmt model uses the reference as a source sequence x and the system translation as a hypothesis y, or vice versa. the segment-level score s is then calculated from token-level log-probabilities: foot_0 by default, prism uses the average of both paraphrasing directions: an overall score for a system can then be calculated as an average over a collection of segments. ## conclusion we have shown that metrics without supervision by human judgments, such as bleu and chrf, tend to be inaccurate under machine-translated references, while trained metrics are more robust. in order to methodically examine this phenomenon, we have trained the prism evaluation metric on a dataset of human judgments. our experiments show that fine-tuning improves the segment-level accuracy of prism on an unseen test set across multiple language pairs and domains, and clearly increases its robustness to machine-translated references. one conclusion to draw from our findings is that post-edited references likely diminish the accuracy of reference-based metrics and should be avoided. a second conclusion is that if it cannot be ruled out that references originate from mt, as is often the case in practice, trained metrics are to be preferred. fine-tuning a metric such as prism on referencebased evaluation can thus be seen as a technique to let the metric make the best out of reference translations in the wild.