Towards a Better Evaluation of Metrics for Machine Translation (2020)

Abstract: An important aspect of machine translation is its evaluation, which can be achieved through the use of a variety of metrics. To compare these metrics, the workshop on statistical machine translation annually evaluates metrics based on their correlation with human judgement. Over the years, methods for measuring correlation with humans have changed, but little research has been performed on what the optimal methods for acquiring human scores are and how human correlation can be measured. In this work, the methods for evaluating metrics at both system- and segment-level are analyzed in detail and their shortcomings are pointed out.

## introduction in the past, machine translation (mt) metrics have been extensively studied and evaluated, at both system-and segment-level @xcite @xcite . when performing systemlevel evaluation, the average score of a mt system is taken into account. segment-level evaluation uses each sentence (segment) separately to compute correlation. the results of these metric evaluations are critical to the way mt metrics are perceived. in particular the correlation with human judgment is of great importance. for this reason, an understanding for the workings of the evaluation method is required. proposals to identify relevant system-level human scores have been discussed @xcite , but no comprehensive analysis on this topic has been conducted. in particular, detailed studies on the segment-level evaluation are neglected, although it is an integral part of the metric evaluation. since the goal of a metric is to evaluate a translation as close as possible to a human's rating, it is important to clearly define the methods of determining human score and the methods of correla-tion measurement. this work aims to present an overview of the methods used in the evaluation, analyze their strengths and weaknesses, and propose solutions to some of the pitfalls of the methods. ## human scores to measure the correlation between the score of a metric and the score of a human, a method of determining human scores is required. thus, a person has to judge the quality of a translated sentence. this is not a simple task, as different people may have different opinions about the exact quality of the translation. another aspect to consider is that in order to calculate correlation, the score must be quantifiable in some way. thus, the methods used to detect human judgment must use a sufficient number of human judges for them to be reproducible. in the workshop on statistical machine translation (wmt), three different methods are used to determine the human score: direct assessment (da) @xcite , relative ranking (rr) @xcite and, in recent years, relative ranking out of direct assessment (darr) @xcite . ## measuring the correlation obtaining human scores is only part of the correlation calculation. the other one is to use both human and metric scores to compute their similarity or correlation. the case, where both human and metric scores are represented by absolute values, is straightforward to compute using methods such as pearson's r or spearman's ρ. however, da relies on a large amount of annotators that cannot always be guaranteed, especially at the segment-level. in the case where rr or darr is used for human scores, this task is not that easy. for this reason, the focus here is on the case where a form of rr is used -typically for the segment-level correlation calculation. as previously mentioned, wmt uses a form of kendall's τ to obtain a correlation given the relative ranking. the coefficient definition in its most general form is shown in equation ( 1 ) to formally define agreement and disagreement, a matrix can be used as described by @xcite . the various matrix formulations that have been used in wmt over the years are shown in the only difference between the three methods is the treatment of ties. on the other hand, incorporating ties while not considering human score ties can also lead to undesirable results. in a middle ground between the penalization and the ignoring of ties is the matrix in since there is no hard penalization for metrics that produce more ties, such metrics are at a disadvantage. for example, a metric with 20 ties and 80 concordant pairs would achieve a correlation of 80/(80 + 20) = 0.8, although all non-tie pairs achieve perfect correlation. on the other hand, a metric that overproduces ties, for example, with 80 ties and 20 concordant pairs, would have a correlation of 20/(20 + 80) = 0.2. it can also be argued that measuring the correlation on metrics with a too high percentage of ties is not significant, since there are too few sentence pairs that are concordant or discordant. one possible solution to the problem is shown in ## discussion the mt metric evaluation is an area that needs further investigation. this work gives an overview of the methods used so far and highlights some of their shortcomings. the system-level assessment currently seems to be good, but the evaluation methods at the segment-level still need to be explored (in particular, if there is not enough da data to directly calculate the correlation at the segment-level): in general, the task of creating a metric evaluation that is fair and reproducible for all metric types remains to be solved and deserves more attention and study.