Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration (2023)

Abstract: Kendall’s tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.

## introduction kendall's τ is a widely used correlation statistic @xcite . it is easy to grasp intuitively, being based on pairwise rank ordering. this makes it complementary to other well-known statistics such as pearson or spearman. in the context of machine translation (mt), kendall plays a key role assessing the performance of evaluation metrics, a process known as metaevaluation: it has been the main statistic for measuring a metric's ability to score segment-level translations in the workshop on machine translation (wmt) metrics shared tasks over the years @xcite . several recent developments in mt-common to other areas of generative ai-have highlighted an important weakness in kendall's τ , namely how it deals with ties ( §4). first, as mt systems get better, they (1) produce more "perfect" outputs, which get assigned the same score by human raters; and (2) necessitate error-based analyses such as mqm @xcite , which often produce tied scores due to integer error counts. second, on the metric side, the use of recently-proposed llm-based metrics @xcite and metrics that model mqm annotations @xcite can also lead to small and discrete score ranges that assign many ties. in this paper, we examine the problems caused by ties in kendall's τ , using data from the wmt metrics tasks. we first show that there are simple phenomena that are not handled properly by any of the existing kendall variants, which mostly differ in how they treat ties ( §5.1). we also demonstrate the possibility of gaming the meta-evaluation by exploiting how ties are handled by existing τ 's, resulting in large improvements in certain evaluation settings ( §5.2). we propose instead to meta-evaluate metrics with a version of pairwise accuracy that is robust to these problems, assigning proper credit for correctly predicting ties ( §6). although there is a modification to τ that is closely related to pairwise accuracy, we argue that the accuracy formulation is easier to interpret, being just the proportion of correctly ranked pairs (including tied pairs). however, pairwise accuracy comes with its own problem, namely that it can discriminate against metrics that rarely assign ties. to counter this, we also propose an algorithm called tie calibration that automatically introduces ties into metric scores in order to optimize its correlation ( §7). we argue, and show empirically, that these two modifications result in a fairer assessment of mt metric performance ( §8.1). finally, we analyze different aspects of pairwise accuracy and tie calibration, including assessing the generalization of tie calibration across datasets ( §8.2), the score ranges where ties are introduced ( §8.3), and how more fine-grained statistics can be used to better understand metric behavior ( §8.4). while our experimental setting is limited to mt metrics, our work should be applicable to metaevaluation for other generative ai metrics with similar characteristics. ## background & related work we begin by justifying our exclusive focus on ranking-based statistics, like kendall's τ , then provide some background on mt metric metaevaluation, and finally contextualize our work by discussing kendall variants. ## analysis setup datasets our analysis is performed on the multidimensional quality metrics (mqm; @xcite ratings collected by the wmt'22 metrics shared task @xcite for three language pairs: @xmath0, @xmath1, and @xmath2. we use the mqm scores as the groundtruth human scores that the automatic metrics' scores are evaluated against. the language pairs have 13-15 systems and around 1300-1900 segments per system with mqm ratings. ## why ties are important there are several motivations for incorporating ties into a ranking-based meta-evaluation statistic like kendall's τ . first, ties in human scores from recent wmt shared tasks are much more trustworthy than they were previously. since wmt'20, the human scores are mqm scores instead of direct assessment (da) scores. the mqm scores come from expert translators and are more reliable than the crowdsourced da scores. as such, ties (or minor differences) between scores are more likely representative of actual ties (or minor differences) in translation quality. second, ties in mqm scores are very common. for instance, up to 53% of possible pairs in en-de have tied mqm scores (see figure 2 : when considering ties, m 1 only incorrectly ranks 1 out of the 6 2 pairs, whereas m 2 incorrectly ranks 6. however, due to how each τ handles ties, only acc eq and τ eq strongly prefer m 1 over m 2 . notably, τ 10 , τ 13 , and τ 14 are unable to distinguish a perfect metric (@xmath3. the acc eq and τ eq statistics are proposed in this work ( §6). els (gemba) predict a large number of ties (see ## shortcomings of kendall's variants the way ties are handled by existing variants of kendall's τ introduces blind spots in the metaevaluation and opens the door for metrics to exploit τ -specific properties to improve correlations. we demonstrate the shortcomings of existing τ 's through a motivational example and experimental analysis. ## evaluating with pairwise accuracy instead of using kendall's τ as the ranking-based meta-evaluation statistic, we propose to use a version of pairwise accuracy that includes ties. we define the pairwise accuracy to be the proportion of all pairs that the metric either ranks correctly or correctly predicts are tied. the equation for our proposal, denoted acc eq ("eq" for equality, as in ties), is included in although there is a modification of kendall's τ that corresponds to acc eq (denoted τ eq in ## tie calibration although we argue that acc eq properly addresses ties in human and metric scores, some metrics do not frequently predict exact ties between translations. regression metrics, such as bleurt @xcite and @xcite , practically never predict tied scores for two different translations (see to address this shortcoming, we propose an algorithm called tie calibration for automatically introducing ties into metric scores so that metrics that do and do not predict ties can be fairly compared. the algorithm is based on the intuition that, although regression metrics do not frequently predict ties, the difference between two translations' scores is sometimes small enough to be considered a tie. tie calibration searches for an value that maximizes a rank-based correlation statistic (e.g., τ or acc eq ) such that any two translations with a difference in score less than is considered to be a tie. 7 our implementation considers all possible differences between the n 2 pairs of translations as candidates for and selects the one that maximizes the desired ranking-based statistic. the algorithm runs in o(n 2 log n), where n is the number of translations. 8 detailed psuedocode for tie calibration is included in appendix d. because tie calibration introduces an optimal number of tie predictions, metrics are not penalized for under-predicting ties, and therefore metrics that do and do not predict ties can be fairly compared. an added benefit of tie calibration is that the resulting optimal improves the interpretability of metric scores. its value can be understood as the threshold for which a difference in metric scores should be considered significant (at least with respect to a specific dataset; see §8.2). henceforth we use * to denote a statistic that has been calculated with tie calibration (e.g., acc * eq ) and * the optimal tie threshold found by the algorithm. discussion. in principle, tie calibration can be used to find an optimal value of any correlation statistic in which the presence of ties changes the value, acc eq being one of them. however, care needs to be taken to ensure that the statistic handles ties in a desirable way. for example, τ 13 omits all ties from its formula, so tie calibration could convert a discordant pair into a tie to improve the value of τ 13 , which, if the human scores are not tied, is undesirable (acc eq would not reward this change). the combination of tie calibration and a statistic that does not properly handle ties may lead to unexpected results. ## analysis in this section, we analyze several different aspects related to our proposal of pairwise accuracy and tie calibration. we address the following questions: ## conclusion in this work, we demonstrated the importance of taking ties into account when calculating rankbased correlation statistics. we argued existing variants of kendall's τ are inadequate for the current state of meta-evaluation. we advocated to instead use pairwise accuracy, which rewards metrics for both predicting correct pair rankings and correctly predicting ties, in combination with a tie calibration procedure that allows for comparing metrics that do and do not predict ties. although our experiments were specific to mt, the methods proposed are generally applicable to any metric meta-evaluation in nlp.