Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies (2024)

Abstract: Ten years ago a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for researchers to develop and retain intuitions about metric deltas that drove earlier research and deployment decisions. This paper investigates the “dynamic range” of a number of modern metrics in an effort to provide a collective understanding of the meaning of differences in scores both within and among metrics; in other words, we ask “what point difference x in metric y is required between two systems for humans to notice?”. We conduct our evaluation on a new large dataset, ToShip23, using it to discover deltas at which metrics achieve system-level differences that are meaningful to humans, which we measure by pairwise system accuracy. We additionally show that this method of establishing delta-accuracy is more stable than the standard use of statistical p-values in regards to testset size. Where data size permits, we also explore the effect of metric deltas and accuracy across finer-grained features such as translation direction, domain, and system closeness.

## introduction a decade ago, the bleu metric served as the default metric for machine translation evaluation. it was not without its criticisms @xcite @xcite or compelling alternatives @xcite , but a combination of adequate performance, robustness to new languages, simplicity, understandability-and also inertia-helped it retain this position. this is no longer the case. bleu's deficiencies quickly became apparent as deep learning approaches to machine translation replaced the earlier symbolic paradigms @xcite . today, a number of metrics-themselves deep-learning basedcompete in an ecosystem where there is no longer any dominant, default metric. this situation creates a problem for researchers working to keep abreast of developments in the field. different metrics, including different models within the same metric family, have different dynamic ranges, i.e., the range of scores one can expect to see. furthermore, the metric delta, i.e., the score difference signifying a meaningful change in performance between two systems, also varies across metrics. it is perhaps understandable that some practitioners therefore continue to use bleu, as well, if only to ground their understanding. this paper attempts to introduce some order and clarity into this situation. we make use of a large, new human evaluation dataset, toship23, to compare the score ranges of metrics on a large number of systems against pairwise system-level accuracy. importantly, we break down these accuracy scores into bins based on metric deltas, which allows us to determine accuracies for each metric as a func-tion of the score differences between two systems. this provides a measure of confidence in the output that is stable across testset size, in contrast to standard statistical significant testing, which becomes more stable as testset size grows. we release a tool that allows a user to easily compare accuracies at different threshold across metrics. 0 in this work we: §3.2 empirically investigate the estimated accuracy for multiple metrics, human ability to perceive quality difference; §3.3 provide thresholds for popular metrics to help reviewers and practitioners interpret results; §4.1 validate our estimated accuracies on wmt testsets and §4.2 investigate the effect of different language groups; §4.3 show that string-based metrics, such as bleu, should never be used to evaluate unrelated systems; §4.4 show that statistical significance testing is insufficient to determine model improvement especially as it is affected by the testset size, but is important for small deltas; §5.1 assess quality of automatic metrics over 6530 system pairs; §5.2 summarize recommendations for machine translation evaluation. ## experimental setup data. we perform experiments related to evaluation of mt outputs based on a proprietary dataset toship23 which is of a magnitude larger than any publicly available data and enables more fine grained glimpse into the metrics behaviour. the dataset is an extended version of toship21 dataset @xcite with details described in appendix b. we also use data from the annual wmt evaluation campaigns to validate our results, specifically the metrics shared task @xcite , to make results replicable. we only use mqm @xcite and da+sqm @xcite subset of human evaluated systems because reference-based da @xcite is suboptimal for the evaluation of modern mt systems @xcite . see ## factors affecting metric deltas we have empirically derived the estimated accuracy for various metrics. in this section, we investigate factors that affect metric delta and show how reliable the thresholds remain under these factors. these include the testset size, dataset and domain selection, and translation direction. additional factors could influence the metric delta, but we lack the data to evaluate these aspects. a key consideration is whether the metric delta is contingent on the underlying absolute values. in other words, we need to determine if a +1 bleu delta varies in reliability based on these absolute values. for instance, does the impact of moving from 20 to 21 bleu differ significantly from a shift from 60 to 61 bleu in different system pairs? ## related work the closest work to ours is @xcite , who investigate the relationship between metric deltas and the p-value of human ranking, concluding that not even 2 bleu points reliably correspondent to human judgement. this aligns with our work that two bleu points reaches an estimated accuracy of only 77.2%. their work also does not consider the directionality of the delta, and consequently they do not penalize situations where humans and metric disagree on which system is better. @xcite found that even statistical significant deltas of up to three bleu points do not reliably correspond to human judgement. in a broad survey, @xcite notes that various community "rules of thumb" about sufficient bleu deltas might be the result of an evolved consensus that has no basis in scientific evidence. similarly, @xcite demonstrated that among system pairs deemed statistically significant by humans and where bleu disagree with humans, the median delta is 1.3 bleu. marie (2022) reinvestigated the wmt 2020 and 2021 results and showed that deltas lower than 2 bleu needs to be tested for statistical significance. automated metrics in nlp and mt have been under scrutiny for a long time. @xcite raised early doubts about bleu. @xcite pointed to failure modes of bleu and suggested it be used in more narrow situations. post (2018) identified a problem with conflicting implementations of bleu and offered a unified solution. the broader field of computer science has been concerned with what is a meaningful acceptance threshold of a metric @xcite . the acceptance thresholds are usually established to trade off risks in types of errors @xcite . @xcite , studying effect sizes in psychology, summarize that effect sizes should be scaled appropriately. alike, @xcite ask what effect size suffices and note its dependence on the variance and that all acceptance thresholds are arbitrary. ## conclusion in this work, we investigated the interpretation of deltas from automatic machine translation metrics. although metrics have different ranges of scores, what ultimately matters to the practitioner is how score deltas are grounded in human ability to perceive those differences, which we judge by pairwise system-level accuracy on a large collection of human judgments. we empirically determined thresholds for popular metrics to align them on accuracy and provide a tool 0 that relates metrics to each other. finally, we showed the importance of using metric-delta accuracy over p-values: the former is stable across testset sizes. we undertook some investigations into subfactors of the data, showing that the results were robust to, for example, translation direction, and also that they generalized to different testsets. these investigations were limited by the data size. for future work, it would be useful to explore deltaaccuracy for different subsets and combinations of features, presuming that enough data were available for the task.