Evaluating Automatic Metrics with Incremental Machine Translation Systems (2024)

Abstract: We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. Since human A/B testing is commonly used, we assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations. Our study not only confirms several prior findings, such as the advantage of neural metrics over non-neural ones, but also explores the debated issue of how MT quality affects metric reliability—an investigation that smaller datasets in previous research could not sufficiently explore. Overall, our research demonstrates the dataset’s value as a testbed for metric evaluation. We release our code.

## introduction automatic metrics for machine translation (mt) are typically assessed by measuring their correlation with or accuracy with respect to human judgments @xcite @xcite . however, human evaluation is resource-intensive and time-consuming, and the number of translation systems included in a meta-evaluation tends to be relatively small. in this study, we explore the use of commercial machine translations, collected weekly over a period of 6 years for 12 translation directions, for the evaluation of mt metrics. given the common use of human a/b testing @xcite , our base assumption is that commercial systems show real improvements over time and that we can assess metrics as to whether they prefer more recent mt outputs. using our dataset, we revisit a number of recent findings in mt metrics research, and find that our dataset supports these. trained metrics, developed to directly learn human judgments @xcite , showed notable advancements in correlating with human judgments compared to non-neural metrics like bleu @xcite . recent studies @xcite also revealed that neural metrics achieved significantly better correlation with human judgments than non-neural ones and were able to generalize to new domains and challenging test sets. in our experiments, we analyze metric scores over time and evaluate metrics' ability to accurately rank mt systems. our findings demonstrate that neural metrics show a more consistent upward trend, and achieve higher accuracy than non-neural metrics. @xcite argued that the correlation between metrics and human judgments significantly decreased when considering only the topperforming systems. to be specific, they assessed the stability of metrics across top-n mt systems, and noticed that the correlation between metric and human scores diminished as n decreased. @xcite highlighted the instability of correlations with small n, and instead employed a rolling window of n systems, moving from the worst to the best systems. due to the limited number of mt systems (typically 10-15 systems) in the datasets, they were unable to confirm that the correlation declines as system quality improves with their approach. due to the much larger size of our dataset (see section 3.1), we can achieve more stable results when using the rolling window approach from @xcite . our findings reveal that a downward trend is the most common, supporting the results of @xcite , although upward or relatively flat trends are also seen in some language pairs. in wmt23 metrics shared task @xcite , human translations received unexpectedly low ratings, which prompted an investigation into using synthetic references as a potential alternative. they found that high-quality synthetic references could produce a stronger correlation between hu-man evaluations and metrics compared to human references. in our study, we reexamine the usefulness of synthetic references with three language pairs and find that synthetic references can result in comparable correlation. ## related work the metrics shared task at wmt @xcite has played a key role in the development and evaluation of automatic metrics. the annual data collected from wmt's comprehensive human evaluation of the translation task provides an ideal foundation for assessing these metrics. in this event, metrics are ranked based on the correlation calculated by comparing their scores to human ratings. machine translation systems that are compared typically come from the same evaluation campaign. an exception to this is the study by @xcite , who study longitudinal improvements in machine translation quality from 2007-2012 with human assessments, finding that translation quality of the top submissions to the wmt shared task indeed rose significantly. our study is similarly longitudinal, but our data stems from a single commercial system, and we do not perform our own human evaluation, but instead assume that improvements over time have been validated by company-internal human a/b testing. instead of evaluating metrics through comparison with human judgement, @xcite explored a complementary approach by correlating metrics with the outcome of downstream tasks. similarly, our study does not use human judgment directly; instead, we evaluate metrics based on their preference for newer mt outputs. ## methods we turn next to describe our data and the metrics we use. ## conclusion our dataset, covering 12 language pairs with at least 56 machine translations each, surpasses previous datasets that typically included only 3 pairs with around 15 machine translations each. based on the assumption that newer translations of a regularly updated commercial system tend to be of a higher quality, we apply the dataset to revisit prior findings on mt metrics. we provide largerscale evidence for debated questions such as the relationship between mt quality and metric reliability-issues that previous research was unable to conclusively resolve on smaller datasets. additionally, the systems are incremental (a baseline compared to improvements developed by the same group), reflecting the most common use case of the metrics. we encourage the use of our dataset for future investigations into mt metrics or the development of mt quality over time.