Metric Score Landscape Challenge ( MSLC 23): Understanding Metrics’ Performance on a Wider Landscape of Translation Quality (2023)

Abstract: The Metric Score Landscape Challenge (MSLC23) dataset aims to gain insight into metric scores on a broader/wider landscape of machine translation (MT) quality. It provides a collection of low- to medium-quality MT output on the WMT23 general task test set. Together with the high quality systems submitted to the general task, this will enable better interpretation of metric scores across a range of different levels of translation quality. With this wider range of MT quality, we also visualize and analyze metric characteristics beyond just correlation.

## introduction under time and human resource constraints, automatic metrics are often used as a proxy of manual evaluation for machine translation (mt) quality. the wmt metrics shared task evaluates how well a variety of automatic metrics correspond to human judgments of mt quality, as evaluated on the wmt general (formerly news) shared task data. those mt systems being evaluated are typically highperforming systems, especially for high-resource language pairs. however, in practice, the lessons learned are applied to a broader range of systems in development, including low-resource and lowquality output. this challenge set foot_0 aims to gain insight into metric scores across a broader mt quality landscape. it provides a collection of low-to mediumquality mt output on the wmt23 general task test set. this serves several purposes. together with the high quality systems submitted to the general task, this will enable more thorough understanding of metric scores across a range of different levels of translation quality: useful knowledge for researchers considering applying these metrics to lower-resource language pairs or lower-performing domains. this challenge set also allows us to explore metric characteristics beyond just correlation, which has been a main focus of past work. by expanding the range of mt quality analyzed, we shed light on some unexpected or under-explored properties of metrics, such as metrics that can distinguish between high quality systems but are not able to differentiate different levels of mt quality on the lower end of the quality scale (or vice versa) and metrics that use their space of scores in very different ways (e.g., discretized, or with specific score ranges with particularly large numbers of ties). we focus on four language pairs: @xmath0, hebrew↔english (he↔en), and @xmath1. three of these correspond to the focus languages of the wmt 2023 metrics shared task @xcite , and they also cover several language families and aspects of translation evaluation (i.e., the paragraph-level evaluation of @xmath2, as well as including a sentence-level out-of-english direction (@xmath3. we combine source and reference data from the news portion of the wmt 2023 general mt task test sets with our challenge set, the low-and medium-quality mt output that we generated to cover a range of mt quality. we begin by describing the training data (section 3.1) and models (section 3.2) used in for constructing our challenge set (section 3.3). we also briefly describe the additional data (section 4) and metrics (section 5) analyzed. in section 6 we analyze the distribution of different metrics over the challenge set. we find that some metrics exhibit strikingly different characteristics on the lowquality systems as compared to the systems submitted to wmt, while others exhibit unexpected characteristics (e.g., large numbers of tied scores) that would not have been apparent from standard correlational analysis or from high-quality wmt submitted systems alone. we conclude by arguing that examining metric characteristics and performance over a wider landscape of mt quality-or indicating clearly when a metric has only been tested on high-quality mt-is an important factor for researchers to consider when building, presenting, and applying new metrics (especially if those metrics will be applied to lower-quality outputs). many other research efforts on meta-evaluation of metrics also focused on their ability to correlate with human judgment. @xcite introduced williams' significance tests for understanding the confidence of the correlation analysis. @xcite pointed out that pearson's correlation is sensitive to outliers and proposed to remove outliers in pearson's correlation analysis at system level. @xcite proposed to use pairwise accuracy to evaluate metrics based on whether the metric's pairwise rankings of two systems agrees with human pairwise rankings. @xcite introduced a tie calibration procedure enabling fair comparison between metrics that do and do not predict ties for pairwise accuracy analysis at the segment level. @xcite and @xcite studied the relationship of metrics' score differences and statistical significance of ranking decision. notably, these works are mostly based on the data released by wmt metrics shared task. that means the translation output scored by the metrics in these work were generated by the participants of the wmt news/general translation shared task, typically consisting of high-quality mt output. there is growing interest in understanding metric performance beyond correlation. @xcite note that neural metrics are not interpretable at the segment level across different language pairs. the wmt metrics shared task introduced the challenge sets subtask @xcite to challenge metrics on particular translation errors, including negation and polarity, word/sentence addition/omission, tokenization, punctuation, numeric expression, casing number swapping, spelling, etc., in order to shed light on metric strengths and weaknesses. the challenge sets created by @xcite ; avramidis et al. (2020); avramidis and macketanz (2022) were more linguistically motivated and covers more than 100 phenomena, including tenses, relative clauses, idioms, focus particles, etc. the aces challenge set @xcite covers 146 translation directions and 68 types of errors, ranging from simple perturbations to more complex errors based on discourse and real-world knowledge. the smaug challenge set @xcite and the hwtsc challenge set @xcite focused on the robustness of metrics on translation errors involving named entities, numeric/date/time entities, etc. we note that even as mt evaluation metrics become better at correlating with human judgment on translation quality for high-quality mt systems, metric performance may be untested on low-to medium-quality mt output. hence, we design the mslc23 challenge set to gain insights of metric behavior on a more complete landscape. ## challenge set the challenge set consists of data translated by mt systems of varying quality. we describe the training data used to build these systems as well as the mt models. ## general mt submissions our challenge set has aimed to cover the range of low-quality mt systems, but to obtain a fuller picture, we also include the metric scores assigned to the systems submitted to the wmt2023 general mt task @xcite in our analysis. for these systems, we have human annotation scores in the form of multidimensional quality metrics (mqm; burchardt, 2013) scores for @xmath4, @xmath5, and @xmath6. ## metrics there are dozens of metrics submitted by the task organizers and participants in wmt23 metrics shared task. under the time and space limitations, we only examine the baseline metrics submitted by the task organizers and the primary metrics submitted by the participants. due to the random shuffling of items in the challenge sets before their delivery to the scorers, we can only examine metrics that produce scores at the segment level, as the systemlevel scores returned do not correspond to the underlying systems in our datasets. we describe the metrics included in this work in appendix f. ## analysis we are interested in metric performance and characteristics at both the segment level and the system level. in the case of @xmath7, the segments are paragraphs, while in all other cases they are typically sentences. for metrics that use the reference, he↔en are scored against refb (a higher-quality reference translation than refa), while @xmath8. for @xmath9, @xmath10, and @xmath11, we have access to human scores for all submitted wmt mt systems (but not for the challenge set systems). these take the form of mqm scores over a consistent subset of the test set. for the remainder of this work unless otherwise noted, in order to make appropriate comparisons between metric scores on the challenge set, metric scores on the submitted wmt mt systems, and the human annotations, we restrict our analysis to only those segments that are in the news domain and that correspond to the set for which we have human annotations (104 paragraphs for @xmath12, 619 segments we use all 516 segments of the test set data that are in the news domain because we do not have any human annotations). ## conclusions this challenge set expands the range of system quality scored by metrics at the shared task. this expanded range of mt quality reveals interesting characteristics and limitations of some new metrics when applied to a broader range of systems. the smaller variations in segment-level scores given by some metrics at the low end of quality could indicate that these metrics struggle to discriminate lowquality mt systems. this is further shown by the observation that some metrics rank the low-quality systems in reverse order at the system level. we have discovered a "universal score" phenomenon for some metrics, where a small subset of nonminimum/maximum distinct scores are assigned to a variety of translation output. this characteristic was not visible in the high-quality mt output alone, highlighting the importance of this type of testing. we also observe diverse behaviors from different metrics on empty string translation. our challenge set serves as a complement to the standard correlation-based analyses and also provides useful information to researchers who are considering using these metrics in low-resource or low-quality domains. we recommend that metric researchers check their metrics' performance on a wider landscape of translation quality or be clear about the limitations of their metrics' testing.