Data Sampling and (In)stability in Machine Translation Evaluation (2023)

Abstract: We analyze the different data sampling approaches used in selecting data for human evaluation and ranking of machine translation systems at the highly influential Conference on Machine Translation (WMT). By using automatic evaluation metrics, we are able to focus on the impact of the data sampling procedure as separate from questions about human annotator consistency. We provide evidence that the latest data sampling approach used at WMT skews the annotated data toward shorter documents, not necessarily representative of the full test set. Lastly, we examine a new data sampling method that uses the available labour budget to sample data in a more representative manner, with the goals of improving representation of various document lengths in the sample and producing more stable rankings of system translation quality.

## introduction human evaluation of machine translation (mt) quality is very labour-intensive, meaning that it is not always possible to have full test sets annotated by human evaluators, for example in large-scale shared tasks like the news/general mt task at the conference on machine translation (wmt) @xcite . the typical practice in such situations is sampling a subset of the test set for human annotation to estimate the quality of mt systems; the rankings of mt systems computed from this are expected to be stable and representative of the full test set. however, in practice, inconsistency and instability in system rankings are observed in human evaluation and are often blamed on human annotator inconsistency. thus, much of the focus on mt human evaluation is on denoising and calibrating human annotations, but there are other sources of error in the data sampling process orthogonal to annotator behavior. throughout wmt's history, the design of data sampling methods has been tangled up with ex-periments on human assessment collection approaches. in the early years when constrastive adequacy/fluency judgments and relative ranking were used @xcite @xcite @xcite @xcite @xcite , the nature of the assessment method (i.e., comparing system outputs directly) ensured that there were overlaps in the segments annotated for the various systems. however, when direct assessment @xcite @xcite was introduced @xcite , the annotation subsets were selected independently per system, which is expected to produce consistent rankings (assuming sufficient annotations). in practice, some years have seen language pairs with very low annotation (sampling coverage of 12.5%) opening the door to scenarios where mt systems could be evaluated and ranked on disjoint sets of sentences, raising questions about fairness and consistency. as wmt added document context to human evaluation @xcite , segments for annotation were sampled at the document level (rather than at segment level). this introduced another source of instability and error by limiting the representation of topics and vocabulary in annotated samples, introducing systematic bias in the samples (e.g., document length), and in some cases even preventing some documents from being sampled @xcite . although @xcite show that system performance can be resilient to adaptive overfitting against the frequently reused evaluation sets in other nlp task, they also show that nlp models' robustness to distribution shift remains a challenge. as mt research moves toward document level, document length may be related to the difficulty of the translation task. the data sampled for gold standard human evaluation should reflect the full test set to support fair system rankings and further analysis of distribution shift effects. as discussed, sample size or coverage is a major factor in how representative the annotation subset is; with high coverage, other sources of error are less likely to cause problems, with low coverage, the errors may have compounding effects on ranking stability. however, sample size is an aspect that may be tightly constrained (e.g., due to funding). in this paper, we study the data sampling methods and resulting instability of system rankings in wmt news/general mt task over the past four years. to focus on data sampling -separate from questions about human annotator consistency -we use automatic mt evaluation metrics to generate these system rankings. we show that system ranking consistency and representation of documents lengths in the sample can be improved by a new data sampling method that uses the available labour budget and balances the desires for document context and representativeness of the sample. ## data sampling methods we divide our discussion of data sampling methods into two orthogonal components: 1) whether the subset of data annotated for each systems is sampled independently per system or once per test set and 2) how the sampling is performed. ## simulation we show the effects of different sampling strategies by scoring segments in the sampled subset and the full test set with automatic metrics and comparing system rankings between the two. ## conclusions we examine three different approaches that wmt has used for sampling segments from test sets for human judgment, performing simulations using automatic metrics in place of human annotations. this allows us to examine a large range of scenarios at low cost, with the risk that it may not be fully representative of human judgment distributions. we demonstrate in simulation that a document budgeted snippet sampling approach finds a balance between providing document context, representation (i.e., better representing the distribution of document lengths), and ranking stability. additionally, we use this analysis to highlight problems and challenges in comparing past human annotation approaches. in particular, large and small variations in annotation procedures are often conflated and collapsed into overly-simplified descriptions that obscure the ways in which they differ from one another; we attempt to untangle some of these. we urge researchers to take care in examiningin isolation and in combination -the effects that various design decisions have on results, in order to build annotation approaches that remove as many sources of error as possible.