On the Stability of System Rankings at WMT (2021)

Abstract: The current approach to collecting human judgments of machine translation quality for the news translation task at WMT – segment rating with document context – is the most recent in a sequence of changes to WMT human annotation protocol. As these annotation protocols have changed over time, they have drifted away from some of the initial statistical assumptions underpinning them, with consequences that call the validity of WMT news task system rankings into question. In simulations based on real data, we show that the rankings can be influenced by the presence of outliers (high- or low-quality systems), resulting in different system rankings and clusterings. We also examine questions of annotation task composition and how ease or difficulty of translating different documents may influence system rankings. We provide discussion of ways to analyze these issues when considering future changes to annotation protocols.

## introduction at the wmt (now conference on machine translation) shared task on news translation, research groups build machine translation systems to accurately translate news data, as tested on test sets of recent news documents. the systems are clustered and ranked on their performance as judged by human annotators. the way that human judgments of translation quality have been collected has varied over the course of wmt's history. in this work, we examine how changes in the collection of human judgments over the last three years have resulted in rankings that are now less robust to the effects of outliers (high-or lowperforming systems) and overall annotation task composition. we replicate the human judgment rankings from 2018-2020, perform simulations for reranking, and examine issues of annotation task composition and translation difficulty. we find that sampling sentences for annotators to annotate by document -intended as a step towards evaluating sentences in context -reintroduces a known problem from the earlier era of relative rankings, namely that systems suffer or benefit in their rankings based on the quality of the other data being rated alongside them in the same annotation tasks. we begin with a discussion of the progression of direct assessment (da) styles employed in wmt evaluations ( §2) and how scoring is performed ( §3), before delving into theoretical and practical understandings of the z-scores used to rank systems ( §4 and §5), including simulations and analysis of specific examples. we also discuss issues around document distribution and translation difficulty ( §6), and close with considerations for downstream impacts ( §7) and future study ( §8). ## historical context in 2016, wmt added direct assessment (da) scoring of system outputs as an investigatory ranking, with relative ranking (rr) remaining the official scoring mechanism (bojar et al., 2016). in relative ranking, five system outputs for a given segment were ranked in comparison to one another, from which pairwise translation comparisons were generated; these were then used to produce overall system rankings by means of the trueskill algorithm (herbrich et al., 2007; sakaguchi et al., 2014). relative ranking can be used to compare systems, but does not provide an absolute score, thus obscuring how close a good system is to a "perfect" translation or, at the other extreme, how poor a system is as compared to others. the following year, 2017, wmt adopted da as its main assessment format on the basis of high pearson correlations between rr and da in the previous year's investigations (bojar et al., 2017). in da (graham et al., 2013, 2014, 2016), annotators provide an absolute numerical score (0-100) for mt output adequacy (at the sentence level or at the document level) using a sliding scale. the use of da has changed since it was first introduced to wmt. in 2016, it was trialed for monolingual evaluations of translation fluency and monolingual evaluations of adequacy. here we provide an overview of changes from the 2016 task to the present, based on findings papers descriptions. bojar et al. (2016) noted that the 2016 version of da assessments has the potential to avoid a known bias of the rr setup. in rr, each rating task consisted of ranking the outputs of five systems on the same input segment, and "a system may suffer more losses if often compared to the reference, and similarly it may benefit from being compared to a poor competitor" (bojar et al., 2011). in the 2016 da setup, translations were annotated in sets of 100, including quality assurance tasks, but each segment was annotated individually, rather than in direct comparison to other system output for the same segment. foot_0 quality assurance tasks can include references (which should score highly), "bad" references (which should score poorly; these are produced by randomly replacing substrings in references to degrade quality), and repeat assessments of a segment (which should be scored consistently). in 2017, da was adopted as the main annotation style, with exact duplicate segment translations being able to be scored just once (rather than once per system that produced them) and with human assessment scores "standardized according to each individual human assessor's overall mean and standard deviation score" (bojar et al., 2017). bojar et al. (2018) describes two setups for the 2018 da tasks, a standard structure (with repeat pairs, "bad" references, and references, as quality assurance) and an alternate setup where an additional constraint was imposed, such that within each 100-translation task, for each input the task would include the corresponding output of all mt systems. this makes a tradeoff between the aim of da (to make absolute score judgments rather than relative ones) and getting a single annotator to provide scores for all systems' output of the same source input (which risks reintroducing some form of relative judgement to the task). this is also the first year that the findings paper explicitly spells out the goal of the way tasks (referred to here using the amazon mechanical turk nomenclature "human intelligence task" or hit) are built in the standard hit structure: [...] within each 100-translation hit, the same proportion of translations are included from each participating system for that language pair. this ensures the final dataset for a given language pair contains roughly equivalent numbers of assessments for each participating system. this serves three purposes for making the evaluation fair. firstly, for the point estimates used to rank systems to be reliable, a sufficient sample size is needed and the most efficient way to reach a sufficient sample size for all systems is to keep total numbers of judgments roughly equal as more and more judgments are collected. secondly, it helps to make the evaluation fair because each system will suffer or benefit equally from an overly lenient/harsh human judge. thirdly, despite da judgments being absolute, it is known that judges "calibrate" the way they use the scale depending on the general observed translation quality. with each hit including all participating systems, this effect is averaged out. @xcite the 2018 shared task also introduced sourcebased da, trialling a bilingual version of the task. rather than scoring mt output against a reference, this version scores it against the source segment, which allows human references to be scored as a "human system" rather than solely as a qa task. they raise a number of potential cautions against drawing strong conclusions, namely that bilingual da is not yet validated, the alternate task structure may introduce biases, the year's sample size for source-based da was smaller than 1,500 judgments per system, and that there may be quality issues with some reference segments. in 2019, wmt introduced additional versions of da (barrault et al., 2019). they used monolingual (reference-based) assessment for translation into english and for language pairs that did not include english at all. for translation out of english, they performed bilingual (source-based) da. the style of da used in previous years is renamed to sr-dc (segment rating without document context), as a new style, sr+dc (segment rating with document context) is introduced. in the new sr+dc style, the full translation of a single docu-ment by a single mt system is shown to the annotator in order (but still scored segment-by-segment); 3 a task consists of multiple such documents. the generation of annotation tasks is described as follows: all documents translated by all systems are pooled, then sampled (without replacement) until up to 70 segments are selected, at which point quality control documents are added, and finally the order of documents in the task is shuffled. barrault et al. (2020) uses both sr-dc and sr+dc styles. ## scoring in order to experiment with questions surrounding human evaluation, it is necessary to understand and be able to replicate the official scores produced by wmt. for the human annotations of interest (segment-level evaluation, with or without document context), there are two main types of scores: raw scores and z-scores, with the latter used as the official ranking. these are presented in a table, ordered by z-score, and clusters of systems deemed statistically significantly different (according to a wilcoxon rank-sum test @xmath0.05) are separated by horizontal lines. @xcite following the approach used at wmt, after removing any hits deemed unacceptable due to quality issues, we calculate raw and z-scores for systems as follows. first, any worker id whose scores have a standard deviation of 0 is removed. given a raw score x generated by the worker with worker id w , its corresponding z-score z is computed as we note that the 2019 and 2020 document context (sr+dc) evaluations differ in their quality assurance (see we reimplemented the scoring system using python and plan to release code for this paper. we were able to exactly replicate the raw scores and z-scores for most of the language pairs of interest from 2018-2020, 7 as well as the significance clusters. 8 see appendix a for details. we use this reimplementation of the wmt scoring scripts in ordownloaded from 2018-2020 wmt websites: http://www. statmt.org/wmt18/results.html , http://www. statmt.org/wmt19/results.html , and http:// www.statmt.org/wmt20/results.html . for 21 language pairs annotated in sr-dc style and 25 in sr+dc style from 2018-2020, we were able to exactly replicate rankings, nearly replicate rankings (e.g., with rounding difference related changes to one significance line), or produce rankings whose differences could be explained by delays in data collection (2020 en-iu). @xcite appendix a provides more details on replication. we use our recalculated rankings and clusters as the starting point for all remaining analysis in this paper. ## understanding z-scores while we've described how the z-score is calculated in the setting of the wmt human annotations, it's important to take a closer look at z-scores to understand how they behave in different scenarios. in this section, we explore z-scores and their underlying assumptions in hypothetical scenarios. given a raw score x, a mean µ, and a standard deviation σ, the z-score (or standard score) is the number of standard deviations above or below the mean that x falls. the z-score for a given raw score x can be computed as follows: this is a linear transformation; the shape of the distribution of z-scores is the same as that of the raw scores, but now with a mean of 0 and a standard deviation of 1. it is a unitless score. intuitively, the z-score provides a potential way of comparing scores from different annotators, but it requires a careful examination of underlying assumptions. if we think of the z-score as a unitless score, perhaps we can think of each annotator as having their own measurement units: we might have a lenient annotator and a harsh annotator, such that a raw score of 50 by the lenient annotator is quite bad while a raw score of 50 for the harsh annotator is actually quite good. in order to directly compare the two annotators' scores, we would like to map them to a shared scale, a unitless z-score. under what assumptions is it appropriate to calculate z-scores to compare annotators' scores? we start with perhaps the most obvious (but frequently unstated) assumption: there exists some latent "quality" of a given translation, which can be judged by a human annotator, such that annotators roughly agree about what constitutes a "good" or a "bad" translation. in practice, human annotators may disagree -for any number of reasons (basile et al., 2021) -about which of two translations of "similar quality" is better, but we assume that the disagreement is not extreme; i.e., we hope that under a correlation coefficient like pearson's r or spearman's ρ, the correlation between annotators' scores would be much closer to 1 than to -1. for the sake of simplicity in the following examples, we will assume there exists a "true" and "objective" score for every translation. suppose that we have some translations with a true mean score of µ and a true standard deviation of σ. a lenient annotator scores all of the translations such that the distribution of their scores has a mean of µ + n and a standard deviation of σ, while a harsh annotator scores all of the translations such that the distribution of their scores has a mean of µ -m and a standard deviation of σ. @xcite when we compute their z-scores, it is easier to directly compare sentence scores, since they are now on the same scale. this seems like a reasonable use of z-scores, but in this scenario annotators are scoring exactly the same data, which doesn't scale to wmt-style annotations; annotators simply don't have time to score all of the data. now suppose that we have two disjoint sets of sentences scored by two different annotators: the set s x of sentences scored by annotator x and the set s y of sentences scored by annotator y . from these raw scores, we can compute µ x and µ y along with σ x and σ y . if µ @xmath1, can we conclude that annotator x is a more lenient annotator than annotator y and resolve this by computing z-scores? not without additional information! imagine that we could see the "true" mean scores of s x and s y , as annotated by a perfect omniscient annotator. it could be the case that the true means are identical and annotator x is indeed more lenient, but it could also be the case that the true mean of the scores in set s x is actually higher. in the latter case, the annotators could be equally lenient, or it is even possible that annotator y could be more lenient! in short, without a shared basis for comparison, we don't know whether computing z-scores is normalizing out annotator differences, differences in the data itself, or a combination. ## z-scores in practice this raises the question: what is happening in practice when we compute z-scores on wmt das? are we really normalizing away inter-annotator differences, or is the normalization also doing something else, such as normalizing away real differences in hit and system quality? if it is the latter, even z-scores for das may suffer the same bias from comparisons to better (or worse) systems. we don't have access to an oracle, and we don't have a direct or reliable way to compute interannotator agreement, because in some collections it is rare that two annotators annotate the same text (and for the appraise data, we only have hit information, not annotator information). however, we can still examine this in the existing data and modifications thereof. bojar et al. (2011) noted that systems might suffer from being compared to the reference too frequently under relative ranking, or might benefit from being compared to particularly poor systems. the same could hold true in da. consider the following toy example: a hit contains 4 sentences, with raw scores of 25, 50, 50, 75, respectively. a sentence with a raw score of 50 in this hit would have a z-score of 0. if, instead, the raw scores were 0, 25, 50, 75, a sentence with a raw score of 50 would have a z-score of 0.39, while for a hit with raw scores of 25, 50, 75, 100, a sentence with a raw score of 50 would have a z-score of -0.39. while it is possible that such a set of scores could reflect differences in annotator behavior, we could also easily imagine that they might reflect differences in hit composition, with one containing only system scores, one containing system scores and a bad reference, and one containing system scores and a (good) reference. ## documents in both sr+dc and sr-dc styles, we don't have a guarantee that every segment-system pair is judged by an annotator, nor that at least one segment from every document-system pair is judged. if we assume approximately uniform translation difficulty across the test set, this isn't necessarily too much of a concern. however, is that really the case, or are some documents "easy" and others "hard"? figure 5 shows a matrix of document-system pairs, with each cell showing the average of all of the segment raw scores for that system-document pair. @xcite the documents are ranked from highest average raw score to lowest average score (top to bottom), while the systems are ranked by highest average raw score to lowest average raw score (left to right). in the leftmost column, we see the "hu-man" system, which has high scores across all documents. if all documents were equally difficult to translate, we would expect to see a gradient along the x-axis (i.e., across systems), with minimal variation along the y-axis (i.e., across documents). what we observe instead in this en-lt pair from 2019 (and across a number of other language pairs) is a rough gradient from the top left to the bottom right (with the exception of the "human" system, which remains strong throughout). this suggests that there are some documents that are "easy" for most systems to translate (top) and some that are "hard" (bottom). this raises a concern: when we attempt to compare two systems of very similar quality, they are not being measured on the same test set. an unlucky sample of documents might see one system judged on a "harder" set of documents, calling the resulting rankings into question. ## downstream consequences while researchers building mt systems for the shared task may view the human judgment rankings as the end result, the rankings are the input to the metrics tasks at wmt. thus the reliability of the rankings has a direct impact on the reliability of the metrics task -which in the long term feeds into mt research as researchers decide which automatic metrics to use for evaluating their systems. in system-level metric evaluation at the wmt metrics shared task, pearson correlations are computed between metric scores and the z-score human rankings (mathur et al., 2020b). note that these correlations are directly between the system average z-scores and the metric scores, and as such do not treat all systems within a given cluster as tied. in practice, this means that even rank-only perturbations in the official ranking can be expected to cause changes to metrics task results. metrics scores are run on the full test set, not the various human-annotated subsets. citing graham et al. (2013), the metrics task papers note that system-level da scores are "consistent and have been found to be reproducible" even though different sets of segments are assessed for each system. however, that work predates the shift to sampling by document, and our analysis of instability and document difficulty suggest revisiting it. recent work has shown that outliers have a concerning impact on metric correlations (mathur et al., 2020a), and organizers have worked to mitigate this (mathur et al., 2020b). this paper is a step towards answering questions raised in mathur et al. (2020b) regarding outliers and unfair advantages. it may seem tempting to remove outliers from human judgment tasks, but this will not solve the other problems and could instead mask their presence. ## proposals for future work the issues discussed in this paper raise concerns about changes to the human evaluation protocols used at wmt and their effects on the validity of wmt system rankings. a partial solution would be to return to sr-dc annotations, perhaps after validation of the 2018 alternate hit structure that guarantees that for every segment in the hit, the hit contains every mt system's output for that sentence. but this may be an unsatisfactory conclusion, and fails to address the interest in pushing mt evaluation toward whole documents. document-level and context-inclusive evaluations are growing in popularity, but there is limited study on document-level assessment methodologies for mt. castilho (2021) examines setups comparable to sr-dc, sr+dc, and document rating with document context (which we omitted from this work), and finds in a controlled experiment using likert scale ratings that a methodology comparable to sr+dc produces higher levels of interannotator agreement and fewer misevaluations than either whole document scores or individual sentences without context. however, that experimental setup does not suffer from the same task composition issues we observe in wmt; in fact these may be orthogonal issues. if the choice is made to use sr+dc style annotations, there are some improvements to consider, but as noted in castilho (2020), it remains "essential to test which methodologies will be best suited for different tasks and domains" prior to adopting them. one option would be to create 2018-alternate-structure style hits with document context, where a hit contains all systems' output for one or more documents. the downside to this is that it would likely require longer hits or hits that only contain a small number of documents; if systems are of similar quality, we might be concerned about annotator fatigue from repetition. the amount of context needed to adequately assess translations is still a question under consideration (castilho et al., 2020; castilho, 2021), which ties into issues of document and hit length. another possibility to consider would be to al-ways normalize over annotators (rather than over hits), but this isn't a solution on its own -it is still necessary to make sure that annotators see comparable distributions of systems and documents, or the same problems will be reintroduced. having annotators do calibration hits, i.e., a set of annotations that all annotators complete, could also be considered. the calibration hits would provide a consistent basis for computing the parameters of an annotator-specific z-score transformation, which could then be applied to the remainder of the annotator's judgments. this could untangle the issue of annotator strictness/leniency, but would still merit study before implementation (as annotator behavior may depend on hit composition, so the z-scores learned in calibration may not be as applicable as one might hope if there is a mismatch between calibration hits and the remainder of the hits). one could also consider additional ways of modeling annotator behavior beyond z-score normalization (paun et al., 2018). a simpler starting point to deal with the issue of different systems being annotated over different documents would be to guarantee that all systems are scored over the same subset of documents. all of these are (partially) orthogonal to the questions of what type of annotation tasks result in the most reliable ratings -whether it be direct assessment, ranking, or detailed error annotation -or questions of annotator skills and knowledge (freitag et al., 2021). ## conclusions we have shown that the current judgment collection methodology at the wmt news translation task results in sr+dc judgments that are more prone to variation on the basis of outliers than sr-dc judgments, and that hit composition issues have helped reintroduce the relative ranking problem of unfair comparisons to the wmt rankings. we examined issues of document difficulty and how this interacts with the decision to sample documents (rather than sentences) for judgment. these issues risk undermining the validity of wmt rankings, with real consequences for mt research and downstream tasks on automatic metrics. in examining these issues, we've also presented several approaches to diving into the wmt ranking data that may be helpful to consider when planning future changes to wmt human judgment collection procedures.