Consistent Human Evaluation of Machine Translation across Language Pairs (2022)

Abstract: Obtaining meaningful quality scores for machine translation systems through human evaluation remains a challenge given the high variability between human evaluators, partly due to subjective expectations for translation quality for different language pairs. We propose a new metric called XSTS that is more focused on semantic equivalence and a cross-lingual calibration method that enables more consistent assessment. We demonstrate the effectiveness of these novel contributions in large scale evaluation studies across up to 14 language pairs, with translation both into and out of English.

## introduction while machine translation systems are typically evaluated with automatic metrics like bleu @xcite , the gold standard for quality assessment is evaluation of machine translation output by human evaluators. in fact, the validity of automatic metrics is justified by correlation to human evaluations. however, in practice individual human evaluators apply very different standards when assessing machine translation output, depending on their expectation of translation quality, their exposure to machine translation output, their language abilities, the presentation of source or reference translation, and vague category descriptions like "mostly correct". this is especially a problem when the goal is to obtain meaningful scores across language pairs, to assess, for instance, if a machine translation system for any given language pair is of sufficiently high quality to be put to use. we address this problem of high variability and cross-lingual consistency by two novel contributions: (1) a new scoring metric xsts that is focused on meaning; and, (2) an evaluation protocol that allows for calibration of scores across evaluators and across language pairs. our studies show that the xsts score yields higher inter-annotator agreement compared against a 5-point raw scale. we also show that our calibration leads to improved correlation of system scores to our subjective expectations of quality based on linguistic and resource aspects as well as improved correlation with automatic scores. ## related work the darpa evaluation of the 1990s tasked human evaluators to assign scores from 1 to 5 to judge the fluency and adequacy of translations @xcite , with vague definitions like much meaning for an adequacy score of 3 or the slightly offensive non-native english for a fluency score of 3. this scale was also used in the first human evaluation of the workshop on statistical machine translation (wmt) @xcite . note that these evaluations aim at a different goal than the one we are concerned with here: their main purpose is to rank the output of different machine translation systems against one another -without the need to report a meaningful score that is an absolute measure of their translation quality. hence, it should come as no surprise that the wmt evaluation then moved towards pairwise comparisons of different system outputs @xcite . for many years, evaluators were asked to rank up to 5 system outputs against each other. due to the problem that for n systems, o(n 2 ) pairwise comparisons need to be done @xcite , recent wmt evaluations switched to direct assessment @xcite . evaluators are required to indicate absolute quality of a machine translated sentence using a slider which is converted into a score on a 100 point scale. such finer grained scores allow for easier normalization of scores between annotators. direct assessment is also used by microsoft for shipping decisions @xcite . google uses a 5-point scale to evaluate their machine translation systems but specifics have not been published. recently, @xcite proposed the multidimensional quality metrics (mqm) framework, rooted in the need for quality assurance for professional translators, that aims at generating meaningful scores. in mqm, fine-grained error categories like omission, register and capitalization are assessed and the error counts per category are combined into a single score. such fine-grained errors can typically only be detected in relatively high-quality translations @xcite . this metric is predominantly used for quality assurance in the translation industry to evaluate translations from professional translations. ## a new metric: xsts we propose a new metric that is inspired by the semantic text similarity metric (sts) used in research on paraphrase detection and textual entailment @xcite . the metric emphasises adequacy rather than fluency. we do this for several reasons but mainly because we deal with many low resource language pairs where preservation of meaning during translation is a pressing challenge. arguably, assessing fluency is also much more subjective and thus leads to higher variance. another reason is that we are interested in evaluating the translation of social media text where the source and reference translation may be disfluent, so lack of fluency should not be counted against machine translation. as in many previously proposed scoring rubrics, we use a 5-point scale. for a detailed definition of the meaning of each score, see figure 1 . there are various ways this metric could be used. the examples in the figure show two english sentences, such as machine translation output and a human reference translation, but our core evaluation protocol presents the source sentence and corresponding machine translation to a bilingual evaluator. different from previous evaluation protocols, xsts asks explicitly about meaning (semantic) correspondence, all the more while obfuscating which sentence is the source and which is the translation. note that the score has a fairly high bar for a score of 4: semantic equivalence, only allowing for differences in style, emphasis, and connotation. this allows us to detect differences in quality at the very high end. we experimented with both this 5 point scale and a reduced scale where the categories 4 and 5 were collapsed. example a (different topics): text 1: john went horseback riding at dawn with a whole group of friends. text 2: sunrise at dawn is a magnificent view to take in if you wake up early enough for it. example b (similar/related topics): text 1: the woman is playing the violin. text 2: the young lady enjoys listening to the guitar. the two sentences share some details, but are not equivalent. some important information related to the primary subject/verb/object differs or is missing, which alters the intent or meaning of the sentence. example a (opposite polarity): example b (word order changes meaning) text 1: they flew out of the nest in groups. text 1: james voted for biden. text 2: they flew into the nest together. text 2: biden voted for james. example c (missing salient information): example d (substitution/change in named entity) text 1: "he is not a suspect anymore." john said. text 1: i bought the book at amazon. text 2: john said he is considered a witness but text 2: the book was purchased at barnes and not a suspect. noble by me. there cannot be any significant conflicts in intent or meaning between the sentences, no matter how long the sentences are. example a (minor details that are not salient to the meaning): text 1: in may 2010, us troops invaded kabul. text 2: the us army invaded kabul on may 7th last year, 2010. example b (minor verb tense and/or unit of measurement differences): text 1: he bought 2 lbs of rice at whole foods. text 2: he buy 1 kg. of rice at wholefoods. example c (small, non-conflicting differences in meaning): text1: she loves eating ripe apples in the fall. text2: she usually eats ripened apple in autumn. example d (omitted non-critical information, but no contradictory info introduced): text1: several of the sailors set out on a rainy tuesday morning. text2: several of the sailors set out on a tuesday morning. example a (different level of formality): text 1: this is europe the so-called human rights country text 2: this is europe, the country of alleged human rights example b (added sense of urgency, advertising style): text1: special bike for more info call 0925279927 text2: special bike for more information call now 0925279927 5 the two sentences are exactly and completely equivalent in meaning and usage expression (e.g., formality level, style, emphasis, potential implication, idioms, common metaphors). example a (same style and level of formality): example b (disfluency is not penalized): text 1: what's up yu'all? text 1: one two three apples oranges green text 2: howdy guys! text 2: one two three apples oranges green ## cross-lingual consistency via calibration sets even after providing evaluators with instruction and training, they still show a large degree of variance in how they apply scores to actual examples of machine translation output. this is especially the case when different language pairs are evaluated, which necessarily requires different evaluators assessing different output. we address this problem with a calibration set. note that we are either evaluating x-english or english-x machine translation systems. in either case, this requires evaluators who are fluent in english. hence, we construct a calibration set by pairing machine translation output from various x-english systems with human reference translations -so that the evaluators compare two english sentences. the sentence pairs are carefully chosen to cover the whole range of scores, based on consistent judgments from prior evaluation rounds. evaluators assess this fixed calibration set in addition to their actual task of assessing translations for their assigned language pair. we then compute the average score each evaluator gives to the calibration set. if this evaluator-specific calibration score is too high, then we conclude that the evaluator is generally too lenient and their scores for the actual task need to be adjusted downward, and vice versa. there are various ways how scores for each evaluator could be adjusted. after exploring various options, we settled on a simple linear shift (with an option for moderating large calibration shifts or shifts near the edges of the scale if desired). to give an example, if the consensus score for the calibration set is 3.0 but an evaluator assigned it a score of 3.2, then we deduct 0.2 from all their scores for the actual evaluation task. ## study design we report on two large-scale human evaluation studies to assess the two novel contributions of this work. the first study compares xsts and its variants against other evaluation methods like a raw 5-point scale modelled (raw) after direct assessment. the second study assesses the effectiveness of our calibration method. language pairs we selected languages with the goal to cover both high-resource languages with good machine translation quality and low-resource languages with weaker machine translation quality. the languages also differ in writing system, morphological complexity, and other linguistic dimensions. see the selection of evaluators evaluators were selected for each language pair and they evaluated both language directions (english-x and x-english). the evaluators were professional translators who were recruited by a translation agency. they had to have at least three years of translation experience, be native speakers of the language x, high level of english proficiency, and pass through a training process (detailed documentation of the task and training examples). we speculate, but have not yet tested, that the xsts protocol may be employable by evaluators with less rigorous translation training than traditional da; so long as they still have high levels of fluency in both languages being evaluated. this could potentially facilitate the evaluation of low-resource languages where qualified annotators may be difficult to source. user interface and training since we are working with language service providers who subcontract the work to professional translators who differ in their technical setup, we do not always have full control over the way text is presented to them and how they register their evaluations. throughout our studies, the employed tools vary from simple spreadsheets to a customized annotation tool similar to the one used in wmt evaluations. machine translation systems most of the machine translation systems used in this studies were trained in-house with fairseq @xcite on public data sets at different times in 2020 and 2021, each designed to optimized translation quality given available data and technology. the most recent system, used in the calibration study, is a 100-language multilingual system, similar to the one developed for the wmt 2021 shared task @xcite . test set the translated sentences to be evaluated are selected from social media messages and wikipedia -the later being part of the flores test set which comprises close to 200 languages at the time of writing @xcite . note that social media messages have the additional challenge of disfluency and creative language variation in the source sentence. ## conclusion we introduced two novel contribution to the human evaluation of machine translation for multiple language pairs and validated their effectiveness in industrial-scale user studies: we proposed figure 2 : adjusted x-english scores using calibration set increases correlation with bleu.