Agree to Disagree: Analysis of Inter-Annotator Disagreements in Human Evaluation of Machine Translation Output (2021)

Abstract: This work describes an analysis of inter-annotator disagreements in human evaluation of machine translation output. The errors in the analysed texts were marked by multiple annotators under guidance of different quality criteria: adequacy, comprehension, and an unspecified generic mixture of adequacy and fluency. Our results show that different criteria result in different disagreements, and indicate that a clear definition of quality criterion can improve the inter-annotator agreement. Furthermore, our results show that for certain linguistic phenomena which are not limited to one or two words (such as word ambiguity or gender) but span over several words or even entire phrases (such as negation or relative clause), disagreements do not necessarily represent “errors” or “noise” but are rather inherent to the evaluation process. %These disagreements are caused by differences in error perception and/or the fact that there is no single correct translation of a text so that multiple solutions are possible. On the other hand, for some other phenomena (such as omission or verb forms) agreement can be easily improved by providing more precise and detailed instructions to the evaluators.

## introduction despite the large number of automatic evaluation metrics designed for machine translation (mt) evaluation 1 which represent invaluable tools for rapid development of mt systems, human assessment of translation quality remains the gold standard, both for evaluating mt systems as well as for developing suitable automatic metrics. human evaluation is typically provided in one of the following ways: assigning an overall quality score to each translated sentence, ranking two or more translations of the same source language sentence from best to worst, or annotating actual translation errors. the errors can be only highlighted (marked) or corrected (postedited), but can also be classified according to a 1 http://www.statmt.org/wmt20/ metrics-task.html given pre-defined scheme, such as mqm scheme. 2 however, human judgments of translation quality show a high degree of variance @xcite , especially for fine-grained error classification based on a detailed error scheme involving many error types @xcite . one of the reasons for the variance is the variety of possible solutions: there is no single objectively correct translation of a given text, but rather a range of possible translations from perfect over good to acceptable. moreover, there is no single universal criterion for translation quality. although all manual evaluations are essentially based on some of the following three criteria: adequacy (meaning preservation; the most frequently used), fluency (grammar of the target language; frequently used) and comprehension (readability; rarely used), the precise definition of the criterion is not always given to the annotators. very often, an unspecified mixture of adequacy and fluency is used. in addition, other factors like target audience, goal of translation, etc. can have influence on evaluator's perception of quality. most publications dealing with human evaluation of mt, such as @xcite @xcite @xcite @xcite , report an overall inter-annotator agreement (iaa) score such as percentage of equal labels, koehn's κ, fleiss' κ, krippendorf's α, or similar. however, less work has been done on analysing the actual disagreements. ranking and assigning overall scores are the mostly used methods and a large amount of annotated data is publicly available (for example in wmt shared tasks 3 ), but these methods do not provide enough information for further analysis. on the other hand, error annotation (with or without classification) does. some publications about error classification calculated separated iaa scores for different error categories and reported that some categories seem to be "more difficult" to agree on than others @xcite . however, to the best of our knowledge, systematic analysis of the relations between different linguistic phenomena and inter-annotator disagreements has not been carried out yet. in this work, we analyse inter-annotator disagreement for a set of different linguistic phenomena. we used two publicly available data sets containing annotated errors without classification. we choose this type of error annotation because we believe that it can better reflect the differences in annotators' perception of errors than error classification: it is not bound to any predefined error scheme which might be tailored for a specific task and/or language pair, so that the evaluators had freedom to mark any part of the text which they perceive as problematic. ## related work while publications usually report only an overall inter-annotator agreement, many researchers analysed the disagreements in natural language processing evaluation from different points of view. @xcite argues that standard iaa coefficients should not represent the only criterion for checking the reliability of human evaluation of natural language generation due to natural variability of human language, and suggest that correlation coefficients should be used. some researchers compared iaa for different evaluation methods, such as ranking vs assigning overall scores. belz and kow (2010) compared the two methods for evaluating natural language generation, and @xcite ; @xcite for machine translation. all reported that evaluators generally agree more in ranking. castilho (2020) compares iaa for three evaluation methods for machine translation, namely ranking, assigning scores and error classification, but only in the context of evaluating isolated sentences vs evaluating larger amounts of text (paragraphs, "documents"). recent studies which investigated disagreements in natural language inference @xcite and semantic annotation @xcite claim that disagreement in natural language evaluation is often expected due to ambiguity and variation of language. therefore, a number of disagreements do not represent "errors" or "noise" but are fully legitimate. on the other hand, @xcite argue that inter-rater disagreements are not necessarily due to inherent properties of the language, but at least in part to the annotation task being underspecified. however, none of the publications analysed the actual disagreements and error perception of different annotators. one of the first publications in this direction deals with iaa for error classification for mt using the mqm error scheme @xcite . they reported that different degree of agreement can be observed for different error types, not always for the same reasons. for example, disagreement is high for word oder errors due to observed error span: while annotators agree that there are problems with word order in the given part of the sentence, they do not agree about exact words which should be marked as errors. on the other hand, evaluators often mixed up "mistranslation" and "terminology" because some evaluators found it difficult to distinguish the difference between the error types. the same error scheme was used in @xcite . although the focus was not on analysis of disagreements, kappa coefficients were presented for different error types showing that the annotators agreed to a large extent on untranslated words as well as on number, gender or case errors, while most disagreements were coming from omissions and tense errors. contrary to @xcite , the agreement for order errors was high, however this analysis was carried out on the sentence level, not on the word level, therefore diminishing the word-level span problem. another publication dealing with different error types reports results for natural language generation @xcite . similarly to @xcite , they report that some error types are more difficult for annotators to agree on than others, as well as the word span plays an important role for disagreements. also they say that some of the observed disagreements could have been resolved by more detailed annotation instructions, while others are more fundamental. to the best of our knowledge, none of the publications analysed inter-annotator disagreements in terms of underlying linguistic phenomena. ## data sets the main part of our experiments was carried out on the "qrev" data set @xcite . 4 the set consists of english user reviews translated into croatian and serbian. for each of the target languages, five different nmt systems were used: three online systems (amazon translate, microsoft bing and google translate) and two in-house systems @xcite based on the sockeye foot_3 implementation, one designed for translating general domain, and the other for translating user reviews. in total, the data set contains outputs of ten different mt systems. two quality criteria were used for highlighting errors: adequacy and comprehension. an important difference between the two criteria is that seeing the source text was required for marking adequacy errors while seeing the source text was forbidden for marking comprehension errors. for both quality aspects, the evaluators were asked to concentrate on problematic parts of the text and to highlight them. for adequacy, they were instructed to highlight parts which entirely or partially change the meaning of the source text. for comprehension, they were asked to mark parts which are impossible or hard to understand. all translations were evaluated in context -the evaluators were seeing entire reviews. in total, 15 evaluators, computational linguistics students and researchers fluent in the source language and native speakers of the target language, participated in the annotation. the largest part of the text (about 3000 sentences) is annotated by two evaluators, while a small part of the text (about 40 sentences) is annotated by three or four evaluators. inter-annotator agreement in terms of krippendorf's α is 0.61 for adequacy errors and 0.51 for comprehension errors. we also worked on a small "hu-manmt " foot_4 @xcite data set consisting of english ted talks translated into german by one in-house mt system. this data set was not created for purposes of mt evaluation, but for improving an nmt system by giving it feedback about errors. since the used loss function did not support omissions and reordering errors, the evaluators are specifically asked not to highlight these two types of errors. the evaluators were not given any specific quality criterion: they were only instructed to "highlight the errors", which usually implies a mixture of adequacy and fluency in mt evaluations, but was not specified. the part of this corpus annotated by multiple evaluators is very small, only 11 sentences, however each of them was annotated by 5-10 evaluators, university students with fluent or native english and german skills. the reported krippendorf's α is much lower, 0.201. due to all above mentioned limitations of the corpus, it is not sufficient to draw any conclusions. still, qualitative inspection of inter-annotator disagreements was helpful to confirm some of the hypotheses based on the results on the qrev corpus. an overview of the two data sets is presented in ## analysis of disagreements in order to analyse the nature of differences in error perception between two evaluators, the first step was to identify the nature and causes of the highlighted errors, namely the underlying linguistic phenomena. the second step was then to calculate the overlap of words perceived as errors by two annotators for each identified underlying phenomenon; higher overlap indicates higher agreement between the annotators. ## results for each of the two quality criteria, the overall overlap (agreement) together with the overlap for the most interesting identified phenomena for the qrev data set are presented in ## conclusions this work attempts to shed light on the differences between error perception of different annotators for evaluation of machine translation output. the main findings are that the quality criterion as well as underlying linguistic phenomena have a strong influence on error perception. for some of the phenomena, such as omission or verb forms, agreement can be increased by providing more detailed annotation instructions. for others, especially those with a large word span such as negation or rephrasing, the differences are inherently subjective and therefore hard to completely avoid. this is, nevertheless, not necessarily bad, but it is important to be aware of it. while improved agreement certainly is an important goal, the exact nature of disagreements where perception of errors does not result in the same annotation can also provide insight into how humans perceive the translation quality. based on these findings, our recommendations for human evaluation are: for named entities: for verb forms: