On the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation (2020)

Abstract: Document-level evaluation of machine translation has raised interest in the community especially since responses to the claims of “human parity” (Toral et al., 2018; Läubli et al., 2018) with document-level human evaluations have been published. Yet, little is known about best practices regarding human evaluation of machine translation at the document-level. This paper presents a comparison of the differences in inter-annotator agreement between quality assessments using sentence and document-level set-ups. We report results of the agreement between professional translators for fluency and adequacy scales, error annotation, and pair-wise ranking, along with the effort needed to perform the different tasks. To best of our knowledge, this is the first study of its kind.

## introduction increasing efforts have been made in order to add discourse into neural machine translation (nmt) systems. however, the results reported for those attempts are somehow limited as the evaluation is still mostly performed at the sentence level, using single references, which are not able to recognise the improvements of those systems. the stateof-the-art automatic evaluation metrics have been shown to underestimate the quality of nmt systems @xcite , and the suitability of these metrics for document-level systems has also been criticised @xcite . for that reason, document-level human evaluation of machine translation (mt) has raised interest in the community recently as it enables the assessment of suprasentential context. in a survey with native speakers, @xcite tested the context span for the translation of 300 sentences in three different domains, namely reviews, subtitles, and literature. over 33% of the sentences tested were found to require more context than the sentence itself to be translated, and from those, 23% required more than two previous sentences to be properly translated. ambiguity, terminology, and gender agreement were the most common issues found to hinder translation. moreover, differences in issues and context span were found between domains. this shows that document-level evaluation enables the assessment of textual cohesion and coherence types of errors which are impossible at times to recognise at sentence level. recent attempts to assess quality at the document-level were described in @xcite and @xcite who independently reassessed the bold claims of mt 'achieving human parity' and found that the lack of extra-sentential context has a great effect on quality assessment, and pointed to a failure of the current best practices in mt evaluation. @xcite used consecutive single sentences to rank translations by two mt systems and a human reference. they found that the evaluators were able to better assess the translations when provided with more context, and moreover, inter-annotator agreement (iaa) between professional translators was higher than that between non-experts. however, this methodology does not discriminate sentence vs document-level set up as single sentences are shown consecutively. @xcite used pairwise rankings of fluency and adequacy to evaluate the quality of mt vs human translation (ht) for document-level texts. the methodology consists of translators choosing the 'best' translation in terms of i) adequacy and ii) fluency, that is, instead of choosing on a scale on how fluent or adequate the translations are, the raters just choose the 'best' one. although not reporting iaa in the main paper, the authors report some iaa scores in the appendix of that work, showing that for fluency, document-level set up has higher iaa than for sentence set-up, but the opposite for adequacy. however, while this evaluation methodology seems appropriate when comparing different translations, it would not be feasible when evaluating a single mt system. after these two papers were published, for the first time the wmt19 attempted a document-level human evaluation for the news domain. in that year, the direct-assessment (da) task asked crowdworkers to give a score (0-100) regarding the accuracy of the translated sentence, where only one mt output is shown each time (no comparison with other mt system). however, conventional kappa cannot be using with da to measure iaa, and so consistency is measured instead, where raters have to pass some quality control criteria @xcite . in light of this, a comparison of iaa between quality assessments on sentence and documentlevel set-ups is needed in order to determine which set-up results in most reliable evaluation. this study presents a small-scale comparison on the differences in iaa between these two methodologies. to the best of our knowledge, this is the first paper to compare iaa for sentence vs document-level setups using the state-of-the-art mt evaluation metrics, namely fluency and adequacy scales, foot_0 error mark-up and pairwise ranking, along with reporting effort indicators. we provide a detailed description of the experimental methodology in section 2. following, we report results in section 3 on the agreement between professional translators for fluency and adequacy scales, error annotation, and pair-wise ranking (in 3.1), along with the effort needed to perform the different tasks (in 3.2). we discuss our results and draw conclusions in section 4, and point out directions for future work. ## methodology five professional english (en) to brazilian portuguese (pt-br) translators were hired to perform the evaluation in terms of (1) fluency, adequacy, and error mark-up using the pet tool @xcite ; and (2) pairwise ranking using google spreadsheet. the choice of pet was due to the fact that the tool is a free toolkit, easy to use, with its ui resembling translation tools, and it is able to handle different evaluations while logging time test set 1 test set 2 av. sentence length (wpd) 316 344 av. sentence length (wps) 20 21 av. sentence count (spd) 15 15 total words 10135 11019 total sentences 500 500 total docs 32 32 the evaluation was carried out in two scenarios: (a) evaluation at the sentence level, showing randomised sentences, one at a time, one score per sentence, and (b) evaluation at a document level, showing randomised documents, one document at a time, one score per document. after each scenario was complete, translators answered a posttask questionnaire about the evaluation and their perceived effort. corpus -we used the english corpora from wmt newstest2019, which has an average document length of 17 sentences (minimum 4 sentences, maximum 30 sentences). in total, 64 full documents were selected (32 per scenario) with 1000 sentences (500 per scenario). we made sure that both scenarios are comparable in terms of sentence and document length, as well as in terms of readability and lexical variation. results on statistics of both data sets in the corpus was then translated from en into pt-br with google translate (for adequacy, fluency, and error mark-up) and also with deepl for the ranking pairwise comparison. 2 translators -five professional translators took part in the evaluation. 3 their professional experience ranges from 6 to 14 years, and three of them have had previous experience with translation evaluation. a warm-up task with 20 sentences was set up so translators could get acquainted with the tasks, guidelines and tools, as well as clarify any doubts about the task. each translator evaluated 1000 sentences, 500 in each scenario and test set. ## discussion this paper attempts to shed light on the differences in iaa between sentence and document-level evaluation scenarios. the experiments performed with five professional translators have tested the stateof-the-art metrics commonly used to assess mt quality with humans, namely the assessment of fluency, adequacy, error mark-up and pairwise ranking @xcite . we note that when evaluating adequacy ( in addition to scores per test sets, it is interesting also to look at the iaa scores by translator pairs. we observe that there is a large difference between t1&t2 who evaluated test set 1 in the sentencelevel scenario, and test set 2 in document-level scenario, against t3&t4 who evaluated the opposite. t1 and t2 tend to disagree more in both test sets for both fluency and adequacy assessments, while t3&t4 have closer iaa scores and higher pearson correlation. the addition of t5's assessments, reported in terms of fleiss kappa, indicate that for the majority of the case, iaa is indeed higher when evaluation is performed at a sentence level. figure 1 shows examples of disagreement between translators. in example (1), t1 assessed the text as containing "little" of the meaning of the source, t2 considered it to contain "all of it", and t5 assessed it as containing "most of it" (adequacy). t1 comments that "many mistranslations of golf/sport terms impaired meaning" and "some unstranslated terms found ('team usa', 'singles')", while t2 thinks that the text contains "minor issues, but the meaning isn't lost", and t5 says "the meaning is compromised by the word-by-word translation". while both t1&t2 agree that the text is "near native" regarding fluency, t5 assess it as having "little fluency", mentioning that fluency is also "compromised by the literal translation of some terms". for t3&t4, the disagreement in the documentlevel (example (2)) is not as strong. while t3 assesses it as containing "most of the meaning", t4 thinks that it contains "little of it" because "there are a couple of plays on words in the source text, a big part of the translation is lost". however both agreed that regarding fluency, example (2) has "little fluency". we speculate that the disagreements at the document-level scenario, especially for the adequacy evaluation, might be connected to the fact that because the texts are made up of "very good", "reasonably" and "poorly" translated sentences which, together, make the text understandable to a certain level, it is harder for translators to be consistent when assigning one single score for a full text. we estimate the percentages of adequacy scores for the document-level scenario as follows: t1&t2&t5 show 0% for score 1 (none of it), 7.29% for score 2, 61.46% for score 3, and 31.25% for score 4 (most of it); while t3&t4 show 4.69% for score 1, 17.19% for score 2, 64.06% for score 3, and 14.06% for score 4. these results show that a great number of scores fall into the middle category which makes it difficult for a consistent evaluation on a document-level scenario. consequently, this type of problem will be persistent when evaluating at a document-level mt systems that operates at the sentence level, because a document translated with sentence-level nmt is still a sequence of translated sentences rather than an entire document translation. we observe that disagreements in the sentencelevel are more often related to ambiguity and lack of context. in example (3), while t1 commented that the translation "failed to use football terminology" and assessed it as containing "none of the meaning", t4 and t5 assessed it as containing "all of the meaning". we speculate that t4 and t5 were unaware that the sentence was about football due to the lack of context, and did not penalise mistranslations such as 'fired' which is better translated as 'chutar' (to kick) and 'box' which should be translated as 'pequena área'. t5 even mentioned that they had problems with the word "hazard" because eve thought "it seems to be a noun as it starts with a capital letter, i could not assess whether "hazard" is a proper noun or just a noun, due to lack of context". example (4) is another example of lack of context, since the pronoun "it" is impossible to identify in the sentence. while t3 decides to rely only on the context given and assess adequacy as "all of it" and fluency as "native", t4 assesses the sentence as containing "little of the meaning" and "little fluency". according to t4, "the context was not enough to translate the pronoun 'it'". this is consistent with findings in @xcite where authors found that over 33% of the surveyed sentences required more context than the sentence itself to be translated. indeed, with the context of previous sentences it is possible to identify that "it" relates to "a radical plan" and therefore the addition of "o plano veria" (the plan would see) in the translation would make it more adequate: interestingly, t1, t2 and t5 who assessed this sentence in context in the document-level scenario agreed that the text was "near native" and contained "most" and "all" of the meaning. this might be another indication that fluency is better assessed at a document level. regarding error mark-up assessment ( finally regarding effort, unfortunately the logged time in pet tool was not decisive, even witht he addition of t5's assessments (due to discrepant results by t1 ( ## conclusions and future work the present work attempts to shed light at the differences in iaa when evaluating mt at the sentence and document levels with a small scale comparison. the main key findings of this comparison is that, a document-level evaluation methodology where translators assign one score per text leads to lower levels of iaa for adequacy, ranking, and error mark-up (when compared to methodologies where translators assign one score per sentence), but it might be useful for fluency assessments. this is consistent with @xcite findings on iaa for pairwise comparison, and previous work on nmt evaluation, where fluency proved to be harder to assess (than adequacy) in sentence-level scenarios. nevertheless, we also speculate that as google translate seems to operate on a sentence-level, a document-level evaluation of adequacy is penalised since a document can be constituted of sentences with different levels of quality. moreover, we consider whether multiple scores per document (sentence pairs, paragraphs, and word-level error tagging) will yield higher levels of iaa when compared to the randomised sentence-level set-up for both sentence and document-levels mt systems. human-evaluation of mt in document-level setups is in its infancy, and therefore, it is essential to test which methodologies will be best suited for different tasks and domains. future work will use more translators and different methodologies, as expressed in the post-task questionnaire and discussed above, with more specific guidelines for context-span issues found in previous works, and the development of test-sets, as well as using document-level mt systems' outputs.