Towards Document-Level Human MT Evaluation: On the Issues of Annotator Agreement, Effort and Misevaluation (2021)

Abstract: Document-level human evaluation of machine translation (MT) has been raising interest in the community. However, little is known about the issues of using document-level methodologies to assess MT quality. In this article, we compare the inter-annotator agreement (IAA) scores, the effort to assess the quality in different document-level methodologies, and the issue of misevaluation when sentences are evaluated out of context.

## introduction the use of machine translation (mt) has now become widespread in many areas thanks to improvements in neural modelling @xcite @xcite . accordingly, researchers have attempted to integrate discourse into neural machine translation (nmt) systems. as a consequence, document-level human evaluation of mt has raised interest in the community as it enables a more detailed assessment of suprasentential context. however, the definition of document-level, in terms of how much of the text needs to be shown, is still unclear. moreover, although a few works have looked into documentlevel evaluation @xcite @xcite , little is known about the issues of using documentlevel methodologies to assess mt quality. the present research attempts to shed light on the differences in inter-annotator agreement (iaa) when evaluating mt with different methodologies, namely random single sentences, sentences in context, and full document scores. we also look into perceived effort from translators when evaluating the translations in the different methodologies. results have shown a good level of iaa with a methodology where translators are able to assess individual sentences within the context of a document compared to a methodology with random sentence assessments, while a methodology where translators give a single score per document yields low iaa. furthermore, we note that misevaluation cases recur in the random single sentences evaluation scenario. ## related work document-level machine translation evaluation has been raising interest in the mt field, however, only a few works have attempted to use documentlevel boundaries for mt evaluation. @xcite asked participants to post-edit and tag single sentences and full paragraphs in terms of cohesion and coherence. their results showed that more post-editing was performed in paragraphs which suggests several issues could only be solved with paragraph-wide context. the authors reported spearman's rank correlation for agreement which showed mixed to low agreement. @xcite used consecutive single sentences to rank translations (in terms of preferred translation) of two mt systems and a human reference. they found that, when provided with more context, evaluators were better able to assess the translations, and moreover, iaa between professional translators was higher than that between nonexperts. however, this methodology did not allow access to the full documents, as sentences were given one by one in order shown in the document. @xcite used pairwise rankings of fluency and adequacy to evaluate the quality of mt against human translation (ht) for documentlevel texts. the methodology consisted of translators choosing the 'best' translated documents in terms of i) adequacy and ii) fluency, that is, instead of choosing on a scale of how fluent or adequate the translations are, the raters just chose the 'best' one. the authors reported some iaa scores in the appendix of that work, showing that for fluency, document-level set-up had higher iaa than sentence set-up, but that the opposite was the case for adequacy. however, this evaluation methodology can only be used when comparing two different translations. castilho et al. (2020) tested the context span for the translation of 300 sentences in three different domains (reviews, subtitles, and literature) in a survey with native speakers. the results showed that over 33% of the sentences tested were found to require more context than the sentence itself to be translated or evaluated, and from those, 23% required more than two previous sentences to be properly evaluated. the most common issues found to hinder translation were ambiguity, terminology, and gender agreement. their results show that cohesion and coherence errors types cannot be recognised at sentence-level at times. in 2019, the fourth conference for machine translation (wmt19) 1 attempted document-level human evaluation for the news domain for the first time @xcite . their direct-assessment (da) 2 task asked crowdworkers to give a score (0-100) regarding the accuracy of the translated sentence, for one mt output. they asked raters to rate i) full documents, ii) single consecutive segments in original sequential order and iii) single random sentences. wmt20 @xcite ) modified the methodology and extended the context span to entire documents, asking raters to score individual segments whilst seeing the entire document, and also to judge the translation of the entire document. however, conventional kappa cannot be used with da to measure iaa, and so consistency is measured instead, where raters have to pass some quality control criteria. in light of this, a comparison of iaa between quality assessments on sentence-and documentlevel set-ups is needed in order to determine which set-up results in the most reliable evaluation. this study is a follow-up of results presented in castilho (2020) where we present a small-scale comparison on the differences in iaa between judgements given in isolated random sentences and entire documents. in the present study, we compare the iaa in evaluation of i) random single sentences, ii) evaluation of individual sentences while translators have 1 wmt is running since 2006 and had always performed evaluation solely at the sentence level until 2019 ( http:// www.statmt.org/wmt19/ ). ## towards a better human evaluation methodology for document-level the recent interest in document-level mt evaluation has raised a few questions in the area. for example, it is still not clear how much context needs to be shown in a document-level evaluation setup. this is important as we need to understand whether there is a pattern regarding how much context is required (in cases when full texts cannot be fully displayed or when they are not available) in order to have a reliable quality assessment and to avoid misevaluation issues. some studies have used consecutive sentences (showing one at time) @xcite , and a few have used full short texts @xcite . @xcite have shown that for a great number of sentences, their successful translation and their mt evaluation requires more than sentence pairs and sometimes even full texts. corroborating these findings, we also observe the need for a wider context in order to solve ambiguities in the evaluation. figure 1 shows examples of context span needed when evaluating translations from @xmath0. 9 to evaluate sentence 105, the translators need 9 the full speech can be found in the appendix i to identify the gender of the speaker in order to know whether "thank you" will be translated into the feminine (obrigada) or masculine (obrigado). for sentences 103 and 104, the translators need to know whether the pronoun "you" refers to singular or plural (você/vocês). moreover, in sentence 104, because of the verb "love", some syntax constructions would need to have the gender of the pronoun "you" determined (as amamos -f, os amamos-m). 10 the issue of gender in 104 might be solved with sentence 102 with the use of the term "young women", as they are the ones who need "to stand up and take the reins", and the speaker knows that "you can do it" ("to stand up and take the reins"). this might imply that the "you" is also female and unlikely to be male, i.e.: and we need strong, smart, confident young women to stand up and take the reins. we know you can do it, paul.× we know you can do it, mary. before sentence 102, it is only in sentence 52 that "you" is clearly identified as "women". regarding number (singular/plural), however, it is still not possible to affirm whether "you" in sentence 104 and 103 refers to singular or plural, with the context of sentence 102, because the one being talked to could still be singular, i.e.: and we need strong, smart, confident young women to stand up and take the reins. we know you can do it, girls. we know you can do it, mary. it is only with sentence 99 that the number of "you" is solved with the term "every single one of you", which indicates that the speaker is talking to more than one person. before that, it is only with sentences 52 and 54 that "you" is again identified as plural. regarding the gender issue in sentence 105, one can claim that sentence 95 "my husband works in figure 1 : examples of context span needed to solve gender and number issues in sentence 104 and 105. the parts in pink relate to the gender of the speaker, red parts relate to the number of "you" (singular/plural), orange parts relate to the gender of "you", and the green parts relate to the resolution of "it". these problems with context span show that it is still uncertain how much context translators need to see in order to identify the issues in the translation and assess translation quality accordingly. we have previously shown (castilho, 2020) that there is a risk of misevaluation when random single sentences are used in evaluation because of the lack of context. we also observe misevaluation issues in the present study, where disagreements in the rs scenario are more often related to ambiguity and lack of context. figure 2 shows two examples of misevaluation for sentences 104 and 105 when assessments were performed in the rs scenario. when comparing the scores assigned to sentence 104 ("we love you") in the rss and scs scenarios, we note that in the rs evaluation, as expected, all translators assessed the mt output as having all the meaning of the source, to be native, and free of er-rors. none of the translators commented on the fact that there are four possible translation for the pronoun "you" in the source (singular/masculine, singular/feminine, plural/masculine, plural/feminine) and only a wider context would determine gender and number in the sentence. translators who assessed sentence 104 in the sc scenario were able to find that the mt was not able to keep the gender agreement in the translation, foot_7 even when they erroneously did not consider the whole context in order to assess the sentence, as it is the case of t5. it is interesting that the scores for adequacy in the sc are quite divergent, while the scores for fluency are more homogeneous. this corroborates findings from our previous work castilho (2020) where we note that disagreements at the document-level are more related to adequacy errors. misevaluation was also observed in sentence 105: similar to the previous sentence, translators assessed the mt output of sentence 105 in the rs scenario as having all the meaning of the source, to be native, and free of errors. however, this time one translator (t4) commented on the fact that only a wider context would determine the gender in the sentence. translators who assessed sentence 105 in the sc scenario were able to find errors in the mt output. again, we note that t5 erroneously does not consider the whole context in order to assess the sentence. similar to sentence 104, the scores for adequacy in the document-level scenario are divergent, while the scores for fluency are more homogeneous. we speculate that the reason methodologies with random single sentences show higher iaa agreement is because raters tend to accept the translation when adequacy is ambiguous but the translation is correct, especially if it is fluent. thus, sentences like 104 and 105 are judged as correct in a scenario where there is no context to tell the evaluator why the translation should be different. therefore, higher iaa scores in rs methodologies do not necessarily mean translators agreed more because the mt output was in fact better. moreover, since nmt systems are known to have improved fluency, these types of misevaluation as shown previously are more likely to happen in a rss set-up. our results have shown that a methodology where translators are able to assess individual sentences within the context of a document (sc scenario) yields good level of iaa compared to rs scenario, while a methodology where translators give one score per document (ds) shows very low level of iaa. moreover, the scs methodology avoids the misevaluation cases which proved to be quite common in the rs evaluation set-ups. ## conclusions and future work the present work attempts to shed light on the differences in iaa when evaluating mt with different methodologies, namely random single sentences, sentences in context, and full document scores. the main finding of this comparison is that, an evaluation methodology where translators judge single random sentences might yield a better annotator agreement at times but with a high cost of misevaluation cases. moreover, a methodology where translators assign one score per text leads to lower iaa and a great level of effort. this corroborates the results seen in castilho (2020) where iaa scores for document-level reaches negative levels, and the level of satisfaction of translators with that methodology is also very low. in turn, evaluating the quality of mt output with individual sentences showed in the context of a document yields not only good iaa scores but avoids the issue of misevaluation which is extremely common in random single sentence evaluation set-up. we believe that a translator will be more inclined to accept as correct an ambiguous but fluent translation. this is problematic for an accurate evaluation of mt quality since it might lead to misevaluation especially when assessing the quality of nmt systems which are known to have an improved fluency level. therefore, we suggest that evaluation set-ups using random single sentences should be avoided. for future work, we will investigate the differences in context span needed for different domains, as well as whether the state-of-the-art metrics for human evaluation of mt (fluency, adequacy, error, ranking) must be modified in order to capture more realistically the quality level of the systems. european association for machine translation through its 2019 sponsorship of activities programme. the adapt centre for digital content technology ( www.adaptcentre.ie ) at dublin city university is funded by the science foundation ireland research centres programme (grant 13/rc/2106) and is co-funded by the european regional development fund.