On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs (2021)

Abstract: Recent studies emphasize the need of document context in human evaluation of machine translations, but little research has been done on the impact of user interfaces on annotator productivity and the reliability of assessments. In this work, we compare human assessment data from the last two WMT evaluation campaigns collected via two different methods for document-level evaluation. Our analysis shows that a document-centric approach to evaluation where the annotator is presented with the entire document context on a screen leads to higher quality segment and document level assessments. It improves the correlation between segment and document scores and increases inter-annotator agreement for document scores but is considerably more time consuming for annotators.

## introduction recently, several studies have suggested that document context is required for the reliable human evaluation of machine-translated documents @xcite . with the improved performance of neural machine translation systems (nmt) over the past years, this is particularly important when assessing the potential for human parity or super-human performance of mt systems @xcite . following these recommendations, the wmt conference on machine translation foot_0 has moved towards adopting and presenting document context in their human evaluation campaigns of 2019 and 2020 @xcite . the wmt campaigns are the largest academic efforts on human evaluation of machine-translated news articles in the field, running yearly since 2007. at wmt19, the previous segment-level direct assessment evaluation @xcite where translated segments were presented to evaluators foot_1 in random order -was extended by introducing "segment ratings with document context" @xcite , and assessments of both, individual segments and entire documents, were collected. in this approach, segments from a single document translated by the same mt system were provided sequentially to evaluators in the order as they appear in the document, only one segment shown at a time (fig. 1a ), followed by the entire document comprised of already scored segments (fig. 1b ). wmt 2020 @xcite implemented a more document-centric approach, displaying the full translated document on a single screen (fig. 1c ) for most of the out-of-english language pairs. while the change was primarily about the user interface (ui), we believe it can impact the quality of document-level evaluation to a large extent. @xcite has noticed potential issues arising from the limited inter-sentential context in the wmt19 method, in which the evaluator does not have continuous access to all segments from the document. unable to revisit previous sentences and never seeing subsequent sentences, the evaluator might forget or lack access to important details necessary to rate the current segment. on the other hand, displaying a long document on a screen can notably increase cognitive load, potentially lowering reliability of assessments over time @xcite , and increase annotation time and costs, especially at the scale of the wmt evaluation campaigns. in this work, we compare human assessment scores collected during the last two wmt evaluation campaigns and analyze the impacts of the user interface changes between these campaigns. we also attempt to determine whether switching to the document-centric ui was an improvement to the human evaluation procedure and should be adopted in future editions of wmt for all language pairs. we examine if and to what extent human raters make use of the document context, estimate the reliability of document ratings collected through both interfaces, and study potential additional costs resulting from the document-centric evaluation at a large scale. ## document context in human evaluation of mt outputs recent research emphasized the importance of document context in human evaluation of machine translation, especially in terms of accessing potential human parity or super-human performance @xcite @xcite . several works have compiled sets of recommendations for document-level evaluation. for example, @xcite recommend evaluation of documents instead of independent sentences as translators tend to judge machine translation more favourably if they cannot identify errors related to textual coherence and cohesion due to lack of context. @xcite have examined the necessary context span needed for evaluation across different domains, and for relatively short documents like news articles, the authors recommend presenting the whole document during the assessment of individual segments. using document context has also been recommended by @xcite who reported that this information was needed for evaluators to rank systems in a contrastive evaluation setting. having the text available during the assessment of fluency or adequacy might be essential for some evaluators who spend more time reading than assessing @xcite . although the literature is consistent about the need of document context in human evaluation of mt, little research has been done on the impact of experimental design and user interfaces on annotator productivity and the reliability of assessments in this context. the existing research on experimental designs for machine translation evaluation focuses on contrasting direct assessments with pairwise rankings @xcite and not on the optimal presentation of the document-level information. however, even the simple ui design decision of aligning document translations on the sentence level impacts efficiency of some evaluators @xcite . with this work, we want to promote that direction of research. ## document-level human evaluation campaigns at wmt during the wmt evaluation campaigns of 2019 and 2020, segment and document-level assessments of document translations were collected, but using different methods and thus user interfaces. both were implemented in the appraise evaluation framework @xcite as a source-based direct assessment task @xcite , i.e. all segments and entire documents were judged on a continuous scale between 0 and 100 by bilingual annotators. ## human assessment data in our experiments, we utilize the human assessment data collected at the wmt19 and wmt20 evaluation campaigns. we limit the data to outof-english language pairs as the into-english evaluation at wmt20 was done using the wmt19 method of reference-based da and assessed by crowd workers instead of translators and researchers. each annotator account provided 200 segment-level scores, and a number of documentlevel scores depending on the length of documents in the annotator's sample. from our analysis, we exclude all documents that contain one or more quality control segments, which constitute about 12% of all segments. 3 we use similar amounts of assessments from both campaigns, as seen in wmt19 wmt20 avg. std. avg. std. ∆ (%) annotation time for a task (200 seg.) 1:06:08 ± 21:47 1:51:09 ± 51:12 +68.1 total time for @xmath0. 02:02 ± 01:00 02:48 ± 01:44 +37.1 total time for @xmath1. 06:54 ± 02:48 12:01 ± 07:53 +74.0 time for single segment score 00:16 ± 00:06 00:24 ± 00:13 +47.4 time for single document score 00:12 ± 00:09 00:06 ± 00:04 -42. 7 ## experiments on wmt data we aim at comparing the wmt19 and wmt20 interfaces for segment and document-level human assessments of mt outputs by analyzing the data that has been collected using both methods. we analyze annotation times, compare correlations of document and averaged segment ratings, and examine the inter-annotator agreement. ## discussion in the presented experiments, we have observed interesting differences in correlation and interannotator agreement for long documents. in wmt19, for long documents, the correlation between segment-level scores and document-level scores significantly decreases, while iaa flattens out and eventually ends up being higher than for wmt20. we think this might be an effect of cognitive overload when annotators are presented with long document translation text pairs without visual help in the form of sentence alignment and similar hints. 5 a large wall of text might discourage annotators and they might fall back to assigning default or less diverse "safe" scores. analyzing annotation times in relation to the document length, presented in figure 4 supports this explanation. the average time of document ratings flattens for documents longer than 20 segments for wmt19, while it increases for wmt20. another non-intuitive observation we have made is that the inter-annotator agreement for averaged segment scores is higher in wmt19 than in wmt20. the agreement for document scores is, as expected, consistently higher for wmt20. if this is not solely attributed to the different data sets used in both campaigns, we would explain it by a tendency of annotators to assign higher scores if they cannot identify errors due to insufficient context our study is conducted post-hoc, i.e. we cannot test for scenarios that were not anticipated during the actual evaluation campaigns. a more conclusive interpretation would require a/b testing with the same sets of documents, translations and annotators used for both evaluation methods. nevertheless, we think that the presented comparison of two wmt evaluation campaigns supports the assumption that the document-centric evaluation conducted during wmt20 produced more reliable document ratings. we believe this to be an important finding because higher quality of collected document assessments should help to avoid statistical issues arising from low statistical power as observed by @xcite . ## summary in this work, we have compared two methods for document-level human evaluation of mt outputs through an analysis of the large-scale human assessment data from wmt evaluation campaigns, consisting of 8 different out-of-english language pairs. our main findings are: our analysis suggests that not only the entire document context is needed for reliable human evaluation of news translations, as recent studies have shown, but that the method in which the context is presented to evaluators is also important for collecting good-quality segment and documentlevel assessments. we conclude that the wmt20 method produces more reliable ratings, and thus can be adopted for future editions of the wmt document-level human evaluation campaigns for all languages. in future work, we plan to strengthen our findings by comparing the wmt19 and wmt20 methods in a/b testing with common sets of documents, translations and annotators for both settings. annual conference of the european association for machine translation, pages 185-194, lisboa, portugal. european association for machine translation. antonio toral, sheila castilho, ke hu, and andy way. 2018. attaining the unattainable? reassessing claims of human parity in neural machine translation. in proceedings of the third conference on machine translation: research papers, pages 113-123, belgium, brussels. association for computational linguistics.