Document-Level Machine Translation Evaluation Project: Methodology, Effort and Inter-Annotator Agreement (2020)

Abstract: Document-level (doc-level) human eval-uation of machine translation (MT) has raised interest in the community after a fewattempts have disproved claims of “human parity” (Toral et al., 2018; Laubli et al.,2018). However, little is known about bestpractices regarding doc-level human evalu-ation. The goal of this project is to identifywhich methodologies better cope with i)the current state-of-the-art (SOTA) humanmetrics, ii) a possible complexity when as-signing a single score to a text consisted of‘good’ and ‘bad’ sentences, iii) a possibletiredness bias in doc-level set-ups, and iv)the difference in inter-annotator agreement(IAA) between sentence and doc-level set-ups.

## introduction although currently an active community is working on developing document-level (doc-level) mt systems, their evaluation has primarily been performed at the sentence level. in 2019, for the first time, wmt19 attempted a doc-level human evaluation for the news domain, after considering criticisms by @xcite and @xcite regarding the current best practices in mt evaluation. both papers independently reassessed the claims of mt "achieving human parity" and found that the lack of extra-sentential context has a great effect on quality assessment. in a recent survey with native speakers, @xcite tested the context span for the translation of three different domains (reviews, subtitles, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. and literature). results show that over 33% of the sentences tested (300 in total) required more content than the sentence itself to be translated, and from those, 23% required more than two previous sentences to be properly translated. some of the issues which the participants found to most hinder the translation include word ambiguity, terminology, and gender agreement. moreover, the authors found that there are differences in issues and context span between domains. this shows that doclevel evaluation enables to assess suprasentential context, textual cohesion and coherence types of errors. in one of the few studies on doc-level evaluation, @xcite use pairwise rankings of fluency and adequacy in which raters give one single score to the full document. for wmt19, the direct assessment task asked crowdworkers to give a single score (0-100) to full documents for accuracy, where only one mt output is shown each time (no comparison with other mt system). with that in mind, this project aims at identifying methodologies to better cope with the sota human metrics, namely ratings of fluency and adequacy, error mark up and ranking evaluations @xcite . we will gauge the complexity when assigning a single score to full texts, since they can consist of 'good' and 'bad' sentences, which could mean that instead of a single score, translators would prefer to give scores to different chunks of the texts while seeing the whole text. we will investigate the difference in iaa between sentence and document level set-ups. furthermore, a possible tiredness bias in doc-level set ups will also be investigated, for example, the extend to which translators judge a long text on the quality of its first sentences. for that end, we will run a series of experiments with the wmt newstest2019, with professional translators. ## methodology the evaluation setup is made up of two sequential stages: (1) fluency/adequacy and error markup, and (2) pairwise ranking. four professional translators will carry out the tasks in two scenarios: (a) evaluation at the sentence level, showing randomised sentences, one at a time, and (b) evaluation at a document level. while scenario a will be the baseline as it follows common practice in mt evaluation, scenario b will show how translators will make the decisions and what influences them when they have to give one score for full texts. after each task, translators will answer a post-task questionnaire about the tasks. the documents and scenarios are randomised to avoid participants evaluating the same source twice. the corpus used is the wmt newstest2019 english corpora, which has an average document length of 17 sentences (minimum 4 sentences, maximum 30 sentences). full documents that amount to 1000 sentences are selected, totalling 64 documents. the english documents are translated into brazilian portuguese with google translate for stage 1, and with both google translate and deepl for stage 2. the choice of language is because as it is the principal researcher's mother tongue, this will make it possible to analyse it more carefully and see possible patterns in the process. moreover, being portuguese a romance language, it is possible that the results of this pilot can be extended to the language family. the tasks are set in two tools. for fluency, adequacy, and error mark up, pet tool @xcite is used as it allows time tracking. for the ranking tasks, an online spreadsheet is used and extension to track time is also implemented. translators are also requested to keep track of their time while performing the evaluation. after stages 1 and 2 are complete, a second round of evaluation will designed. this time, doclevel evaluation will be performed with translators giving one score per chunks/sentence in the text while having access to the full document. that way we will be able to compare effort and iaa between the two methodologies for doc-level. ## final remarks this project aims at shedding light at methodology, effort and iaa and systematically improve human evaluation of mt at the doc-level. preliminary results will be available by june 2020, and data sets will be fully available at the end of the project.