DELA Project: Document-level Machine Translation Evaluation (2022)

Abstract: This paper presents the results of the DELA Project. We describe the testing of context span for document-level evaluation, construction of a document-level corpus, and context position, as well as the latest developments of the project when looking at human and automatic evaluation metrics for document-level evaluation.

## introduction the challenge of evaluating translations in context has been raising interest in the machine translation (mt) field. however, the definition of what constitutes a document-level (doc-level) mt evaluation, in terms of how much of the text needs to be shown, is still unclear @xcite . few works have taken into account doc-level human evaluation @xcite , and one common practice is the usage of test suites with context-aware markers. however, test suites with document-level boundaries are still scarce @xcite . the main objective of the dela project is to define best practices for doc-level mt evaluation, and test the existing human and automatic sentence-level evaluation metrics to the doclevel. we present here the results from the project to date, as well as the upcoming research to be carried out. ## doc-level evaluation methodology in @xcite , we tested the differences in inter-annotator agreement (iaa) between single-sentence and doc-level setups. first, translators evaluated the mt output in terms of fluency, adequacy, ranking and error annotation in: (i) one score per single isolated sentence, and (ii) one score per document. then, the doc-level setup was modified, and translators evaluated (i) random single sentences, (ii) individual sentences with access to the full source and mt output, and (iii) full documents. results showed that assessing individual sentences within the context of a document yields a higher iaa compared to the random single-sentence methodology, while when translators give one score per document, iaa is much lower. assigning one score per sentence in context avoids misevaluation cases, extremely common in the random sentences-based evaluation setups. 1 the higher iaa agreement in the random single-sentence setup is because raters tend to accept the translation when adequacy is ambiguous but the translation is correct, especially if it is fluent. using the issues found in @xcite , we developed the dela corpus, a doc-level corpus annotated with context-aware issues when translating from english into portuguese, namely gender, number, ellipsis, reference, lexical ambiguity, and terminology @xcite . the corpus contains 60 full documents and was compiled with six different domains: subtitles, literary, news, reviews, medical, and legislation; and can be used as a challenge test set, training/testing corpus for mt and quality estimation, and for deep linguistic analysis of context issues. foot_1 ## examining context-related issues using the dela corpus, we examine the shortest context span necessary to solve the issues annotated in the corpus, and categorise the types of contexts according to their position, and report the i) context position, and ii) context length we find that the shortest context span might appear in different positions in the document including preceding, following, global, world knowledge. the average length depends on the issue types as well as the domain. the results show that the standard approach of relying on only two preceding sentences as context might not be enough depending on the domain and issue types. ## latest developments the dela project, running until september 2022, will focus now on the human and automatic evaluation metrics for mt, testing and developing new ways to use them for doc-level evaluation. doc-level human and automatic evaluation metrics: the focus of the dela project is to answer the following research questions: i) are the stateof-the-art (sota) human and automatic evaluation metrics able to capture the quality level of the doc-level systems realistically?; and ii) can/should they be modified or do new ones are needed? a series of experiments with the sota human metrics are being carried out, informed by the best methodologies found in previous results. with that, we will determine whether these metrics can be used in doc-level evaluations, or if new metrics should (and could) be developed. the doc-level human evaluation will inform automatic metrics to