WMT 20 Document-Level Markable Error Exploration (2020)

Abstract: Even though sentence-centric metrics are used widely in machine translation evaluation, document-level performance is at least equally important for professional usage. In this paper, we bring attention to detailed document-level evaluation focused on markables (expressions bearing most of the document meaning) and the negative impact of various markable error phenomena on the translation. For an annotation experiment of two phases, we chose Czech and English documents translated by systems submitted to WMT20 News Translation Task. These documents are from the News, Audit and Lease domains. We show that the quality and also the kind of errors varies significantly among the domains. This systematic variance is in contrast to the automatic evaluation results. We inspect which specific markables are problematic for MT systems and conclude with an analysis of the effect of markable error types on the MT performance measured by humans and automatic evaluation tools.

## introduction this paper presents the results of our test suite for wmt20 news translation task. 1 the conclusion of @xcite , a last year's similar effort, states that expert knowledge is vital for correct and comprehensible translation of professional domains, such as audits or lease agreements. furthermore, even mt systems which make fewer mistakes and score above others in both automatic and manual evaluations are prone to making fatal errors related to markable conflicts, which render the whole document translation unusable. in this study, we aim to organize and describe a more detailed study with a higher number of annotators. we show three evaluation approaches: (1) automatic evaluation, (2) fluency and adequacy per document line and (3) detailed markable phenomena evaluation. we compare the results of this evaluation across the three domains and try to explain why all of these evaluations do not produce the same ordering of mt systems by performance. this paper is organized accordingly: section 1.1 defines the term "markable", section 1.2 describes the examined documents and section 2 introduces the two phases of our annotation experiment and shows the annotator user interface in section 2.3. in section 3, we discuss the results from both phases and also automatic evaluation. the main results of this examination are shown in section 3.5 and specific markable examples are discussed in section 4. we conclude in section 5. ## annotation setup for both phases of this experiment, we used 10 native czech annotators with english proficiency. none of them were professional audit or legal translators. because each annotator annotated only one or two documents, the aggregated results across domains, labelled as total, are of less significance than the results in individual domains. ## annotation examples in the following figures (figure 5 , figure 6 and figure 7 ) we show annotated excerpts with bleu, fluency, adequacy and markable phenomena severities. references are here to convey the czech source segment meanings. they were not shown to the annotators. examined markables are underlined. reference: this supplement no. 1 is written and signed in 2 (in words in words in words in words in words in words in words in words in words in words in words in words in words in words in words in words in words: two) copies, each of which is valid for the original. ## conclusion in this article, we compared three approaches to document translation evaluation. we saw that nonexpert annotators rate most mt systems higher than reference with fluency and adequacy, but reference ranks better than most of them when inspecting markable phenomena and their severity. inspecting specific instances in detail, we found out that mt systems made errors in terms of markables, which no human translator would do. relating the current observation with the impression last year, we conclude that annotators lacking in-depth domain knowledge are not reliable for annotating on the rather broad scales of fluency and adequacy but they are capable of spotting term translation errors in the markable style of evaluation. this is important news because expert annotators can not be always secured. unfortunately, the inter-annotator agreement remains generally low, possibly due to a high cognitive load with many systems annotated. we further examined these markable phenomena and showed that especially sense, other grammar and terminology kinds of errors negatively influence the fluency and adequacy the most. for bleu the variables of highest importance were non-translated and conflicting errors. in future work, we would like to examine more of the kinds of markable errors in modern mt systems and their influence on the translation quality. this description could then help researches focus on specific parts of their mt systems. furthermore, we would like to explore possible automated metrics, which would help in determining whether the document meaning remained intact with respect to markables. annotating markables appears to be easier for human annotators and more reliable for non-expert ones, and the results gave us more insight into the systems' performance than the fluency-adequacy method.