DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues (2021)

Abstract: Recently, the Machine Translation (MT) community has become more interested in document-level evaluation especially in light of reactions to claims of “human parity”, since examining the quality at the level of the document rather than at the sentence level allows for the assessment of suprasentential context, providing a more reliable evaluation. This paper presents a document-level corpus annotated in English with context-aware issues that arise when translating from English into Brazilian Portuguese, namely ellipsis, gender, lexical ambiguity, number, reference, and terminology, with six different domains. The corpus can be used as a challenge test set for evaluation and as a training/testing corpus for MT as well as for deep linguistic analysis of context issues. To the best of our knowledge, this is the first corpus of its kind.

## introduction machine translation (mt) is now widely used in a variety of fields, mainly due to advancements in neural models @xcite @xcite . as a result of these recent advances, scientists have been increasingly attempting to include discourse into neural machine translation (nmt) systems @xcite . thus, researchers started to consider a more suitable evaluation for these document-level systems as the standard mt automatic evaluation metrics have been shown to underestimate the quality of nmt systems @xcite and the appropriateness of these metrics for document-level systems has been challenged (smith, 2017) since they are not sensitive to their improvements @xcite . accordingly, document-level human evaluation of mt has attracted the community's attention since it allows for a more thorough examination of the output quality with context. while a few works have taken into account document-level human evaluation @xcite @xcite @xcite , one common practice for document-level evaluation is the usage of test suites with context-aware markers @xcite @xcite @xcite . however, the concept of documentlevel evaluation, in terms of how much text must be shown, remains uncertain @xcite . while most research on document-level mt evaluation works with contrastive pairs, very few works have tried to use full documents for human evaluation @xcite @xcite and challenge test sets @xcite . methodologies for assessing mt at the document-level have been looked into @xcite as well as the types of issues that come with different methodologies @xcite . we present a document-level corpus annotated with context-aware issues when translating from english (en) into brazilian-portuguese (pt-br). in total, 60 documents from six different domains (literary, subtitles, news, reviews, medical, and europarl) were annotated with context-aware issues, namely gender, number, ellipsis, reference, lexical ambiguity, and terminology. the corpus can be used as a challenge test set for the evaluation and as a training/testing corpus for mt and quality estimation, as well as for deep linguistic analysis of context issues. moreover, we believe that the annotation can be also used for close-related languages such as spanish. ## related work document-level mt evaluation has attracted interest in the field as it allows for the evaluation of suprasentential content, which in turn, provides more meaningful insights on the mt output. however, the definition of what constitutes a documentlevel mt evaluation is still unclear @xcite . context plays an important role as it is widely used in translation and interpreting literature @xcite , although it lacks a precise definition for practical purposes, including in everyday work of a professional translator @xcite . for melby and foster (2010, p 3), context in translation could be studied "either for the purpose of analysing existing translations or for the purpose of improving the production of new translations". for the authors, context can be categorised into nontext (non-linguistic variables) and text (linguistic aspects), where the latter is divided into four aspects of context: relating to the source text: co-text (the version of the document itself) and chron-text (past and future versions); and relating to other text: rel-text (monolingual related texts) and bi-tex (bilingual related texts). in this work, we adopt melby and foster's view of context that is important to the analysis of translations, and focus (i) on the co-text, i.e. the boundaries within the document translated, and (ii) in the non-text, where the name of the authors, speakers, and products have an effect on the translation. in a survey with native speakers, @xcite tested the context span for the translation of 300 sentences in three different domains, namely reviews, subtitles, and literature. the results showed that over 33% of the sentences tested were found to require more context than the sentence itself to be translated or evaluated, and from those, 23% required more than two previous sentences to be properly evaluated. the authors found that ambiguity, terminology, and gender agreement were the most common issues to hinder translation. moreover, differences in issues and context span were found between domains. their recommendations include to show whole documents when possible, include information on text type, topic, product, hotel and movie names in case of reviews, and include visual context whenever possible @xcite . this shows that document-level evaluation enables the assessment of textual cohesion and coherence types of errors which are impossible at times to recognise at sentence level. regarding overall mt evaluation, a few attempts have been made to perform human evaluation with document-level set-ups. @xcite compared sentence-level evaluation versus documentlevel evaluation with pairwise rankings of fluency and adequacy to evaluate the quality of mt against human translation (ht) with professional translators. their results show that document-level raters clearly preferred ht over mt, especially in terms of fluency. the authors argue that documentlevel evaluation enables the identification of certain types of errors, such as ambiguous words, or errors related to textual cohesion and coherence. the conference for machine translation (wmt), which has been running since 2006 and only evaluated sentences, attempted documentlevel human evaluation for the news domain for the first time in 2019 @xcite . their direct assessment (da) @xcite required crowdworkers to assign a score (0-100) to each sentence. they asked raters to evaluate (i) whole texts, (ii) single consecutive segments in their original order, and (iii) single random phrases. in the following year, wmt20 changed the approach and expanded the context span to include full papers, requiring raters to evaluate specific segments while seeing the complete document, as well as to assess the content's translation @xcite . @xcite tested for the differences in inter-annotator agreement (iaa) between single sentence and document-level set-ups. in castilho (2020), the author asked translators to evaluate the mt output of freely available online systems in terms of fluency, adequacy (likert scale), ranking and error annotation in two different set-ups: (i) translators give one score per single isolated sentence, and (ii) translators give one score per document. the results showed that iaa scores for the document-level set-up reached negative levels, and the level of satisfaction of translators with that methodology was also very low. nonetheless, it avoided cases of misevaluation that happen in isolated single sentences. following on from that work, @xcite modifies the document-level set-up and re-runs the experiment with more translators, where she compares the iaa in evaluation of (i) random single sentences, (ii) evaluation of individual sentences where translators have access to the full source and mt output, and (iii) evaluation of full documents. results showed that a methodology where translators assess individual sentences within the context of a document yields a good level of iaa compared to the random single-sentence methodology, while a methodology where translators give one score per document shows a very low level of iaa. the author demonstrates that the methodology of assigning one score per sentence in context avoids misevaluation cases which are extremely common in the random sentencesbased evaluation set-ups. moreover, the author posits that the higher iaa agreement in the random single sentence set-up is because "raters tend to accept the translation when adequacy is ambiguous but the translation is correct, especially if it is fluent" @xcite , and asserts that the single random sentence evaluation method should be avoided as the misevaluation issue is especially problematic when assessing the quality of nmt systems as they have an improved fluency level. one current way of evaluating document-level issues is the use of test suites designed to better evaluate translation of the addressed discourse-level phenomena. commonly, these test suites are contrastive, that is, each sample sentence in the test has both correct and wrong translations for a given phenomena @xcite @xcite @xcite . the higher the accuracy of the model in rating correct translations over incorrect ones, the better the quality is deemed to be. test suites with document-level boundaries are still scarce, e.g. @xcite present a test suite designed to evaluate coherence when testing mt models trained for the news domain on audit reports, and @xcite designed a document-level test suite to assess three documentlevel discourse phenomena, namely information structure, discourse connectives, and alternative lexicalisation of connectives. given the above, the need to move toward document-level methodologies in mt is indisputable. moreover, with the lack of resources for the topic of document-level mt, the documentlevel corpus annotated with context-aware issues presented here can be used as a challenge test set for evaluation and as a training/testing corpus for mt as well as for deep linguistic analysis of context issues. ## corpus compilation the corpus was collected from a variety of freely available sources. following a pre-determined list of context issues found in @xcite that hindered the translation of single sentences and sentence pairs, the annotators searched for challenging english texts for the mt systems when translating into pt-br. in total, 60 full documents (57217 tokens) were collected from six different domains: literary, subtitles, news, reviews, medical, and legislation (europarl). ## methodology for annotation following literature on document-level test suites (see section 2), together with issues found when trying to define how much context span is needed to translate and evaluate mt @xcite , we compiled a list of context-aware issues that are challenging for mt when translating from en into pt-br to be annotated: when evaluating the translation of the source sentence given by 3 different mt systems (google translate foot_7 (gg), microsoft bing foot_8 (ms) and deepl foot_9 (dpl) the translator has to evaluate all three systems' outputs as correct: gg: "e obrigado pelo caso." (masculine, police case) ms: "e obrigado pelo estojo." (masculine, pencil case) dpl: "e obrigado pela caixa." (masculine, box) that is because without a wider context, it is impossible to know the correct translation or the sentence, which should be: ht: "e obrigada pela capa." (feminine, phone case) therefore, the issues tagged in the corpus are issues that might arise in the translation of sentences when the full context is not given. annotators used different mt systems to help check for issues that would go unnoticed when only looking at the source text. moreover, a few modifications to the source text were performed in order to add those issues and make the translation more challenging for mt, such as modifying the gender, substituting the name of a product for 'it', splitting a sentence into two, etc. these modifications are explained in the spreadsheet file for each line modified, so researchers can decide if they can use or not documents that had the source modified. ## conclusion we have presented a document-level corpus annotated with context-aware issues when translating from en into pt-br, namely gender, number, ellipsis, reference, lexical ambiguity, and terminology. this first version of the corpus contains 60 documents, with 3680 sentences, in six different domains: subtitles, literary, news, reviews, medical and legislation. to the best of our knowledge, this is the first corpus of its kind. 24 with the rise in nmt quality and the claims of human parity, the need to move towards a more fine-grained evaluation involving the whole document is beyond question. moreover, with the lack of resources for the document-level mt area, this document-level corpus can be used as a challenge 24 the corpus and guidelines will be freely available at https://github.com/sheilacastilho/ dela-project test set for evaluation and as a training/testing corpus for mt as well as for deep linguistic analysis of context issues. we believe that the annotation can be also used for closely-related languages such as spanish. we intend to increase the corpus, adding more documents, domains and more context-aware issues. the full translation into pt-br is ongoing, and we want to annotate it for other languages, starting with the romance language family.