Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023)

Abstract: Several recent papers claim to have achieved human parity at sentence-level machine translation (MT)—especially between high-resource language pairs. In response, the MT community has, in part, shifted its focus to document-level translation. Translating documents requires a deeper understanding of the structure and meaning of text, which is often captured by various kinds of discourse phenomena such as consistency, coherence, and cohesion. However, this renders conventional sentence-level MT evaluation benchmarks inadequate for evaluating the performance of context-aware MT systems. This paperpresents a new dataset with rich discourse annotations, built upon the large-scale parallel corpus BWB introduced in Jiang et al. (2022a). The new BWB annotation introduces four extra evaluation aspects, i.e., entity, terminology, coreference, and quotation, covering 15,095 entity mentions in both languages. Using these annotations, we systematically investigate the similarities and differences between the discourse structures of source and target languages, and the challenges they pose to MT. We discover that MT outputs differ fundamentally from human translations in terms of their latent discourse structures. This gives us a new perspective on the challenges and opportunities in document-level MT. We make our resource publicly available to spur future research in document-level MT and its generalization to other language translation tasks.

## introduction the field of machine translation (mt) has made tremendous strides in recent years thanks to neural machine translation (nmt) models that can utilize massive quantities of parallel training data @xcite . unfortunately, most parallel corpora are aligned only at the sentence level, and document-level translation has remained limited to small-scale studies @xcite @xcite . there are, however, inherent differences between sentence-level translation and documentlevel translation, as documents consist of complex discourse structures that go beyond the mere concatenation of individual sentences. three key discourse features are particularly important in document-level translation. first, the translations of named entities have to be consistent. for example, in fig. 1 , the same terminologies are not consistently referred to with the same translations (i.e., weibo vs. micro-blog, qiao lian vs. joe vs. joe love), and as a result, the sentence-level mt system fails to capture discourse dependencies across sentences. second, the coreference relationship in the source language needs to be preserved. in particular, the relations between entities and their pronominal anaphora should be preserved, as well as the transition of discourse centers foot_0 . in fig. 1 and fig. 8 , coreference chains are color-coded, visually demonstrating the preservation of anaphoric referential relations and the transition chains of centers. third, the conversational structure, such as transitions between speakers, must be maintained. inferring latent discourse structures from documents is essential for translation coherence because of these discourse features. as a result, conventional sentence-level mt pipelines that do not consider context are unable to generate natural and coherent translations @xcite . while efforts have been made to develop context-aware nmt models over the past few years @xcite tiedemann and source reference mt (1) 乔恋攥紧了拳头，垂下了头。 qiao lian clenched her fists and lowered her head. joe clenched his fist and bowed his head. (2) 其实他说得对。 actually, he was right. in fact, he's right. (3) 自己就是一个蠢货，竟然会[...]。 she was indeed an idiot, as only an idiot would [...] i am a fool, even will [...] (5) 她点进去，发现是凉粉群，所有人都在@她[...] she logged into her account and saw that a large number of fans in the liang fan group had tagged her.[...] she nodded in and found it was a cold powder group, and everyone was on her.[...] (7) 【川 流 不 息：乔 恋， 快 看微博头 条 ！微博头 条！】 [chuan forever: qiao lian, look at the headlines on weibo, quickly!] chuan-flowing: joe love, quickly look at the micro-blogging headlines! weibo headlines? (8) 她微微一愣，拿起手机，登陆微博，在看到头条 的时候，整个人一下子愣住了！ she froze momentarily, then picked up her cell phone and logged into weibo. when she saw the headlines, her entire body immediately froze over again! she took a slight look, picked up the phone, landed on the micro-blog, when she saw the headlines, the whole person suddenly choked! figure 1 : selected examples of the annotations in bwb-test. the mentions in the same coreference chain are marked with the same color. pronoun omissions are marked with . the mistranslated verbs are marked with teal, and the mistranslated named entities are underlined. (7) is a quotation with the speaker annotated as chuan forever. the full chapter is given in fig. 8 . mt is the output of a transformer-based sentence-level machine translation system. @xcite @xcite @xcite , a reliable evaluation method for document-level mt has not yet been developed. traditional mt evaluation metrics, such as bleu @xcite , ter @xcite , meteor @xcite , are limited to evaluating translation quality at the sentence level and do not consider contextual information. even with the use of more complex neural evaluation methods, e.g., @xcite , @xcite and bertscore @xcite , it is not possible to accurately assess the coherence of translations within the context of a document. blonde @xcite has attempted to evaluate documentlevel translation quality and obtained a higher correlation with human evaluation. yet, it is a coarsegrained automatic evaluation metric based on surface syntactic matching, thus being unable to evaluate deeper discourse features, such as coreferences and quotations. @xcite propose to concatenate context to extend pretrained metrics to the document level. however, its lack of interpretability leaves it unclear as to why it assigns a particular score to a given translation, which is a major reason why some mt researchers are reluctant to employ learned metrics in order to evaluate their mt systems @xcite . to address this issue, we have annotated a new benchmark based on the large-scale bilingual parallel corpus bwb proposed in previous work @xcite . this benchmark includes four evaluation aspects annotated on both source and target sides: entity, terminology, coreference, and quota-tion. it consists of 80 documents across multiple fiction genres, with 15,095 mentions covered in 150,287 words in total. using these annotations, we systematically evaluate the latent discourse structures of mt models. we show that, while context-aware mt systems perform better than phrase-and sentence-level ones in terms of entity translation, they still lag behind human translation in terms of proper noun and personal name translation. in addition, we demonstrate that humans are far more adept at preserving coreference chains than mt models. the main contributions of our paper are: ## corpus we draw our source material for annotation from the texts in the test set of bwb @xcite , which consists of 65,107 words, 2,633 sentences in 80 different documents drawn from 6 web novels across different genres. the distribution of genres in bwb are shown in fig. 2 . the statistics of bwb are shown in tab. 1. we refer the readers to app. d for the dataset creation details. ## annotation in this section, we describe the annotation scheme and the process of annotating the corpus, and provide examples of how the annotations can be used to study discourse phenomena in the two languages. the annotation was conducted by eight professional translators. a total of 8,585 mentions on the english side and 6,070 mentions on the chinese side have been annotated. ## bilingual analysis we conduct a thorough bilingual analysis of the novel evaluation aspects using the new annotation. the following two research questions are being investigated: entity types and terminology. tab. 4 demonstrates the distribution of entity annotations and term annotations in both chinese and english in the bwb test set. first, the bwb test set is dominated by person and facility entities, with a much lower proportion of geo-political entities. in addition, terminology entities and person entities are repeated more frequently, requiring better translation consistency. an important sanity check from our analysis is that the number and distribution of entities in chinese and english are similar. this suggests that at the discourse level, the information being conveyed in these two languages is largely the same. pronouns. pronoun translation has been the focus of discourse-level mt evaluation @xcite . we compare the numbers of different types of pronouns in chinese and english in tab. 5. as can be seen, chinese has significantly fewer pronouns due to its pronoun-dropping property. this poses additional challenges for nmt systems, as they must be able to resolve anaphoric references. in addition, tab. 5 reveals a notable difference in the frequency of neuter pronouns. english exhibits a higher prevalence of neuter pronouns, possibly due to the presence of a larger number of expletive subjects in the language. coreference. we further conduct analyses to investigate the differences and similarities between coreference behaviors in chinese and english. error type # description an entity 43.3% error(s) due to the mistranslation of named entities. term 19.2% error(s) caused by the mistranslation of terminologies. coref 34.0% error(s) caused by coreference resolution failure(s). zeropro 17.3% error(s) caused by the omission of pronoun(s). quote 1.1% error(s) caused by the misinterpretation of quotation(s). fig. 3a examines the distribution in distances to the nearest antecedent for both chinese and english. the average distance between antecedents and anaphora in english is shorter than in chinese. this may be connected to the more common use of pronoun ellipsis in chinese-when referring to closer antecedents, pronouns are often omitted. fig. 3b illustrates the distance between the first and last mention of an entity within each coreference chain. it can be observed that, although english coreference chains tend to be longer in general compared to chinese coreference chains, the distribution of these lengths is consistent. this represents another language-independent discourse feature in addition to entity distribution. the spread distributions of the two languages are further depicted in fig. 8 and fig. 9 . finally, we present an analysis of the size of coreference chains in fig. 3c . our results indicate that the number of mentions in english coreference chains tends to be larger than in chinese. challenges for mt. so far our analyses have revealed that the chinese source language and english target language exhibit several distinct linguistic characteristics, leading to various discourse phenomena, including the omission of pronouns in chinese and the use of expletive subjects in english, as well as shorter english coreference chains, among others. these discourse phenomena present challenges for mt. in order to systematically analyze translation errors caused by these phenomena, we conducted a study in which four professional translation experts compared reference translations to those generated by a commercial mt system google translate. the annotation guidelines are as follows: the results, summarized in tab. 6, indicate that named entity translation is the most significant issue, with terminology translation, coreference resolution failures, and pronoun omission also being notable problems. this analysis emphasizes the importance of accurately inferring latent discourse structures in the translation of long texts. ## exploring discourse features of mt systems and human translation the existence of a bilingual corpus with discourse annotations allows us to test the performance of both human translations and mt systemsincluding the popularly used commercial systems and those trained on the in-domain dataset bwb. the following 6 mt systems are adapted: foot_1 overall quality. the quality of the translations produced by each system was first evaluated using a range of automatic translation metrics, including bleu, meteor, bertscore, comet, and blonde. the results of this evaluation are presented in tab. 7. in addition, we conducted a human evaluation of the translations, the results of which are shown in fig. 4 . 3 the large gap between the performances of ht and mt indicates that the genre of bwb, i.e., literary translation, is challenging for mt, and nmt systems are far beneath human parity. doc performs significantly better than sent, suggesting that bwb contains rich discourse phenomena that can only be translated accurately when the context is taken into account. it is also worth noting that even though pe is the post-edit of the relatively poor-performing system bing, it still achieves surprisingly better performance than doc at the document level. this observation confirms ## conclusion in this paper, we introduced the bilingual parallel corpus bwb-test, which includes annotations of various discourse phenomena. our analysis of these annotations revealed the significant challenges posed by discourse phenomena to mt. therefore, we advocate for greater attention on discourse coherence and consistency of the outputs of nmt models. the main discourse challenges faced by mt include entity consistency, entity recognition, anaphoric information loss, and coreference. additionally, the bwb corpus, with its rich discourse annotations, serves as a valuable resource for a variety of purposes, including studying the transferability of named entity recognition and coreference resolution, as well as the development of multilingual structured prediction models.