A Test Suite for Evaluating Discourse Phenomena in Document-level Neural Machine Translation (2020)

Abstract: The need to evaluate the ability of context-aware neural machine translation (NMT) models in dealing with specific discourse phenomena arises in document-level NMT. However, test sets that satisfy this need are rare. In this paper, we propose a test suite to evaluate three common discourse phenomena in English-Chinese translation: pronoun, discourse connective and ellipsis where discourse divergences lie across the two languages. The test suite contains 1,200 instances, 400 for each type of discourse phenomena. We perform both automatic and human evaluation with three state-of-the-art context-aware NMT models on the proposed test suite. Results suggest that our test suite can be used as a challenging benchmark test bed for evaluating document-level NMT. The test suite will be publicly available soon.

## introduction document-level nmt has attracted extensive interest in recent years. different from sentence-level nmt models, discourse-level models need to not only cope with intra-sentence dependencies, but also incorporate context beyond current sentence into context-aware translation. inter-sentence links usually exhibit a wide variety of discourse phenomena: coreference, lexical cohesion, coherence, discourse relations, etc. the quality of a documentlevel nmt model therefore can be evaluated based on its ability in dealing with these discourse phenomena. widely-used automatic evaluation metrics, e.g., bleu @xcite , normally consider fragments in a local window for translation quality assessment, while cross-sentence discourse links are usually neglected. hence, for document-level models, current automatic evaluation metrics may be not a reasonably good fit for evaluation. one possible alternative is using manually-created test suites which are composed of carefully selected examples with discourse phenomena @xcite . such test suites @xcite @xcite @xcite have been constructed for several language pairs, such as english-czech, english-german, english-russian, french-german, but few in english-chinese translation. in this paper, we propose a test suite aiming at english-chinese discourse phenomena evaluation. three frequent discourse phenomena in english-chinese translation are selected in our test suite, namely pronoun, discourse connective and ellipsis, each of which forms an individual test set. we choose examples from the opensubtitles @xcite to construct the three test sets. unlike corpora from news domain, this corpus is more conversational and colloquial. we use this test suite to evaluate several typical context-aware nmt models. the experiment results show that our test suite can evaluate the ability of nmt models in dealing with discourse phenomena and that it is still very challenging for current context-aware models to capture different discourse phenomena. ## related work research on the evaluation of document-level machine translation is usually on specific discourse phenomena. a few test suites and methods have been designed for evaluating nmt from the perspective of discourse phenomena. for pronoun translation evaluation, recent test sets on pronoun evaluation have consisted of contrastive pairs. @xcite provide 50 example blocks of english-french contrastive pairs. @xcite have also created contrastive pairs of pronoun "it" in english-german translation. contrastive test sets allow us to automatically evaluate document-level nmt by only judging whether the evaluated model can choose the correct translation against the wrong from each contrastive pair according to their model score. however, this is an indirect rather than a direct way to evaluate the ability of context-aware nmt in modeling discourse phenomena as we do not evaluate the actual translations generated by these nmt systems. to evaluate discourse connective translation, @xcite propose act (accuracy of connective translation) to evaluate connective translation. for french-english discourse relation and discourse connective translation assessment, smith and specia (2018) use pretrained bilingual embeddings of discourse connectives. @xcite investigates conjunction disambiguation in english-german and french-german translation. for the evaluation on ellipsis translation, @xcite explore contrastive examples to evaluate the verb phrase ellipsis and morphological inflection in english-russian translation. in our work, we also investigate verb ellipsis in english-chinese translation. ## test sets we choose three types of discourse phenomena, i.e., pronoun, discourse connective and ellipsis, as they appear frequently in english-chinese documentlevel nmt. in the following parts, we will introduce corpus construction and then the three test sets separately. ## experiment we used the proposed test suite as a benchmark test bed to evaluate state-of-the-art context-aware nmt models against the three types of discourse phenomena. ## conclusion we have presented a discourse-level test suite for the evaluation of context-aware neural machine translation. we constructed 1,200 instances for three types of discourse phenomena in english-chinese translation, 400 instances per discourse phenomenon. our experiments with three stateof-the-art document-level nmt models suggest that ellipsis is the most challenging discourse issue among the three test sets.