Challenges in Context-Aware Neural Machine Translation (2023)

Abstract: Context-aware neural machine translation, a paradigm that involves leveraging information beyond sentence-level context to resolve inter-sentential discourse dependencies and improve document-level translation quality, has given rise to a number of recent techniques. However, despite well-reasoned intuitions, most context-aware translation models show only modest improvements over sentence-level systems. In this work, we investigate and present several core challenges that impede progress within the field, relating to discourse phenomena, context usage, model architectures, and document-level evaluation. To address these problems, we propose a more realistic setting for document-level translation, called paragraph-to-paragraph (PARA2PARA) translation, and collect a new dataset of Chinese-English novels to promote future research.

## introduction neural machine translation (nmt) has garnered considerable scientific interest and commercial success in recent years, with current state-of-the-art systems approaching or exceeding human quality for a few resource-rich languages when translating individual sentences @xcite @xcite . despite the strong empirical performance of such systems, the independence assumption that underlies sentencelevel nmt raises several issues. certain textual elements, such as coreference @xcite , lexical cohesion @xcite , or lexical disambiguation @xcite are impossible to correctly translate without access to linguistic cues that exist beyond the present sentence @xcite . when evaluating documents rather than individual sentences, the adequacy and fluency of professional human † equal contribution. 1 we release the paper's code and dataset here: https: //github.com/linghao-jin/canmt-challenges . despite some efforts to meaningfully exploit inter-sentential information, many context-aware (or interchangeably, document-level) nmt systems only show meager gains across sentence-level and document-level translation metrics @xcite @xcite @xcite @xcite . performance improvements against sentence-level baselines on overall translation accuracy, pronoun resolution, or lexical cohesion become less pronounced when context-aware systems are trained on realistic, high-resourced settings @xcite , casting doubt on the efficacy of such approaches. in this paper, we conduct a thorough empirical analysis and present some key obstacles that hinder progress in this domain: ## background the high-level objective of sentence-level machine translation is to model the sentence-level conditional probability p (y|x), in which the source and target sentences @xmath0, ..., x m ), @xmath1, ..., y n ) are textual sequences of respective lengths m and n . under the dominant paradigm of neural machine translation @xcite , the conditional probability p θ (y|x) is typically decomposed into the following auto-regressive formulation (with θ denoting parameterized weights): equation 1 implies that when predicting the target token y n , the model could only access the current source sentence, x, as well as all previously translated tokens in the current target sentence, @xmath2. translating sentences in a document in such an isolated fashion, without any extra-sentential information that lies beyond sentence boundaries, has been found to produce syntactically valid, but semantically inconsistent text @xcite . to remedy this, context-aware neural machine translation considers a document d that entails a set of logically cohesive source sentences @xmath3, x 2 , ..., x d }, and a parallel set of target sentences @xmath4, y 2 , ..., y d }. under a left-to-right translation schema, the model computes the probability of translating the source sentence x i conditioned on the context c i , wherein 0 ≤ @xmath5: in practice, there are multiple ways to formulate c i . passing in c @xmath6. throughout this paper, we explore two concatenation-based setups first presented by @xcite . the one-to-two (1-2) setup prepends the preceding target sentence to the current target sentence (c @xmath7, denoting sentence boundaries with @xmath8. the two-to-two (2-2) setup incorporates additional context from the previous source sentence (c @xmath9, y i-1 }). the target context is integrated in the same manner as in one-to-two. in order to investigate the importance of context after the current sentence, we also explore a three-to-one (3-1) setting, wherein we introduce additional source-side context by concatenating the previous and subsequent sentences to the current one (c @xmath10, x i+1 }), and do not incorporate any target context. ## challenges we identify key obstacles that account for the lack of progress in this field, based on a careful empirical analysis over a range of language pairs, model architectures, concatenation schemas, and document-level phenomena. foot_1 4.1 discourse phenomena is sparse in surrounding context. contextual sparsity is a bottleneck to documentlevel neural machine translation that manifests in two forms @xcite . first, the majority of words within a sentence can be accurately translated without additional access to inter-sentential information; context poses as a weak training signal and its presence has not been found to substantially boost translation performance. second, only a few words in neighboring sentences may actually contribute to the disambiguation of current tokens at translation time. we investigate contextual sparsity via a finegrained analysis on the bwb (jiang et al., 2022) test set, which has been manually tagged with specific discourse-level phenomena. 4 specifically, we use it to probe nmt models' ability to exploit long-range context by analyzing the frequency of particular discourse phenomena that can only be resolved with context. for the manual analysis, we randomly sample 200 discourse-annotated instances from the test set and ask bilingual annotators who are fluent in chinese and english to identify and count instances that contain a particular context-dependent discourse phenomenon. annotators are asked to discern if the following document-level discourse phenomena exist in each sentence pair: chinese, but must be included in the english translation. we next examine how far the useful context tends to be from the cross-lingually ambiguous sentence. taking d as the sentence distance, the majority of discourse phenomena can be disambiguated based on the nearest context sentence (@xmath11. specifically, the necessary information for tense, ambiguity, and discourse markers can almost always be found by @xmath12, whereas relevant context for pronoun ellipses and lexical cohesion tends to be more spread out. hardly any useful information can be found in very distant context (@xmath13. a significant fraction (40.5%) of sentences in the sampled test set can be translated independently, i.e., without access to inter-sentential information. correspondingly, we notice that many sentences across document-level data are not lengthy with discourse-level phenomena, but rather simple constructions. figure 1 indicates that the majority of sentences are relatively short in bwb and iwslt-17, ranging from 20-50 characters (chinese) or 10-30 words (french and german). ## para2para translation a recurrent theme throughout our analyses is that existing datasets are not conducive to meaningful context usage in document-level translation. the majority of datasets used in the literature of document-level nmt are aligned at the sentence level, which is artificial in design and not reflective of how documents are translated in practice. as such, paragraph-level parallel data (figure 2 ) may be more suited for document-level nmt and provide richer contextual training signals. recent work have turned toward literary translation as a challenging, realistic setting for document-level translation @xcite @xcite , given that literary texts typically contain complex discourse structures that mandate a document-level frame of reference. as figure 2 illustrates, sentence alignment is not always feasible when translating literature. @xcite finds that language models can effectively exploit document-level context and cause fewer discourse-level translation errors based on human evaluation, when the paragraph is taken as the minimal discourse-level unit. to promote future research on document-level translation in a realistic setting, we collect professional english and chinese translations of classic novels, and format the data by manually correcting paragraph-level alignments. the para2para dataset consists of 10,545 parallel paragraphs across six novels from the public domain. 9 to our knowledge, the only other paragraph-aligned, parallel dataset sourcing from the literary domain is par3 @xcite , which uses google translate and fine-tuned @xcite to automatically generate reference translations. in contrast, the source and target paragraphs in our dataset are culled from professional translations. 10 we then benchmark the dataset under two experimental settings for @xmath14: i). a standard closed-domain setup, in which both the training and testing data are sourced from the same novels; ii). a more challenging open-domain setup, wherein two novels are held and used as only the test set. we experiment with training a transformer-based model on para2para data from scratch (none), as well as incorporating pretrained baselines, in which the model is first trained on the sentence-level wmt17 zh-en dataset @xcite , before further fine-tuning on the para2para data, using the following backbone architectures: on the closed-domain setting, lightconv big yields the highest score across all three metrics. open-domain results are mixed: as expected, scores are lower across the board as this setting is challenging. xfmr big has the best bleu and discourse marker f1-score on blonde, although all pre-training baselines perform similarly. light-conv big performs the best on pronoun, entity, and tense on blonde and has the highest comet score. ## conclusion despite machine-human parity at the sentence level, nmt still lags behind human translation on long collections of text, motivating the need for contextaware systems that leverage signals beyond the current sentence boundary. in this work, we highlight and discuss key obstacles that hinder momentum in context-aware nmt. we find that training signals that improve document-level discourse phenomena occur infrequently in surrounding context, and that most sentences can be accurately translated in isolation. another challenge is that context benefits the resolution of some discourse phenomena over others. a context-agnostic transformer baseline is already competitive against contextaware settings, and replacing the transformer's self-attention mechanism with a more complex long-range mechanism does not significantly improve translation performance. we also note the need for a generalizable document-level translation metric. finally, we make the case for paragraphaligned translation, and release a new para2para dataset, alongside baseline results, to encourage further efforts in this direction.