Improving Long Context Document-Level Machine Translation (2023)

Abstract: Document-level context for neural machine translation (NMT) is crucial to improve the translation consistency and cohesion, the translation of ambiguous inputs, as well as several other linguistic phenomena. Many works have been published on the topic of document-level NMT, but most restrict the system to only local context, typically including just the one or two preceding sentences as additional information. This might be enough to resolve some ambiguous inputs, but it is probably not sufficient to capture some document-level information like the topic or style of a conversation. When increasing the context size beyond just the local context, there are two challenges: (i) the memory usage increases exponentially (ii) the translation performance starts to degrade. We argue that the widely-used attention mechanism is responsible for both issues. Therefore, we propose a constrained attention variant that focuses the attention on the most relevant parts of the sequence, while simultaneously reducing the memory consumption. For evaluation, we utilize targeted test sets in combination with novel evaluation techniques to analyze the translations in regards to specific discourse-related phenomena. We find that our approach is a good compromise between sentence-level NMT vs attending to the full context, especially in low resource scenarios.

## introduction machine translation (mt) is the task of mapping some input text onto the corresponding translation in the target language. mt systems typically operate on the sentence-level and utilize neural networks trained on large amounts of bilingual data @xcite . these neural machine translation (nmt) systems perform remarkably well on many domains and language pairs, sometimes even on par with professional human translators. however, when the automatic translations are evalu-ated on the document-level (e.g. the translation of a whole paragraph or conversation is evaluated), they reveal shortcomings regarding consistency in style, entity-translation or correct inference of the gender, among other things @xcite @xcite . the goal of document-level nmt is to resolve these shortcomings by including context information as additional input when translating a sentence. in recent years, many works have been published on the topic of document-level nmt. however, most of these works focus only on including a few surrounding sentences as context. when the context size is increased beyond that, typically a degradation of overall translation performance is reported. additionally, the transformer architecture as the quasi standard in nmt seems sub optimal to handle long sequences as input/output, since the memory complexity increases quadratically with the sequence length. this is due to the attention mechanism, where each token in a sequence needs to attend to all other tokens. in this work, we propose a constrained attention variant for the task of document-level nmt. the idea is to reduce the memory consumption while at the same time focusing the attention of the system onto the most relevant parts of the sequence. our contributions are two-fold: ## related work many works have been published on the topic of document-level nmt. the widely used baseline approach consists of simply concatenating a few adjacent sentences and feeding this as an input to the mt system, without modifying the system architecture in any way @xcite @xcite @xcite . also, several modifications to this baseline concatenation approach have been proposed. @xcite introduce segment embeddings and also partially constrain the attention to the tokens of the current sentence. zhang et al. (2020) propose to calculate the self-attention both on the sentence-and on the document-level and then combine the two representations. fernandes et al. (2021) and lei et al. (2022) both mask out tokens in the current sentence to increase context utilization while yang et al. (2023) remove tokens from the context if they are not attended. typically, slight improvements in bleu are reported as well as more significant improvements on targeted test sets e.g. for coreference resolution. apart from the simple concatenation method, there exist other approaches to document-level nmt. they include using a single documentembedding vector @xcite @xcite , multiple encoders @xcite @xcite @xcite , hierarchical attention @xcite @xcite , translation caches @xcite @xcite or dynamic evaluation @xcite . however, these approaches are less versatile and require significant changes to the model architecture, often introducing a significant amount of additional parameters. furthermore, recent works have concluded that the baseline concatenation approach first proposed by @xcite performs as good -if not better -than these more complicated approaches @xcite . while the concatenation approach works well for short context sizes, when used with a larger number of context sentences, typically performance degradation is reported: @xcite saw a severe performance degradation when using input sequences with a length of 250 tokens. @xcite could not get their system to converge when using context sizes of up to 512 tokens. they improve training stability by adding additional monolingual data via pre-training. @xcite also report that their systems with context length of more than 256 tokens fail to converge. they propose to partially constrain the attention to the current sentence, similar to @xcite . @xcite try to translate full documents with the concatenation approach but could not get their system to converge during training. their solution is to mix document-and sentence-level data, which reportedly improves system convergence. @xcite report severe performance degradation for context sizes longer than 512 tokens. they argue this is due to insufficient positional information and improve performance by repeatedly injecting this information during the encoding process. however, increasing the context size seems to not always result in performance degradation. in their works, junczys-dowmunt (2019) and @xcite train systems with a context size of up to 1000 tokens without degradation in translation quality, which stands in contrast to the works mentioned above and which we will discuss again in the context of our own results. we want to point out that all of the approaches mentioned above still have the problem of quadratically increasing resource requirements, which poses a big challenge even on modern hardware. since our proposed approach consists of modifying the attention matrix in the model architecture, we give a brief overview of previous works related to this concept. the works of @xcite , @xcite and @xcite are most closely related and were already mentioned above. all three papers restrict the attention (partially) to the current sentence and combine sentence-and document-level attention context vectors for the final output. however, this means all approaches still suffer from the quadratic dependency on the number of input tokens. @xcite were among the first to propose using the attention concept for the task of mt. they also proposed using a sliding-window with target-to-source alignment for attention similar to us. however, they only work on sentence-level nmt and to the best of our knowledge, this approach was never before transferred to document-level nmt. @xcite and @xcite both extend the approach of @xcite while still working solely on sentence-level nmt. our approach is also related to the utilization of relative positional encoding, which was introduced by @xcite and later extended by @xcite to be applicable for cross-attention. the work by @xcite should also be mentioned, where they pre-select a subset of source tokens on which to perform attention on. again, all of the above mentioned works only perform experiments on sentence-level nmt. the works of @xcite al. (2019), sukhbaatar et al. (2019) and guo et al. (2020) are also related, since they use attention windows similar to us for tasks other than mt. finally, we briefly want to touch on the subject of automatic evaluation of document-level mt systems. many works only report results on general mt metrics like bleu @xcite , sometimes matching n-grams across sentenceboundaries. however, it has been argued that these metrics do not capture well the very specific improvements that could be expected by including document-level context and that the reported improvements rather come from regularization effects and comparing to sub optimal baseline performance @xcite @xcite . several targeted test suites have been released to better assess the improvements gained by document-level nmt @xcite @xcite . these test suites have some limitations, for example they are language-specific and they are based on just scoring predefined contrastive examples without scoring the actual translations. more recently, @xcite and @xcite have released frameworks that allow to score the actual mt hypotheses in regards to their consistency regarding specific aspects of the translation. ## methodology here, we explain the baseline concatenation approach (section 3.1), the more refined method that we are comparing ourselves against (section 3.2) as well as our own approach (section 3.3). we also discuss our different evaluation approaches in section 3.5. ## experiments we perform experiments on three document-level translation benchmarks. we call them news (@xmath0, ted (@xmath1. news is a collection of news articles, ted is a collection of transcribed ted talks and their respective translations and os consists of subtitles for movies and tv shows. especially the latter holds many examples for discourse between different entities. for the details regarding data conditions, preparation and training, we refer to appendix a.2. ## conclusion in this work, we focus on methods to increase the context-size for document-level nmt systems. we point out the shortcomings of the baseline approaches to long-context document-level nmt and in turn propose to modify the attention component to be more focused and also to be more memory efficient. we compare our approach against approaches from literature on multiple translation tasks and using different targeted evaluation methods. we confirm the improved memory efficiency of the proposed method. we find that for some discourse phenomena like pronoun translation, the longer context information is not necessary. for other aspects, like consistent style translation, the longer context is very beneficial. it seems that the baseline concatenation approach needs large amounts of training data to perform well for larger context sizes. we conclude that our approach performs among the best across all tasks and evaluation methods, with the additional benefit of reduced memory consumption for long input sequences.