Self-Retrieval from Distant Contexts for Document-Level Machine Translation (2025)

Abstract: Document-level machine translation is a challenging task, as it requires modeling both short-range and long-range dependencies to maintain the coherence and cohesion of the generated translation. However, these dependencies are sparse, and most context-augmented translation systems resort to two equally unsatisfactory options: either to include maximally long contexts, hoping that the useful dependencies are not lost in the noise; or to use limited local contexts, at the risk of missing relevant information. In this work, we study a self-retrieval-augmented machine translation framework (Self-RAMT), aimed at informing translation decisions with informative local and global contexts dynamically extracted from the source and target texts. We examine the effectiveness of this method using three large language models, considering three criteria for context selection. We carry out experiments on TED talks as well as parallel scientific articles, considering three translation directions. Our results show that integrating distant contexts with Self-RAMT improves translation quality as measured by reference-based scores and consistency metrics.

## introduction document-level machine translation (dlmt) is a challenging task, as it requires modeling both shortrange and long-range dependencies to maintain the coherence and cohesion of the generated translation. inter-sentential contexts are indispensable for the handling of phenomena such as co-reference, lexical consistency, textual coherence and cohesiveness, which continue to be challenging for long document translation @xcite @xcite . numerous approaches, reviewed in @xcite , have been proposed to integrate these contexts. they include segment concatenation @xcite @xcite , architecture adaptation @xcite @xcite , training strategy optimization @xcite @xcite , and multi-pass refinement @xcite @xcite . past work also shows that various sources of contextual information contribute differently to translation quality; the local source and target context is the main resource for handling anaphoric references and word-sense disambiguation information @xcite , whereas the global context, especially on the target side, holds information likely to improve coherence and cohesiveness of the full translated document @xcite . recent generative models such as llama3 @xcite and @xcite can process inputs up to hundreds of thousands of tokens, creating new possibilities for the inclusion of the whole source text, as well as already translated target segments, in the translation context. it however remains an open question whether such architectures, relying on the selfattention-mechanism @xcite , are effectively able to identify relevant long-range dependencies and actually improve dlmt @xcite . this is because inter-sentential dependencies can be sparsely distributed within a document, whereas self-attention generates dense patterns spreading out over the entire past text @xcite . therefore, most approaches to dlmt still consider a limited context window size, usually up to 1024 tokens or a fixed number of sentences. in order to capture long-distance dependencies without requiring the attention mechanism to handle overly long contexts, we propose selfretrieval augmentation for machine translation (self-ramt), aiming to take into account both local and global dependencies, regardless of the document length. in our approach, inter-sentential dependencies are precomputed for the full source document to identify past relevant segments for each translation unit. as soon as they are translated, these segments and their translation become available to inform the subsequent translation choices. in our implementation, which uses large language models (llms), such dynamic contexts are taken into account through in-context learning (icl). two scenarios are considered: (a) one where incontext examples correspond to correct translations, as in online learning, where the input sentences, once incrementally post-edited by a human translator, become available to revise the model (álvaro peris and casacuberta, 2019), foot_0 and (b) a fully automatic setup, with imperfect in-context translations, requiring no human intervention. in this context, our main research questions are as follows: (i) how to best identify and retrieve useful context segments, (ii) what improvement to mt quality do these retrieved contexts bring, and (iii) to what extent distant (as opposed to local) contexts actually enhance translation scores. we compare three criteria for context selection (cosine similarity with respect to labse embeddings (cos), best match 25 (bm25) and point-wise mutual information (pmi)) and carry out experiments on three llms in three language directions (english to german (en-de), french (en-fr), and chinese (en-zh)), analyzing the impact of contexts, especially distant ones. experimental data includes both ted talks @xcite and a new dedicated parallel test set, mersenne, consisting of scientific articles for the en-fr direction. scientific articles offer an interesting use case to study term consistency in long document translation. our investigation reveals that distant contexts retrieved with pmi provide valuable information that increases translation metric scores as well as term consistency. we make our code and data available. 2 2 related work document-level mt dlmt research broadly falls into two categories: doc2sent, which involves translating each sentence individually using intradocument source and/or target context to aid translation, and doc2doc, which involves translating multiple sentences at once @xcite @xcite . doc2doc approaches represent a simple strategy for effective context integration, maintaining better consistency and coherence than doc2sent methods @xcite . however, doc2doc methods struggle to process very long sequences of sentences, as relevant information is sparse in global contexts @xcite , which can lead to the degradation of translation quality or omitted sentences @xcite @xcite . to address this problem, several approaches have been explored, including context-aware attention @xcite @xcite , which selects important contexts according to the attention distribution, and dynamic context selection @xcite , which applies a reward model to identify varying numbers of useful context sentences within the local context, constrained by the complexity of reinforced training. llm-based dlmt various llm-based methods have been proposed for dlmt. several works explore zero-shot prompting and icl for doc2doc mt @xcite and study the best way to train llms for dlmt @xcite @xcite , illustrating the importance of high-quality in-context demonstrations and fine-tuning parallel corpora. llms have also been used as post-editors, either through fine-tuning @xcite @xcite or prompting for iterative translation refinement @xcite . when it comes to integrating context, llms offer greater flexibility than traditional neural machine translation models. various multi-aspect prompting techniques have been proposed to enhance the input by incorporating or automatically summarizing relevant information from the context. for instance, delta @xcite builds a dynamic context for each source sentence, heavily relying on llm components to extract and assemble relevant information from the available source and target texts (including proper nouns, bilingual summaries of local contexts and relevant past sentences within a predefined context window). however, delta is computationally costly (a lot more so than self-ramt), due to the multiple steps required for dynamic context extraction. sent2sent++ @xcite incorporates two types of contexts: a static part, consisting of an automatically generated bilingual summary of the full source document, and a dynamic part, composed of the previous source and target sentences. however, the use of static, automatic summaries (a) has the effect of potentially changing the words of the context, which can be detrimental to lexical consistency and (b) does not ensure that the most relevant information is accessible for each source sentence, especially as documents increase in length. retrieval-augmented mt choosing which context to be included in mt can be seen as a type of retrieval-augmented mt. in past works, retrievalaugmented mt systems have mostly been designed to mimic the use of translation memories by translators, which has a long history in mt @xcite . recent implementations of this idea for neural models encode the target side of relevant example(s) together with the source sentence in an extended translation context @xcite @xcite @xcite . variants, relying on both the source and target sides of the retrieved example(s) are proposed by @xcite and @xcite . llm-based mt systems seamlessly accommodate examples through in-context learning, where examples of the translation task (the source and target sides of parallel samples) @xcite are inserted into the prompt. the optimal selection of in-context examples has also been the focus of recent research @xcite @xcite @xcite @xcite , also analyzed by @xcite and @xcite . a key difference with self-ramt is that these methods retrieve examples from external resources, instead of the input sequence, with the aim to find similar examples that can be easily edited. several retrieval-augmented architectures have also been proposed, e.g., by @xcite , to retrieve relevant contextual information from very long input documents. these approaches have been evaluated in language modeling tasks, but, to the best of our knowledge, have not yet been applied to mt. ## examining context selection strategies in this section, we aim to answer the following questions: (a) how effective are the context selection scores to identify relevant contexts? and (b) how similar are the retrieved segments when method to assess the context selection criteria for their sensitivity to coherence, we challenge their ability to distinguish sentences extracted from the same documents from other noise segments. starting with a set of document pairs (x 1 , x 2 ) both containing n sentences in the same language, we randomly shuffle sentences from x 1 with those of x 2 , resulting in a combined document x 1,2 . we then retrieve, from the first 2n -1 segments of x 1,2 , the k most relevant segments for the last sentence (x 2n ), using each relevance score, and compute the extraction error rate r, defined as the proportion of selected sentences that do not belong to the same document as x 2n . for a set of n documents {x 1,2 l , @xmath0. . . n }, from which we retrieve the k context sentences {c l,1 , . . . , c l,k } for each x 2n , r is computed as follows: results for question (a), comparing relevance scores we perform context-aware evaluations on iwslt, reported in note that this problem does not arise when we retrieve automatic translations, where we see the benefits of self-ramt more clearly. the evaluation results for mersenne and iwslt translated with reference and automatic translations as context (using eurollm) are given in tables 5 and 6 . these results lead to the same conclusion that pmi is a good criterion for retrieving relevant past segments. as the ltcr scores rank the context selection methods differently across test sets, we conduct a follow-up case study presented in section 6.3, which better highlights the contribution of selected contexts to term consistency. the complete scores for all mt systems are in tables 11 and 12 alyze the contribution of relevant contexts (especially distant ones) to the translation quality, we bin sentences according to the distance (in number of sentences) to their most distant selected context. for example, all selected contexts of sentences in the group 0-20 are retrieved from the past 20 sentences, while there are contexts more distant than 64 sentences for members of the group 64-256. we observe that about half of the retrieved contexts are in the past 20 sentences, while a sizable portion of contexts are chosen in more distant part of the document (in the 20 -80 range); more remote sentences, with a distance larger than 80 are also frequently selected. there is a clear variance between relevance scores: cos retrieves closer segments on average, whereas pmi and bm25 are more likely to extract more remote sentences. distant contexts matter for each group of sentences, we now compare the translations generated with pmi and with the baseline methods. translation scores are in ## conclusion to achieve better document-level machine translation for extra long documents, we propose the self-ramt framework. by retrieving relevant sentences in a local translation memory composed 14 lines @xcite of sentences that have already been translated, and possibly post-edited, in the same document, we expect to generate translations that are globally more consistent. we carry out experiments to translate full ted talks and a novel parallel corpus consisting of scientific articles, using three llms with in context learning, to integrate the retrieved past contexts. we further analyze the influence of distant contexts on translation quality, according to their distance from the current sentence. our findings show that incorporating distant contexts, selected using context criteria such as pmi, can be useful for better lexical consistency. distant context also seem to improve the general translation quality, as measured by reference-based scores.