Investigating Length Issues in Document-level Machine Translation (2025)

Abstract: Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we challenge the ability of MT systems to handle texts comprising up to several thousands of tokens. We design and implement a new approach designed to precisely measure the effect of length increments on MT outputs. Our experiments with two representative architectures unambiguously show that (a) translation performance decreases with the length of the input text; (b) the position of sentences within the document matters and translation quality is higher for sentences occurring earlier in a document. We further show that manipulating the distribution of document lengths and of positional embeddings only marginally mitigates such problems. Our results suggest that even though document-level MT is computationally feasible, it does not yet match the performance of sentence-based MT.

## introduction statistical and neural machine translation (mt) architectures (koehn, 2020) have been designed to process isolated sentences, limiting their ability to properly handle discourse phenomena, such as coherence and cohesion, the modelling of which requires longer contexts @xcite . a first step to address this shortcoming has been to augment the source and/or the target side with a couple of preceding sentences @xcite . multiple approaches to encode and fully exploit such extended contexts have been proposed @xcite @xcite and have been shown to improve the ability of mt engines to preserve local discourse coherence and cohesiveness through word-sense disambiguation or the resolution of anaphoric references @xcite . most of these approaches continue to process texts on a per-sentence basis with an extended context, even though attempts have also been made to process continuous chunks of texts comprising several sentences @xcite @xcite @xcite . the ability of today's neural mt modelsrelying on encoder-decoder or decoder-only architectures-to handle large context lengths, up to thousands of tokens @xcite , opens new perspectives to go beyond context-augmented mt and develop fully-fledged document-level mt, where the entire document context is available at once, and where the target text is generated in a single pass. 1 two main technical novelties have made this possible: (a) more efficient computation in the attention layers @xcite and (b) changes in the design of positional encodings (pes). in particular, replacing the sinusoidal absolute pes (apes) of @xcite with methods like al-ibi @xcite and rope @xcite , which lend themselves well to length extrapolation @xcite , seems to make today's transformers amenable to the processing of arbitrarily long contexts @xcite . in this work, we challenge the ability of contemporary mt models to effectively handle long spans of texts. for this, we develop a new methodology for assessing the impact of length variations on mt performance. we perform a series of controlled experiments with two representative neural mt systems, where the same documents are processed by chunks of increasing lengths in a document-level manner and show that (a) mt performance tends to degrade with the length of the source document, (b) length issues happen even for in-distribution lengths and get worse when extrapolating to unseen document lengths, and (c) most of the degradation happens in the final parts of the translation. hypothesising that this may be due to a mismatch between the distribution of train and test pes, we explore a possible mitigation, which flattens the distribution of pes during training. we observe a consistent improvement of automatic metric scores for the ape-based vanilla encoder-decoder model, while the rope based decoder-only model remains mostly unaffected. in summary, our main contributions are: (a) a new approach to the detection and diagnosis of length issues in document-level mt, (b) a new variant of the shape @xcite ) method, which improves the distribution of pes during training, and (c) a confirmation that (perhaps for lack of an appropriate documentlevel evaluation tool) sentence-level mt remains a strong baseline in most settings. translation of discourse phenomena, such as lexical consistency, reference, and word sense disambiguation, requires inter-sentential context @xcite @xcite . this has motivated the integration of extended (local) contexts in doc2sent models. such approaches include concatenation-based methods @xcite ; architecture adaptations to process context in different components of the same encoder @xcite , in a dedicated encoder @xcite , or via hierarchical attention networks @xcite @xcite ; cache-based methods using a short-term mt memory @xcite @xcite and multi-pass decoding algorithms @xcite @xcite . translating sentence by sentence, even with augmented contexts, still fails to capture phenomena related to coherence and consistency @xcite , motivating doc2doc approaches to process documents as a whole. this can be done with concatenation-based methods @xcite @xcite , along with sliding window attention @xcite and group attention @xcite to address the issue of quadratic complexity. other strategies include focusing on improving training through data augmentation with a balanced length distribution @xcite and richer context-dependent phenomena @xcite , or on better training strategies with multilingual denoising pretraining @xcite , adapted loss functions @xcite , and enriched positional encodings @xcite . multiple methods have recently emerged for large language models (llms) @xcite , which also show a decline in translation quality as input length increases @xcite . 2 these include a two-stage training recipe with the use of a monolingual corpus and high-quality parallel documents @xcite , and applying llms as post-editors @xcite . ## manipulating the distribution of pes a basic requirement for document-level systems is that they should be trained, or at least fine-tuned, with long text inputs, ideally with complete documents. using the empirical document length distribution may however not be ideal, as it yields very skewed distributions of pes where small position indices are over-represented. we discuss two approaches to obtain more balanced distributions. ## conclusion in this work, we have studied the ability of current mt architectures to handle long input texts, ideally entire documents, and to translate them holistically. our analyses are based on systematic comparisons of translation outputs computed with varying input lengths, which are then evaluated with two automatic metrics. they consistently show that, even when the test document lengths match that of the training set and remain within the model limits, the translation scores tend to decrease with the source length, a degradation that mostly impacts sentences occurring far from the beginning of the document. we also show that manipulating the training distribution of lengths or pes has a positive effect for ape-based models, which vanishes in rope-based models like towerbase. these results finally confirm the robustness of sentence-level baselines. they hint at the need to improve existing models to truly benefit from the potential of documentlevel mt, for instance by constraining the attention mechanism to simulate a form of sentence alignment, by improving the memorization capacities of existing architectures, or by ensuring that the generation algorithm does not eventually get trapped in repetition loops. these are some of the directions we wish to explore in future work.