Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models (2025)

Abstract: Neural Machine Translation (NMT) models have traditionally used Sinusoidal Positional Embeddings (PEs), which often struggle to capture long-range dependencies and are inefficient for handling extended context or document-level translation tasks. This work addresses the challenge of transitioning pre-trained NMT models from absolute Sinusoidal PEs to Relative PEs, such as RoPE and ALiBi, without compromising performance. We demonstrate that parameter-efficient fine-tuning, using only a small amount of high-quality data, can successfully facilitate this transition. Experimental results indicate that switching from Sinusoidal to Relative PEs results in competitive translation quality on sentence-level evaluation benchmarks. Additionally, models trained with RoPE consistently outperform those using ALiBi and Sinusoidal PEs on document-level benchmarks across both string-based metrics and qualitative evaluations. Moreover, we find that a small amount of long-context data in a few languages is sufficient for cross-lingual length generalization, thereby inducing long-context capabilities.

## introduction neural machine translation (nmt) models have become critical tools in many natural language processing (nlp) applications, especially for document-level translation. while nmt models perform well on sentence-level tasks, they often struggle with longer sequences, where preserving contextual coherence across sentences is essential. this limitation is largely due to their reliance on absolute positional embedding (pe) methods, such as sinusoidal pes, which can limit the model's ability to extrapolate effectively to longer contexts. recent advancements in relative pes, such as ro-tary positional embeddings (rope) and at-* equal contribution tention with linear biases (alibi), have demonstrated superior performance in handling longer sequences. these relative positional embedding techniques enable models to generalize better to long contexts, making them well-suited for document-level translation tasks. however, most state-of-the-art nmt models were originally trained using sinusoidal embeddings, and retraining them with newer methods like rope or alibi is computationally expensive. this presents a crucial question: can we modify the positional embeddings in pre-trained nmt models to improve long-context translation without significant retraining or performance degradation? addressing this problem would enable nmt models to translate long documents more effectively, improving translation coherence and quality across paragraphs and sections. moreover, longcontext nmt models could have broader applications, such as generating synthetic data for multilingual pre-training. such data is often needed in scenarios where natural multilingual corpora are limited, especially for low-resource languages. however, current sentence-level nmt systems are suboptimal for this task as they lack the capacity to capture broader contextual dependencies in long documents. in this work, we investigate whether it is possible to replace the positional embeddings in pre-trained nmt models with relative positional embeddings post-hoc. our study explores both parameterefficient fine-tuning (peft) methods and full finetuning (fft) strategies to restore model performance while enabling long-context generalization. in the remainder of this paper, we present the following contributions: our findings indicate that positional embeddings in pre-trained nmt models can be modified and adapted without compromising performance, to efficiently induce long-context capabilities. positional embedding methods transformerbased models require explicit positional information, as all timesteps are processed in parallel. to address this, @xcite proposed an empirical absolute pe approach, using a fourier series with alternating sine and cosine terms. @xcite also experimented with learnable pes, but these did not show significant performance differences compared to absolute sinusoidal embeddings. later works @xcite demonstrated the advantages of relative pes in transformers. @xcite discussed alibi, a simple relative pe strategy that adds relative token distances to the attention matrix, enabling models to generalize to longer sequences during inference, even when trained on shorter sequences. rotary positional embeddings (rope) @xcite introduced the concept of rotating vectors according to their positions to encode relative positional dependencies directly into the self-attention mechanism. recent works @xcite @xcite @xcite have also proposed adaptations of rope for much longer sequences. in this study, we focus on the standard variant of rope, though our method is generalizable to other variants. ## methodology in this section, we discuss the dataset curation, model training and evaluation procedures. ## selfattention, crossattention, feedforward 4 experiments we use the fairseq foot_4 framework @xcite for all of our experiments. all the finetuning experiments were conducted on 8 × h100 80gb gpus, and every model was trained till convergence with early-stopping. the exact hyperparameter sets are mentioned in ## results and discussions in this section, we discuss the results that aim to understand whether positional embeddings can be efficiently swapped posthoc in a trained model. ## additional experiments to assess the scalability of our approach, we extend our methodology to the larger 1b indictrans2 model. for consistency, we reuse the same training data and hyperparameters as described in section 3.2.1 and ## conclusion in this study, we demonstrated that standard nmt systems can be efficiently retrofitted with relative positional embeddings such as rope and alibi, enabling them to handle longer contexts. moreover, we showed that this retrofitting can be achieved through parameter-efficient fine-tuning using minimal document-level data, even across a few languages, with evidence of cross-lingual transfer. although our experiments focused on the machine translation (mt) task, we believe that this approach can generalize to other encoder-decoder and encoder-only models, potentially improving their long-context capabilities in a wide range of applications.