Context-Aware or Context-Insensitive? Assessing LLM s’ Performance in Document-Level Translation (2025)

Abstract: Large language models (LLMs) are increasingly strong contenders in machine translation. In this work, we focus on document-level translation, where some words cannot be translated without context from outside the sentence. Specifically, we investigate the ability of prominent LLMs to utilize the document context during translation through a perturbation analysis (analyzing models’ robustness to perturbed and randomized document context) and an attribution analysis (examining the contribution of relevant context to the translation). We conduct an extensive evaluation across nine LLMs from diverse model families and training paradigms, including translation-specialized LLMs, alongside two encoder-decoder transformer baselines. We find that LLMs’ improved document-translation performance compared to encoder-decoder models is not reflected in pronoun translation performance. Our analysis highlight the need for context-aware finetuning of LLMs with a focus on relevant parts of the context to improve their reliability for document-level translation.

## introduction language normally consists of collocated, structured, coherent groups of sentences referred to as a discourse @xcite . discourse properties that go beyond an individual sentence include the frequency and distribution of words within a document, topical, functional and discourse coherence patterns, and the use of reduced expressions. these properties have stimulated a good deal of machine translation research in the 1990s, aimed at endowing machine-translated target texts with the same properties as their source texts @xcite . since then, there has been a growing interest in document-level translation, mainly focused on document-level influences on lexical choice, and developing methods, annotated resources and assessment metrics for discourse-level machine translation @xcite . large language models (llms) show promise on multiple language technologies, with recent models specially finetuned for machine translation @xcite . @xcite suggest that translation llms have potential to be the new paradigm for document-level translation. while such work focuses only on assessing translation quality using metrics such as bleu or comet, our work investigates how models utilize context in translation. inspired by @xcite , we follow an interpretable approach towards context utilization evaluation. in particular, we focus on answering two main questions: how sensitive llms are to the correct context, and how well they utilize the relevant parts of context. for context sensitivity assessment, we compare the general and discourse-phenomena-specific @xcite translation performance of llms under the gold context setup to a perturbed context setup. for relevant-context utilization assessment, we perform a finer-grained evaluation. we look at models' internals using attribution methods @xcite to quantify the contribution of relevant context to the translation. context utilization in machine translation has been explored in encoder-decoder models, such as by @xcite , who developed an end-to-end interpretability framework to assess context-aware translation. to the best of our knowledge, we are the first to explore context utilization in translation llms via perturbation and attribution methods. our main findings can be summarized in the following: the structure of our paper is as follows: §2 provides an overview of the analyses conducted, while §3 outlines the experimental setup. in §4, we present and discuss the results of our experiments. a review of additional related work is included in §5, and we present our conclusions and suggestions for future work in §6. finally, §7 addresses the limitations of our research and our ethical considerations are detailed in §8. ## analysis overview this section presents an overview of the analyses we conducted. like mohammed and niculae (2024), we perform a perturbation analysis on translation quality and pronoun resolution accuracy. moreover, we examine model mechanics through an attribution analysis via interpretability methods. ## conclusion we use interpretability tools (perturbation and attribution techniques) to analyze llms' contextutilization in document-level translation. our experiments suggest that multilingual pretraining and translation-specific finetuning of llms pushes state-of-the-art translation performance beyond encoder-decoder models. however, we highlight that looking at discourse phenomena performance, llms show room for improvement. we argue that more care is needed before adopting llms as the new standard for document-level translation, and more focused evaluation must be considered. future research directions include enhancing context-aware translation capabilities of llms, potentially through explicit finetuning, and creating datasets with supporting-context annotations for other discourse phenomena to enable extending context-utilization analysis to those phenomena.