Context Is Ubiquitous, but Rarely Changes Judgments: Revisiting Document-Level MT Evaluation (2025)

Abstract: As sentence-level performance in modern Machine Translation (MT) has plateaued, reliable document-level evaluation is increasingly needed. While the recent FALCON framework with pragmatic features offers a promising direction, its reliability and reproducibility are unclear. We address this gap through human evaluation, analyzing sources of low inter-annotator agreement and identifying key factors. Based on these findings, we introduce H-FALCON, a Human-centered refinement of FALCON. Our experiments show that, even with limited annotator consensus, FALCON achieves correlations comparable to or better than standard sentence-level protocols.Furthermore, we find that contextual information is inherent in all sentences, challenging the view that only some require it. This suggests that prior estimates such as “n% of sentences require context” may stem from methodological artifacts. At the same time, we show that while context is pervasive, not all of it directly influences human judgment.

## introduction the conventional approach to automatic machine translation (mt) evaluation has focused primarily on sentence-level analysis, emphasizing lexical overlap or n-gram similarity, as seen in bleu @xcite , @xcite , and chrf @xcite . more recent methods account for semantic similarity through embedding-based metrics such as bertscore @xcite and comet @xcite , while llm-based (large language model) metrics, including @xcite and meta-metrics @xcite , demonstrate improved alignment with human judgments. despite these advances, their scope remains confined to sentence-level evaluation, failing to capture discourse phenomena such as cohesion, coreference, consistency, and pragmatic adequacy. documentlevel metrics have been proposed @xcite @xcite ), but they typically target narrow aspects of discourse and lack comprehensive coverage. human evaluation at the document level poses additional challenges due to the complexity of quantifying context-dependent phenomena. approaches that rely only on overt discourse markers risk underestimating the role of context @xcite . furthermore, protocols vary in context length, annotation granularity, and guideline specificity @xcite . the resulting cognitive burden on evaluators can lead to longer annotation times and reduced inter-annotator agreement (iaa) @xcite @xcite . collectively, these factors render documentlevel evaluation both methodologically complex and resource-intensive, limiting its adoption in mt research and practice @xcite . to address this gap, the falcon framework (functional assessment of language and contextuality in narratives; kim 2025) integrates pragmatic features into a structured document-level protocol, with llms as judges. however, its human evaluation component remains underdeveloped and untested for reproducibility and reliability. we therefore conduct a meta-evaluation of fal-con through human assessments with professional translators, and extend the protocol by introducing h-falcon, a reproducible and streamlined human evaluation framework. our contributions are as follows: ## related works document-level evaluation is not merely a scaledup version of sentence-level evaluation; it captures translation phenomena that rely on extended context, such as coreference resolution, lexical cohesion, discourse connectives, and pragmatic intent @xcite . these features recur across the document, with evidence distributed over multiple segments, shaping a distinctive atmosphere or nuance (halliday and matthiessen, 2004). evaluating such phenomena enables a more accurate assessment of mt systems that appear statistically indistinguishable at the sentence level @xcite . this section reviews prior efforts in both manual ( § 2.1) and automatic ( § 2.2) evaluation of document-level phenomena, including falcon ( § 2.3). ## experiment setup we conduct direct human evaluation of falcon across two tasks and assess whether the current experimental design yields reproducible human judgments ( §4). using these gold annotations, we ## meta-evaluation of falcon the highest human iaa in our configuration is κ = 0.50 for task i and @xmath0.61 for task ii. using these gold scores as reference, we evaluate the reliability of falcon as an llm-as-judge framework. as baselines, we test multiple proprietary models-openai' s gpt-o3, o4-mini, and the baseline from kim (2025), 4.1-mini. model performance is assessed using the same reproducibility metrics defined in §3.4, complemented by accuracy for task i, where the output is a single categorical label, and micro-f1 for task ii, where multiple labels must be selected simultaneously. ## refined protocol: h-falcon the current protocol of falcon suffers from ambiguous definitions of context and limited reproducibility in skill selection, calling into question its central hypotheses. building on these findings, we identify three structural limitations of falcon: unclear translation objectives for human evaluators, the rigid requirement to assign exactly three skills per sentence, and the lack of adaptability to the domain and language pair. to address them, we propose h-falcon (humancentered falcon), grounded in two revised hypotheses: (i) every sentence is influenced by context, and (ii) judges should flexibly decide the number of translation skills. ## conclusion our findings challenge prevailing assumptions in mt evaluation by demonstrating that contextual information, though modest in magnitude, is both universal and consequential for human judgment. operationalizing this insight, h-falcon provides a reproducible, context-aware evaluation protocol that aligns as closely with human preferences as traditional sentence-level approaches. these results underscore the need to move beyond narrow, sentence-bounded metrics toward richer documentlevel assessments that capture the pragmatic realities of translation quality. as mt performance converges at the sentence level, such holistic, contextsensitive evaluation will be essential for driving the next phase of progress in the field.