Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation (2025)

Abstract: Large language models (LLMs) have excelled in various NLP tasks, including machine translation (MT), yet most studies focus on sentence-level translation. This work investigates the inherent capability of instruction-tuned LLMs for document-level translation (docMT). Unlike prior approaches that require specialized techniques, we evaluate LLMs by directly prompting them to translate entire documents in a single pass. Our results show that this method improves translation quality compared to translating sentences separately, even without document-level fine-tuning. However, this advantage is not reflected in BLEU scores, which often favor sentence-based translations. We propose using the LLM-as-a-judge paradigm for evaluation, where GPT-4 is used to assess document coherence, accuracy, and fluency in a more nuanced way than n-gram-based metrics. Overall, our work demonstrates that instruction-tuned LLMs can effectively leverage document context for translation. However, we caution against using BLEU scores for evaluating docMT, as they often provide misleading outcomes, failing to capture the quality of document-level translation.

## introduction large language models (llms) have demonstrated exceptional performance across a wide range of natural language processing tasks @xcite @xcite . in the realm of machine translation (mt), recent findings also suggest that llmbased models rival dedicated commercial systems like google translate, particularly in translating high-resource languages @xcite @xcite . nonetheless, most research has focused only on sentence-level translation. while some studies have begun to explore document-level translation (docmt) with llms, there is a prevailing belief that directly applying instruction-tuned llms to docmt performs poorly without specialized training and prompting techniques, largely due to the limited availability of document-level content in instruction-tuning datasets @xcite @xcite . however, their conclusions are frequently drawn from n-gram-based metrics without thorough analysis to substantiate the models' true performance. in this work, we conduct an in-depth investigation into the inherent capabilities of instructiontuned llms in handling docmt tasks. unlike previous studies that explore special tricks, such as multi-turn inference @xcite , we directly prompt llms to translate entire documents in a single pass. comparing this method to a simpler baseline that translates individual sentences separately and then stitches them together, we can evaluate whether instruction-tuned llms can leverage their inherent ability to incorporate documentlevel context and improve translation quality. a key challenge in our research is the evaluation of document-level machine translation (docmt). traditional metrics foot_0 like bleu foot_1 , chrf, and ter @xcite @xcite , though widely used, often poorly correlate with human judgment @xcite , especially in docmt, where maintaining coherence and logical flow across a document is essential-something n-gram overlap struggles to capture. metrics like ctt, azpt, and blonde @xcite address specific aspects such as terminology consistency and zero-pronoun accuracy, but still rely heavily on word matching and symbolic statistics. we argue that an ideal docmt metric should be (1) context-aware-capturing document-level coherence and accuracy, (2) structured-evaluating aspects such as fluency, accuracy, and coherence separately, and (3) interpretable-explicitly identifying translation errors for clear, objective human evaluation. to this end, we design a gpt-4-based evaluation pipeline to provide deeper insights into the docmt capabilities of llms. ## problem settings given a document containing l source sentences @xmath0, • • • , x l }, the goal of docmt is to generate its translation @xmath1, • • • , y l ′ } as a sequence of sentences in the target language. in this work, we explore two approaches for generating translations using instruction-tuned llms: the doc approach is designed to capture intersentence dependencies by considering the full document context, potentially leading to more coherent and accurate translations. however, this approach requires the llm to process and generate longer sequences of text, which can increase the risk of cumulative errors, especially if the model has not been explicitly optimized for document-level translation. ## bleu-based evaluation document-level bleu (d-bleu, @xcite is widely used for evaluating translations in docmt. however, we notice that it is sensitive to overly lengthy generation, which can be problematic as llms sometimes overgenerate. we find that even minor overgeneration can significantly affect the final d-bleu score. 4 we argue that documents are generally independent units, so they should be weighted equally in the evaluation. we, therefore, propose an alternative, avgbleu, defined as: here, n is the number of documents, and y ref and y pred represent the reference document translations and the predicted translations, respectively. this allows us to calculate the average bleu score (avgbleu) for the entire dataset, providing a comprehensive measure of translation quality. evaluation setup. for evaluation, we use the test set from wmt22 @xcite , which includes sentence-level reference translations along with annotated document boundaries. documentlevel references are obtained by concatenating the corresponding sentence translations. we cover four translation directions in our evaluation: german (de) and chinese (zh) translated to and from english (en). specific dataset statistics are presented in results. vicuna-13b-16k as the only two exceptions. the specific value of k that yields the highest avg-bleu score varies depending on the translation direction, however, on average, st3 achieves the highest score overall. while independently translated sentences yield better avgbleu scores than document translations done in one pass by llms, manual inspection reveals that st[k] translations often contain more redundancy, literal translations, and disjointed phrasing. while these translations may achieve higher avgbleu scores, we find that doc translations result in more fluent, readable, and cohesive output. this raises concerns about how much avgbleu can be trusted as a metric for evaluating docmt. ## llm-as-a-judge evaluation maruf et al. (2021) outlines various discourse phenomena that should be considered when evaluating document-level translations, such as cohesion and the use of discourse connectives. in the past, automatic evaluation of these aspects was difficult due to the need for deep semantic understanding, and evaluations typically focused on one aspect at a time using specialized test sets @xcite @xcite . inspired by the "llm-as-a-judge" approach @xcite , we aim to assess mul-tiple aspects simultaneously using a strong llm. evaluation setup. we design four (sub) metrics: fluency is rated on a scale of 1 to 5, with higher being better. since fluency can be evaluated solely based on the translated text, we present only the model's outputs to gpt-4 for this assessment, decoupling fluency from metrics that require consideration of source and reference texts. content errors refer to translation mistakes such as mistranslations, omissions, or additions. we instruct gpt-4 (gpt-4-0613) to output a list containing all identified mistakes. the ce score is determined by the length of this list, and report the average ce score over the test set. 5 cohesion errors are further divided into two subcategories: lexical (le) and grammatical (ge), which affect text connection and the logic flow, respectively. le includes incorrect vocabulary usage, missing synonyms, or overuse of certain terms that disrupt the flow. ge includes pronouns, conjunctions, and sentence-linking structure mistakes. similar to ce, we prompt gpt-4 to generate a list of identified errors, with the score corresponding to the length of the list. other settings, such as translation directions and the models of interest, remain consistent with section 3. due to the cost associated with using gpt-4, we sample 70 documents per translation direction from the wmt22 dataset for our evaluation. results. the results with en-zh are shown in to gain a deeper understanding of how these metrics correlates with each other, we compute the pearson correlation coefficients (pcc) among those metrics and visualize them in figure 1 , as well as translation directions, showing that bleu score has poor correlation with those discourselevel phenomena metrics. other translation directions also exhibit low correlation results in appendix e. therefore, we suggest not using bleu score for docmt since it fails to account for discourse-level phenomena, and even worse, it often produces misleading results-such as suggesting that sentence translations are better. human agreement. while some judgments by the llm-as-a-judge may appear reasonable, certain nuances may still be misinterpreted due to unique human perspectives. to validate the alignment between our llm-as-a-judge paradigm and human evaluations, we conducted experiments to assess agreement. for each model in both st3 and doc, we used 10 samples per translation direction and asked human evaluators to respond with a simple "yes" or "no" regarding their agreement with the llm-as-a-judge's assessments according to our metrics. our manual evaluation confirmed a strong alignment between human judgments and the llm-asa-judge paradigm. as shown in ## conclusion in this work, we investigate the performance of instruction-tuned llms in document-level machine translation (docmt), comparing the translation of entire documents in a single pass to translating individual sentences that are then concatenated. our findings show that translating entire documents yields better results, as the model can capture inter-sentence dependencies and maintain discourse coherence, even without explicit fine-tuning for docmt tasks. however, evaluating these improvements is challenging. traditional metrics like d-bleu fail to consider discourse-level phenomena, often favoring sentence-level translations and producing misleading results. to address this limitation, we propose the llm-as-a-judge approach, utilizing gpt-4 to assess specific aspects of discourse through tailored prompts. this method enhances interpretability and can be adapted for evaluating translation quality in other domains.