Long-context Reference-based MT Quality Estimation (2025)

Abstract: In this paper, we present our submission to the Tenth Conference on Machine Translation (WMT25) Shared Task on Automated Translation Quality Evaluation. Our systems are built upon the COMET framework and trained to predict segment-level ESA scores using augmented long-context data. To construct long-context training examples, we concatenate multiple in-domain sentences and compute a weighted average of their scores. We further integrate human judgment datasets MQM, SQM, and DA) through score normalisation and train multilingual models on the source, hypothesis, and reference translations. Experimental results demonstrate that incorporating long-context information yields higher correlations with human judgments compared to models trained exclusively on short segments.

## introduction the automatic evaluation of machine translation (mt) is a crucial component of mt research and development. while expert-based human evaluation remains the gold standard, automatic evaluation offers fast and scalable judgments, enabling rapid feedback for optimizing model parameters. traditionally, automatic mt evaluation metrics have relied on basic lexical-level features, such as counting matching n-grams between the mt hypothesis and the reference translation. metrics such as bleu @xcite , meteor @xcite , and chrf @xcite remain popular due to their lightweight design and computational efficiency @xcite . more recently, neural metrics (either trained on human annotations or based on pre-trained language models) have demonstrated superior capability in comparing and assessing mt quality, often outperforming traditional lexical-based metrics @xcite . these neural approaches leverage largescale multilingual data during training and achieve strong performance even when translations diverge lexically from the reference. this paper presents dcu_adapt's submission to the wmt25 foot_0 mt evaluation shared task. the primary focus of this year's task is on systems capable of evaluating translation quality in context, where the context spans entire documents or multiple consecutive segments. we participated in the segment-level quality score prediction track for english-czech, english-russian, english-japanese, and english-chinese, employing models based on the comet framework @xcite . in our contribution to the shared task, we explore methods for leveraging synthetic data alongside the capabilities of pre-trained and cross-lingual models to predict mt quality estimates for longsequence or multi-sentence units. human judgments of mt quality are typically available as short segment-level scores, such as da @xcite , mqm @xcite , and sqm @xcite . recent pre-trained models support larger context windows and can handle long-sequence inputs, improving discourse-level resolution @xcite )-albeit at the cost of increased memory and computational requirements. however, most existing automatic evaluation metrics (aems) predict scores at the sentence level, and those designed for document-level evaluation often perform only shallow context integration during inference @xcite . we propose a data augmentation strategy to train multilingual models on long-context annotated data, enabling them to better exploit broader context and reduce inconsistencies caused by sentence-level ambiguity. the exploration of context in mt is a wellestablished topic and, in recent years, has become a focal point, driven by the need to incorporate context into both mt systems and their evalua-tion methodologies @xcite @xcite @xcite . there is now broad consensus on the value of document-level evaluation. since 2019, wmt has conducted human evaluations at the document level, providing evaluators with access to context even when collecting segment-level ratings @xcite @xcite . research indicates that the appropriate context span is critical for reliable mt evaluation, with @xcite showing that incorporating relevant context spans can yield more accurate assessments of translation quality, thereby improving the evaluation process. several techniques have been proposed to extend evaluation to the document level or to incorporate multi-sentence context into automatic evaluation metrics @xcite @xcite @xcite . for this shared task submission, we use the estimator model from the comet framework @xcite , which learns mt quality from human evaluation data such as mqm and da. to create long-context training data, we combine multiple annotations using a weighted average alongside the original annotations. our experiments show promising progress toward improved correlation in multi-sentence-level mt quality estimation. fine-tuning multilingual embedding models demonstrates that it is possible to achieve high correlations with human judgments when evaluating long segments, rather than relying solely on sentence-level score predictions. we release the data and code produced during this research. ## corpora we used the human annotations from previous wmt shared tasks @xcite @xcite for training our models, which includes human annotations from mqm, da, and sqm. we train and evaluate our models for english (en) to czech (cs), japanese (ja), chinese (zh), and russian (ru) language pairs. mqm scores are derived from error annotations and can range from -∞ to 100. since our goal is to predict esa scores @xcite , which range between 0 and 100, we normalise (using equation 1, where x is original and x ′ is normalised score) and rescale the scores to the [0, 1] interval for training models on the combined dataset. we create augmented training data for long-span mt quality estimation by taking a weighted average of segment-level scores. during augmentation, we concatenate multiple samples (i.e., 2, 3, 4, and 5 segments) to form long-span texts. to construct the long-span mt evaluation dataset, adjacent short segments are concatenated, and a document-level quality score is computed as a length-weighted average of their original scores. the weighting is based on the total number of characters in the source and machine-translated texts, as formalized in equation 2 . let s 1 , s 2 are short segments (e.g., sourcetranslation pairs), and raw 1 , raw 2 human evaluation scores for these segments. c 1 and c 2 are the total character count of each segment e.g., c @xmath0. then the document-level score (raw doc ) is calculated as: this equation foot_1 computes a weighted average of two segment-level scores, where the weight is determined by the combined character count of the source and mt for each segment. longer segments contribute more to the final score, reflecting their higher informational content. the augmentation process increased the average segment length of dataset from 16.84 words per segment to 52.99 words per segment. since the augmented segments were added to the original dataset, the overall size of the dataset increased by a factor of two. we then create training, test, and validation sets by randomly sampling segments from the training data. the statistics of the final training, validation, and test sets are shown in ## experimental setup our system is built on top of the comet package, utilizing the comet-train and comet-score commands to train and evaluate our models. we fine-tuned the pre-trained model unbabel/wmt22-comet-da, originally trained on da data, as well as the multilingual pretrained model facebookai/xlm-roberta-base @xcite , using a100 gpu. the xlm-roberta model was pre-trained on 2.5tb of filtered commoncrawl data containing 100 languages and has approximately 279 million parameters. we trained the models for up to 5 epochs and employed early stopping when the spearman correlation on the development set did not improve for two consecutive evaluations. for each language pair, the augmented dataset contained five times the original data; however, due to limited gpu memory, we faced out-of-memory issues and restricted training to augmentations with up to two segments. the training process with the augmented dataset took approximately 10 hours for each model. we trained one baseline model (the fine-tuned version of wmt22-comet-da) and three main models (one primary and two secondary submissions) using the data augmentation approach, retaining only the last two checkpoints for each. spearman correlation on the test split was used to select the best checkpoint per language pair, and this best model was used for the final submission. for official wmt test25 3 evaluation, the full segment was used since it fell within the maximum sequence length (512) supported by both the baseline and fine-tuned models. comet scores typically range between 0 and 1, but can sometimes exceed 1, indicating exceptionally high-quality segments. to align with the esa metric's scoring strategy @xcite , we upscaled and rounded the scores to a range between 0 and 100. ## results as described in section 2, our experiments use the normalized versions of multiple human annotations collected from previous wmt shared tasks. to evaluate and compare our approach, we applied a similar augmentation method to construct long-sequence test data, incorporating annotations from da, mqm, and sqm. our baseline sentence-level quality estimation models are wmt22-comet-da (referred to as comet-22) and bertscore. comet-22-ls, a fine-tuned version of wmt22-comet-da on long-span data, and roberta-ls, fine-tuned from facebookai/xlm-roberta-base, both serve as long-sequence quality score prediction models. following @xcite , the models were trained on triplets of (source, hypothesis, reference) and output a score between 0 and 1 reflecting the translation quality relative to both source and reference. we used the pearson correlation coefficient to evaluate the models' performance. segment-level pearson correlations on the self-test set are presented in tables 2 and 3 . our results indicate that metrics from models trained on long-context inputs generally outperform sentence-level metrics, in some cases by a significant margin. the annotation-wise segment-level correlation results in these results suggest that combining multiple segments within the same document or domain is more effective than independently scoring segmented sentences and averaging their scores @xcite . the improvement may be attributed to the model's ability to capture contextual information across long-sequence segments, thereby enabling more context-aware quality estimation. however, handling longer text sequences poses challenges due to the input size limitations of the underlying models, as highlighted by @xcite . this necessitates careful segmentation and score-averaging strategies to compute scores at the paragraph or document level. we also conduct a preliminary analysis of the score distributions (figure 1 ) and find that sentence-level baseline scores (comet-22) are mostly concentrated between 60 and 100, with a pronounced peak around 90. in contrast, the roberta-ls model, trained on multisentence inputs, produces a more widely spread score distribution that better reflects the variability typically observed in human judgments @xcite . this wider spread may be due to the long-span training data being based on weighted average scores that encompass a broader range of score scales. by contrast, the narrower distribution of comet-22 scores could stem from the characteristics of the human-annotated data on which it was trained, where non-expert evaluators have been shown to assign disproportionately higher fluency and adequacy ratings, resulting in smaller score gaps and reduced variance compared to expert assessments @xcite . this indicates that models trained with longer context are more sensitive to subtle quality differences, reflecting a more nuanced understanding of translation quality. ## related work in recent years, metrics based on large pre-trained models have emerged as strong alternatives to traditional n-gram-based approaches, enabling better capture of semantic similarity between words beyond mere lexical matching. these metrics broadly fall into two categories: embedding-based metrics and fine-tuned metrics. embedding-based metrics typically represent an advancement over n-gram matching by using dense word representations in an embedding space to compute scores that reflect semantic similarity between reference and hypothesis segments. notable examples include yisi-1 @xcite , mover-score @xcite , and bertscore @xcite , which leverage embedding models for soft alignment between two segments to capture semantic similarity effectively. fine-tuned metrics, on the other hand, involve learnable models such as ruse @xcite , bleurt @xcite , and comet @xcite that directly optimize underlying embedding models to maximize correlation with human judgments. these models have demonstrated promising results in producing reliable quality scores for test sets such as da or mqm. while most of these metrics perform referencebased evaluation, recent advancements leveraging highly multilingual pre-trained encoders like multilingual bert @xcite and roberta @xcite have enabled reference-less systems to show encouraging correlations with human judgments @xcite . most automatic evaluation approaches rely on decontextualized assessments, where translations are judged at the sentence level. however, sentences are often inherently ambiguous, and incorporating document-level context has been shown to be beneficial for both mt and its evaluation @xcite @xcite . consequently, a few automatic metrics have been developed to extend evaluation beyond the word or sentence level @xcite . these metrics aim to address discourselevel phenomena such as lexical consistency, coherence, ellipsis, and pronoun resolution @xcite . however, existing methods typically use a limited number of surrounding sentences as context, allowing models to incorporate neighboring information when embedding each sentence and computing scores at the sentence level @xcite @xcite . in contrast, long-sequence or document-level evaluation processes the entire segment as a single input, enabling deeper discourse-level resolution and offering a promising yet still underexplored avenue for improving alignment with human judgments. ## conclusion in this paper, we present dcu_adapt's contribution to the wmt25 mt evaluation shared task. we leverage the comet framework and train regression models to predict esa quality scores. in line with the shared task goals, we augment the provided training data and optimize our models to evaluate long, multi-sentence units of text. by fine-tuning multilingual models for cross-lingual transfer, we utilize source, reference, and hypothesis as inputs. our primary submission -a finetuned pre-trained model trained on augmented data -demonstrates higher or otherwise competitive correlation levels with human judgments across multiple languages. further investigation comparing long-text evaluation after segmentation with sentence-level evaluation is a promising direction for future work. the data and code produced during this shared task participation are available at: https://github.com/sami-haq99/caemt/ tree/main/wmt-2025-submission .