x COMET : Transparent Machine Translation Evaluation through Fine-grained Error Detection (2024)

Abstract: Widely used learned metrics for machine translation evaluation, such as COMET and BLEURT, estimate the quality of a translation hypothesis by providing a single sentence-level score. As such, they offer little insight into translation errors (e.g., what are the errors and what is their severity). On the other hand, generative large language models (LLMs) are amplifying the adoption of more granular strategies to evaluation, attempting to detail and categorize translation errors. In this work, we introduce xCOMET, an open-source learned metric designed to bridge the gap between these approaches. xCOMET integrates both sentence-level evaluation and error span detection capabilities, exhibiting state-of-the-art performance across all types of evaluation (sentence-level, system-level, and error span detection). Moreover, it does so while highlighting and categorizing error spans, thus enriching the quality assessment. We also provide a robustness analysis with stress tests, and show that xCOMET is largely capable of identifying localized critical errors and hallucinations.

## introduction automatic metrics for machine translation evaluation are widely used by researchers and practitioners to evaluate the quality of translations and the systems generating them. notably, learned neural metrics, such as comet @xcite and bleurt @xcite , have demonstrated significant improvements in terms of correlation with human judgments when compared to traditional metrics like bleu @xcite @xcite . these metrics are trained to regress on scores obtained through human annotations, by predicting a single sentence-level score representing the quality of the translation hypothesis. however, these single scores do not offer a detailed view into translation errors (e.g., it is not immediate which words or spans of words are wrongly translated). moreover, as they are obtained by making use of highly complex pre-trained models, they can be difficult to interpret @xcite . one appealing strategy to bring a more detailed view into translation errors is to obtain finer-grained information on error spans through highlighting them and indicating their severity @xcite @xcite . in fact, this is the strategy adopted in recent work that has employed generative large language models (llms) for machine translation evaluation: (i) identify errors within a given translation, subsequently (ii) categorize these errors according to their severity, and finally (iii) infer a sentence-level score from the predicted errors @xcite . however, these methods still lag behind dedicated learned metrics when using open llms, such as the llama models @xcite . as it stands, competitive performance with generative strategies remains contingent on utilizing large proprietary, closed llms such as palm-2 and gpt-4 @xcite . in this work, we bridge the gap between these two approaches to machine translation evaluation by introducing xcomet: a learned metric that simultaneously performs sentence-level evaluation and error span detection. through extensive experiments, we show that our metrics leverage the strengths of both paradigms: they achieve state-of-the-art performance in all relevant vectors of evaluation (sentence-level, system-level, and error span prediction), while offering, via the predicted error spans, a lens through which we can analyze translation errors and better interpret the sentence-level scores. we achieve this by employing a curriculum during training that is focused on leveraging high-quality publicly available data at both the sentence-and error span level, complemented by synthetic data constructed to enhance the metric's robustness. moreover, xcomet is a unified metric @xcite , supporting all modes of evaluation within a single model. this enables the metric to be used for quality estimation (when no reference is available), or for reference-only evaluation, similarly to bleurt (when a source is not provided). crucially, xcomet also provides sentence-level scores that are directly inferred from the predicted error spans, in the style of automqm @xcite and instructscore @xcite . our contributions can be summarized as follows: in post-edits (pe), professional translators are tasked with ''fixing'' a given translation, making minimal edits to improve its quality. using this edited translation-often termed post-edit-we can evaluate the machine translation output by quantifying the number of edits, thus gauging the initial translation's quality @xcite . direct assessments (da) @xcite ) are a simple and widely-used evaluation method. annotators-non-expert bilingual speakers or professional translators-are asked to annotate each translation with a score ranging from 0 to 100 to reflect its adequacy and fluency, where a score of 100 corresponds to a perfect translation, and 0 corresponds to a completely inadequate one. the multidimensional quality metrics (mqm) framework @xcite , on the other hand, offers a more comprehensive and systematic approach to mt evaluation. professional translators highlight errors-typically in the form of error spans-within translations, attributing them severity ratings (e.g., minor, major, or critical) and categorical labels (e.g., fluency, accuracy). figure 1 illustrates one such annotation. mqm annotations have gained prominence in recent years due to their capacity to offer detailed insights into translation errors, facilitating more fine-grained and accurate comparisons between translation systems @xcite . as such, the field of automatic evaluation of mt has increasingly favored comparisons using mqm annotations over traditional da and pe methodologies @xcite @xcite . ## problem statement an automatic metric for translation evaluation aims at predicting the quality of a translated sentence, t, in light of a reference translation, r, for a given source sentence, s. here, we focus specifically on neural metrics that make use of a neural model, and typically operate under one of the following evaluation scenarios: the model evaluates the translation by jointly processing it with both the source and the reference (comet is an example of such a metric); in essence, the model's input sequence consists of the translation t paired with some additional input-either r, [r, s], or s-derived from the scenarios above. given this input, the model may predict the quality of the translation at different granularities, e.g., sentence-level or word(span)-level. ## design and methodology of xcomet in this section, we describe the methodology behind xcomet, outlining its model architecture, training settings and corpora, and learning curriculum. we detail how the model is designed to perform both regression and error span detection while adopting a unified input approach for enhanced flexibility and performance. ## correlations with human judgments in this section, we present a standard performance analysis of our metrics in terms of correlations with human judgments. overall, we find xcomet to be a state-of-the-art in sentence-level and error span prediction, being competitive with generative llms in terms of system-level evaluation. sentence-level evaluation. system-level evaluation. ## robustness of xcomet to pathological translations we have shown that xcomet metrics exhibit state-of-the-art correlations with human judgements when evaluating on high-quality mqm annotations. however, these mqm annotations are often highly unbalanced and contain little to no major or critical errors. as such, they may not offer a full picture of the metrics' performance. in this section, we shift our focus to studying how xcomet metrics behave when evaluating translations with localized major or critical errors, and highly pathological translations, such as hallucinations. ## ablations on design choices we now address relevant questions about the development of xcomet through ablations on design choices. ablations are run with xcomet-xl on the source: das teilabonnement für international tätige juristen. translation: the sub-sub-sub-sub-scription for international lawyers. reference: partial subscriptions for internationally active lawyers. oscillatory hall. with xcomet error span predictions: the crit: sub-sub-sub-sub-scription for international lawyers. source: empfehlenswert gleich mit der zimmerreservierung zu buchen! translation: the staff were very friendly and helpful. the room was clean and comfortable. reference: we recommend booking your treatments together with the hotel booking! fully detached hall. with xcomet error span predictions: crit: the staff were very friendly and helpful. the room was clean and comfortable. ## conclusions we introduced xcomet, a novel suite of metrics for machine translation evaluation that combines sentence-level prediction with fine-grained error span prediction. through extensive experiments, we have shown that xcomet is a state-of-the-art metric at all relevant vectors of evaluation: sentence-level, system-level, and error span prediction. notably, through xcomet's capabilities to predict error spans, we can not only obtain useful signals for downstream prediction (either directly through error span prediction or by informing sentence-level scores) but also gain access to a lens through which we can better understand and interpret its predictions. we also stress-tested the metrics by assessing how they score localized critical errors and hallucinations: the metrics identify the vast majority of localized errors and can appropriately penalize the severity of hallucinations. we hope xcomet can serve as a step towards more informed machine translation evaluation.