The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics (2023)

Abstract: Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, “black boxes” returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: https://github.com/Unbabel/COMET/tree/explainable-metrics

## introduction reference-based neural metrics for machine translation evaluation are achieving evergrowing success, demonstrating superior results over traditional lexical overlap-based metrics, such as bleu @xcite and chrf @xcite , in terms of both their correlation with human ratings and their robustness across diverse domains @xcite @xcite @xcite . however, lexical overlapbased metrics remain popular for evaluating the performance and progress of translation systems and algorithms. concerns regarding trust and interpretability may help explain this @xcite : contrary to traditional metrics, neural metrics are considered "black boxes" as they often use figure 1 : illustration of our approach. in this example, the metric assigns the translation a low score. we aim to better understand this sentence-level assessment by examining the correspondence between our token-level explanations and human annotated error spans. while some recent work has focus on explaining the predictions made by reference-free quality estimation (qe) systems @xcite , explaining reference-based metrics has remained a largely overlooked problem @xcite . it is an open question whether the observations from studies of explainable qe carry over to this scenario. thus, in this work, we fill that gap by turning to state-of-theart reference-based metrics-we aim to interpret their decision-making process by exploiting the fact that these metrics show consistently good correlations with multidimentional quality metrics (mqm) @xcite @xcite , which are fine-grained quality assessments that result from experts identifying error spans in translation outputs @xcite . we hypothesize that reference-based metrics leverage this tokenlevel information to produce sentence-level scores. to test this hypothesis, we assess whether our explanations -measures of token-level importance obtained via attribution and input attribution methods such as attention weights and gradient scores @xcite ) -align with human-annotated spans @xcite @xcite , as illustrated in figure 1 . our analysis focuses on two main vectors: (i) understanding the impact of the reference information on the quality of the explanations; and (ii) finding whether the explanations can help to identify potential weaknesses in the metrics. our main contributions are: • we provide a comparison between multiple explainability methods for different metrics on all types of evaluation: src-only, ref-only, and src+ref joint evaluation. ## explaining neural metrics we aim to explain sentence-level quality assessments of reference-based metrics by producing token-level explanations that align to translation errors. in what follows, we describe the metrics and how we produce the explanations that we study. ## experimental setting mqm annotations. we use mqm annotations from the wmt 2021 metrics shared task (freitag et al., 2021), foot_1 covering three language pairs -english-german (@xmath0, english-russian (@xmath1, and chinese-english (@xmath2: news and ted talks. for each incorrect translation, human experts marked the corresponding error spans. in our framework, these error spans should align with the words that the attribution methods assign higher importance to. ## conclusions and future work in this paper, we investigated the use of explainability methods to better understand widely used neural metrics for machine translation evaluation, such as comet and unite. concretely, we analyzed how explanations are impacted by the reference information, and how they can be used to reveal weaknesses of these metrics. our analysis shows that the quality of the explanations is tightly related to the underlying metric architecture. interestingly, we also provide evidence that neural metrics like comet may heavily rely on reference information over source information. additionally, we show that explanations can be used to reveal reference-based metrics weaknesses such as failing to severely penalize localized critical errors. this opens up promising opportunities for future research on leveraging explanations to diagnose reference-based metrics errors. to support these studies, we call for future datasets illustrating critical errors (e.g., challenge sets @xcite ) to be accompanied by annotated error spans.