BMX : Boosting Natural Language Generation Metrics with Explainability (2024)

Abstract: State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, we explicitly leverage explanations to boost the metrics’ performance. In particular, we perceive feature importance explanations as word-level scores, which we convert, via power means, into a segment-level score. We then combine this segment-level score with the original metric to obtain a better metric. Our tests show improvements for multiple metrics across MT and summarization datasets. While improvements on machine translation are small, they are strong for summarization. Notably, BMX with the LIME explainer and preselected parameters achieves an average improvement of 0.087 points in Spearman correlation on the system-level evaluation of SummEval.

## introduction modern language model (lm) based natural language generation (nlg) metrics achieve astonishing results in grading machine generated sentences like humans would (e.g., @xcite @xcite . as most language models are blackbox components, some recent works started to explore the explainability of lm-based metrics (e.g. @xcite @xcite @xcite . this exploration, for example, contributes to the foundation of ethical machine learning (e.g. fort and couillault, 2016; european_commission, 2019). 1 we make our code available at: https://github. com/gringham/bmx our work is motivated by an intriguing duality that we note between segment-level metrics and their explainability through feature importance techniques, e.g., @xcite : segment-level metrics foot_0 return a single score indicating the quality of a generated segment. feature importance explanations foot_1 increase the granularity of this score, by assigning additional wordlevel scores. these granular scores capture additional information about the generated text and about the metric that processed it, as, e.g., explored by the eval4nlp21 shared task @xcite and the wmt22 quality estimation shared task @xcite . on the other hand, in recent multidimensional quality metrics (mqm) datasets, word-level error annotations are converted into segment-level scores using heuristic functions @xcite . likewise, metrics like bertscore @xcite and bartscore @xcite build their segment-level scores upon word-level scores. in other words, we note the duality that feature importance techniques produce word-level scores from segment-level scores and heuristics can aggregate word-level scores into segment-level scores. figure 1 gives an example of this duality for machine translation (mt), where a german source sentence "ich habe einen hund" was wrongly translated into "i have a cat". on the right side, a segment-level score of 0.6 is assigned by a metric. on the left side, a feature importance explainer is used to explain this score by assigning word-level scores to each input token. instead of displaying the scores, we use colors to describe the concept. the red words would likely achieve a low importance score, as they are translated incorrectly. the duality arises as the feature importance scores can be recombined into a new segment-level score (here using power-means). in this work, we explore whether this duality leads to iterative improvements of segment-and word-level scores, with a focus on segment-level scores as these are the main goal of modern metrics. we propose boosting natural language generation metrics with explainability (bmx), a method that directly leverages word-level explanations to improve the original segment-level score of a metric. specifically, the approach aggregates wordlevel feature importance explanations using power means @xcite and combines them with the original score using a linear combination. to obtain the explanations, we leverage modelagnostic explainability techniques, allowing application to any nlg metric. while we consider nlg (especially mt and summarization) as 'natural use case', other regression and classification tasks follow similar settings, which makes our approach more generally applicable. for example, in sentiment classification, feature importance techniques might assign high importance scores to tokens with positive sentiment. hence, aggregating these scores could further inform a classification decision. we evaluate bmx with several metrics and explainability techniques on 5 mt datasets (3 for exploration + 2 held out for testing), as well as 2 summarization datasets, and discuss conditions for its failure and success. our work makes the following contributions: ## approach nlg metrics grade a generated text, also referred to as hypothesis, by comparing it to a ground truth. for mt, the ground truth could be a human written reference translation or the original text in source language. for summarization, the ground truth could be a human written reference summary or the source text that is being summarized. given a pair of ground truth segment @xmath0, ..., g n ⟩ and this score can be interpreted as, for example, the adequacy/accuracy of the generation of h given g. our algorithm consists of three steps: (1) compute feature importance explanations, (2) aggregate explanation scores, and (3) combine the aggregated explanations with the original score. ## experiment setup in this section, we describe the datasets, metrics, explainers and aggregation methods that we evaluate in §4 and their parameter configurations. ## results in this section, we evaluate the effectiveness of bmx by correlating the results with human judgments of mt and summarization quality annotated in the datasets described in §3.1. to start, we calibrate the parameters p and w. calibrating p and w we perform a grid search on the calibration sets (see §3.1) to determine the parameters w and p for our evaluation of bmx on the evaluation sets. for p, we test 600 equally spaced values in [-30, +30] and for w, we test 6 equally spaced values in [0, 1] (where @xmath1. this results in 3000 bmx configurations (without @xmath2. next, we evaluate all p-w-metric-explainer combinations on the respective calibration set(s). specifically, for the mt calibration sets we evaluate with segment-level pearson correlation (see eq. 1) for each language pair, and for summarization we evaluate with system-level kendall correlation. for our evaluation, we select the median of the p and w values that led to any increase over the original correlation on the calibration set(s) for each metric-explainer combination. 9 our approach of selecting p and w is rather simple. future work might consider more sophisticated ways of optimization, such as considering the areas of highest increase in the grid search or even learning a model to set the parameters based on input segments. figure 2 shows exemplary box-plots of p and w for xbertscore, to illustrate the distributions we select from. ## analysis in this section, we compare bmx to a fine-tuned metric on a summeval split, analyze the failure in realsumm and explore the stability of the metric when using the lime explainer. comparison to fine-tuning a metric we use the out-of-the-box training script of bartscore to fine-tune bartscore on the reference-hypothesis pairs of the first calibration split of summeval. then, we evaluate the fine-tuned metric, the original metric and bmx on the first test split and compare the results (see figure 3 ). the tuned metric has a better coherence than the original metric and bmx, however, all other aspects are worse than original. bmx has the highest correlation in all other dimensions, which shows that it can use the small-scale training set more efficiently. mt failure analysis for some settings, for example with comet, changes are extremely small. to understand bmx' internal workings, we plot the human scores and the two factors of the linear combination (the original score and the aggregated feature importance scores) for comet on wmt22 cs-en (see the figure in appendix g). the scores are ordered by the human scores from high to low and normalized by z-scoring. we find that many scores that were aggregated from the explanations are uniform, with few outliers. hence, adding them to the original comet will hardly change the results. future work could further explore the causes. as the system-level correlation decreased for some test setups on the mqm dataset, we further suspect that the transfer of p and w from the calibration sets to the evaluation set did not work out well, resulting in decreased correlations. to test this, we perform another grid-search on p and w and analyze whether other parameter settings would have performed better. the analysis shows that, even for comet, the best parameter choice could lead to improvements of over 0.07 kendall points, with a choice of @xmath3.2 and a good selection of p (see the figure in appendix 7). for transquest, the improvements can be over 0.06 kendall points in en-de with @xmath4.8. determining p and w in an 4 . a choice of @xmath5.3 (over 100% improvement). for bartscore, the correlation could be improved by over 0.05 with the correct selection (see appendix f). determining the values in a similar stratification setting as with summeval might thus have led to better results. ## conclusion we have presented bmx: boosting natural language generation metrics with explainability, a novel approach that leverages the duality of nlg metrics and feature importance explanations to boost the metrics' performance. bmx leverages model-agnostic explainability techniques, so that it can be applied to any nlg metric. additionally, it requires no supervision once the initial parameters for p and w are set, which might benefit fully unsupervised or weakly supervised approaches to inducing evaluation metrics @xcite . our tests show consistent improvements for multiple configurations on all tested datasets. notably, we demonstrate strong improvements for summarization with 0.074 points in kendall correlation on the system-level evaluation of summeval, being significant on many test splits. on realsumm, bmx is not as strong, but our analysis shows that a better choice of p and w could lead to strong improvements on this dataset as well. to the best of our knowledge, our approach is the first to leverage the duality of segment-level mte metrics and their feature-importance explanations directly and we believe that it can lead a step forward towards integrating metrics with explainability. future work should also consider to which degree bmx can improve the explainability of metrics and apply our framework to other regression and classification tasks, beyond mt and summarization metrics. future work should also examine how to effectively leverage higher-level iterations.