BUMP : A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics (2023)

Abstract: The proliferation of automatic faithfulness metrics for summarization has produced a need for benchmarks to evaluate them. While existing benchmarks measure the correlation with human judgements of faithfulness on model-generated summaries, they are insufficient for diagnosing whether metrics are: 1) consistent, i.e., indicate lower faithfulness as errors are introduced into a summary, 2) effective on human-written texts, and 3) sensitive to different error types (as summaries can contain multiple errors). To address these needs, we present a benchmark of unfaithful minimal pairs (BUMP), a dataset of 889 human-written, minimally different summary pairs, where a single error is introduced to a summary from the CNN/DailyMail dataset to produce an unfaithful summary. We find BUMP complements existing benchmarks in a number of ways: 1) the summaries in BUMP are harder to discriminate and less probable under SOTA summarization models, 2) unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics, and reveals that the most discriminative metrics tend not to be the most consistent, and 3) unlike datasets containing generated summaries with multiple errors, BUMP enables the measurement of metrics’ performance on individual error types.

## introduction although modern abstractive summarization systems have improved in their ability to produce fluent text @xcite , their ability to generate text that is factually grounded in the source article remains an issue @xcite . this phenomenon has inspired the nlp community to develop faithfulness evaluation metrics @xcite @xcite that automatically measure the extent to which abstractive summarization systems produce summaries that contain information that cannot be verified by the source article. as the number of these automatic faithfulness metrics has increased, there has arisen a corresponding need for benchmarks that evaluate their relative strengths. to satisfy this need, researchers have developed datasets such as frank @xcite and true @xcite that are comprised of model-generated summaries along with human-annotated faithfulness levels. although these datasets are useful for evaluating the degree to which faithfulness metrics correlate with human judgements and can discriminate unfaithful summaries, a number of factors limit the conclusions that can be drawn from them. for one, because model summaries can vary in terms of length, content, and number of errors, these benchmarks are ill-suited for drawing conclusions about the consistency @xcite of metrics, i.e., whether their scores indicate lower faithfulness as summaries become increasingly unfaithful, as well as their sensitivity to specific types of errors since summaries can contain multiple errors. furthermore, as the summaries are machine-generated, these benchmarks cannot evaluate whether metrics can detect human-written unfaithful summaries. to enable research on these topics, we present bump-a benchmark of unfaithful minimal pairsa dataset of 889 minimally different summary pairs where all unfaithful summaries are generated by human annotators. as illustrated in figure 1 , given an article and its reference summary, we ask a human annotator to edit the reference summary in a minimal way such that the edited summary exhibits one unfaithful error. we design two tasks for performance comparisons: 1) taxonomy-based edits, where a specific unfaithfulness error type is required according to our proposed taxonomy, and 2) freestyle edits, where no error type constraints are imposed. the motivation behind the first task setting is to ensure that different error types are adequately represented in our dataset, while the second task setting is important for understanding article: manchester city and chelsea are set to battle it out for the signature of west ham left-back aaron cresswell this summer... west ham snapped up cresswell for £2million from ipswich last summer but he has adapted to the top tier with relative ease and attracted the eye of the division's champions league clubs... england stars jack wilshere, jordan henderson and liverpool contract rebel raheem sterling are also on city's wanted list to fulfil the quota. premier league sides must name eight homegrown players in their 25-man squad registered at the beginning of the season. a player is deemed homegrown if they have spent at least three years at any english club before the age of 21, regardless of nationality. ## related work standard evaluation metrics for text generation tasks, e.g., bleu and rouge, do not correlate well with human judgements of factual alignment in summarization settings @xcite . this has motivated the development of automated faithfulness metrics that quantify factual alignment through methods that either: use nli to measure the entailment degree between the source article and summary @xcite @xcite , compare summary probabilities when relevant information is removed from the source @xcite , or use question answering models to measure if questions derived from the source can be answered by the summary and vice versa @xcite @xcite . existing faithfulness metric evaluations use one of two classes of benchmarks: 1) machinegenerated summaries paired with human-annotated faithfulness levels @xcite @xcite , and 2) summary pairs pertaining to the same article where one summary is faithful and the other is unfaithful @xcite . while both classes can evaluate a metric's ability to discriminate unfaithful summaries, the latter additionally allows for consistency tests, i.e., whether metrics assign higher values to more faithful summaries. the bump dataset belongs to the second class of benchmarks; however, it has a number of unique properties. first, unlike both @xcite and @xcite , the unfaithful summaries in bump are human-written. in addition, the unfaithful summaries in bump are minimally different, in the sense that only a single error differentiates the faithful and unfaithful summary. as shown in section 5, this produces summary pairs that are substantially more challenging for metrics to differentiate. inspired by the use of minimal pairs to diagnose linguistic knowledge of language models @xcite , the benefit of this approach is that it allows targeted tests of a metric's consistency on different types of errors (section 3.2) while minimizing the effect of confounding factors. therefore, unlike other benchmarks with error type annotations @xcite , results on bump are not complicated by issues such as multiple errors appearing in the same summary. ## benchmark of unfaithful minimal pairs (bump) two annotation tasks are designed for bump, where task 1 is taxonomy-based (a specific error type is required for the edited summary), and task 2 allows freestyle edits (i.e., no error type constraints are imposed). in this section, we first describe how data sources are selected to build bump (3.1), and then describe the details of the two annotation tasks (3.2 and 3.3). ## meta-evaluation of faithfulness evaluation metrics in this section, we first describe the faithfulness evaluation metrics benchmarked on bump (4.1). article (partial) reference summary required error type edited summary corrected error type ... the drugs, whose value is estimated at more than @xmath0105 million. officers arrested one venezuelan and two spanish citizens on board the vessel. intrinsic circumstance error the value of the drugs is estimated at more than $250 million . officers arrested one venezuelan and two spanish citizens on board the vessel. intrinsic entity error lightning, floods and a deluge of hailstones descended on st louis tuesday as powerful storms pummeled the mid-united states. roads around the missouri city were flooded in the intense downpour, with one town recording more than two inches of rain in half an hour. ... st louis was hit tuesday by flash floods. a nearby town had more than two inches of rain in less than half an hour. extrinsic predicate error st louis pummeled tuesday by flash floods . a nearby town had more than two inches of rain in less than half an hour. intrinsic predicate error then meta-evaluation protocols are discussed (4.2). ## results we report and analyze the performance of faithfulness metrics in this section using meta-evaluation protocol consistency and roc auc. consistency. the consistency studies of the two tasks foot_2 for all the metrics are reported in roc auc. roc auc scores are presented in we additionally observe that the relative rankings of roc auc scores across error types and task settings are largely consistent with the relative rankings of consistency scores. specifically, we again observe that on a per metric basis: 1) roc auc scores are generally lower for task 2 than task 1 (particularly for entity errors), and 2) metrics generally show worse performance on intrinsic errors than extrinsic ones. for our two meta-evaluation protocols, consistency is suitable for the pairwise ranking of two summaries for a given input article, while roc auc is more adequate in evaluating the absolute capacity of unfaithful summary detection. if a metric has high consistency but low roc auc, it implies that the scores for predicted faithful and unfaithful summaries overlap frequently. such overlap makes it challenging to establish a clear decision boundary for classifications. hence, to improve the classification capability of metrics with high consistency, more calibration is needed to increase the score gap between faithful and unfaithful summaries. ## analysis of bump in this section, we conduct more analysis of bump by studying how bump differs from other benchmarks, followed by a qualitative analysis of the detection difficulty between tasks 1 and 2. comparison with model-generated unfaithful summaries. we compare the generation probabilities of our edited summaries to those of summaries generated from beam search by a bartbased summarizer (trained using the training data of cnn/dailymail) for the same set of documents in our dataset. we report the difference of these generation probabilities normalized by the text length in figure 2 , where we find our edited summaries are much different from model generations in terms of the model generation probabilities. this highlights that existing metrics may not work well on summaries of various styles and experiments are needed to verify their effectiveness in humangenerated unfaithful summaries. furthermore, we compare our roc auc scores with those in existing datasets as shown in true @xcite . in bump, faithful and unfaithful samples under each error type are balanced for both tasks 1 and 2. therefore, for a fair comparison, we pick qags-c @xcite (also a balanced dataset on cnn/dailymail) in true. in qualitative analysis. we provide a qualitative analysis of examples that demonstrate the increased difficulty of task 2. the examples are provided in for the first two examples of task 2 in ## conclusion in this paper, we presented a benchmark of unfaithful minimal pairs (bump) to evaluate faithfulness metrics. unlike prior work where all unfaithful summaries are model generated, each unfaithful summary in bump is generated by minimal human edits to introduce one unfaithful error given a reference summary. through our experiments, we found that bump complements existing benchmarks in a number of ways. first, we found that the summaries in bump are harder to discriminate and less probable under sota summarization models. second, we used bump to measure the consistency of metrics, which cannot be readily measured using other benchmarks. this analysis revealed a discrepancy between the discriminability and consistency of existing metrics, highlighting an important area for future faithfulness metric research to address. finally, we used bump to study faithfulness metrics' performance on individual error types-where our minimal-pair-inspired setup helped control for conclusions being conflated across multiple error types-which revealed that sensitivity to intrinsic errors is another important area for future research to focus on.