Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics (2024)

Abstract: Automatic metrics are used as proxies to evaluate abstractive summarization systems when human annotations are too expensive. To be useful, these metrics should be fine-grained, show a high correlation with human annotations, and ideally be independant of reference quality; however, most standard evaluation metrics for summarization are reference-based, and existing reference-free metrics correlates poorly with relevance, especially on summaries of longer documents. In this paper, we introduce a reference-free metric that correlates well with human evaluated relevance, while being very cheap to compute. We show that this metric can also be used along reference-based metrics to improve their robustness in low quality reference settings.

## introduction given an input source, an abstractive summarization system should output a summary that is short, relevant, readable and consistent with the source. to reflect this, fine-grained human evaluations are split into different scores @xcite , such as fluency, faithfulness (sometimes called factual consistency), coherence and relevance. fluency measures the linguistic quality of individual sentences, eg if they contain no grammatical errors. coherence gauges if sentences in a summary are well-organized and well-structured. faithfulness, or factual consistency, considers factual alignment between a summary and the source. relevance is the measure of whether a summary contains the main ideas from the source. automatic summarization metrics are intended to capture one or multiple of these qualities @xcite , and used as a proxy to evaluate summarization systems when human annotations are too expensive. these metrics can be compared on their different attributes such as the reliance on one or multiple references, the cost of inference @xcite , the dataset-agnosticism @xcite and their correlations with human judgment at systemlevel @xcite or summary-level. in this work, we introduce a new reference-free metric foot_0 that intends to capture the relevance of machine summaries using n-gram importance weighting. we rate n-grams of the source documents relative to how much semantic meaning they express, as measured by tf-idf @xcite , and score summaries according to their weighted lexical overlap with these n-grams. we show that this metric is complementary to other metrics and can be mixed with referencebased metrics to alleviate their sensitivity to noisy and low quality references. ## limits of reference-based evaluation lexical overlap scores such as bleu or rouge work under the implicit assumption that reference summaries are mostly extractive and contain no errors. this assumption is challenged by a study conducted by @xcite on hallucinated content in abstractive summaries. in human written summaries from the xsum dataset, 76.9% of the gold references were found to have at least one hallucinated word. summarization methods can trade abstractiveness for faithfulness, creating a faithfulnessabstractiveness tradeoff curve that was illustrated and studied by @xcite . they show that some metrics are more sensitive to the summary abstractiveness than others. in the context of translations, translationese refers to source language artifacts found in both human and machine translations. this phenomenon is similar to extractive segments in summaries, as it is an artifact of the source document that can be mitigated through paraphrasing. @xcite demonstrated that reference translations in machine translation datasets tend to exhibit this translationese language. they addressed this by creating new references through paraphrasing the existing ones. when tested, systems produced much lower bleu scores with the paraphrased references compared to the translationese ones, but the correlation with human judgment was higher. they observed that with translationese references, the n-grams with the highest match rates resulted from translations adhering to the source sentence structure. in contrast, using the paraphrased references, the most-matched n-grams were related to the semantic meaning of the sentence. following a translationese -extractiveness analogy, we assume that with highly extractive references, the most matched n-grams between proposed and reference summaries are artifacts of the extractiveness of the summaries. more abstractive references will yield much lower rouge scores, but might correlate better with human judgement. we propose to use n-gram importance weighting methods, such as tf-idf @xcite or bm-25 @xcite , to extract the n-grams expressing most of the semantic meaning of the source document. we believe that these ngrams should appear in relevant summaries, and are not artifacts of extractiveness. ## proposed metric let w t,d,d be the importance of a n-gram t in a document d from a corpus d, defined as given a proposed summary ŝ of a document @xmath0, we compute the metric: with n d,d the upper ceiling of the sum of weights, used to normalize the score: by design this score will be maximized for a summary consisting of the full document. to alleviate this issue, we penalize longer summaries by multiplying with a term accounting for the length of the summary |ŝ| relative to the length of the document |d|: α ŝ,@xmath1, |d|) 2 . we observe that this length penalty not only resolves the issue related 2 the choice for f is illustrated in appendix a, figure 4 to the scoring of entire documents but also shows a stronger correlation with human judgment at the system level. it is relatively straightforward to devise a trivial heuristic that achieves a high score by employing the same n-gram importance weighting method to generate an extractive summary, with access to the full corpus. we do not consider this point to be a substantial issue, as such heuristic will result in a low score on metrics that measure other aspects of an abstractive summary, such as fluency. ## experiments for our experiments, we work with different datasets of human evaluation of summarization systems. summeval @xcite ) contains human evaluations for 23 systems, each with 100 summaries of news article from the cnn/dailymail dataset. coherence, consistency, fluency and relevance are evaluated by experts and crowd-source workers. arxiv and govreport @xcite contain annotations for 12 summarization systems, evaluated on 18 long documents for each dataset. human evaluators rated the factual consistency and the relevance of the machine summaries. rose @xcite is a benchmark consisting of 12 summarization systems evaluated on 100 news article from cnn/dailymail. each summary is annotated with different protocols, we are using the reference-based and reference-free human evaluations. we describe the choice of settings for our metric in appendix a, which takes into account systemlevel correlations on the four datasets, as well as the range of values taken by the metric. ## conclusion and future works in this work, we introduce a new reference-free metric based on importance-weighted n-gram overlap between the summary and the source. we demonstrated that it has high correlations with human judgement and can be used alongside other metrics to improve them and mitigate their sensitivity to low-quality references. the prospects for future research include further exploration of the behaviour of reference-based, reference-free and hybrid metrics with references of varying quality, as well as potential extensions to multimodal settings such as the evaluation of vision-language systems.