Re-Examining Summarization Evaluation across Multiple Quality Criteria (2023)

Abstract: The common practice for assessing automatic evaluation metrics is to measure the correlation between their induced system rankings and those obtained by reliable human evaluation, where a higher correlation indicates a better metric. Yet, an intricate setting arises when an NLP task is evaluated by multiple Quality Criteria (QCs), like for text summarization where prominent criteria including relevance, consistency, fluency and coherence. In this paper, we challenge the soundness of this methodology when multiple QCs are involved, concretely for the summarization case. First, we show that the allegedly best metrics for certain QCs actually do not perform well, failing to detect even drastic summary corruptions with respect to the considered QC. To explain this, we show that some of the high correlations obtained in the multi-QC setup are spurious. Finally, we propose a procedure that may help detecting this effect. Overall, our findings highlight the need for further investigating metric evaluation methodologies for the multiple-QC case.

## introduction the broad interest in text generation has triggered significant progress in the development of respective automatic evaluation metrics, complementing costly manual evaluation. in order to assess and rank various automatic metrics, the conventional setup compares the ranking of various systems according to metric scores and human scores, where a high correlation between them indicates an effective metric. in some cases, a task requires evaluation by multiple quality criteria (qcs), as is prominently the case for text summarization, where common criteria include coherence, fluency, faithfulness and saliency. in such cases, comparing metric scores to human scores separately for each criterion should supposedly determine which metrics * work was done as an intern at amazon. ## background and related work a high-quality summary should satisfy several requirements, including preserving the salient information of the source, being faithful to the source, and being coherent. the duc benchmarks @xcite made an early attempt at capturing these requirements with a human evaluation procedure that assessed several readability qualities as well as content responsiveness @xcite . later, additional benchmarks focused on content quality (e.g., @xcite , tac (nist, 2008)) or linguistic quality @xcite separately. @xcite laid out four quality criteria (qcs) on which to assess summary quality, which were reinforced in the summeval summary evaluation benchmark @xcite , annotated over the cnn/dailymail @xcite summarization dataset. described briefly, the four qcs are: relevance, measuring the importance of summary content with respect to the source; consistency, measuring the faithfulness of the summary content to the source; fluency, measuring the linguistic quality of individual sentences in the summary; and coherence, measuring the quality of the collective structure of the sentences in the summary. a summarization system is rated for each qc by averaging human scores (1 to 5 scale) over all input instances. thus, multiple systems can be scored and ranked separately for each qc in accordance with the respective human scores. to assess the effectiveness of an automatic evaluation metric, it is first applied on the outputs of several summarization systems, yielding a respective score per system (averaged over all instances). the metric performance for a certain qc is then measured as the correlation between these metric scores and the corresponding human scores for that qc. the described meta-evaluation procedure enables to rank multiple evaluation metrics per each qc separately, as illustrated in some studies examined other aspects of this meta-evaluation scheme. @xcite and @xcite showed that meta-evaluation using old systems or datasets, like those of duc and tac, yield erroneous trends on modern summarization benchmarks. @xcite investigated the preciseness of correlations between metrics and human annotations in meta-evaluation benchmarks, and proposed approaches to improve the level of confidence. finally, @xcite discussed ways of improving the reliability of system-level correlations in meta-evaluation. ## metric robustness to summary corruptions in this section, we aim to investigate whether a metric that was found to strongly correlate with human scores for a certain qc according to summeval reliably measures this specific qc. specifically, we suggest that the performance of metrics -that were designed to measure relevance -on other qcs (e.g., using rouge to measure fluency) may be questionable. to examine this, we artificially corrupt summeval @xcite system summaries in various forms, each specifically designed to degrade a single qc. the corruption is expected to have a large impact on the corresponding qc measurements, while having minimal effect on other qcs. accordingly, we may conclude that a metric that fails to penalize corrupted summaries does not actually measure the specific qc for which corruption was introduced. in what follows, we experiment with the following corruptions of the system summaries with respect to each qc (except for relevance which is the qc most metrics were designed to capture). fluency: all verbs are replaced with their lemma form, resulting in ungrammatical sentences. coherence: sentences are randomly shuffled to disrupt the structure of the summary. this corruption is inspired by the shuffle test @xcite used to evaluate whether models can detect incoherent text. consistency: all person named entities (using spacy ner, @xcite are replaced with different persons from the source document. this is in fact a common factual mistake of models @xcite . an example for each corruption type can be found in to test the meta-evaluation of metrics presented in summeval, we examine the sensitivity of the best metric for each qc, according to summeval results (from to validate the aforementioned assumption, that our corruptions should actually cause a ranking downgrade, we manually annotated a sample of 9 systems that contain corrupted summaries and found that all system rankings were downgraded after corruption. (more details can be found in appendix b.) since the best automatic metric did not reflect this change in ranking, we conjecture that summeval meta-evaluation scores, even when they appear to be high, are not reliable. in the next section, we investigate the possibility that this qc anchor coherence consistency fluency relevance coherence 1.00 .28 / .24 / .11 .48 / .25 / .06 .69 / .53 / .27 consistency .28 / .24 / .07 1.00 .60 / .36 / .11 .41 / .25 / .07 fluency .48 / .25 / .08 .60 / .36 / .18 1.00 .61 / .21 / .09 relevance .69 / .53 / .21 .41 / .25 / .06 .61 / .21 / .04 1.00