COSMIC : Mutual Information for Task-Agnostic Summarization Evaluation (2024)

Abstract: Assessing the quality of summarizers poses significant challenges—gold summaries are hard to obtain and their suitability depends on the use context of the summarization system. Who is the user of the system, and what do they intend to do with the summary? In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries while preserving task outcomes. We theoretically establish both a lower and upper bound on the expected error rate of these tasks, which depends on the mutual information between source texts and generated summaries. We introduce COSMIC, a practical implementation of this metric, and demonstrate its strong correlation with human judgment-based metrics, as well as its effectiveness in predicting downstream task performance. Comparative analyses against established metrics like BERTScore and ROUGE highlight the competitive performance of COSMIC.

## introduction assessing the quality of summarizers in different settings, tasks, and datasets is critical for better understanding these models and for studying their strengths and weaknesses. in many text generation scenarios, assessing model quality is arduous and resource-intensive, often necessitating human annotations and evaluations. consequently, developing automatic metrics that align closely with human judgments is paramount @xcite @xcite . standard automatic evaluation methods for summarization rely on the idea that a good summary should have some (semantic) overlap with either a gold standard or the source text @xcite . they leverage similarity metrics such as bleu @xcite or rouge @xcite to evaluate the quality of the generated summaries. however, these methods often do not correlate well with human judgements @xcite . to enhance alignment with human judgment and capitalize on recent advancements in large language models, recent efforts have concentrated on learned metrics for scoring summaries @xcite @xcite . for instance, @xcite introduced six new learned metrics by finetuning a pretrained mt5 model @xcite to predict human judgment along different axes. another line of research focuses on reconstruction-based metrics, such as blanc @xcite and information difference @xcite , the former evaluating the actual reconstruction error of the source text with and without a summary, and the latter evaluating the information gain obtained when conditioning the generative model on the summary. standard evaluation methods, therefore, suffer from two major shortcomings. they rarely formally define a clear notion of quality and the methods used to evaluate said notion lack theoretical foundations. this leads to potential discrepancies between the intended evaluation and what is actually measured, leading to a lack of validity of the evaluation methods. we introduce a task-oriented evaluation setup where summaries are meant to allow an agent to perform downstream tasks without reading the longer source text. for example, if a political advisor drafts a briefing on a subject to enable a politician to make informed decisions, our evaluation considers the advisor successful if the decisions made using the summary align with those made using the initial source text @xcite . providing such a well-defined notion of success enables careful analysis of the validity of the evaluation for different possible use cases. in addition, it enables us to perform mechanical evaluations of our method, reducing the noise from the evaluation process. furthermore, many summarization techniques typically operate under the assumption that the downstream task is unknown during the summarization process, yet an implicit notion of summary quality exists. however, this assumption is seldom explicitly addressed or substantiated. in our task-oriented approach, we provide an informationtheoretic rationale for the existence of such a metric. we demonstrate that it essentially involves assessing the mutual information (mi) between the source texts' distribution and the summaries generated by a given summarizer. this paper does not aim to study the syntactic, semantic, and pragmatic aspects of features of summary information, which are essential for capturing the rich notion of information in human communication. for instance, we do not account for the summaries' fluency, grammatical correctness, or coherence. instead, our approach emphasizes how effectively, in the best-case scenario, the output of a summarizer can be used to perform downstream tasks. the results in this paper suggest that the formal definition of mutual information successfully achieves this goal. our approach leverages embeddings to abstract the surface form of text, following recent studies by @xcite and @xcite . we evaluate the quality of our approach in two ways. first, we show that summarizers that induce a summary distribution with higher mi with the source texts' distribution are higher quality in the following sense -they tend to produce summaries that preserve outcomes on downstream tasks as compared to using the source texts. second, we compare the mi to metrics trained on human judgments and show that it displays consistent correlations. our results are consistent with and extend previous work that leverages mi to understand relationships in datasets @xcite , predictive performance of representations @xcite and tool to construct or evaluate representations @xcite , models and generative processes. contributions. our contributions are threefold: ## related work assessing the quality of summaries poses a unique challenge due to the contextual nature of 'quality', influenced by factors such as the audience, topic, and intended purpose of the summary. even expertly crafted human summaries considered "gold standard" in one context may be perceived as subpar in a different setting @xcite . reference-free summary evaluation. referencefree evaluation methods mostly rely on comparing the content of the summary with the content of the source text @xcite and they rely on common overlap metrics such as rouge @xcite , bleu @xcite or bertscore @xcite . however, most reference-free metrics show some important limitations @xcite : they lack theoretical grounding. work such as blanc @xcite and information difference @xcite are closely related to our work. the former evaluates the actual reconstruction error of the source text with and without a summary, while the latter evaluates the information difference evaluated by the generative model. however, both lack the theoretical justification we introduce in section 3 and only evaluate the information based on the generative model. while these methods appear intuitive, they lack of theoretical support for their approaches, relying solely on empirical results and correlations with human evaluations. thus, the results are often tied to a dataset and are affected by variance from the human evaluation. conversely, we offer a theoretical framework applicable to any dataset and measure a tangible, well-defined success criterion: do the tasks yield the same output when performed on summaries as they do on the source texts? embedding-based evaluation. mauve @xcite first proposed a new informationtheoretic metric to compare two text distributions based on embedding clustering; more recent work showed that the crucial element was the clustering step @xcite . they show that while embeddings (and the clusters they form) do not capture fluency or grammatical correction, they do grasp meaning and coherence, making them excellent tools for evaluation. ## experimental settings while the bounds derived in section 4.1 provide certain theoretical guarantees regarding the mutual information, their practical consequences must still be evaluated empirically. we present here two evaluation methods. first, we show that mi effectively predicts whether performing a downstream on the summaries would lead to the same outcome as if it were performed on the source text. this validates our task-oriented evaluation approach for summaries. furthermore, we contrast our metric with metrics trained to emulate human judgments across various dimensions. remarkably, we observe that even without any training, our metric closely aligns with human preferences. datasets. we select three summarization datasets for the english language: cnn/dailymail @xcite , xsum @xcite and multinews @xcite and perform all evaluations on all datasets. we provide additional experiments in french and spanish using the mlsum @xcite and xlsum @xcite datasets. we report mixed results due to the lack of efficient multilingual embedders in section e.2. ## experimental results correlation with downstream tasks performance. in comparing summarizers with cosmic. in figure 3 , we compare the mi of different models and report their size. we observe that ood models -trained on arxiv or medical data -perform very poorly, whereas in distribution models such as bart display significantly higher mi. interestingly enough, the size of the model does not seem to be a good predictor of mi. in appendix f we explore further use of the mutual information to compare the informativeness of different summarizers between themselves to construct a hierarchy based on their mutual informativeness. ## discussion of quantitative results our findings reveal that various conventional metrics do not consistently align with the effectiveness on downstream tasks. as illustrated in figure 4 , different metrics exhibit distinct behaviors and correlations with other metrics and the performance of downstream tasks. ## summary and discussion we introduced a task-oriented setting for summary evaluation in which we can derive a principled and clear notion of the quality of a summarizer: the expected risk of performing a task on a summary instead of the source text. we connected this risk theoretically to the mutual information of the source texts and generated summaries and we bounded the risk on both sides using the mutual information. even if these bounds are task-agnostic and thus potentially loose, we demonstrated experimentally that the mutual information indeed correlated well with the risk. high mutual information indicates that a task performed in the summaries is likely to produce the same outcome as in the source texts. cosmic is therefore theoretically grounded in a reasonable task-oriented scenario. our ability to estimate this mutual information practically and its correlation with downstream task performance further underscores its significance. our proposed method extends beyond summarization systems and could also contribute to the broader field of multi-modal generation evaluation @xcite .