P re S umm: Predicting Summarization Performance Without Summarizing (2025)

Abstract: Despite recent advancements in automatic summarization, state-of-the-art models do not summarize all documents equally well, raising the question: why? While prior research has extensively analyzed summarization models, little attention has been given to the role of document characteristics in influencing summarization performance.In this work, we explore two key research questions. First, do documents exhibit consistent summarization quality across multiple systems? If so, can we predict a document’s summarization performance without generating a summary? We answer both questions affirmatively and introduce PreSumm, a novel task in which a system predicts summarization performance based solely on the source document. Our analysis sheds light on common properties of documents with low PreSumm scores, revealing that they often suffer from coherence issues, complex content, or a lack of a clear main theme.In addition, we demonstrate PreSumm’s practical utility in two key applications: improving hybrid summarization workflows by identifying documents that require manual summarization and enhancing dataset quality by filtering outliers and noisy documents.Overall, our findings highlight the critical role of document properties in summarization performance and offer insights into the limitations of current systems that could serve as the basis for future improvements.

## introduction recent years have witnessed a remarkable proliferation of summarization models, with many achieving impressive performance on widely used benchmarks. these models, often rooted in large-scale language modeling, represent a significant leap forward in natural language processing capabilities. despite these advancements, not all documents are equally well summarized by these models. some documents yield better summaries, while others do not. as many summarization models share inherent design principles, operational mechanisms, and therefore limitations, we hypothesize that documents that are 'difficult' for one system to summarize tend to also be 'difficult' for other systems. if this hypothesis holds, it suggests that certain intrinsic properties of source documents systematically hinder summarization performance. while prior work has extensively analyzed the characteristics of summarization models @xcite , there is a lack of research on how document properties influence performance across systems. ## task definition our task is to predict the average performance of summarization systems on a given document, using only the document as input. averaging performance across multiple systems can reveal key properties of the document itself, while minimizing the influence of system-specific variability or noise. formally, let d represent a corpus of n text passages, where each document is denoted as d i . presumm aims to estimate the average quality of summaries generated by multiple systems for each document. specifically, consider m systems, denoted as s 1 , . . . , s m , where system s i produces a summary for document d j that is scored as s i,j . the goal of presumm is to first be able to generate a function f : @xmath0: this score, d * j , reflects the average quality of the summaries generated by different systems for document d j . we could leverage this score to rank the documents by their potential performance. to evaluate the model's performance on ranking documents, we measure the correlation between the predicted scores and the gold-standard scores, following standard practices for assessing summarization metrics @xcite . ## experiments we examine the ability of several methods to rank the documents according to the average score across all systems, d * @xmath1,j . in this section we present simple baseline models based on document-based features (section 5.1) and several supervised models trained on our training data (section 5.2). in addition, we compare these methods to conventional summarization automatic metrics that utilize not only the document itself, but also the generated summary or even a reference summary (section 5.3). ## extrinsic evaluation through downstream tasks in this section, we explore practical applications that benefit from predicting in advance the summarization model performance over documents. additionally, these applications serve as extrinsic evaluations of our model. specifically, we examine two use cases: (1) identifying in advance the documents where models perform poorly to enable manual summarization in hybrid systems, and (2) filtering out noisy documents in a multi-document summarization setting. ## analysis in this section, we aim to investigate presummreg to better understand the properties that make a document less likely to be successfully summarized by various summarization systems. to that end, we examine the influence of document-based features over presummreg by measuring correlation (section 7.1). then, we conducted a manual analysis to reveal more insights and explain the automatic results (section 7.2). additional analysis examines how presumm deals with different corruptions can be found in appendix b. ## conclusion in this work, we introduced presumm, a novel approach that opens up new research avenues in understanding the structural features that make a document less likely to be summarized successfully. our findings suggest that documents that are more complex to read for humans are also ranked low by presumm models, implying the centrality of this feature for summarization models. we hope these insights will contribute to the design of more robust and effective models in the future.