Beyond Paraphrasing: Analyzing Summarization Abstractiveness and Reasoning (2025)

Abstract: While there have been many studies analyzing the ability of LLMs to solve problems through reasoning, their application of reasoning in summarization remains largely unexamined. This study explores whether reasoning is essential to summarization by investigating three questions: (1) Do humans frequently use reasoning to generate new summary content? (2) Do summarization models exhibit the same reasoning patterns as humans? (3) Should summarization models integrate more complex reasoning abilities? Our findings reveal that while human summaries often contain reasoning-based information, system-generated summaries rarely contain this same information. This suggests that models struggle to effectively apply reasoning, even when it could improve summary quality. We advocate for the development of models that incorporate deeper reasoning and abstractiveness, and we release our annotated data to support future research.

## introduction in recent decades, the amount of textual information available has grown exponentially, creating a pressing need for automatic systems that can process this information and derive meaningful conclusions from it. recent advances in large language models (llms) have shown remarkable progress in handling tasks that appear to require reasoningnamely, deriving conclusions not explicitly stated in the text. for instance, llms have demonstrated strong performance in question answering tasks that involve background knowledge and inference @xcite . yet, despite these advances, the role of reasoning in generic summarization remains largely underexplored. a key question arises: can and should automatic summaries incorporate new conclusions that go beyond the information explicitly present in the source? traditionally, research in automatic summarization has focused on information selection and paraphrasing @xcite @xcite . one widely used quality measure to human-like summaries has been abstractiveness-the degree to which a summary uses its "own words" rather than copying source text. with the emergence of llms, summarization systems have achieved substantial gains not only in content selection but also in producing highly fluent and abstractive outputs comparable to humanwritten summaries @xcite . these advances invite a deeper investigation into the next frontier: can automatic summaries perform reasoning, deriving conclusions like humans do? in principle, the ability to reason during summarization could enhance content focus and informativeness, enabling the generation of summaries that emphasize the most salient insights rather than merely restating information. to investigate this, we outline several research questions: (1) do humans rely on reasoning to create summary content, and if so, how often? (2) do models employ the same reasoning as humans, or do they bypass it? (3) should we aim to incorporate reasoning abilities in summarization modeling? to address these questions, we began with a manual annotation of human-generated summaries. we identified three common operations that humans perform when rewriting selected text for summaries (will be defined clearly in section 3): paraphrasing, generalization, and drawing conclusions. the latter two operations, generalization and conclusion, change the semantic meaning from the source to the summary and are considered to require reasoning. 1 we manually classified matching text spans between summaries and source texts according to these levels of abstractiveness and found that approximately 25% of human summary spans involve such reasoning. however, our manual evaluation of systemgenerated summaries revealed a different pattern. despite high overall evaluation scores with respect to reference summaries, these systems predominantly matched the reference with paraphrased text spans. crucially, reference spans that require reasoning to extract important information were underrepresented in systems summaries. in other words, while these models seem to display reasoning abilities in other tasks, they tend to avoid using them correctly in summarization, where output is often deemed acceptable without it, despite the fact that this omission can reduce the quality of the summary. this analysis highlights the importance of reasoning in summarization and calls for the development of new models that better integrate reasoning and different levels of abstraction. to facilitate further research, we are releasing the manual annotations and data. 2 2 related work ## abstractiveness levels there are a few ways in which source information can be utilized in the process of summarization. the simplest approach is to copy the information directly and reword or restructure it to fit the rest of the summary. generalizing certain parts of the source material can help compress information even further by reducing the amount of detail and level of specificity. sometimes, it is necessary to add entirely new information drawn from the source through reasonable conclusions to avoid relying on reader inference. we use these observations regarding the various uses of source information to define abstractiveness levels. first, we define a span-level matching between the information in the summary and its corresponding evidence from the source. having these matching pairs allow to analyze the abstractiveness level performed in the summary. following @xcite the spans are standalone facts that are usually formed into a proposition, where the source span entails the summary span. we also required tight matching, where each source token that adds additional information that the summary is not based on, is omitted. given these pairs, the abstractiveness levels are defined as follows: paraphrase. bi-directional entailment between the summary span and the document span. that is, both sides share the same information generalization. the summary and document spans are event-coreferred foot_1 . the summary span does not explicitly mention specific details but instead uses broader terms that encompass those details. as a result, while the source span entails the summary span, the summary span does not fully entail the source span. conclusion. the summary span adds new information that is not mentioned explicitly in the source but derived from it. accordingly, while the source span entails the summary span, the summary does not entail the source in full, and they are not eventcoreferred. the examples in ## annotation process in order to understand how often different levels of abstractiveness appear in human written summaries, we annotate reference summaries from the news and review domains. this annotation was performed manually by an expert annotator. ## system-generated summaries we observed a substantial presence of generalization and conclusion actions in human-written reference summaries, raising questions about whether automatic summarization systems are capable of generating similar types of information. however, due to the lack of available alignment data between system-generated summaries and their corresponding source documents, data required for our annotation process, we were unable to apply the same fine-grained annotation procedure to system summaries as we did for human references. instead, we analyzed how well system-generated summaries align with reference summaries across different abstractiveness levels. our findings reveal that system outputs tend to match paraphrasebased reference spans far more frequently than generation-or conclusion-based spans. in other words, the high similarity scores that system summaries achieve relative to reference summaries primarily stem from their ability to produce effective paraphrases, rather than from generating novel conclusions or inferred content akin to those written by humans. to examine this phenomenon, we conducted an abstractiveness-aware manual evaluation inspired by the pyramid method @xcite . we selected 10 topics from multinews, 10 from fewsum, and 6 from duc 2004. sys building on these matches, we computed a recall score for each abstractiveness level, reflecting the proportion of reference spans of a given type that were successfully reproduced by the system. as shown in these findings suggest that current summarization models excel at reproducing paraphrased content but struggle to incorporate reasoning-based or conclusion-oriented information that match the reference summary, particularly when such reasoning is not explicitly required by the input. it is important to note that, in a few cases, models did select the same source information as the reference summary did, but because they employed a different level of abstraction, their output was too distant from the reference to be considered a match. examples of this phenomenon are provided in ## conclusion in this work, we analyzed different abstraction levels in summarization, and found that while humans use reasoning to derive information which improves the focus and clarity of summaries, models are still lagging behind. we release our data and annotation to facilitate research in this direction and the development of summarization models that incorporate better reasoning abilities.