S umm I t: Iterative Text Summarization via C hat GPT (2023)

Abstract: Existing text summarization systems have made significant progress in recent years, but typically generate summaries in a single step. The one-shot summarization setting is sometimes inadequate, however, as the generated summary may contain hallucinations or overlook important details related to the reader’s interests. In this paper, we address this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, closely resembling the iterative process humans undertake when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We evaluate the performance of our framework on three benchmark summarization datasets through empirical and qualitative analyses. We also conduct a human evaluation to validate the effectiveness of the model’s refinements and find a potential issue of over-correction.

## introduction document summarization aims to compress text material while retaining its most salient information. with the increasing amount of publicly available text data, the significance of automated summarization techniques has amplified. recent advancements in summarization systems, leveraging neural networks and pre-trained language models, have demonstrated notable progress @xcite @xcite @xcite . the above summarization systems are all built end-toend and follow a one-shot paradigm that generates summaries in a single step. on the contrary, humans often write text through an evolutionary pro1 we will release our codebase at https://github.com/ hpzhang94/summ_it cess characterized by multiple iterations of drafting and editing @xcite . these end-to-end summarization systems encounter multiple challenges. firstly, they frequently suffer from the issue of hallucination, resulting in the generation of ungrammatical or factually incorrect content @xcite . secondly, these systems are often optimized using imperfect reference summaries, and widely adopted evaluation metrics like rouge may not accurately assess summary quality. thirdly, most of these systems lack controllability, as they only produce a single generic summary conditionally on the input document. in practice, instead of a single condensed version of the entire document, generating summaries that cater to specific aspects or queries would be more beneficial to meet the diverse requirements of users. the emergence of advanced instruction-tuned large language models (llms), such as chatgpt foot_0 , has presented exciting possibilities for summarization systems by exhibiting strong zero-shot performance in various downstream tasks. a recent study by @xcite compared gpt-3 with traditional fine-tuning methods and found that despite lower rouge scores, human annotators preferred the gpt-3 generated summaries. another comprehensive analysis by @xcite focused on large language models for news summarization and revealed that the quality of generated summaries is already on par with those created by humans. furthermore, @xcite demonstrated the utilization of llms like gpt-4 as an effective natural language generation evaluator, showing a higher correlation with humans in the summarization task compared to previous reference-based methods. the advent of llms also introduces new opportunities for summarization beyond the traditional one-shot generation setting. in this paper, we introduce summit, a framework that leverages large language models for iterative text summarization. instead of generating summaries in a single step, our framework enables the model to iteratively refine the generated summary through self-evaluation and feedback, resembling the human process of drafting and revising summaries. according to our experiments, the rationale generation and summary refinement in summit can be guided effectively with in-context learning, eliminating the need for supervised training or reinforcement learning processes. additionally, we explore the potential benefits of incorporating knowledge and topic extractors to enhance summary faithfulness and controllability. we instantiate summit with chatgpt as the backbone, and the automatic evaluation results on three benchmark datasets demonstrate the effectiveness of summit in improving summary quality, faithfulness, and controllability within only a few iterations. furthermore, we conduct a human evaluation to validate the iterative refinements quality and identify a potential over-correction issue. we summarize the contributions of this paper as follows: ## conclusion in this paper, we propose a new framework for text summarization by iteratively refining summaries with model feedback. our framework is fully built upon large language models and doesn't require supervised training or reinforcement learning alignment. we also demonstrate that the system improves faithfulness and controllability by incorporating knowledge and topic extractors. we conduct extensive experiments and analyses on three benchmark datasets and experimental results show that our iterative summarization system outperforms the one-shot generation setting systems with llm, which demonstrates the effectiveness of our method. our human evaluation finds that the summary refinement by our framework can clearly follow the self-evaluation feedback, but is highly biased toward its own evaluation criteria, rather than human judgment. we believe the potential issue could be addressed with human-in-the-loop feedback. we hope the insights gained from this work can guide future research in building more powerful llm-based summarization systems.