Extractive Summarization via C hat GPT for Faithful Summary Generation (2023)

Abstract: Extractive summarization is a crucial task in natural language processing that aims to condense long documents into shorter versions by directly extracting sentences. The recent introduction of large language models has attracted significant interest in the NLP community due to its remarkable performance on a wide range of downstream tasks. This paper first presents a thorough evaluation of ChatGPT’s performance on extractive summarization and compares it with traditional fine-tuning methods on various benchmark datasets. Our experimental analysis reveals that ChatGPT exhibits inferior extractive summarization performance in terms of ROUGE scores compared to existing supervised systems, while achieving higher performance based on LLM-based evaluation metrics. In addition, we explore the effectiveness of in-context learning and chain-of-thought reasoning for enhancing its performance. Furthermore, we find that applying an extract-then-generate pipeline with ChatGPT yields significant performance improvements over abstractive baselines in terms of summary faithfulness. These observations highlight potential directions for enhancing ChatGPT’s capabilities in faithful summarization using two-stage approaches.

## introduction document summarization aims to compress text material while retaining its most salient information. with the increasing amount of publicly available text data, automatic summarization approaches have become increasingly important. these approaches can be broadly classified into two categories: abstractive and extractive summarization. while abstractive methods @xcite have the advantage of producing flexible and less redundant summaries, they often struggle with generating ungrammatical or even nonfactual contents @xcite . in contrast, extractive summarization directly selects sentences from the source document to form the summary, resulting in summaries that are grammatically correct and faithful to the original text. the growing interest in applying advanced large language models (llm) such as chatgpt foot_0 for text summarization tasks has sparked significant attention. a recent study by @xcite compared gpt-3 with traditional fine-tuning methods and found that, despite lower rouge scores, human annotators preferred the gpt-3 generated text. another study by @xcite conducted a comprehensive analysis of large language models for news summarization and found that the generated summaries were comparable to those produced by humans. however, existing research @xcite has only focused on abstractive summary approaches, and the performance of chatgpt for extractive summarization remains an open question. moreover, the hallucination problem has dramatically hindered the practical use of abstractive summarization systems, highlighting the need to explore extractive summarization with llms for faithful summaries. in this study, we comprehensively evaluate chat-gpt's performance on extractive summarization and investigate the effectiveness of in-context learning and chain-of-thought explanation approaches. our experimental analysis demonstrates that chat-gpt exhibits inferior extractive summarization performance in terms of rouge scores compared to existing supervised systems, while achieving higher performance based on llm-based evaluation metrics. additionally, we observe that using an extract-then-generate pipeline with chatgpt yields large performance improvements over abstractive baselines in terms of summary faithfulness. the main contributions of this paper are: 1) this study represents the first attempt to extend the ap-plication of chatgpt to extractive summarization and evaluate its performance. 2) we investigate the effectiveness of in-context learning and chain-ofthought reasoning approaches for extractive summarization using chatgpt. 3) we further extend the extraction step to abstractive summarization and find that the extract-then-generate framework could improve the generated summary faithfulness by a large margin compared to abstractive-only baselines without hurting summary qualities. ## related work most extractive summarization works formulate the task as a sequence classification problem and use sequential neural models with diverse encoders such as recurrent neural networks @xcite and pre-trained language models @xcite . another group of works formulated extractive summarization as a node classification problem and applied graph neural networks to model inter-sentence dependencies @xcite @xcite . several studies also explored the use of large language models @xcite for summarization. @xcite found that while the former obtained slightly lower rouge scores, human evaluators preferred them. likewise, @xcite reported that large language modelgenerated summaries were on par with humanwritten summaries in the news domain. in addition, @xcite explored the limits of chatgpt on query-based summarization other than generic summarization. luo et al. (2023) explored the use of chatgpt as a factual inconsistency evaluator for abstractive text summarization. zhang et al. (2023c) proposed a self-evaluation and revisement framework with chatgpt. while most of the existing research has focused on abstractive summarization, this work aims to investigate the applicability of chatgpt to extractive summarization and examine whether extractive methods could enhance abstractive summarization faithfulness. ## conclusion this paper presents a thorough evaluation of chat-gpt's performance on extractive summarization across four benchmark datasets. the results indicate chatgpt's strong potential for the task and the possibility of generating factual summaries using the extract-generate framework. overall, this study suggests that chatgpt is a powerful tool for text summarization, and we hope the insights gained from this work can guide future research in this area.