Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLM s (2024)

Abstract: Extractive summarization plays a pivotal role in natural language processing due to its wide-range applications in summarizing diverse content efficiently, while also being faithful to the original content. Despite significant advancement achieved in extractive summarization by Large Language Models (LLMs), these summaries frequently exhibit incoherence. An important aspect of the coherent summary is its readability for intended users. Although there have been many datasets and benchmarks proposed for creating coherent extractive summaries, none of them currently incorporate user intent to improve coherence in extractive summarization. Motivated by this, we propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback, offering valuable insights into how to improve coherence in extractive summaries. We utilize this dataset for aligning LLMs through supervised fine-tuning with natural language human feedback to enhance the coherence of their generated summaries. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (~10% Rouge-L) in terms of producing coherent summaries. We further utilize human feedback to benchmark results over instruction-tuned models such as FLAN-T5 which resulted in several interesting findings.

## introduction with the increasing amount of information, the significance of automatic summarization has grown exponentially. summarization techniques can be broadly classified into two categories: (i) extractive, and (ii) abstractive. the abstractive methods @xcite often focus on the semantic meaning of the text, giving a summary by creating a new set of sentences. however, these methods often struggle with generating ungrammatical or even nonfactual contents @xcite . in contrast, extractive methods focus on selecting meaningful phrases/sentences from the given text, giving a summary that is faithful to the original content, hence it has a range of real-world applications @xcite . for instance, tasks such as video shortening, and legal document summarization require precision and adherence to specific details from original text, and extractive methods are more suitable for these tasks. however extractive summarization often generates summaries that lack coherence, and coherence is a crucial attribute of text summarization since it holds a significant connection to user experience. thus, our work aims to improve coherence in extractive summarization. with the advent of llms such as gpt-4, llama-2 @xcite , and falcon @xcite , there is a significant advancement in gen-erating extractive summaries @xcite . for extractive summarization, coherence is often measured through the interconnection among sentences and ease of readability for users. past attempts have been made to improve and quantify coherence in extractive summarization @xcite @xcite 2 , however, these attempts do not consider user-specific intent (i.e., ease of readability while preserving important information). thus, we approach the concept of coherence through the lens of user-specific intent (figure 1 ). to this end, we propose a comprehensive dataset with a systematic collection of natural language feedback to improve coherence in model-generated summaries, and human-annotated extractive coherent summaries. to the best of the authors' knowledge, this dataset represents the initial effort to align the coherence in a summary with user intent. to develop the proposed dataset, we hire expert annotators to accurately annotate data for our task. for the annotation, the objective is two-fold: (1) to create a coherent summary by extracting important sentences from a source document that effectively captures the key aspects of the document, and (2) to provide feedback (i.e, natural language explanations) on the steps to go from the model summary to the gold coherent summary. we annotate this data across five categories: news, debate, tv show, meeting, and dialogue. our annotation process consists of three phases (detailed discussion in §2). each data instance collected in our dataset consists @xmath0, initial model summary, feedback, gold coherent summary, @xmath1. we utilize the proposed dataset for aligning widely used open-source llms to generate more coherent extractive summaries via supervised fine-tuning: (i) two decoder-only models, i.e., falcon-40b and llama-2-13b, and (ii) three en-coder+decoder models, i.e., flan-t5, tk-instruct, and t5. we develop a baseline and propose two different supervised fine-tuning strategies with human feedback (details are presented in §3). we measure the performance in terms of rouge-l. rouge-l assesses the syntactic and semantic similarity between the generated and the gold coherent summary, indicating their proximity. we also provide human judgments in terms of the coherence of generated summaries by baseline and proposed approach. experimental results reveal that 2 detailed related work is presented in app. b the proposed models show absolute improvement of ∼ 10% rouge-l over baselines. furthermore, human evaluation shows a preference for extractive summaries from our approach, often rating them as more coherent. this indicates that aligning the model with user feedback improves coherence. furthermore, a thorough analysis of the results reveals several interesting findings. we hope that our findings facilitate future research for improving coherence in extractive summarization. ## data collection our annotation process consists of three phases. first, we randomly select a source text for annotation across five different categories from publicly available datasets. second, we prompt a large language model to create coherent summaries for selected source text. finally, we hire expert annotators to review generated summaries and provide natural language feedback/explanations to improve coherence in generated summaries. ## conclusions this paper introduced a comprehensive dataset designed to improve coherence in extractive summarization while integrating natural language feedback from human users across five different categories. utilizing this dataset, we conducted evaluations using various llms, and initial experimental outcomes demonstrate an enhancement in model performance, with ∼ 10% improvement in coherence achieved through fine-tuning with human feedback. moreover, our analysis highlights the potential for performance advancements in instructiontuned models through pre-finetuning based on user feedback. we believe that both the dataset and the findings derived from this work will serve as valuable tools for future research in this direction.