Learning to Summarize from LLM -generated Feedback (2025)

Abstract: Developing effective text summarizers remains a challenge due to issues like hallucinations, key information omissions, and verbosity in LLM-generated summaries. This work explores using LLM-generated feedback to improve summary quality by aligning the summaries with human preferences for faithfulness, completeness, and conciseness. We introduce FeedSum, a large-scale dataset containing multi-dimensional LLM feedback on summaries of varying quality across diverse domains. Our experiments show how feedback quality, dimensionality, and granularity influence preference learning, revealing that high-quality, multi-dimensional, fine-grained feedback significantly improves summary generation. We also compare two methods for using this feedback: supervised fine-tuning and direct preference optimization. Finally, we introduce SummLlama3-8b, a model that outperforms the nearly 10x larger Llama3-70b-instruct in generating human-preferred summaries, demonstrating that smaller models can achieve superior performance with appropriate training. The full dataset and SummLlama3-8B model are available at https://huggingface.co/datasets/DISLab/FeedSum and https://huggingface.co/DISLab/SummLlama3-8B.

## introduction developing an effective text summarizer has long been a challenge, as summaries generated by language models often fall short of human standards @xcite @xcite . while large language models (llms) have greatly improved the coherence and fluency of summaries @xcite , persistent issues remain, such as unfaithful statements (hallucinations), omission of key information (low complete-ness), and verbosity (low conciseness) in the summaries @xcite . an ideal approach would involve providing expert-level summary examples for language models to imitate. however, creating such high-quality summaries is both labor-intensive and difficult to scale effectively. a better alternative is to leverage feedback on the summaries through reinforcement learning from human feedback (rlhf) @xcite . rlhf avoids the need to write an ideal summary by having users select their preferred response from candidate summaries of the same document. yet, human involvement still poses scalability challenges, particularly when acquiring high-quality, fine-grained human feedback across multiple dimensions, such as faithfulness and completeness. for example, @xcite reports that the cost of obtaining finegrained human feedback for these three dimensions exceeded $30k for just 2,025 summaries. in this work, we address this challenge by utilizing llm-generated feedback, known as rl from ai feedback (rlaif) @xcite , with a special focus on text summarization. our approach shifts focus to the relatively unexplored area of leveraging llm-generated feedback to enhance summary quality, whereas most existing research in summarization has primarily concentrated on using llms to evaluate summaries. @xcite @xcite . specifically, our goal is to produce humanpreferred summaries by exploiting llm feedback with respect to the three core dimensions, namely faithfulness, ensuring summaries are consistent with original documents; completeness, encompassing all key-facts foot_0 ; and conciseness, maintaining a succinct and focused summary. we focus on these three dimensions, as llms already excel in other [chosen] the effectiveness of llm feedback on preference learning can vary based on three factors, including the quality of the feedback (e.g., low vs. high), its dimensionality (e.g., single vs. multiple), and its level of granularity in scoring (e.g., coarse vs. fine). to explore the impact of these factors, we configure four different setups (see through extensive experiments with feedsum, we provide insights into learning to summarize from llm feedback, focusing on three key open questions: (q1) the impact of these three factors on the effectiveness of preference learning; (q2) an analysis on the effectiveness of each dimension in multi-dimensional feedback; and (q3) a comparison between two approaches for utilizing llm-generated feedback: supervised fine-tuning (sft) and direct preference optimization (dpo). our main contributions are: (1) we create and release feedsum, the first large-scale summarization dataset for preference learning, featuring high diversity in inputs, summaries, and feedback; (2) we examine how different configurations of llmgenerated feedback impact preference learning, and the importance of feedback quality, dimensionality, and granularity. 2 related work preference optimization. preference optimization plays a crucial role in bridging the gap between human intent and the outputs generated by llms @xcite @xcite . the predominant methods are ppo @xcite foot_2 , which fits a reward model to optimize llms to generate responses that receive high reward, and dpo @xcite , which directly optimizes the llms' outputs based on preference data without relying on an explicit reward model. these methods have demonstrated effectiveness in aligning llms with human preferences, particularly in mitigating hallucination, harmful outputs, and biased contents @xcite @xcite @xcite , and llm feedback, synfacedit @xcite . the numbers in the parenthesis are the number of input domains (in "# of domain") and summarizers used for summary generation (in "summary gen"). ## data and experiment details high-level overview. we overview the overall pipeline from data creation to preference learning, as in figure 1 , following three distinct steps: step 1. input sourcing and summary generation: input documents are extracted from 7 diverse sources, varying in domain, length, and type. summaries are then generated using 13 non-llms, open-source llms, and proprietary llms, producing a wide range of summary quality. ## analysis of llm-generated feedback the configurations c1-c4 for llm-based feedback generation show significant differences in feedback quality and their impact on constructing chosen-rejected summary pairs. ## main experiment test set. the test set is constructed by randomly sampling 200 documents from the test split of feed-sum's seven source datasets, totaling 1.4k. both automated and human evaluations are conducted to evaluate summarizers on this set. for automated evaluation, we use llama3-80b-instruct foot_4 as the backbone of finesure @xcite . for human evaluation foot_5 , we perform a fact verification task for faithfulness and a key-fact alignment task for completeness and conciseness, following the work @xcite . three annotators are assigned for each task, recruited through amazon mechanical turk. details of the automated and human evaluation can be found in appendix c.3. ## conclusion this work presents a framework for improving text summarization using llm-generated feedback. we demonstrate that this approach is the most effective when the feedback is high-quality, multidimensional, and assessed at a fine-grained level. our experiments show that dpo significantly outperforms sft variants in utilizing such feedback. additionally, we provide insights into the alignment trade-offs in summarization, the impact of feedback size, and the advantages of our llmgenerated feedback over existing human and synthetic alternatives. as part of our contribution, we open-sourced both the feedsum dataset and the summllama model on hugging face.