A lign S um: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preference (2024)

Abstract: Text summarization tasks commonly employ Pre-trained Language Models (PLMs) to fit diverse standard datasets. While these PLMs excel in automatic evaluations, they frequently underperform in human evaluations, indicating a deviation between their generated summaries and human summarization preferences. This discrepancy is likely due to the low quality of fine-tuning datasets and the limited availability of high-quality human-annotated data that reflect true human preference. To address this challenge, we introduce a novel human summarization preference alignment framework AlignSum. This framework consists of three parts: Firstly, we construct a Data Pymarid with extractive, abstractive, and human-annotated summary data. Secondly, we conduct the Gaussian Resampling to remove summaries with extreme lengths. Finally, we implement the two-stage hierarchical fine-tuning with Data Pymarid after Gaussian Resampling. We apply AlignSum to PLMs on the human-annotated CNN/DailyMail and BBC XSum datasets. Experiments show that with AlignSum, PLMs like BART-Large surpass 175B GPT-3 in both automatic and human evaluations. This demonstrates that AlignSum significantly enhances the alignment of language models with human summarization preferences.

## introduction text summarization is a pivotal component of natural language processing, striving to produce coherent and concise summaries of textual documents @xcite @xcite . it can be categorized into two main styles: extractive summarization @xcite @xcite involves selecting significant 528*( 528*(/ *(9@xmath057/dujh %5,2 *37z&r7 ## alignsum: summarization preference alignment framework we first formalize the summarization task: given a document @xmath1...d n with length n, the goal is to generate a summary @xmath2...s m with length m, and usually m ≪ n. our proposed preference alignment framework consists of three parts: data pyramid (dp), gaussian resampling, and twostage hierarchical fine-tuning (hft). figure 2 shows the overall framework: firstly, we construct the data pyramid using various methods such as extraction, llm generation, and human annotation. secondly, as the source data have different summary lengths, plms with this data would lead to inconsistent summary lengths. a two-stage hierarchical fine-tuning strategy: initially training the plms on extractive and abstractive data to fit the general domain, followed by fine-tuning the justly fine-tuned plms on humanannotated data to align with human preference. details will be introduced in the following parts. ## conclusion we propose a novel human summarization preference alignment framework alignsum including data pymarid, gaussian resampling, and twostage hierarchical fine-tuning to align plms with human preference. experiments demonstrate the effectiveness of our framework and narrow the gaps between automatic and human evaluation of plms.