Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method (2023)

Abstract: Automatic summarization generates concise summaries that contain key ideas of source documents. As the most mainstream datasets for the news sub-domain, CNN/DailyMail and BBC XSum have been widely used for performance benchmarking. However, the reference summaries of those datasets turn out to be noisy, mainly in terms of factual hallucination and information redundancy. To address this challenge, we first annotate new expert-writing Element-aware test sets following the “Lasswell Communication Model” proposed by Lasswell, allowing reference summaries to focus on more fine-grained news elements objectively and comprehensively. Utilizing the new test sets, we observe the surprising zero-shot summary ability of LLMs, which addresses the issue of the inconsistent results between human preference and automatic evaluation metrics of LLMs’ zero-shot summaries in prior work. Further, we propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step, which helps them integrate more fine-grained details of source documents into the final summaries that correlate with the human writing mindset. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two datasets, respectively. Dataset and code are publicly available at https://github.com/Alsace08/SumCoT .

## introduction automatic summarization is a challenging text generation task that condenses the source text into a few coherent and abstract sentences. in recent years, the study of summarization has evolved with supervised learning based on sequence-to-sequence architectures @xcite @xcite and transfer learning based on pre-trained language models (de-figure 1 : case comparisons for our element-aware summary and original dataset-specific summary. news elements have been highlighted with different color shadows. it is clear that our element-aware summary covers more comprehensive elements, and the logical connection between the elements is smoother. @xcite @xcite . existing studies commonly train or fine-tune language models on largescale corpus @xcite @xcite , so superior performance is often reported by measuring the lexical overlap (e.g. rouge @xcite ) with golden summaries @xcite @xcite , which reflects the fit degree to these standard datasets. however, some standard datasets have shown to be noise-enriched, mainly in terms of information redundancy @xcite and factual hallucination @xcite . meanwhile, sufficient experiments have shown that reference summaries in these standard datasets perform poorly on human assessment dimensions, especially coherence, consistency, and relevance @xcite . to fill this gap, this work releases expert-writing element-aware summary test sets. in professional news writing, core elements such as character, time, place, event, etc., are indispensable. this theory named "lasswell communication model" was first ## preliminary comparison: zero-shot llms versus fine-tuned plms in this section, we preliminarily compare existing strong llms and plms upon our elementaware test sets, designed to analyze the general summary capabilities of zero-shot llms and finetuned plms from a more fine-grained perspective. ## towards element-oriented summary: chain-of-thought method we have analyzed the summary writing ability of zero-shot gpt-3 and other fine-tuned plms in §3. we see that gpt-3 performs surprisingly well on our element-aware test sets. the results compellingly show that gpt-3 has great potential for fine-grained zero-shot summary writing. inspired by the prevalence of the chain-of-thought (cot) technique in llms @xcite @xcite , we can further enhance the summarization ability of llms by leveraging a cot-based method (sumcot). sumcot elicits llms to focus on news core elements, thereby generating element-aware summaries step by step. the pipeline and example have been illustrated in figure 2 .