Benchmarking Large Language Models for News Summarization (2024)

Abstract: Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.

## introduction large language models (llms) have shown promising results in zero-/few-shot tasks across a wide range of domains @xcite @xcite and have raised significant interest for their potential for automatic summarization @xcite . however, the design decisions contributing to its success in summarization remain poorly understood, and while prior work has shown that llms outperform the prior state of the art, it remains unclear whether their outputs are comparable to human writers. examining these questions is crucial for advancing future research in automatic summarization. to answer the first question, we perform a systematic evaluation of ten diverse llms with human evaluation on news summarization; our evaluation identifies instruction tuning to be the key to zero-shot summarization capability. in contrast, self-supervised learning alone cannot induce strong summarization performance in the zero-shot setting (figure 1 ). in fact, even a 350m parameter instruction-tuned gpt-3 can perform on par with the 175b parameter gpt-3. to benchmark llms, we evaluated the standard cnn/dm @xcite and xsum datasets @xcite but found that existing reference summaries caused several issues. the reference summaries in these benchmarks were originally created in a different use context and, when evaluated as part of a generic news summarization benchmark, human annotators judge them to be worse than the outputs of most automatic systems (figure 1 ). when computing automatic metrics using these references, their poor quality reduces the correlation between metric results and human judgment. not only does this make evaluation difficult, but it also degrades the performance of systems that take supervision either through finetuning or few-shot prompting and makes comparison difficult. to address the quality issues of reference summaries and better understand how llms compare to human summary writers, we recruit freelance writers from upwork 1 to re-annotate 100 articles from the test set of cnn/dm and xsum. comparing the best performing llm, instruct davinci, to the freelance writers, we find that the instruct davinci summaries are much more extractive. by manually annotating the summarization operations @xcite used in these summaries, we find that instruct davinci paraphrases much less frequently although it is able to combine copied segments coherently. given their stylistic differences, we recruit annotators to compare the instruct davinci summaries to those written by freelance writers. on aggregate, we find that instruct davinci is rated as comparable to the freelance writers. examination of the annotations from each individual rater shows that every rater has their own consistent preference for either instruct davinci or the freelance writers. together, our work makes the following key contributions. first, we identify instruction tuning, instead of model scale, as the key to llms' summarization capability. second, we show that reference summaries used in xsum, which are simply the first sentence of the news article, are judged by humans to be worse than the best llmgenerated summaries. third, to address these issues with references, we collect better quality summaries from freelance writers and we show that the best llm is rated as comparable to upwork freelance writers. in combination, these results call into question recent claims made about llm summarization. in particular, summarization progress cannot be measured using reference-based metrics applied on xsum. furthermore, the question of whether fine-tuned, few-shot, or zero-shot models perform better remains an open question due to the poor quality of training data. to encourage future work on improved evaluations, we release the high-quality summaries written by freelance writers and the evaluation data on 18 model settings and two datasets as resources. 2 2 background and related work ## human evaluation on news summarization benchmarks in this section, we use human evaluation to systematically benchmark a diverse set of ten llms on news summarization. we observe that instruction tuning is the key to strong summarization capability and reference summaries in current benchmarks may underestimate few-shot or finetuning performance. ## comparing to freelance writers in section 3, we see that the low-quality reference summaries make studying and benchmarking llms difficult. in this section, we address this by recruiting upwork freelance writers to collect higher-quality summaries. with this data, we aim to answer two important questions. first, we would like to know whether the best llm has reached human-level performance and how the summaries written by the best llm differ from the ones written by humans. second, we aim to examine the correlation between reference-based metrics and human judgments when the metrics are calculated using our higher-quality reference summaries. ## discussion implication for model development. in this study, we systematically evaluate a diverse set of llms and find that instruction tuning contributes the most to llms' summarization capability. we believe that there is much research beyond our benchmarking effort that needs to be done to better understand the effect of instruction tuning. here we hypothesize three aspects that could account for the success of instruction tuning. first, the quality of the summarization data used in instruction tuning can serve an important role. our findings in section 3 show that currently, we are finetuning language models on lowquality training data, which can account for their ineffectiveness. at this point, we cannot rule out the possibility that when finetuned on higher quality data, finetuned lms may perform much better. second, the learning algorithm used for instruction tuning can be important @xcite . while the exact training details are unknown, the success of instruct davinci might be credited to ''learning from human feedback'' (lhf; @xcite . contrary to supervised finetuning that trains systems on written summaries, learning from human feedback trains systems from binary labels of human preferences. as we observe in section 4.2, there is a discrepancy in how annotators write and rate summaries. while it is possible that lhf has merits over the supervised learning/finetuning approach in exploiting this discrepancy, more analysis is needed to validate this hypothesis. third, multi-task learning can be important. instruct davinci is trained on a diverse distribution of inputs and many previous studies have confirmed the effectiveness of multi-task learning. we look forward to understanding how summarization benefits from learning on other tasks. ## conclusion in this work, we conducted a comprehensive human evaluation of ten llms, across the two most popular news summarization benchmarks. through our experiments, we find that the stateof-the-art llm performs on par with summaries written by freelance writers, with instruction tuning being the key factor for success. beyond these findings, our work highlights the crucial role of good reference summaries in both summarization model development and evaluation. unless the reference quality issue is addressed, comparing zero-shot, few-shot, and finetuning performance will remain an open question, and the current benchmarks will provide limited value when used with reference-based evaluation. even when we address the quality issue and conduct a human evaluation with high-quality references, we observe a significant amount of individual variation from our annotator pool. due to these factors, evaluations for single document news summarization may be reaching their limits.