Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024)

Abstract: While large language models (LLMs) can already achieve strong performance on standard generic summarization benchmarks, their performance on more complex summarization task settings is less studied. Therefore, we benchmark LLMs on instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. To this end, we curate an evaluation-only dataset for this task setting and conduct human evaluations of five LLM-based systems to assess their instruction-following capabilities in controllable summarization. We then benchmark LLM-based automatic evaluation for this task with 4 different evaluation protocols and 11 LLMs, resulting in 40 evaluation methods. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) no LLM-based evaluation methods can achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation capabilities. We make our collected benchmark InstruSum publicly available to facilitate future research in this direction.

## introduction recent studies @xcite @xcite have found that large language models (llms), e.g., gpt-3.5 @xcite , can achieve state-of-the-art, even human-level performance on widely used summarization benchmarks such as the cnn/dailymail dataset @xcite . moreover, there are signs that llm performance is saturated on the task of generic summarization, since on these benchmarks (1) llms with varying * equal contribution -1977: ford chooses bridgend for its new engine plant after competition from other european locations. ## human annotation collection we curate a human evaluation benchmark for inscontrollable summarization with two steps: (1) dataset creation and (2) system output evaluation. ## can we automate ins-controllable summarization benchmarking? after evaluating llms as summary evaluators, we explore their potential for automating inscontrollable summarization benchmarking. ## conclusions in this work, we benchmarked large language models for instruction controllable summary generation and evaluation, and presented a new benchmark dataset, instrusum. we found that several llms have already shown promising performance in generating ins-controllable summaries. however, they lack robust holistic capabilities for this task since they still make a considerable amount of errors in their summaries and they can not reliability evaluate different candidate summaries. furthermore, we notice large gaps between the performance of different generations of llms on both ins-controllable summary generation and evaluation. as we believe our proposed ins-controllable summarization setting is more realistic and can provide better usability, we call for future work along this direction to make the text summarization systems more beneficial to the actual users.