On Context Utilization in Summarization with Large Language Models (2024)

Abstract: Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries. Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens. However, in question answering, language models exhibit uneven utilization of their input context. They tend to favor the initial and final segments, resulting in a U-shaped performance pattern concerning where the answer is located within the input. This bias raises concerns, particularly in summarization where crucial content may be dispersed throughout the source document(s). Besides, in summarization, mapping facts from the source to the summary is not trivial as salient content is usually re-phrased. In this paper, we conduct the first comprehensive study on context utilization and position bias in summarization. Our analysis encompasses 6 LLMs, 10 datasets, and 5 evaluation metrics. We introduce a new evaluation benchmark called MiddleSum on the which we benchmark two alternative inference methods to alleviate position bias: hierarchical summarization and incremental summarization. Our code and data can be found here: https://github.com/ntunlp/MiddleSum.

## introduction large language models (llms) have drastically transformed the landscape of nlp recently @xcite . with instruction tuning @xcite , llms made a major leap forward in conditional (prompted) content generation, and can generate satisfying outputs without the need to fine-tune on a specific task. in abstractive summarization specifically, this approach has arguably opened a new paradigm: summaries generated by llms are highly fluent, grammatical and relevant @xcite . despite noticeably lower scores on automatic metrics such as rouge @xcite or bertscore @xcite , summaries generated by llms are largely preferred by humans over summaries from state-of-the-art fine-tuned models like brio @xcite . in fact, on xsum, gpt-3.5 summaries are even on par with re-annotated human-written summaries, and much better than the dataset's original ground-truth, according to human evaluators @xcite . llms also show promising capability in evaluating summaries generated by other systems, including llms @xcite @xcite . despite this success, a few major technological bottlenecks remain with llms, including the maximum length of their context window. the standard context window length for open-source llms is 2k tokens @xcite @xcite , which drastically limits their usefulness for long-input summarization @xcite . several techniques were proposed to extend the context window, including alibi @xcite , lex @xcite , position interpolation @xcite and yarn @xcite . while some of them claim up to 100k+ tokens processing capacity @xcite , it remains unclear how much such methods help on long-context summarization. scaling up context length would only succeed if a key question gets addressed first: do llms make proper use of their entire context? recent work @xcite suggested that, surprisingly, such a simple assumption may not hold: through experiments on multi-document question answering and key-value retrieval, the authors find that llms mostly focus on the beginning and end of the (long) context window. plotting performance with regards to the position of the important information exhibits a u-shape, with performance high at first (beginning of the source), then dropping, and rising again at the end. worryingly, in the middle of the context window, llms' performance can drop to even below random chance, calling for greater examination of llms' behaviors with regard to the position of information within the source. in this work, we investigate in depth how llms use their context window in abstractive summarization. unlike in question-answering, mapping facts in the output to a specific snippet in the source is not straightforward in abstractive summarization, due to the high-level of re-phrasing and compression. we conduct a large-scale study with 6 llms, 10 datasets covering many aspects of summarization, and 5 highly diverse automatic metrics. our contributions are threefold: ## experimental setup datasets we cover a broad set of diverse abstractive summarization tasks, varying length and domain. we include 5 datasets of standard length (source is below 2k tokens, which always fits in the context window): (i) cnn/dailymail @xcite , (ii) xsum @xcite , (iii) reddit-tifu @xcite , (iv) samsum @xcite , and (v) multi-xscience @xcite . we also include another 5 long-input summarization datasets: (i) arxiv and (ii) pubmed @xcite , (iii) govreport @xcite , (iv) summscreenfd @xcite , and (v) multi-news @xcite . a highlevel view of each dataset is shown in ## experiments in this section, we describe a series of experiments aimed at understanding how llms treat information in their input depending on the position. ## conclusion behind the recent hype around llms and their amazing instruction following and content generation capabilities, our study showcases a major weakness in abstractive summarization: llms suffer from the middle curse and struggle to use information in the middle of their context window. llms do not make a consistent use of their context window as they mostly look at the beginning and (to a lesser extent) the end, which at first glance may be hidden by the prevalent lead bias in summarization datasets. extending context window beyond 4k tokens, which has been an intense area of focus lately, is not justified in the current inference and evaluation setup in abstractive summarization. we benchmarked two alternative inference methods (hierarchical inference and incremental inference) on middlesum, an evaluation subset designed to showcase the middle curse. despite encouraging results on scientific paper datasets, these methods are far from a silver bullet to the middle curse. we call for a better evaluation of llms, which accounts for the salient spans of the source which are effectively being processed.