Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias (2024)

Abstract: We characterize and study zero-shot abstractive summarization in Large Language Models (LLMs) by measuring position bias, which we propose as a general formulation of the more restrictive lead bias phenomenon studied previously in the literature. Position bias captures the tendency of a model unfairly prioritizing information from certain parts of the input text over others, leading to undesirable behavior. Through numerous experiments on four diverse real-world datasets, we study position bias in multiple LLM models such as GPT 3.5-Turbo, Llama-2, and Dolly-v2, as well as state-of-the-art pretrained encoder-decoder abstractive summarization models such as Pegasus and BART. Our findings lead to novel insights and discussion on performance and position bias of models for zero-shot summarization tasks.

## introduction deep learning based abstractive text summarization models and large language models (llms) have shown remarkable progress in generating concise and coherent summaries from input articles that are comparable to human-written summaries @xcite . building upon this research, we aim to quantitatively measure summarization performance of llms (and pretrained encoder-decoder models for reference) by proposing position bias, which is a novel and general formulation of the lead bias phenomenon @xcite . position bias refers to the tendency of models to prioritize information from certain parts of the source text, potentially overlooking crucial details in other parts of the input article. while position bias has been studied previously in the literature as lead bias, we posit that lead bias is a specific case of position bias. most prior works in this domain aim to propose methods that either incorporate or alleviate lead bias in models for improved performance @xcite without a thorough analysis of the problem itself. it is also important to note that a formal definition for lead bias is still currently lacking in related work. in contrast to lead bias, position bias seeks to decipher if models are over-utilizing sentences from any section(s) of the input articles, instead of just the leading segment. moreover, a model's output summary can only be considered positionally biased if it overwhelmingly uses sentences from section(s) of the input that the human-written (or gold) summaries do not use themselves. for instance, if gold summaries for a dataset are lead biased and the model generates lead biased summaries, this is desirable behavior and cannot constitute position bias. in this scenario, if the model were to generate tail biased summaries, it would be regarded as position bias. an example of a positionally biased summary is shown in figure 1 . we show how position bias can be empirically estimated by generating a distributional mapping between summary sentences and the article sentences used to generate the summary. then, position bias can be measured using a metric such as wasserstein distance @xcite between the model generated summary distribution and the gold summary distribution. position bias measurements augmented with traditional metrics such as rouge scores @xcite can provide a more holistic eval-uation of zero-shot summarization models. in summary, we make the following contributions in this work: ## related works related work has studied the more specific phenomenon of lead bias in summarization. both @xcite and @xcite propose approaches and architectural changes to models that can reduce lead bias in extractive summarization, where summary sentences are selected directly from the source text. in contrast, in our work we study position bias more generally in abstractive summarization. interestingly, @xcite seek to leverage lead biased pre-training to improve performance on news articles, which are known to be lead biased. prior work has also analyzed llms for their performance as zero-shot abstractive summarizers @xcite . goyal et al. (2022) study gpt-3 specifically in the context of news summarization and zhang et al. (2023) benchmark the summarization performance of multiple llms on the cnn/dm and xsum datasets. tam et al. (2023) study the factuality of summaries generated by llms and shen et al. (2023) use gpt 3.5-turbo for evaluating summaries generated by other models. in vision, model bias has been investigated for video summarization @xcite . unlike our work, none of these have studied position 1 https://platform.openai.com/docs/models/gpt-3-5 2 https://huggingface.co/meta-llama/llama-2-13b-chat-hf bias of zero-shot summarization in llms. ## results for experiments, we consider the cnn/dm, xsum, news, and reddit datasets. all datasets are different, in terms of domain, inherent position biases, or article and gold summary length. we choose only instruction-tuned llms as they are more performant at summarization @xcite and we cover different model sizes: gpt 3.5-t is large (175b params), llama-2 is mid-size (13b params), and dolly-v2 is small (7b params). we also consider sota pretrained encoder-decoder models such as pegasus and bart, although these models are not performant unless fine-tuned (i.e. manyshot learning). all experiments are done on the test set of datasets (more details in appendix f). we first visualize the positional distributions generated using our mapping procedure for the gold summaries and model generated summaries in figure 2 with @xmath0. as can be seen, cnn and news contain lead biased gold summaries and reddit and xsum are positionally uniformly distributed. it can also be seen that the llms tend to have low position bias on cnn, news, and reddit datasets. however, for xsum, which constitutes the extreme summarization setting, llms as well as bart/pegasus exhibit much more lead bias, which is absent from the gold summaries. in xsum, the articles are up to 286 sentences long, and summary lengths are required to be between 1-2 sentences long. this is the largest jump from @xmath1. comparing even with reddit where article lengths are up to 23 sentences long and summaries are 1-17 sentences long, it is evident that xsum poses a unique challenge for summarization models. next, in figure 3 , we measure model performance using the r 1 score between gold and generated summaries (results for r 2 , r l are provided in appendix b and follow similar trends) and position bias using wasserstein distance between the positional distributions of gold and generated summaries of figure 2 . as is evident, llms attain the highest rouge scores on all datasets, and tend to have very low position bias, with the exception of xsum. however, even for xsum, llms achieve excellent performance on rouge. bart/pegasus tend to have low performance and bart is also heavily position biased across all datasets. ## discussion insights on model performance and biases. choice of mapping function ϕ. for experiments in the paper we use tf-idf vector similarities as ϕ. in our preliminary experiments, we did not observe significant differences for other choices. we provide these additional results when the r 1 score is used instead and compare with the original results (appendix d). future work can analyze this choice of ϕ as well as effect of other values of k. ## conclusion we analyze zero-shot abstractive summarization by llms via a novel formulation of position bias. position bias measures the tendency of models to generate summaries which overtly and unfairly utilize certain portions of input text over others. through extensive experiments on the cnn/dm, xsum, reddit, news datasets, as well as various models (gpt 3.5-t, llama-2, dolly-v2, pegasus, bart), we obtain novel insights about model performance and position bias that contribute to a deeper understanding of the challenges and opportunities in leveraging llms for effective abstractive summarization.