From Sparse to Dense: GPT -4 Summarization with Chain of Density Prompting (2023)

Abstract: Selecting the “right” amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a “Chain of Density” (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace (https://huggingface.co/datasets/griffin/chain_of_density).

## introduction automatic summarization has come a long way in the past few years, largely due to a paradigm shift away from supervised fine-tuning on labeled datasets to zero-shot prompting with large language models (llms), such as @xcite . without additional training, careful prompting can enable fine-grained control over summary characteristics, such as length @xcite , topics @xcite , and style @xcite . an overlooked aspect is the information density of an summary. in theory, as a compression of another text, a summary should be denser-containing a higher concentration of information-than the source document. given the high latency of llm decoding @xcite , covering more information in fewer 1 https://huggingface.co/datasets/ griffin/chain_of_density ## chain of density prompting prompt. our goal is to generate a set of summaries with gpt-4 with varying levels of information density, while controlling for length, which has proven to be a strong confounder when evaluating summaries @xcite . to do this, we formulate a single chain of density (cod) prompt, whereby an initial summary is generated and made increasingly entity dense. specifically, for a fixed number of turns, a set of unique salient entities from the source text are identified and fused into the previous summary without increasing the length. the first summary is entity-sparse as it focuses on only 1-3 initial entities. to maintain the same length while increasing the number of entities covered, abstraction, fusion, and compression is explicitly encouraged, rather than dropping meaningful content from previous summaries. figure 2 displays the prompt along with an example output. rather than be prescriptive about the types of entities, we simply define a missing entity as: ## statistics direct statistics (tokens, entities, entity density) are ones directly controlled for by cod, while indirect statistics are expected byproducts of densification. ## results to better understand the tradeoffs present with cod summaries, we conduct a preference-based human study and a rating-based evaluation with gpt-4. human preferences. we conduct a human evaluation to assess the impact of densification on human assessments of overall quality. specifically, the first four authors of the paper were presented with randomly shuffled cod summaries, along with the articles, for the same 100 articles (5 steps * 100 = 500 total summaries). based on the definition of a "good summary" from @xcite ( based on the average density of step 3 summaries, we can roughly infer a preferred entity density of ∼ 0.15 across the cod candidates. from ## conclusion we study the impact of summary densification on human preferences of overall quality. we find that a degree of densification is preferred, yet, when summaries contain too many entities per token, it is very difficult maintain readability and coherence. we open-source annotated test set as well as a larger un-annotated training set for further research into the topic of fixed-length, variable density summarization.