Enhancing Incremental Summarization with Structured Representations (2024)

Abstract: Large language models (LLMs) often struggle with processing extensive input contexts, which can lead to redundant, inaccurate, or incoherent summaries. Recent methods have used unstructured memory to incrementally process these contexts, but they still suffer from information overload due to the volume of unstructured data handled. In our study, we introduce structured knowledge representations (GU_json), which significantly improve summarization performance by 40% and 14% across two public datasets. Most notably, we propose the Chain-of-Key strategy (CoK_json) that dynamically updates or augments these representations with new information, rather than recreating the structured memory for each new source. This method further enhances performance by 7% and 4% on the datasets.

## introduction individuals commonly use large language models (llms) to summarize content from sources like webpages, books, and articles @xcite @xcite . this aids in efficiently processing large volumes of information, influencing daily decision-making tasks. despite their potential, llms often struggle with processing extensive contexts, leading to redundancy or inaccuracies @xcite . recent research integrates unstructured memory systems @xcite and fine-tunes models for larger context windows @xcite . however, unstructured memory formats often result in oversized memories that overload the model, impairing its processing and summarization abilities. @xcite proposes a self-controlled memory architecture to manage information via heuristics, yet unstructured memory complicates figure 1 : example summaries generated based on a text memory representation and a structured memory representation, respectively, given the n paragraphs in the original texts. red color marks the values that are not included in plain text summary. to address summarization tasks over extensive input contexts, we introduce cok, the chain-of-key update based on json structured memory representations. on one hand, this approach leverages two key aspects: (1) json's capability to organize information into distinct, easily accessible segments, facilitating efficient updates and retrievals-offering superior expansion and nesting capabilities compared to markdown tables or graphs @xcite ; (2) the prevalence of json in llm pretraining data enhances llms' ability to understand and generate structured json content @xcite foot_0 . consequently, as shown in figure 1 , json-based representations produce more well-structured summaries than plain text, avoiding verbosity and retaining critical content across all sections. on the other hand, instead of requiring llms to recreate complete structured representations for all previously encountered knowledge upon encountering a new information source, our method dynamically identifies which new information needs to be added or updated within the existing structures. this reduces the cognitive load on llms, enabling them to maintain and process complex information more effectively and produce more accurate and comprehensive summaries, as illustrated in figure 1 , where summaries using the cok strategy effectively retain essential knowledge, unlike unstructured summaries that often miss key details (highlighted in red). ## methodology overview. the goal of the summarization task is to generate a summary s t from a sequence of documents @xmath0, d 1 , . . . , d t that encapsulates the essential contents of all documents up to time t. we primarily approach the task within an incremental framework: incremental summarization. this process involves continuously refining the summary by integrating new information from each subsequent document. each document d i may contain new, relevant information that contributes to an ongoing topic. the task requires producing a summary s t at each time step t, encompassing all critical information accumulated, thereby reflecting the key insights from the document sequence. our methodology utilizes a structured representation to efficiently manage and update the summary dynamically with each new document. ## experimental setup dataset. we evaluate our methods using two datasets: sumie @xcite and booookscore @xcite . the sumie dataset is designed to assess the incremental entity summarization capabilities of llms. the booookscore dataset is aimed at long document summarization and includes 100 recently published books, with some books exceeding 100k tokens. due to our models' 6k token context window size, each book is segmented into 2k-token chunks. for additional dataset details, see appendix c. ## conclusion and discussion in this paper, we introduce the chain-of-key method, which uses structured memory representations and leverages llm's step-by-step reasoning to dramatically improve performance on two summarization tasks, surpassing strong baselines. json demonstrates its superiority in organizing knowledge for incremental summarization foot_1 . challenges remain in filtering out trivial details and focusing on crucial information within structured summaries. developing heuristics based on structured properties to better highlight key information is an area for future research.