From Moments to Milestones: Incremental Timeline Summarization Leveraging Large Language Models (2024)

Abstract: Timeline summarization (TLS) is essential for distilling coherent narratives from a vast collection of texts, tracing the progression of events and topics over time. Prior research typically focuses on either event or topic timeline summarization, neglecting the potential synergy of these two forms. In this study, we bridge this gap by introducing a novel approach that leverages large language models (LLMs) for generating both event and topic timelines. Our approach diverges from conventional TLS by prioritizing event detection, leveraging LLMs as pseudo-oracles for incremental event clustering and the construction of timelines from a text stream. As a result, it produces a more interpretable pipeline. Empirical evaluation across four TLS benchmarks reveals that our approach outperforms the best prior published approaches, highlighting the potential of LLMs in timeline summarization for real-world applications.

## introduction condensing a vast collection of texts into comprehensible summaries is a crucial and challenging task. timeline summarization (tls) represents a specialized topic within this area, focusing on distilling event narratives from many texts to summarize the development of specific events or topics over time. research on tls to date is split into two primary categories, each with its objective and granularity. for event timeline summarization @xcite , each timeline is a temporally sorted list of momentary updates about a certain event. the task for event tls is to generate abstractive summaries of events over time from a text stream such as tweets. on the other hand, topic timeline summarization @xcite @xcite aims to systematically arrange timestamped descriptions of milestone events from a corpus of documents, such as compiling a timeline of a public figure's career from historical news articles. as illustrated in figure 1 , event tls equips users with detailed summaries to follow the evolution of particular incidents, while topic tls facilitates rapid understanding of milestone events about a topic. while the task of timeline summarization is essential, it has not garnered as much attention in recent research. prevailing studies have mainly focused on either event or topic-level tls, overlooking the potential synergy between the two. though distinct in their objectives, inputs, and outputs, both forms of tls converge on the fundamental goal of distilling and structuring events from a vast collection of texts. recognizing this shared purpose, our study seeks to bridge this gap by emphasizing event clustering as the common subtask for both event and topic tls. in contrast to the retrospective approach common in prior studies, we approach tls in a streaming context, envisioning that both event and topic timelines should evolve as time progresses and as new texts are received. in this context, the text stream can be envisaged as an evolving event graph, where nodes represent entities (e.g., tweets, event descriptions) and edges indicate common event associations. to align with the dynamic nature of real-world events, we advocate for an incremental approach that is more suited to the continual evolution of timelines. we propose llm-tls, a novel approach that harnesses large language models (llms) for the incremental generation and summarization of timelines. large language models, pre-trained on vast text corpora, have proven effective in knowledgeintensive tasks and are comparable to crowd workers in data annotation tasks @xcite . recent studies have also unveiled the potential of llms in roles traditionally fulfilled by large-scale manual efforts. @xcite . inspired by these advances, we use llms as pseudo-oracles to emulate crowdsourced event clustering for tls. crowdsourced clustering @xcite typically entails grouping n items into k clusters, based on responses to pairwise queries from crowd workers, such as "are items i and j part of the same cluster?" similarly, we propose using llms as pseudooracles to resolve queries such as "do events i and j refer to the same event?" or "does the new tweet i pertain to the same event as the timeline j?" this approach is grounded in the hypothesis that the knowledge encapsulated within llms enables them to discern the relatedness between event descriptions, thus serving as a viable proxy for human judgment in this context. our contributions are summarized as follows: ## problem definition our problem definition for event tls @xcite and topic tls (ghalandari and ifrim, 2020) dovetails with the respective event and topic tls datasets used in prior research. ## method we propose llm-tls, a novel approach that leverages a large language model (specifically llama2-13b @xcite ) to address the challenges in event and topic tls. our method involves a two-step iterative process: cluster and summarize, to construct coherent timelines. event tls focuses on the extraction of event timelines from a noisy text stream, while topic tls focuses on including key timestamps in timelines. however, for a broad topic where many events can occur within a single day, previous date selection methods struggle to select key dates for a timeline. hence, we advocate for a strategy that mirrors human judgment, prioritizing the selection of milestone events. the adaptability of our event clustering component is crucial, as it caters to the distinct requirements of both event and topic tls. we model a text stream as an evolving event graph, where nodes are entities (e.g., tweets, event descriptions), and edges indicate common event as- ## results & analysis 6.1 event llm-tls performance clustering evaluation the evaluation presented in ## discussion why not just use gpt-4? we explored gpt-4's ability to tackle timeline extraction by prompting it to cluster a set of tweets. due to its output length limitation foot_6 , we sampled 10 timelines with noisy tweets per trial. as shown in figure 4 , across five trials, gpt-4's average f1 score was 56.65 with a standard deviation of 16.08, suggesting significant fluctuation in handling noisy streaming data. conversely, llm-tls with llama2-13b performed much better and more consistently, achieving an average f1 score of 87.26 with a much lower standard deviation of 4.95. ## conclusion in summary, we propose a novel approach that utilizes large language models (llms) for the purpose of incremental timeline summarization. our approach addresses both event and topic timeline summarization (tls) in a streaming context, which is more reflective of real-world scenarios. this represents a significant advance in the field, as demonstrated by extensive experiments conducted on multiple datasets. our findings not only reveal the untapped potential of llms in enhancing timeline summarization, but also lay the foundation for further exploration of llms in understanding and organizing temporal information.