Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding (2024)

Abstract: The digital landscape is rapidly evolving with an ever-increasing volume of online news, emphasizing the need for swift and precise analysis of complex events.We refer to the complex events composed of many news articles over an extended period as Temporal Complex Event (TCE). This paper proposes a novel approach using Large Language Models (LLMs) to systematically extract and analyze the event chain within TCE, characterized by their key points and timestamps. We establish a benchmark, named TCELongBench, to evaluate the proficiency of LLMs in handling temporal dynamics and understanding extensive text. This benchmark encompasses three distinct tasks - reading comprehension, temporal sequencing, and future event forecasting. In the experiment, we leverage retrieval-augmented generation (RAG) method and LLMs with long context window to deal with lengthy news articles of TCE. Our findings indicate that models with suitable retrievers exhibit comparable performance with those utilizing long context window.

## introduction in today's digital age, the flood of online news highlights the urgent need for quick and precise event analysis. prior work in topic detection has mainly clustered news articles by representation similarity to identify stories from news streams @xcite . extending this approach, our focus shifts to the temporal dynamics of these stories, which we term temporal complex events (tce) @xcite . tces consist of semantically related articles that together narrate the development of various entities over time (refer to figure 1 ). understanding the genesis and evolution of tce, as well as predicting future developments, holds considerable significance for meeting the practical needs of decision-makers, stakeholders, and even the general public interest. existing research in complex event analysis has made significant strides but is constrained by inadequate natural language processing (nlp) techniques. some works @xcite aims at provide concise insights into real-word events, utilizing data mining method or human-curated datasets. another line of works @xcite further tracks the temporal progression of complex events by converting news articles into structured data, such as temporal knowledge graphs (tkgs). the information extraction (ie) methods involved, however, tend to be costly and error-prone. interestingly, how can modern powerful nlp models be applied to complex event analysis, and the extent to which they are aware of its temporal dynamics, remain challenging to determine. in this paper, inspired by the extensive success of llms across various nlp challenges, we delve into their suitability for analyzing tces and assess their prowess in understanding temporal and long contexts. first, llms typically have a limitation in input length, e.g. 4,096 tokens, while a tce may span tens of news articles and then tens of thousands of tokens (i.e., an average of 29 articles and 18,589 tokens in our experimental datasets). even if longer context window enables llms to take in all articles, existing works @xcite have demonstrated their inferior performance with lengthy context. second, llms, pre-trained for next token prediction, sometimes fall short in temporal reasoning tasks @xcite . for tce analysis, this limitation becomes apparent as it necessitates precise event-timestamp correlation and a deep understanding of chronological and causal connections. furthermore, building on top of lengthy past events and their temporal relations, their potential for predicting future events is still under-explored. to this end, we propose a llm-based pipeline for tce outline extraction, and build a large-scale benchmark tcelongbench (tlb) for comprehensive investigation. inspired by @xcite , we aim at providing a coherent and chronological representation of tce, i.e. outline with a timeline. we apply a hierarchical summarization framework and then leverage llm's in-context learning (icl) ability @xcite to extract key points on each day, in the form of sentences. after de-duplication, key points across all timestamps constitute the outline of tce. based on these, we build tcelongbench for temporal, long context evaluation. it contains 88,821 question answering (qa) pairs from 2,289 tces, tailored to three distinct tasks: tlb-detail qa, which tests llms' ability to find evidence across numerous articles; tlb-order qa, focusing on understanding temporal sequences; and tlb-forecast qa, challenging llms to predict future events based on past information. to ensure dataset integrity, we employed a generate-thenverify paradigm, leading to a dataset with an 88% quality rating across human evaluation metrics. in our analysis, we employed both retrievalaugmented generation (rag) methods and llms optimized for long contexts to navigate the extensive narratives typical of tces. our findings reveal that (1) while retrievers are crucial for rag methods, their effectiveness is variable; (2) long-context models excel in managing long temporal sequences but may lead to inferior performance; and (3) models equipped with apt retrievers can match the performance of those designed for long contexts. to sum up, our contributions are threefold: • we leverage llms to extract the outlines and form event chains of tces. a more recent work @xcite formulates a report generation task around complex events using llms, but falls short in large-scale datasets and quantitative analysis on the report quality. however, before delving into long text generation, we aim at evaluating the llm's capability of understanding temporal, long text in tce, and contribute a qa dataset for quantitative comparisons of various baselines. related benchmarks. there are two strands of benchmarks related to tcelongbench. first, temporal reasoning benchmarks @xcite @xcite ) mostly focus on event-time, event-event and/or time-time relations of chronicles in wikipedia. for example, tram @xcite encom-passes ten temporal reasoning tasks, including temporal ordering without any context. forecastqa @xcite are proposed to develop methods for event forecasting with large volumes of unstructured text data. second, long text understanding benchmarks @xcite @xcite aim at evaluating long text modeling with multiple tasks, such as summarization, question answering, code completion, etc. in contrast, tcelongbench evaluates the model's understanding of tces from three tasks, requiring temporal reasoning, long text understanding as well as forecasting abilities. ## task definition existing work has identified tces from news articles by clustering their semantic embeddings concatednated with temporal indexes @xcite . each tce has n timestamps, i.e. a timeline @xmath0: @xmath1, n]}, and news articles a @xmath2: @xmath3, n]}, where a k is the set of news articles on t k . on each timestamp t k , we extract j k number of key points from a k , expressed as p @xmath4,k , . . . , p j k ,k }. each key point is a concise and informative sentence. the collection of key points across all timestamps forms the tce's outline @xmath5: @xmath6, n]}. note that news articles accessible to models are a n-1 = {a k : @xmath7, n -1]} in our experiment as a n is used for generating forecasting questions. tlb-detail. this is a reading comprehension task aiming at testing the model's ability to locate and understand detailed information across numerous articles. the input is a question q, a set of shuffled choices @xmath8: @xmath9, 4]}, and a n-1 , while the output is a choice c @xmath10. tlb-order. this is an ordering task aiming at testing a model's ability to capture the event-event relations across timestamps. the input is a set of shuffled choices @xmath11: @xmath12, r]} and a n-1 , while the output is the chronological order of the choices {c o 1 , . . . , c o r }. tlb-forecast. this is a forecasting task aiming at testing a model's ability to predict future event given historical data. we have two settings of answering forecasting questions, multi-choice and open-domain. in multi-choice setting, the input is a question q, a set of shuffled choices @xmath13: @xmath14; the output is a choice c @xmath15. in open-domain setting, we only have question q and a n-1 as the input, while the output is open for llms. for each question in tlb-detail and tlbforecast, the text span that supports its correct answer lies in the gold article a gold on t gold . while a gold in tlb-detail follows a @xmath16, the a gold in tlb-forecast is within a n , not accessible during evaluation. moreover, articles on t gold except a gold may offer supporting evidence to the correct answer, suggesting that identifying t gold , rather than precisely matching a gold , is also pivotal in determining the correct answer. ## outline extraction inspired by @xcite al. (2023) and rashkin et al. (2020) , we propose a llm-based outline extraction pipeline, which tersely organizes the primary content of tces along with a clear timeline. outline in our work consists of key points from all timestamps, each of which is a concise and informative sentence. these key points represent tces with suitable granularity, recording the detailed actions of entities and unfolding the whole story over the timelines. neither the fine-grained tkg nor event mention (phrase) could capture the intricate relations of multiple entities within tces. our llm-based outline extraction pipeline consists of three parts, summarization, key point generation and key point filtering (figure 2 (1)). initially, we implement a hierarchical summarization framework to filter out extraneous peripheral events, using xgen-7b-8k-inst @xcite . this framework operates as follows: on each timestamp t k , we summarize each news article within a k to distill their essential contents, and then summarize these articles' summaries to obtain the central event on t k . consequently, we compile the daily summaries across all timestamps as @xmath17: @xmath18, n]}. we then leverage llm's icl ability to partition daily summaries into key points. we design a few-shot prompt ( finally, we implement a filtering mechanism to enhance the quality of timeline. we eliminate redundant key points that duplicate previously conveyed information, by calculating two similarity scores using sup-simcse-bert @xcite and quora-distilroberta @xcite . if any of the similarity scores between p i,m on t m and p j,k on t k exceeds predefined thresholds, i.e. 0.8, we discard the key point in later position, i.e. p i,m , since t @xmath20. subsequently, we obtain the tce's outline @xmath21: @xmath22, n]}. ## dataset generation and analysis based on our extracted outlines, we construct qa datasets in tcelongbench, under a generate-thenverify paradigm. we also show the summary statistics and human evaluation results. ## conclusion in this work, we introduced a llm-based framework for outline extraction of tce and established tcelongbench to evaluate llms' capability of temporal understanding and long text comprehension. our approach involved three tasks target-ing reading comprehension, temporal sequencing, and future event forecasting, and conducted experiments across two foundational models: llms leveraging rag method and llms with long context windows. while our experiments provided valuable insights into llms' abilities in tce analysis, future research is essential, particularly in content generation tasks @xcite , to unlock the full potential of llms in complex narrative understanding.