Using LLM for Improving Key Event Discovery: Temporal-Guided News Stream Clustering with Event Summaries (2023)

Abstract: Understanding and characterizing the discus- sions around key events in news streams is important for analyzing political discourse. In this work, we study the problem of identification of such key events and the news articles associated with those events from news streams. We propose a generic framework for news stream clustering that analyzes the temporal trend of news articles to automatically extract the underlying key news events that draw significant media attention. We characterize such key events by generating event summaries, based on which we form document clusters in an unsupervised fashion. We evaluate our simple yet effective framework, and show that it produces more coherent event-focused clusters. To demonstrate the utility of our approach, and facilitate future research along the line, we use our framework to construct KeyEvents, a dataset of 40k articles with 611 key events from 11 topics.

## introduction analyzing the dynamics of discussions within the stream of news coverage has been an important tool for researchers to visualize and characterize media discourse around a topic @xcite @xcite @xcite @xcite . news media discourse is typically centered around real-world events that catch media attention and gives rise to news reports streams. with the vast, ever-growing amount of news information available, we need automatic ways for identifying such key events. in this paper, we study the problem of identifying and characterizing key events from a large collection of news articles. since the number of news events is usually not known in advance, past works have typically formulated the problem as a form of non-parametric clustering of news articles, using hierarchical dirichlet processes @xcite or stream clustering @xcite @xcite . rather than relying on the output of such clustering algorithms directly, we view the discovered clusters as event candidates, and leverage recent advances in large language modeling (llm) @xcite to characterize these candidates and reason about their validity. from a bird's eye view, the process is related to past work on interactive clustering @xcite @xcite , but instead of using human feedback to shape the emergent clusters, we rely on llm inference. we propose a framework for clustering an archive of news articles into temporally motivated news events. a high-level overview of our approach is shown in figure 1 . we first retrieve relevant issue-specific articles (details about the document retrieval module are in app a) and perform temporal analysis to identify "peaks", in which the number of articles is significantly higher. we then use hdbscan @xcite a nonparametric clustering algorithm to generate candidate event clusters. we then characterize the candidate clusters by performing few-shot multidocument summarization of the top-k articles assigned to each cluster, identify inconsistent clusters by assessing the (dis)agreement between the summary and each article individually, and redundant clusters by assessing the similarity between cluster pairs' summaries (details in sec. 2.1). these lowquality candidates are removed, resulting in higher quality event clusters. we demonstrate this property over the nela dataset @xcite and show the improvement both in terms of event coherence and document mapping quality. 2 event discovery and article inference ## experiments and results we conduct experiments on the nela-dataset, which is a large collection of news articles (see ap-pendix a). using our document retrieval module, we collect a total of 335k relevant news articles on 11 contemporary issues foot_1 . the application of temporal filters reduces the article count to 90k, which is the basis for our analysis. the retrieved articles are mapped to a four-way {left, right, center, and conspiracy-pseudoscience} political rating. details about the dataset, document retrieval module, and four-way political rating can found in appendix a. evaluation metrics. we evaluate our framework's ability to create coherent event clusters at the desired granularity with three automatic metrics inspired by @xcite . given an event e i and the top-10 relevant entities v e @xmath0..10] to e i by tf-idf, entity purity measures the percentage of the documents that mention at least one of the top-10 entities; coverage counts the percentage of documents accounted for in the cluster assignments. in addition, entity coherence considers cooccurrences of central entity pairs in the clustered documents to measure coherency for an event. baselines. we compare our method's performance against various competitive topic model as baselines. we consider lda @xcite in two different settings -lda, and lda (temporal). the topics are estimated individually at each temporal peak for lda (temporal), whereas the topics are estimated across ## broader impact our method and the resulting keyevents dataset could be useful for analyzing political discourse across different ideologies. as a simple case study, we illustrate how the portrayal of events varies for different political ideologies. we take an entitybased approach (rashkin et al., 2016; field and tsvetkov, 2019; roy et al., 2021) and analyze mentions of joe manchin, a democratic senator and the chair of senate energy committee, in climate change articles. fig. 2 shows that left-leaning articles mention him significantly more than the other two ideologies in some of the events (e.g. the 5 th , 9 th , and 14 th ). analyzing these events' articles show that left leaning articles criticize his ties to the coal industry and opposition to climate change legislation, while fewer (or no) mentions in articles with other ideology leanings under the same events. different ideologies also persist different sentiments when mentioning the same entity. in biden's executive actions on climate change (16 th event in fig. 2 ), articles from different ideologies have comparable mention frequencies of joe manchin. we prompt gpt-3.5 to classify the sentiment expressed towards him (positive, neutral, negative). interestingly, none of the articles from any ideology expresses a positive sentiment; 86% of the articles from the left endure a negative attitude towards him, whereas only 38% and 0% of the articles from the center and the right have negative sentiments. this distinction shows that even the same entities could be portrayed differently within each event to strengthen the beliefs along their political lines. ## conclusion we present a framework for key events identification and showed that events generated from our approach were coherent through quantitative measures, and human evaluation. we also presented a simple qualitative study to showcase the potential of key events, for investigating various political perspectives under nuanced settings.