C risis LTLS um: A Benchmark for Local Crisis Event Timeline Extraction and Summarization (2022)

Abstract: Social media has increasingly played a key role in emergency response: first responders can use public posts to better react to ongoing crisis events and deploy the necessary resources where they are most needed. Timeline extraction and abstractive summarization are critical technical tasks to leverage large numbers of social media posts about events. Unfortunately, there are few datasets for benchmarking technical approaches for those tasks. This paper presents , the largest dataset of local crisis event timelines available to date. contains 1,000 crisis event timelines across four domains: wildfires, local fires, traffic, and storms. We built using a semi-automated cluster-then-refine approach to collect data from the public Twitter stream. Our initial experiments indicate a significant gap between the performance of strong baselines compared to the human performance on both tasks.Our dataset, code, and models are publicly available (https://github.com/CrisisLTLSum/CrisisTimelines).

## introduction we present crisisltlsum, the first dataset on extraction and summarization of local crisis event timelines from twitter. an example of an annotated timeline in crisisltlsum is shown in figure 1 . a timeline is a chronologically sorted set of posts, where each brings in new information or updates about an ongoing event (such as a fire, storm, or traffic incident). crisisltlsum supports two complex downstream tasks: timeline extraction and timeline summarization. as shown in figure 1 , the timeline extraction task is formalized as: given a seed tweet as the initial mention of a crisis event, extract relevant tweets with updates on the same event from the incoming noisy tweet stream. this task is crucial for real-time event tracking. figure 1 : this is a sample annotated timeline from cri-sisltlsum. the noisy timeline is the set of tweets sorted chronologically. the first tweet is the seed of the event. means that the tweet is annotated to be part of the timeline and indicates that the tweet is excluded. the reason for the exclusion is written under the mark. the timeline summarization task aims to generate abstractive summaries of evolving events by aggregating important details from temporal and incremental information. crisisltlsum can facilitate research in two directions: 1) nlp for social good (crisis domain), and 2) natural language inference and generation, i.e., timeline extraction and summarization tasks. here, we discuss the importance and the differences of crisisltlsum compared to previous work for both of these aspects. towards the first direction, the extraction of real-time crisis-relevant information from microblogs @xcite plays a vital role in providing time-sensitive information to help first responders understand ongoing situations and plan relief efforts accordingly @xcite . cri-sisltlsum goes beyond the task of categorizing each single crisis-relevant post independently @xcite @xcite @xcite and enables a more challenging task for extracting new updates of an ongoing crisis event from incoming posts and summarizing them with respect to the important event details. this can help provide time-sensitive updates while avoiding missing critical information in the bulk of the posts in microblogs due to the high volume of redundant and noisy information @xcite . to the best of our knowledge, this is the first annotated dataset for such an extraction task, while this problem has been tackled before in unsupervised settings @xcite . moreover, we focus on the extraction of local crisis events. the term "local" indicates that an event is bound to an exact location, such as a building, a street, or a county, and usually lasts for a short period. building a corpus of local crisis events is particularly useful for first responders but also challenging because the timelines of these events are often not captured in existing knowledge sources. this means one has to design mechanisms for automatically detecting and tracking events directly from the twitter stream, which is especially hard for existing clustering methods @xcite given the low number of available tweets for each local event. for the second point, crisisltlsum enables nlp research on the complex tasks of timeline extraction and abstractive summarization. these tasks are particularly challenging in the context of social media. first, the process of identifying and extracting relevant updates for a specific event has to contend with the large volume of noise @xcite and informal tone @xcite compared to other domains such as news. additionally, summarizing an on-going event helps toward a quick and better understanding of its progress. this requires a good level of abstraction with important details covered and properly presented (e.g., the temporal order of event evolution). crisisltlsum is the first dataset to provide human-written timeline summaries to support research in this direction. crisisltlsum is developed through a two-step semi-automated process to create 1,000 local crisis timelines from the public twitter stream. to our best knowledge, this is the first timeline dataset focusing on "local" crisis events with the largest number of unique events. the contributions of this paper are as follows: our work in this paper is related to two main directions of crisis domain datasets for nlp and timeline summarization. crisis datasets for nlp: prior research has investigated generating datasets from online social media (e.g., twitter) on large scale crisis events, while providing labels for event categories @xcite , humanitarian types and sub-types @xcite @xcite @xcite , actionable information @xcite , or witness levels @xcite of each crisis related post. while existing datasets on crisis event timelines @xcite @xcite are limited to a small set of large-scale events, crisisltlsum covers a thousand timelines compared to only tens of events covered by each of the existing datasets. additionally, we further go beyond the simple tweet categorization by enabling the extraction of information that include updates over the events' progress. timeline summarization: timeline summarization (tls) was firstly proposed in @xcite , which extracts a single sentence from the news stream of an event topic. in general, the tls task aims to summarize the target's evolution (e.g., a topic or an entity) in a timeline @xcite . existing approaches of tls are mainly based on extractive methods, which are often grouped into several categories. for instance, update summarization @xcite aims to update the previous summary given new information at a later time, while timeline generation @xcite @xcite aims to generate itemized summaries as the timeline, where each item is extracted by finding important temporal points (e.g., spikes, changes or clusters) and selecting representative sentences. another category, temporal summarization, was first proposed in the trec shared task @xcite with follow-up work @xcite , which targets extracting sentences from a large volume of news streams and social media posts as updates for large events. temporal summarization is close to the first task (timeline extraction) proposed in crisisltlsum. there have been a few recent works on abstractive timeline summarization across different domains, e.g., biography @xcite , narratives @xcite , and news headlines @xcite , where the humanwritten summaries are directly collected from the web. the abstractive summarization goal is to generate a set of sentences summarizing the context of interest without taking the exact words or phrases from the original text but rather by combining them and summarizing the important content. to our best knowledge, crisisltlsum is the first to provide human-written summaries for crisis event timelines collected from noisy social media stream. recent research @xcite has also investigated the summarization task based on tweets in other domains which essentially do not reflect the challenges in the summarization of an evolving event. ## crisisltlsum collection this section presents our semi-automated approach to collect crisisltlsum. we first extract clusters of tweets as noisy timelines and then refine them via human annotation to get clean timelines that only include non-redundant, informative, and relevant tweets. ## crisisltlsum statistics & analysis in this section, we cover comprehensive statistics and analysis of crisisltlsum to further elaborate on the statistical characteristics of our dataset. ## experiments here, we first formalize the definitions of the downstream tasks. we then propose a set of existing naive and advanced models to serve as a baseline for these tasks. next, we analyze the performance of the baselines on each task and provide error analysis for the best performing baseline to indicate the remaining major challenges of this dataset for future research. ## conclusion we presented crisisltlsum, the first dataset on local crisis timelines extracted from twitter and the first to provide human-written summaries on information extracted from twitter. we showed that crisisltlsum supports two downstream tasks: timeline extraction and timeline summarization. our experiments with sota baselines indicate that both of these tasks are challenging and encourage future research. our dataset further provides a resource for developing methods on utilizing microblogs toward aiding first-responders in evaluating ongoing crisis events. in future, we plan to explore models that can solve both tasks in a joint setting by extracting new information from each update point and then summarizing those. this dataset can also be expanded by additional annotation to enable abstractive entity-based understanding of the event flow @xcite .