Background Summarization of Event Timelines (2023)

Abstract: Generating concise summaries of news events is a challenging natural language processing task. While journalists often curate timelines to highlight key sub-events, newcomers to a news event face challenges in catching up on its historical context. In this paper, we address this need by introducing the task of background news summarization, which complements each timeline update with a background summary of relevant preceding events. We construct a dataset by merging existing timeline datasets and asking human annotators to write a background summary for each timestep of each news event. We establish strong baseline performance using state-of-the-art summarization systems and propose a query-focused variant to generate background summaries. To evaluate background summary quality, we present a question-answering-based evaluation metric, Background Utility Score (BUS), which measures the percentage of questions about a current event timestep that a background summary answers. Our experiments show the effectiveness of instruction fine-tuned systems such as Flan-T5, in addition to strong zero-shot performance using GPT-3.5.

## introduction events such as natural disasters, political conflicts, and elections are extensively covered by news agencies and followed by readers throughout the world. generating concise summaries of these events is a challenging nlp task @xcite . for popular news stories, journalists curate retrospective timelines that highlight key sub-events on a timeline. however, for a new observer of a specific major event, catching up on the historical context needed to understand the significance of the sub-event update can be a challenging ordeal. to this end, we present the task of background news summarization that complements each update in a timeline with a background summary. timeline summarization is the task of automatically extracting event timelines from a collection of news articles @xcite . update summarization @xcite involves summarizing a set of recent articles, assuming the reader is already familiar with a set of background articles. it has since been studied in shared tasks that track events in newswire @xcite and twitter feeds @xcite . our novel proposed task of background summarization presents an orthogonal use case for the update summarization task. a background summary provides sufficient historical context to the reader to help them understand the latest news update. it summarizes what has happened previously, in order to explain the background of the current news update. background summaries allow the reader to quickly grasp the historical context of an event without having to read through potentially hundreds of news articles or long timelines regarding a specific event. one application would be to contextualize short-text content (e.g., tweets) with background information from news articles about the events. in addition to providing much-needed context, this can be useful for verifying the factuality of the events described in the tweet (e.g., twitter community notes). in a news-centric conversational ai setting, a background summary may be generated to answer a user request to "tell me what i need to know to understand this event". to construct a dataset for background summarization, we identify existing timeline datasets centered around major news events. specifically, we select the widely-used timeline17 @xcite , crisis @xcite , and social timeline @xcite datasets -identifying 14 major news events from these datasets and prepare a single timeline of events for each major event. june @xcite june @xcite dr. conrad murray finds jackson unconscious in the bedroom of his los angeles mansion. paramedics are called to the house while he performs cpr, according to a recording of the 911 call. murray travels with the singer in an ambulance to ucla medical center where jackson later dies. dr. murray is charged with involuntary manslaughter. he pleads not guilty. released on @xmath075,000 bail, he was allowed to resume practicing medicine, but was banned from administering anesthetic agents, "specifically propofol". ... dr. conrad murray is sentenced to four years in county jail. judge michael pastor says the evidence showed him to be guilty of a "continuous pattern of lies and deceit". on june 25th, 2009, dr. conrad murray found pop star michael jackson unconscious at his los angeles mansion. ... later died at ucla medical center. ... whose death by acute propofol intoxication was ruled a homicide. murray pleaded not guilty to the charge of involuntary manslaughter, ... prosecutors alleged that dr. murray acted negligently in giving jackson a lethal dose of propofol, ... the defense denied ... saying the star caused his own death while murray was out of the room. ... dr. conrad murray was found guilty of involuntary manslaughter and was remanded in custody without bail until sentencing. figure 1 : an illustration of the background summarization task. this is a snapshot from the timeline of the michael jackson's death event. the timeline above shows four news updates between june 25, 2009, and november 29, 2011. each update is complemented with a background summary that provides sufficient historical context to the events and entities described in the update. we highlight phrases from the background that provide context to specific phrases in the update text. the original datasets included multiple timelines for major events, collected from different news agencies. for simplifying our annotation process, we merged all the timelines for a given major event to create a single timeline. we present these timelines to expert annotators and collect background summaries at each timestep for each news event. figure 1 provides an example from the timeline of michael jackson's death. the timeline starts on june 25th, 2009 with a news update. the following updates on feb 8, june 25, and nov 29 are substantiated using background summaries that provide historical context regarding the event. on the resulting annotated dataset, we experiment with a suite of summarization systems based on flan-t5 @xcite , longt5 @xcite , and gpt-3.5 @xcite . we propose to formulate the background summarization task as two different variants: (1) in a generic summarization setting, we generate a background summary for the current update at time step t by using a concatenation of the past updates at time steps 1 through t -1. (2) in a query-focused setting @xcite , we utilize the update at time step t as a query. while the back-ground summary does not include content from the current update, the current update can still be used for conditioning the summarization of past updates. this could potentially improve the utility of the backgrounds. for the query-focused summarization, we explore multiple variants of the query and present a comparison of flan-t5 and gpt-3.5. our experiments indicate that the fine-tuned flan-t5 system outperforms gpt-3.5 on the standard rouge metric while underperforming on factuality metrics. this illustrates the challenges associated with automatic metrics for summarization. @xcite and @xcite highlight the difficulties in using automatic metrics for comparing fine-tuned system summaries against zero-shot summaries from high-quality large language models (llms). accordingly, we also present a novel question-answering-based evaluation of background summaries in this work that measures the utility of a given background summary to an update. first, we prompt gpt-3.5 to generate questions about the background of events and entities mentioned in the update. second, we extract answers from the background summaries to measure their effectiveness in providing suffi-cient historical context to the update. our proposed background utility score (bus) measures the percentage of questions about current news updates that are answerable by a background summary. our human evaluation results show the effectiveness of bus. our key contributions are as follows: ## related work events in the real world are often complex, consisting of numerous threads @xcite , and are reported by a large number of news agencies across the world. tracking these events and providing important and useful updates to users has been the focus of many works in natural language processing and information retrieval communities @xcite . we highlight two specific variants of the event summarization task below, timeline summarization: given a corpus of documents and a query, the task is to retrospectively extract important events from the documents and place them along a timeline @xcite . a typical query consists of major events such as the haitian earthquake or the bp oil spill. datasets rely on timelines compiled by news journalists from agencies such as bbc, reuters, and the new york times, amongst others. notable datasets for this task include timeline17 @xcite , crisis @xcite , social timeline @xcite , entities dataset @xcite , and tls-newsroom @xcite . update summarization: dang and owczarzak (2009) first proposed the task of update summarization. given two sets of documents a & b, the task is to generate a query-focused update summary of the document set b assuming the user of the summary has already read the documents from set a. this task has since been studied on documents from newswire @xcite and twitter feeds @xcite . in contrast to the timeline summarization task, systems do not have access to the documents from the future. updating users about critical news events in real-time is very important to news and government agencies @xcite . however, comprehending these updates can be challenging for new readers. our proposed task of background summarization serves a complimentary purpose to updates. background summarization: hayashi et al. (2020) proposed the task of disentangled paper summarization, in which two separate summaries are generated for an academic paper: one summary describing the paper contribution, and another summarizing paper context. a related setting for news events can be a disentangled summarization of updates and backgrounds of events. chen et al. (2022) presented a dataset of tv series transcripts and human-written recaps (summscreen). similar to a background in our task setup, recaps can help viewers understand the current episode. a key distinction is that a recap typically provides information from the most recent episode(s) only, but it does not provide general context to the story. in contrast, background summaries often include information from the very first update to put the current event in context. some prior works have studied the impact of background knowledge in the standard summarization task setup @xcite . a typical summarization setup requires a system to generate a summary of a collection of documents while ignoring any background knowledge already known to the receiver (or reader). peyrard and west (2020) used a broader definition of background. in their setup, the background constitutes a document collection that the user is already familiar with, similar to the document set a from the above definition of update summarization. our definition of background constitutes a summary of previous updates in a given event that are directly relevant to the current, most recent update. the time period provides the overall span of the event timeline. the length of the timeline, the average word count of the (rewritten) updates, and newly annotated backgrounds are specified in the final columns. long-form summarization: our proposed task often contains long timelines of events, requiring systems to perform long-form summarization. there is a growing effort in the community to improve long-range summarization systems. this includes works on book summarization @xcite , meeting summarization @xcite , tv script summarization @xcite and evaluation of long-form summarization systems @xcite . ## background summarization event timelines help readers keep track of updates regarding major news events. they provide a concise overview of the event's progress over time, without the need to read through hundreds or thousands of news articles written about the event. however, for long-lasting events, keeping track of all the sub-event threads can pose a major challenge for the user @xcite . we postulate that complementing each update with a short background summary regarding the event's past can assist the user in understanding the update. our approach is inspired by the standard inverted pyramid structure of news articles @xcite . typically, news articles consist of new newsworthy information at the top, followed by further details about the story, and end with necessary background information. this background information helps the reader gain a full perspective of the news story. in this work, we extend this to news timelines. ## experiments for our background summarization task, we experiment with three summarization systems, flan-t5 @xcite , longt5 @xcite , and gpt-3.5 @xcite . flan-t5: an instruction fine-tuned version of t5 @xcite . we use flan-t5-xl with a maximum source length of 512 tokens. 4 longt5: a sparse attention variant of t5 that utilizes two efficient attentions, local and transientglobal. source length can be significantly longer than the standard 512 token limits of a t5-based system ( §3.2). we use the long-t5-tglobal-xl with a maximum source length of 4096. foot_4 gpt-3.5: a variant of the instructgpt model @xcite optimized for dialogue using reinforcement learning with human feedback. we use this model in a zero-shot setting. we set a maximum source length of 3696. 6 we explore both generic and query-focused summarization settings ( §3.1). in the query-focused setting, we use the current update (u t ) as an additional input to the summarization system. generic: we use a task prefix 'summarize:' to instruct t5-based systems. for gpt-3.5, we use a task suffix, 'provide a short summary of the above article.' query-focused: the input for the t5-based systems follows the template, 'generate a short queryfocused summary of the background. query: ## bus analysis our human evaluation results showed variance amongst turkers (figure 3 ). this is in line with the observations made by prior work on standard summarization datasets @xcite . while human evaluation can be very useful, past work highlighted the difficulties in choosing evaluation dimensions and task design @xcite . @xcite recommends using an evaluation setup based on how users utilize the system in practice. to this end, we analyze the effectiveness of bus ( §3.5) in measuring the real-world utility of background summaries. results: figure 4 provides the best-worst vote counts on the same 1,000 updates from test set using gpt-3. ## conclusion & future work to help readers follow long and complex event timelines, we propose the task of news background summarization. we compliment each update in the timeline with a background summary that provides sufficient context to the readers. we present an expert-annotated dataset for this task with over 1,100 background summaries from three annotators. on this dataset, we benchmark a suite of state-ofthe-art summarization systems (flan-t5, longt5, and gpt-3.5). our results show the zero-shot gpt-3.5 system outperforms the fine-tuned systems on the factuality metrics while underperforming on rouge. given the lack of a metric that accurately captures the utility of a background summary to the news reader, we propose a novel qa-based metric, bus, which measures the percentage of questions about the updates that are answerable from the respective background summaries. for future work, we plan to explore background summarization directly from news articles instead of past updates. sub-events previously considered unimportant but directly consequential to the latest news update can be captured in this setup. we are also interested in benchmarking aspect-based summarization systems for our task.