Examining the State-of-the-Art in News Timeline Summarization (2020)

Abstract: Previous work on automatic news timeline summarization (TLS) leaves an unclear picture about how this task can generally be approached and how well it is currently solved. This is mostly due to the focus on individual subtasks, such as date selection and date summarization, and to the previous lack of appropriate evaluation metrics for the full TLS task. In this paper, we compare different TLS strategies using appropriate evaluation frameworks, and propose a simple and effective combination of methods that improves over the stateof-the-art on all tested benchmarks. For a more robust evaluation, we also present a new TLS dataset, which is larger and spans longer time periods than previous datasets.

## introduction timelines of news events can be useful to condense long-ranging news topics and can help us understand how current major events follow from prior events. timeline summarization (tls) aims to automatically create such timelines, i.e., temporally ordered time-stamped textual summaries of events focused on a particular topic. while tls has been studied before, most works treat it as a combination of two individual subtasks, 1) date selection and 2) date summarization, and only focus on one of these at a time @xcite . however, these subtasks are almost never evaluated in combination, which leaves an unclear picture of how well tls is being solved in general. furthermore, previously used evaluation metrics for the date selection and timeline summarization tasks are not appropriate since they do not consider the temporal alignment in the evaluation. just until recently, there were no established experimental settings and appropriate metrics for the full tls task @xcite . date summary 2001-11-29 enron could cost dutch group $195m 2001-11-30 1,100 uk jobs go in enron collapse 2001-12-02 barclays: enron bankruptcy will not affect business 2002-01-15 as enron scandal spreads, us starts to question cash for influence culture [...] 2004-07-08 jury indicts lay for inflating enron earnings 2006-05-25 former enron bosses found guilty 2006-07-05 enron founder lay dies 2008-02-22 us prison beckons british bankers who got cosy with enron in this paper, we examine existing strategies for the full tls task and how well they actually work. we identify three high-level approaches: 1) direct summarization treats tls like text summarization, e.g., by selecting a small subset of sentences from a massive collection of news articles; 2) the date-wise approach first selects salient dates and then summarizes each date; 3) event detection first detects events, e.g., via clustering, selects salient events and summarizes these individually. the current state-of-the-art method is based on direct summarization @xcite . we therefore focus on testing the two remaining strategies, which have not been appropriately evaluated yet and allow for better scalability. we propose a simple method to improve date summarization for the date-wise approach. the method uses temporal expressions (textual references to dates) to derive date vectors, which in turn help to filter candidate sentences to summarize particular dates. with this modification, the date-wise approach obtains improved state-of-the-art results on all tested datasets. we also propose an event-based approach via clustering, which outperforms @xcite on one of three tested datasets. we use purpose-build evaluation metrics for evaluating timelines introduced by @xcite . for a more robust evaluation, we also present a new dataset for tls, which is significantly larger than previous datasets in terms of the number of individual topics and time span. we summarize our contributions as follows: ## related work timeline summarization for news articles has received some attention in the last two decades (swan and @xcite @xcite @xcite @xcite @xcite @xcite @xcite . the task is commonly split into date selection and date summarization subtasks. ## strategies for timeline summarization problem definition we define the tls setup and task as follows. given is a set of news articles a, a set of query keyphrases q, and a ground-truth (reference) timeline r, with l dates that are associated with k sentences on average, i.e., @xmath0. the task is to construct a (system) timeline s that contains m sentences, assigned to an arbitrary number of dates. a simpler and stricter setting can also be used, in which s must contain exactly l dates with k sentences each. ## dataset tran et al. introduced the 17 timelines (t17) @xcite and the crisis @xcite datasets for timeline summarization from news articles. however, we see the need for better benchmarks due to 1) a small number of topics in the t17 and crisis datasets (9 and 4 topics respectively), and 2) relatively short time span, ranging from a few months to 2 years. therefore, we build a new tls dataset, called entities, that contains more topics (47) and longer time-ranges per topic, e.g., decades of news articles. in the following, we describe how we obtain ground-truth timelines and input article collections for this dataset. ground-truth timelines: we obtain groundtruth timelines from cnn fast facts foot_4 , which has a collection of several hundred timelines grouped in categories, e.g., 'people' or 'disasters'. we pick all timelines of the 'people' category and a small number from other categories. queries: for each ground-truth timeline, we define a set of query keyphrases q. by default, we use the original title of the timeline as the keyphrase. for people entities, we use the last token of the title to capture surnames only, which increases the coverage. we manually inspect the resulting sets of keyphrases and correct these if necessary. input articles: for each entity from the groundtruth timelines, we search for news articles using theguardian api foot_5 . we use this source because it provides access to all published articles starting from 1999. we search for articles that have exact matches of the queries in the article body. the timespan for the article search is set so that it extends the ground-truth timeline by 10% of its days before its first and after its last date. adjustments and filtering: the ground-truth timelines are modified to be usable for tls and to ensure they do not contain data not present in the document collection: afterwards, we remove all topics from the dataset that do not fulfill the following criteria: dataset characteristics: tables 2 and 3 give an overview of properties of the two existing datasets and our new dataset, and mostly show averaged values over tasks in a dataset. an individual task corresponds to one ground-truth timeline that a tls algorithm aims to simulate. #p ubdates refers to the number of days in an article collection a on which any articles are published. the compression ratio w.r.t. sentences ("comp. ratio (sents)") is m divided by the total number of sentences in a, and the compression ratio w.r.t dates is l divided by #p ubdates. "avg. date cov" refers to the average coverage of dates in the ground-truth timeline r by the articles in a. this can be counted by using publication dates in a, ("published"), or by textual date references to dates within articles in a ("mentioned"). the fact that there are generally more ground-truth dates covered in textual date references compared to publication dates suggests making use of these date mentions. t17 has longer (l), and more detailed (k) timelines than the other datasets, crisis has more articles per task, and entities has more topics, publication dates and longer time periods per task. ## conclusion in this study, we have compared and proposed different strategies to construct timeline summaries of long-ranging news topics: the previous stateof-the-art method based on direct summarization, a date-wise approach, and a clustering-based approach. by exploiting temporal expressions, we have improved the date-wise approach and yielded new state-of-the-art results on all tested datasets. hence, we showed that an expensive combinatorial search over all sentences in a document collection is not necessary to achieve good results for news tls. for a more robust and diverse evaluation, we have constructed a new tls dataset with a much larger number of topics and with longer time-spans than in previous datasets. most of the generated timelines are still far from oracle timeline extractors and leave large gaps for improvements. potential future directions include a more principled use of our proposed heuristic for detecting content relevant to specific dates, the use of abstractive techniques, a more effective treatment of the redundancy challenge, and extending the new dataset with multiple sources. lu wang, claire cardie, and galen marchetti. 2015. socially-informed timeline generation for complex events. in proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies. william yang wang, yashar mehdad, dragomir r radev, and amanda stent. 2016. a low-rank approximation approach to learning joint embeddings of news stories and images for timeline summarization.. in hlt-naacl. 58-68. zhenhua wang, lidan shou, ke chen, gang chen, and sharad mehrotra. 2014. on summarization and timeline generation for evolutionary tweet streams. ieee transactions on knowledge and data engineering 27, 5 (2014), 1301-1315. rui yan, liang kong, congrui huang, xiaojun wan, xiaoming li, and yan zhang. 2011a. timeline generation through evolutionary trans-temporal summarization. in proceedings of the conference on empirical methods in natural language processing. association for computational linguistics, 433-443. rui yan, xiaojun wan, jahna otterbacher, liang kong, xiaoming li, and yan zhang. 2011b. evolutionary timeline summarization: a balanced optimization framework via iterative substitution. in proceedings of the 34th international acm sigir conference on research and development in information retrieval. acm, 745-754.