N ews E dits: A News Article Revision Dataset and a Novel Document-Level Reasoning Challenge (2022)

Abstract: News article revision histories provide clues to narrative and factual evolution in news articles. To facilitate analysis of this evolution, we present the first publicly available dataset of news revision histories, NewsEdits. Our dataset is large-scale and multilingual; it contains 1.2 million articles with 4.6 million versions from over 22 English- and French-language newspaper sources based in three countries, spanning 15 years of coverage (2006-2021).We define article-level edit actions: Addition, Deletion, Edit and Refactor, and develop a high-accuracy extraction algorithm to identify these actions. To underscore the factual nature of many edit actions, we conduct analyses showing that added and deleted sentences are more likely to contain updating events, main content and quotes than unchanged sentences. Finally, to explore whether edit actions are predictable, we introduce three novel tasks aimed at predicting actions performed during version updates. We show that these tasks are possible for expert humans but are challenging for large NLP models. We hope this can spur research in narrative framing and help provide predictive tools for journalists chasing breaking news.

## introduction revision histories gathered from various natural language domains like wikipedia @xcite , wikihow @xcite and student learner essays @xcite have primarily been studied to explore stylistic changes, such as grammatical error correction @xcite and argumentation design @xcite . however, deeper questions about content updates and narrative evolution are underexplored: which facts are uncertain and likely to be changed? which events are likely to update? what voices and perspectives are needed to complete a narrative? existing edits corpora do not address these questions due to the nature of previously studied domains: as shown in @xcite , the distribution of edits in other domains, like wikipedia, tend to focus on syntax or style edits. in this work, we introduce a novel domain for revision histories, news article revision histories which, we show, covers the updating of events. many edits in news either (1) incorporate new information (2) update events or (3) broaden perspectives (section 3). our dataset, newsedits, contains 1.2 million articles with 4.6 million versions. we develop a document-level view for studying revisions and define four edit actions to characterize changes between versions: sentence addition, deletion, edit and refactor (i.e. the sentence is moved within a document). we introduce algorithms for identifying these actions. we count over 40 million edits, additions, deletions or refactors in newsedits. we argue that news is an important, practical medium to study questions about narrative, factual and stylistic development. this is because, we hypothesize, there are consistent patterns in the way articles update in the breaking news cycle @xcite . to prove this hypothesis, we show that updates are predictable. we design three tasks: (1) "predict whether an article will be updated," (2) "predict how much of an article will updated," (3) "predict sentence-level edit actions." we show that current large language model (llm)-based predictors provide a strong baseline above random guessing in most tasks, though expert human journalists perform significantly better. our insights are twofold: (a) article updates are predictable and follow common patterns which humans are able to discern (b) significant modeling progress is needed to address the questions outlined above. see section 4.6 for more details. finally, we show that the newsedits dataset can bring value to a number of specific, ongoing research directions: event-temporal relation extraction @xcite , article link prediction @xcite , factguided updates @xcite , misinformation detection @xcite , headline generation @xcite and author attribution @xcite , as well as numerous directions in computational journalism @xcite and communications fields @xcite . our contributions are the following: ## exploratory analysis we extract all edit actions in our dataset using methods described in the previous section. statistics on the total number of operations are shown in in figure 3a , we show that edit-actions in an article's early versions are primarily adding or updating information: new articles tend to have roughly 20% of their sentences edited, 10% added and few deleted. this fits a pattern of breaking news lifecy- one way to measure this is to explore whether edit-actions update the events in a story @xcite . we focus on pairs of edited sentences. we randomly sample edits from documents in our corpus (@xmath0,329 pairs) and extract events using ma et al. (2021)'s model. we find that edited sentence pairs are more likely to contain events (43.5%) than unchanged sentences (31.4%). further, we find that 37.1% of edited sentences with events contain different across versions. we give a sample of pairs in taken together, we have shown in this analysis that factual updates drive many of the edit operations that we have constructed to describe newsedits revision histories. next, we will measure how predictable these update patterns are. ## predictive analysis on newsedits as shown in section 3, many edit-actions show breaking news patterns, which usher (2018) observed follow common update patterns. now, we explore how predictable these operations are, to address whether future work on the fundamental research questions addressed in section 1 around narrative design is feasible. in this section, we outline three tasks that involve predicting the future states of articles based on the current state. these tasks, we hypothesize, outline several modeling challenges: (1) identify indicators of uncertainty used in news writing 10 (ekström et al., 2021), (2) identify informational incompleteness, like source representation @xcite and (3) identify prototypical event patterns @xcite . these are all strategies that ex-pert human evaluators used when performing our tasks (section 4.6). the tasks range from easier to harder, based on the sparsity of the data available for each task and the dimensionality of the prediction. we show that they are predictable but present a challenge for current language modeling approaches: expert humans perform these tasks much more accurately than llm-based baselines. in addition to serving a model-probing and dataexplanatory purpose, these tasks are also practical: journalists told us in interviews that being able to perform these predictive tasks could help newsrooms allocate reporting resources in a breaking news scenario. foot_9 ## related work a significant contribution of this work, we feel, is the introduction of a large corpus of news edits into revision-history research and the framing of questions around sentence-level edit-actions. despite the centrality of news writing in nlp @xcite @xcite , we know of no academic corpus of news revision histories. two works that analyze news edits to predict article quality @xcite do not release their datasets. 21 wikinews foot_19 articles and editorannotations have been used for document summarization (bravo-marquez and manriquez, 2012), timeline synthesis @xcite , word-identification @xcite and entity salience @xcite . however, we are not aware of any work using wikinews revision histories. we did not include wikinews because its collaborative community edits differ from professional news edits. since at least 2006, internet activists have tracked changes made to major digital news articles @xcite . newsdiffs.org, newssniffer and diffengine are platforms which researchers have used to study instances of gender and racial bias in article drafts, 23 @xcite @xcite shifting portrayals of social events, @xcite and lack of media transparency @xcite . these tools collect article versions from rss feeds and the internet archive. major newspa-pers 24 and thousands of government websites foot_22 are being analyzed. we use diffengine and newssniffer to construct newsedits. wikihow @xcite and source code diffs @xcite @xcite @xcite use revision histories from domains and for purposes different than ours. many tasks have benefited from studying wikipedia revisions, like text simplification @xcite , textual entailment (zanzotto and pennacchiotti, 2010), discourse learning @xcite and grammatical error correction @xcite . however, most tasks focus on word-level edit operations to explore sentence-level changes. ours focuses on sentence-level operations to explore document-level changes. research in student learner essays focuses on editing revisions made during essay-writing @xcite @xcite . researchers categorize the intention and effects of each edit @xcite , but do not try to predict edits. ## conclusion in this work, we have introduced the first largescale dataset of news edits, extracted edit-actions, and shown that many were fact-based. we showed that edit-actions are predictable by experts but challenging for current lm-backed classifierss. going forward, we will develop a schema describing the types of edits. we are inspired by the wikipedia intentions schema developed by @xcite , and are working in collaboration with journalists to further clarify the differences. this development will help to clarify the nature of these edits as well as focus further directions of inquiry.