FRUIT : Faithfully Reflecting Updated Information in Text (2022)

Abstract: Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent. While automated writing assistants could potentially ease this burden, the problem of suggesting edits grounded in external knowledge has been under-explored. In this paper, we introduce the novel generation task of *faithfully reflecting updated information in text* (FRUIT) where the goal is to update an existing article given new evidence. We release the FRUIT-WIKI dataset, a collection of over 170K distantly supervised data produced from pairs of Wikipedia snapshots, along with our data generation pipeline and a gold evaluation set of 914 instances whose edits are guaranteed to be supported by the evidence. We provide benchmark results for popular generation systems as well as EDIT5 – a T5-based approach tailored to editing we introduce that establishes the state of the art. Our analysis shows that developing models that can update articles faithfully requires new capabilities for neural generation models, and opens doors to many new applications.

## introduction information changes on a constant basis. every day, athletes are traded to new teams, and musicians and actors produce new albums and tv shows. maintaining textual knowledge bases to keep track of these changes requires considerable community effort. for instance, a team of 120k volunteer editors make 120 edits to english wikipedia every minute, and write 600 new articles a day. 2 as the knowledge base grows, the amount of maintenance effort is compounded by the need to keep the knowledge base consistent; e.g., each edit may render information in one of the existing 6.3m+ articles obsolete. assistive writing technologies have the potential to substantially reduce the burden of keeping text corpora up to date and consistent. however, existing work has mainly focused on correcting grammar @xcite , reducing repetitive typing @xcite , and following rhetorical directives @xcite , whereas the problem of producing edits grounded in external knowledge has received little attention @xcite . in contrast, numerous works have developed systems for distilling external knowledge into text (e.g., wikipedia article generation) by treating the problem as multi-document summarization @xcite or data-to-text generation @xcite . however, these systems are not useful for updating existing texts as they can only generate text from scratch. to help endow writing assistants with grounded editing capabilities, we introduce the novel generation task of faithfully reflecting updated information in text (fruit), where the goal is to incorporate new information into an existing piece of text. an illustration is provided in figure 1 . given an outdated wikipedia article and collection of new information about the article's subject, fruit requires updating the existing text so that it is consistent with the new information, as well as adding text to reflect new salient facts, e.g., in figure 1 , the first sentence is updated to reflect that tom kristensson now drives in the junior world championship, and new sentences are added to reflect his achievements in 2019 and 2020. fruit presents several unique challenges. first, unlike many generation tasks, models cannot obtain good performance by solely relying on their parametric world knowledge. whenever the provided evidence contradicts parametric knowledge, the model must prefer the evidence, which recent work has shown is difficult for pretrained language figure 1: illustration of the fruit task. an outdated original article and relevant new information are provided as inputs, and the goal is to generate the updated article. in this example, the original article about tom kristensson was written in 2020, and the new information is comprised of updated information about tom kristensson that has been added to other wikipedia articles between 2020 and 2021. given these inputs, the goal is to produce the updated 2021 version of article. models need to identify the relevant supporting facts (orange and teal) to generate faithful updates while ignoring superfluous information (grey). models @xcite . second, the generated text needs to be faithful to both the original article and the new evidence, except when evidence invalidates information in the existing article. finally, this task requires models to jointly read and analyze evidence from both textual and tabular sources and determine which is relevant and which can be ignored, thus combining challenging aspects of both multi-document summarization and data-to-text generation. to facilitate research on this task, we release the fruit-wiki dataset, a collection of over 170k distantly supervised ("silver") update-evidence pairs. this dataset is produced by comparing pairs of english wikipedia snapshots to identify updates to an article between two snapshots, and associating information from the other articles that supports these updates under a distant supervision assumption. as there is no guarantee that updates in the later wikipedia snapshots can be supported by the collected evidence, we also collect a "gold" evaluation set of 914 human annotated update-evidence pairs where unsupported claims have been removed without disturbing fluency. we train and validate our models using silver data and then evaluate the final performance using gold data. we establish initial benchmark results for a number of trivial and neural sequence-to-sequence base-lines. we also introduce edit5, a t5-based model specially adapted for grounded editing, which establishes state-of-the-art performance on fruit-wiki. through an extensive set of analyses, we identify a number of failure modes needed to be improved upon in order to obtain better performance on fruit-wiki, as well as other interesting topics for future work on this task. we additionally release our data collection pipeline to allow researchers to produce data from future wikipedia snapshots and other languages, which we show to produce high-quality silver data. ## dataset collection and analysis as discussed in the introduction, keeping track of new information and then updating articles to reflect that information requires a massive amount of manual effort. thus, in order to scalably collect sufficient data for training and evaluating fruit systems, some amount of automation is likely required. in this section we introduce the fruit-wiki dataset and associated data collection pipeline, which allows the automatic collection of high-quality training and evaluation data for fruit from pairs of wikipedia snapshots. ## methods in this section we introduce baseline methods to establish initial benchmark results on fruit-wiki. we consider trivial approaches that copy task inputs, as well as t5, a neural sequence-to-sequence baseline which has shown strong performance on related tasks such as summarization @xcite we additionally introduce edit5, a variant of t5 that produces a sequence of edits instead of the entire updated text, and employs additional tweaks to improve performance. ## results and analysis baseline results on the gold evaluation data are provided in although our models are trained on silver data, they still obtain good performance on the gold evaluation set. this shows the high quality of our silver data collection pipeline, and t5's ability to generate reasonable updates based on the evidence. for the t5 baselines, we find that adding evidence to the input results significant increase in all metrics, demonstrating that using the evidence is crucial to obtaining good performance. edit5 obtains additional 3-5% absolute increase in all performance metrics compared to t5, establishing edit5 as a strong baseline for future systems to be compared against. the reduction of unsupported entity tokens implies that edit5 hallucinates less frequently than t5 models. results are provided for different model sizes to illustrate how performance scales with parameter counts. example output an example edit5 output is provided in figure 4 , and additional outputs in appendix e. the examples illustrate important features of the task. in figure 4 the goal is to update the wikipedia article for holli sullivan to reflect her new role of secretary of state of indiana. in the reference, this information is reflected in an updated version of the first sentence as well as in a newly added last sentence. an additional sentence is added after the first sentence paraphrasing the introduction of the source article, which describes sullivan's previous position as a member of the indiana house of representatives. in the edit5 output for this example, information is only added at the end of the article. while the model correctly states that sullivan was appointed to be secretary of state by governor eric holcomb, as well as includes additional context surrounding sullivan's appointment that is paraphrased from the evidence, there are some issues with the output. first, because the first sentence of the article is not updated there is conflicting information about sullivan's current position. second, the added sentence hallucinates that sullivan was appointed in january 2020 when she was actually appointed in march 2021, a fact that directly appears in the evidence. categorizing errors to better understand the types of errors made by edit5, we review a random sample of 100 of its predictions on the gold evaluation data and categorize them as either: grounded updates, meaning all generated claims are supported, ungrounded updates, meaning at least one unsupported claim appears in the output, or no updates, meaning the model did not predict any updates. for grounded updates we additionally keep track of how many updates include additional content not present in the ground truth update, or are missing content that appears in the ground truth update. for ungrounded updates we track whether an incorrect number/date appears in the update, the model distorted evidence, i.e., paraphrased or combined claims in the evidence in a way that changed their meaning, or hallucinated new claims unrelated to the evidence. the results of this analysis are presented in ## conclusion and future work in this work we introduced fruit, a novel text generation task where the goal is to update an article to reflect new information about its subject. to enable research on this task, we formulated a pipeline for extracting weakly supervised training and evaluation data from pairs of wikipedia snapshots, and collected data for the years 2019-2020 and 2020-2021, as well as human annotated gold evaluation data. we additionally provided results for several strong baselines, that demonstrate both the feasibility of this task, as well as strong correlation between gold and distantly supervised data evaluation performance that establishes the trustworthiness of future data produced using our pipeline. this work lays the foundation for future research into making faithful updates to entries in textual knowledge bases. one limitation of this work is that the metrics we use to evaluate faithfulness are all entity-centric, and thus may overstate the performance of models whose edits include the correct entities but misspecify the relations between them or other facets of the evidence. accordingly, one promising direction for future work on this task is to develop more robust metrics for measuring faithfulness. an additional promising direction for future work is to consider open settings of evidence collection, where other forms of updated information such as excerpts from news articles could be used to justify edits in place of updates to other entries in the same knowledge base. relatedly, we also recommend studying this task in streaming settings, where updates arrive in sequential fashion, in addition to the batch setting considered in this work. finally, there are a number of promising directions for improving model performance on this task. in particular, copy mechanisms @xcite have been widely used in data-to-text tasks @xcite , and may help mitigate issues such as the mistranscribed dates we saw in section 5.