D iff QG : Generating Questions to Summarize Factual Changes (2023)

Abstract: Identifying the difference between two versions of the same article is useful to update knowledge bases and to understand how articles evolve. Paired texts occur naturally in diverse situations: reporters write similar news stories and maintainers of authoritative websites must keep their information up to date. We propose representing factual changes between paired documents as question-answer pairs, where the answer to the same question differs between two versions. We find that question-answer pairs can flexibly and concisely capture the updated contents. Provided with paired documents, annotators identify questions that are answered by one passage but answered differently or cannot be answered by the other. We release DiffQG which consists of 759 QA pairs and 1153 examples of paired passages with no factual change. These questions are intended to be both unambiguous and information-seeking and involve complex edits, pushing beyond the capabilities of current question generation and factual change detection systems. Our dataset summarizes the changes between two versions of the document as questions and answers, studying automatic update summarization in a novel way.

## introduction given a pair of statements, how can we identify the difference in their information content? this problem has existed in different forms across nlp research, such as recognizing textual entailment @xcite and natural language inference @xcite . the initial focus of this type of research was finding the logical implication relations between sentences. more recently, specialized entailment-like resources and models have been applied to fact verification @xcite with applications to science, education and journalism. this trend has figure 1 : diffqg consists of paired wikipedia passages that correspond to factual edits. the goal is to generate a discriminating question given an answer span such that the question is answerable by one of the passages but not the other or yields different answers. wikipedia revisions across time provide a large scale and highly available source of sentence pairs, leading to new resources such as wikiatomiced-its @xcite and vitaminc @xcite . however, prior work is limited to minimal changes that concern only a single factual addition or change. we introduce diffqg, a manually annotated dataset spanning changes over multiple years. diffqg consists of paired passages with complex factual changes including multiple additions and deletions within the same example. additionally, it provides a way to interpret the prediction in the form of a discriminative question-answer pair that identifies the change. question-answer pairs provide a semi-structured summary of a change: more flexible than knowledge graph triples and more useful than free-form text. for instance, question-answer pairs can represent different types of updates: a new prime minister may update an answer, while a new type of minister would add an entirely new question. question generation (qg) is a new nlp task that consists of generating a question that a provided document answers. there are various successful applications of this approach, including augmenting datasets to train question answering systems @xcite , capturing implicit information written about text @xcite , and building soft knowledge bases @xcite . previous work in qg treated the underlying passages as static @xcite , while real life documents are constantly updated @xcite . as the source corpus is updated, new question-answer pairs must be added and existing ones must be updated. diffqg thus addresses two challenges simultaneously: providing an interpretable summarization of factual changes and updating soft knowledge bases consisting of question answer pairs. we hope that this dataset can also help evaluate the quality of qg models in producing natural, semantically correct, unambiguous, and information-seeking questions. the dataset and code for our experiments will be open sourced. 1 our contributions are the following: (a) we introduce diffqg, an expert-annotated evaluation dataset that consists of questions that summarize the difference between two passages. to the best of our knowledge, no prior dataset exists that covers such long and complicated edits. (b) we propose a set of metrics that can be used to measure improvements in question generation or factual change detection. (c) we evaluate a comprehensive set of baselines that surface the shortcomings of current systems. ## diffqg task the goal of diffqg is to capture how two similar passages differ from each other using questionanswer pairs. in particular, given a base passage x b and a target passage x t , where x t and x b are different versions of the same article, we aim to generate discriminating questions q t . for each q @xmath0, the information to deduce the corresponding answer span a @xmath1. to limit the scope of possible questions, each answer span a @xmath2. while a t could also be a substring of x b , x b must be missing the required information to deduce a t is the correct answer. alternatively, there could be a corresponding answer span a b , which is the answer resulting from answering q t with x b . note that we consider paraphases of a t , such as lexicalizing numbers and using alternate entity names, as equivalent answers. this discriminating question has certain additional requirements: it should be seeking factual information and stand-alone @xcite (i.e., interpretable when presented by itself without the passage). it is possible that no such discriminating question can be written. the annotators only mark that there is no factual change when they are fairly confident that there is no new information about the answer span in the target passage. consider the following example: annotators are informed that the goal of the process is to collect disambiguated and informationseeking queries that can be answered with one passage but not with the other. by disambiguated queries, we mean queries that refer to roughly a single answer without any context. for instance, "who won two gold medals in the 2012 olympics?" could refer to several different people, and questions of the form "how many medals did he win in the 2012 olympics?" are not answerable at all without the presence of the john doe passage. information-seeking queries are ones where the questioner would not need to know the answer in advance for the question to make sense. this is related to the original goals of natural questions @xcite and corresponds to the cranfield-style questions described by @xcite . as an example, "what did al capone's mother do for a living?" seems like an information-seeking query. on the other hand, "which italian-american gangster's mother was a seamstress?" does not: why would the questioner assume that such a person even exists unless they already knew the answer? we describe the annotation process to acquire such discriminating question set in the next section. ## motivation in the previous section, we described diffqg and its annotation procedure. as mentioned, the purpose of diffqg is to detect and describe factual changes. in particular, diffqg is a rough measurement of a model's ability to automatically construct a database of question-answer pairs that encapsulate the changes. there are many possible formats that could be used as an alternative to summarize factual changes, such as paragraphs, knowledge base triples, or individual claims. while paragraphs can contain nuance, they lack atomicity. it is thus difficult to tell what exactly changed or otherwise compare two changes to each other. this makes them less useful as a database. on the other hand, knowledge base triples are limiting in the types of factual changes that can be described: regardless of the exact setup, the nodes and relations come from some form of fixed vocabulary that may require discarding interesting changes. for instance, changes related to a set of entities, date ranges, various numbers, or abstract information may all be challenging. another alternative method would be a list of claims, similar to vitamin-c @xcite . this method is also atomic and more flexible than knowledge base triples. however, question-answer pairs have a few advantages. first, question-answer pairs are semi-structured information, forming a loose key-value pair. factual edits may change the answer to an existing question or add information corresponding to an entirely new information, requiring a new question. conversely, claims are more difficult to relate to each other. finally, question-answer pairs are interesting because question answering is interesting. previous work has seen the use of a database of questionanswer pairs as a method to improve question answering performance @xcite . a good method for automatically creating and updating such a database thus seems quite useful. as factual corpora change over time, we envision constructing such a database to require iterative updates. ## metrics diffqg can be used to measure performance on three related tasks. factual change detection given an example consisting of a base passage, target passage, and answer span, the goal is to determine whether there exists a valid differentiating question. in other words, whether there is new information about this answer span that is present in the target passage when compared to the base passage. to measure this, we report accuracy, precision, recall and f1 score over the existence of a differentiating question in our annotations. note that always predicting no change achieves 60.3% accuracy but 0% f1, but random guessing corresponds to 44.1% f1. ## methods as mentioned, diffqg can be thought of as a composition of two tasks: factual change detection and discriminating question generation. our simple baseline systems thus treat this as a pipeline, first predicting whether or not there is a factual change and then generating a discriminating question if there is. we also present baseline models that solve both tasks jointly with a single prediction. our methods are illustrated in figure 3 . note that none of our methods use any part of diffqg as training data, as the dataset is only intended to be used for evaluation. models are instead trained on larger existing datasets for question generation and factual change detection. ## results and discussion we present results separately for factual change detection, question generation and the full system. we also report results separately for the overall performance and the performance on only the subset of questions that are human written; those sentences labeled as edit or context in the annotation phase. selected examples with model outputs are provided in appendix a.1 to illustrate the capabilities and typical errors baseline. ## conclusion in this work, we introduce the diffqg task and dataset to evaluate the ability of nlp systems to summarize changes between two related passages via question generation. we present several heuristic and model baselines as well as a set of metrics to measure performance on the dataset. the dif-fqg task requires models to identify changes in factual relationships and ignore other stylistic edits. we find that existing approaches struggle under these conditions. models trained to perform factual change detection and question generation jointly sometimes fail to understand even simple edits. we hope this work finds value in future research on this important problem.