I f QA : A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023)

Abstract: Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability. To address this void, we introduce the first such dataset, named IfQA, where each question is based on a counterfactual presupposition via an “if” clause. Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that the IfQA dataset is highly challenging for existing open-domain QA methods, including supervised retrieve-then-read pipeline methods (F1 score 44.5), as well as recent few-shot approaches such as chain-of-thought prompting with ChatGPT (F1 score 57.2). We hope the unique challenges posed by IfQA will push open-domain QA research on both retrieval and reasoning fronts, while also helping endow counterfactual reasoning abilities to today’s language understanding models.

## introduction counterfactual reasoning captures human tendency to create possible alternatives to past events and imagine the consequences of something that is contrary to what actually happened or is factually true @xcite . take, for example, the business arena where a corporate leadership team might rigorously analyze the potential ripple effects had they opted for an alternative investment strategy (baron, * the majority of the work was completed during wenhao's internship at the allen institute for ai. 2000; @xcite . counterfactual reasoning has long been considered a necessary part of a complete system for ai. however, few nlp resources aim at gauging the effectiveness of such reasoning capabilities in ai models, especially for the opendomain qa task. instead, existing formulations of open-domain qa tasks mainly focus on questions whose answer can be deduced directly from global, factual knowledge (e.g., what was the occupation of lovely rita according to the song by the beatles?) available on the internet @xcite @xcite . counterfactual presupposition in open-domain qa can be viewed as a causal intervention. such intervention entails altering the outcome of events based on the given presuppositions, while obeying the human readers' shared background knowledge of how the world works. to answer such questions, models must go beyond retrieving direct factual knowledge from the web. they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. although some recent work has attempted to answer questions based on counterfactual evidence in the reading comprehension setting @xcite , or identified and corrected a false presupposition in a given question @xcite , none of existing works have been developed for evaluating and improving counterfactual reasoning capabilities in open-domain qa scenarios. to fill this gap, we present a novel benchmark dataset, named ifqa, where each of over 3,800 questions is based on a counterfactual presupposition defined via an "if" clause. two examples are given in figure 1 . ifqa combines causal inference questions with factual text sources that are comprehensible to a layman without an understanding of formal causation. it also allows us to evaluate the capabilities and limitations of recent advances in qa methods in the context of counterfactual reasoning. ifqa introduces new challenges in both retrieval and reading. for example, to answer the 2nd example question in figure 1 , "if the movement of the earth's crust caused the height of mount everest to drop by 300 meters, which mountain would be the highest mountain in the world?", the search and reasoning process can be divided into four steps: (i) retrieve documents relevant to the current height of mount everest (8,848 metres); (ii) calculate the height based the counterfactual presupposition (8,848-300=8,548 metres); (iii) retrieve documents relevant to the current second-highest mountain in the world (k2: 8,611 metres); and (iv) compare the heights of lowered mount everest and k2, then generate the answer (k2). ## conclusion we introduce ifqa, a novel dataset with 3,800 questions, each of which is based on a counterfactual presupposition and has an "if" clause. our empirical analysis reveals that ifqa is challenging for existing open-domain qa methods in both retrieval and reasoning process. it thus forms a valuable resource to push open-domain qa research on both retrieval and counterfactual reasoning fronts.