CREPE : Open-Domain Question Answering with False Presuppositions (2023)

Abstract: When asking about unfamiliar topics, information seeking users often pose questions with false presuppositions. Most existing question answering (QA) datasets, in contrast, assume all questions have well defined answers. We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections. Through extensive baseline experiments, we show that adaptations of existing open-domain QA models can find presuppositions moderately well, but struggle when predicting whether a presupposition is factually correct. This is in large part due to difficulty in retrieving relevant evidence passages from a large text corpus. CREPE provides a benchmark to study question answering in the wild, and our analyses provide avenues for future work in better modeling and further studying the task.

## introduction when an information-seeking user poses a question about the topic they are unfamiliar with, they can often introduce false presuppositions @xcite which are assumed but not directly stated. for instance, the question in figure 1 incorrectly presupposes that the equal and opposite reactions in newton's law apply to the same object. although such a question is unanswerable, we might still hope to identify the confusion and explain it to the user. this functionality goes well beyond prior open-domain qa task formulations, which focus on questions with a valid direct answer @xcite or that are unanswerable from lack of evidence @xcite @xcite . while overly brief paraphrases of the third law, like "action equals reaction" might have caused confusion among generations of students: the "action" and "reaction" apply to different bodies. ## dataset: crepe our task requires the model to be given a question q and to (a) identify whether q has any false presuppositions, and (b) if yes, generate the false presupposition as well as the correction. we have three criteria in constructing the dataset: c1. naturalness of the questions: whether the questions in the data are written by real, information-seeking users. c2. validity of the presupposition: whether the identified presupposition is highly likely made by the question writer. c3. correctness and adequacy of the information: we first describe the source of questions (section 3.1) which address c1. we then describe the process of annotation (section 3.2) that addresses c2 and c3. finally, we present detailed, qualitative analysis of the data (section 3.3). the formal definition of the task and metrics are in section 4. ## task setup the model is given a question q, and is required to perform the following subtasks: ## experiments: detection this section discusses baseline experiments for the detection subtask; section 6 discusses baseline experiments for the writing subtask. ## conclusion we introduced crepe: the first benchmark for the identification and correction of false presuppositions in the open-domain setup. crepe consists of 8,400 user questions, 25% of which contain false presuppositions and are paired with their corrections. our detailed analysis highlights challenges in solving the task, including (1) retrieval of evidence that identifies false presupposition, (2) identification of implicit and subtle presuppositions, and (3) generating correction that is accurate and adequately explains how and why the presupposition is false. we hope our benchmark adds to the problem of open-domain, open-ended question answering, inviting researchers to build models to study questions with false presuppositions. further, we suggest future work to develop better models, explore approaches to address inherent debatability of the judgment, and evaluation of the model generation.