Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021)

Abstract: A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of ∼ 66%.

## introduction developing models that successfully reason over multiple parts of their input has attracted substantial attention recently, leading to the creation of many multi-step reasoning question answering (qa) benchmarks @xcite @xcite @xcite . commonly, the language of questions in such benchmarks explicitly describes the process for deriving the answer. for instance (figure 1 , q2), the question was aristotle alive when the laptop was invented? explicitly specifies the required reasoning steps. however, in real-life questions, reasoning is often implicit. for example, the question did aristotle use a laptop? (q1) can be answered using the same steps, but the model must infer the strategy for answering the question-temporal comparison, in this case. answering implicit questions poses several challenges compared to answering their explicit counterparts. first, retrieving the context is difficult as there is little overlap between the question and its context (figure 1 , q1 and 'e' ). moreover, questions tend to be short, lowering the possibility of the model exploiting shortcuts in the language of the question. in this work, we introduce strategyqa, a boolean qa benchmark focusing on implicit multi-hop reasoning for strategy questions, where a strategy is the ability to infer from a question its atomic subquestions. in contrast to previous benchmarks @xcite , questions in strategyqa are not limited to predefined decomposition patterns and cover a wide range of strategies that humans apply when answering questions. eliciting strategy questions using crowdsourcing is non-trivial. first, authoring such questions requires creativity. past work often collected multi-hop questions by showing workers an entire context, which led to limited creativity and high lexical overlap between questions and contexts and consequently to reasoning shortcuts @xcite ). an alternative approach, applied in natural questions figure 1 : questions in strategyqa (q1) require implicit decomposition into reasoning steps (d), for which we annotate supporting evidence from wikipedia (e). this is in contrast to multi-step questions that explicitly specify the reasoning process (q2). @xcite and ms-marco @xcite , overcomes this by collecting real user questions. however, can we elicit creative questions independently of the context and without access to users? second, an important property in strategyqa is that questions entail diverse strategies. while the example in figure 1 necessitates temporal reasoning, there are many possible strategies for answering questions ( to overcome these difficulties, we use the following techniques in our pipeline for eliciting strategy questions: (a) we prime crowd workers with random wikipedia terms that serve as a minimal context to inspire their imagination and increase their creativity; (b) we use a large set of annotators to increase question diversity, limiting the number of questions a single annotator can write; and (c) we continuously train adversarial models during data collection, slowly increasing the difficulty in question writing and preventing recurring patterns @xcite . beyond the questions, as part of strategyqa, we annotated: (a) question decompositions: a sequence of steps sufficient for answering the question ('d' in figure 1 ), and (b) evidence paragraphs: wikipedia paragraphs that contain the answer to each decomposition step ('e' in figure 1 ). strategyqa is the first qa dataset to provide decompositions and evidence annotations for each individual step of the reasoning process. our analysis shows that strategyqa necessitates reasoning on a wide variety of knowledge domains (physics, geography, etc.) and logical operations (e.g., number comparison). moreover, experiments show that strategyqa poses a combined challenge of retrieval and qa, and while humans perform well on these questions, even strong systems struggle to answer them. in summary, the contributions of this work are: the dataset and codebase are publicly available at https://allenai.org/data/strategyqa . ## data collection pipeline our goal is to establish a procedure for collecting strategy questions and their annotations at scale. to this end, we build a multi-step crowdsourcing 1 pipeline designed for encouraging worker creativity, while preventing biases in the data. 1 we use amazon mechanical turk as our framework. we break the data collection into three tasks: question writing ( §3.1), question decomposition ( §3.2), and evidence matching ( §3.3). in addition, we implement mechanisms for quality assurance ( §3.4). an overview of the data collection pipeline is in figure 2 . ## the strategyqa dataset we run our pipeline on 1,799 wikipedia terms, allowing a maximum of 5 questions per term. we update our online fine-tuned solvers (fntd) every 1k questions. every question is decomposed once, and evidence is matched for each decomposition by 3 different workers. the cost of annotating a full example is $4. to encourage diversity in strategies used in the questions, we recruited new workers throughout data collection. moreover, periodic updates of the online solvers prevent workers from exploiting shortcuts, since the solvers adapt to the training distribution. overall, there were 29 question writers, 19 decomposers, and 54 evidence matchers participating in the data collection. we collected 2,835 questions, out of which 55 were marked as having an incorrect answer during sqd ( §3.2). this results in a collection of 2,780 verified strategy questions, for which we create an annotator-based data split @xcite . we now describe the dataset statistics ( §4.1), analyze the quality of the examples ( §4.2), and explore the reasoning skills in strategyqa ( §4.3). ## experimental evaluation in this section, we conduct experiments to answer the following questions: a) how well do pre-trained language models (lms) answer answer accuracy 87% strategy match 86% decomposition usage 14% average # searches 1. 25 ## related work prior work has typically let annotators write questions based on an entire context @xcite @xcite @xcite . in this work, we prime annotators with minimal information (few tokens) and let them use their imagination and own wording to create questions. a related priming method was recently proposed by @xcite , who used the first 100 characters of a wikipedia page. among multi-hop reasoning datasets, our dataset stands out in that it requires implicit decompositions. two recent datasets @xcite have considered questions requiring implicit facts. however, they are limited to specific domain strategies, while in our work we seek diversity in this aspect. most multi-hop reasoning datasets do not fully annotate question decomposition @xcite @xcite . this issue has prompted recent work to create question decompositions for existing datasets @xcite , and to train models that generate question decompositions @xcite @xcite . in this work, we annotate question decompositions as part of the data collection. ## conclusion we present strategyqa, the first dataset of implicit multi-step questions requiring a widerange of reasoning skills. to build strategyqa, we introduced a novel annotation pipeline for eliciting creative questions that use simple language, but cover a challenging range of diverse strategies. questions in strategyqa are annotated with decomposition into reasoning steps and evidence paragraphs, to guide the ongoing research towards addressing implicit multi-hop reasoning.