♫ M u S i Q ue: Multihop Questions via Single-hop Question Composition (2022)

Abstract: Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, requires proper multihop reasoning? To this end, we introduce a bottom–up approach that systematically selects composable pairs of single-hop questions that are connected, that is, where one reasoning step critically relies on information from another. This bottom–up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting k-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2–4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3× increase in human–machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30-point drop in F1). We further add unanswerable contrast questions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.1

## introduction multihop qa datasets are designed to support the development and evaluation of models that perform multiple steps of reasoning in order to answer a question. recent work, however, shows that on existing datasets, models often need not even connect information across all supporting facts, foot_1 because they can exploit reasoning shortcuts and other artifacts to find the correct answers and obtain high scores @xcite @xcite . such shortcuts arise from various factors, such as overly specific sub-questions, train-test leakage, and insufficient distractors. these factors allow models to circumvent connected reasoning-they need not read the context to find answers to previous sub-question(s) or use these answers to answer the later sub-questions that depend on them. the left hand side of fig. 1 illustrates an instance of this problem in an actual question (q) taken from the hotpotqa dataset @xcite . this question has the over-specification issue. at first glance, it appears to require a model to identify kurt vonnegut as the author of armageddon in retrospect, and then use this information to answer the final question about the famous satire novel he authored. however, this framing of the question is insufficient to enforce that models must perform connected multihop reasoning to arrive at the correct answer. a model can, in fact, find the correct answer to this question from the context without finding the answer to q1. this is because, even if a model does not know that a1 refers to kurt vonnegut, there happens to be only one person best known for a satirical novel mentioned in the context. contrast this with the question on the right (q'), which cannot be answered by simply returning a novel that someone was best known for. there are three possible answers in the context and choosing between them requires knowing which author is referenced. this is a desirable multihop question that requires connected reasoning. figure 1 : generating connected multihop questions by composing carefully chosen pairs of single-hop questions. left: a hotpotqa question that would have been filtered out by our approach for not requiring connected reasoning; it can be answered using just q2 without knowing the answer to q1 (since there is only one person mentioned in the context as being best known for a satirical novel). right: a connected question that forces models to reason through both intended hops (since there are multiple people mentioned in the context as being best known for some novel). prior work has characterized such reasoning, where a model arrives at the correct answer without using all supporting facts, as disconnected reasoning @xcite . while this characterization enables filtering or automatically transforming existing datasets @xcite , we ask a different question: how can we construct a new multihop dataset that, by design, enforces connected reasoning? we make two main contributions towards this: we hope our bottom-up multihop dataset construction methodology and our challenging datasets with a mixed number of hops will help develop proper multihop reasoning systems and decomposition-based models. unanswerable qa. prior works have used unanswerable questions for robust reasoning in single-hop @xcite and multihop @xcite ) settings. iirc @xcite focuses on open-domain qa where the unanswerable questions are identified by crowdsourcing questions where relevant knowledge couldn't be retrieved from wikipedia. our idea to make unanswerable multihop questions by removing support paragraphs is most similar to @xcite . while they rely on annotations (potentially incomplete) to identify these support paragraphs, we can use the bridge entities to remove any potential support paragraphs (containing the bridge entity) and better ensure unanswerability. ## multihop reasoning desiderata multihop question answering can be seen as a sequence of inter-dependent reasoning steps leading to the answer. in its most general form, these reasoning steps and the dependencies can be viewed as directed acyclic graph (dag), g q . each node q i in this graph represents a reasoning step or a ''hop'', for example, a single-hop question in multihop qa or a kb relation traversal in graph-based kbqa. an edge (q j , q i ) ∈ edges(g q ) indicates that the reasoning step q i relies critically on the output of the predecessor step q j . for example, in fig. 1 , the single-hop question q2 depends on the answer to q1 , and the graph g q is a linear chain @xmath0. given this framing, a key desirable property for multihop reasoning is connected reasoning: performing each step q i correctly should require the output of all its predecessor steps q j . analytical intuition: suppose a model m can answer each q i correctly with probability p, and it can also answer q i without the output of all its predecessor steps with probability @xmath1. for simplicity, we assume these probabilities are independent across various q i . m can correctly answer a k-hop question q by identifying and performing all its k reasoning steps. this will succeed with probability at most p k . alternatively, as an extreme case, it can ''cheat'' by identifying and performing only the last step q k (the ''end question'') without considering the output of q k-1 (or other steps) at all. this could succeed with probability as much as r, which does not decrease with k and is thus undesirable when constructing multihop datasets. our goal is to create multihop questions that enforce connected reasoning, that is, where r p and, in particular, @xmath2, so that models have an incentive to perform all k reasoning steps. not surprisingly, the connected reasoning property is often not satisfied by existing datasets @xcite @xcite , and never optimized for during dataset construction. as a consequence, models are able to exploit artifacts in existing datasets that allow them to achieve high scores while bypassing some of the reasoning steps, thus negating the main purpose of building multihop datasets. prior work @xcite has attempted to measure the extent of connected reasoning in current models and datasets. however, due to the design of existing datasets, this approach is only able to measure this by ablating the pre-requisites of each reasoning step, namely, the supporting facts. rather than only measure, we propose a method to construct multihop qa datasets that directly optimize for this condition. consider question q on the left-hand side of fig. 1 . it can be answered in two steps, q1 and q2. however, the information in q2 itself is sufficient to uniquely identify a2 from the context, even without considering a1. that is, while there is an intended dependency between q1 and q2, q2 can be answered correctly without requiring the output of its predecessor question q1. our approach constructs multihop questions that prevent this issue, and thereby require the desired connected reasoning. specifically, we carefully choose which single-hop questions to compose and what context to use such that each constituent single-hop question necessitates the answers from one or more previous questions. ## connected reasoning via composition the central issue we want to address is ensuring connected reasoning. our solution is to use a bottom-up approach where we compose multihop questions from a large pool of single-hop questions. as we show later, this approach allows us to explore a large space of multihop questions and carefully select ones that require connected reasoning. additionally, with each multihop question, we will have associated constituent questions, their answers and supporting paragraphs, which can help develop more interpretable models. here we describe the high-level process and describe the specifics in the next section. ## dataset construction pipeline the high-level schematic of the pipeline is shown in fig. 2 . we begin with a large set of rc singlehop questions from 5 english wikipedia-based datasets, squad @xcite , natural questions @xcite , mlqa (en-en) @xcite , t-rex @xcite , and zero shot re @xcite , where instances are of the form (q i , p i , a i ) referring to the question, the associated paragraph, and the answer, respectively. for natural questions, as the context is very long (entire wikipedia page), we use the annotated long answer (usually a paragraph) from the dataset as the context, and the annotated short answer as the answer. then, we take the following two steps: s1. find good single-hop questions. even a tolerably small percentage of issues in single-hop questions can compound into an intolerably large percentage in the composed multihop questions. to mitigate this, we first remove questions that are likely annotation errors. because manually identifying such questions at scale is laborious, we use a model-based approach. we remove the questions for which none of five large trained qa models foot_2 can predict the associated answer @xmath3. furthermore, we remove (i) erroneous questions where the answer spans are not in the context, (ii) questions @xmath4, and (iii) questions @xmath5. ## empirical findings we now discuss our findings, demonstrating that musique is a challenging multihop dataset that is harder to cheat on than existing datasets ( §8.1) and that the steps in the musique construction pipeline are individually valuable ( §8.2). finally, we explore avenues for future work ( §8.3). for hq and 2w, we report validation set performance. for -ans and -full, ## conclusion constructing multihop datasets is a tricky process. it can introduce shortcuts and artifacts that models can exploit to circumvent the need for multihop reasoning. a bottom-up process of constructing multihop from single-hop questions allows systematic exploration of a large space of multihop candidates and greater control over which questions we compose. we showed how to use such a carefully controlled process to create a challenging dataset that, by design, requires connected reasoning by reducing potential reasoning shortcuts, minimizing train-test leakage, and including harder distractor contexts. empirical results show that -ans has a substantially higher human-model gap and is significantly less cheatable via disconnected reasoning than previous datasets. the dataset also comes with unanswerable questions, and question decompositions which we hope spurs further work in developing models that get right answers for the right reasons.