Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering (2022)

Abstract: Generative question answering (QA) models generate answers to questions either solely based on the parameters of the model (the closed-book setting) or additionally retrieving relevant evidence (the open-book setting). Generative QA models can answer some relatively complex questions, but the mechanism through which they do so is still poorly understood. We perform several studies aimed at better understanding the multi-hop reasoning capabilities of generative QA models. First, we decompose multi-hop questions into multiple corresponding single-hop questions, and find marked inconsistency in QA models’ answers on these pairs of ostensibly identical question chains. Second, we find that models lack zero-shot multi-hop reasoning ability: when trained only on single-hop questions, models generalize poorly to multi-hop questions. Finally, we demonstrate that it is possible to improve models’ zero-shot multi-hop reasoning capacity through two methods that approximate real multi-hop natural language (NL) questions by training on either concatenation of single-hop questions or logical forms (SPARQL). In sum, these results demonstrate that multi-hop reasoning does not emerge naturally in generative QA models, but can be encouraged by advances in training or modeling techniques. Code is available at https://github.com/jzbjyb/multihop .

## introduction empowered by large-scale pre-trained language models (lms) @xcite @xcite , recent years have seen much progress on generative question answering (qa), where lms generate answers given questions in an end-to-end fashion. while most works only demonstrate the performance of such generative qa models on simple questions @xcite , there has been some indication that these models can also answer complex questions that theoretically require multi-hop reasoning @xcite , sometimes to an impressive degree. for example, brown et al. (2020) demonstrate strong performance of lms on multi-hop reasoning tasks such as drop @xcite which requires discrete reasoning and numeracy. on the other hand, many argue that lm-based qa models are not actually performing any reasoning, and rather performing (sophisticated) pattern matching and data memorization @xcite . simultaneously, in the context of extractive qa models that select answers from the provided context, several works have demonstrated that they can leverage superficial signals to return correct answers even when the context does not contain all the supporting facts @xcite in this paper, we perform a closer examination of the multi-hop reasoning capabilities of generative qa models. to do so, we take multi-hop questions and their component single-hop questions to directly query generative qa models, studying their multi-hop reasoning ability. specifically, we use multi-hop questions from the complexwebquestions @xcite and hotpotqa @xcite datasets as the testbed, and generate decomposed single-hop questions using heuristics ( § 2.2). we examine two types of generative qa models, namely closedbook @xcite and open-book @xcite @xcite qa models that either do not or do refer to external knowledge when generating the answer respectively. specifically, we use unifiedqa @xcite as a representative closed-book model, and rag @xcite as a representative open-book model ( § 2.1). we first ask: rq1 is the correctness of decomposed single-hop questions a necessary and sufficient condition for correctness of multi-hop questions? ( § 3.2) are answers to multi-hop and chains of decomposed questions consistent? ( § 3.3) rq2 do models trained on single-hop questions demonstrate zero-shot generalization to multihop questions? ( § 4) we find that generative qa models, even those close to the state-of-the-art, generally do not demonstrate robust multi-hop reasoning abilities, with success on multi-hop questions largely a result of taking shortcuts rather than true multi-hop reasoning. zero-shot multi-hop reasoning ability does not emerge naturally from training on single-hop questions, which motivates our final question: rq3 can we improve models' zero-shot multihop reasoning capacity by training on approximations of real multi-hop questions? ( § 4) motivated by the fact that pre-training on massive text endows lms with the ability to identify semantically similar expressions, our first method uses concatenated decomposed single-hop questions to approximate real multi-hop questions. our second method is inspired by recent work teaching lms complex reasoning capabilities through neural execution of logical forms, e.g. by training neural models to execute sql queries @xcite . we hypothesize that the ability to perform multihop reasoning can also be potentially learned from logical forms without reliance on nl questions. to this end, we propose to use sparql, a standard query language over knowledge bases, as our logical forms to endow generative qa models with the ability to perform multi-hop reasoning, and examine whether learning to execute sparql transfers to the ability to answer nl multi-hop questions. both methods lead to significant improvement on zero-shot multi-hop reasoning performance, and further improvements are obtained when both are combined, opening possibilities for future work ( § 6). ## generative question answering in this section, we briefly introduce generative qa models and multi-hop qa datasets. then we elaborate on how we use multi-hop and decomposed questions to perform experiments. ## improving zero-shot multi-hop reasoning capability in this section, we first examine lms' zero-shot capacity for multi-hop reasoning when they are not trained on multi-hop nl questions. compositional generalization ability (lake and baroni, 2018) is required in this case to generalize from single-hop to multi-hop questions. unsurprisingly, generative qa models perform poorly in this setting, with almost half performance degradation. since multihop nl questions are expensive to obtain, one natural question is "is it possible to improve the multihop reasoning ability using only single-hop nl questions, or even without any nl questions?" we design two methods to achieve this goal. motivated by the fact that unifiedqa and rag models are initialized with language models pre-trained on massive text corpora, which endows them with the ability to identify semantically similar expressions, our first method uses concatenated decomposed nl questions (i.e., [q 1 , q * 2 ]) to approximate the real nl multi-hop question q, and fine-tunes models on them. the second is inspired by recent progress in teaching lms complex reasoning capabilities by executing logical forms neurally. for example, @xcite formulate the execution of sql over tables as a seq2seq task where the input is a logical form string associated with a table and the output is answers @xcite . we hypothesize that in our multi-hop qa setting, the ability to perform multi-hop reasoning can also be potentially learned from logical forms without reliance on any nl question. to this end, we propose to use sparql, which is a standard query language over knowledge bases, as our logical forms. we then examine whether the ability to answer questions expressed in these sparql queries is transferable to nl multi-hop questions. the advantage of using sparql for training is that sparql queries can be easier and cheaper to obtain or generate than nl. for example we can use existing query logs, 7 or use manual sparql queries as templates and replace entities/relation to generate more queries. 8 our observation sheds light on potential improvement on multi-hop reasoning using many sparql query-answer pairs. ## conclusion in this paper, we examined the multi-hop reasoning capabilities of generative qa models, finding that overall models take shortcuts when answering multi-hop questions, not demonstrating convincing multi-hop reasoning capability. when trained only on single-hop questions, models generalize poorly to multi-hop questions, while approximation using the concatenation of single-hop questions and sparql queries improves the multi-hop performance significantly. further directions include better approximations of multi-hop questions and advanced modeling techniques that encourage compositional ability.