Inferring Implicit Relations in Complex Questions with Language Models (2022)

Abstract: A prominent challenge for modern language understanding systems is the ability to answer implicit reasoning questions, where the required reasoning steps for answering the question are not mentioned in the text explicitly. In this work, we investigate why current models struggle with implicit reasoning question answering (QA) tasks, by decoupling inference of reasoning steps from their execution.We define a new task of implicit relation inference and construct a benchmark, IMPLICITRELATIONS, where given a question, a model should output a list of concept-relation pairs, where the relations describe the implicit reasoning steps required for answering the question.Using IMPLICITRELATIONS, we evaluate models from the GPT-3 family and find that, while these models struggle on the implicit reasoning QA task, they often succeed at inferring implicit relations.This suggests that the challenge in implicit reasoning questions does not stem from the need to plan a reasoning strategy alone, but to do it while also retrieving and reasoning over relevant information.

## introduction a longstanding goal of language understanding has been to develop systems that can reason, i.e., integrate multiple pieces of information to reach a conclusion @xcite . this has sparked interest in question answering (qa) benchmarks that require such reasoning @xcite @xcite . one particularly challenging case is questions that require implicit reasoning, that is, where the evidence for answering the question is not mentioned explicitly. consider the question "does santa claus work during summer?". this question requires implicit reasoning since it involves knowing when the holiday associated with santa occurs, but this is not evident from the question. recent advances in qa @xcite have steered attention towards would a stool be useful for a lusotitan to reach the top of an almond tree? ⟨lusotitan , height⟩ ⟨almond tree , height⟩ implicit reasoning qa benchmarks such as strat-egyqa @xcite , opencsr @xcite , commonsenseqa 2.0 @xcite , creak @xcite , and realfp @xcite , which span a wide range of domains and reasoning skills. still, implicit reasoning remains an open challenge, even for large language models (lms) such as gpt-3 and palm @xcite @xcite . answering implicit reasoning questions can be viewed as a two-step process: (a) inferring simple sub-questions necessary for answering the question, and (b) retrieving the relevant knowledge pieces (i.e., answering sub-questions) and reasoning over them to derive the answer. figure 1 illustrates this decoupling. to answer the shown question, we need to use knowledge about the lusotitan dinosaur and almond trees to infer that the relevant sub-questions concern their heights. we refer to the relation height, which is not mentioned in the question as an implicit relation. once implicit relations are inferred, we can retrieve the relevant facts and deduce that the answer is 'false', as a lusotitan is in this work, we put a spotlight on implicit relations and investigate the ability of language models to infer them as a necessary (albeit insufficient) step for answering implicit reasoning questions. we first define implicit relations, and show that they can be reliably annotated through crowdsourcing (example annotated implicit relations are in figure 1 and we use our benchmark to evaluate the ability of large lms to infer implicit relations, since they are known to acquire substantial amounts of knowledge and common sense with scale @xcite @xcite , but struggle with implicit reasoning questions. specifically, we evaluate models from the gpt-3 family using in-context learning, where the model is fixed and only a few examples are given as context. we find that large lms perform well on this task, with a 175b parameter model recovering 0.53-0.59 of the implicit relations across datasets, outperforming a baseline by 21-40 points. this is robust across methods for sampling in-context examples, and even in cross-data scenarios where in-context examples are sampled from a different dataset than the target question. however, inferring implicit relations does not improve accuracy on the downstream qa task, even when gold relations are provided. this suggests that the challenge of implicit reasoning questions is not primarily due to implicit relation inference, but possibly due to the need to also retrieve information and reason over it. to conclude, in this work we propose the notion of implicit relations, and construct the implic-itrelations evaluation benchmark for testing the ability of models to infer them from questions. we evaluate large lms and show that they infer implicit relations fairly well, while still falling short of answering implicit reasoning questions. our work facilitates future work on improving implicit relation inference, and sheds light on the factors relevant for developing models that can handle implicit reasoning. from a broader perspective, our work joins recent community efforts to highlight the ubiquity of missing and implicit elements in natural language @xcite @xcite . 1 ## implicit relations we now define the notion of implicit relations in the context of complex question answering. complex questions are questions that require multiple steps of reasoning in order to be answered @xcite @xcite . for example, the question "was linnaeus alive when darwin published origin of species?" involves fetching two dates and then comparing them (figure 2 ). a prominent challenge in complex qa, that attracted substantial attention recently @xcite @xcite ; did linnaeus edit darwin's draft of origin of species? was linnaeus alive when darwin published origin of species? s 1 when did carl linnaeus pass away? s 2 when was origin of species first published? s 3 is #2 before #1? implicit reasoning explicit reasoning decomposition d implicit relations i ⟨origin of species , year published ⟩ ⟨linnaeus , year of death ⟩ i 1 i 2 retrieval retrieval logical figure 2 : an example of explicit and implicit reasoning questions that share the same question decomposition, along with the implicit relations derived from the retrieval steps in the decomposition. @xcite @xcite , is cases where the reasoning steps are implicit and should be inferred from the question. for instance, "did linnaeus edit darwin's draft of origin of species?" involves the same reasoning steps, but they are not mentioned explicitly. thus, the former question is an explicit reasoning question, while the latter is an implicit one (figure 2 ). @xcite proposed qdmr as a meaning representation for complex questions, where a complex question q is decomposed into a sequence of m reasoning steps @xmath0, . . . , s m ), and each step s i corresponds to a simple natural language question. answering the simple questions one-by-one yields the final answer (see decomposition in figure 2 ). @xcite collected decompositions for implicit reasoning questions as part of the strategyqa dataset, where importantly, inferring the sequence of reasoning steps is challenging due to their implicit nature. in addition to generating effective decompositions for implicit reasoning questions, we find an additional challenge in evaluating these decompositions when represented as a sequence of sub-questions. specifically, @xcite distinguished two types of reasoning steps in a decomposition -retrieval steps, which require retrieval of facts (s 1 and s 2 in figure 2 ), and logical steps, which perform logical reasoning over previous results (s 3 in figure 2 ). in this work, we observe that a key ingredient in inferring decompositions is to identify the implicit relations that are necessary for answering the question. concretely, each retrieval step in a question decomposition can typically be represented as concept-relation pair ⟨c, r⟩, where c is a sequence of tokens from the question that refer to a concept, and r is a relation of that concept. for example, the concept c in step s 2 in figure 2 is "origin of species", and the relation r is its publication year. based on this observation, we provide the following definition for implicit relations in complex qa. let q be an implicit reasoning question, and denote by @xmath1, ..., s m ) its decomposition into a sequence of m reasoning steps. let {s i 1 , . . . , s in } be the subset of retrieval steps in the decomposition d. we define the implicit relations for answering q as the set @xmath2, r 1 ⟩, ..., ⟨c n , r n ⟩} of concept-relation pairs, where each concept-relation pair corresponds to a particular retrieval step. in the next sections, we will use this definition to construct a task for probing the ability of models to infer implicit relations ( §3- §5), and investigate why they struggle on the downstream qa task ( §6). ## the implicitrelations benchmark in this section, we describe the process of creating implicitrelations, a benchmark for evaluating the ability of models to infer implicit relations. ## experimental setting we now turn to evaluate the ability of large lms to infer implicit relations in questions. to this end, we use examples from implicitrelations in a few-shot in-context learning setting @xcite , where given several input-output examples and a test input, the lm is expected to generate the required output. we focus on this setup following the recent progress in in-context learning, specifically for tasks that involve general commonsense reasoning @xcite . ## large lms can infer implicit relations next, we perform additional experiments to (a) further substantiate the ability of lms to infer implicit relations ( §5.1), and (b) test the effect of model scale on performance ( §5.2). ## related work recent work utilized the ability of large lms to generate intermediate reasoning steps for improving performance on qa tasks @xcite @xcite . @xcite introduced 'chainof-thought' prompting to elicit intermediate reasoning steps along with answers from lms, which improved performance on several reasoning tasks. conversely, we propose a task and benchmark for evaluating the ability of lms to infer the intermediate reasoning steps themselves. prior work has dealt with reasoning abilities in lms @xcite @xcite by fine-tuning lms to generate additional knowledge for reasoning questions. we contribute to this effort by evaluating the in-context ability to infer implicit relations with large lms. implicit relations are closely related to question decomposition, which have been used in past work to improve performance on questions that require reasoning @xcite @xcite . we contribute to this research direction by defining implicit relations pairs, which provide a structured representation of the decomposed sub-questions and allow us to examine how language models infer reasoning steps. several works in narrative understanding @xcite @xcite have attempted to assess a model's implicit reasoning capabilities using different methods, such as assessing the solution path to unanswerable questions and and narrative understanding through question-answering. despite their different approaches, these studies are relevant to our cause. ## conclusion we propose the task of implicit relation inference, which decouples inference of reasoning steps from their execution. we introduce implicitrela-tions, a benchmark that includes more than 2,000 annotated implicit relations. we show large lms can infer implicit relations across multiple types of questions and reasoning skills, but this success does not translate to an improvement in answering implicit reasoning questions. our work sheds light on capabilities missing from large lms for addressing implicit reasoning questions, and provides a valuable resource for improving the ability of models to infer implicit relations.