Can We Guide a Multi-Hop Reasoning Language Model to Incrementally Learn at Each Single-Hop? (2022)

Abstract: Despite the success of state-of-the-art pre-trained language models (PLMs) on a series of multi-hop reasoning tasks, they still suffer from their limited abilities to transfer learning from simple to complex tasks and vice-versa. We argue that one step forward to overcome this limitation is to better understand the behavioral trend of PLMs at each hop over the inference chain. Our critical underlying idea is to mimic human-style reasoning: we envision the multi-hop reasoning process as a sequence of explicit single-hop reasoning steps. To endow PLMs with incremental reasoning skills, we propose a set of inference strategies on relevant facts and distractors allowing us to build automatically generated training datasets. Using the SHINRA and ConceptNet resources jointly, we empirically show the effectiveness of our proposal on multiple-choice question answering and reading comprehension, with a relative improvement in terms of accuracy of 68.4% and 16.0% w.r.t. classic PLMs, respectively.

## introduction recent developments have shown that models based on transformers @xcite have emerged as effective soft reasoners over language @xcite . to teach transformers the ability to emulate reasoning, they are trained on knowledge encoded in the form of natural language statements generally built upon explicit rules @xcite or symbolic facts that refer to triples in knowledge graphs (kg) @xcite . in addition, the reasoning skills of these models can successfully combine explicit natural language statements with implicit knowledge acquired during pre-training @xcite . in particular, many state-of-the-art pre-trained language models (plms), such as bert @xcite or roberta @xcite , have been successfully used in multi-hop reasoning problems includ-ing multi-hop question-answering (qa) tasks @xcite @xcite and multihop reading comprehension (rc) @xcite . training multi-hop reasoning specifically implies a two-step process: 1) distinguish -within a context-the relevant facts from the distractors to be used for reasoning; both relevant facts and distractors are generally expressed as statements in natural language using linguistic patterns @xcite ; 2) reasoning over a sequence of relevant facts leading to chains of reasoning @xcite . a common approach to teaching plms to solve a multi-hop reasoning task is to convert the structural reasoning task into subtasks that model the sequence of reasoning tasks. for instance, @xcite and @xcite rely on a multitasking training strategy @xcite that uses training instances mixing different depths of reasoning steps (hops). more precisely, to teach a model solving a k-hop reasoning problem, it is trained to simultaneously solve single i-hop (1 ≤ @xmath0. in the same line, @xcite and @xcite carry out several steps of single-hop reading comprehension to simulate multi-hop reasoning. however, while yielding impressive results, it is still unclear if plms endowed with multi-hop reasoning skills really leverage the learned skills at each single-hop depth level along the reasoning chain. more specifically, our work is motivated by observing that plms yield unpredictable results while performing multi-hop reasoning. for instance, previous studies show that the performance of plms degrades substantially even with a slight increase in the number of hops in the underlying reasoning tasks @xcite . this result indicates that multi-hop models at lower depths struggle to transfer information to deeperhop models, giving rise to the compositionality generalization @xcite issue from simpler to complex tasks. in this work, we advocate that a better understanding of the inherent relationships between the different single-hop reasoning models allows the design of more predictable models. we seek to answer three main questions. first, grounded in previous findings in the literature @xcite showing the compositionality generalization issue, do single-hop reasoning models incrementally learn? (rq1). we construct large probe datasets using shinra @xcite , conceptnet @xcite , and rule-takers @xcite using single-depths of inference to train and probe single-hop models and compare their performance. overall, our findings confirm the prevalence of the compositionality generalization issue from complex to less complex multi-hop reasoning tasks. second, inspired by the human reasoning style to solve complex problems based on simpler ones @xcite , can plms be guided toward incremental reasoning? (rq2). specifically, we propose a generic and automatic methodology for generating training probe datasets that endow plms with reasoning capabilities over a sequence of single-hop steps. we particularly investigate the impact of using distractor generation strategies. our empirical results show that we can guide plms to incrementally reason by leveraging classic approaches with a gain of up to 7.98 accuracy. these training datasets are publicly available 1 . finally, grounded on previous findings revealing that plms trained on one specific reasoning task improve their performance on different and unrelated reasoning tasks @xcite , do qa tasks leverage incrementally trained reasoning models? (rq3). for the multi-hop qa task, in particular, the results show that our approach quickly adapts to obtain an accuracy of 54.74 compared to 52.03 from state-of-the-art ## methodology in this section, we first introduce the basic definitions and notations used in our proposal and then present the data probe generation methodology. ## results and discussion to address rq1 and rq2, we used the k-hop s-rt with @xmath1, 1, 2, 3, 5} and the {@xmath2, 1-hop, and 2-hop shinet dataset. all the shinet datasets are composed of the object, relationship, and inference distractors. regarding the inference distractors, we evaluated with the three strategies: i-inf, s-inf, g-inf. the default setting uses the (s-inf) strategy unless it is explicitly mentioned otherwise. note that for each shinet strategy-based dataset, we have train and test partitions. as recommended in the literature @xcite , to avoid biases in these datasets that lead to an overestimation of the reasoning capabilities of plms, we applied on the test partitions the aflite algorithm @xcite ) that finds machine-detectable embeddings associations to reduce biases. we used optimal parameters after grid-search: n (classifiers) = 64, m(samples) = 1000, top-@xmath3, @xmath4.75. comparing the original and filtered datasets, we approximately filtered 45% of the total samples. to address rq3, we used the mcqa @xcite , and race @xcite . mcqa is composed of 193,000 entries. each entry is composed of a question and five possible answers, including reasoning tasks such as hypernymy, hyponymy, synonymy detection, and word sense disambiguation. race @xcite consists of nearly 28,000 passages and 100,000 questions divided into middle and high school sets and up to four possible answers. ## related work our main study focused on multi-hop reasoning. recent studies have proposed solutions using decomposition-aggregation approaches @xcite by combining or extending different model architectures to leverage reasoning performance @xcite @xcite , creating explained reasoning paths @xcite or using chain of thought prompting @xcite . in contrast, we focus on leveraging the inner reasoning skills of plms, benefiting from their internal architecture and knowledge captured in pre-training. we argue that our results may be a solid alternative to a standard plm in this kind of work. there are recent demonstrations that trained plms can perform simple reasoning tasks @xcite . even if these models are not naturally good solvers of complex tasks such as multi-hop reasoning @xcite , they are capable of learning when trained on such tasks @xcite . however, these training setups propose mixing different depth levels of reasoning, leading to unpredictable results, and, thus, a lack of model interpretability. they do not let recognize the knowledge captured at each hop level and whether acquired knowledge, if any, is actually reused to solve higher-hop level reasoning. we proposed a single-hop design that lets us analyze the actual contribution of each reasoning level. although our work is inspired by the previous literature, it is different from @xcite and @xcite , as they evaluate the inner reasoning capabilities of plms in simple reasoning tasks, but we evaluate incrementally trained plms for multi-hop reasoning performed on nlp downstream tasks. similarly, @xcite evaluate the model reasoning skills controlling the data given in pre-training. in contrast, we analyze how the training data and elements in the context affect the task in a multi-hop scenario. ## conclusions transformers have been recently gaining increasing attention for reasoning tasks over language. in this paper, we have specifically studied whether we can endow plms used in multi-hop reasoning tasks with the ability to incrementally acquire knowledge by following the inference path over the sequence of hops. our underlying objective is to control the training of plms better, leading to more understandable and predictable multi-hop reasoning models. in particular, we have complemented previous findings in the literature by showing that plms trained on 1-hop reasoning tasks can extrapolate the reasoning to 2-hops but that 2-hop reasoning models struggle to generalize over slightly simpler 1-hop tasks. keeping in mind the human-style reasoning from simpler to complex tasks, we advocate incremental reasoning over the structure of the inference path as a step forward. we provide a training data generation strategy that relies critically on inference distractors connecting intermediate relevant facts in the reasoning path. by applying our approach, our models achieve higher or similar performance trends than fine-tuning multi-hop models but consume fewer resources. furthermore, we show that the incrementally trained multi-hop plms are transferable to other qa-based tasks. although our experimental settings are limited to low depths of inference (@xmath5, 2), our findings show both the feasibility and the benefit of incremental reasoning and open new research op-portunities. we may potentially extend this work toward the benchmarking of multi-hop reasoning interpretability with the design of baseline models, dataset generation strategies with upper bounds, and evaluation metrics including, but not limited to, inference path recall.