Unification-based Reconstruction of Multi-hop Explanations for Science Questions (2021)

Abstract: This paper presents a novel framework for reconstructing multi-hop explanations in science Question Answering (QA). While existing approaches for multi-hop reasoning build explanations considering each question in isolation, we propose a method to leverage explanatory patterns emerging in a corpus of scientific explanations. Specifically, the framework ranks a set of atomic facts by integrating lexical relevance with the notion of unification power, estimated analysing explanations for similar questions in the corpus. An extensive evaluation is performed on the Worldtree corpus, integrating k-NN clustering and Information Retrieval (IR) techniques. We present the following conclusions: (1) The proposed method achieves results competitive with Transformers, yet being orders of magnitude faster, a feature that makes it scalable to large explanatory corpora (2) The unification-based mechanism has a key role in reducing semantic drift, contributing to the reconstruction of many hops explanations (6 or more facts) and the ranking of complex inference facts (+12.0 Mean Average Precision) (3) Crucially, the constructed explanations can support downstream QA models, improving the accuracy of BERT by up to 10% overall.

## introduction answering multiple-choice science questions has become an established benchmark for testing natural language understanding and complex reasoning in question answering (qa) @xcite @xcite . in parallel with other nlp research areas, a crucial requirement emerging in recent years is explainability @xcite @xcite . to boost automatic methods of inference, it is necessary not * equal contribution only to measure the performance on answer prediction, but also the ability of a qa system to provide explanations for the underlying reasoning process. the need for explainability and a quantitative methodology for its evaluation have conducted to the creation of shared tasks on explanation reconstruction @xcite using corpora of explanations such as worldtree @xcite . given a science question, explanation reconstruction consists in regenerating the gold explanation that supports the correct answer through the combination of a series of atomic facts. while most of the existing benchmarks for multi-hop qa require the composition of only 2 supporting sentences or paragraphs (e.g. qasc @xcite , hotpotqa @xcite ), the explanation reconstruction task requires the aggregation of an average of 6 facts (and as many as ≈20), making it particularly hard for multi-hop reasoning models. moreover, the structure of the explanations affects the complexity of the reconstruction task. explanations for science questions are typically composed of two main parts: a grounding part, containing knowledge about concrete concepts in the question, and a core scientific part, including general scientific statements and laws. consider the following question and answer pair from worldtree @xcite : an explanation that justifies a is composed using the following sentences from the corpus: (f 1 ) a stick is a kind of object; (f 2 ) to rub together means to move against; (f 3 ) friction is a kind of force; (f 4 ) friction occurs when two objects' surfaces move against each other; (f 5 ) friction causes the temperature of an object to increase. the explanation contains a set of concrete sentences that are conceptually connected with q and a (f 1 ,f 2 and f 3 ), along with a set of abstract facts that require multi-hop inference (f 4 and f 5 ). previous work has shown that constructing long explanations is challenging due to semantic drift -i.e. the tendency of composing out-ofcontext inference chains as the number of hops increases @xcite . while existing approaches build explanations considering each question in isolation @xcite , we hypothesise that semantic drift can be tackled by leveraging explanatory patterns emerging in clusters of similar questions. in science, a given statement is considered explanatory to the extent it performs unification @xcite @xcite , that is showing how a set of initially disconnected phenomena are the expression of the same regularity. an example of unification is newton's law of universal gravitation, which unifies the motion of planets and falling bodies on earth showing that all bodies with mass obey the same law. since the explanatory power of a given statement depends on the number of unified phenomena, highly explanatory facts tend to create unification patterns -i.e. similar phenomena require similar explanations. coming back to our example, we hypothesise that the relevance of abstract statements requiring multihop inference, such as f 4 ("friction occurs when two objects' surfaces move against each other"), can be estimated by taking into account the unification power. following these observations, we present a framework that ranks atomic facts through the combination of two scoring functions: an extensive evaluation is performed on the worldtree corpus @xcite , adopting a combination of k-nn clustering and information retrieval (ir) techniques. we present the following conclusions: to the best of our knowledge, we are the first to propose a method that leverages unification patterns for the reconstruction of multi-hop explanations, and empirically demonstrate their impact on semantic drift and downstream question answering. ## related work explanations for science questions. reconstructing explanations for science questions can be reduced to a multi-hop inference problem, where multiple pieces of evidence have to be aggregated to arrive at the final answer @xcite @xcite . aggregation methods based on lexical overlaps and explicit constraints suffer from semantic drift @xcite -i.e. the tendency of composing spurious inference chains leading to wrong conclusions. one way to contain semantic drift is to leverage common explanatory patterns in explanationcentred corpora @xcite . transformers @xcite represent the state-of-the-art for explanation reconstruction in this setting @xcite . however, these models require high computational resources that prevent their applicability to large corpora. on the other hand, approaches based on ir techniques are readily scalable. the approach described in this paper preserves the scalability of ir methods, obtaining, at the same time, performances competitive with transformers. thanks to this feature, the framework can be flexibly applied in combination with downstream question answering models. our findings are in line with previous work in different qa settings @xcite , which highlights the positive impact of explanations and supporting facts on the final answer prediction task. in parallel with science qa, the development of models for explanation generation is being explored in different nlp tasks, ranging from open domain question answering @xcite , to textual entailment @xcite and natural language premise selection @xcite . ## explanation reconstruction as a ranking problem a multiple-choice science question @xmath0, c} is a tuple composed by a question q and a set of candidate answers @xmath1, c 2 , . . . , c n }. given an hypothesis h j defined as the concatenation of q with a candidate answer c @xmath2, the task of explanation reconstruction consists in selecting a set of atomic facts from a knowledge base e @xmath3, f 2 , . . . , f n } that support and justify h j . in this paper, we adopt a methodology that relies on the existence of a corpus of explanations. a corpus of explanations is composed of two distinct knowledge sources: in this setting, the explanation reconstruction task for an unseen hypothesis h j can be modelled as a ranking problem @xcite . specifically, given an hypothesis h j the algorithm to solve the task is divided into three macro steps: ## empirical evaluation we carried out an empirical evaluation on the worldtree corpus @xcite , a subset of the arc dataset @xcite q and c j the system has to retrieve the scientific facts describing how friction occurs and produces heat across objects. the corpus classifies these facts (f 3 , f 4 ) as central. grounding explanations like "stick is a kind of object" (f 1 ) link question and answer to the central explanations. lexical glues such as "to rub; to rub together means to mover against" (f 2 ) are used to fill lexical gaps between sentences. additionally, the corpus divides the facts belonging to f kb into three inference categories: retrieval type, inference supporting type, and complex inference type. taxonomic knowledge and properties such as "stick is a kind of object" (f 1 ) and "friction is a kind of force" (f 5 ) are classified as retrieval type. facts describing actions, affordances, and requirements such as "friction occurs when two object's surfaces move against each other" (f 3 ) are grouped under the inference supporting types. knowledge about causality, description of processes and if-then conditions such as "friction causes the temperature of an object to increase" (f 4 ) is classified as complex inference. we implement relevance and unification score adopting tf-idf/bm25 vectors and cosine similarity function (i.e. 1 -cos( x, y)). specifically, the rs model uses tf-idf/bm25 to compute the relevance function for each fact in f kb (i.e. rs(h j , f i ) function in equation 1 ) while the us model adopts tf-idf/bm25 to assign similarity scores to the hypotheses in e kb (i.e. sim(h j , h z ) function in equation 2 ). for reproducibility, the code is available at the following url: https://github.com/ai-systems/ unification_reconstruction_explanations . additional details can be found in the supplementary material. ## conclusion this paper proposed a novel framework for multihop explanation reconstruction based on explanatory unification. an extensive evaluation on the worldtree corpus led to the following conclusions: for the calculation of the unification score (equation 2). the values adopted for these parameters are as follows: