METGEN : A Module-Based Entailment Tree Generation Framework for Answer Explanation (2022)

Abstract: Knowing the reasoning chains from knowledge to the predicted answers can help construct an explainable question answering (QA) system. Advances on QA explanation propose to explain the answers with entailment trees composed of multiple entailment steps. While current work proposes to generate entailment trees with end-to-end generative models, the steps in the generated trees are not constrained and could be unreliable. In this paper, we propose METGEN, a Module-based Entailment Tree GENeration framework that has multiple modules and a reasoning controller. Given a question and several supporting knowledge, METGEN can iteratively generate the entailment tree by conducting single-step entailment with separate modules and selecting the reasoning flow with the controller. As each module is guided to perform a specific type of entailment reasoning, the steps generated by METGEN are more reliable and valid. Experiment results on the standard benchmark show that METGEN can outperform previous state-of-the-art models with only 9% of the parameters.

## introduction explanation is recognized as a key factor toward responsible ai systems @xcite . in the context of question answering (qa), providing an explanation of the predicted answers can help improve the understandability, debuggability, and trustworthiness of qa models. great efforts have been devoted to revealing how the models predict the answers and give explanations in various forms, including showing an attention map over passages @xcite , giving a snippet of textual evidence @xcite , and selecting answer-supporting sentences @xcite . among all explanation forms, the entailment trees @xcite provide the most detailed and informative explanation by exposing the chains of reasoning from the knowledge to the predictions. as shown in figure 1 (a) and (c), given a hypothesis (summarizing a question+answer pair) and supporting facts (retrieved from a corpus), the goal is to generate an entailment tree where each non-leaf node is an entailment of its children. providing a valid entailment tree would help users to understand how the hypothesis is proved, obtain novel intermediate conclusions from the basic knowledge, and gain detailed information to support decision making. to generate the entailment trees, @xcite propose entailmentwriter, an end-to-end sequence-to-sequence generative model, trained by maximizing the generation likelihood of the linearized gold trees. however, they do not have an explicit strategy to constrain the validity of every single step and the tree structure. thus, the steps are not guaranteed to satisfy the reasoning rules and could be incorrect and unreliable. for example, the step conclusion may not be entailed by the input premises or simply repeat one of the input premises @xcite . furthermore, although their outputs are trees that can indicate the reasoning chains, the mapping mechanisms from the inputs to the trees remain implicit and invisible. to tackle the above problems, we propose met-gen, a module-based framework to generate entailment trees in a more explicit approach and constrain the entailment steps with reasoning rules. as shown in figure 1 (b), given the target hypothesis and known facts, metgen first uses the reasoning controller to select some steps that can help get closer to the hypothesis. subsequently, metgen executes the selected steps with single-step entailment modules and adds the generated intermediate facts into the known facts for the next round of reasoning. through this iterative approach, metgen proves the hypothesis step by step and generates the overall entailment tree. each module in metgen is a generative model that can perform a specific type of entailment reasoning (e.g., making a substitution inference). to guide the modules to generate correct and sound conclusions, we train the modules with wellformed synthetic data containing the corresponding logical regularities of the reasoning types @xcite . inspired by the forward chaining and backward chaining algorithms in logic programming @xcite , we adopt both deductive and abductive modules to execute forward and backward reasoning steps, respectively. experiments on the standard benchmark entail-mentbank @xcite show that metgen can outperform the previous best model with 9.0% of the model parameters. manual evaluation results demonstrate that metgen can generate more reliable steps. further experiments under the datascarce setting and cross-dataset setting (on eqasc and eobqa @xcite ) show that metgen is more data-efficient and has better generalization capability compared with the baselines. ## task definition as shown in figure 1 , the inputs are a hypothesis h and some fact sentences @xmath0, s 2 , . . . , s n } (including both relevant and irrelevant ones) expressing knowledge. h is a declarative sentence derived from a question+answer pair and can be proved by the knowledge in s. the desired output is a valid entailment tree t with the root node being h, the leaves being facts selected from s, and the intermediate nodes being novel intermediate facts (e.g., i 1 , i 2 ). t is considered valid if each non-leaf node is a valid entailment (a conclusion that "a person would typically infer" @xcite ) of its immediate children. we denote the annotated gold tree as t gold and its leaf facts as s gold . following @xcite , we consider three increasingly difficult tasks with different s: task1 (no-distractor): @xmath1, task2 (distractor): @xmath2, task3 (full-corpus): @xmath3. ## metgen figure 2 illustrates the reasoning process of met-gen. we reason one step at a time and iteratively generate the entailment trees. in each iteration, given a reasoning state (e.g., the initial state r 0 : h ⇐ s, where we aim to prove h using s), the reasoning controller selects promising steps, including forward deductive steps and backward abductive ones. we then use the corresponding modules to perform single-step entailment on the selected steps and generate novel intermediate facts. finally, we use the controller to verify the generated facts and select the correct states to perform further reasoning. we introduce details about the module design, reasoning controller, and reasoning algorithm in sec 4.1, 4.2, and 4.3, respectively. ## experiments we conduct experiments on entailmentbank @xcite , the first dataset supporting qa explanations in the form of the entailment tree. entail-mentbank contains 1,840 entailment trees, each of which corresponds to a question from the arc dataset @xcite . on average, each tree contains 7.6 nodes across 3.2 steps. summary statistics are shown in ## conclusion we propose metgen, a module-based framework to generate the entailment trees for explaining answers. metgen reasons with single-step entailment modules and the reasoning controller. experiments on entailmentbank benchmark show met-gen can generate valid trees with reliable steps and achieve sota performance.