RLET : A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees (2022)

Abstract: Interpreting the reasoning process from questions to answers poses a challenge in approaching explainable QA. A recently proposed structured reasoning format, entailment tree, manages to offer explicit logical deductions with entailment steps in a tree structure. To generate entailment trees, prior single pass sequence-to-sequence models lack visible internal decision probability, while stepwise approaches are supervised with extracted single step data and cannot model the tree as a whole. In this work, we propose RLET, a Reinforcement Learning based Entailment Tree generation framework, which is trained utilising the cumulative signals across the whole tree. RLET iteratively performs single step reasoning with sentence selection and deduction generation modules, from which the training signal is accumulated across the tree with elaborately designed aligned reward function that is consistent with the evaluation. To the best of our knowledge, we are the first to introduce RL into the entailment tree generation task. Experiments on three settings of the EntailmentBank dataset demonstrate the strength of using RL framework.

## introduction reasoning over explicitly given knowledge and generating detailed deduction steps are important challenges towards the goal of automated reasoning in ai community @xcite @xcite . interpreting the reasoning process in qa systems can provide humanunderstandable inspections into the logical deductions and help ensure the soundness and reliability of the models @xcite . as shown in figure 1 , given the question "what keeps mars in orbit around the sun?", a reasoning process can explain why the answer "gravity" is predicted, thereby strengthening the trustworthiness. recent work in the field of explainable qa includes extracting keywords or sentences that indicate the answers @xcite , constructing multi-hop explanations from knowledge graph @xcite and generating freeform explanations and reasoning chains @xcite @xcite @xcite . among various explanation approaches, multi-step entailment trees @xcite build tree-structured reasoning chains and explicitly demonstrate the deduction steps from given knowledge to the hypothesis, which can offer interpretability in more detail (figure 1 ). one line of previous work considers the entailment trees as linearised sequences and adopt sequence-to-sequence (seq2seq) models to generate the entire reasoning chain in a single pass with all given sentences as input @xcite . in these approaches, the internal steps lack trustworthiness because the probability distribution remains invisible and not controllable. besides, each deduction step has access to the full input so it might contain information leaked across all input sentences instead of only deducing over its own premises. to alleviate these problems, subsequent work splits the tree into multiple steps and trains the model to perform single step reasoning @xcite @xcite . these models, however, are trained with isolated single-step data, which ignores the dependencies between steps as a whole chain. since new intermediate sentences will be generated at each step, every action poses an impact on its subsequent decisions. the absence of such consideration in training will lead to the inconsistency with evaluation which scores the chain in a comprehensive way. the reasoning chain is a sequence of discrete actions. as such, we address the above issues by presenting rlet, a reinforcement learning (rl) based entailment tree generation framework that models the entire reasoning chain as a markov decision process (mdp). specifically, we decompose the task into two parts: sentence selection and deduction generation. at each step, the model will first select two sentences (including both given facts and generated intermediate conclusions) for composition, and the deduction generation will combine them into a new intermediate conclusion and add it to the next step. after constructing a whole chain, each step will receive a reward depending on its contribution to the overall correctness and validity of the entire tree. enjoying the convenience of crafting reward functions in rl, we can flexibly assign evaluationconsistent rewards to the steps, instead of purely relying on exact match with the gold tree. in such a way, model behaviors can be manipulated with more flexibility, getting rid of the rigorous chronological match with ground truth. supervised by the cumulative rewards, the model is encouraged to find the optimal policy that leads to greater good. such advantages can not only bring improvements on post-hoc explanation modeling, but also benefit interpretable model decision making, where the reasoning process is integrated into the inference process. extensive experiments on three settings of the benchmark dataset entailmentbank @xcite demonstrate that our approach outper-forms existing baseline models on structure correctness and verify the strength of embracing the end-to-end rl framework by modeling the entire reasoning chain. to our knowledge, we are the first to introduce rl framework into the task of entailment tree generation, taking care of the entire chain with the aid of flexible reward design that aligns with the final evaluation metric, bridging the gap between training objective and evaluation results. foot_0 2 method our goal is to provide a step-by-step reasoning process for commonsense science questions, with prior knowledge of the question, the correct answer and a set of fact sentences. we first describe the task formulation and then explain each part in detail. ## conclusion we presented rlet, a rl-based entailment tree generation framework, which contains sentences selection and deduction generation modules and can be trained with cumulative signals across the entire reasoning tree. experiments show that rlet outperforms existing baselines on structure correctness and is applicable in practical scenarios. future directions include applying rl framework on other stepwise methods with more stable and sophisticated rl algorithms.