SEER : Facilitating Structured Reasoning and Explanation via Reinforcement Learning (2024)

Abstract: Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems. However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges. Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps. Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning. In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation. Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps. In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps. Extensive experiments show that SEER significantly outperforms state-of-the-art methods, achieving an absolute improvement of 6.9% over RL-based methods on EntailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance.

## introduction navigating machines to understand and articulate the thought process from posing a question to arriving at an answer has been a long-term pursuit in the ai community @xcite . current qa explainable systems adeptly furnish brief supporting evidence @xcite . however, they often fail to clarify the reasoning process from prior knowledge structured explanation ℎ: sturdy wood would be the best for making a table facts 𝑥𝑥 1 : wood is usually sturdy 𝑥𝑥 2 : a table is a kind of furniture 𝑥𝑥 3 : making furniture requires sturdy materials 𝑥𝑥 4 : wood is a kind of natural material or corpus 𝑖𝑖 1 : wood is usually a sturdy natural material 𝑖𝑖 2 : making a table requires sturdy materials intermediate conclusions reasoning steps 𝑖𝑖 1 𝑖𝑖 2 ℎ 𝑥𝑥 3 𝑥𝑥 4 𝑥𝑥 2 𝑥𝑥 1 q: which natural material is best for making a table? a: sturdy wood 𝑥𝑥 3 𝑥𝑥 4 𝑖𝑖 2 𝑥𝑥 2 𝑥𝑥 1 𝑖𝑖 1 & & & ℎ 𝑖𝑖 1 𝑖𝑖 2 figure 1 : an example of structured explanation. given a hypothesis h (a declarative sentence derived from a question-answer pair) and a set of facts (or corpus), the goal is to generate a structured explanation, which delineates the reasoning process from facts to the hypothesis. to the derived answer. by elucidating the reasoning process of answers generation from the language models, we can greatly improve interpretability, trustworthiness, and debuggability @xcite . as illustrated in figure 1 , when generating answers for the question "which natural material is best for making a table?", the reasoning process with structured explanations, such as entailment trees @xcite or reasoning graphs @xcite , explains why "sturdy wood" is the best answer. deriving such complex structured explanations poses a great challenge. previous methods @xcite consider structured explanations as linearized sequences and generate the entire reasoning process in one go. however, these methods lack controllability and may hallucinate unreliable reasoning steps. to ad- @xcite , supervised method, fame @xcite , and our approach. our contributions are summarized as follows: ## conclusions we propose seer, a novel approach that facilitates structured reasoning and explanation via rl. to our knowledge, seer is the first general framework capable of enhancing chained, tree-based, and graph-based structured reasoning. our structurebased return precisely delineates the hierarchical and branching structure inherent in structured reasoning, effectively facilitating reasoning ability. furthermore, seer employs a generative model to represent the policy and refines the reward function, ingeniously circumventing the limitations of existing works. comprehensive experimental results demonstrate that seer significantly outperforms state-of-the-art methods and exhibits outstanding cross-dataset generalization performance.