Neural Natural Logic Inference for Interpretable Question Answering (2021)

Abstract: Many open-domain question answering problems can be cast as a textual entailment task, where a question and candidate answers are concatenated to form hypotheses. A QA system then determines if the supporting knowledge bases, regarded as potential premises, entail the hypotheses. In this paper, we investigate a neural-symbolic QA approach that integrates natural logic reasoning within deep learning architectures, towards developing effective and yet explainable question answering models. The proposed model gradually bridges a hypothesis and candidate premises following natural logic inference steps to build proof paths. Entailment scores between the acquired intermediate hypotheses and candidate premises are measured to determine if a premise entails the hypothesis. As the natural logic reasoning process forms a tree-like, hierarchical structure, we embed hypotheses and premises in a Hyperbolic space rather than Euclidean space to acquire more precise representations. Empirically, our method outperforms prior work on answering multiple-choice science questions, achieving the best results on two publicly available datasets. The natural logic inference process inherently provides evidence to help explain the prediction process.

## introduction question answering (qa) is an important real-life nlp application but also a challenging task for assessing how well ai systems understand human language and perform reasoning to answer questions. a main challenge of qa is that the answers often do not explicitly exist in a supporting knowledge base but instead need to be inferred from it. prior work @xcite has viewed qa as a textual entailment problem performed on a large premise set, where a question and candidate answers are formulated as hypotheses that need to be proved. neural networks have recently become the mainstream models for qa @xcite @xcite . most of the models, however, are unable to give explainable inference results. developing effective and yet explainable question answering models has attracted more attention @xcite @xcite @xcite . in this paper, we investigate a neural-symbolic qa approach that integrates natural logic reasoning @xcite @xcite within deep learning architectures for qa, aiming to keep the backbone of inference based on the natural logic formalism, while integrating neural networks to make the systems powerful and robust. conventional natural logic has been designed for natural language inference and question answering @xcite . as opposed to performing deduction on an abstract logical form, e.g., first-order logic (fol) or its fragments, in which obtaining representation for abstract logic forms is known to face many thorny challenges, natural logic provides a formal proof framework based on the monotonicity calculus or projectivity. we present the neural natural logic inference (neunli) framework for question answering. the core idea of neunli is bridging a hypothesis and candidate premises by following natural logic inference steps and incorporating neural models to help build the proof paths. neunli first converts a question and candidate answers to form declarative sentences, namely hypotheses. it then rewrites these original hypotheses to obtain intermediate hypotheses and repeats this process to construct a proof tree for each question-answer pair. since the reasoning process forms a tree-like, hierarchical structure @xcite , it can lead to structural distortion when learning embeddings for hypotheses and premises in the euclidean space @xcite . additionally, natural language text exhibits hierarchical structure in a variety of respects @xcite . neunli projects the question and answer embeddings to the hyperbolic space. for a proof tree, neunli computes an entailment score between tree nodes and candidate premises in a hyperbolic space and use that to help select the answer. we demonstrate modelling entailment score in the hyperbolic space improves the performance. to train the above process in an end-to-end differentiable manner, we utilize the gumbel-softmax technique @xcite , which can effectively approximate the discrete variable, as an approximation of the non-differentiable selecting process of candidate mutations. in summary, the contributions of our work are as follows: (1) we introduce a novel framework neunli, which combines the advantages of natural logic and deep neural networks for question answering. (2) our proposed model provides step-by-step explanation for how the prediction was derived. (3) the proposed model achieves new state-of-the-art performance on two qa datasets. we provide detailed analyses demonstrating how the model works to achieve the improvement. the code is released at https://github.com/shijihao/neunli . ## method in this paper we propose the neural natural logic inference (neunli) framework, aiming to combine the advantages of natural logic and deep neural networks for question answering, which builds explainability in the model and leverages the powerful capacity and robustness of neural models. figure 2 depicts the overall architecture of neunli; the pseudocode of neunli is listed in algorithm 1. in the following subsections, we discuss neunli in detail. as the starting point, given a question sentence, "in new york state, the longest period of daylight occurs during which month" and candidate answers, neunli converts the question and each return: si answer (say, "june") to a declarative hypothesis sentence h i , i.e., "in new york state, the longest period of daylight occurs during june". ## experiment results we list the test accuracy of baseline methods and neunli on two test sets in ## related work question answering systems that integrate deep learning methods have made great progress in recent years @xcite @xcite . many works first adopt learnable encoders for sentence representation like convolutional encoders @xcite , recurrent encoders @xcite and transformers @xcite . then an interaction layer is devised to calculate the semantic similarity, which is the main difference in many models. @xcite utilize a multi-layered perceptron to combine the cnn encoded representations. @xcite perform a soft-attention alignment to measure word similarity between the question and the answer. though neural networks-based models make great advances in qa, they are short of illustrating the step-by-step prediction derivation process, where the logic-based method is adept @xcite @xcite , which differs from the widely used attention mechanism @xcite . @xcite proposed a natural logic inference framework to utilize natural logic to conduct interpretable question answering and viewed the open-domain question answering as a textual entailment problem. our neunli is inspired by natural logic inference but can achieve better performance by modeling the contextual information during natural logic proving using two pre-trained language models and training the whole process in an end-to-end fashion. ## conclusion in this work, we explore the feasibility of combining natural logic with neural networks for interpretable question answering. we present an end-toend differentiable method for learning the parameters as well as the structure of natural logical rules, which is capable of considering the contextual information while conducting natural logic-based reasoning. experimental results on the regents science exam of the aristo dataset show that our proposed model could bring improvements over baseline methods.