Explanation Regeneration via Multi-Hop ILP Inference over Knowledge Base (2020)

Abstract: Textgraphs 2020 Workshop organized a shared task on ‘Explanation Regeneration’ that required reconstructing gold explanations for elementary science questions. This work describes our submission to the task which is based on multiple components: a BERT baseline ranking, an Integer Linear Program (ILP) based re-scoring and a regression model for re-ranking the explanation facts. Our system achieved a Mean Average Precision score of 0.3659.

## introduction question answering (qa) has been a long standing challenge in the field of natural language processing with considerable recent focus on machine reading comprehension @xcite , complex question answering @xcite , open domain and commonsense question answering @xcite that require piecing together chunks of information in order to infer the correct answers -also known as multi-hop inferencing. there has been a rapid rise in the development of such qa datasets as well as deep learning-based models for solving them but most of these models are ineffective in explaining why a model chooses a particular answer for a question. explanation regeneration is the task of generating simple sentence explanations for complex scientific phenomena related question answers. it is a multi-hop inference task wherein the gold explanation is formed by chaining together individual facts from a knowledge base (kb) ordered such that they form correct reasoning behind the answer to a question. it can be posed both as a ranking problem where we iteratively rank relevant facts with respect to a question and as a graph traversal problem, where we "hop" from some starting fact to other related facts until we have enough facts to infer the answer. to regenerate the correct chain of explanation facts, we have developed a system that can rank and score the facts in a kb by inferring over the graph of question, its correct answer and the relevant kb facts. integer linear programming has been used as an effective approach in answering questions over semistructured tables @xcite , tuples from kb @xcite , and semantic abstractions of paragraphs of text @xcite . we use this approach to further rank and score relevant facts forming an explanation for a question answer pair. to find the relevant kb facts, we first rank each fact in the kb with respect to its bidirectional contextual representation with the question and its correct answer using a bert baseline @xcite . we then choose a set of top-k ranked facts forming a chain and create an integer linear program with variables and constraints consisting of a graph of nodes and edges created from constituents of question, answer and the fact chain. the ilp maximizes the graph that has maximum alignment between the edges for re-scoring each fact in the chain. the fact scores from the ilp model are further combined with the help of a regression model to finally re-rank each fact in the top-k set of explanation facts. the paper is organized as follows: section 2 provides task description, followed by details of our system components in section 3, detailed evaluation results in section 4 and conclusion in section 5. ## task description the task of explanation regeneration(jansen and ustalov, 2020) is based on multiple choice elementary and middle school science exam question answers taken from arc @xcite dataset and supplemented with a set of curated explanation facts that explain the correct answer to each question. the explanation facts are atomic sentences from a kb and link a question and its correct answer in a chainlike manner that can be understood easily by a 5-year-old. to answer a question correctly, appropriate explanation facts must be retrieved from the kb and chained in the correct order. ## system description the complete pipelined system in shown in figure 2 that takes the input dataset, generates a baseline ranking of facts from which the top-k are given as input to the ilp program which generates scores for them, followed by generation of combined scores by the regression model and a final sorted list of scores for each question answer pair. ## evaluation results for evaluation, we consider following baseline models and compare their mean average precision (map) scores on test data with our system: ## conclusion and future work explanation regeneration is a multi-hop inferencing task that requires chaining together kb facts which can form an explanation for the correct answer to a question. ranking each fact individually in the kb is not enough to solve such a task and requires deep probing into finding links between explanation facts, question and its answer. to this end, we devised an ilp that can infer such links and score and rank a chain of explanation facts with the regression model. with the current system in place, we plan to generate first order logic based semantic graph structures of question, answer and explanation facts and use the alignment scores between them instead of direct phrasal entailment scores between constituents in the ilp model. we are also exploring leveraging information from more semantic resources like con-ceptnet @xcite and framenet @xcite instead of only wordnet based entailment scoring and ilp parameter tuning for improving regression ranking scores.