Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering (2020)

Abstract: Despite the rapid progress in multihop question-answering (QA), models still have trouble explaining why an answer is correct, with limited explanation training data available to learn from. To address this, we introduce three explanation datasets in which explanations formed from corpus facts are annotated. Our first dataset, eQASC contains over 98K explanation annotations for the multihop question answering dataset QASC, and is the first that annotates multiple candidate explanations for each answer. The second dataset eQASC-perturbed is constructed by crowd-sourcing perturbations (while preserving their validity) of a subset of explanations in QASC, to test consistency and generalization of explanation prediction models. The third dataset eOBQA is constructed by adding explanation annotations to the OBQA dataset to test generalization of models trained on eQASC. We show that this data can be used to significantly improve explanation quality (+14% absolute F1 over a strong retrieval baseline) using a BERT-based classifier, but still behind the upper bound, offering a new challenge for future research. We also explore a delexicalized chain representation in which repeated noun phrases are replaced by variables, thus turning them into generalized reasoning chains (for example: “X is a Y” AND “Y has Z” IMPLIES “X has Z”). We find that generalized chains maintain performance while also being more robust to certain perturbations.

## introduction while neural systems have become remarkably adept at question answering (qa), e.g., @xcite , their ability to explain those answers remains limited. this creates a barrier for deploying qa systems in practical settings, and q: what can cause a forest fire? ## related work in the context of qa, there are multiple notions of explanation/justification, including showing an authoritative, answer-bearing sentence @xcite , a collection of text snippets supporting an answer @xcite , an attention map over a passage @xcite , a synthesized phrase connecting question and answer @xcite , or the syntactic pattern used to locate the answer @xcite . these methods are primarily designed for answers to "lookup" questions, to explain where and how an answer was found in a corpus. for questions requiring inference, the focus of this paper, an explanation is often taken as the chain of steps (typically sentences) leading to an answer. hotpotqa's support task goes partway towards this by asking for answer-supporting sentences (but not how they combine) @xcite . the r4c dataset takes this further, annotating how (and which) hotpotqa supporting sentences chain together @xcite . however, in hotpotqa and r4c, the decomposition (hence structure of the explanation) is evident in the question @xcite , simplifying the task. more recently, multihop datasets where the decomposition is not evident have appeared, e.g., @xcite , obqa @xcite , and qasc @xcite , posing more a realistic explanation challenge. however, explanation annotations are sparse (only qasc contains a single gold explanation per question), limiting their support for both training and evaluation of explanation systems, hence motivating this work. finally there are a few human-authored explanation datasets. e-snli @xcite adds crowdsourced explanations to snli entailment problems @xcite , and @xcite adds explanations for com-monsenseqa questions @xcite . this work differs from ours in two ways. first, the authored explanations are single-hop, directly linking a question to an answer. second, the datasets were primarily designed for (explanation) language generation, while our goal is to assemble explanations from an authoritative corpus so that they have credible provenance. our work is quite different from prior work focusing on textual entailment. our goal is not to decide if a sentence is entailed, but to identify a valid explanation for why. for example, a scitail @xcite model may predict that "metals conduct heat" entails "a spoon transmits energy", but not offer an explanation as to why. our work fills this gap by providing an explanation (e.g., "spoon is made of metal", "heat is energy" from a larger retrieved context). similarly, another entailmentbased dataset is fever @xcite , testing where a larger context entails a claim. however, the fever task requires finding a context sentence that simply paraphrases the claim, rather than a reasoned-based explanation from more general statements -the aim of this work. ## explanation datasets we now present our new datasets, first describing how we construct candidate chains for each qa pair, and then how they were annotated. ## learning to score chains our full approach to explaining an answer has two steps, namely candidate chain retrieval followed by chain scoring, to find the highest-ranked chain supporting an answer. for chain retrieval, we assume the same procedure described earlier to identify candidate chains. for chain scoring, we train a bert-based model to distinguish valid chains from invalid ones, using the training data collected in the eqasc dataset, as we now describe. we evaluate using all three collected datasets. we also evaluate two different ways of presenting the chain to the model to score (both train and test): (a) in its original form (with q+a flipped to a declarative sentence), (b) in a generalized form, where repeated noun phrases are variabilized. our interest is how well a model can perform, both to assess practical use and as a baseline for further improvement; and how the two different chain representations impact performance. ## experiments for training, we use the annotated chains in the train split of eqasc alongwith the 'gold' chains provided in the qasc dataset (qsc gold chains are always considered valid reasoning chains). we try two different ways of presenting chains to the model, namely the original and generalized chain representations (grcs), thus produce two models that we refer to as bert-chain and bert-grc respectively. in earlier experiments, we did not find using chains for negative answer options (which are all invalid chains) to be useful (see section 6.3), so we use chains for correct answer options only. we use allennlp @xcite toolkit to code our models. we test on all the three proposed datasets. since we are interested in finding explanations for the correct answer, we ignore the incorrect answer chains for the purpose of testing (they still accompany the dataset and can be used as additional training data since they are invalid reasoning chains by definition: section 6.3). for eqasc and eobqa, we evaluate in two ways: first, treating the task as classification, we measure f1 and auc-roc (below). second, treating the task as ranking the set of explanations for each answer, we measure p@1 and normalized discounted cumulative gain (ndmc) (also below). we use the trained model's probability of a chain being valid to rank the retrieved candidate chains for a given question and answer. ## summary explaining answers to multihop questions is important for understanding why an answer may be correct, but there is currently a dearth of suitable, annotated data. to address this, and promote progress in explanation, we contribute three new explanation datasets, including one with over 98k annotated reasoning chains -by far the largest repository of annotated, corpus-derived explanations to date. we also have shown this data can significantly improve explanation quality on both in-domain (qasc) and out-of-domain (obqa) tasks. finally, we have proposed and explored using a lightweight method to achieve a delexicalized representation of reasoning chains. while preserving explanation quality (despite removing details), this representation appears to be more robust to certain perturbations.