E xplanations for C ommonsense QA : N ew D ataset and M odels (2021)

Abstract: CommonsenseQA (CQA) (Talmor et al., 2019) dataset was recently released to advance the research on common-sense question answering (QA) task. Whereas the prior work has mostly focused on proposing QA models for this dataset, our aim is to retrieve as well as generate explanation for a given (question, correct answer choice, incorrect answer choices) tuple from this dataset. Our explanation definition is based on certain desiderata, and translates an explanation into a set of positive and negative common-sense properties (aka facts) which not only explain the correct answer choice but also refute the incorrect ones. We human-annotate a first-of-its-kind dataset (called ECQA) of positive and negative properties, as well as free-flow explanations, for 11K QA pairs taken from the CQA dataset. We propose a latent representation based property retrieval model as well as a GPT-2 based property generation model with a novel two step fine-tuning procedure. We also propose a free-flow explanation generation model. Extensive experiments show that our retrieval model beats BM25 baseline by a relative gain of 100% in F_1 score, property generation model achieves a respectable F_1 score of 36.4, and free-flow generation model achieves a similarity score of 61.9, where last two scores are based on a human correlated semantic similarity metric.

## introduction the field of automated question answering (qa) has witnessed a rapid progress in the past few years, sometimes beating even human performance @xcite . the reasons behind this trend include (i) emergence of large-sized qa datasets such as squad @xcite , hotpotqa @xcite , commonsenseqa @xcite , naturalqa @xcite , etc., and (ii) emergence of powerful, large scale, pre- where is a frisbee in play likely to be? answer choices: outside park roof tree air our explanation: positives properties 1) a frisbee is a concave plastic disc designed for skimming through the air as an outdoor game. negative properties 1) a frisbee can be outside anytime, even while not in play. ## related work bulk of the recent literature on automated qa is focused on either (i) proposing a new kind of dataset @xcite @xcite @xcite @xcite @xcite , or (ii) proposing a model with improved answer accuracy @xcite @xcite . as far as explanation in qa is concerned, we can either (i) explain the model's predicted answer, or (ii) explain the given gold answer without worrying about the model. for certain qa tasks (e.g. kbqa, mathqa, vqa), former explanation task is more meaningful. for other qa tasks (e.g. common-sense qa, scienceqa), the later form of explanation may be more meaningful. in both, one of the key challenge is to ground the definition of explanation. knowledge-base qa task @xcite ) requires the qa model to output a logical query (e.g. sparql or sql) which is then executed over the underlying kb to get the answer. this logical query itself serves as an explanation. the mathqa task @xcite requires the model to output a theorem-like proof, program, or algebraic construct which is executed to get the answer. again, such a theorem serves as an explanation. for scienceqa task, an expla- ## explanations for commonsenseqa the broad idea behind explaining common-sense qa is to capture how humans would justify if a qa pair is presented to them. however, grounding a precise definition for this human justification is still hard due to subjectivity @xcite . furthermore, depending on the type of reasoning involved in the qa task, form and shape of an explanation may vary. though, it is hard to give a single definition of the explanation for qa pairs coming from the cqa dataset, we believe one can still approach this by means of putting forward desiderata or desired characteristics of a well-formed explanation: comprehensive: any information or reasoning, which is necessary to explain the answer should be present. this requires writing common-sense facts that are not present in the question but are essential for explanation. refutation complete: while it should explain why an answer choice is correct, it should also explain why rest of the choices are incorrect or not best suited as answer. minimal: it should not contain any irrelevant or redundant information, especially the ones which are already present in the question. coherent: all the facts and statements should be written in a coherent and free-flow form to get a meaningful and natural explanation. ## ecqa dataset we partnered with a private firm to crowdsource the annotations in property set (s) format for the cqa dataset. the firm utilized their in-house annotation and quality control teams for this purpose. for each question in the cqa dataset, an annotator was shown the question, its target concept (as given in cqa), all five answer choices, and the correct answer choice. as described earlier, the annotators were then asked to write the following: a set s + of positive properties, another set s -of negative properties and a free-flowing english explanation using the facts encapsulated in sets s + and s -. each question in the cqa dataset comes with a label called target concept. we sorted all the questions according to their target concepts and provided questions of the same target concept to a single annotator. this prevented from conflicting statements appearing in positive and negative properties, and also helped speed up the annotation. an outcome of this exercise is shown in while it is difficult to guarantee that annotated property set is comprehensive, we tried to ensure it by asking annotators writing at least one property for each answer choice. we also asked them to write simple sentences by breaking down the complex sentences into two or more so that it helps in maintaining minimality. for the comprehensiveness and minimality of the final free-flow explanation, we explicitly asked them to include everything that appear in properties and avoid introducing anything from question and answer choices. the dataset quality at the ground level was ensured by a separate team of the partner firm, and random checks were performed by the authors as well. ## explanation retrieval this section describes our proposed explanation retriever (xr) system to retrieve s + and s -property sets from a given property corpus for a given question. xr consists of two modules -(i) property ranker, and (ii) property selector. the experimentation code and trained models for this and the following section are publicly released. foot_5 ## explanation generation in this section we will describe our proposed gpt-2 @xcite xgp to generate common-sense properties, and (ii) xgf to generate the free-flow explanations across all the answer choices. in all our experiments, we use random sampling to generate the output tokens using gpt-2 and report average numbers over 3 different runs. ## conclusion and future work we have presented desiderata of what constitutes an explanation in the case of common-sense qa. based on it, we generated a human-annotated explanation dataset ecqa for commonsenseqa. we have also proposed models to retrieve and generate common-sense facts required to justify the answer choice. we have publicly released our crowdsourced ecqa dataset and code/models. in future work, we plan to explore directions to design rlbased schemes for joint training of property ranker and property selector components in the xr system and joint training of xgp and xgf-ii to generate free-flow explanation. another direction is to improve the accuracy and interpretability of the existing models for commonsenseqa using the ecqa dataset.