E xpla G raphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021)

Abstract: Recent commonsense-reasoning tasks are typically discriminative in nature, where a model answers a multiple-choice question for a certain context. Discriminative tasks are limiting because they fail to adequately evaluate the model’s ability to reason and explain predictions with underlying commonsense knowledge. They also allow such models to use reasoning shortcuts and not be “right for the right reasons”. In this work, we present ExplaGraphs, a new generative and structured commonsense-reasoning task (and an associated dataset) of explanation graph generation for stance prediction. Specifically, given a belief and an argument, a model has to predict if the argument supports or counters the belief and also generate a commonsense-augmented graph that serves as non-trivial, complete, and unambiguous explanation for the predicted stance. We collect explanation graphs through a novel Create-Verify-And-Refine graph collection framework that improves the graph quality (up to 90%) via multiple rounds of verification and refinement. A significant 79% of our graphs contain external commonsense nodes with diverse structures and reasoning depths. Next, we propose a multi-level evaluation framework, consisting of automatic metrics and human evaluation, that check for the structural and semantic correctness of the generated graphs and their degree of match with ground-truth graphs. Finally, we present several structured, commonsense-augmented, and text generation models as strong starting points for this explanation graph generation task, and observe that there is a large gap with human performance, thereby encouraging future work for this new challenging task.

## introduction current state-of-the-art commonsense reasoning (csr) @xcite models are typi-1 explagraphs dataset will be publicly available at https://explagraphs.github.io/ . cally trained and evaluated on discriminative tasks, in which a model answers a multiple-choice question for a certain context @xcite @xcite . while pretrained language models perform well on these tasks @xcite , this setup limits the exploration and evaluation of a model's ability to reason and explain its predictions with relevant commonsense knowledge, thereby allowing models to solve tasks by using shortcuts, statistical biases or annotation artifacts @xcite . thus, we emphasize the importance of generative csr capability, in which a model has to compose and reveal the plausible commonsense knowledge required to solve a reasoning task. moreover, structured (e.g., graph-based) commonsense explanations, unlike unstructured natural language explanations, can more explicitly explain and evaluate the reasoning structures of the model by visually laying out the relevant context and commonsense knowledge edges, chains, and subgraphs. we propose explagraphs, a new generative and structured csr task (in english) of explana-tion graph generation for stance prediction on debate topics. specifically, our task requires a model to predict whether a certain argument supports or counters a belief, but correspondingly, also generate a commonsense explanation graph that explicitly lays out the reasoning process involved in inferring the predicted stance. consider fig. 1 showing two examples with belief, argument, and stance (support or counter) from our benchmarking dataset collected for this task. each example requires understanding social, cultural, or taxonomic commonsense knowledge about debate topics in order to infer the correct stance. the example on the left requires the knowledge that "children" are "still developing" and hence not capable of making an "important decision" like "cosmetic surgery" which has "consequences". given this knowledge, one can understand that the argument is counter to the belief. we represent this knowledge in the form of a commonsense explanation graph. graphs are efficient for representing explanations due to multiple reasons: (1) unlike a chain of facts @xcite @xcite , they can capture complex dependencies between facts, while also avoiding redundancy (e.g., "factory farming causes food and millions desire food" forms a "vstructure"), (2) unlike natural language explanations @xcite @xcite @xcite , it is easier to impose task-specific constraints on graphs (e.g., connectivity, acyclicity), that eventually help in better quality control during data collection (sec. 4) and designing structural validity metrics for model-evaluation (sec. 6), and (3) unlike semi-structured templates @xcite or extractive rationales @xcite @xcite , they allow for more flexibility and expressiveness. graphs can encode any reasoning structure and the nodes are not limited to just phrases from the context. as shown in fig. 1 , our explanations are connected directed acyclic graphs (dags), in which the nodes are either internal concepts (short phrases from the belief or argument), or external commonsense concepts (dashedred), essential for connecting the internal concepts in a way that the stance is inferred. the edges are labeled with commonsense relations chosen from a pre-defined set. while some edges might not necessarily be factual (e.g., "factory farming; has context; necessary"), note that such edges are essential in the context for composing an explanation that is indicative of the stance. semantically, our graphs are extended structured arguments, augmented with commonsense knowledge. we construct a benchmarking dataset for our task through a novel create-verify-and-refine graph collection framework. these graphs serve as nontrivial (not paraphrasing the belief as an edge), complete (explicitly connects the argument to the belief) and unambiguous (infers the target stance) explanations for the task (sec. 3). the graph quality is iteratively improved (up to 90%) through multiple verification and refinement rounds. 79% of our graphs contain external commonsense nodes, indicating that commonsense is a critical component of our task. explanation graph generation poses several syntactic and semantic challenges like predicting the internal nodes, generating the external concepts and predicting and labeling the edges in a way that leads to a connected dag. finally, the graph should unambiguously infer the target stance. we next present a multi-level evaluation framework for our task (sec. 6, fig. 4 ), consisting of diverse automatic metrics and human evaluation. the evaluation framework checks for stance and graph consistency along with the structural and semantic correctness of explanation graphs, both locally by evaluating the importance of each edge and globally by the graph's ability to reveal the target stance. furthermore, we propose graph-matching metrics like graph edit distance @xcite and ones that extend text-generation metrics for graphs (based on multiple test graphs in our dataset). lastly, as some strong initial baseline models for this new task, we propose a commonsense-augmented structured prediction model that predicts nodes and edges jointly and enforces global graph constraints (e.g., connectivity) through an integer linear program (ilp). we also experiment with bart @xcite and t5 @xcite based models, and show that all these models have difficulty in generating meaningful graph explanations for our challenging task, leaving a large gap between model and human performance. overall, our main contributions are: we encourage researchers to use our benchmark as a way to improve and explore structured commonsense reasoning capabilities of models. commonsense reasoning benchmarks: a large variety of csr tasks have been developed recently, including commonsense extraction @xcite , next situation prediction @xcite , cultural, social, and physical commonsense understanding @xcite @xcite @xcite , pronoun disambigua-tion @xcite , abductive commonsense reasoning @xcite and general commonsense @xcite @xcite . while there is an abundance of discriminative commonsense tasks, there are few recent works in generative commonsense tasks. e.g., @xcite generates unstructured commonsense text, and eigen @xcite considers event influence graph generation. instead, our work focuses on generating commonsense-augmented explanation graphs. ## explagraphs task definition we propose explagraphs, a new generative and structured commonsense-reasoning task, where given a belief about a topic and an argument, a model has to (1) infer the stance (support/counter), and (2) generate the corresponding commonsense explanation graph that explains the inferred stance (fig. 1 ). our primary focus in this work is on the second sub-task that requires generative commonsense reasoning. the explanation graph is a connected and directed acyclic graph, where each node is a concept (short english phrase). concepts are either internal (part of the belief or the argument) or external (part of neither but essential for filling in any knowledge gap between the belief and the argument). each directed edge connects two concepts and is labeled with one of the pre-defined commonsense relations. these relations are chosen based on conceptnet @xcite with three modifications -(1) removing some generic relations like "related to", (2) merging some relations that have similar meanings (e.g. "synonym of" and "similar to"), (3) adding a negated counterpart ("not desires") for every non-negated relation ("desires"), to enable easy construction of support and counter explanations and a balanced set between negated and non-negated relations (see appendix for full list). semantically, our explanation graphs are commonsense-augmented structured arguments that explicitly support or counter the belief. all subjective claims in the graph are assumed to be true for inferring the stance. an explanation graph is correct if it is both structurally and semantically correct. structural correctness of graphs: in order to ensure the structural validity of an explanation graph, we define certain constraints on the graph which not only ensure better quality control during our data collection (sec. 4) but also simplify the evaluation (sec. 6), given the open-ended nature of our task. note that most of these constraints are only possible to impose because of the explicit graphical structure of these explanations. semantic correctness of graphs: we define the semantic correctness of explanation graphs as follows. first, all facts in the graph, individually, should be semantically coherent. second, the graph should be non-trivial, complete and unambiguous. we call a graph non-trivial if it uses the argument to arrive at the belief and does not use fact(s) which are mere paraphrases of the belief. e.g., for a belief "factory farming should be banned", if the explanation graph contains facts like "(factory farming; desires; banned)", then it is only paraphrasing the belief to explain why the belief holds, hence making the graph incorrect. instead, it should be augmenting the argument with commonsense knowledge like our graph in fig. 1 . a complete graph is one which explicitly connects the argument to the belief and no other commonsense knowledge is needed to understand why it supports or counters the belief. e.g., in fig. 1 , the fact "(necessary; not desires; banned)" makes the explanation complete by explicitly connecting back to the belief. we call a graph unambiguous if it, as a whole, infers the target stance and only that stance. we revisit these definitions of structural and semantic correctness when evaluating the quality of human-written graphs (sec. 4.2) as well as model-generated graphs (sec. 6). ## dataset collection we collect explagraphs data in two stages via crowdsourcing on amazon mechanical turk (fig. 2 ). in stage 1 (left of fig. 2 ), we collect instances of belief, argument and their corresponding stance. in stage 2 (right of fig. 2 ), we collect the corresponding commonsense explanation graph for each (belief, argument, stance) sample. ## dataset analysis explagraphs consists of a total of 3166 samples (see ## evaluation metrics explanation graphs can be represented in multiple correct ways with varying levels of specificity and different graphical structures. a single concept can also be paraphrased differently. thus, we design a 3-level evaluation pipeline (see fig. 4 ). level 1 -stance accuracy (sa): all models for our task predict both the stance label and the commonsense explanation graph. in level 1, we report the stance prediction accuracy which ensures that the explanation graph is consistent with the predicted stance. samples with a correctly predicted stance are then passed to the next levels that check for the quality of the generated explanation graphs. level 2 -structural correctness accuracy of graphs (stca): as per our task definition in sec. 3, for an explanation graph to be correct, it first has to be structurally correct. hence, we compute the fraction of structurally correct graphs (connected dags with at least three edges and at least two concepts from the belief and at least two from the argument). samples with correct stances and structurally correct graphs are then evaluated in level 3 for: (1) semantic correctness, (2) match with gt graphs, and (3) edge importance. level 3 -semantic correctness accuracy of graphs (seca): identifying semantic correctness of a graph requires following our human verification process discussed in sec. 4.2. a graph is semantically correct if all its edges are semantically coherent and given the belief, the unambiguously inferred stance from the graph matches the original stance. however, both these aspects are challenging because they require understanding the underlying semantics and reasoning through the graph. carrying this out by humans at a large scale is also expensive. thus, following previous works @xcite @xcite , we propose an automatic model-based metric that given a belief-graph pair, predicts the label between incorrect, support, and counter. specifically, we fine-tune roberta @xcite on the beliefs and corresponding human-verified graphs from our data collection phase. graphs are fed as concatenated edges to the model. since the space of incorrect graphs is potentially huge, we augment our training data with synthetically created incorrect graphs (by randomly adding, removing, or replacing edges) from already correct (support/counter) graphs. note that our automatic metric is not meant to replace human evaluation. thus, for completeness, we still perform human evaluation and show human-metric correlation for seca (sec. 8). level 3 -g-bertscore (g-bs): we also introduce a matching metric that quantifies the degree of match between the ground-truth and the predicted graphs. we call this g-bertscore, designed as an extension of a text generation metric, bertscore (zhang* et al., 2020) for graph-matching. we consider graphs as a set of edges and solve a matching problem that finds the best assignment between the edges in the gold graph and those in the predicted graph. each edge is treated as a sentence and the scoring function between a pair of gold and predicted edges is given by bertscore. 3 given the best assignment and the overall matching score, we compute precision, recall and report f1 as our g-bertscore metric. on the test set, we consider the best match across all ground-truth graphs. level 3 -graph edit distance (ged): as a more interpretable graph matching metric, we use graph edit distance @xcite to compute the distance between the predicted graph and the gold graph. formally, ged measures the number of edit operations (addition, deletion, and replacement of nodes and edges) for transforming the predicted graph to a graph isomorphic to the gold graph. the cost of each edit operation bart roberta (finetuned on conceptnet) factory farming ... millions necessary internal node prediction edge prediction (finetuned on conceptnet) ... graph constraints -ilp |r| cartesian @xmath0... banned is chosen to be 1. the ged for each sample is normalized between 0 and 1 by an appropriate normalizing constant (upper bound of ged). thus, the samples with either incorrect stances or structurally incorrect graphs will have a maximum normalized ged of 1 while samples whose graphs match exactly will have a score of 0. the overall ged is given by the average of the sample-wise geds. lower ged indicates that the predicted graphs match more closely with the gold graphs. ## models following prior work on explanation generation @xcite , we experiment with two broad families of models -(1) reasoning (first-graph-then-stance) models that first predict the explanation graph by conditioning on the belief and the argument. then it augments the belief and the argument with the generated graph to predict the stance, (2) rationalizing (first-stance-then-graph) models that first predict the stance, followed by generating graphs as post-hoc explanations. in both these types of models, the stance prediction happens through a fine-tuned roberta. for graph generation, we first propose a commonsense-augmented structured model (described next). we also experiment with state-of-the-art text generation models like bart and t5 that generate graphs as linearized strings. during training, edges in the graphs are ordered according to the depth-first-traversal (dfs) order of the nodes. see appendix for details on fine-tuning bart and t5 for graph generation. commonsense-augmented structured prediction model: next, as another baseline, we present a commonsense-augmented structured prediction model. as shown in fig. 5 , it has the following four modules: (a) internal nodes prediction: it identifies the concepts (nodes) from the belief or the argument. we pose this task as a sequence-tagging problem where given a sequence of tokens from the belief and argument, each token is classified into one of the three classes {b-n, i-n, o} denoting the beginning, inside and outside of a node respectively. we build this module on top of a pre-trained roberta @xcite by feeding in the concatenated belief and argument and having a standard 2-layer classifier at the top. (b) external commonsense nodes generation: we build this module separately by fine-tuning a pre-trained bart @xcite model that conditions on the concatenated belief and argument and generates a sequence of commonsense concepts. (c) edge prediction: we pose this as a multi-way classification problem in which given a pair of nodes, the module has to classify the edge into one relation (or no edge). this module is conditioned on the node prediction module to enable learning edges between the set of chosen nodes only and also for optimizing both modules jointly. specifically, given the set of node representations from the node module, we construct the edge representa-tions for all possible edges which are then passed to a 2-layer classifier for prediction. to augment our model with external commonsense, we first finetune roberta and the edge module on concept-net @xcite . (d) enforcing graph constraints: our final loss sums the cross-entropy losses from the node and edge module. following saha et al. (2020), during inference, we ensure connectivity and acyclicity in the explanation graph through an integer linear program. see appendix sec. c.3 for a more formal description of the model. 4 8 experiments and analysis in explanation impact (ra vs re): ra models predict the stance first without the graph, while the re models predict the stance conditioned on the generated graph. re models' drop in sa points to their overall limitations in generating helpful explanations. in fact, conditioning on such graphs makes the model less confident of its stance predictions. metrics' upper bound: while the stance accuracy (sa) is sufficiently high for all models, they obtain a significantly low semantic correctness accuracy (seca) for graphs (between 10-20%). to obtain an upper bound on our metrics (last row), we treat ground-truth graphs as predictions and find that they not only aid in stance prediction (sa increases from 87% to 91%) but also obtain a high 83% seca. given the large gap (>60%) between human and model performance, we hope our dataset will encourage future work on better model development for explanation graph generation. human-metric correlation for seca: while we develop an initial automatic metric for seca, it still is a challenging problem and hence human evaluation for the same is necessary. in order to show human-metric correlation for seca, we perform human evaluation (using the exact mechanism of human-written graph verification, discussed in sec. 4.2) of all structurally-correct generated graphs. encouragingly, we find that our modelbased metric (seca column) correlates well with humans (last column), with re-sp being the best model. the human verification labels match with the seca model's predictions 68% of the time. analysis of generated graphs: fig. 6 shows two randomly chosen graphs generated by re-sp containing external commonsense nodes like "positive effects", "good for society". while the first graph is correct, the second graph chooses the wrong relations for certain edges (in red), thus pointing to its lack of commonsense. overall, we find a large fraction of incorrect graphs contain incoherent facts or facts not adhering to human commonsense. ## discussion and future work we show the promise of explanation graphs by considering the task of stance prediction as a motivating use-case because it is representative of many sentence-pair inference tasks (consider the belief as the premise, argument as the hypothesis and the support/counter labels as entailment/contradiction). we believe that our definition of explanation graphs (sec. 3) is quite generic and should extend naturally to any nlu task, e.g., the internal nodes are concepts that are part of a context (context could mean premise-hypothesis for nli, passage for sentiment classification, passage-question for qa, etc), the external nodes refer to concepts that are not part of the context, the edges are semantic relations between concepts, and the dag-like constraints ensure the presence of explicit reasoning structures. although we choose a pre-defined set of relations for our task that can adequately represent most commonsense facts, the relations can be updated/adapted for a different task. given the potential of explanation graphs in improving the explainability of many reasoning tasks, we hope future work can further explore their applicability in different scenarios. ## conclusion we proposed explagraphs, a new generative and structured commonsense-reasoning task (and a benchmarking dataset) on explanation graph generation for stance prediction. additionally, we proposed automatic evaluation metrics and an initial structured model for explagraphs, demonstrating its difficulty in generating high-quality commonsense-augmented graphical explanations, and encouraging future work on better graph-based commonsense explanation generation.