Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2020)

Abstract: To increase trust in artificial intelligence systems, a promising research direction consists of designing neural models capable of generating natural language explanations for their predictions. In this work, we show that such models are nonetheless prone to generating mutually inconsistent explanations, such as ”Because there is a dog in the image.” and ”Because there is no dog in the [same] image.”, exposing flaws in either the decision-making process of the model or in the generation of the explanations. We introduce a simple yet effective adversarial framework for sanity checking models against the generation of inconsistent natural language explanations. Moreover, as part of the framework, we address the problem of adversarial attacks with full target sequences, a scenario that was not previously addressed in sequence-to-sequence attacks. Finally, we apply our framework on a state-of-the-art neural natural language inference model that provides natural language explanations for its predictions. Our framework shows that this model is capable of generating a significant number of inconsistent explanations.

## introduction in order to explain the predictions produced by accurate yet black-box neural models, a growing number of works propose extending these models with natural language explanation generation modules, thus obtaining models that explain themselves in human language @xcite @xcite @xcite . in this work, we first draw attention to the fact that such models, while appealing, are nonetheless prone to generating inconsistent explanations. we define two explanations to be inconsistent if they provide contradictory arguments about the instances and predictions that they aim to explain. for example, consider a visual question answering (vqa) task @xcite and two instances where the image is the same but the questions are different, say "is there an animal in the image?" and "can you see a husky in the image?". if for the first instance a model predicts "yes." and generates the explanation "because there is a dog in the image.", while for the second instance the same model predicts "no." and generates the explanation "because there is no dog in the image.", then the model is producing inconsistent explanations. inconsistent explanations reveal at least one of the following undesired behaviors: (i) at least one of the explanations is not faithfully describing the decision mechanism of the model, or (ii) the model relied on a faulty decision mechanism for at least one of the instances. note that, for a pair of inconsistent explanations, further investigation would be needed to conclude which of these two behaviors is the actual one (and might vary for each instance). indeed, a pair of inconsistent explanations does not necessarily imply at least one unfaithful explanation. in our previous example, if the image contains a dog, it is possible that the model identifies the dog when it processes the image together with the first question, and that the model does not identify the dog when it processes the image together with the second question, hence both explanations would faithfully reflect the decision mechanism of the model even if they are inconsistent. similarly, a pair of inconsistent explanations does not necessarily imply that the model relies on a faulty decision mechanism, because the explanations may not faithfully describe the decision mechanism of the model. we here will not investigate the problem of identifying which of the two undesired behaviors is true for a pair of inconsistent explanations. as part of our framework, we address the problem of adversarial attacks with full target sequences, a scenario that has not been previously addressed in sequence-to-sequence attacks, and which can be useful for other areas, such as dialog systems. finally, we apply our framework on a state-of-theart neural natural language inference model that generates natural language explanations for its decisions @xcite . we show that this model can generate a significant number of inconsistent explanations. ## method given a model m that can jointly produce predictions and natural language explanations, we propose a framework that, for any given instance x, attempts to generate new instances for which the model produces explanations that are inconsistent with the explanation produced for x; we refer to the latter as e m (x). we approach the problem in two high-level steps. given an instance x, (a) we create a list of explanations that are inconsistent with the explanation generated by the model on x, and (b) given an inconsistent explanation from the list created in a, we find an input that causes the model to generate this precise inconsistent explanation. setup. our setup has three desired properties that make it different from commonly researched adversarial settings in natural language processing: to our knowledge, this work is the first to tackle this problem setting, especially due to the challenging requirement of generating a full target sequence -see section 4 for comparison with existing works. context-dependent inconsistencies. in certain tasks, instances consist of a context (such as an image or a paragraph), and some assessment to be made about the context (such as a question or a hypothesis). since explanations may refer (sometimes implicitly) to the context, the assessment of whether two explanations are inconsistent may also depend on it. for example, in vqa, the inconsistency of the two explanations "because there is a dog in the image." and "because there is no dog in the image." depends on the image. however, if the image is the same, the two explanations are inconsistent regardless of which questions were asked on that image. for such a reason, given an instance x, we differentiate between parts of the instance that will remain fixed in our method (referred to as context parts and denoted as x c ) and parts of the instance that our method will vary in order to obtain inconsistencies (referred to as variable parts and denoted as x v ). hence, @xmath0, x v ). in our vqa example, x c is the image, and x v is the question. stand-alone inconsistencies. furthermore, we note that there are cases for which explanations are inconsistent regardless of the input. for example, explanations formed purely of background knowledge such as "a woman is a person." and "a woman is not a person." 1 are always inconsistent (and sometimes outrageous), regardless of the instances that lead to them. for these cases, our method can treat the whole input as variable, i.e., x @xmath1. steps. our adversarial framework consists of the following steps: while this step requires a degree of specific adjustment to the task at hand, we consider it a small price to pay to ensure that the deployed system is coherent. also, note that this step can eventually be automated, for example, by training a neural network to generate task-specific inconsistencies after crowd-sourcing a dataset of inconsistent explanations for a task at hand -we leave this as future work. finally, to execute step (2d), our framework currently checks for an exact string match between a reverse explanation and any of the inconsistent explanations created at step (2a). alternatively, one can train a model to identify if a pair of explanations forms an inconsistency, which we also leave as future work. ## experiments we consider the task of natural language inference (nli) @xcite , which consists of detecting whether a pair of sentences, called premise and hypothesis, are in a relation of: entailment, if the premise entails the hypothesis; contradiction, if the premise contradicts the hypothesis; or neutral, if neither entailment nor contradiction holds. for example, a pair with premise "two doctors perform surgery on patient." and hypothesis "two doctors are performing surgery on a man." constitutes a neutral pair. the snli corpus @xcite of ∼570k such human-written instances enabled a plethora of works on this task @xcite @xcite . recently, @xcite augmented snli with crowd-sourced free-form explanations of the ground-truth label, called e-snli. an explanation from e-snli for the neutral pair above is "not every patient is a man.". their best model for generating explanations, called explainthenpredictattention (hereafter called etpa), is a sequence-to-sequence attention model that uses two bidirectional lstm networks @xcite for encoding the premise and hypothesis, and an lstm decoder for generating the explanation while separately attending over the tokens of the premise and hypothesis. subsequently, they predict the label solely based on the explanation via a separately trained network, which maps an explanation to a label. we show that our framework is able to make etpa foot_0 generate a significant number of inconsistent explanations. we highlight that our final goal is not a label attack, even if, for this particular model in which the label is predicted solely from the explanation, we implicitly also have a label attack with high probability. 3 in our experiments, we set x c as the premise (as this represents the given context in this task) and x v as the hypothesis. however, note that due to the nature of snli for which decisions are based mostly on commonsense knowledge, the explanations are most of the time independent of the premise, such as "a dog is an animal." -hence, it would be possible to also reverse the premise and not just the hypothesis; we leave this as future work. for the revexpl model, we use the same neural architecture and hyperparameters used by @xcite for etpa. revexpl takes as input a premise-explanation pair, and produce a hypothesis. our trained revexpl model is able to reconstruct exactly the same (according to string matching) hypothesis with 32.78% test accuracy. creating i e . to execute step (2a), we employ negation and swapping explanations. for negation, we simply remove the tokens "not" and "n't" if they are present. if these tokens appear more than once in an explanation, we create multiple inconsistencies by removing only one occurrence at a time. we do not attempt to add negation tokens, as this may result in grammatically incorrect sentences. for swapping explanations, we note that the explanations in e-snli largely follow a set of label-specific templates. this is a natural consequence of the task and the snli dataset and not a requirement in the collection of the e-snli. for example, annotators often used "one cannot x and y simultaneously." to explain a contradiction, "just because x, doesn't mean y." for neutral, or "x implies y." for entailment. since any two labels are mutually exclusive, transforming an explanation from one template to a template of another label should automatically create an inconsistency. for example, for the explanation of the contradiction "one cannot eat and sleep simultaneously.", we match x to "eat" and y to "sleep", and create the inconsistent explanation "eat implies sleep." using the entailment template "x implies y.". thus, for each label, we created a list of the most used templates that we manually identified among e-snli, which can be found in appendix a. a running example of creating inconsistent explanations by swapping is given in appendix a.1. if there is no negation and no template match, we discarded the instance. in our experiments, we only discarded 2.6% of the snli test set. we note that this procedure may result in grammatically or semantically incorrect inconsistent explanations. however, as we will see below, our revexpl performed well in generating correct and relevant reverse hypotheses even when its input explanations were not correct. this is not surprising, because revexpl has been trained to output ground-truth hypotheses. the rest of the steps follow as described in ( 2b ) -(2d). results and discussion. we identified a total of 1044 pairs of inconsistent explanations starting from the snli test set, which contains 9824 instances. first, we noticed that there are, on average, 1.93 ± 1.77 distinct reverse hypotheses giving rise to a pair of inconsistent explanation. since the hypotheses are distinct, each of these instances is a separate valid adversarial inputs. however, if one is strictly interested in the number of distinct pairs of inconsistent explanations, then, after eliminating duplications, we obtain 540 pairs of such inconsistencies. secondly, since the generation of natural language is always best evaluated by humans, we manually annotated 100 random distinct pairs. we found that 82% of the reverse hypotheses form realistic instances together with the premise. we also found that the majority of the unrealistic instances are due to a repetition of a token in the hypothesis. for example, "a kid is riding a helmet with a helmet on training." is a generated reverse hypothesis which is just one token away from a perfectly valid hypothesis. given our estimation of 82% to be inconsistencies caused by realistic reverse hypotheses, we obtained a total of ∼443 distinct pairs of inconsistent explanations. while this means that our procedure only has a success rate of ∼4.51%, it is nonetheless alarming that this very simple and under-optimized adversarial framework detects a significant number of inconsistencies on a model trained on ∼570k examples. in manual scanning. we were curious to what extent one can find inconsistencies via a brute-force manual scanning. we performed three such experiments, with no success. on the contrary, we noticed a good level of robustness against inconsistencies when scanning through the generic adversarial hypotheses introduced by @xcite . the details are in appendix c. ## related work an increasing amount of work focuses on providing natural language, free-form explanations @xcite @xcite as a more comprehensive and user-friendly alternative to other forms of explainability, such as feature-based explanations @xcite . in this work, we bring awareness to the risk of generating inconsistent explanations. @xcite identify the risk of mentioning attributes from a strong class prior without any evidence being present in the input. ## summary and outlook we drew attention that models generating natural language explanations are prone to producing inconsistent explanations. this concern is general and can have a large practical impact. for example, users would likely not accept a self-driving car if its explanation module is prone to state that "the car accelerates because there are people crossing the intersection.". we introduced a generic framework for identifying such inconsistencies and showed that the best existing model on e-snli can generate a significant number of inconsistencies. future work will focus on developing more advanced procedures for detecting inconsistencies, and on building robust models that do not generate inconsistencies.