Gradient-based Analysis of NLP Models is Manipulable (2020)

Abstract: Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, the fact that they directly reflect the model internals. In this paper, however, we demonstrate that the gradients of a model are easily manipulable, and thus bring into question the reliability of gradient-based analyses. In particular, we merge the layers of a target model with a Facade Model that overwhelms the gradients without affecting the predictions. This Facade Model can be trained to have gradients that are misleading and irrelevant to the task, such as focusing only on the stop words in the input. On a variety of NLP tasks (sentiment analysis, NLI, and QA), we show that the merged model effectively fools different analysis tools: saliency maps differ significantly from the original model’s, input reduction keeps more irrelevant input tokens, and adversarial perturbations identify unimportant tokens as being highly important.

## introduction it is becoming increasingly important to understand the reasoning behind the predictions of nlp models. post-hoc explanation techniques are useful for such insights, for example, to evaluate whether a model is doing the "right thing" before deployment @xcite , to increase human trust into black box systems @xcite , and to help diagnose model biases @xcite . recent work, however, has shown that explanation techniques can be unstable and, more importantly, can be manipulated to hide the actual reasoning of the * first two authors contributed equally. a solid examination of the male midlife crisis. ## gradient-based model analysis in this section, we introduce notation and provide an overview of gradient-based analysis methods. ## manipulating model gradients in this section, we describe how to modify neural nlp models in order to manipulate the results of gradient-based analysis techniques. ## experiment setup in this section, we describe the tasks, the types of facade models, and the original models that we use in our experiments (source code is available at http://ucinlp.github.io/facade ). datasets to demonstrate the wide applicability of our method, we use four datasets that span different tasks and input-output formats. three of the datasets are selected from the popular tasks of sentiment analysis (binary stanford sentiment treebank @xcite , natural language inference @xcite , and question answering @xcite . we select sentiment analysis and question answering because they are widely used in practice, their models are highly accurate @xcite , and they have been used in past interpretability work @xcite @xcite . we select nli because it is challenging and one where models often learn undesirable "shortcuts" @xcite . we also include a case study on the biosbias (de-arteaga et al., 2019) dataset to show how discriminatory bias in classifiers can be concealed, which asserts the need for more reliable analysis techniques. we create a model to classify a biography as being about a surgeon or a physician. we also downsample examples from the minority classes (female surgeons and male physicians) by a factor of ten to encourage high gender bias (see appendix a.4 for further details). ## results in this section, we evaluate the ability of our approach to manipulate popular gradient-based analysis methods. we focus on the techniques present in allennlp interpret @xcite as described in section 2.2. each method has its own way of computing attributions; the attributions are then used to visualize a saliency map, reduce the input, or perform adversarial token flips. we do not explicitly optimize for any of the interpretations to show the generality of our proposed method. ## discussion downsides of an adversarial approach our proposed approach provides a mechanism for an adversary to hide the biases of their model (at least from gradient-based analyses). the goal of our work is not to aid malicious actors. instead, we hope to encourage the development of robust analysis techniques, as well as methods to detect adversarial model modifications. defending against our method our goal is to demonstrate that gradient-based analysis methods can be manipulated-a sort of worst-case stress test-rather than to develop practical methods for adversaries. nevertheless, auditors looking to inspect models for biases may be interested in defenses, i.e., ways to detect or remove our gradient manipulation. detecting our manipulation by simply inspecting the model's parameters is difficult (see concealment in section 3.3). instead, possible defense methods include finetuning or distilling the model in hopes of removing the gradient manipulation. unfortunately, doing so would change the underlying model. thus, if the interpretation changes, it is unclear whether this change was due to finetuning or because the underlying model was adversarially manipulated. we leave a further investigation of defenses to future work. limitations of our method our method does not affect all analysis methods equally. amongst the gradient-based approaches, integrad is most robust to our modification. furthermore, nongradient-based approaches, e.g., black-box analysis using lime @xcite , anchors @xcite , and shap (lundberg and lee, 2017), will be unaffected by misleading gradients. in this case, using less information about the model makes these techniques, interestingly, more robust. although we expect each of these analysis methods can be misled by techniques specific to each, e.g., @xcite fool lime/shap and our regularization is effective against gradient-based methods, it is unclear whether these strategies can be combined, i.e. a single model that can fool all analysis techniques. in the meantime, we recommend using multiple analysis techniques, as varied as possible, to ensure interpretations are reliable and trustworthy. ## conclusions gradient-based analysis is ubiquitous in natural language processing: they are simple, model-agnostic, and closely approximate the model behavior. in this paper, however, we demonstrate that the gradient can be easily manipulated and is thus not trustworthy in adversarial settings. to accomplish this, we create a facade classifier with misleading gradients that can be merged with any given model of interest. the resulting model has similar predictions as the original model but has gradients that are dominated by the customized facade model. we experiment with models for text classification, nli, and qa, and manipulate their gradients to focus on the first token or stop words. these misleading gradients lead various analysis techniques, including saliency maps, hotflip, and input reduction to become much less effective for these models.