Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing (2021)

Abstract: Interpretability methods like Integrated Gradient and LIME are popular choices for explaining natural language model predictions with relative word importance scores. These interpretations need to be robust for trustworthy NLP applications in high-stake areas like medicine or finance. Our paper demonstrates how interpretations can be manipulated by making simple word perturbations on an input text. Via a small portion of word-level swaps, these adversarial perturbations aim to make the resulting text semantically and spatially similar to its seed input (therefore sharing similar interpretations). Simultaneously, the generated examples achieve the same prediction label as the seed yet are given a substantially different explanation by the interpretation methods. Our experiments generate fragile interpretations to attack two SOTA interpretation methods, across three popular Transformer models and on three different NLP datasets. We observe that the rank order correlation and top-K intersection score drops by over 20% when less than 10% of words are perturbed on average. Further, rank-order correlation keeps decreasing as more words get perturbed. Furthermore, we demonstrate that candidates generated from our method have good quality metrics.

## introduction recently, the use of natural language processing (nlp) has gained popularity in many securityrelevant tasks like fake news identification @xcite , authorship identification @xcite , toxic content detection @xcite , and for text-based automated privacy policy understanding @xcite . since interpretations of nlp predictions have become necessary building blocks of the sota deep nlp workflow, explanations have the potential to mislead human users into trusting a problematic interpretation. how-ever, there has been little analysis of the reliability and robustness of the explanation techniques, especially in high-stake settings, making their utility for critical applications unclear. research has shown that it is possible to disrupt and even manipulate interpretations in deep neural networks @xcite . the core idea in this literature centers around "fragile interpretations". @xcite defined that an interpretation is fragile if, for a given input, it is possible to generate perturbed input that achieves the same prediction label as the seed, yet is given a substantially different interpretation. fragility limits how much we can trust and learn from specific interpretations. an adversary for "fragile interpretations" could manipulate the input to draw attention away from relevant words or onto desired features. such input manipulation might be especially hard to detect because the actual labels have not changed. the literature includes two relevant groups: (1) to conduct model manipulations @xcite (details in sec. 2), and (2) to manipulate input samples @xcite . there has been little attention studying fragile interpretations via input manipulation in deep nlp. in this paper, we propose a simple algorithm "ex-plainfooler" that can make small adversarial perturbations on text inputs and demonstrate fragility of interpretations. we focus on optimizing two objective metrics -"l2 norm" or a proposed "delta lom", searching for small word-swap-based input manipulation to produce misleading interpretations and using semantic-oriented constraints to constrain the manipulations. figure 3 provides one example perturbation process. in summary, this paper provides the following contributions: • our input perturbation optimizes to increase the objective metric ("l2 norm" or "delta lom") that measures difference between the original and generated interpretations. the lom score captures the approximate center "position" of an interpretation and summarizes it to a scalar. ## related work interpretation methods: several interpretation methods have been proposed @xcite @xcite to calculate feature importance scores. two well-known methods in this area are integrated gradients (ig) @xcite and local interpretable model explanations (lime) @xcite . ig computes the scores by summing up the gradients along a path from the baseline to the input in a fixed number of steps and subsequently multiplied by the input itself. ig overcomes the saturation problem discussed in @xcite . on the other hand, lime is a completely black-box approach which explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction by training the model on perturbations generated around the input. fragile interpretations more recently, several works have focused on discussing the robustness of the said interpretations. studies have demonstrated that the interpretations generated are not robust and can be easily manipulated due to high dimensionality of networks. @xcite @xcite . multiple other works have tried to fix the problem by making interpretations robust @xcite . @xcite demonstrated that it is possible to introduce a new model over the original and alter gradients, to fool gradient-based interpretation methods. similarly, @xcite showed that black-box interpretation methods can also be fooled by allowing an adversarial classifier component. more recently, @xcite demonstrated empirically. that interpretability methods produce varying results on the same models but differently initialized. adversarial examples that fool nlp predictions: adversarial examples are inputs to a predictive machine learning model that are maliciously designed to fool the model predictions @xcite . multiple recent works have focused on applying the concept of adversarial examples on language inputs, including (1) attacks by character substitution @xcite @xcite ; (2) attacks by paraphrase @xcite ; (c) attacks by synonym substitution @xcite @xcite ; (d) attacks by word insertion or removal @xcite ; (e) attacks by limiting l p distance in a latent embedding space @xcite . our proposed algorithm is closely connected to the textfooler algorithm @xcite that searches for input perturbations to achieve mis-classification. differently, we optimize the "l2 norm" and "location of mass (lom)" objective directly on the input space for fragile explanations. by a discussion on the search strategy to optimize the objective metrics. subsequently, we discuss the interpretation method choices and end with the final candidates' selection procedure and pseudocode for our algorithm (algorithm1). we denote a text input as x and its word importance score vector (from a specific interpretation strategy on a particular nlp model) using notation i. ## conclusions literature sees a growing emphasis on interpretation techniques for explaining nlp model predictions. our work demonstrates a novel algorithm that generates perturbed inputs that provide evidence of fragile interpretations. we demonstrate the effectiveness of our approach across three different models, with one of them adversarially trained. our results show that it is possible to attack interpretations using simple input-level word swaps under certain constraints. we also demonstrate that both black and white-box interpretability approaches (lime and integrated gradient) show fragility in their derived interpretations. we hope our findings can pave lights for future studies on defending against problem of fragile interpretations in nlp.