F OOLING E XPLANATIONS IN T EXT C LASSIFIERS
A BSTRACT
State-of-the-art text classiﬁcation models are becoming increasingly reliant on deep neural networks (DNNs).
Due to their black-box nature, faithful and robust explanation methods need to accompany classiﬁers for deployment in reallife scenarios.
However, it has been shown in vision applications that explanation methods are susceptible to local, imperceptible perturbations that can signiﬁcantly alter the explanations without changing the predicted classes.
We show here that the existence of such perturbations extends to text classiﬁers as well.
Speciﬁcally, we introduce T EXT E XPLANATION F OOLER (TEF), a novel explanation attack algorithm that alters text input samples imperceptibly so that the outcome of widely-used explanation methods changes considerably while leaving classiﬁer predictions unchanged.
We evaluate the performance of the attribution robustness estimation performance in TEF on ﬁve sequence classiﬁcation datasets, utilizing three DNN architectures and three transformer architectures for each dataset.
TEF can signiﬁcantly decrease the correlation between unchanged and perturbed input attributions, which shows that all models and explanation methods are susceptible to TEF perturbations.
Moreover, we evaluate how the perturbations transfer to other model architectures and attribution methods, and show that TEF perturbations are also effective in scenarios where the target model and explanation method are unknown.
Finally, we introduce a semi-universal attack that is able to compute fast, computationally light perturbations with no knowledge of the attacked classiﬁer nor explanation method.
Overall, our work shows that explanations in text classiﬁers are very fragile and users need to carefully address their robustness before relying on them in critical applications.
I NTRODUCTION
Deep neural networks (DNNs) have undoubtedly become the state-of-the-art architectures for many existing machine learning tasks (Choi et al., 2016).
Yet, their black-box nature has raised the need for developing methods to mitigate the lack of interpretability caused by their increased complexity (Simonyan et al., 2013; Zeiler & Fergus, 2014; Hendricks et al., 2018;
Bahdanau et al., 2014).
These methods give intuitive, easily understandable explanations that do not require signiﬁcant domain knowledge.
This is especially desired in safety-critical scenarios, such as healthcare, where the users of such DNNs - the medical professionals for instance - need to understand the decision process and reasoning behind it.
However, they have been shown to lack local robustness towards carefully crafted, imperceptible perturbations in the input (Ghorbani et al., 2019).
While resulting in the same predictions, these altered inputs yield signiﬁcantly different explanations and attributions maps (Figure 1).
Interpretation methods fragile towards small input perturbations not only fail to provide faithful explanations, a desiderata commonly required in explainable AI (Jacovi & Goldberg, 2020), but also damages user trust in DNNs and prevents them from being deployed on high-stakes, safety-
Original sample
TEF sample
PCC [CLS] romanians pitch rumsfeld on base location — mihail kogalniceanu air base , romania - to entice the us military to make a home here , what better symbolic appeal could the romanian government make than to rename a street here quot;george washington boulevard ?
[SEP]
{+F ( s, World ) = 
+}
.[CLS]
romanians pitch clinton on base places — mihail kogalniceanu air base , rumania - to entice the us military to make a home here , what better symbolic appeal could the romanian government make than to rename a street here quot;george washington boulevard ?
[SEP] F ( s, World ) = 0 .-0.
[CLS] forgettable horror – more gory than psychological – with a highly satisfying quotient of friday - night excitement and milla power .
[SEP]
[Equation]
[CLS] forgettable horror – more gory than psychological – with a highly satisfying quotient of friday - night arousal and milla wattage .
[SEP] F ( s , Pos . )
= 0 .0.
critical applications, such as in healthcare (Adadi & Berrada, 2020).
The previously described phenomenon has been widely studied in the image domain by Etmann et al.
; Moosavi-Dezfooli et al. (2019b) or Ivankay et al.
However, in discrete-input domains like text, there has been limited progress.
This is especially problematic given the increased reliance on and fragility of attention mechanisms as inherently explainable methods, as stated in
Ghaeini et al. Therefore, we summarize our contributions as follows: • We provide a novel baseline black-box adversarial attack, T EXT E XPLANATION F OOLER (TEF) to estimate the local robustness of explanations in text classiﬁcation problems • We evaluate attribution robustness on widely used, state-of-the-art text datasets and model architectures, showing that explanation methods’ output can be signiﬁcantly altered with our attack • We provide insight into the transfer capability of TEF on different models and explanation methods as well as introduce semi-universal adversarial perturbations to alter explanations without requiring access to the model at attack-time
P RELIMINARIES 2. R ELATED W ORK
Adversarial attacks that alter the inference outcomes in DNNs have been widely studied both in the image (Goodfellow et al., 2014; Carlini & Wagner, 2017; Moosavi-Dezfooli et al., 2016;
Modas et al., 2019) and text domain (Ebrahimi et al., 2017; Sun et al., 2020; Jin et al., 2019;
Yang et al., 2020).
Methods to alleviate the networks susceptibility to such attacks have also been proposed, including the works of Madry et al.
(2017); Moosavi-Dezfooli et al.
(2019a)
; Buckman et al.
(2018) or Cisse et al.
However, it has recently been shown by authors Ghorbani et al.
(2019) that, in addition to DNN predictions, widely-used explanation methods also lack robustness to targeted, imperceptible alterations of the input.
These attacks change the outcomes of such explanation methods signiﬁcantly, while predictions of the DNNs are unaltered.
This violates the Prediction Assumption of faithful explanations and crucially degrades user trust in such explanation methods, as significantly different interpretations are provided for similar inputs and outputs (Jacovi & Goldberg, 2020).
The aforementioned phenomenon of fragile explanations has mostly been investigated in the image domain (Etmann et al., 2019;
Singh et al., 2019; Chen et al., 2019; Ivankay et al., 2020), with less focus on discrete input spaces like text.
However, faithful and robust interpretations are arguably equally important in the discrete text domain, for instance in electronic health record classiﬁcation (Girardi et al., 2018) or precision medicine (Binder et al., 2021), where critical decisions often need to be based on DNN explanations.
The work of La Malfa et al.
(2021) constructs inherently robust explanations for NLP models, however only towards perturbations in the embedding space, not the input space that adversaries can operate on.
Moreover, they do not give a method to evaluate robustness of already existing explanation algorithms.
The authors of Feng et al.
(2018) show that interpretation methods in NLP lack completeness (Sundararajan et al., 2017) by removing words deemed least important by explanation methods.
Moreover, attention mechanisms (Bahdanau et al., 2014) have been increasingly relied on as inherently interpretable systems.
However recent work has questioned their faithfulness and plausibility (Serrano & Smith, 2019; Jain & Wallace, 2019; Wiegreffe & Pinter, 2019) by proving that they often do not highlight input components that are most important to a DNN decision.
We, paralleled by the very recent work of Sinha et al. (2021), are the ﬁrst to show that imperceptible perturbations in the input space can alter the outcome of explanations of text classiﬁers signiﬁcantly, giving an efﬁcient attack to estimate explanation robustness.
However, our work is the ﬁrst to give an extensive evaluation of the robustness of widely-used explanation methods on large datasets, comparing several state-of-the-art architectures.
Further, we are the ﬁrst to address robustness of attention weights in transformer architectures and to provide insight on transfer capabilities and universal attacks.
2. B ACKGROUND
Let S = { s 1 , s 2 , ..., s N } be a dataset of N text samples
s i , each with a label from a predeﬁned set of labels
L = { l 1 , l 2 , ..., l | L | } .
Each sample s i contains a sequence of tokens (or words)
w i taken from a discrete vocabulary set W = { w 1 , w 2 , ..., w | W | } .
A generic sequence classiﬁer then consist of a non-injective, non-surjective embedding function E : S !
R d ⇥ p , E ( s ) = X , which maps the input sample s to its embedding matrix X , and a function f : R d ⇥ p !
R |
L | , f ( X )
=
o , representing a (DNN) classiﬁer function.
d and p denote the embedding dimension and sequence length respectively.
Let F : S !
R |
L | , F ( s ) =
f  E be the full sequence classiﬁer with ﬁnal prediction y = arg max i 2 { 1: | L |} o i .
We deﬁne an attribution map as A : S !
R p , A ( s , F, l ) =
a that assigns a scalar value to each input token w i in the text sample s , resulting in the attribution vector a 2 R p .
This vector represents each token’s inﬂuence towards the prediction outcome y of classiﬁer F .
Our work considers three widely-used attribution methods in text classiﬁcation, namely Saliency Maps (S) (Simonyan et al., 2013), Integrated Gradients (IG) (Sundararajan et al., 2017) and Attention (A) (Bahdanau et al., 2014), deﬁned in the following Equations (1), (2) and (3) respectively.
[Equation]
where B denotes the null matrix 0 d ⇥ p , f is the classiﬁer function of F , ↵ a scaling factor and
X = E ( s ) .
r X f denotes the matrix-derivative of f to X , as deﬁned in Goodfellow et al.
An attention head is a layer that transforms its inputs into scores e and calculates its output by linear combination of each input score, with coefﬁcients normalized to a distribution.
These coefﬁcients are the attention weights A Att i ( s , F, l ) denoted in Equation (3).
It is commonly agreed to give intuitive explanations on how much the model attends to the given inputs through its attention weights (Jacovi & Goldberg, 2020).
M ETHODS
In this section, we describe our novel method T EXT E XPLANATION F OOLER (TEF) to estimate attribution robustness (AR) in sequence classiﬁcation problems.
Speciﬁcally, we deﬁne the problem formulation, introduce our threat model and present the algorithm used to alter explanations by imperceptibly changing the inputs.
3. P ROBLEM F ORMULATION
Given an input text samples s and ˜ s , labels l ; a text classiﬁer F with embedding function E and classiﬁer function f ; and attribution method A , we deﬁne attribution robustness (also explanation robustness , AR) as written in Equation (4).
[Equation]
with
[Equation]
where d denotes a distance measure between the attributions ˜ a and a of the the two input samples s and ˜ s .
The rest of the notation is kept as in Section 2.
Equation (4) quantiﬁes how different the attributions of two input samples are, given the constraint in Equation (5) that enforces the inputs having the same prediction outcome.
The attribution robustness estimation is then solved utilizing the following Equation (6).
[Equation]
where s adv denotes the solution to the estimation, i.e. the adversarial input, which also minimizes
AR deﬁned in Equation (4).
s denotes the original, unperturbed input and ˜ s the perturbed input, optimized during estimation.
The solution s adv gives a robustness estimate by ﬁnding an input that maximizes the distance between original attribution A ( s , F, l ) and adversarial attribution A (˜ s , F, l ) within a local neighbourhood of s .
The more dissimilar these maps are, the less robust the attribution method is.
The local neighbourhood is deﬁned by both linguistic constraints described in the next section that encourage semantic proximity to the original text and the perturbed samples having the same prediction outcome as the unperturbed ones, see Equation (5).
This formulation is backed by current research (Ghorbani et al., 2019;
Dombrowski et al., 2019;
Ivankay et al., 2020) and the
Prediction Assumption of faithful explanations (Jacovi & Goldberg, 2020).
3. T HREAT M ODEL AND A TTACK
Input : Input sentence s with predicted class l , classiﬁer F , attribution A , attribution distance d , number of synonyms N , maximum perturbation ratio ⇢ max
Output : Adversarial sentence s adv 1: s adv s , d max 0 , r 2: for w i 2 s do 3:
I w i =
d ⇥
A ( s w i ! 0 , F, l ) , A ( s , F, l ) ⇤ 4: for w j 2 h w 1 , ..., w | s | i | I
w m  1  I w m 8 m{ 2 , ..., | s |} do 5: if w j 2 S Stopwords then 6: continue 7:
C j
SynonymEmbeddings ( w j , N ) 8:
C j
POSFilter ( w j , C j , s ) 9: for c k 2 C j do 10: ˜ s w j !
c k
Replace w j in s adv with c k 11: if arg max i 2 { 1: | L |}
F (˜ s w j !
c k ) =
l then 12: ˜
d d ⇥
A (˜ s w i !
c k , F, l ) , A ( s , F, l ) ⇤ 13: if ˜ d > d max then 14: s adv ˜ s
w i !
c
k 15: d max ˜ d 16: r r +17: if ⇢ = r
| s | + 1 > ⇢ max then 18: break
We deﬁne our algorithm to estimate AR as a black-box attack.
It only queries the model to obtain its output logits and the accompanied explanations of the inference process.
The model might access its gradients to compute explanations, but the attack only utilizes the resulting explanations, no gradient or architectural information.
We restrict the valid input perturbations to token substitutions, speciﬁcally insertions and deletions of tokens are forbidden, as they alter the input lengths.
Algorithm 1 contains the schematic code for TEF, consisting of the following two steps.
Step 1 - Word importance ranking
First, an importance ranking is extracted for each token of the input sample.
Speciﬁcally, we compute I w i =
d ⇥
A ( s w i ! 0 , F, l ) , A ( s , F, l ) ⇤ for each token i in s , where s w i !
denotes the input sequence s with the i -th word masked to the zero embedding token.
The input tokens are then sorted by the I w i values in a decreasing fashion.
Then, high importance words are prioritized during substitution.
[Table]
Step 2 - Candidate selection For each word w i in s sequentially, a set of substitution candidates
C of N elements is extracted.
This candidate set is constructed from the counter-ﬁtted GloVe (Pennington et al., 2014) synonym embeddings by the authors of Mrkˇsi´c et al.
The candidates are then ﬁltered by Part-Of-Speech (POS), tagged by SpaCy (Honnibal et al., 2020), only allowing replacements with equal POS.
Stop words are also discarded from C .
A ﬁnal selection as replacement for w i is then made to be the c k 2 C that maximizes d ⇥ A [Table] ˜
[Table]
i [Table] c k , F, l ) , A
[Table]
s
[Table]
⇤ .
The algorithm is aborted when the number of replacements to sentence length exceeds the maximum value ⇢ max .
E XPERIMENTS AND R ESULTS
In this section [Table] we present an extensive evaluation of our attribution robustness (AR) estimation attack, TEF, for sequence classiﬁcation problems.
We examine the performance of TEF and study the impact of different factors on its robustness evaluation performance.
We ﬁnd that our attack effectively reduces the correlation of original and attacked attributions on all datasets and models.
Moreover, we describe our transfer and semi-universal attacks and examine their robustness estimation performance, showing that even under circumstances where the model and explainer are unknown to the attacker, TEF perturbations transferred from other models decrease attribution robustness effectively.
4. M ODELS , DATASETS AND EVALUATION
Our TEF attack is evaluated on ﬁve commonly used public sequence classiﬁcation datasets, AG’s
News (Zhang et al., 2015), MR reviews (Zhang et al., 2015), IMDB Movie Reviews (Maas et al., 2011), Fake News Dataset 1 and Yelp (Asghar, 2016).
We train six different word embedding-based architectures for each dataset, namely a CNN, an LSTM, an LSTM containing a single attention layer with one head (LSTMAtt) and three state-of-the-art ﬁnetuned transformer-based architectures,
BERT (Li et al., 2020), RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019).
Table 1 contains a summary of our model performances as well as details on the datasets.
The text samples are tokenized with the default English SpaCy (Honnibal et al., 2020) tokenizer for the CNN, LSTM and LSTMAtt models and embedded with the pretrained GloVe 6B 300-dimensional word vectors (Pennington et al., 2014).
The transformer-based models use their own pretrained tokenizers and embeddings.
We use PyTorch (Paszke et al., 2019) with Captum (Kokhlikyan et al., 2020) to implement our models and explainers and the Huggingface Transformers library (Wolf et al., 2020) to ﬁnetune the transformer architectures on our datasets.
We evaluate the robustness of three commonly used explanation methods in natural language processing with our TEF attack.
These are Saliency Maps (S), Integrated Gradients (IG) and the Attention mechanism (A), deﬁned in Section 2.
We use S and IG in combination with all our architectures, Attention only with LSTMAtt, BERT, RoBERTa and XLNet.
During the attack, we set the attribution distance d of Equation (6) to be d (˜ a , a ) = 1  PCC(˜ a , a ) + 1, with PCC denoting the
Pearson Correlation Coefﬁcient (Pearson, 1895) of original and adversarial attributions ˜ a and a .
We then report the standard Pearson Correlation Coefﬁcient (PCC), Kendall’s Rank Order Correlation (ROC) (Kendall, 1938), Spearman’s Correlation Coefﬁcient (SCC) (Myers & Sirois, 2004) and
the
Top-10%, Top-30% and Top-50% intersections to measure AR in Equation (4).
These are common
LSTMAtt - Attention (A) on IMDB
[Figure]
⇢
[Figure] ACC:
[Figure]
ACC: 0.
⇢ 0.
⇢ 3.
BERT - Integrated Gradients (IG) on Yelp
[Figure]
⇢
[Figure] ACC [Figure] [Figure]
ACC: 0.
⇢ 0.
⇢ 7.
RoBERTa - Saliency Maps (S) on MR [Figure]
⇢
[Figure]
ACC
[Figure]
ACC:
[Figure]
⇢ 0.
⇢ 8.
XLNet - Attention (A) on Fake News
[Figure]
[Figure] [Figure]
[Figure]
metrics that correspond to human measures of AR (Ghorbani et al., 2019; Dombrowski et al., 2019; Ivankay et al., 2020). Additionally, in order to quantify imperceptibility of perturbations, the semantic similarity of adversarially perturbed and unchanged sentences is reported, along with the relative increase of average perplexity of the perturbed samples, given by the GPT-2 (Radford et al., 2019) language model. Semantic similarity (Sem. sim.) is measured by the cosine distance between the embeddings produced by the Universal Sentence Encoder (USE) (Cer et al., 2018).
This is a stateof-the-art sentence embedding widely used in adversarial attacks on text (Sun et al., 2020; Jin et al., 2019).
Perplexity increase (PPL inc.) indicates how much the likelihood of the perturbed data has decreased, given a language model, and is often used to validate language models (Keselj, 2009).
Due to the lack of related work in this ﬁeld, we compare the AR estimation performance of TEF to our R ANDOM
A TTACK (RA) baseline.
RA serves as an agnostic attack, utilizes a random word importance ranking in Step 1 of TEF and selects a random synonym in the ﬁnal selection in Step 2.
POS and stop word ﬁlters (see Section 3) are still utilized in RA to keep linguistic constraints intact.
PCC of LSTMAtt - A on AG’s News PCC of LSTM - IG on MR
[Figure]
4. R OBUSTNESS OF EXPLANATIONS
Attribution robustness estimation.
In order to estimate the attribution robustness (AR) of the aforementioned models and explainers, we vary the parameter ⇢ max of TEF, which denotes the maximum ratio of perturbed tokens in the input sample.
A larger ⇢ max value leads to lower attribution correlation, as potentially more words are substituted in the input.
We then capture the aforementioned metrics PCC, ROC, SCC, Sem.
sim, Top-10%/30%/50% intersections and PPL inc. to evaluate AR.
Additionally, in order to quantify performance of our attack over the whole operation interval of 0  ⇢ max  0 . 4 , we compute the Area under the Pearson Correlation Curve (ACC).
A lower value of ACC corresponds to lower robustness overall, as correlation values are lower.
We note that a particular value of ⇢ max does not guarantee that all input samples have exactly ⇢ max ratio of perturbed tokens.
Therefore, we quantize our samples based on their actual, resulting perturbed ratio ⇢ such that samples with similar ⇢ are grouped together.
These bins are computed per dataset, ensuring the comparability of resulting curves and ACCs for each plot.
Moreover, we choose the number of candidates in Step 2 of TEF to be N =
| C | = 15 , as it is a good trade-off between TEF estimation performance and attack run time.
As expected, we ﬁnd that TEF is able to signiﬁcantly outperform the baseline provided by RA in terms of all AR metrics, on all datasets, models and explanation methods considered in this work.
A subset of these results is shown in Figure 2, the rest can be found in Appendix A.2.
Moreover, we do not ﬁnd that any architecture is signiﬁcantly more robust to TEF perturbations for explainers S and IG.
However, the self-attention mechanism of transformers seems to be more robust to perturbations than non-transformer-based architectures and explanations.
The semantic similarity decreases with increasing ⇢ and stays above 0 .
in most cases.
This, together with the fact that resulting samples share predictions with the non-perturbed ones effectively highlights that the explanations given by these models and attribution methods lack faithfulness .
Ablation study.
In addition to the fully random attack described in the previous paragraph, we compare TEF to our semi-random attacks R ANDOM I MPORTANCE (RI) and
R ANDOM S YNONYM (RS).
We randomize the word importance ranking of TEF (RI) but keep the selection of best ﬁnal synonym, and we randomize the ﬁnal synonym selection of TEF (RS) but keep the word importance ranking respectively.
Figure 3 shows our ﬁndings for these experiments, along with comparisons to
R ANDOM
A TTACK (RA).
The PCC curves and the ACC values show that RI consistently outperforms
RS in terms of PCC over the whole operation interval of ⇢ .
Moreover, the impact of word importance ranking diminishes with increasing ⇢ , especially for shorter datasets like MR.
This can be observed by RS performing closer to RA for high ⇢ values.
BERT’s attention layers and heads.
BERT’s attention weights can be used to help gain insight into a models prediction by understanding which parts of the input are most attended to (Vig, 2019).
Our BERT models have 12 layers with 12 attention heads (144 heads in total), each producing a distribution of attention weights over its inputs and outputs.
Estimating the AR of all heads together is not useful, as effects would average out.
Therefore, we run TEF to estimate the robustness of each head separately.
Figure 4 contains the average PCCs of the attention weights before and after perturbing the inputs with TEF.
We ﬁnd that attention weights in later layers tend to be more susceptible to input perturbations than earlier layers.
Moreover, heads within a layer tend to be comparably robust.
We leave a thorough, theoretical analysis of this phenomenon to future work.
We conclude that
[Figure]
MR
IMDB
Yelp
L.
H.
L.
H.
L.
H.
L.
H.
[Figure]
the increasing reliance on attention weights to provide inherent interpretations to BERT predictions needs careful investigation, especially in safety-critical applications.
4. T RANSFERABILITY AND SEMI - UNIVERSAL PERTURBATIONS
Transferability of perturbations to models and explanations.
The adversary does not necessarily possess information about the deployec model nor the exact method to produce the accompanying explanations.
Therefore, it is crucial for systems to be as resistant to transfer attacks as possible in order to evade perturbations constructed on similar models and explanations.
Thus, we examine how our classiﬁers and attribution methods react to transfer attacks computed with
TEF.
We alter the input samples for a given model and explanation method with TEF, then evaluate the PCC of attributions on the same samples but different architectures and explainers.
The results are found in Figure 5.
We observe that transfer attacks perform better than RA, some even by approx.
4 in terms of average PCC decrease in the operation area of ⇢ ⇡ 0 .
However, as expected, they signiﬁcantly fall short of the performance of TEF.
Therefore, we conclude that transferring TEF perturbations across models and explainers effectively highlights fragility of explanations, but TEF provides tighter AR bounds without transfer.
Semi-universal perturbations.
In this section, we take a step towards deﬁning universal perturbations, similarly to the work of the authors Moosavi-Dezfooli et al.
and Gao & Oates (2019).
These provide fast and computationally cheap perturbations during attack time that are able to mislead classiﬁers with pre-computed perturbations.
However, we attack the explanations of text classiﬁers, instead of their predictions and call our perturbations semi-universal attack policies .
We split the test dataset into two equally sized parts, the attack set and the evaluation set.
We utilize the former for constructing our semi-universal attack policies and the latter to evaluate how effectively our semi-universal attack alters the attributions maps of our models.
First, for each sample in the attack set, we compute the optimal TEF perturbation for all our models and explainers.
We then extract statistics of these perturbations, which are the most common replacement and the replacement frequency for each replaced token and sort these by decreasing frequency. These are our semi-universal attack policies , seen in Figure 6. During this construction [Figure]
146k goldman said 131k
[Figure]
glows
PCC of LSTM - IG on AG’s News
[Figure]
phase, we query the model for predictions and explanations, as we compute optimal TEF perturbations.
Second, we evaluate our semi-universal attack that utilizes the aforementioned policies to alter explanations of classiﬁers.
The inputs to this attack are a text sample, a semi-universal policy and a maximum perturbed ratio ⇢ max .
The attack iterates over the policy, starting with the token in the ﬁrst row and ﬁnishing with the last.
Whenever the current token is found in the input text sample, it is replaced with the replacement token in the list.
If the perturbed ratio exceeds ⇢ max , the attack is aborted.
In such a way, perturbed inputs are created without querying the model during attack time.
The actual perturbation for each text sample depends on the sample, hence the name semi-universal attack policy .
The resulting samples are then evaluated on a given model and explanation method.
Representative results are given in Figure 7.
We conclude that our semi-universal policies are effective in reducing attribution correlation when the adversary has no access to the target model and explanation method, as indicated by the lower ACC values of the semi-universal PCC curves.
C ONCLUSION
In this work, we introduced a novel black-box attack called T EXT E XPLANATION F OOLER , that successfully perturbs input data such that the outcome of popular explanation methods in sequence classiﬁcation, but not the prediction of the classiﬁer.
This attack provides a baseline estimator for attribution robustness and highlights the lack of robustness of current explanation methods.
We compared it to the random attack, showing its superior performance to it on ﬁve different, widely used text classiﬁcation datasets.
Moreover, the transfer capabilities of the attack are evaluated.
Finally, we showed the existence of semi-universal perturbation policies that are capable of altering explanations without querying the model during attack-time, even without having access to perturbations for those models.
In future work, we plan to examine whether a similar white-box attack that has access to model gradients can improve robustness estimation.
Moreover, instead of synonym embeddings, we plan to use BERT-based masked language models to extract possible candidates, further improving imperceptible word substitutions.