Original: B ETTER SAMPLING IN EXPLANATION METHODS CAN PREVENT DIESELGATE - LIKE DECEPTION
Revised: B ETTER SAMPLING IN EXPLANATION METHODS CAN PREVENT DIESELGATE - LIKE DECEPTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning models are used in many sensitive areas where besides predic- tive accuracy, their comprehensibility is also important.
Revised: Machine learning models are used in many sensitive areas where, besides predictive accuracy, their comprehensibility is also important.

Original: Interpretability of prediction models is necessary to determine their biases and causes of errors and is anecessary prerequisite for users’ conﬁdence.
Revised: Interpretability of prediction models is necessary to determine their biases and causes of errors and is a prerequisite for users’ conﬁdence.

Original: For complex state-of-the-art black-box models post-hoc model-independent explanation techniques are an established solution.
Revised: For complex state-of-the-art black -box models, post-hoc model-independent explanation techniques are an established solution.

Original: Popular and effective techniques, such as IME, LIME and SHAP, use perturbation of instance features to explain individual predictions.
Revised: Popular and effective techniques, such as IME, LIME, and SHAP, use perturbation of instance features to explain individual predictions.

Original: Recently, Slack et al.
Revised: Recently, Slack et al.

Original: (2020) put their robustness into question by showing that their outcomes can be manipulated due to poor perturbation sampling employed.
Revised: (2020) put their robustness into question by showing that their outcomes can be manipulated due to poor perturbation sampling employed.

Original: This weakness would allow dieselgate type cheating of owners of sensitive models who could deceive inspection and hide potentially unethical or illegal biases existing in their predictive models.
Revised: This weakness would allow dieselgate type cheating of owners of sensitive models who could deceive inspection and hide potentially unethical or illegal biases existing in their predictive models.

Original: This could undermine public trust in machine learning models and give rise to legal restrictions on their use.
Revised: This could undermine public trust in machine learning models and give rise to legal restrictions on their use.

Original: We show that better sampling in these explanation methods prevents malicious manipulations.
Revised: We show that better sampling in these explanation methods prevents malicious manipulations.

Original: The proposed sampling uses data generators that learn the training set distribution and generate new perturbation instances much more similar to the training set.
Revised: The proposed sampling uses data generators that learn the training set distribution and generate new perturbation instances much more similar to the training set.

Original: We show that the improved sampling increases therobustness of the LIME and SHAP, while  previously untested method IME is already the most robust of all.
Revised: We show that the improved sampling increases the LIME and SHAP’s robustness, while the previously untested method IME is already the most robust of all.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models are used in many areas where besides predictive performance, their comprehensibility is also important, e.g., in healthcare, legal domain, banking, insurance, consultancy, etc.
Revised: Machine learning models are used in many areas where besides predictive performance, their comprehensibility is also important, e.g., in healthcare, legal domain, banking, insurance, consultancy, etc.

Original: Users in those areas often do not trust a machine learning model if they do not understand why it made a given decision.
Revised: Users in those areas often do not trust a machine learning model if they do not understand why it made a given decision.

Original: Some models, such as decision trees, linear regression, and na ¨ ıve Bayes, are intrinsically easier to understand due to the simple representation used.
Revised: Some models, such as decision trees, linear regression, and na ¨ ıve Bayes, are intrinsically easier to understand due to the simple representation used.

Original: However, complex models,which are mostly used in practice due to better accuracy, are incomprehensible and behave like black boxes, e.g., neural networks, support vector machines, random forests, and boosting.
Revised: However, complex models, mostly used in practice due to better accuracy, are incomprehensible and behave like black boxes, e.g., neural networks, support vector machines, random forests, and boosting.

Original: For these models, the area of explainable artiﬁcial intelligence (XAI) has developed post-hoc explanation methods that are model-independent and determine the importance of each feature for the predicted outcome.
Revised: For these models, the area of explainable artiﬁcial intelligence (XAI) has developed post-hoc explanation methods that are model-independent and determine the importance of each feature for the predicted outcome.

Original: Frequently used methods of this type are IME (ˇStrumbelj & Kononenko, 2013), LIME (Ribeiro et al., 2016), and SHAP (Lundberg & Lee, 2017).
Revised: Frequently used methods of this type are IME (ˇStrumbelj & Kononenko, 2013), LIME (Ribeiro et al., 2016), and SHAP (Lundberg & Lee, 2017).

Original: To determine the features’ importance, these methods use perturbation sampling.
Revised: To determine the features’ importance, these methods use perturbation sampling.

Original: Slack et al.
Revised: Slack et al.

Original: (2020) recently noticed that the data distribution obtained in this way is signiﬁcantly different from the original distribution of the training data as we illustrate in Figure 1a.
Revised: (2020) recently noticed that the data distribution obtained in this way is signiﬁcantly different from the original distribution of the training data as we illustrate in Figure 1a.

Original: They showed that this can be a serious weakness of these methods.
Revised: They showed that this can be a serious weakness of these methods.

Original: Possibility to manipulate the post-hoc explanation methods is a critical problem for the ML community, as the reliability and robustness of explanation methods are essential for their use and public acceptance.
Revised: The possibility to manipulate the post-hoc explanation methods is a critical problem for the ML community, as the reliability and robustness of explanation methods are essential for their use and public acceptance.

Original: These methods are used in the interpretation of otherwise black-box models,they help in debuggingof models, and reveal models’ biases, thereby establishing trust in their behaviour.
Revised: These methods are used to interpret otherwise black-box models, help in debugging models, and reveal models’ biases, thereby establishing trust in their behavior.

Original: Non-robust explanation methods that can be manipulated can lead to catastrophic consequences, as explanations do not detect racist, sexist, or otherwise biased model if theowner of the model  wants to hide these biases.
Revised: Non-robust explanation methods that can be manipulated can lead to catastrophic consequences, as explanations do not detect racist, sexist, or otherwise biased models if the model owner wants to hide these biases.

Original: This would enable dieselgate-like cheating where owners of sensitive prediction models could hide the socially, morally, or legally unacceptable biases present in their models.
Revised: This would enable dieselgate-like cheating where owners of sensitive prediction models could hide the socially, morally, or legally unacceptable biases present in their models.

Original: As the schema of the attack on explanation methods 
Revised: As the schema of the attack on explanation methods on Figure 1b shows,

Original: [Figure]
Revised: [Figure]

Original: on Figure 1b shows, owners of prediction models could detect when their models are examined and return unbiased predictions in this case and biased predictions in normal use.
Revised: owners of prediction models could detect when their models are examined and return unbiased predictions in this case and biased predictions in normal use.

Original: This could have serious consequences in areas where the reliability and fairnessof predictive models are essential, e g., in healthcare or banking.
Revised: This could have serious consequences in areas where predictive models’ reliability and fairness are essential, e g., in healthcare or banking.

Original: Such weaknesses can undermine users’ trust in machine learning models in general and slow down technological progress.
Revised: Such weaknesses can undermine users’ trust in machine learning models in general and slow down technological progress.

Original: In this work, we propose to change the main perturbation-based explanation methods and make them more resistant to manipulation attempts.
Revised: In this work we propose to change the main perturbation-based explanation methods and make them more resistant to manipulation attempts.

Original: In our solution, the problematic perturbation-based sampling is replaced with more advanced sampling, which uses modern data generators that better capture the distribution of the training dataset.
Revised: In our solution, the problematic perturbation-based sampling is replaced with more advanced sampling, which uses modern data generators that better capture the distribution of the training dataset.

Original: We test three generators, the RBF network based generator (Robnik-ˇSikonja, 2016), random forest-based generator, available in R library semiArtiﬁcial (Robnik- ˇ Sikonja, 2019), as well as the generator using variational autoencoders (Miok et al., 2019).
Revised: We test three generators, the RBF network based generator (Robnik- ˇ Sikonja, 2016), random forest-based generator, available in R library semiArtiﬁcial (Robnik- ˇ Sikonja, 2019) as well as the generator using variational autoencoders (Miok et al., 2019).

Original: We show that the modiﬁed gLIME and gSHAP methods are much more robust than their original versions.
Revised: We show that the modiﬁed gLIME and gSHAP methods are much more robust than their original versions.

Original: For the IME method, which previously was not analyzed, we show that it is already quite robust.
Revised: For the IME method, which previously was not analyzed, we show that it is already quite robust.

Original: We release the modiﬁed explanation methods under the open-source license 
Revised: We release the modiﬁed explanation methods under the open-source license 1 .

Revised: In this work, we use the term robustness of the explanation method as a notion of resilience against adversarial attacks, i.e. as the ability of an explanation method to recognize the biased classiﬁer in an adversary environment.

Revised: This type of robustness could be more formally deﬁned as the number of instances where the adversarial model’s bias was correctly recognized.

Revised: We focus on the robustness concerning the attacks described in Slack et al.

Revised: There are other notions of robustness in explanation methods; e.g., (Alvarez-Melis & Jaakkola, 2018) deﬁne the robustness of the explanations in the sense that similar inputs should give rise to similar explanations.

Original: The remainder of the paper is organized as follows.
Revised: The remainder of the paper is organized as follows.

Original: In Section 2, we present the necessary background and related work on explanation methods, attacks on them, and data generators.
Revised: In Section 2, we present the necessary background and related work on explanation methods, attacks on them, and data generators.

Original: In Section 3, we propose a defence against the described weaknesses of explanation methods and in Section 4, we empirically evaluate the proposed solution.
Revised: In Section 3, we propose a defense against the described weaknesses of explanation methods, and in Section 4, we empirically evaluate the proposed solution.

Original: In Section 5, we draw conclusions and present ideas for further work.
Revised: In Section 5, we draw conclusions and present ideas for further work.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: In this section, we ﬁrst brieﬂy describe the background on post-hoc explanation methods and attacks on them, followed by data generators and related works on  robustness of explanation methods.
Revised: In this section, we ﬁrst brieﬂy describe the background on post-hoc explanation methods and attacks on them, followed by data generators and related works on the robustness of explanation methods.

Original: 2. P OST - HOC EXPLANATION METHODS
Revised: 2. P OST - HOC EXPLANATION METHODS

Original: The current state-of-the-art perturbation-based explanation methods, IME, LIME, and SHAP, explain predictions for individual instances.
Revised: The current state-of-the-art perturbation-based explanation methods, IME, LIME, and SHAP, explain predictions for individual instances.

Original: To form an explanation of a given instance, they measure the difference in prediction between the original instance and its neighbouring instances, obtained with perturbation sampling.
Revised: To form an explanation of a given instance, they measure the difference in prediction between the original instance and its neighboring instances, obtained with perturbation sampling.

Original: Using the generated instances, the LIME method builds a local interpretable model, e.g., a linear model.
Revised: Using the generated instances, the LIME method builds a local interpretable model, e.g., a linear model.

Original: Themethods SHAP and IME  determine the impact of the features as Shapley values from the coalitional game theory (Shapley, 1988).
Revised: The SHAP and IME methods determine the impact of the features as Shapley values from the coalitional game theory (Shapley, 1988).

Original: In this way, they assure that the produced explanations obey the four Shapley fairness axioms (ˇStrumbelj & Kononenko, 2013).
Revised: In this way, they assure that the produced explanations obey the four Shapley fairness axioms (ˇStrumbelj & Kononenko, 2013).

Original: Due to the exponential time complexity of Shapley value calculation, both methods try to approximate them.
Revised: Due to the exponential time complexity of Shapley value calculation, both methods try to approximate them.

Original: The three methods are explained in detail in the above references, and a formal overview is presented in Appendix A, while below, we present a brief description.
Revised: The three methods are explained in detail in the above references, and a formal overview is presented in Appendix A, while below, we present a brief description.

Original: In our exposition of the explanation methods, we denote with f the predictive model and with x the instance we are explaining.
Revised: In our exposition of the explanation methods, we denote with f the predictive model and with x the instance we are explaining.

Original: Explanations of instances with the LIME method is obtained with an interpretable model g .
Revised: Explanations of instances with the LIME method is obtained with an interpretable model g .

Original: The model g has to be both locally accurate (so that it can obtain correct features contributions) and simple (so that it is interpretable).
Revised: The model g has to be both locally accurate (so that it can obtain correct feature contributions) and simple (so that it is interpretable).

Original: Therefore in LIME g is a linear regression model trained on the instances sampled in the neighbourhood of the explained instance x , which is ﬁrst normalized.
Revised: Therefore in LIME, g is a linear regression model trained on the instances sampled in the neighborhood of the explained instance x , which is ﬁrst normalized.

Original: The perturbed explanation instances are generated by adding Gaussian noise to each feature of x independently.
Revised: The perturbed explanation instances are generated by adding Gaussian noise to each feature of x independently.

Original: Using linear models to determine the contributions of features, LIME is fast but may produce poor explanations for instances close to complex decision boundaries.
Revised: Using linear models to determine the contributions of features, LIME is fast but may produce poor explanations for instances close to complex decision boundaries.

Original: SHAP estimates Shapley values using the approach of LIME (i.e. local linear models computed with the weighted least squares algorithm) but using a different neighbourhood and loss function for local accuracy which assure that Shapley values are obtained.
Revised: SHAP estimates Shapley values using the approach of LIME (i.e. local linear models computed with the weighted least squares algorithm) but using a different neighborhood and loss function for local accuracy, which ensures that Shapley values are obtained.

Original: The sampling of explanation instances in
Revised: The sampling of explanation instances in

Original: SHAP relies on the distribution set D , obtained from the training set, by computing the centroids of k-means clusters.
Revised: SHAP relies on the distribution set D , obtained from the training set, by computing the centroids of k-means clusters.

Original: IME computes Shapley values using  pure sampling approach (no proxy with an interpretable model).
Revised: IME computes Shapley values using a pure sampling approach (no proxy with an interpretable model).

Original: Its sampling replaces part of the explained instance with existing values of selected features and then averages differences in the prediction of explained instance and samples.
Revised: Its sampling replaces part of the explained instance with existing values of selected features and then averages differences in the prediction of explained instance and samples.

Original: This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency.
Revised: This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency.

Original: 2. A TTACKS ON EXPLANATIONS
Revised: 2. A TTACKS ON EXPLANATIONS

Original: The idea of the attack on explanations presented in Slack et al.
Revised: The idea of the attack on explanations presented in Slack et al.

Original: (2020) is that the attacker creates an adversarial (biased) model whose behaviour it wants to hide from explanation methods, e.g., a racist model that does not grant credits to blacks.
Revised: (2020) is that the attacker creates an adversarial (biased) model whose behavior it wants to hide from explanation methods, e.g., a racist model that does not grant credits to blacks.

Original: As Figure 1b illustrates, a part of the attacker’s adversarial model is an unbiased model, which, e.g., does not take disputed attributes such as race into account.
Revised: As Figure 1b illustrates, a part of the attacker’s adversarial model is an unbiased model, which, e.g., does not take disputed attributes such as race into account.

Original: The adversarial model tries to manipulate the explanation method by behaving like an unbiased model on perturbed instances used for the explanation – in this case, it uses theoutput of the unbiased model.
Revised: The adversarial model tries to manipulate the explanation method by behaving like an unbiased model on perturbed instances used for the explanation – in this case, it uses the unbiased model’s output.

Original: On instances from the original distributions that are a part ofa normal model use, the adversarial model uses theoutput of the biased model.
Revised: On instances from the original distributions that are a part of normal model use, the adversarial model uses the biased model’s output.

Original: To distinguish between the original and perturbation-sampled data, the adversarial model contains a decision model that selects the right type of model (biased or unbiased) based on the predicted type of instances.
Revised: To distinguish between the original and perturbation-sampled data, the adversarial model contains a decision model that selects the right type of model (biased or unbiased) based on the predicted type of instances.

Original: The reason that the decision function can distinguish between the original data and sampled instances are the shortcomings of explanation methods that use the perturbation sampling (see Figure 1a).
Revised: The reason that the decision function can distinguish between the original data and sampled instances are the shortcomings of explanation methods that use the perturbation sampling (see Figure 1a).

Original: For that graph, we created new instances based on the COMPAS dataset (see Section 4) with the Gaussian noise as in the LIME explanation method.
Revised: For that graph, we created new instances based on the COMPAS dataset (see Section 4) with the Gaussian noise as in the LIME explanation method.

Original: We mapped the original and sampled instances to 2D space using the PCA transformation.
Revised: We mapped the original and sampled instances to 2D space using the PCA transformation.

Original: We can see that the sampled data is distributed differently than the original data.
Revised: We can see that the sampled data is distributed differently than the original data.

Original: As the difference is revealed with PCA, other machine learning algorithms can also effectively distinguish between sampled and original data.
Revised: As the difference is revealed with PCA, other machine learning algorithms can also effectively distinguish between sampled and original data.

Original: The attack can be formally deﬁned in the following way.
Revised: The attack can be formally deﬁned in the following way.

Original: The attacker has a biased model b and a sample S from the distribution X dist .
Revised: The attacker has a biased model b and a sample S from the distribution X dist .

Original: The attacker has no additional information about the distribution X dist .
Revised: The attacker has no additional information about the distribution X dist .

Original: The attacker’s goal is to produce an adversarial model e that behaves like b on the data from X dist but does not reveal b ’s bias to the explanation methods.
Revised: The attacker’s goal is to produce an adversarial model e that behaves like b on the data from X dist but does not reveal b ’s bias to the explanation methods.

Original: We assume that the attacker also has an unbiased model ψ that hides the f ’s bias.
Revised: We assume that the attacker also has an unbiased model ψ that hides the f ’s bias.

Original: The attacker creates a decision model d , that should output 1 if the input instance x is from X dist and 0 otherwise.
Revised: The attacker creates a decision model d , that should output 1 if the input instance x is from X dist and 0 otherwise.

Original: The model d is trained on S and generated perturbation samples.
Revised: The model d is trained on S and generated perturbation samples.

Original: The attacker creates the adversarial model e with its output deﬁned by the following equation:
Revised: The attacker creates the adversarial model e with its output deﬁned by the following equation:

Original: [Equation]
Revised: [Equation]

Original: 2. D ATA GENERATORS
Revised: 2. D ATA GENERATORS

Original: We used three different data generators based on different algorithms, modelling the distribution of the training set: variational autoencoder with Monte Carlo dropout (Miok et al., 2019), RBF network (Robnik-ˇSikonja, 2016), and random forest ensemble (Robnik-ˇSikonja, 2019).
Revised: We used three different data generators based on different algorithms, modeling the distribution of the training set: variational autoencoder with Monte Carlo dropout (Miok et al., 2019), RBF network (Robnik-ˇSikonja, 2016), and random forest ensemble (Robnik-ˇSikonja, 2019).

Original: In the remainder of the paper, we refer to the listed generators 
Revised: In the remainder of the paper, we refer to the listed generators consecutively as MCD-VAE, rbfDataGen, and

Original:  Autoencoder (AE) consists of two neural networks called encoder and decoder.
Revised: TreeEnsemble. Autoencoder (AE) consists of two neural networks called encoder and decoder.

Original: It aims to compress the input instances by passing them through the encoder and then reconstructing them to the original values with the decoder.
Revised: It aims to compress the input instances by passing them through the encoder and then reconstructing them to the original values with the decoder.

Original: Once the AE is trained, it can be used to generate new instances.
Revised: Once the AE is trained, it can be used to generate new instances.

Original: Variational autoencoder (Doersch, 2016) is a special type of autoencoder, where the vectors z in the latent dimension (output of the encoder and input of the decoder) are normally distributed.
Revised: Variational autoencoder (Doersch, 2016) is a special type of autoencoder, where the vectors z in the latent dimension (output of the encoder and input of the decoder) are normally distributed.

Original: Encoder is therefore approximating the posterior distribution p ( z | x ) , where we assume p ( z | x ) ∼ N ( µ x , Σ x ) .
Revised: Encoder is therefore approximating the posterior distribution p ( z | x ) , where we assume p ( z | x ) ∼ N ( µ x , Σ x ) .

Original: The generator proposed by Miok et al.
Revised: The generator proposed by Miok et al.

Original: (2019) uses the Monte Carlo dropout (Gal & Ghahramani, 2016) on the trained decoder.
Revised: (2019) uses the Monte Carlo dropout (Gal & Ghahramani, 2016) on the trained decoder.

Original: The idea of this generator is to propagate the instance x through the encoder to obtain its latent encoding z .
Revised: The idea of this generator is to propagate the instance x through the encoder to obtain its latent encoding z .

Original: This can be propagated many times through the decoder, obtaining every time a different result due to the Monte Carlo dropout, but preserving similarity to the original instance x .
Revised: This can be propagated many times through the decoder, obtaining every time a different result due to the Monte Carlo dropout but preserving similarity to the original instance x .

Original: The RBF network (Moody & Darken, 1989) uses Gaussian kernels as hidden layer units in a neural network.
Revised: The RBF network (Moody & Darken, 1989) uses Gaussian kernels as hidden layer units in a neural network.

Original: Once the  parametersof the network are learned, the rbfDataGen generator (RobnikˇSikonja, 2016) can sample from the normal distributions, deﬁned with obtained Gaussian kernels, to generate new instances.
Revised: Once the network’s parameters are learned, the rbfDataGen generator (Robnik- ˇ Sikonja, 2016) can sample from the normal distributions, deﬁned with obtained Gaussian kernels, to generate new instances.

Original: The TreeEnsemble generator (Robnik-ˇSikonja, 2019) builds a set of random trees (forest) that describe the data.
Revised: The TreeEnsemble generator (Robnik- ˇ Sikonja, 2019) builds a set of random trees (forest) that describe the data.

Original: When generating new instances, the generator traverses from the root to the leaves of a randomly chosen tree, setting values of features in the decision nodes on the way.
Revised: When generating new instances, the generator traverses from the root to the leaves of a randomly chosen tree, setting values of features in the decision nodes on the way.

Original: When reaching a leaf, it assumes that it has captured the dependencies between features.
Revised: When reaching a leaf, it assumes that it has captured the dependencies between features.

Original: The remaining features can, therefore, be generated independently according to the observed empirical distribution in this leaf.
Revised: Therefore, the remaining features can be generated independently according to the observed empirical distribution in this leaf.

Original: For each generated instance, all attributes can be generated in one leaf, or another tree can be randomly selected where unassigned feature values are ﬁlled in.
Revised: For each generated instance, all attributes can be generated in one leaf, or another tree can be randomly selected where unassigned feature values are ﬁlled in.

Original: By selecting different trees, different features are set in the interior nodes and leaves.
Revised: By selecting different trees, different features are set in the interior nodes and leaves.

Original: 2. R ELATED WORK ON ROBUSTNESS OF EXPLANATIONS
Revised: 2. R ELATED WORK ON ROBUSTNESS OF EXPLANATIONS

Original: The adversarial attacks on perturbation based explanation methods were proposed by Slack et al.
Revised: The adversarial attacks on perturbation based explanation methods were proposed by Slack et al.

Original: (2020), who show that LIME and SHAP are vulnerable due to the perturbation based sampling used.
Revised: (2020), who show that LIME and SHAP are vulnerable due to the perturbation based sampling used.

Original: We propose the solution to the exposed weakness in SHAP and IME based on better sampling using data generators adapted to the training set distribution.
Revised: We propose the solution to the exposed weakness in SHAP and IME based on better sampling using data generators adapted to the training set distribution.

Original: In general, the robustness of explanation methods has been so far poorly researched.
Revised: In general, the robustness of explanation methods has been so far poorly researched.

Original: There are claims that post-hoc explanation methods shall not be blindly trusted, as they can mislead users (deliberately or not) and disguise gender and racial discrimination (Lipton, 2016).
Revised: There are claims that post-hoc explanation methods shall not be blindly trusted, as they can mislead users (deliberately or not) and disguise gender and racial discrimination (Lipton, 2016).

Original: Selbst & Barocas (2018) and Kroll et al.
Revised: Selbst & Barocas (2018) and Kroll et al.

Original: (2017) showed that even if a model is completely transparent, it is hard to detect and prevent bias due to the existence of correlated variables.
Revised: (2017) showed that even if a model is completely transparent, it is hard to detect and prevent bias due to the existence of correlated variables.

Original: Speciﬁcally, for deep neural networks and images, there exist adversarial attacks on saliency map based interpretation of predictions, which can hide the model’s bias (Dombrowski et al., 2019; Heo et al., 2019; Ghorbani et al., 2019).
Revised: Speciﬁcally, for deep neural networks and images, there exist adversarial attacks on saliency map based interpretation of predictions, which can hide the model’s bias (Dombrowski et al., 2019; Heo et al., 2019; Ghorbani et al., 2019).

Original: Dimanov et al.
Revised: Dimanov et al.

Original: (2020) showed that a bias of a neural network could be hidden from post-hoc explanation methods by training a modiﬁed classiﬁer, that has similar performance to the original one but the importance of the chosen feature is signiﬁcantly lower.
Revised: (2020) showed that a bias of a neural network could be hidden from post-hoc explanation methods by training a modiﬁed classiﬁer that has similar performance to the original one, but the importance of the chosen feature is signiﬁcantly lower.

Original: The kNN-based explanation method, proposed by Chakraborty et al.
Revised: The kNN-based explanation method, proposed by Chakraborty et al.

Original: (2020), tries to overcomes the inadequate perturbation based sampling used in explanation methods by ﬁnding similar instances to the explained one in the training set instead of generating new samples.
Revised: (2020), tries to overcomes the inadequate perturbation based sampling used in explanation methods by ﬁnding similar instances to the explained one in the training set instead of generating new samples.

Original: This solution is inadequate for realistic problems as the problem space is not dense enough to get reliable explanations.
Revised: This solution is inadequate for realistic problems as the problem space is not dense enough to get reliable explanations.

Original: Our defence of current post-hoc methods is based on superior sampling, which has not yet been tried.
Revised: Our defense of current post-hoc methods is based on superior sampling, which has not yet been tried.

Revised: Saito et al.

Revised: (2020) use the neural CT-GAN model to generate more realistic samples for LIME and prevent the attacks described in Slack et al.

Original: We arealso not aware of any other defences against the adversarial attacks on post-hoc explanations.
Revised: We are not aware of any other defenses against the adversarial attacks on post-hoc explanations.

Original:  R OBUSTNESS THROUGH BETTER SAMPLING
Revised:  R OBUSTNESS THROUGH BETTER SAMPLING

Original: We propose the defence against the adversarial attacks on explanation methods that replaces the problematic perturbation sampling with a better one, thereby making the explanation methods more robust.
Revised: We propose the defense against the adversarial attacks on explanation methods that replaces the problematic perturbation sampling with a better one, thereby making the explanation methods more robust.

Original: We want to generate the explanation data in such a way that the attacker cannot determine whether an instance is sampled or obtained from the original data.
Revised: We want to generate the explanation data in such a way that the attacker cannot determine whether an instance is sampled or obtained from the original data.

Original: With an improved sampling, the adversarial model shown in Figure 1b shall notbe able to determine whether the instance x was generated by the explanation method, or it is the original instance the model has to label.
Revised: With an improved sampling, the adversarial model shown in Figure 1b shall not determine whether the instance x was generated by the explanation method, or it is the original instance the model has to label.

Original: With a better data generator, the adversarial model cannot adjust its output properly and the deception, described in Section 2.2, becomes ineffective.
Revised: With a better data generator, the adversarial model cannot adjust its output properly and the deception, described in Section 2.2, becomes ineffective.

Original: The reason for the described weakness of LIME and SHAP is inadequate sampling used in these methods.
Revised: The reason for the described weakness of LIME and SHAP is inadequate sampling used in these methods.

Original: Recall that LIME samples new instances by adding Gaussian noise to the normalized feature values.
Revised: Recall that LIME samples new instances by adding Gaussian noise to the normalized feature values.

Original: SHAP samples new instances from clusters obtained in advance with the k-means algorithm from the training data.
Revised: SHAP samples new instances from clusters obtained in advance with the k-means algorithm from the training data.

Original: Instead of using the Gaussian noise with LIME, we generate explanation samples for each instance with one of the three better data generators, MCD-VAE, rbfDataGen, or TreeEnsemble (see Section 2.3).
Revised: Instead of using the Gaussian noise with LIME, we generate explanation samples for each instance with one of the three better data generators, MCD-VAE, rbfDataGen, or TreeEnsemble (see Section 2.3).

Original: We call the improved explanation methods gLIME and gSHAP (g stands for generatorbased).
Revised: We call the improved explanation methods gLIME and gSHAP (g stands for generatorbased).

Original: Using better generators in the explanation methods, the decision function in the adversarial model will less likely determine which predicted instances are original and which are generated for the explanation purposes.
Revised: Using better generators in the explanation methods, the decision function in the adversarial model will less likely determine which predicted instances are original and which are generated for the explanation purposes.

Original: Concerning gLIME, we generate data in the vicinity of the given instance using MCD-VAE, as the
Revised: Concerning gLIME, we generate data in the vicinity of the given instance using MCD-VAE, as the

Original: LIME method builds a local model.
Revised: LIME method builds a local model.

Original: Using the TreeEnsemble and rbfDataGen generators, we do not generate data in the neighbourhood of the given instance but leave it to the proximity measure of the
Revised: Using the TreeEnsemble and rbfDataGen generators, we do not generate data in the neighborhood of the given instance but leave it to the proximity measure of the

Original: LIME method to give higher weights to instances closer to the explained one.
Revised: LIME method to give higher weights to instances closer to the explained one.

Original: In SHAP, the perturbation sampling replaces the values of hidden features in the explained instance with the values from the distribution set D .
Revised: In SHAP, the perturbation sampling replaces the values of hidden features in the explained instance with the values from the distribution set D .

Original: The problem of this approach is that it ignores the dependencies between features.
Revised: The problem with this approach is that it ignores the dependencies between features.

Original: For example, in a simple dataset with two features, house size and house price, let us assume that we hide the house price, but not the house size.
Revised: For example, in a simple dataset with two features, house size, and house price, let us assume that we hide the house price, but not the house size.

Original: These two features are not independent, because the price of a house increases with its size.
Revised: These two features are not independent because the price of a house increases with its size.

Original: Suppose we are explaining an instance that represents a large house.
Revised: Suppose we are explaining an instance that represents a large house.

Original: Disregarding the dependency, using the sampled set D ,
Revised: Disregarding the dependency, using the sampled set D ,

Original: SHAP creates several instances with a low price, as such instances appeared in the training set.
Revised: SHAP creates several instances with a low price, as such instances appeared in the training set.

Original: In this way, the sampled set contains many instances with a small price assigned to a large house, from which the attacker can determine that these instances were created in perturbation sampling and serve only for the explanation.
Revised: In this way, the sampled set contains many instances with a small price assigned to a large house, from which the attacker can determine that these instances were created in perturbation sampling and serve only for the explanation.

Original: In the proposed gSHAP, using the MCD-VAE and TreeEnsemble generators, the distribution set D is generated in the vicinity of the explained instance.
Revised: In the proposed gSHAP, using the MCD-VAE and TreeEnsemble generators, the distribution set D is generated in the vicinity of the explained instance.

Original: In the sampled instances, some feature values of the explained instance are replaced with the generated values, but the well-informed generators take into account dependencies between features detected in the original distribution.
Revised: In the sampled instances, some feature values of the explained instance are replaced with the generated values, but the well-informed generators consider dependencies between features detected in the original distribution.

Original: This will make the distribution of the generated instances very similar to the original distribution.
Revised: This will make the distribution of the generated instances very similar to the original distribution.

Original: In our example, the proposed approach generates new instances around the instance that represents a large house, and most of these houses will be large.
Revised: In our example, the proposed approach generates new instances around the instance representing a large house, and most of these houses will be large.

Original: As the trained generators capture thedependencies from the original dataset, these instances will also have higher prices.
Revised: As the trained generators capture the original dataset’s dependencies, these instances will also have higher prices.

Original: This will make it difﬁcult for the attacker to recognize the generated instances, used in explanations.
Revised: This will make it difﬁcult for the attacker to recognize the generated instances used in explanations.

Original: The advantage of generating the distribution set close to the observed instance is demonstrated in Appendix B.
Revised: The advantage of generating the distribution set close to the observed instance is demonstrated in Appendix B.

Original: The rbfDataGen generator does not support the generation of instances around a given instance.
Revised: The rbfDataGen generator does not support the generation of instances around a given instance.

Original: We, therefore, generate the sampled set based on the whole training set and not for each instance separately (we have this option also for TreeEnsemble).
Revised: Therefore, we generate the sampled set based on the whole training set and not for each instance separately (we have this option also for TreeEnsemble).

Original: This is worse than generating the distribution set around the explained instance but still better than generating it using the k-means sampling in SHAP.
Revised: This is worse than generating the distribution set around the explained instance but still better than generating it using the k-means sampling in SHAP.

Original: There are at least three advantages.
Revised: There are at least three advantages.

Original: First, the generated distribution set D can be larger.
Revised: First, the generated distribution set D can be larger.

Original: The size of the k-means distribution set cannot be arbitrarily large, because it is limited with the number of clusters in the training set.
Revised: The size of the k-means distribution set cannot be arbitrarily large because it is limited by the number of clusters in the training set.

Original: Second, the centroids of the clusters obtained by the k-means algorithm do not necessarily provide a good summary of the training set and may not be valid instances from training set distribution.
Revised: Second, the centroids of the clusters obtained by the k-means algorithm do not necessarily provide a good summary of the training set and may not be valid instances from training set distribution.

Original: They also do not capture well the density of instances (e.g., most of the data from the training set could be grouped in one cluster).
Revised: They also do not capture well the density of instances (e.g., most of the data from the training set could be grouped in one cluster).

Original: Third, using the proposed generators,
Revised: Third, using the proposed generators,

Original: SHAP becomes easier to use compared to the k-means generator, where users have to determine  the number of clusters, while the data generators we tested can be used in the default mode without parameters,
Revised: SHAP becomes easier to use compared to the k-means generator, where users have to determine [Table] the number of clusters, while the data generators we tested can be used in the default mode without parameters,

Original:  E VALUATION
Revised:  E VALUATION

Original: To evaluate the proposed improvements in the explanation methods, we ﬁrst present the used datasets in Section 4.1, followed by the experiments.
Revised: To evaluate the proposed improvements in the explanation methods, we ﬁrst present the used datasets in Section 4.1, followed by the experiments.

Original: We test the robustness of gLIME, gSHAP, and gIME against the adversarial models.
Revised: In Section 4.2, we test the robustness of gLIME, gSHAP, and gIME against the adversarial models.

Original: To be more realistic, we equipped the adversarial models with the[Table] same improved data generators we used in the explanation methods.
Revised: To be more realistic, we equipped the adversarial models with the same improved data generators we used in the explanation methods.

Original: It is reasonably to assume that attackers could also use better data generators when preparing their decision function as this would make their attacks much stronger.
Revised: It is reasonable to assume that attackers could also use better data generators when preparing their decision function, making their attacks much stronger.

Original: As the evaluation metric for the success of deception, we use the proportion of instances where the adversarial model deceived the explanation methods so that they did not detect sensitive attributes as being important in the prediction models.
Revised: As the evaluation metric for the success of deception, we use the proportion of instances where the adversarial model deceived the explanation methods so that they did not detect sensitive attributes as being important in the prediction models.

Revised: In Section 4.3, we test if enhanced generators produce different explanations than the original ones.

Revised: As the attacker might decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method, we test different thresholds of the decision function d from Equation (1) (currently set to 0 . 5 ).

Revised: We report on this analysis in Section 4.4.

Original: 4. S ENSITIVE DATASETS PRONE TO DECEPTION
Revised: 4. S ENSITIVE DATASETS PRONE TO DECEPTION

Original: Following (Slack et al., 2020), we conducted our experiments on three data sets from domains where a biased classiﬁer could pose a critical problem, such as granting a credit, prediction of crime recidivism, and prediction of violent crime rate.
Revised: Following (Slack et al., 2020), we conducted our experiments on three data sets from domains where a biased classiﬁer could pose a critical problem, such as granting a credit, predicting crime recidivism, and predicting the violent crime rate.

Original: The basic information on the data sets are presented in Table 1.
Revised: The basic information on the data sets is presented in Table 1.

Original: The statistics were collected after removing the missing values from the data sets, and before we encoded categorical features as one-hot-encoded vectors.
Revised: The statistics were collected after removing the missing values from the data sets and before we encoded categorical features as one-hot-encoded vectors.

Original: COMPAS (Correctional Offender Management Proﬁling for Alternative Sanctions) is a risk assessment used by the courts in some US states to determine the crime recurrence risk of a defendant.
Revised: COMPAS (Correctional Offender Management Proﬁling for Alternative Sanctions) is a risk assessment used by the courts in some US states to determine the crime recurrence risk of a defendant.

Original: The dataset (Angwin et al., 2016)) includes criminal history, time in prison, demographic data (age, gender, race), and COMPAS risk assessment of the individual.
Revised: The dataset (Angwin et al., 2016)) includes criminal history, time in prison, demographic data (age, gender, race), and COMPAS risk assessment of the individual.

Original: The dataset contains data of 6, defendants from Broward Couty, Florida.
Revised: The dataset contains data of 6, defendants from Broward Couty, Florida.

Original: The sensitive attribute in this dataset (the one  which the adversarial model will be biased on) isthe race.
Revised: The sensitive attribute in this dataset (the one on which the adversarial model will be biased) is race.

Original: African Americans, whom biased model associates with a high risk of recidivism, represent 51 .
Revised: African Americans, whom biased model associates with a high risk of recidivism, represent 51 .

Original: 4% of instances from the data set.
Revised: 4% of instances from the data set.

Original: The target variablein this set is the COMPAS score, which is divided into two classes: a high and low risk.
Revised: This set’s target variable is the COMPAS score, which is divided into two classes: a high and low risk.

Original: The majority class is the high risk, which represents 81 .
Revised: The majority class is the high risk, which represents 81 .

Original: 4% of the instances.
Revised: 4% of the instances.

Original: The German Credit dataset (German for the rest of the paper) from the UCI repository (Dua & Graff, 2019) includes ﬁnancial (bank account information, loan history, loan application information, etc.)
Revised: The German Credit dataset (German for the rest of the paper) from the UCI repository (Dua & Graff, 2019) includes ﬁnancial (bank account information, loan history, loan application information, etc.)

Original: and demographic data (gender, age, marital status, etc.)
Revised: and demographic data (gender, age, marital status, etc.)

Original: for 1,000 loan applicants.
Revised: for 1,000 loan applicants.

Original: A sensitive attribute in this data set is gender.
Revised: A sensitive attribute in this data set is gender.

Original: Men, whom the biased model associates with a low-risk, represent 69% of instances.
Revised: Men, whom the biased model associates with a low-risk, represent 69% of instances.

Original: The target variable is the loan risk assessment,which is divided into two classes: a good and  bad customer.
Revised: The target variable is the loan risk assessment, divided into two classes: a good and a bad customer.

Original: The majority class is a good customer, which represents 70% of instances.
Revised: The majority class is a good customer, which represents 70% of instances.

Original: Communities and Crime (CC) data set (Redmond & Baveja, 2002) contains data about the rate of violent crime in US communities in 1994 .
Revised: Communities and Crime (CC) data set (Redmond & Baveja, 2002) contains data about the rate of violent crime in US communities in 1994 .

Original: Each instance represents one community.
Revised: Each instance represents one community.

Original: The features are numerical and represent the percentage of the  populationin the community with a certain property or the average of population in the community.
Revised: The features are numerical and represent the percentage of the community’s population with a certain property or the average of population in the community.

Original: Features include socio-economic (e.g., education, house size, etc.) and demographic (race, age) data.
Revised: Features include socio-economic (e.g., education, house size, etc.) and demographic (race, age) data.

Original: The sensitive attribute is the percentage of the white race.
Revised: The sensitive attribute is the percentage of the white race.

Original: The biased model links instances where the percentageof whites is above average to a low rate of violent crime.
Revised: The biased model links instances where whites’ percentage is above average to a low rate of violent crime.

Original: The target variable is the rate of violent crime, which is divided into two classes: high and low. Both classes are equally represented in the data set.
Revised: The target variable is the rate of violent crime

Original: 4. R OBUSTNESS OF EXPLANATION METHODS
Revised: 4. R OBUSTNESS OF EXPLANATION METHODS

Original: To evaluate the robustness of explanation methods with added generators (i.e. gLIME, gSHAP, and gIME), we split the data into training and evaluation set in the ratio 90% : 10% .
Revised: To evaluate the robustness of explanation methods with added generators (i.e. gLIME, gSHAP, and gIME), we split the data into training and evaluation set in the ratio 90% : 10% .

Original: We used the same training set for  training of adversarial models and explanation methods.
Revised: We used the same training set for the training of adversarial models and explanation methods.

Original: We encoded categorical features as one-hot-encoded vectors.
Revised: We encoded categorical features as one-hot-encoded vectors.

Original: We simulated the adversarial attack for every combination of generators used in explanation methods and adversarial models (except for method IME, where we did not use rbfDataGen which is unable to generate instances in the neighbourhood of a given instance).
Revised: We simulated the adversarial attack for every combination of generators used in explanation methods and adversarial models (except for method IME, where we did not use rbfDataGen, which cannot generate instances in the neighborhood of a given instance).

Original: When testing SHAP, we used two variants of the TreeEnsemble generator for explanation: generating new instances around the explained instance and generating new instances according to the whole training set distribution.
Revised: When testing SHAP, we used two variants of the TreeEnsemble generator for explanation: generating new instances around the explained instance and generating new instances according to the whole training set distribution.

Original: In the LIME testing, we used only the whole training set variant of TreeEnsemble inside the explanation.
Revised: In the LIME testing, we used only the whole training set variant of TreeEnsemble inside the explanation.

Revised: In the

Original: In the IME testing, we only used the variant of Tree Ensemble that ﬁlls in the hidden values (called TEnsFillIn variant).
Revised: IME testing, we only used the variant of Tree Ensemble that ﬁlls in the hidden values (called TEnsFillIn variant).

Original: For trainingof the adversarial models, we used the whole training set based variant of TreeEnsemble, except for IME, where we used the TEnsFillIn variant.
Revised: For training the adversarial models, we used the whole training set based variant of TreeEnsemble, except for IME, where we used the TEnsFillIn variant.

Original: These choices reﬂect the capabilities of different explanation method and attempt to make both defence and attack realistic (as strong as possible).
Revised: These choices reﬂect the capabilities of different explanation method and attempt to make both defense and attack realistic (as strong as possible).

Revised: More details on training the decision model d inside the adversarial model can be found in Appendix D.

Original: In all cases, the biased model b (see Section 2.2) was a simple function depending only on the value of the sensitive feature.
Revised: In all cases, the biased model b (see Section 2.2) was a simple function depending only on the value of the sensitive feature.

Original: The unbiased model ψ depended only on the values of unrelated features.
Revised: The unbiased model ψ depended only on the values of unrelated features.

Original: The sensitive and unrelated features are shown in the right-hand side of Table 1.
Revised: The sensitive and unrelated features are shown on the right-hand side of Table 1.

Original: Features random and random2 were uniformly randomly generated from the { 0 , 1 } set.
Revised: Features random and random2 were uniformly randomly generated from the { 0 , 1 } set.

Original: On COMPAS and CC, we simulated two attacks, one with a single unrelated feature (the result of ψ depends only on the value of the unrelated feature 1), and another one with two unrelated features.
Revised: On COMPAS and CC, we simulated two attacks, one with a single unrelated feature (the result of ψ depends only on the value of the unrelated feature 1), and another one with two unrelated features.

Original: For every instance in the evaluation set, we recorded the most important feature according to the used explanation method (i.e. the feature with the largest absolute value of the contribution as determined by the explanation method).
Revised: For every instance in the evaluation set, we recorded the most important feature according to the used explanation method (i.e. the feature with the largest absolute value of the contribution as determined by the explanation method).

Original: The results are shown as a heatmap in Figure 2.
Revised: The results are shown as a heatmap in Figure 2.

Original: The green colour means that the explanation method was deceived in less than 30% of cases, and the red means that it was deceived in more than 70% of cases.
Revised: The green color means that the explanation method was deceived in less than 30% of cases (i.e. the sensitive feature was recognized as the most important one in more than 70 % of the cases as the scale on the heatmap suggests), and the red means that it was deceived in more than 70% of cases (the sensitive feature was recognized as the most important one in less than 30 % of the cases as the scale on the heatmap suggests).

Original: We consider deception successful if the sensitive feature was not recognized as the most important by the explanation method.
Revised: We consider deception successful if the sensitive feature was not recognized as the most important by the explanation method (the sensitive features are the only relevant features in biased models b ).

Original: The gLIME method is more robust with the addition of rbfDataGen and TreeEnsemble than LIME and less robust with the addition of MCD-VAE.
Revised: The gLIME method is more robust with the addition of rbfDataGen and TreeEnsemble than LIME and less robust with the addition of MCD-VAE.

Original: This suggests that parameters for MCD-VAE were not well-chosen, and this pattern repeats for other explanation methods.
Revised: This suggests that parameters for MCD-VAE were not well-chosen, and this pattern repeats for other explanation methods.

Original: Both TreeEnsemble and rbfDataGen make gLIME considerably more robust on COMPAS and German datasets, but not on the CC dataset.
Revised: Both TreeEnsemble and rbfDataGen make gLIME considerably more robust on COMPAS and German datasets, but not on the CC dataset.

Original: We believe the reason for that is that features in CC are strongly interdependent, and many represent the same attribute as a fraction of value, e.g., we have the share of the white population, the share of the Asian population, and so on.
Revised: We believe the reason for that is that features in CC are strongly interdependent, and many represent the same attribute as a fraction of value, e.g., we have the share of the white population, the share of the Asian population, and so on.

Original: This interdependence dictates that all fractions have to add up to 1 .
Revised: This interdependence dictates that all fractions have to add up to 1 .

Original: The data generators are unlikely to capture so strong conditional dependencies, but the  decision functionin the adversarial model is likely to approximate it.
Revised: The data generators are unlikely to capture such strong conditional dependencies, but the adversarial model’s decision function is likely to approximate it.

Original: The gSHAP method is most robust when using the TreeEnsemble generator, but it shows less robust behaviour than the gLIME method.
Revised: The gSHAP method is most robust when using the TreeEnsemble generator, but it shows less robust behavior than the gLIME method.

Original: The reason for that could be in the feature value replacement strategy used by the SHAP and gSHAP methods, which change only some of the feature’s values with the help of the distribution set.
Revised: The reason for that could be in the feature value replacement strategy used by the SHAP and gSHAP methods, which change only some of the feature’s values with the help of the distribution set.

Original: This can lead to out of distribution instances if the instances in the distribution set are not close enough to the explained instance.
Revised: This can lead to out-of-distribution instances if the instances in the distribution set are not close enough to the explained instance.

Original: With gLIME, we generate complete instances which are more likely to be in the distribution of the training set.
Revised: With gLIME, we generate complete instances that are more likely to be in the distribution of the training set.

Original: IME is quite robust even with the perturbation sampling, as we expected.
Revised: IME is quite robust even with the perturbation sampling, as we expected.

Original: This suggests that IME is the most robust of all three tested explanation methods and shall be considered the  methodof choice in sensitive situations.
Revised: This suggests that IME is the most robust of all three tested explanation methods and shall be considered the chosen method in sensitive situations.

Original: The gIME results when using the TreeEnsemble generator (TEnsFillIN variant) are comparable to theresults of the original IME variant.
Revised: The gIME results when using the TreeEnsemble generator (TEnsFillIn variant) are comparable to the original IME variant results.

Original: This suggests that sampling from a smaller data set, which represents the neighbourhood of the explained instance does not decrease the robustness of the method.
Revised: This suggests that sampling from a smaller data set, which represents the neighborhood of the explained instance, does not decrease the method’s robustness. 4. C OMPARING EXPLANATIONS OF ORIGINAL AND MODIFIED METHODS

Revised: We checked if improved data generators affect explanations in a non-adversary environment.

Revised: We compared the explanations of original and modiﬁed methods for four different classiﬁcation models on three different datasets.

Revised: As a measure of differences, we use the mean of the absolute difference between the original and enhanced explanations over the whole evaluation set.

Original: C ONCLUSIONS
Revised: Details of these experiments are reported in Appendix F, below we shortly summarize the results, contained in Table The differences between original LIME and gLIME explanations are considerable (see the top part of Table 4). This is not surprising since LIME ﬁts local linear models in its local explanations, which can strongly differ even for small perturbations of the model’s input. The differences in

Revised: SHAP and IME explanations are small (the average MAD is almost negligible).

Revised: We can conclude that explanations of gSHAP and gIME are not signiﬁcantly different from SHAP and IME in the non-adversary environment.

Revised: 4. B EHAVIOR OF ADVERSARIAL CLASSIFIER WITH DIFFERENT PREDICTION THRESHOLDS

Revised: The attacker might be wary of being discovered by the regulator and decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method.

Revised: We express different levels of attacker’s conservatism by changing the decision function threshold d from Equation (1), where it is currently set to 0 .

Revised: We tested the robustness of modiﬁed explanation methods on the COMPAS dataset with adversarial models that use different certainty thresholds for deploying the biased classiﬁer (simulating different levels of aggressiveness).

Revised: For results and a more detailed description of the experiment, see Appendix G. Below, we shortly summarize the results.

Revised: Even with different thresholds, gIME is still the most robust of the three explanation methods, and treeEnsemble still gives the best results as the data generator.

Revised: The sensitive feature is recognized as the most important more often when the threshold is lower, which is expected as in this case the adversarial model behaves more aggressively.

Revised: While the percentage of the instances on which the racist behavior of the adversarial model is recognized drops with higher values of the thresholds, it remains high enough to warn the regulator about the problematic behavior of the prediction model (especially in the case of gSHAP and gIME using treeEnsemble as data generator).

Revised: We can conclude that the enhanced explanation methods remain robust enough, even with more conservative adversarial models.

Revised:  C ONCLUSIONS

Original: We presented the defence against adversarial attacks on explanation methods.
Revised: We presented the defense against adversarial attacks on explanation methods.

Original: The attacks exploit the shortcomings of perturbation sampling in post-hoc explanation methods.
Revised: The attacks exploit the shortcomings of perturbation sampling in post-hoc explanation methods.

Original: This sampling used in these methods produces instances too different from the original distribution of the training set.
Revised: This sampling used in these methods produces instances too different from the original distribution of the training set.

Original: This allows unethical owners of biased prediction models to detect which instances are meant for explanation and label them in an unbiased way.
Revised: This allows unethical owners of biased prediction models to detect which instances are meant for explanation and label them in an unbiased way.

Original: We replaced the perturbation sampling with data generators that better capture the distribution of a given data set.
Revised: We replaced the perturbation sampling with data generators that better capture the distribution of a given data set.

Original: This prevents the detection of instances used in explanation and disarms attackers.
Revised: This prevents the detection of instances used in explanation and disarms attackers.

Original: We have shown that the modiﬁed gLIME and gSHAP explanation methods, which use better data generators, are more robust than the original variants, while IME is already quite robust. 
Revised: We have shown that the modiﬁed gLIME and gSHAP explanation methods, which use better data generators, are more robust than the original variants, while IME is already quite robust. The difference in explanation values between original and enhanced gSHAP and gIME is negligible, while for gLIME, it is considerable.

Original: Our preliminary results in Appendix C show that using the TreeEnsemble generator, the gIME methods converges faster and requires from 30-50% fewer samples.
Revised: Our preliminary results in Appendix C show that using the TreeEnsemble generator, the gIME method converges faster and requires from 30-50% fewer samples.

Original: The effectiveness of the proposed defence depends on the choice of the data generator and its parameters.
Revised: The effectiveness of the proposed defense depends on the choice of the data generator and its parameters.

Original: While the TreeEnsemble generator turned out the most effective in our evaluation, in practice several variants might need toto be tested to get a robust explanation method.
Revised: While the TreeEnsemble generator turned out the most effective in our evaluation, in practice, several variants might need to be tested to get a robust explanation method.

Original: Inspecting authorities  be aware of the need for good data generators and makethe access to training data of sensitive prediction models a legal requirement.
Revised: Inspecting authorities shall be aware of the need for good data generators and make access to training data of sensitive prediction models a legal requirement.

Original: Luckily, even a few non-deceived instances would be enough to raise an alarm aboutthe unethical model
Revised: Luckily, even a few non-deceived instances would be enough to raise the alarm about unethical models.

Original: This work opens a range of possibilities for further research.
Revised: This work opens a range of possibilities for further research.

Original: The proposed defence and attacks shall be tested on other data sets with  different number and types of features, with missing data and on different types of problems such as text, images, and graphs.
Revised: The proposed defense and attacks shall be tested on other data sets with a different number and types of features, with missing data, and on different types of problems such as text, images, and graphs.

Original: The work on useful generators shall be extended to ﬁnd time-efﬁcient generators with easy to set parameters and  ability to generate new instances in the vicinity of a given one.
Revised: The work on useful generators shall be extended to ﬁnd time-efﬁcient generators with easy to set parameters and the ability to generate new instances in the vicinity of a given one.

Original: Generative adversarial networks (Goodfellow et al., 2014) may be a promising line of research.
Revised: Generative adversarial networks (Goodfellow et al., 2014) may be a promising line of research.

Original: The TreeEnsemble generator, currently written in pure R, shall be rewritten in a more efﬁcient programming language.
Revised: The TreeEnsemble generator, currently written in pure R, shall be rewritten in a more efﬁcient programming language.

Original: The MCD-VAE generator shall be studied to allow automatic selection of reasonable parameters for a given dataset.
Revised: The MCD-VAE generator shall be studied to allow automatic selection of reasonable parameters for a given dataset.

Original: In SHAP sampling, we could ﬁrst ﬁx the values of the features we want to keep, and the values of the others would be generated using the TreeEnsemble generator.
Revised: In SHAP sampling, we could ﬁrst ﬁx the values of the features we want to keep, and the values of the others would be generated using the TreeEnsemble generator.

Original: A D ETAILS ON POST - HOC EXPLANATION METHODS
Revised: A D ETAILS ON POST - HOC EXPLANATION METHODS

Original: For the sake of completeness, we present further details on the explanation methods LIME
Revised: For the sake of completeness, we present further details on the explanation methods LIME

Original: (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), and IME (ˇStrumbelj & Kononenko, 2013).
Revised: (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), and IME ( ˇ Strumbelj & Kononenko, 2013).

Original: Their complete description can be found in the above-stated references.
Revised: Their complete description can be found in the above-stated references.

Original: In our exposition of the explanation methods, we denote with f the predictive model, with x the instance we are explaining, and with n the number of features describing x .
Revised: In our exposition of the explanation methods, we denote with f the predictive model, with x the instance we are explaining, and with n the number of features describing x .

Original: A. LIME
Revised: A. LIME

Original: Explanations of instances with the LIME method is obtained with an interpretable model g .
Revised: Explanations of instances with the LIME method is obtained with an interpretable model g .

Original: The model g has to be both locally accurate (so that it can obtain correct features contributions) and simple (so that it is interpretable).
Revised: The model g has to be both locally accurate (so that it can obtain correct feature contributions) and simple (so that it is interpretable).

Original: The explanation of the instance x for the predictive model f , obtained with the LIME method, is deﬁned with the following equation:
Revised: The explanation of the instance x for the predictive model f , obtained with the LIME method, is deﬁned with the following equation:

Original: [Equation]
Revised: [Equation]

Original: where Ω( g ) denotes the measure of complexity for interpretable model, π x ( z ) denotes the proximity measure between x and generated instances z , L ( f, g, π x ) denotes the measure of local ﬁdelity of interpretable model g to the prediction model f , and G denotes the set of interpretable models.
Revised: where Ω( g ) denotes the measure of complexity for interpretable model, π x ( z ) denotes the proximity measure between x and generated instances z , L ( f, g, π x ) denotes the measure of local ﬁdelity of interpretable model g to the prediction model f , and G denotes the set of interpretable models.

Original: We use the linear version of the LIME method, where G represents the set of linear models.
Revised: We use the linear version of the LIME method, where G represents the set of linear models.

Original: With x (cid:48)
Revised: With x (cid:48)

Original: we denote the normalized presentation of instance x , i.e. the numerical attributes have their mean set to 0 and variance to 1, and the categorical attributes contain value 1 if they have the same value as the exoplained instance and 0 otherwise.
Revised: we denote the normalized presentation of instance x , i.e. the numerical attributes have their mean set to 0 and variance to 1, and the categorical attributes contain value 1 if they have the same value as the exoplained instance and 0 otherwise.

Original: The proximity function π x is deﬁned on the normalized instances (hence we use the notation π x (cid:48) ), and uses the exponential kernel: π x (cid:48)
Revised: The proximity function π x is deﬁned on the normalized instances (hence we use the notation π x (cid:48) ), and uses the exponential kernel: π x (cid:48)

Original: ( z (cid:48) )
Revised: ( z (cid:48) )

Original: =
Revised: =

Original: e d ( x (cid:48) ,z (cid:48) ) 
Revised: e d ( x (cid:48) ,z (cid:48) ) [Equation]

Original: σ, where d ( x (cid:48) , z (cid:48) ) denotes a distance measure between x (cid:48) and z (cid:48) .
Revised: [Equation] where d ( x (cid:48) , z (cid:48) ) denotes a distance measure between x (cid:48) and z (cid:48) .

Original: The local ﬁdelity measure L from
Revised: The local ﬁdelity measure L from

Original: Equation (2) is deﬁned as:
Revised: Equation (2) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where Z denotes the set of samples.
Revised: where Z denotes the set of samples.

Original: In LIME, each generated sample is obtained by adding Gaussian noise to each feature of x (cid:48) independently.
Revised: In LIME, each generated sample is obtained by adding Gaussian noise to each feature of x (cid:48) independently.

Original: Using linear models as the set of interpretable models, LIME is relatively fast but may produce poor explanations for instances close to complex decision boundaries.
Revised: Using linear models as the set of interpretable models, LIME is relatively fast but may produce poor explanations for instances close to complex decision boundaries.

Original: A. SHAP
Revised: A. SHAP

Original: We refer to SHAP as the method called Kernel SHAP by (Lundberg & Lee, 2017).
Revised: We refer to SHAP as the method called Kernel SHAP by (Lundberg & Lee, 2017).

Original: SHAP essentially estimates Shapley values using the approach of LIME, which means that the calculation of explanations is fast due to the use of local linear models computed with the weighted least squares algorithm.
Revised: SHAP essentially estimates Shapley values using LIME’s approach, which means that the calculation of explanations is fast due to the use of local linear models computed with the weighted least squares algorithm.

Original: The explanation of the instance x with the SHAP method are feature contributions φ
Revised: The explanation of the instance x with the SHAP method are feature contributions φ

Original: i , i = 1 , 2 , ..., n that are coefﬁcients of the linear model g ( x (cid:48) )
Revised: i , i = 1 , 2 , ..., n that are coefﬁcients of the linear model g ( x (cid:48) )

Original: =
Revised: =

Original: φ 0 + (cid:80) ni
Revised: φ 0 + (cid:80) ni

Original: =1 φ i · x (cid:48)
Revised: =1 φ i · x (cid:48)

Original: i , obtained with LIME, where x (cid:48) i ∈ { 0 , 1 } for i ∈ { 1 , 2 , ..., n } .
Revised: i , obtained with LIME, where x (cid:48) i ∈ { 0 , 1 } for i ∈ { 1 , 2 , ..., n } .

Original: As LIME does not compute Shapley values, this property is assured with the proper selection of functions Ω( g ) , π x (cid:48)
Revised: As LIME does not compute Shapley values, this property is assured with the proper selection of functions Ω( g ) , π x (cid:48)

Original: ( z (cid:48) ) , and L ( f, g, π x (cid:48) ) .
Revised: ( z (cid:48) ) , and L ( f, g, π x (cid:48) ) .

Original: Let us ﬁrst deﬁne the function
Revised: Let us ﬁrst deﬁne the function

Original: h x ( z (cid:48) ) that maps from the { 0 , 1 } n to the original feature space.
Revised: h x ( z (cid:48) ) that maps from the { 0 , 1 } n to the original feature space.

Original: The function h x is deﬁned implicitly with the equation f ( h x ( z (cid:48) ))
Revised: The function h x is deﬁned implicitly with the equation f ( h x ( z (cid:48) ))

Original: =
Revised: =

Original: E [ f ( z ) | z i = x i ∀ i ∈ { j ; z (cid:48)
Revised: E [ f ( z ) | z i = x i ∀ i ∈ { j ; z (cid:48)

Original: j = 1 } ] .
Revised: j = 1 } ] .

Original: This is the expected value of the model f when we do not know the values of features at indices where z (cid:48) equals 0 (these values are hidden).
Revised: This is the expected value of the model f when we do not know the values of features at indices where z (cid:48) equals 0 (these values are hidden).

Original: Functions Ω( g ) , π x (cid:48)
Revised: Functions Ω( g ) , π x (cid:48)

Original: ( z (cid:48) ) and L ( f, g, π x (cid:48) )
Revised: ( z (cid:48) ) and L ( f, g, π x (cid:48) )

Original: that enforce the computation of Shapley values are:
Revised: that enforce the computation of Shapley values are:

Original: [Equation]
Revised: [Equation]

Original: where | z (cid:48) | denotes the number of nonzero features of z (cid:48) and Z ⊆ 2 { 0 , 1 } n .
Revised: where | z (cid:48) | denotes the number of nonzero features of z (cid:48) and Z ⊆ 2 { [Equation] , 1 } n .

Original: The main purpose of the sampling in this method is to determine the value f ( h x ( z (cid:48) )) because in general predictive models cannot work with hidden values.
Revised: The main purpose of the sampling in this method is to determine the value f ( h x ( z (cid:48) )) because in general predictive models cannot work with hidden values.

Original: To determine f ( h x ( z (cid:48) )) , SHAP uses the distribution set D which we obtain from the training set.
Revised: To determine f ( h x ( z (cid:48) )) , SHAP uses the distribution set D which we obtain from the training set.

Original: For D , SHAP takes the centroids of clusters, obtained by  k-means clustering algorithm on the training set.
Revised: For D , SHAP takes the centroids of clusters obtained by the k-means clustering algorithm on the training set.

Original: The number of clusters is set by the user.
Revised: The number of clusters is set by the user.

Original: Value of f ( h x ( z (cid:48) )) is determined by the following sampling:
Revised: Value of f ( h x ( z (cid:48) )) is determined by the following sampling:

Original: [Equation]
Revised: [Equation]

Original: where x [ x i =
Revised: where x [ x i =

Original: d i ,z (cid:48)
Revised: d i ,z (cid:48)

Original: i =0] denotes instance x with features that are 0 in z (cid:48) being set to the feature values from d .
Revised: i =0] denotes instance x with features that are 0 in z (cid:48) being set to the feature values from d .

Original: A. IME
Revised: A. IME

Original: The explanation of x with the method IME are feature contributions φ
Revised: The explanation of x with the method IME are feature contributions φ

Original: i , i = 1 , 2 , ..., n .
Revised: i , i = 1 , 2 , ..., n .

Original: ˇStrumbelj &
Revised: ˇStrumbelj &

Original: Kononenko (2013) shoved that Shapley values are the only solution that takes into account contributions of interactions for every subset of features in a fair way.
Revised: Kononenko (2013) shoved that Shapley values are the only solution that takes into account contributions of interactions for every subset of features in a fair way.

Original: The i -th feature contribution can be calculated with the following expresiion:
Revised: The i -th feature contribution can be calculated with the following expresiion:

Original: [Equation]
Revised: [Equation]

Original: where S n denotes a group of permutations of n elements, X denotes the training set instances,
Revised: where S n denotes a group of permutations of n elements, X denotes the training set instances,

Original: Pre i ( π ) represents the set of indices that precedes i in the permutation π , i.e. Pre i ( π )
Revised: Pre i ( π ) represents the set of indices that precedes i in the permutation π , i.e. Pre i ( π )

Original: = { j ; π ( j ) <
Revised: = { j ; π ( j ) <

Original: π ( i ) } ).
Revised: π ( i ) } ).

Original: Let p ( w ) denote the probability of the instance w in X and let w [ formula ] denote the instance w with some of its features values changed according to the formula .
Revised: Let p ( w ) denote the probability of the instance w in X and let w [ formula ] denote the instance w with some of its features values changed according to the formula .

Original: To calculate φ i ( x ) , we have to go through |X| · n ! iterations, which can be slow.
Revised: To calculate φ i ( x ) , we have to go through |X| · n ! iterations, which can be slow.

Original: Therefore, the method IME uses the following sampling.
Revised: Therefore, the method IME uses the following sampling.

Original: The sampling population of i -th feature is V π,w =
Revised: The sampling population of i -th feature is V π,w =

Original: ( f ( w [ w i
Revised: ( f ( w [ w i

Original: =
Revised: =

Original: x
Revised: x

Original: i ,i ∈
Revised: i ,i ∈

Original: Pre i ( π ) ∪{ i } ] ) − f ( w [ w i = x
Revised: Pre i ( π ) ∪{ i } ] ) − f ( w [ w i = x

Original: i ,i ∈
Revised: i ,i ∈

Original: Pre i ( π )] )) for every combination of the permutation π and instance w .
Revised: Pre i ( π )] )) for every combination of the permutation π and instance w .

Original: IME draws m i samples V 1 , ..., V m i at random with repetition.
Revised: IME draws m i samples V 1 , ..., V m i at random with repetition.

Original: The estimate of φ i ( x ) is deﬁned with the equation:
Revised: The estimate of φ i ( x ) is deﬁned with the equation:

Original: [Equation]
Revised: [Equation]

Original: Contrary to SHAP, IME does not use approximation with linear models which computethe contributions of all features at once but has to compute the Shapley values by averaging over  large enough sample for each feature separately.
Revised: Contrary to SHAP, IME does not use approximation with linear models, which compute all features’ contributions at once but has to compute the Shapley values by averaging over a large enough sample for each feature separately.

Original: This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency in the space of normalized features.
Revised: This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency in the space of normalized features.

Original: B D EMONSTRATION OF BETTER SAMPLING IN SHAP
Revised: B D EMONSTRATION OF BETTER SAMPLING IN SHAP

Original: The graphs in Figure 3 show the PCA based 2D space of the evaluation part of the COMPAS dataset (see Section 4 for the dataset description).
Revised: The graphs in Figure 3 show the PCA based 2D space of the evaluation part of the COMPAS dataset (see Section 4 for the dataset description).

Original: The left-hand side shows the SHAP-generated sampled instances using the k-means algorithm (14 clusters determined by the silhouette score (Rousseeuw (1987)).
Revised: The left-hand side shows the SHAP-generated sampled instances using the k-means algorithm (14 clusters determined by the silhouette score (Rousseeuw (1987)).

Original: The sample produced with the MCD-VAE generator in gSHAP is shown on the right-hand side.
Revised: The sample produced with the MCD-VAE generator in gSHAP is shown on the right-hand side.

Original: This sample is much more similar to the original distribution compared to the SHAP sampling.
Revised: This sample is much more similar to the original distribution compared to the SHAP sampling.

Original: C I MPROVED IME CONVERGENCE RATE WITH THE T REE E NSEMBLE GENERATOR
Revised: C I MPROVED IME CONVERGENCE RATE WITH THE T REE E NSEMBLE GENERATOR

Original: Preliminary, we tested howthe use of better generators affects the convergence rate of the IME explanation method.
Revised: Preliminary, we tested how better generators affect the convergence rate of the IME explanation method.

Original: The preliminary results in Table 2 show a signiﬁcant reduction in the number of needed samples and a slight increase in the error compared to the original perturbation sampling.
Revised: The preliminary results in Table 2 show a signiﬁcant reduction in the number of needed samples and a slight increase in the error compared to the original perturbation sampling.

Original: Note that the error measure deployed is biased in favor of the perturbation sampling, which was used to determine the gold standard.
Revised: Note that the error measure deployed is biased in favor of the perturbation sampling, which was used to determine the gold standard.

Original: COMPAS dataset Error # samples Classiﬁer Perturb.
Revised: This was determined with the sampling population’s variance, as described in ˇStrumbelj & Kononenko (2010).

Original: TEnsFillIn Perturb. TEnsFillIn Reduction %
Revised: D T RAINING DISCRIMINATOR FUNCTION OF THE ATTACKER

Original: CA %
Revised: Details of training attacker’s decision models d (see Figure 1b) is described in Algorithms 1, 2, and 3. We used a slightly different algorithm for each of the three explanation methods, LIME, SHAP,

Original: [Table]
Revised: [Table]

Revised: compared to perturbations.

Revised: The

Revised: CA stands for the classiﬁcation accuracy of the explained classiﬁer on the evaluation set.

Original: Linear SVM 0.
Revised: and IME, as each method uses a different sampling. Algorithms ﬁrst create out-of-distribution instances by method-speciﬁc sampling. The training sets for decision models are created by labeling the created instances with 0 ; the instances from sample S (to which the attacker has access) from distribution X dist are labeled with 1 . Finally, the machine learning model dModel is trained on this training set and returned as d .

Original: Random forest 0.
Revised: In our experiments, we used random forest classiﬁer as dModel and the training part of each evaluation dataset as S . E H EATMAPS AS TABLES We present the information contained in Figure 2 in a more detailed tabular form in Table 3. F C OMPARING EXPLANATIONS OF ORIGINAL AND MODIFIED METHODS

Original: Neural network 0.
Revised: We check if improved data generators affect explanations in non-adversary environment. We split the dataset into training and evaluation set in the ratio 90% : 10% , and trained four classiﬁers from Python scikit-learn (Pedregosa et al. (2011)) library: Gaussian naive Bayes, linear SVC (SVM for classiﬁcation), random forest, and neural network.

Original: German dataset
Revised: We explained the predictions of each classiﬁer on the evaluation set with every combination of explanation methods and generators used in the adversarial attack experiments. For instances in the evaluation set, we measured the mean absolute if gen == KMeans or gen == rbfDataGen or gen == treeEnsemble then if gen == MCD − V AE or gen ==

Original: Error # samples
Revised: treeEnsembleFill then w ← gen.newdata (1 , x ) // Generate an instance w in the vicinity of x endelse // KMeans, treeEnsemble or rbfDataGen w ← take a random instance from D end M ← choose a random subset of { 1 , 2 , ..., len ( x ) } // Choose random features x [ M ] ← w [ M ] // Replace the values of chosen features in x with values from w as in SHAP method X ← X ∪ ( x, 0) //

Original: Classiﬁer
Revised: Add out of distribution instance endfor i = 1 to m do // Add instances from distribution X ← X ∪ ( x i , 1) end d ← dModel.fit ( X ) // Fit model dModel to set X and save it in d return d difference (MAD) of modiﬁed explanation methods, deﬁned with the following equation:

Original: Perturb.
Revised: [Equation]

Original: TEnsFillIn
Revised: where φ geni ( x ) and φ i ( x ) represent the explanations of i -th feature returned by the modiﬁed and original explanation method, respectively (recall that n denotes the number of features in the data set). We experimented on three datasets. The COMPAS dataset is described in Section 4.1. In addition to that, we used synthetic dataset condInd from Robnik- ˇSikonja & Kononenko (2008), and Ionosphere do

Original: Perturb. TEnsFillIn
Revised: X ← X ∪ ( x i , 1) //

Original: Reduction % CA % [Table] [Table]
Revised: Add an instance from distribution for do /

Original: Linear
Revised: Add nSamples distribution instances w ← gen.newdata (1 , x i ) // Generate an instance w in the vicinity of x i b 1 ← w // First out of distribution instance b 2 ← w // Second out of distribution instance π ← choose a random permutation from S i )

Original: [Table]
Revised: .e

Original: [Table] [Table] [Table]
Revised: a random permutation of x ’s features idx ← choose a random number from { 1 , 2 , ..., len ( x i ) }

Original: [Table]
Revised: M 1 ← { k ∈ { 1 , 2 , ..., len ( x i ) } , π ( k ) <

Original: network    /
Revised: π ( idx ) } //

Revised: Features that precede idx in permutation π

Original: [Table] dataset Error # samples
Revised: M 2 ← M 1 ∪ { idx } b 1 ← x i [ M 1 ] //

Original: Classiﬁer
Revised: Vector b 1 as

Original: Perturb. TEnsFillIn
Revised: IME method (ˇStrumbelj

Original: Perturb.
Revised: Kononenko (2013)) b 2 ← x i [ M 2 ] // Vector b 2 as in IME method (ˇStrumbelj Kononenko (2013))

Original: TEnsFillIn
Revised: X ← X ∪ { ( b 1 ,

Original: Reduction %
Revised: , ( b 2 , 0) } // Add out of distribution instances end end d ← dModel.fit ( X

Original: CA %
Revised: // Fit model dModel to set X save it in d return d dataset from UCI repository (Dua & Graff (2019)). Both datasets represent a binary classiﬁcation problem. Apart from the target variable, condInd consists of 8 binary features, while Ionosphere consists of 34 numerical attributes. The condInd datasets contains 2000 instances and Ionosphere contains 351 instances.

Original: [Table] [Table] [Table]
Revised: The results are shown in Table 4.

Original: Linear
Revised: The differences between original LIME and gLIME explanations are considerable (see the top table). This is not surprising since LIME ﬁts local linear models in its local explanations, which can strongly differ even for small perturbations of the model’s input.

Original: [Table]
Revised: SHAP and IME explanations are very similar (the average MAD is almost negligible).

Original: [Table]
Revised: We can conclude that explanations of gSHAP and gIME are not signiﬁcantly different from SHAP and IME in the non-adversary environment. G A DVERSARIAL CLASSIFIERS WITH DIFFERENT PREDICTION THRESHOLDS In a realistic scenario, the attacker might be wary of being discovered by the regulator and decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method.

Original: [Table]
Revised: In these circumstances, the number of times when the sensitive variable is used inside the model will also depend on how aggressively the biased classiﬁer is used. This reﬂection leads to another metric for the defense’s success, which measures the proportion of instances when the sensitive variable is identiﬁed when it is being used, i.e. at different levels of conservatism. We express different levels of attacker’s conservatism by changing the threshold of the decision function d . The output of the adversarial model e t , which uses threshold t is now deﬁned with the equation:

Original: [Table]
Revised: [Equation]

Original: forest 0.
Revised: where d prob ( x ) denotes the probability that x is from distribution X dist , returned by decision model d .

Original: [Table]
Revised: Threshold t therefore represents the required certainty of the attacker that given instance is from X dist before it deploys the biased classiﬁer. Larger values of t will result in biased classiﬁer being deployed fewer times but with larger certainty that it has to be used.

Original: [Table]
Revised: If we use t = 0 .

Original: [Table]
Revised: 5 in Equation (8), we get the adversarial model that behaves identically to the one, described with Equation (1) in Section 2.2. In this experiment, we used only the COMPAS dataset, which we split into the training and evaluation set in ratio 90% : 10% . We performed the same experiment as in section 4.2, but with different thresholds for the adversarial models.

Original: network 0.
Revised: We used ﬁve different thresholds: 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 and 0 .

Original: [Table]
Revised: We ﬁrst checked how different thresholds affect the deployment of the biased model. For each generator used in the adversarial model’s training and the certainty threshold used by the adversarial model, we measured the proportion of instances from the evaluation set, where the biased classiﬁer was deployed. The results are shown in Table 5. As expected, the percentage of instances where the biased classiﬁer is deployed decreases with increasing t .

Revised: The biased classiﬁer is deployed less frequently when using treeEnsemble and rbfDataGen in its training, which is also expected, as these two generators give the best results inside the explanation methods.

Revised: This means that for these generators, it is more difﬁcult for an attacker to distinguish between the actual instances and generated instances.

Original: [Table]
Revised: The biased classiﬁer is deployed the most often in the case of the LIME adversarial model and least often in the SHAP adversarial model. Next, we tested the proportion of instances where the sensitive feature (in case of COMPAS dataset this is race ) was recognized as the most important feature by the explanation method on two different subsets of instances: instances from the evaluation set on which the biased classiﬁer was deployed and the whole evaluation set. We measured these numbers for all combinations of thresholds and explanation methods. The results are shown in Tables 6 (gLIME), 7 (gSHAP) and 8 (gIME) for gLIME, gSHAP, and gIME, respectively.

Revised: We can observe the same pattern as in section 4.2: gIME is still the most robust, and the treeEnsemble generator still gives the best results.

Revised: The sensitive attribute is recognized as the most important feature more often when the decision threshold values are lower, which is expected as in these cases, the adversarial model behaves more aggressively.

Revised: Whilst the percentage of the instances on which the racist behavior of the adversarial model is recognized drops with higher thresholds, it remains high enough to warn the regulator about the problematic behavior of the prediction model (especially in the case of gSHAP and gIME using treeEnsemble as data generator).

Revised: From that, we can conclude that modiﬁed explanation methods remain robust enough, even with more conservative adversaries.