F OOL SHAP WITH S TEALTHILY B IASED S AMPLING .
Anonymous authors
Paper under double-blind review
A BSTRACT
SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a speciﬁc input versus a background distribution.
Recent studies have shown that they can be manipulated by malicious adversaries to produce arbitrary desired explanations.
However, existing attacks focus solely on altering the black-box model itself.
In this paper, we propose a complementary family of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution.
In the context of fairness audit, we show that our attack can reduce the importance of a sensitive feature when explaining the difference in outcomes between groups while remaining undetected.
More precisely, experiments performed on real-world datasets showed that our attack could yield up to an 90% relative decrease in amplitude of the sensitive feature attribution while remaining undetected.
These results highlight the manipulability of SHAP explanations and encourage auditors to treat them with skepticism.
I NTRODUCTION
As Machine Learning (ML) gets more and more ubiquitous in high-stake decision contexts (e.g., healthcare, ﬁnance, and justice), concerns about its potential to lead to discriminatory models are becoming prominent.
The use of auditing toolkits (Adebayo et al., 2016; Saleiro et al., 2018; Bellamy et al., 2018) is getting popular to circumvent the use of unfair models.
However, although auditing toolkits can help model designers in promoting fairness, they can also be manipulated to mislead both the end-users and external auditors.
For instance, a recent study of Fukuchi et al.
(2020) has shown that malicious model designers can produce a benchmark dataset as fake “evidence” of the fairness of the model even though the model itself is unfair.
Another approach to assess the fairness of ML systems is to explain their outcome in a post hoc manner (Guidotti et al., 2018).
For instance, SHAP (Lundberg & Lee, 2017) has risen in popularity as a means to extract model-agnostic local feature attributions.
Feature attributions are meant to convey how much the model relies on certain features to make a decision at some speciﬁc input.
The use of feature attributions for fairness auditing is desirable for cases where the interest is on the direct impact of the sensitive attributes on the output of the model.
One such situation is in the context of causal fairness (Chikahara et al., 2021).
In some practical cases, the outputs cannot be independent from the sensitive attribute unless we sacriﬁce much of prediction accuracy.
For example, any decisions based on physical strength are statistically correlated to gender due to biological nature.
The problem in such a situation is not the statistical bias (such as demographic parity), but whether the decision is based on the physical strength or gender, i.e. the attributions of each feature.
The focus of this study is on manipulating the feature attributions so that the dependence on sensitive attributions is hidden and the audits are misled as if the model is fair even if it is not the case.
Recently, several studies reported that such a manipulation is possible, e.g. by modifying the black-box model to be explained (Slack et al., 2020; Begley et al., 2020; Dimanov et al., 2020), by manipulating the computation algorithms of feature attributions (Aïvodji et al., 2019), and by poisoning the data distribution (Baniecki et al., 2021; Baniecki & Biecek, 2022).
With these ﬁndings in mind, the current possible advice to the auditors is not to rely solely on the reported feature attributions for fairness auditing.
A question then arises about what “evidence” we can expect in addition to the feature attributions, and whether they can be valid “evidence” of fairness.
In this study, we show that we can craft fake “evidence” of fairness for SHAP explanations, which provides the ﬁrst negative answer to the last question.
In particular, we show that we can produce not only manipulated feature attributions but also a benchmark dataset as the fake “evidence” of fairness.
The benchmark dataset ensures the external auditors reproduce the reported feature attributions using the existing SHAP library.
In our study, we leverage the idea of stealthily biased sampling introduced by Fukuchi et al.
to cherry-pick which data points to be included in the benchmark.
Moreover, the use of stealthily biased sampling allows us to keep the manipulation undetected by making the distribution of the benchmark sufﬁciently close to the true data distribution.
Figure 1 illustrates the impact of our attack in an explanation scenario with the Adult Income dataset.
[Figure]
Our contributions can be summarized as follows:
• Theoretically, we formalize a notion of foreground distribution that can be used to extend Local Shapley Values (LSV) to Global Shapley Values (GSV), which can be used to decompose fairness metrics among the features (Section 2.2).
Moreover, we formalize the task of manipulating the GSV as a Minimum Cost Flow (MCF) problem (Section 4).
• Experimentally (Section 5), we illustrate the impact of the proposed manipulation attack on a synthetic dataset and four popular datasets, namely Adult Income, COMPAS, Marketing, and Communities.
We observed that the proposed attack can reduce the importance of a sensitive feature while keeping the data manipulation undetected by the audit.
Our results indicate that SHAP explanations are not robust and can be manipulated when it comes to explaining the difference in outcomes between groups.
Even worse, our results conﬁrm we can craft a benchmark dataset so that the manipulated feature attributions are reproducible by external audits.
Henceforth, we alert auditors to treat post-hoc explanations methods with skepticism even if it is accompanied by some additional evidence.
S HAPLEY V ALUES 2. L OCAL S HAPLEY V ALUES
Shapley values are omnipresent in post-hoc explainability because of their fundamental mathematical properties (Shapley, 1953) and their implementation in the popular SHAP Python library (Lundberg & Lee, 2017).
SHAP provides local explanations in the form of feature attributions i.e. given an input of interest x , SHAP returns a score φ i P R for each feature i “ 1 , 2 , . . .
, d .
These scores are meant to convey how much the model f relies on feature i to make its decision
f p x q .
Shapley values have a long background in coalitional game theory, where multiple players collaborate toward a common outcome.
In the context of explaining model decisions, the players are the input features and the common outcome is the model output
f p x q .
In coalitional games, players (features) are either present or absent.
Since one cannot physically remove an input feature once the model has already been ﬁtted,
SHAP removes features by replacing them with a baseline value z .
This leads to the Local Shapley
Value (LSV) φ i p f, x , z q which respect the so-called efﬁciency axiom (Lundberg & Lee, 2017)
[Equation]
Simply put, the difference between the model prediction at x and the baseline z is shared among the different features.
Additional details on the computation of LSV are presented in Appendix B.1.
2. G LOBAL S HAPLEY V ALUES
LSV are local in the sense that they explain the prediction at a speciﬁc x and rely on a single baseline input z .
Since model auditing requires a more global analysis of model behavior, we must understand the predictions at multiple inputs x „ F sampled from a distribution F called the foreground .
Moreover, because the choice of baseline is somewhat ambiguous, the baselines are sampled z „ B from a distribution B colloquially referred to as the background .
Taking inspiration from Begley et al.
, we can compute Global Shapley Values (GSV) by averaging LSV over both foreground and background distributions.
Deﬁnition 2.1.
[Equation]
Proposition 2.1.
The GSV have the following property
[Equation]
2. M ONTE -C ARLO E STIMATES
In practice, computing expectations w.r.t the whole background and foreground distributions may be prohibitive and hence Monte-Carlo estimates are used.
For instance, when a dataset is used to represent a background distribution, many explainers in the SHAP library such as the ExactExplainer and TreeExplainer will subsample this dataset 1 by selecting 100 instances uniformly at random when the size of the dataset exceeds 100.
More formally, let
[Equation]
represent a categorical distribution over a ﬁnite set of input examples S , where δ p¨q is the dirac probability measure, w j ě 0 @ j , and ř j ω j “ 1 .
Estimating expectations with Monte-Carlo amounts to sampling M instances
[Equation]
and computing the plug-in estimate
[Equation]
When a set of samples is a singleton ( e.g.
S 1 “ t z p j q u ), we shall use the convention p Φ p f, S 0 , t z p j q uq ” p Φ p f, S 0 , z p j q q to improve readability.
In Appendix B.2, p Φ p f, S 0 , S 1 q is shown to be a consistent and asymptotically normal estimate of Φ p f, F , B q meaning that one can compute approximate conﬁdence intervals around p Φ to capture Φ with high probability.
In practice, the estimates p Φ are employed as the model explanation which we see as a vulnerability.
As discussed in
Section 4, the Monte-Carlo estimation is the key ingredient that allow us to manipulate the GSV in favor of a dishonest entity.
A UDIT S CENARIO
This section introduces an audit scenario to which the proposed attack of SHAP can apply.
This scenario involves two parties: a company and an audit.
The company has a dataset D “ tp x p i q , y p
i q qu Ni “with x p i q P R d and y p i q P t 0 , 1 u that contains N input-target tuples and also has a model f : X Ñ r 0 , 1 s that is meant to be deployed in society.
The binary feature with index s ( i.e x s P t 0 , 1 u ) represents a sensitive feature with respect to which the model should not explicitly discriminate.
Both the data D and the model f are highly private so the company is very careful when providing information about them to the audit.
Hence, f is a black box from the point of view of the audit.
At ﬁrst, the audit asks the company for the necessary data to compute fairness metrics e.g. the Demographic Parity (Dwork et al., 2012), the Predictive Equality (Corbett-Davies et al., 2017), or the Equal Opportunity (Hardt et al., 2016).
Note that our attack would apply as long as the
[Table] [Table]
py f ( x )
E [ f ( x ) | x s = 0] E [ f ( x ) | x s = 1] x s =x s
[Figure] [Figure]
x ´ s ff
F
B fairness metric is a difference in model expectations over subgroups.
For simplicity, the audit decides to compute the Demographic Parity
[Equation]
and henceforth demands access to the model outputs for all inputs with different values of the sensitive feature i.e f p D 0 q and f p D 1 q , where D 0 “ t x p i q : x p i
[Equation]
s “ 0 u and D 1 “ t x p i q : x p i q s “ 1 u are subsets of the input data of sizes N 0 and N 1 respectively.
Doing so does not force the company to share values of features other than x s nor does it requires direct access to the inner workings of the proprietary model.
Hence this demand respects privacy requirements and the company will accept to share the model outputs across all instances, see Figure 2a.
At this point, the audit conﬁrms that the model is indeed biased in favor of x s “ 1 and puts in question the ability of the company to deploy such a model.
Now, the company argues that, although the model exhibits a disparity in outcomes, it does not mean that the model explicitly uses the feature
x s to make its decision.
If such is the case, then the disparity could be explained by other features statistically associated with x s .
Some of these other features may be acceptable grounds for decisions.
To verify such a claim, the audit decides to employ post-hoc techniques to explain the disparity.
Since the model is a black-box, the audits shall compute the GSVs.
The foreground F and background B are chosen to be the data distributions conditioned on x s “ 0 and x s “ 1 respectively
[Equation]
According to Equation 3, the resulting GSVs will sum up of the demographic parity (cf.
Equation 7).
If the sensitive feature has a large negative GSV Φ s , then this would mean that the model is explicitly relying on x s to make its decisions and the company would be forbidden from deploying the model.
If the GSV has a small amplitude, however, the company could still argue in favour of deploying the model in spite of having disparate outcomes.
Indeed, the difference in outcomes by the model could be attributed to other more acceptable features.
See Figure 2b for a toy example illustrating this reasoning.
To compute the GSV, the audit demands the two datasets of inputs D 0 and D 1 , as well as the ability to query the black box f at arbitrary points.
Because of privacy concerns on sharing values of x across the whole dataset, and because GSV must be estimated with Monte-Carlo, both parties agree that the company shall only provide subsets S
Ă D 0 and S 1 Ă D 1 of size M to the audit so they can compute a Monte-Carlo estimate p Φ p f, S 0 , S 1 q .
The company ﬁrst estimate GSV on their own by choosing S 0 , S 1 uniformly at random from F and B (cf.
Equation 5) and observe that p Φ s indeed has a large negative value.
They realize they must carefully select which data points will be sent, otherwise, the audit may observe the explicit bias toward x s “ 1 and the model will not be deployed.
Moreover, the company understands that the audit currently has access to the data f p D 0 q and f p D 1 q representing the model predictions on the whole dataset (see Figure 2a).
Therefore, if the company does not share subsets S 0 , S 1 that where chosen uniformly at random from D 0 , D 1 , it is possible for the audit to detect this fraud by doing a statistical test comparing f p S 0 q to f p D 0 q and f p S 1 q to f p D 1 q .
The company needs a method to select misleading subsets S 1 0 , S 1 1 in a manner that manipulates the
GSV in their favour, while remaining undetected by the audit.
Such a method is the subject of the next section.
F OOL SHAP WITH S TEALTHILY B IASED S AMPLING 4. M ANIPULATION
To fool the audit, the company can decide to indeed sample S 1 0 uniformly at random S 1 0 „ F M .
Then, given this choice of foreground sub-sample, they can manipulate the background distribution to cherry-pick M instances.
Formally, the company must compute a non-uniform background distribution B 1 ω : “ C p D 1 , ω q with ω ‰ 1 { N 1 from which to sample M points S 1 1 „ B 1 M ω .
To make the model look fairer, the company needs the p Φ s computed with these cherry-picked points to have a small magnitude.
Proposition 4.1.
Let S 1 0 be ﬁxed , and let p Ñ represent convergence in probability as the size M of the set S 1 1 „ B 1 M ω increases, we have
[Equation]
We note that the coefﬁcients p Φ s p f, S 1 0 , z p j q q in Equation 9 are tractable and can be computed and stored by the company.
We discuss in more detail how to compute them in Appendix B.3.
Remember that to manipulate the GSV, the company must tune the weights ω such that the explanation p Φ s is manipulated while ensuring that the non-uniform distribution B 1 ω remains similar to the original B .
Otherwise, the fraud could be detected by the audit.
In this work, the notion of similarity between distributions will be captured by the Wasserstein distance in output space.
Deﬁnition 4.1 (Wassertein Distance) .
Any probability measure π over D 1 ˆ D 1 is called a coupling measure between B and B 1 ω , denoted π P ∆ p B , B 1 ω q , if 1 { N 1 “ ř j π ij and ω j “ ř i π ij .
The
Wassertein distance between B and B 1 ω mapped to the output-space is deﬁned as
[Equation]
representing the cost of the optimal transport plan that distributes the probability mass from one distribution to the other.
We propose the Algorithm 1 to compute the non-uniform weights ω by minimizing the magnitude of the GSV while maintaining a small Wasserstein distance.
The trade-off between attribution manipulation and proximity to the data is tuned via a hyper-parameter λ ą 0 .
We show in the
Appendix A.2 that the optimization problem at line 5 of Algorithm 1 can be reformulated as a
Minimum Cost Flow (MCF) and hence can be solved in polynomial time (more precisely r O p N 2 .
5 1 q as in Fukuchi et al.
Algorithm 1 Compute non-uniform weights 1: procedure COMPUTE _ WEIGHTS ( D 1 , (cid:32)p Φ s p f, S 1 0 , z p j q q ( j , λ ) 2: β : “ sign r ř z p
j q P D 1 p Φ s p f, S 1 0 ,
z p j q q s 3:
B : “ C p D 1 , 1 { N 1 q Ź Unmanipulated background 4:
B 1 ω : “ C p D 1 , ω
q Ź Manipulated background as a function of ω
5: ω “ arg min ω β ř z p j q P D 1 ω j p Φ s p f, S 1 0 , z p j q q ` λ W p B , B 1 ω q Ź Optimization Problem 6: return ω ;
4. D ETECTION
We now discuss ways the audit can detect manipulation of the sampling procedure.
Recall that the audit has previously been given access to f p D 1 q , f p D
q representing the model outputs across all instances in the private dataset.
The audit will then be given sub-samples S 1 1 , S 1 0 of D 1 , Don which they can compute the output of the model and compare with f p D 1 q , f p D 0 q .
To assess whether or not the sub-samples provided by the company were sampled uniformly at random, the audit has to conduct statistical tests.
The null hypothesis of these tests will be that S 1 1 , S 1 0 were sampled uniformly at random from D 1 , D 0 .
The detection Algorithm 2 with signiﬁcance α uses both the Kolmogorov-Smirnov and Wald tests with Bonferonni corrections (i.e. the α { 4 terms in the
Algorithm).
The Kolmogorov-Smirnov and Wald tests are discussed in more details in appendix C.
1: procedure DETECT _ FRAUD ( f p D 0 q , f p D 1 q , f p S 1 0 q , f p S 1 1 q , α , M ) 2: for i “ 0 , 1 do 3: f p S
i q „ C p f p D i q , 1 { N i q M Ź Subsample without cheating.
4: p-value-KS “ KS p
f p S i q , f p S 1 i q q Ź KS test comparing f p S
i q and f p S 1 i q 5: p-value-Wald “ Wald p f p S 1 i q , f p D i q q Ź Wald test 6: if p-value-KS ă α { 4 or p-value-Wald ă α { 4 then Ź Reject the null hypothesis 7: return8: return 0;
4. W HOLE PROCEDURE
The procedure returning the subsets S 1 0 , S 1 1 is presented in Algorithm 3.
It conducts a log-space search between λ min and λ max for the λ hyper-parameter (line 6) in order to explore the space of possible attacks.
For each value of λ , the attacker runs Algorithm 1 to obtain B 1 ω (line 7-8), then repeatedly samples S 1 1 „ B 1 M ω (line 11) and attempts to detect the fraud via the detection
algorithm (line 12).
The attacker will choose B 1 ω that reduces the magnitude of p Φ s the most while having a detection rate below some threshold τ (line 13).
An example of search over λ on a real-world dataset is presented in Figure 3.
One of the limitations of the proposed methodology is that we only manipulate a single sensitive attribute.
In appendix E.1, we present a possible extension of Algorithm 1 to handle multiple sensitive attributes and present preliminary results of its effectiveness.
Algorithm 3 Fool SHAP 1: procedure F OOL _SHAP( f, D 0 , D 1 , M, λ min , λ max , τ, α ) 2:
S
[Table]
0 „ C p D 0 , 1 { N 0 q M Ź S 1 0 is sampled without cheating 3:
Compute p
Φ s p f, S 1 0 ,
z p j q q @
z p
j q P DŹ cf.
Section B.
B ‹ “ C p D 1 , 1 { N 1 q 5: Φ ‹ s “ 1 { Nř z p j q P D 1 p Φ s p f, S 1 0 ,
z p
j q q Ź Initialize the solution 6: for λ “ λ min , . . .
, λ max do 7: ω “ COMPUTE _ WEIGHTS ( D 1 , (cid:32)p Φ s p f, S 1 0 , z p j q q ( j , λ ) 8:
B 1 ω “ C p D 1 , ω q 9:
Detection_rate “10: for rep “ 1 , . . .
, 100 do Ź Detect the manipulation 11:
S 1 1 „ B 1 M ω 12:
Detection_rate += D ETECT _ FRAUD ( f p D 0 q , f p D 1 q , f p S 1 0 q , f p S 1 1 q , α , M ) 13: if | ř z p
[Equation]
q P D 1 ω j p Φ s p f, S 1 0 , z p j q q| ă | Φ ‹ s | and Detection_rate ă 100 τ then 14:
[Equation]
[Equation]
Ź Update the solution 16:
S
[Equation]
Ź Cherry-pick by sampling from the non-uniform background 17: return S 1 0 , S
[Figure]
4. C ONTRIBUTIONS
We clarify our contributions.
The ﬁrst technique to fool SHAP with perturbations of the background distribution was conducted by via a genetic algorithm Baniecki & Biecek (2022).
However, the crossover and mutation operations it employs to preturb data do not take into account feature correlations and can therefore generate unrealistic data.
Moreover, the objective to minimize does not enforce a similarity between the original and manipulated backgrounds.
We show in Appendix E.3 that these two limitations lead to systematic detections of the fraud.
Hence, our contribution are two-fold
• By perturbing the background via non-uniform weights over pre-existing instances ( i e. B 1 ω : “ C p D 1 , ω q ) rather than a genetic algorithm, we avoid the issue of non-realistic data. • By adding the Wasserstein distance to the optimization objective, we can control the similarity between the original and manipulated backgrounds.
Since the technique introduced in Fukuchi et al.
(2020) also leverages a non-uniform distribution over data points and the Wasserstein distance, it makes sense to adapt it for fooling SHAP . Still, the approach of Fukuchi et al.
is different from ours. Indeed, in their work, they minimize the
Wasserstein distance while enforcing a hard constraint on the number of instances that land on the different bins for the target and sensitive feature, That way, they can set the Demographic Parity to any given value while staying as close as to the original data. In our setting of manipulating the model explanation, we leave the Demographic Parity intact and instead manipulate its feature attribution. From an mathematical point of view, we now minimize a Shapley value with a soft constraint on the
Wasserstein distance.
E XPERIMENTS 5. T OY EXPERIMENT
The task is predicting whether an individual will be hired for a job that requires carrying heavy objects.
The causal graph for this toy data is presented in Figure 4 (left).
We observe that sex ( S ) inﬂuences height ( H ), and that both these features inﬂuence the Muscular Mass ( M ).
In the end, the hiring decisions ( Y ) are only based on the two attributes relevant to the job: H and M .
Also, two noise features N 1 , N 2 were added.
More details and justiﬁcations for this causal graph are discussed in Appendix D.1.
Since strength and height (the two important qualiﬁcations for applicants) are correlated with sex, any model f that ﬁts the data will exhibit some disparity in hiring rates between sexes.
Although, if the decisions made by the model do not rely strongly on feature S , the company can argue in favor of deployment.
GSVs are used by the audit to measure the amount by which the model relies on the sex feature, see Figure 4 (Middle).
By manipulating SHAP with M “ 200 , the company is able to reduce the amplitude of the GSV of feature S .
More importantly, the audit is not able to detect that the provided samples S 1 0 , S 1 1 were cherry-picked, see Figure 4 (Right).
[Figure]
5. D ATASETS
We consider four standard datasets from the FAccT literature, namely COMPAS, Adult-Income,
Marketing, and Communities.
• COMPAS regroups 6,150 records from criminal offenders in Florida collected from 2013 and 2014.
This binary classiﬁcation task consists in predicting which individual will re-offend within two years.
The sensitive feature s is race with values x s “ 0 for African-American and x s “for Caucasian.
• Adult Income contains demographic attributes of 48,842 individuals from the 1994 U.S. census.
It is a binary classiﬁcation problem with the goal of predicting whether or not a particular person makes more than 50K USD per year.
The sensitive feature s in this dataset is gender , which took values x s “ 0 for female, and x s “ 1 for male.
• Marketing involves information of 41,175 customers of a Portuguese bank and the binary classiﬁcation task is to predict who will subscribe to a term deposit.
The sensitive attribute is age and took values x s “ 0 for age 30-60 , and x s “ 1 for age not30- • Communities & Crime contains per-capita violent crimes
for 1994 different communities in the US.
The binary classiﬁcation task is to predict which communities have crimes below the median rate.
The sensitive attribute is PercentWhite and took values x s “ 0 for
PercentWhite<90% , and x s “ 1 for PercentWhite>=90% .
Three models were considered for the two datasets: Multi-Layered Perceptrons (MLP), Random
Forests (RF), and eXtreme Gradient Boosted trees (XGB).
One model of each type was ﬁtted on each dataset for 5 different train/test splits seeds, resulting in 60 models total.
Values of the test set accuracy and demographic parity for each model type and dataset are presented in Appendix D.2.
We note that all demographic parity values are negative.
5. D ETECTOR C ALIBRATION
All probabilities should be under
[Table]
Detector calibration refers to the assessment that, assuming the null hypothesis to be true, the probability of rejecting it (i.e. false positive) should be bounded by the signiﬁcance level α .
Remember that the null hypothesis is that the sets S 1 0 , S 1 1 provided by the company are sampled uniformly from
D 0 , D 1 .
Hence, to test the detector, the audit can sample their own subsets f p S 0 q , f p S 1 q uniformly from f p D 0 q , f p D 1 q , run the detection algorithm, and count the number of detection over 1000 repeats.
Table 1 presents the probabilities of false positives over the ﬁve train-test splits using a signiﬁcance level α “ 5% .
We observe that the probability of false positives is indeed bounded by α for all model types and datasets implying that the detector employed by the audit is calibrated.
5. A TTACK R ESULTS AND D ISCUSSION
[Figure]
The ﬁrst step of the attack (line 3 of Algorithm 3) requires that the company run SHAP on their own and compute the necessary coefﬁcients to run Algorithm 1.
For the COMPAS and Adults datasets, the ExactExplainer of SHAP was used.
Since Marketing and Communities contain more than 15 features, and since the ExactExplainer scales exponentially with the number of features, we were restricted to using the TreeExplainer (Lundberg et al., 2020) on these datasets.
The
TreeExplainer avoids the exponential cost of
Shapley values but is only applicable to tree-based models such as RFs and XGBs.
Therefore, we could not conduct the attack on MLPs ﬁtted on Marketing and Communities.
The following step is to solve the MCF for various values of λ (line 7 of Algorithm 3).
As stated previously, solving the MCF can be done in polynomial time in terms of N 1 , which was tractable for a small dataset like COMPAS and Communities, but not for larger datasets like Adult and Marketing.
To solve this issue, as was done in Fukuchi et al.
(2020), we compute the manipulated weights multiple times using 5 bootstrap sub-samples of D 1 of size 2000 to obtain a set of weights ω r 1 s , ω r 2 s , . . .
, ω r 5 s which we average to obtain the ﬁnal weights ω .
Results of 46 attacks with M “ 200 are shown in Figure 5.
Speciﬁc examples of the conducted attacks are presented in Appendix E.2.
As a point of reference, we also show the results for a brute-force approach consisting of repeatedly sampling S 1 1 „ B M in a honest manner, and choosing the set S 1leading to the smallest attribution of the sensitive attribute.
To make comparisons to our attack more meaningful, the brute-force method was only allowed to run for the same amount of time it took to search for the non-uniform weights ω (about 30-180 seconds).
We note that, across all datasets, Fool
SHAP leads to greater reductions of the sensitive feature attribution compared to brute-force search.
This result highlights the beneﬁt of perturbing the background distribution compared to brute-force sampling.
Now focusing on Fool SHAP, for the datasets COMPAS and Marketing, we observe median reductions in amplitudes of about 90% .
This means that our attack can considerably reduce the perceived importance of the sensitive attribute.
For the Adult and Communities datasets, the median reduction in amplitude is about 50% meaning that we typically reduce by half the importance of the sensitive attribute.
Still, looking at the maximum reduction in amplitude for both datasets, we note that one attack managed to reduce the amplitude by 90% .
This implies that there is a luck factor to the degree of success of the attack, which is to be expected from data-driven attacks.
Finally, the audit was consistently unable to detect the fraud using statistical tests.
This observation raises concerns about the risk that SHAP explanations can be attacked to return not only manipulated attributions but also non-detectable fake evidence of fairness.
C ONCLUSION
To conclude, we proposed a novel attack on Shapley values that does not require modifying the model but rather manipulates the sampling procedure that estimates expectations w.r.t the background distributions.
We show on a toy example and four fairness datasets that our attack can reduce the importance of a sensitive feature when explaining the difference in outcomes between groups using
SHAP .
Moreover, the sampling manipulation is hard to detect by an audit that is given limited access to the data and model.
These results raise concerns about the viability of using Shapley values to assess model fairness.
We leave as future work the use of Shapley values to decompose other fairness metrics such as predictive equality and equal opportunity.
Moreover, we wish to move to use cases beyond fairness, as we believe that the vulnerability of Shapley values that was demonstrated can apply to many other properties such as safety and security.
E THICS S TATEMENT
The main objective of this work is to raise awareness about the risk of manipulation of SHAP explanations and their undetectability.
As such, it aims at exposing the potential negative societal impacts of relying on such explanations.
It remains however possible that malicious model producers could use this attack to mislead end users or cheat during an audit.
However, we believe this paper makes a signiﬁcant step toward increasing the vigilance of the community and fostering the development of trustworthy explanations methods.
Furthermore, by showing how fairness can be manipulated in explanation contexts, this work contributes to the research on the certiﬁcation of the fairness of automated decision-making systems
R EPRODUCIBILITY S TATEMENT
The source code of all our experiments is will be made available online.
Moreover, experimental details are provided in appendix D.2 for the interested reader