On the Impact of Adversarially Robust Models on Algorithmic Recourse
Abstract
The widespread deployment of machine learning models in various high-stakes settings has underscored the need for ensuring that individuals who are adversely impacted by model predictions are provided with a means for recourse.
To this end, several algorithms have been proposed in recent literature to generate recourses.
Recent research has also demonstrated that the recourses generated by these algorithms often correspond to adversarial examples.
This key finding emphasizes the need for a deeper understanding of the impact of adversarially robust models (which are designed to guard against adversarial examples) on algorithmic recourse.
In this work, we make one of the first attempts at studying the impact of adversarially robust models on algorithmic recourse.
We theoretically and empirically analyze the cost (ease of implementation) and validity (probability of obtaining a positive model prediction) of the recourses output by state-of-the-art algorithms when the underlying models are adversarially robust.
More specifically, we construct theoretical bounds on the differences between the cost and the validity of the recourses generated by various state-of-the-art algorithms when the underlying models are adversarially robust vs. non-robust.
We also carry out extensive empirical analysis with multiple real-world datasets to not only validate our theoretical results, but also analyze the impact of varying degrees of model robustness on the cost and validity of the resulting recourses.
Our theoretical and empirical analyses demonstrate that adversarially robust models significantly increase the cost and reduce the validity of the resulting recourses, thereby shedding light on the inherent trade-offs between achieving adversarial robustness in predictive models and providing easy-to-implement and reliable algorithmic recourse.
Introduction
As machine learning (ML) models are increasingly being deployed in high-stakes domains such as banking, healthcare, and criminal justice, it becomes critical to ensure that individuals who have been adversely impacted (e.g., loan denied) by the predictions of these models are provided with a means for recourse.
To this end, several techniques have been proposed in recent literature to provide recourses to affected individuals by generating counterfactual explanations which highlight what features need to be changed and by how much in order to flip a model’s prediction [49, 9, 42, 30, 21, 20, 47] 1 .
For instance, [49] proposed a gradient-based approach which returns the nearest counterfactual resulting in the desired prediction.
[42] proposed an integer programming-based approach to obtain actionable recourses for linear classifiers.
More recently, [21, 22] leveraged the causal structure of the underlying data for generating recourses [3, 27, 30].
Prior research has also theoretically and empirically analyzed the properties of the recourses generated by state-of-the-art algorithms.
For instance, several recent works [33, 30, 10, 40] demonstrated that the recourses output by state-of-the-art algorithms are not robust to small perturbations to input instances, underlying model parameters, and to the recourses themselves.
More recently, [28] demonstrated that the recourses output by state-of-the-art algorithms are very similar to adversarial examples.
This finding is critical because there have been several efforts in the literature on adversarial ML [17, 24, 4] to build adversarially robust models that are not susceptible to adversarial examples.
However, the impact of such models on the quality and the correctness of the recourses output by state-of-the-art algorithms remains unexplored.
The aforementioned connections between adversarial examples and recourses underscore the need for a deeper investigation of the impact of adversarially robust models (which are designed to guard against adversarial examples) on algorithmic recourse.
Such an investigation becomes particularly critical as the need for adversarial robustness of predictive models as well as the ability to obtain easy-to-implement and reliable recourses have often been touted as the cornerstones of trustworthy and safe ML both by prior research as well as recent regulations [48, 15].
However, there is no prior work that investigates the relationship and/or the trade-offs between these two critical pillars of trustworthy and safe ML.
In this work, we address the aforementioned gaps by making the first ever attempt at studying the impact of adversarially robust models on algorithmic recourse.
We theoretically and empirically analyze the cost (ease of implementation) and validity (probability of obtaining a positive model prediction) of the recourses output by state-of-the-art algorithms when the underlying models are adversarially robust.
More specifically, we construct theoretical bounds on the differences between the cost and the validity of the recourses generated by various state-of-the-art algorithms (e.g., gradientbased [49, 25] and manifold-based [31] methods) when the underlying models are adversarially robust vs. non-robust (See Section 3).
To this end, we first derive theoretical bounds on the differences between the weights (parameters) of adversarially robust vs. non-robust models and then leverage these to bound the differences in the costs and validity of the recourses corresponding to these two sets of models.
We also carried out extensive empirical analysis with multiple real-world datasets from diverse domains.
This analysis not only validated our theoretical bounds, but also unearthed several interesting insights pertaining to the relationship between adversarial robustness of predictive models and algorithmic recourse.
More specifically, we found that the cost differences between the recourses corresponding to adversarially robust vs. non-robust models increase as the degree of robustness of the adversarially robust models increases.
We also observed that the validity of recourses worsens as the degree of robustness of the underlying models increases.
We further probed these insights by visualizing the resulting recourses in low dimensions using t-SNE plots, and found that the number of valid recourses around a given instance reduces as the degree of robustness of the underlying model increases.
This work lies in the intersection of Algorithmic Recourse methods and Adversarial Robustness.
Please refer Appendix A for a detailed discussion of related works.
Algorithmic Recourse.
Several approaches have been proposed in recent literature to provide recourses to affected individuals [9, 49, 42, 45, 30, 27, 18, 22, 8].
These approaches can be broadly categorized along the following dimensions [47]: type of the underlying predictive model (e.g., tree based vs. differentiable classifier), type of access they require to the underlying predictive model (e.g., black box vs. gradient access), whether they encourage sparsity in counterfactuals (i.e., only a small number of features should be changed), whether counterfactuals should lie on the data manifold , whether the underlying causal relationships should be accounted for when generating counterfactuals, and whether the output produced by the method should be multiple diverse counterfactuals or a single counterfactual.
In addition, [34] also studied how to generate global, interpretable summaries of counterfactual explanations.
Some recent works also demonstrated that the recourses output by stateof-the-art techniques might not be robust, i.e., small perturbations to the original instance [11, 38], the underlying model [41, 33], or the recourse [32] itself may render the previously prescribed recourse(s) invalid.
These works also formulated and solved minimax optimization problems to find robust recourses to address the aforementioned challenges.
Adversarial Examples and Robustness.
Prior works have shown that complex machine learning models, such as deep neural networks, are vulnerable to small changes in input [39].
This behavior of predictive models allows for generating adversarial examples (AEs) by adding infinitesimal changes to input targeted to achieve adversary-selected outcomes [39, 14].
Prior works have proposed several techniques to generate AEs using varying degrees of access to the model, training data, and the training procedure [6].
While gradient-based methods [14, 24] return the smallest input perturbations which flip the label as adversarial examples, generative methods [50] constrain the search for adversarial examples to the training data-manifold.
Finally, some methods [7] generate adversarial examples for non-differentiable and non-decomposable measures in complex domains such as speech recognition and image segmentation.
Prior works have shown that Empirical Risk
Minimization (ERM) does not yield models that are robust to adversarial examples [14, 24].
Hence, to reliably train adversarially robust models, [26] proposed the adversarial training objective which minimizes the worst-case loss within some ϵ -ball perturbation region around the input instances.
Intersections between Adversarial ML and Model Explanations.
There has been a growing interest in studying the intersection of adversarial ML and model explainability [15].
Among all the existing works focusing on this intersection, two explorations are relevant to our work [36, 29].
[36] studied the interplay between adversarial robustness and post hoc explanations [36] and demonstrated that gradient-based feature attribution methods (e.g., vanilla gradients, gradient times input, integrated gradients, smoothgrad) may severely violate the primary assumption of attribution – features with higher attribution are more important for model prediction – in case of non-robust models.
However, their results also demonstrate that such a violation does not occur when the underlying models are robust to ℓ 2 and ℓ inf input perturbations.
More recently, [28] demonstrated that recourses generated by certain state-of-the-art methods are very similar to adversarial examples, and also argued that the methods proposed to output recourses and adversarial examples are designed with similar goals of changing the input minimally in order to achieve the desired outcome.
While the aforementioned works explored the connections between adversarial ML and model explanations, none of these works focus on analyzing the impact of adversarially robust models on the recourses output by state-of-the-art algorithms.
Preliminaries
Notation.
In this work, we denote a classifier f : X → Y mapping features x ∈ X to labels y ∈ Y , where x is a d -dimensional feature vector.
We define a non-linear activation function ϕ ( · ) such that f ( x ) =
ϕ ( h ( x )) , where h ( x ) is the logits.
In addition, we represent the non-robust and adversarially robust models using f NR ( x ) and f R ( x ) .
Below we describe the methodological frameworks used for comparing recourses generated from non-robust and adversarially robust models.
Adversarially Robust models.
Despite the superior performance of machine learning (ML) models, they are susceptible to adversarial examples (AEs), i.e., inputs generated by adding infinitesimal perturbations to the original samples targeted to change prediction label [1].
One standard approach to ameliorate this problem is training a model using adversarial training which minimizes the worst-case loss within some perturbation region (the perturbation model) [23].
In particular, for a classifier, f parameterized by weights θ , loss function ℓ ( · ) , and training data { x i , y i } i = { 1 , 2 ,...,n } ∈ D train , the optimization problem of minimizing the worst-case loss within ℓ p − norm perturbation with radius ϵ is:
[Equation]
where D train denotes the training dataset and ∆ p,ϵ = { δ : ∥ δ ∥ p ≤ ϵ } is the ℓ p ball with radius ϵ centered around sample x .
Algorithmic Recourse.
One of the ways in which recourse can be realized is by explaining to affected individuals what features in their profile need to change and by how much in order to obtain a positive outcome.
Counterfactual explanations which essentially capture the aforementioned information can therefore be used to provide recourse.
The terms "counterfactual explanations" and "algorithmic recourse" have, in fact, become synonymous in recent literature [19, 43, 46].
More specifically, algorithms that try to find algorithmic recourses do so by finding a counterfactual x ′ =
x
+ ζ that is closest to the original instance x and change the model’s prediction f
( x + ζ ) to the target label.
Next, we describe three methods we use to understand the implications of adversarially robust models on algorithmic recourses.
Score CounterFactual Explanations (SCFE).
Given the classifier f ( x ) =
ϕ ( h ( x )) and a distance function d : R d × R d → R + , [49] define the problem of generating a recourse x ′ =
x + ζ for sample x by minimizing the following objective:
[Equation]
where s is the target score for x ′ , λ is the regularization coefficient, and d ( · ) is the distance between sample x and its counterfactual counterpart x ′ .
C-CHVAE.
Given a Variational AutoEncoder (VAE) model with encoder E γ and decoder G θ trained on the original data distribution D train , C-CHVAE [31] aims to generate recourses in the latent space
Z , where E γ : X → Z .
The encoder E γ transforms a given sample x into a latent representation z ∈ Z and the decoder G θ takes z as input and produces ˆ x as similar as possible to x .
To this end, given a sample x , C-CHVAE generates the recourse ζ using the following objective function:
[Equation]
such that
[Equation]
where E γ allows to search for counterfactuals in the data manifold and G θ projects the latent counterfactuals to the feature space.
Growing Spheres Method (GSM).
While the above techniques directly optimize specific objective functions for generating counterfactuals, GSM [25] uses a search-based algorithm to generate recourses by randomly sampling points around the original instance x until a sample with the target label is found.
In particular, GSM first draws an ℓ 2 -sphere around a given instance x , randomly samples point within that sphere, and checks whether any sampled points result in target prediction.
Finally, they contract or expand the sphere until a (sparse) counterfactual is found and finally returned.
GSM defines a minimization problem using a function c : X × X
→ R + , where c ( x , x ′ ) is the cost of moving from instance x to counterfactual x ′ .
[Equation]
where x ′ is sampled from the ℓ 2 -ball around x such that f ( x ′ )
̸
= f ( x ) , c
( x , x ′ )
=
∥ x ′ − x ∥ 2 + γ ∥ x ′ − x ∥ 0 , and γ ∈ R + is the weight associated to the sparsity objective.
Theoretical Analysis
Next, we carry out a detailed theoretical analysis to bound the cost and validity differences of recourses generated by state-of-the-art recourse methods when the underlying models are adversarially robust vs. non-robust.
More specifically, we compare the cost differences w.r.t.
the recourses obtained using 1) gradient-based methods such as SCFE [49](Sec. 3.1.1) and 2) manifold-based methods such as
C-CHVAE [31] (Sec. 3.1.2).
Finally, we show that the validity of algorithmic recourse generated using existing methods for robust models is lower compared to that of non-robust models (Sec. 3.2).
3.1 Cost Analysis
The cost of a generated algorithmic recourse is defined as the distance (e.g., ℓ 1 or ℓ 2 distance) between the input instance x and the counterfactual x ′ obtained using a state-of-the-art recourse finding method [47].
Algorithmic recourses with lower costs are considered better since they enable minimal changes to input to achieve the desired outcome.
Here, we theoretically analyze the cost difference of generating recourses using algorithmic recourse methods when the underlying models are non-robust and adversarially robust.
3.1.1 Gradient-based method: SCFE
Next, we carry out a detailed theoretical analysis to bound the cost and validity difference of recourses generated by state-of-the-art recourse methods when the underlying models are adversarially robust vs. non-robust.
Here, we derive the lower and upper bound for the cost difference of recourses generated by the
SCFE [49] method when the underlying models are adversarially robust vs. non-robust.
Following previous works [13, 16, 29, 35, 43], we focus on locally linear model approximations as this lays the foundation for understanding non-linear model behavior.
For the cost difference, we first define the closed-form solution for the optimal cost ζ ∗ required to generate a recourse.
Definition 1.
(Optimal Cost from [29])
For a given scoring function f with weights w the SCFE method generates a recourse x ′ for an input x using cost ζ such that:
[Equation]
where m = s − h
( x ′ ) is the target residual, s is the target score for x , h ( x ) is a local linear score approximation, and λ is a given hyperparameter.
Theorem 1.
(Cost difference for SCFE)
For a given instance x , let x ′ NR and x ′ R be the recourse generated using Wachter’s algorithm for non-robust and adversarially robust models.
Then, for a normalized Lipschitz activation function σ ( · ) , the cost difference for the recourse generated for both models can be bounded as:
[Equation]
where w NR and w R are the weights of the non-robust and adversarially robust models, λ is the regularization coefficient in Wachter’s algorithm, m NR = s − h NR ( x ′ ) , m R = s
− h R ( x ′ ) are the target residuals for robust ( f R ( x ) =
ϕ ( h R ( x )) ) and non-robust models ( f NR ( x ) = ϕ ( h NR ( x )) ), respectively. Proof Sketch.
We derive the cost difference of recourses generated for non-robust and adversarially robust models by comparing their optimal solutions.
Similar to [29], the upper bound results follow from Cauchy-Schwartz and triangle inequality.
In addition, we also leverage reverse triangle inequality to derive a lower bound for the recourse difference.
See Appendix B.1 for the complete proof.
The equality of Equation 6 entails that the upper bound of the recourse difference will have a tighter bound if the ℓ 2 -norms of the weights w R and w B are bounded, and the lower bound of the recourse difference will be tighter if the output score of the non-robust and adversarially robust models is similar for the given sample x .
3.1.2 Manifold-based method: C-CHVAE
We extend our analysis of bounding the cost difference of generated recourses using manifold-based methods for non-robust and adversarially robust models.
In particular, we leverage C-CHVAE [31] that leverages variational autoencoders to generate counterfactuals.
For a fair comparison, we assume that both models use the same encoder I γ and decoder G θ networks for learning the latent space of the given input space X .
Definition 2.
An encoder model E is L -Lipschitz if ∀ z 1 , z 2 ∈ Z , we have:
[Equation]
Using Definition 7, we now derive the lower and upper bounds of the cost difference of recourses generated for non-robust and adversarially robust models.
Theorem 2.
(Cost difference for C-CHVAE)
Let z NR and z R be the generated recourse from CCHVAE [31] method in the latent space using an L-Lipschitz generative model G ( · ) for a non-robust and adversarially robust model.
Then, by definition of C-CHVAE, x NR = G ( z NR )=
x + ζ NR and x R = G ( z R )=
x + ζ R are the corresponding recourses in the input space.
The cost difference between the recourses can then be bounded as:
[Equation]
where L is the Lipschitz constant of the generative model, and r ∗ NR and r ∗ R be the corresponding radii chosen by the algorithm such that they successfully return a recourse for the non-robust and adversarially robust model.
Proof Sketch.
The proof follows from Definition 7 and the triangle inequality.
It shows that the cost difference for generating recourses using C-CHVAE is bounded by the product of the Lipschitz constant of the generative model and the radii chosen by the C-CHVAE to generate counterfactuals for the underlying non-robust and adversarially robust models.
See Appendix B.2 for detailed proof.
3.2 Validity Analysis
The validity of a given recourse x ′ is defined as the probability that it results in the desired outcome [47], denoted by Pr ( f ( x ′ )
= 1) .
Below, we analyze the validity of the recourses by first deriving the upper bound of the difference in non-robust and adversarially robust model weights, and then use this lemma to show that the validity of non-robust model is higher than for the adversarially robust model.
Lemma 1.
(Difference between non-robust and adversarially robust model weights)
For a given instance x , let w NR and w R be weights of the non-robust and adversarially robust model.
Then, for a normalized Lipschitz activation function σ ( · ) , the difference in the weights ∆ w can be bounded as:
[Equation]
[Equation]
) where η is the learning rate, ϵ is the ℓ 2 -norm perturbation ball, y is the label for x , n is the total number of training epochs, and d is the dimension of the input features.
Proof Sketch.
We derive the upper bound of the difference in non-robust and adversarially robust model weights, denoted by ∆ w , and show that it is proportional to the dimension of the input features times the ℓ 2 perturbation ball around the sample x .
See Appendix B.3 for the detailed proof.
Next, we show that the probability of a recourse action resulting in the desired outcome is greater for a non-robust model compared to that of the adversarially robust model.
Theorem 3.
(Validity Comparison)
For a given instance x ∈ R d and desired target label denoted by unity, let x R and x NR be the counterfactuals for adversarially robust f R ( x ) and non-robust f NR ( x ) models respectively.
Then, Pr ( f NR ( x NR ) = 1) ≥ Pr ( f R ( x R ) = 1) if | f NR ( x R ) − f NR ( x NR ) |≤ nη ( y ∥ x T ∥ 2 + ϵ √ d ) ∥ x R ∥ , where η is the learning rate, ϵ is the ℓ 2 -norm perturbation ball, y is the label for x , and n is the total number of training epochs.
Proof Sketch.
We derive the difference between the probability that a valid recourse exists for a non-robust and adversarially robust model.
Using data inequalities and Cauchy-Schwartz, we show that the condition for the validity is dependent on the weight difference of the models (Lemma 1).
See Appendix B.4 for the detailed proof.
Experimental Evaluation
In this section, we empirically analyze the impact of adversarially robust models on the cost and validity of recourses.
First, we empirically validate our theoretical bounds on differences between the cost and validity of recourses output by state-of-the-art recourse generation algorithms when the underlying models are adversarially robust vs. non-robust.
Second, we carry out further empirical analysis to assess the differences in cost and validity of the resulting recourses as the degree of the adversarial robustness of the underlying model changes on three real-world datasets.
4.1 Experimental Setup
Here, we describe the datasets used for our empirical analysis along with the predictive models, algorithmic recourse generation methods, and the evaluation metrics.
Datasets.
We use three real-world datasets for our experiments: 1) The German Credit datasetcomprises demographic (age, gender), personal (marital status), and financial (income, credit duration) features from 1000 credit applicants, with each sample labeled as "good" or "bad" depending on their credit risk.
The task is to successfully predict if a given individual is a "good" or "bad" customer in terms of associated credit risk.
The Adult dataset 3 contains demographic (e.g., age, race, and gender), education (degree), employment (occupation, hours-per week), personal (marital status, relationship), and financial (capital gain/loss) features for 48,842 individuals.
The task is to predict if an individual’s income exceeds $50K per year.
The COMPAS dataset 4 has criminal records and demographics features for 18,876 defendants who got released on bail at the U.S state courts during
[Figure]
The dataset is designed to train a binary classifier to classify defendants into bail (i.e., unlikely to commit a violent crime if released) vs. no bail (i.e., likely to commit a violent crime).
Predictive models.
We generate recourses for the non-robust and adversarially robust version of
Logistic Regression (linear) and Neural Networks (non-linear) models.
We use two linear layers with sigmoid activation functions as our predictor and set the number of nodes in the intermediate layers to twice the number of nodes in the input layer, which is the size of the input dimension in each dataset.
Algorithmic Recourse Methods.
We analyze the cost and validity for non-robust and adversarially robust models w.r.t.
three popular classes of recourse generation methods, namely, gradient-based (SCFE), manifold-based (C-CHVAE), and random search-based (GSM) methods (described in Sec. 2).
Evaluation metrics.
To concretely measure the impact of adversarial robustness on algorithmic recourse, we analyze the difference between cost and validity metrics for recourses generated using non-robust and adversarially robust model.
To quantify the cost, we measure the average cost incurred to act upon the prescribed recourses across all test-set instances, i.e., Cost ( x , x ′ ) =
[Equation]
x ′ ∥ 2 , where x is the input and x ′ is its corresponding recourse.
To measure validity, we compute the probability of the generated recourse resulting in the desired outcome
, i
e., Validity ( x , x ′ ) =
|{ x ′ : f ( x ′ )
=
1 ∩ x ′ =
g ( x,f ) }| |D test | , where g ( x, f ) returns recourses for input x and predictive model f .
Implementation details.
We train non-robust and adversarially robust predictive models from two popular model classes (logistic regression and neural networks) for all three datasets.
In the case of adversarially robust models, we adopt the commonly used min-max optimization objective for adversarial training using varying degree of robustness, i.e., ϵ ∈ { 0 , 0 .
Note that the model trained with ϵ =0 is the non-robust model.
Following [2] and [12], we preprocessed the input data by removing categorical features for efficient training of our models.
We follow [29] to set the hyperparameters for the algorithmic recourse methods.
4.2 Empirical Analysis
Next, we describe the experiments that we carried out to understand the impact of adversarial robustness of predictive models on algorithmic recourse.
More specifically, we will discuss (1) empirical verification of our theoretical bounds, (2) empirical analysis of the differences between the costs of recourses corresponding to non-robust vs. adversarially robust models, and (3) empirical analysis to compare the validity of the recourses corresponding to non-robust vs. adversarially robust models.
Empirical Verification of Theoretical Bounds.
We empirically validate our theoretical findings from Section 3 on real-world datasets.
In particular, we first estimate the empirical bounds (RHS of
Theorems 1-2) for each instance in the test set by plugging the corresponding values of the parameters in the theorems and compare them with the empirical estimates of the cost differences between recourses generated using gradient- and manifold-based recourse methods (LHS of Theorems 1-2).
[Figure]
Figure 1 show the results obtained from the aforementioned analysis of cost differences.
We observe that our bounds are tight, and the empirical estimates fall well within our theoretical bounds.
Similarly, we observe that the validity of the non-robust model, as denoted by Pr ( f NR ( x ) = 1) in Theorem 3, was higher than the validity of the adversarially robust model for all the test samples in Adult, German
Credit, COMPAS datasets, following the condition in Theorem 3 for a large number of training iterations used for training adversarially robust models with ϵ ∈ { 0 , 0 .
Cost Analysis.
To analyze the impact of adversarial robustness on the cost of recourses, we compute the difference between the cost for obtaining a recourse using non-robust and adversarially robust model and plotted this difference for varying degrees of robustness ϵ .
Results in Figure 2 show a significant increase in incurred costs to find algorithmic recourse for adversarially robust models compared to the non-robust model for all the datasets with increasing degrees of robustness.
We observe a similar trend for the case of logistic regression, as shown in Figure 6 in Appendix C.
Further, we observe a relatively smoother increasing trend for cost differences in the case of SCFE compared to others, which can be attributed to the stochasticity present in C-CHVAE and GSM.
We also observe a higher cost difference in SCFE for most datasets, which could result from the larger sample size used in C-CHVAE and GSM.
We observe a similar trend in cost differences when the sample size per iteration is reduced, which also resulted in more iterations to find recourse.
Validity Analysis.
To analyze the impact of adversarial robustness on the validity of recourses, we compute the fraction of recourses resulting in the desired outcome, generated using non-robust and adversarially robust model under resource constraints, and plot it against varying degrees of robustness ϵ .
Results in Figure 3 show that there is an even stronger impact of adversarial training on validity for neural networks trained on the three datasets.
We observe a similar pattern for the case of the logistic regression model trained on the three datasets, shown in Appendix C. On average, we observe that the validity drops to zero for models adversarially trained with ϵ > 0 .
To understand this further, we use t-SNE visualization [44] – a non-linear dimensionality reduction technique – to map points in the dataset to two-dimensional space and demonstrate a gradual decline in valid recourses around a local neighborhood with increasing ϵ in Figure 4, where x and y be the names of reduced dimensions.
This decline suggests that a large number of recourses in the neighborhood of the input sample are now being classified with the same class as the input.
Hence, this supports our hypothesis that adversarially robust models severely impact the validity of recourses and make the recourse search computationally expensive.
Conclusion
In this work, we theoretically and empirically analyzed the impact of adversarially robust models on algorithmic recourse.
We theoretically bounded the differences between the costs of the recourses output by two state-of-the-art counterfactual explanation methods (SCFE and C-CHVAE)
when
[Figure]
the underlying models are adversarially robust vs. non-robust.
In addition, we also bounded the differences between the validity of the recourses corresponding to adversarially robust and nonrobust models.
We empirically validated our theoretical results using three real-world datasets (Adult, COMPAS, and German Credit) and two popular model classes (neural networks and logistic regression).
Our theoretical and empirical analyses demonstrated that adversarially robust models significantly increase the cost and reduce the validity of the resulting recourses, thereby highlighting the inherent trade-offs between achieving adversarial robustness in predictive models and providing easy-to-implement and reliable algorithmic recourses.
Our work also paves the way for several interesting future research directions at the intersection of algorithmic recourse and adversarial robustness in predictive models.
For instance, given the aforementioned trade-offs, it would be interesting to develop novel techniques which enable end users to navigate these trade-offs based on their personal preferences – e.g., an end user may choose to sacrifice the adversarial robustness of the underlying model in order to secure lower cost recourses.  Acknowledgements
The authors would like to thank the anonymous reviewers for their helpful feedback and all the funding agencies listed below for supporting this work.
This work is supported in part by the NSF awards IIS-2008461 and IIS-2040989, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and D 3 Institute at Harvard. HL would like to thank Sujatha and Mohan Lakkaraju for their continued support and encouragement.
The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.
A Related Works
Algorithmic Recourse.
Several approaches have been proposed in recent literature to provide recourses to affected individuals [9, 49, 42, 45, 30, 27, 18, 22, 8].
These approaches can be broadly categorized along the following dimensions [47]: type of the underlying predictive model (e.g., tree based vs. differentiable classifier), type of access they require to the underlying predictive model (e.g., black box vs. gradient access), whether they encourage sparsity in counterfactuals (i.e., only a small number of features should be changed), whether counterfactuals should lie on the data manifold , whether the underlying causal relationships should be accounted for when generating counterfactuals, and whether the output produced by the method should be multiple diverse counterfactuals or a single counterfactual.
In addition, [34] also studied how to generate global, interpretable summaries of counterfactual explanations.
Some recent works also demonstrated that the recourses output by stateof-the-art techniques might not be robust, i.e., small perturbations to the original instance [11, 38], the underlying model [41, 33], or the recourse [32] itself may render the previously prescribed recourse(s) invalid.
These works also formulated and solved minimax optimization problems to find robust recourses to address the aforementioned challenges.
Adversarial Examples and Robustness.
Prior works have shown that complex machine learning models, such as deep neural networks, are vulnerable to small changes in input [39].
This behavior of predictive models allows for generating adversarial examples (AEs) by adding infinitesimal changes to input targeted to achieve adversary-selected outcomes [39, 14].
Prior works have proposed several techniques to generate AEs using varying degrees of access to the model, training data, and the training procedure [6].
While gradient-based methods [14, 24] return the smallest input perturbations which flip the label as adversarial examples, generative methods [50] constrain the search for adversarial examples to the training data-manifold.
Finally, some methods [7] generate adversarial examples for non-differentiable and non-decomposable measures in complex domains such as speech recognition and image segmentation.
Prior works have shown that Empirical Risk
Minimization (ERM) does not yield models that are robust to adversarial examples [14, 24].
Hence, to reliably train adversarially robust models, [26] proposed the adversarial training objective which minimizes the worst-case loss within some ϵ -ball perturbation region around the input instances.
Intersections between Adversarial ML and Model Explanations.
There has been a growing interest in studying the intersection of adversarial ML and model explainability [15].
Among all the existing works focusing on this intersection, two explorations are relevant to our work [36, 29].
[36] studied the interplay between adversarial robustness and post hoc explanations [36] and demonstrated that gradient-based feature attribution methods (e.g., vanilla gradients, gradient times input, integrated gradients, smoothgrad) may severely violate the primary assumption of attribution – features with higher attribution are more important for model prediction – in case of non-robust models.
However, their results also demonstrate that such a violation does not occur when the underlying models are robust to ℓ 2 and ℓ inf input perturbations.
More recently, [28] demonstrated that recourses generated by certain state-of-the-art methods are very similar to adversarial examples, and also argued that the methods proposed to output recourses and adversarial examples are designed with similar goals of changing the input minimally in order to achieve the desired outcome.
While the aforementioned works explored the connections between adversarial ML and model explanations, none of these works focus on analyzing the impact of adversarially robust models on the recourses output by state-of-the-art algorithms.
B Proof for Theorems in Section
Here, we provide detailed proofs of the Lemmas and Theorems defined in Section 3.
B.1 Proof for Theorem
Theorem 1.
(Cost difference for SCFE)
For a given instance x , let x ′ NR and x ′ R be the recourse generated using Wachter’s algorithm for the non-robust and adversarially robust models.
Then, for a normalized Lipschitz activation function σ ( · ) , the difference in the recourse for both models can be bounded as:
[Equation]
where w NR and w R are the weights of the non-robust and adversarially robust models, λ is the regularization coefficient in Wachter’s algorithm, φ ( · ) is a function that measures the shift in the model weights using the target and predicted scores.
Proof.
Following the definition of SCFE in Equation 2, we can find a counterfactual sample x ′ that is "closest" to the original instance x by minimizing the following objective:
[Equation]
where s is the target score, λ is the regularization coefficient, and d ( · ) is the distance between the original and counterfactual sample x ′ .
Lower bound.
Using Lemma 1, the optimal cost for generating a valid recourse for a non-robust ( ζ ∗ NR ) and adversarially robust ( ζ ∗ R ) model can be written as:
[Equation]
[Equation]
(13) where m NR = s − w T NR x and m R =
s − w T R x .
Subtracting and taking ℓ 2 -norm on both sides of Eqn.
12 and Eqn. 13, we get:
[Equation]
Upper bound.
Again, using the optimal recourse cost (Definition 1) [Equation] we can derive the upper bound of the cost difference for generating recourses using non-robust and adversarially robust models:
[Equation]
Triangle Inequality)
Note that the difference between the target and the predicted score for both non-robust and adversarially robust models is upper bounded by a term that is always positive.
Hence, we get: ∥
[Equation]
B.2 Proof for Theorem
Theorem 2.
(Cost difference for C-CHVAE)
Let z NR and z R be the solution returned by the CCHVAE [31] algorithmic recourse method by sampling from ℓ p -norm ball in the latent space using an L-Lipschitz generative model G ( · ) for a non-robust and adversarially robust model.
By definition of the recourse method, let x NR
= G ( z NR ) and x R = G ( z R ) be the corresponding recourses in the input space.
The difference between them can then be bounded as:
[Equation]
(14) where L is the Lipschitz constant of the generative model, and r ∗ NR and r ∗ R be the corresponding radii chosen by the algorithm such that they successfully return a recourse for the non-robust and adversarially robust model.
Proof.
From the formulation of the counterfactual algorithm, we can write the difference between x R and x NR as:
[Equation]
Lower bound.
Here, we present a lower bound on the ℓ p norm of the cost difference between a baseline and robust model.
Using Equation 15, we get:
[Equation]
(since ∥ a − b ∥ p ≥ ∥ a ∥ p −∥ b
[Equation] [Equation]
37]) where z is the latent space representation for the original point x , r ∗ R and r ∗ NR are the radius of the ℓ p -norm for generating samples from the robust and baseline model.
Note that using the radius of the ℓ p norm in the above equation provides a tighter lower bound.
Upper bound.
Using Equation 15, we can derive the upper bound using Lemma 1 and the triangle inequality.
[Equation]
triangle inequality)
[Equation]
[Equation]
where r ∗ R and r ∗ NR is the radius of the ℓ p -norm for generating samples from the robust and baseline model, respectively.
B.3 Proof for Lemma
Lemma 1.
(Difference between non-robust and adversarially robust model weights)
For a given instance x , let w NR and w R be weights of the non-robust and adversarially robust model.
Then, for a normalized Lipschitz activation function σ ( · ) , the difference in the weights ∆ w can be bounded as:
[Equation]
where η is the learning rate, ϵ is the ℓ 2 -norm perturbation ball, y is the label for x , n is the total number of training epochs, and d is the dimension of the input features.
Proof.
Without loss of generality, we consider the case of binary classification which uses the binary cross entropy or logistic loss.
Let us denote the baseline and robust models as f NR ( x )=
w TNR x and f R ( x )= w TR x , where we have removed the bias term for simplicity.
We consider the class label as y ∈ { +1 , − 1 } , and loss function L ( f ( x ))
=
log (1 + exp( − y.f ( x ))) .
Note that an adversarially robust model
f R ( x ) is commonly trained using a min-max objective, where the inner maximization problem is given by:
[Equation]
where δ is the adversarial perturbation added to a given sample x and ϵ denotes the the perturbation norm ball around x .
Since our loss function is monotonic decreasing, the maximization of the loss function applied to a scalar is equivalent to just minimizing the scalar quantity itself, i.e.,
[Equation]
The optimal solution to min ∥ δ ∥≤ ϵ y · w TR δ is given by − ϵ ∥ w TR ∥ 1 [23].
Therefore, instead of solving the min-max problem for an adversarially robust model, we can convert it to a pure minimization problem, i.e.,
[Equation]
Correspondingly, the minimization objective for a baseline model is given by min w NR L (cid:0) y · ( w T NR x ) (cid:1) .
Looking into the training dynamics under gradient descent, we can define the weights at epoch ‘t’ for a baseline and robust model as a function of the Jacobian of the loss function with respect to their corresponding weights, i.e.,
[Equation]
[Equation]
where η is the learning rate of the gradient descent optimizer, w 0 is the weight initialization of both models.
[Equation]
where sign ( x ) return +1, -1, 0 for x > 0 , x < 0 , x = 0 respectively and σ ( x )= 1 1+exp( − x ) is the sigmoid function.
Let us denote the weights of the baseline and robust model at the n -th iteration as w n NR and w n R , respectively.
Hence, we can define the n -th step of the gradient-descent for both models as:
[Equation] [Equation]
where η is the learning rate of the gradient descent optimizer.
Taking n = 1 , we get:
[Equation] [Equation]
where w 0 NR and w 0 R are the same initial weights for the baseline and robust models.
Subtracting both equations, we get:
[Equation]
Similarly, for n = 2 and using Equation 30, we get the following relation:
[Equation]
Using the above equations
[Equation]
we can now write the difference between the weights of the baseline and robust models at the n -th iteration as:
[Equation]
(Using σ ( a − b ) ≤ σ
[Equation]
a ) for b > 0 )
[Equation]
Using ℓ 2 -norm on both sides, we get:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Inequality)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(since the term inside ∥·∥ 2 is a scalar)
[Equation]
B.4 Validity
Theorem 3.
(Validity Comparison)
For a given instance x ∈ R d and desired target label denoted by unity, let x R and x NR be the counterfactuals for adversarially robust f R ( x ) and non-robust f NR ( x ) models respectively.
Then, Pr ( f NR ( x NR ) = 1) ≥ Pr ( f R ( x R ) = 1) if | f NR ( x R ) − f NR ( x NR ) |≤ nη ( y ∥ x T ∥ 2 + ϵ √ d ) ∥ x R ∥ , where η is the learning rate, ϵ is the ℓ 2 -norm perturbation ball, y is the label for x , and n is the total number of training epochs.
Proof.
In a logistic regression case, Pr ( f ( x ) = 1) =
e w T x 1+ e w T x , which is the sigmoid of the model output.
Next, we derive the difference in probability of a valid recourse from non-robust and adversarially robust model:
[Equation]
Since (1 + e w T R x R )(1 + e w T NR x NR ) > 0 , so Pr ( f NR ( x NR ) = 1) ≥ Pr ( f R ( x R ) = 1)) occurs when, e w T NR x
[Equation]
T
[Equation]
x
[Equation] [Equation]
(Taking natural logarithm on both sides)
[Equation] [Equation]
(Taking norm on both sides)
[Equation] [Equation]
(Using Cauchy-Schwartz) (cid:13)(cid:13)
[Equation]
NR (
[Equation]
(From Lemma 1) | f NR
[Equation]
C Additional Experimental Results
In this section
[Figure]
we have plots for cost differences, validity
[Figure]
and adversarial accuracy for the two logistic regression and neural network models trained on three real-world datasets.
[Figure]
[Figure] [Figure]
[Figure] [Figure]