C OVARIANCE -R OBUST M INIMAX P ROBABILITY M ACHINES FOR A LGORITHMIC R ECOURSE
Anonymous authors
Paper under double-blind review
A BSTRACT
Algorithmic recourse is rising as a prominent technique to promote the explainability and transparency of the predictive model in ethical machine learning.
Existing approaches to algorithmic recourse often assume an invariant predictive model; however, this model, in reality, is usually updated temporally upon the input of new data.
Thus, a recourse that is valid respective to the present model may become in valid for the future model.
To resolve this issue, we propose a pipeline to generate a model-agnostic recourse that is robust to model shifts.
Our pipeline ﬁrst estimates a linear surrogate of the non linear (black-box) model using covariance-robust minimax probability machines (MPM); then, the recourse is generated with respect to this robust linear surrogate.
We show that the covariance-robust MPM recovers popular regularization schemes, including (cid:96) 2 -regularization and class-reweighting.
We also show that our covariance-robust MPM pushes the decision boundary in an intuitive manner, which facilitates an interpretable generation of a robust recourse.
The numerical results demonstrate the usefulness and robustness of our pipeline.
I NTRODUCTION
The recent prevalence of machine learning (ML) in supporting consequential decisions involving humans such as loan approval (Moscato et al., 2021), job hiring (Cohen et al., 2019; Schumann et al., 2020), and criminal justice (Brayne & Christin, 2021) urges the need of transparent ML systems with explanations and feedback to users (Doshi-Velez & Kim, 2017; Miller, 2019).
One popular and emerging approach to providing feedback is the algorithmic recourse (Ustun et al., 2019).
A recourse suggests how the input instance should be modiﬁed to alter the outcome of a predictive model.
Consider a speciﬁc scenario in which an individual is rejected from receiving a loan by a ﬁnancial institution’s ML model.
Recently, it has become a legal necessity to provide explanations and recommendations to the individual so that they can improve their situation and obtain a loan in the future (GDPR, Voigt & Von dem Bussche (2017)).
For example, an explanation can be “increase the income to $5000” or “reduce the debt/asset ratio to below 20%”.
Leveraging the recourses, ﬁnancial institutions can assess the reliability of their ML predictive models and increase user engagement through actionable feedback and acceptance guarantee if they fulﬁll the requirements.
To construct plausible and meaningful recourses, one must assess and strike a balance between conﬂicting criteria.
They can be: (1) validity , a recourse should effectively reverse the unfavorable prediction of the model into a favorable one, (2) proximity , recourse should be close to the original input instance to alleviate the efforts required, and thus to encourage the adoption of the recourse, (3) actionability , prescribed modiﬁcations should follow causal laws of our society (Ustun et al., 2019;
Karimi et al., 2021); for example, one can not modify their race or decrease their age.
Various techniques were proposed to devise algorithmic recourses for a given predictive model, extensive surveys are provided in (Karimi et al., 2020a; Stepin et al., 2021; Pawelczyk et al., 2021;
Verma et al., 2020).
Wachter et al.
(2017) introduced the deﬁnition of counterfactual explanations and proposed a gradient-based approach to ﬁnd the nearest instance that yields a favorable outcome.
Ustun et al.
(2019) proposed a mixed integer programming formulation (AR) that can ﬁnd recourses for a linear classiﬁer with a ﬂexible design of the actionability constraints.
Alternatively, Karimi et al.
(2021; 2020b) investigated the nearest recourse through the lens of minimal intervention to take causal relationships between features into account.
Recent works including Russell (2019) and Mothilal et al.
(2020) also studied the problem of generating a menu of diverse recourses to provide multiple possibilities that users might choose.
The aforementioned methods rely on an assumption of an invariant predictive model.
Nevertheless, machine learning models are usually re-trained or re-calibrated as new data arrive.
Thus, a valid recourse at present may become invalid in the future, leading to an exemplary case where a rejected applicant may spend efforts to improve their income and reapply for a loan, but then is rejected (again) simply because the ML model has been updated.
This leads to a potential inefﬁciency due to the waste of resources and loss of trust in the recommendation and in the ML system (Rudin, 2019).
Studying this phenomenon, Rawal et al.
(2020) described several types of model shifts related to the correction, temporal, and geospatial shifts from data.
They pointed out that the recourses, even constructed with state-of-the-art algorithms, are vulnerable to distributional shifts in the model’s parameters.
Pawelczyk et al.
(2020) study counterfactual explanations under predictive multiplicity and its relation to the difference in the way two classiﬁers treat predicted individuals.
Black et al.
(2021) then show that the constructed recourses might be invalid even for the model retrained with different initial conditions such as weight initialization and leave-one-out variations in data.
Recently,
Upadhyay et al.
(2021) leveraged robust optimization to propose ROAR - a framework for generating recourses that are robust to shifts in the predictive model, which is assumed to be a linear classiﬁer.
Despite the promising results, existing methods are often restricted to the linear classiﬁers setting to be able to introduce actionability or robustness (Ustun et al., 2019; Russell, 2019; Upadhyay et al., 2021; Rawal et al., 2020).
For non-linear classiﬁers, a linear surrogate method such as LIME (Ribeiro et al., 2016) is used to approximate the local decision boundary of the black-box classiﬁers; the recourse is then generated respectively to the (linear) surrogate model instead of the nonlinear model.
LIME is well-known for explaining predictions of black-box ML models by ﬁtting a reweighted linear regression model to the perturbed samples around an input instance.
In the recourse literature, LIME is the most common linear surrogate for the local decision boundary of the black-box models (Ustun et al., 2019; Upadhyay et al., 2021).
Unfortunately, the LIME surrogate has several limitations.
Firstly, Laugel et al.
and White &
Garcez (2019) showed that LIME may not be faithful to the underlying models because LIME might be inﬂuenced by input features at a global scale rather than a local scale.
Secondly, explanations generated by perturbation-based methods are also well-known to be sensitive to the original input and the synthesized perturbations (Alvarez-Melis & Jaakkola, 2018; Ghorbani et al., 2019; Slack et al., 2020; 2021; Agarwal et al., 2021;
Laugel et al., 2018).
Several works have been proposed to overcome these issues.
Laugel et al.
(2018) and Vlassopoulos et al.
(2020) proposed alternative sampling procedures that generate sample instances in the neighborhood of the closest counterfactual to ﬁt a local surrogate.
White & Garcez (2019) integrated counterfactual explanation to local surrogate models to introduce a novel ﬁdelity measure of an explanation.
Later, Garreau & von Luxburg (2020) and Agarwal et al.
(2021) analyzed theoretically the stability 1 of LIME, especially in the low sampling size regime.
Zhao et al.
(2021) leveraged Bayesian reasoning to improve the consistency in repeated explanations of a single prediction.
Nevertheless, the impact and effectiveness of these surrogates on the recourse generation are still unknown.
Contributions.
We revisit the recourse generation scheme through surrogate models.
We propose a novel model-agnostic pipeline that facilitates the generation of robust and actionable recourses.
The core innovation in our pipeline is the use of the covariance-robust minimax probability machines (MPM) as a linear surrogate of the nonlinear black-box ML model.
Additionally, we contribute • to the ﬁeld of MPM and robust classiﬁer: We propose and analyze in detail the covariance-robust
MPMs in which the set of possible perturbations of the covariance matrices are prescribed using distances on the space of positive semideﬁnite matrices.
Motivated by the statistical distances between Gaussian distributions, we show that the covariance-robustness induces and connects to two prominent regularization schemes of the nominal MPM:
– if the distance is motivated by the Bures distance, we recover the (cid:96) 2 -regularization, – if the distance is motivated by the Fisher-Rao distance, we recover class reweighting schemes. [Figure]
While prior works showed that distributionally robust optimization (DRO) with optimal transport distance recovers norm regularization ( -Abadeh et al., 2019b) and f -divergence DRO leads to reweighting (Duchi & Namkoong, 2019), this extends the connections to the
• to the ﬁeld of robust algorithmic recourse: We propose an intuitive and interpretable approach to generate robust recourse.
We show that, by calibrating the radii of the ambiguity sets in a proper manner, the covariance-robust MPM shifts the separation hyperplane towards the favorable class.
As a consequence, our recourse exhibits robustness to model shifts and it is also lenient to incorporate mixed-integer constraints to promote actionability.
This paper unfolds as follows.
In Section 2, we delineate our explanation framework using MPM.
Section 3 dives deeper into the MPM problem and its robustiﬁcation.
Section 4-5 construct two types of covariance-robust MPM using the Bures and Fisher-Rao distance on the space of covariance matrices.
In Section 6, we demonstrate empirically that the covariance-robust MPM provides a competitive approximation of the local decision boundary, and improves the robustness of the recourse subject to model shifts. All proofs are relegated to the appendix.
R ECOURSE G ENERATION F RAMEWORK
Throughout this paper, we assume that the covariate space is X =
R d
and we have a binary label space
Y = {− 1 , +1 } .
Without any loss of generality, we assume that label − 1 encodes the unfavorable decision, while +1 encodes the favorable one.
Given a speciﬁc classiﬁer and an input x 0 with an unfavorable predicted outcome, the goal of this paper is to ﬁnd a recourse recommendation for xthat has a high probability of being classiﬁed into a favorable group, subject to possible shifts in the parameters underlying the classiﬁer.
Such recourse is termed a robust recourse.
Our robust recourse generator consists of three components (see Figure 1 for a schematic view):
(i) a local sampler: we use a similar procedure as in Vlassopoulos et al.
(2020) and Laugel et al.
Given an instance x 0 , we choose k nearest counterfactuals x 1 , . . .
, x k from the training data that have the opposite label to x 0 .
For each x i , we perform a line search to ﬁnd a point x b,i that is on the decision boundary and the line segment between x 0 and x i .
Among x b,i , we choose the nearest point x b to x 0 and sample uniformly in an (cid:96) 2 -neighborhood with radius r p around x b .
We then query the black-box classiﬁer to obtain the predicted labels of the synthetic samples.
(ii) a linear surrogate using (covariance-robust)
MPM:
We use the synthetic samples to estimate the moment information ( (cid:98) µ y , (cid:98) Σ y ) of the covariate conditional on each predicted class y .
We then train a covariance-robust MPM parametrized by θ ϕ to approximate the local decision boundary of the ML model.
(iii) a recourse search: Basically, we can apply any existing recourse search for linear models on top of the linear surrogate θ ϕ dictated by the covariance-robust MPMs to ﬁnd a robust recourse.
In this paper, we use a simple projection onto the hyperplane prescribed by θ ϕ for simplicity and AR (Ustun et al., 2019), which is a MIP-based framework, to promote actionable recourses.
Center to the success of our pipeline is the possibility of shifting the MPM classiﬁcation hyperplane toward the region of the favorable class, which induces robust recourse with respect to model shifts in a geometrically-intuitive manner (see Remark 4.6 for a detailed discussion).
It is imperative to note a clear distinction between our pipeline and the existing method of ROAR (Upadhyay et al., 2021): ROAR uses a non-robust surrogate in Step (ii) and then formulates a min-max optimization problem in Step (iii) for recourse search, whilst our pipeline uses a robust surrogate in Step (ii) and then employs a simple recourse search in Step (iii).
Note that mixed-integer formulations can be injected in Step (iii) to generate more realistic robust recourses in our pipeline.
On the contrary, mixed-integer constraints are not easy to be integrated into the min-max formulation of ROAR.
Subsequently, Sections 3-5 describe in detail different methods to build robust surrogates in Step (ii).
The application of recourse research in Step (iii) is provided in the experiment section (Section 6.2).
(C OVARIANCE -R OBUST ) MPM
MPM is a binary classiﬁcation framework pioneered by Lanckriet et al.
and extended to Quadratic MPM in (Lanckriet et al., 2003).
For each class y ∈ Y , MPM makes no assumption on the speciﬁc parametric form of the (conditional) distribution (cid:98) P y of X | Y
=
y .
Instead, MPM assumes that we can identify (cid:98) P y only up to the ﬁrst two moments, i.e. , it assumes that (cid:98) P y has mean vector (cid:98) µ y ∈ R d and covariance matrix (cid:98) Σ y ∈ S
d + , denoted (cid:98) P y ∼ ( (cid:98) µ y , (cid:98) Σ y ) .
These moments can be estimated from the samples synthesized from the boundary sampler.
The goal of MPM is to ﬁnd a (non-trivial) linear classiﬁer that minimizes the maximum misclassiﬁcation rate among classes.
To this end, we consider the family of linear classiﬁers parametrized by θ = ( w, b )
∈ R
d +1 , w (cid:54) = 0 with classiﬁcation rule
[Equation]
where ( w, b ) is the slope and intercept.
The MPM solves the min-max optimization problem
[Equation]
where we deﬁne the feasible set Θ (cid:44) { θ = ( w, b )
∈ R d +1 : w (cid:54) = 0 } .
Notice that the constraints w (cid:54) = 0 eliminate trivial solutions to the classiﬁcation problem.
To derive the MPM, we deﬁne the set of feasible slopes W
[Equation]
which is a hyperplane in R d .
The main instrument for solving (1) is the following result from (Lanckriet et al., 2001, §2) which provides the form of its optimal solution.
Lemma 3.1 (Optimal solution) .
Let (cid:98) w be an optimal solution to the second-order cone program
[Equation]
then (cid:98) θ =
( (cid:98) w, (cid:98) b ) solves the MPM problem (1) , where (cid:98) κ
= (cid:0) (cid:80) y ∈Y (cid:113) (cid:98)
[Table]
In this paper, we refer to the second-order cone program (2) as the nominal MPM problem, because the MPM is fully determined by the solution to (2).
We next discuss the covariance-robust MPM.
3. Q UADRATIC MPM
In a practical setting, it is likely that the covariance matrices (cid:98) Σ y are misspeciﬁed, for example, due to low sample size, statistical estimation error, or corrupted data.
To hedge against these mismatches,
Lanckriet et al.
(2003) proposed to add another layer of robustness by allowing the mean vectors and the covariance matrices of the conditional distributions to be chosen (adversarially) in a prescribed set, which we call the ambiguity set.
They showed that perturbing the mean vectors does not change the optimal classiﬁer.
In this paper, we, therefore, perturb only the covariance matrices.
More speciﬁcally, we allow the conditional distribution P y to be in the ambiguity set
[Equation]
where ϕ is a measure of dissimilarity between covariance matrices.
The distributionally robust minimax probability machine is formulated as
[Equation]
Previously, Lanckriet et al.
(2003) considered the robust MPM with moment uncertainty, in which the covariance matrix is perturbed using the quadratic divergence.
Deﬁnition 3.2 (Quadratic divergence) .
Given two positive semideﬁnite matrices Σ , (cid:98) Σ ∈ S d + , the quadratic divergence between them is Q (Σ (cid:107) (cid:98) Σ) =
Tr (cid:2) (Σ − (cid:98) Σ) 2 (cid:3) .
The divergence Q is the squared Frobenius norm of Σ − (cid:98) Σ ; thus Q is non-negative and vanishes to zero if and only if Σ = (cid:98) Σ , so it is a divergence on S d + .
The Quadratic MPM has the below form (Lanckriet et al., 2003).
Theorem 3.3 (Quadratic MPM) .
Suppose that ϕ ≡ Q .
Let w Q be a solution to the problem
[Equation]
Then θ Q = ( w Q , b Q ) solves the distributionally robust MPM problem (3) , with κ Q =
[Equation]
The Quadratic MPM can be considered as a regularization of the nominal problem (2): each matrix (cid:98) Σ y is added with a diagonal matrix √ ρ y I , making the matrix better conditioned.
This is equivalently known as inverse regularization, which ensures invertibility when (cid:98) Σ y is low-rank and ρ y > 0 .
3. C OVARIANCE -R OBUST MPM
While Lanckriet et al.
(2003) focused only on the quadratic divergence, their results can be generalized to the covariance-robust MPM with a general divergence ϕ .
For any y ∈ Y , deﬁne τ
y : R d → R as
[Equation]
We are now ready to give a generalized reformulation of problem (3).
Proposition 3.4 (Covariance-robust MPM) .
Let w ϕ
be the optimal solution to the problem
[Equation]
then θ ϕ
= ( w ϕ , b ϕ ) solves the distributionally robust MPM problem (3) , where
[Equation]
3. E QUIVALENCE UNDER G AUSSIAN A SSUMPTIONS
While the quadratic divergence Q in Deﬁnition 3.2 is attractive for its tractability, it is not statistically meaningful.
More speciﬁcally, it does not coincide with any distance between probability distributions with the corresponding covariance information.
In this paper, we consider discrepancy measures ϕ that arise as a statistical distance between Gaussian distributions.
To this goal, we ﬁrst need to show that the covariance-robust MPM is invariant with the Gaussian assumption.
Deﬁne a parametric ambiguity set constructed on the space of Gaussian distribution of the form
[Equation]
wherein any distribution is Gaussian.
Consider the Gaussian distributionally robust MPM problem
[Equation]
Proposition 3.5 (Gaussian equivalence) .
The optimizer θ ϕ = ( w ϕ , b ϕ ) in Proposition 3.4 also solves the Gaussian parametric covariance-robust MPM problem (6) .
Proposition 3.5 justiﬁes the use of divergences induced by a distance between normal distributions.
We study several constructions of the covariance-robust MPM in the subsequent sections.
B URES MPM
We ﬁrst explore the case where ϕ is the Bures divergence whose deﬁnition is as follows.
Deﬁnition 4.1 (Bures divergence) .
Given two positive semi-deﬁnite matrices Σ , (cid:98) Σ ∈ S d + , the Bures divergence between them is [Equation]
[Equation]
It can be shown that B is symmetric and non-negative, and it vanishes to zero if and only if Σ = (cid:98) Σ .
As such, B is a divergence on the space of positive semideﬁnite matrices.
Moreover, B also equals the squared type-2 Wasserstein distance between two Gaussian distributions with the same mean vector and covariance matrices Σ and (cid:98) Σ (Olkin & Pukelsheim, 1982; Givens & Shortt, 1984; Gelbrich, 1990).
Next, we assert the form of the Bures MPM.
Theorem 4.2 (Bures MPM) .
Suppose that ϕ ≡ B .
Let w B be the solution of the following problem
[Equation]
Then θ B = ( w B , b B ) is the optimal solution of the distributionally robust MPM problem (3) , where [Equation] y ∈Y [Equation]
[Equation]
Σ y w
[Equation]
+ [Equation] y ∈Y [Equation]
y (cid:1)
[Equation]
w B (cid:107) 2 (cid:19) −, and
[Equation] w B (cid:107) [Equation] Theorem 4.2 unveils a fundamental connection between robustness and regularization: if we construct the ambiguity sets for the covariance matrices using the Bures divergence, the resulting optimization problem (7) is an l 2 -regularization of the nominal problem (2).
This connection aligns with previous observations highlighting the equivalence between regularization schemes and optimal transport robustness (Shaﬁeezadeh-Abadeh et al., 2019a; Blanchet et al., 2019).
To prove Theorem 4.2, we provide a result that asserts the analytical form of τ B y ( w ) .
Proposition 4.3 (Bures divergence) .
If ϕ ≡ B , then τ B y ( w ) =
ρ y (cid:107) w (cid:107) 2 + (cid:113) w (cid:62) (cid:98) Σ
y w for all y ∈ Y .
The proof of Theorem 4.2 follows by combining Propositions 3.4 and 4.3.
Next, we study the asymptotic form of the Bures MPM as the radii of the ambiguity sets grow.
Note that problem (7) depends only on the sum of the radii, but not on the individual values of each radius.
Let ρ = (cid:80) y ∈Y ρ y be their sum, it sufﬁces to study when ρ grows to inﬁnity.
To this end, denoted by w B ρ the optimal solution of problem (7) parametrized by ρ .
The next result provides the analysis of the asymptotic value of w B ρ as ρ → ∞ .
Proposition 4.4 (Bures asymptotic hyperplane) .
Fix y ∈ Y , let − y be its opposite class and suppose that ρ − y remains constant.
Let w B
ρ
y be the optimal solution of (7) parametrized by ρ y .
As ρ y → ∞ ,
[Equation]
Notice that as ρ y → ∞ , κ B τ B y ( w B ρ y ) → 1 .
Thus, b B ∞ = ( w B ∞ ) (cid:62) (cid:98) µ y − y for any y ∈ Y .
The asymptotic hyperplane deﬁned by w B ∞ is thus characterized by the linear equation w B ∞ x − w B ∞ (cid:98) µ y + y = 0 , which identiﬁes a hyperplane passing through (cid:98) µ − y as (cid:80) y ∈Y yw (cid:62) µ y = 1 . Moreover, we note that the asymptotic hyperplane does not depend on the covariance matrices.
Remark 4.5 (Quadratic asymptotic hyperplane) .
It is provable that the solution of problem (4) converges to w B ∞ as ρ = (cid:80)
y ρ y tends to inﬁnity.
Thus, Quadratic MPM (4) and Bures MPM (7) are asymptotically equivalent even though they induce different regularizations of the nominal MPM (2) .
Remark 4.6 (Geometric intuition for robust recourse) .
Figure 3 visualizes the Bures MPM hyperplanes by varying the radii.
Notice that the hyperplane drifts toward the favorable (+1) class as the uncertainty of the unfavorable (-1) covariance matrix increases.
Thus, the recourse generated w.r.t.
the green hyperplane is more robust compared to that generated w.r.t.
the red one.
By calibrating the radii, we shift the hyperplane and obtain robust recourses at different robustness-cost trade-offs.
F ISHER -R AO MPM
We now explore the case where ϕ is the Fisher-Rao distance which is deﬁned as follows.
Deﬁnition 5.1 (Fisher-Rao distance) .
Given two positive deﬁnite matrices Σ , (cid:98) Σ ∈ S d ++ , the FisherRao distance between them is F (Σ , (cid:98) Σ) =
(cid:107) log( (cid:98) Σ
− 12 Σ (cid:98) Σ − 12 ) (cid:107) 2 , where log( · ) is the matrix logarithm.
The Fisher-Rao distance enjoys many nice properties.
In particular, it is invariant to inversion and congruence, i.e. , for any Σ , (cid:98) Σ ∈ S d ++ and invertible A ∈ R d × d , F (Σ , (cid:98) Σ) =
F (Σ − 1 , (cid:98) Σ − 1 ) =
F ( A Σ A (cid:62) , A (cid:98)
Σ A (cid:62) ) .
Such invariances are especially statistically meaningful as it implies that the results remain unchanged if we reparametrize the problem with an inverse covariance matrix (instead of the covariance matrix) or apply a change of basis to the data space X .
It is shown that F is the unique Riemannian distance (up to scaling) on the cone S d
++ with such invariances (Savage, 1982).
Next, we assert the form of the Fisher-Rao MPM.
Theorem 5.2 (Fisher-Rao MPM) .
Suppose that ϕ ≡ F .
Let w F be the solution of the problem
[Equation]
Then θ F = ( w F , b F ) is the optimal solution of the distributionally robust MPM problem (3) , where κ F =
[Equation]
(cid:88) [Equation] Theorem 5.2 divulges another foundational connection between robustness and regularization: if we construct the ambiguity sets for the covariance matrices using the Fisher-Rao distance, the resulting optimization problem (8) is a reweighted version of the nominal problem (2).
Each term ( w (cid:62) (cid:98) Σ y w )is assigned a weight exp( ρ y / 2) , which is proportional to the radius ρ y .
This connection aligns with previous observations highlighting the equivalence between reweighting schemes and distributional robustness (Ben-Tal et al., 2013; Bayraksan & Love, 2015; Namkoong & Duchi, 2017; Hashimoto et al., 2018).
To prove Theorem 5.2, we derive an analytical expression of τ F y ( w ) .
Proposition 5.3 (Fisher-Rao distance) .
If ϕ ≡ F , then τ F y ( w ) =
exp (cid:0) ρ y 2 (cid:1) ( w (cid:62) (cid:98) Σ w ) 12 for all y ∈ Y .
The proof of Theorem 5.2 follows by combining the results from Proposition 3.4 and Proposition 5.3.
Next, we study the asymptotic form of the Fisher-Rao MPM.
Proposition 5.4 (Fisher-Rao asymptotic hyperplane) .
Fix y ∈ Y , let − y be its opposite class and suppose that ρ − y remains constant.
Let w F ρ y be the optimal solution of (8) parametrized by ρ y .
Let a (cid:44) (cid:80) y [Equation] y (cid:98) µ y , then as ρ y → ∞ , [Equation]
Contrary to the Bures MPM, the asymptotic hyperplane of Fisher-Rao MPM depends explicitly on the covariance matrix (cid:98) Σ y . The boundary prescribed by the Fisher-Rao MPM can be shifted through an appropriate calibration of the radii ρ y .
Thus, Fisher-Rao MPM can generate a robust algorithmic recourse in a geometrically-intuitive manner.
N UMERICAL E XPERIMENTS
We conduct comprehensive experiments to highlight the performance of our models.
We ﬁrst compare the ﬁdelity and stability of our surrogates with LIME.
We then compare the quality of our recourses against two popular baselines: ROAR (Upadhyay et al., 2021) and AR (Ustun et al., 2019).
Classiﬁer.
We use three-layer MLP with 20, 50, 20 nodes and ReLU activation functions in each consecutive layer.
We use a sigmoid function in the last layer to produce probabilities.
Dataset.
We evaluate our framework using popular real-world datasets for algorithmic recourse:
German Credit (Dua & Graff, 2017; Groemping, 2019), Small Bussiness Administration (SBA)
(Li et al., 2018), and Student performance (Cortez & Silva, 2008).
Each dataset contains two sets of data (the present data - D 1 and the shifted data D 2 ).
The shifted dataset D 2 could capture the correction shift (German credit), the temporal shift (SBA), or the geospatial shift (Student).
For each dataset, we use 80% of the instances in the present data D 1 to train an underlying predictive model and the remaining instances are used to generate and evaluate recourses.
The shifted data D 2 is used to train future classiﬁers in Section 6.2.
The main text contains the results for the German and Student datasets.
Further results, including the SBA and the synthetic data, are provided in Appendix A.
6. F IDELITY AND S TABILITY OF THE S URROGATE M ODELS
We evaluate the performance of different surrogate models with respect to the current classiﬁer.
We compare our methods: QUAD-MPM in (3.3), the BW-MPM in (4.2) and the FR-MPM in (5.2) against the popular linear surrogate LIME (Ribeiro et al., 2016) under the following metrics:
[Figure]
Stability.
We use the procedure in Agarwal et al.
(2021) to measure the stability of the surrogate models with respect to small perturbations in the input instance.
For a given instance x , we draw a set U x of 10 neighbors of x from N ( x, 0 .
001 I ) independently. We use the abovementioned methods to ﬁnd the linear surrogate θ x
[Equation] = ( w x (cid:48) , b x (cid:48) ) for each x (cid:48) ∈ U x .
We report the maximum distance of the explanations of x (cid:48) to that of x ; precisely, the stability formula is
[Equation]
Fidelity.
We use the LocalFid criterion as in Laugel et al.
(2018) to measure the ﬁdelity of a local surrogate model.
For a given instance x , we draw a set V x of 1000 instances uniformly from an l 2 -ball of radius r ﬁd centered on x .
The local ﬁdelity of the surrogate θ x is measured as:
[Equation]
where f ( · ) is the original classiﬁer and C θ ( · ) is the linear surrogate classiﬁer.
Basically, LocalFid measures the fraction of instances where the output class of f and C θ x agree.
Here, we set r ﬁd to 10 % of the maximum distance between instances in the training data.
Note that V x is for evaluation only and independent from the perturbation samples used to train the local surrogate.
To generate the MPM’s surrogates, we choose 10 nearest counterfactuals of x 0 in training data to ﬁnd x b .
We set the perturbation radius r p to 5% of the maximum distance between instances in the training data and set ρ +1 = 0 , ρ − 1 = 1 . 0 .
For LIME, we use the default parameters recommended in the
LIME source code and return θ = ( w, b − 0 .
5) as the LIME’s surrogate, similar to Laugel et al.
We vary the number of perturbation samples in a range of [500 , 10000] to measure the ﬁdelity and sensitivity of constructed surrogates under low sampling sizes.
The results on German and Student datasets are shown in Figure 4.
The results show the superiority of MPM’s surrogates compared to LIME in both local ﬁdelity and stability metrics.
Meanwhile, FR-MPM provides higher-ﬁdelity surrogates compared to QUAD-MPM and BW-MPM.
6. R OBUSTNESS OF R ECOURSES
We now study the robustness to model shifts of recourses and its trade-off against the recourse cost.
Metrics.
We use an (cid:96) 1 -distance as the cost function.
We deﬁne the current validity as the validity of the generated recourses with respect to the given classiﬁer.
To measure the validity of recourses under the model shifts, we sample 80% instances of the shifted data D 2 100 times to train 100 ‘future’ MLP classiﬁers.
We then report the future validity of recourse as the fraction of the future classiﬁers with respect to which the recourse is valid.
[Table]
Baselines.
The experiment in Section 6.1 suggests that the Fisher-Rao model can be used to represent the class of covariance-robust MPM.
We use FRMPM as the linear surrogate θ F = ( w F , b F ) and a simple projection to generate recourse by ﬁnding x r = arg min {(cid:107) x − x 0 (cid:107) 1 : x (cid:62) w F + b F ≥ 0 } .
We compare the above method, namely FRMPM-PROJ, against strong baselines, including
Wachter (Wachter et al., 2017) and ROAR (Upadhyay et al., 2021) using LIME (Ribeiro et al., 2016), CLIME (Agarwal et al., 2021), and
LIMELS (Laugel et al., 2018) 2 as surrogates.
Comparisons with ROAR using the vanilla MPM and SVM surrogates are in Appendix A.2.
[Figure]
Cost-validity trade-off.
We ﬁx the number of perturbation samples to 1000 and vary the ambiguity size ( ρ +1 = 0 , ρ − 1 ∈ [0 , 10] for FR-MPMPROJ and δ max ∈ [0 , 0 . 2] for the uncertainty size of ROAR).
We then plot the Pareto frontiers of the cost-validity trade-off in Figure 5.
Generally, increasing the ambiguity size will increase both the current and future validity of recourses, but induces a sacriﬁce in the cost.
This result is consistent with the analysis in Rawal et al.
However, the frontiers of FR-MPM-PROJ dominate the frontiers of LIME-ROAR, CLIME-ROAR, and
LIMELS-ROAR.
Note that we use CARLA’s implementation with default parameters for Wachter and its higher cost compared to the linear surrogate’s methods on the Student dataset is consistent with results in Upadhyay et al.
and Pawelczyk et al.
Table 1 shows the performance of FR-MPM-PROJ at radii ρ +1 = 0 , ρ − 1 = 10 and of ROAR-related methods at uncertainty size δ max = 0 .
Here, the FR-MPM-PROJ has similar future validity compared to LIMELS-ROAR, but at a lower cost and higher current validity.
Further results and discussion are in Appendix A.2.
[Table]
Actionability.
By introducing robustness into the linear surrogate model, our method can use
AR (Ustun et al., 2019) as a recourse search mechanism to promote ﬂexible actionability constraints.
In Table 2, we compare the AR performance using different surrogate models (LIME, CLIME,
LIMELS, SVM, MPMs).
We consider the recourse under some actionability constraints such as immutable race, gender, or non-decrease age (see Appendix A.1 for more details on the actionability constraints).
The result shows that AR using FR-MPM as the linear surrogate increases both the current and future validity substantially compared to other surrogates.
The results for other datasets are presented in Appendix A.
R EPRODUCIBILITY S TATEMENT
In order to foster reproducibility, we have released all source code and scripts used to replicate our experimental results at https://anonymous.4open.science/r/mpm-recourse .
The repository includes source code, datasets, conﬁgurations, and instructions; thus one could reproduce our results with several commands.
We use the original authors’ implementations of for LIME 3 (Ribeiro et al., 2016) and AR 4 (Ustun et al., 2019).
We use a well-known CARLA’s implementation 5 for Wachter (Wachter et al., 2017).
Since we cannot ﬁnd the open-source code for CLIME (Agarwal et al., 2021), LIMELS (Laugel et al., 2018), and ROAR (Upadhyay et al., 2021), we implement according to their papers.
CLIME and
LIMELS are adapted from LIME’s source code while ROAR is adapted from CARLA’s source code for Wachter.
The hyperparameter conﬁgurations for our methods and other baseline are clearly stated in Section A,
Appendix A.1 and Appendix A.2 and also stored in the repository.
The surrogates sharing the same local sampler will be ensured to have the same random seed, therefore, have the same synthesized samples.
The hyperparameters that affect baselines’ performance such as λ and the probabilistic threshold of Wachter and ROAR will also be studied in Appendix A.2.
The remaining proofs and theoretical claims are provided in Appendix B.
A E XPERIMENTS A. E XPERIMENTAL D ETAILS
Classiﬁer.
We use a three-layer MLP with 20, 50, and 20 nodes and a ReLU activation in each consecutive layer.
We use the sigmoid function in the last layer to produce probabilities.
To train this classiﬁer, we use the binary cross-entropy, solved using the Adam optimizer and 1000 epochs.
Datasets.
We provide more details about synthetic and real-world datasets.
For synthetic data, we generate 2-dimensional data by sampling instances uniformly in a rectangle x = ( x 1 , x 2 ) ∈ [ − 2 , 4] ×
[ − 2 , 7] .
Each sample is labeled using the following function:
[Equation]
where ε is a random noise.
We generate a present data set D 1 with ε = 0 and a shifted data set D 2 with ε ∼ N (0 , 1) .
The decision boundary of the MLP classiﬁer for current synthetic data is illustrated in Figure 6.
The detail of three real-world datasets are listed below: i German Credit (Dua & Graff, 2017).
The dataset contains the information (e.g. age, gender, ﬁnancial status,...) of 1000 customers who took a loan from a bank.
The classiﬁcation task is to determine the risk (good or bad) of an individual.
There is another version of this dataset regarding corrections of coding error (Groemping, 2019).
We use the corrected version of this dataset as shifted data to capture the correction shift.
The features we used in this dataset include ‘duration’, ‘amount’, ‘personal status sex’, and ‘age’.
When considering actionability constraints (Section 6.2), we set ‘personal status sex’ as immutable and ‘age’ to be non-decrease.
ii Small Bussiness Administration (SBA)
(Li et al., 2018).
This
data includes 2,102 observations with historical data of small business loan approvals from 1987 to 2014.
We divide this dataset into two datasets (one is instances from 1989 - 2006 and one is instances from 2006 - 2014) to capture temporal shifts.
We use the following features: selected, ‘Term’, ‘NoEmp’, ‘CreateJob’, ‘RetainedJob’, ‘UrbanRural’, ‘ChgOffPrinGr’, ‘GrAppv’, ‘SBA Appv’, ‘New’, ‘RealEstate’, ‘Portion’, ‘Recession’.
When considering actionability constraints, we set ‘UrbanRural’ as immutable.
iii Student performance (Cortez & Silva, 2008).
This data includes the performance records ofstudents in two schools: Gabriel Pereira (GP) and Mousinho da Silveira (MS).
The classiﬁcation task is to determine if their ﬁnal score is above average or not.
We split this dataset into two sets in two schools to capture geospatial shifts.
The features we used are: ‘age’, ‘Medu’, ‘Fedu’, ‘studytime’, ‘famsup’, ‘higher’, ‘internet’, ‘romantic’, ‘freetime’, ‘goout’, ‘health’, ‘absences’, ‘G1’, ‘G2’.
When considering actionability constraints, we set ‘romantic’ as immutable and ‘age’ to be non-decreased.
For categorical features, we use one-hot encoding to convert them to binary features, similar to Mothilal et al.
We also normalize continuous features to zero mean and unit variance before training the classiﬁer.
The performance of the classiﬁer is reported in Table 3.
[Table]
A. A DDITIONAL E XPERIMENTAL R ESULTS
Local ﬁdelity and stability experiments.
Here, we provide benchmarks of local ﬁdelity and stability (Section 6.1) on the synthetic and SBA datasets in Figure 7.
{+We also run with a different setting for the stability and local ﬁdelity metric to assess if the results are sensitive to the parameter choices. Speciﬁcally, we sample 10 neighbors in the distribution N ( x, 0 . 
 I ) instead of N ( x, 0 .+}
001 I ) to measure the stability. Meanwhile, we set r fid to 20% and the radius r p to 10% of the maximum distance between instances in the training data. The result is shown in Figure [Table]
[Figure]
[Figure]
[Table]
The cost-validity trade-off.
Here [Table] we provide more detail about the settings of the baselines and the complementary results of the experiments in Section 6.2 in the main paper [Table]
[Figure] [Figure] For Wachter’s implementation, we use CARLA’s source code 6 (Pawelczyk et al., 2021), which employs an adaptive scheme to adjust the hyperparameter ( λ ) if no valid recourse is found.
We adopt this implementation for ROAR and set the initial λ to 0 .
1 as suggested in (Upadhyay et al., 2021).
Regarding the surrogate models, we use the open-source code with the default settings for
LIME 7 (Ribeiro et al., 2016).
We adapt this source code accordingly for CLIME’s implementation (Agarwal et al., 2021).
LIMELS, SVM, and MPM-related surrogates use the same boundary sampling procedure (with the same seed), in which we set the number of counterfactuals k = 10 and the number of perturbation samples is 1000 .
Figure 9 shows the Pareto frontiers of the cost-validity trade-off on the synthetic and three real-world datasets.
Table 4 shows the performance of FR-MPM-PROJ at radii ρ +1 = 0 , ρ − 1 = 10 and of ROAR-related methods at uncertainty size δ max = 0 .
2 on synthetic, SBA, and Student datasets.
The hyperparameter λ is set to 0 .
1 for both Wachter and ROAR-related methods.
In FR-MPM-PROJ and ROAR-related methods, the trade-off between cost and validity can be observed when increasing the ambiguity size ( ρ − 1 for FR-MPM and δ max for ROAR-related methods),
[Figure]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] similar to the analysis in (Rawal et al., 2020) and the experimental results in (Upadhyay et al., 2021;
Black et al., 2021).
Generally, the Pareto frontiers of FR-MPM-PROJ dominate the frontiers of
ROAR-related methods on all evaluated datasets.
In other words, with the same cost (or validity), our method will provide recourses with a higher validity (or lower cost) compared to ROAR.
Tabledemonstrates that our method has similar validity but a much smaller cost than ROAR on Synthetic and German datasets.
Meanwhile, our method achieves higher validity with reasonable cost on SBA and Student datasets.
Comparing other baselines, LIMELS-ROAR performs slightly better compared to LIME-ROAR and
CLIME-ROAR.
Wachter provides the recourses with high current validity but is vulnerable to model shifts, resulting in poor future validity.
Wachter has the lowest cost for the synthetic and German datasets but a higher cost for the SBA and Student datasets compared to the linear surrogate-based methods.
This is consistent with the results in (Upadhyay et al., 2021) since the objective function of
Wachter might be non-convex when the classiﬁer is an MLP network.
Comparison with ROAR using the vanilla MPM and SVM as the surrogate model.
Here, we compare the FR-MPM-PROJ with ROAR using the vanilla MPM and SVM as the surrogate model.
Both vanilla MPM and SVM use the same boundary sample procedure (with the same seed number) as FR-MPM.
The settings are similar to the experiment in Section 6.2.
The result shown in Figuredemonstrates the merits of our method.
Actionability.
We provide the complementary result of the actionability experiment on the synthetic,
SBA, and Student datasets in Table 5.
Using AR with FR-MPM produces the recourses with higher value in both the current and future validity compared to AR using other surrogates.
Ablation study.
We conduct an ablation study to understand the contribution of each stage in our method.
Figure 10 shows the comparison of our method with ROAR using vanilla MPM and SVM as the local surrogate.
Figure 11 shows the Pareto frontiers of FR-MPM-PROJ compared to its ablations by alternating the FR-MPM with other surrogates (LIME, MPM) or alternating the projection with [Figure] [Figure] u t u r e
V a l i d i t y
[Figure]
Wachter.
Particularly, we compare our method with LIME-ROBUST-PROJ, which uses LIME as the surrogate model and then solves the robustiﬁed projection:
[Equation]
where ( w, b ) is the weight and bias of LIME’s surrogate [Figure] δ max is similar to the uncertainty size of
ROAR (Upadhyay et al., 2021).
The recourses are generated with respect to the MLP classiﬁer on synthetic and three real-world datasets.
This result demonstrates the usefulness of the FR-MPM in promoting the generation of robust recourses.
Note that, for ρ − = 0 and ρ + = 0 , the hyperplane of the FR-MPM classiﬁer recovers the vanilla MPM’s hyperplane.
Comparison with the probabilistic shiftings
One might attempt to increase the probabilistic threshold (usually set to 0 . 5 ) at which a sample is considered ‘favorable’ to generate robust recourses.
In Figure 12, we compare the proposed method with Wachter, LIME, and MPM with various probabilistic thresholds in the range [0 . 5 , 0 . 9] .
It can be seen that FR-MPM-PROJ consistently achieves the best performance compared to other baselines.
Interestingly, Wachter improves its future validity signiﬁcantly on the synthetic and German datasets as the threshold increases.
However, our method still dominates Wachter in all datasets.
Robust recourses with MPM’s variants.
We compare FR-MPM-PROJ with QUAD-MPM-PROJ and BW-MPM-PROJ.
The settings are similar in the cost-validity trade-off experiments in Section 6.2.
[Figure]
The number of samples is 1000.
The results are shown in Figure 13.
Generally, all three MPM-related methods perform better than LIME-ROAR and Wachter.
However, FR-MPM-PROJ is better than QUAD-MPM-PROJ and BW-MPM-PROJ, especially in synthetic and Student datasets [Figure]
Bures, and Fisher-Rao distances. When ρ +1 = ρ − 1 = 0 , all hyperplanes coincide and recover the vanilla MPM. All hyperplanes move towards the favorable class as the radius for the un favorable class ρ − 1 increases. At ρ − 1 = 10 in
Subﬁgure (c), the hyperplanes of Quadratic and Bures MPMs come close together which is distinct from the Fisher-Rao MPM’s hyperplane.
Notice that the Fisher-Rao MPM in Subﬁgure (c) tends to position in parallel to the major axis of the un favorable covariance matrix, which shows the dependence on (cid:98) Σ − 1 , see Proposition 5.4.
The Bures and Quad MPM hyperplanes in Subﬁgure (c) do not show any dependence on the covariance matrix, which aligns with the results in Proposition 4.4.
A. C OVARIANCE - ROBUST MPM S WITH D IFFERENT D IVERGENCES
In this section, we discuss the variants of covariance-robust MPMs with different distances and provide guidance for choosing the surrogate model in practice, especially at a low sample size.
Proposition 4.4 and Remark 4.5 showed that Quadratic MPM and Bures MPM coincide when one of the radii ρ y grows to inﬁnity and they are independent of the covariance matrices (cid:98) Σ y .
Meanwhile, the asymptotic hyperplane of the Fisher-Rao MPM when ρ y → ∞ aligns with axes of the covariance matrices (cid:98) Σ
y (see Proposition 5.4 and Figure 14).
It suggests that the Fisher-Rao MPM is not a suitable surrogate at low sample sizes as it relies on the estimate of the covariance matrices.
On the other hand, when the number of samples is sufﬁcient to estimate the covariance matrices accurately,
Fisher-Rao MPM would be better than Quadratic MPM and Bures MPM as it takes the geometry of the data into account when robustifying the surrogate.
We probe the performance of MPMs with different distances at low sample sizes to demonstrate our claim above.
[Figure]
Local ﬁdelity.
We probe the local ﬁdelity at low sample sizes and plot the result in Figure 15. The experiment settings are similar to those in Section 6.1. The number of samples is set in the range of [50 , 1000] . We also measure the average condition number of estimated covariance matrices for both positive and negative classes in Figure 15a.
It can be seen that the covariance matrices are ill-conditioned at 50 samples on SBA and Student datasets.
The ﬁdelity of FR-MPM is just slightly better than QUAD-MPM and BW-MPM.
When the number of samples increased, FR-MPM beneﬁted the most, and the gap between FR-MPM and QUAD-MPM (or BW-MPM) became more signiﬁcant.
It supports our claim that FR-MPM would better approximate the decision boundary when the number of samples is sufﬁcient for estimating the covariance matrices.
[Figure]
Robust recourses. We revisit the recourse generation with covariance-robust MPMs using quadratic distance (QUAD-MPM-PROJ) and Fisher-Rao distance (FR-MPM-PROJ), at which the surrogate is estimated with 50 and 1000 synthesized samples. We omit the comparison with Bures distance to ease the presentation as it behaves asymptotically like the MPM using the quadratic distance.
The results are shown in Figure 16.
The results showed that QUAD-MPM-PROJ would be better at a low sample size. When increasing the number of samples, the recourses constructed with Fisher-Rao MPM exhibit a better cost-validity trade-off. This result is consistent with our previous observation in the local ﬁdelity experiment.
B P ROOFS B. P ROOFS OF S ECTION
Proof of Proposition 3.4.
The covariance-robust MPM problem
[Equation]
is equivalent to
[Equation]
where we used the classiﬁcation rule that C θ ( x ) = sign( w (cid:62) x − b ) if and only if y ( w (cid:62) X − b ) ≥and that the feasible set takes the form Θ (cid:44) { θ = ( w, b ) ∈ R d +1 : w (cid:54) = 0 } . We claim that w = 0 is never optimal.
To see this, take w = 0 . Then, [Equation] which is independent of the random variable X and is either 0 or 1 no matter what b we choose.
Therefore,
[Equation]
and hence α = 0 , which is never optimal.
So, the domain of w can be relaxed from R d \ { 0 } to R d .
Problem (9) can then be further re-written as [Equation] Notice that here, equivalency means the optimal solution ( w (cid:63) , b (cid:63) ) of (10) will constitute the optimal solution θ (cid:63) =
( w (cid:63) , b (cid:63) ) of the original min-max-max problem.
Moreover, recall the deﬁnition of the ambiguity set
[Equation]
where P y ∼ ( (cid:98) µ y , Σ y ) means that the the distribution P y has mean (cid:98) µ y and covariance Σ y .
In other words, each element P y in the ambiguity U ϕy ( (cid:98) P y ) is determined by ﬁrst choosing a covariance matrix Σ y satisfying the divergence constraint ϕ (Σ y (cid:107) (cid:98) Σ y ) ≤
ρ y
and then picking a distribution P y having mean (cid:98) µ y and covariance Σ y .
Therefore, the worst-case probability admits a two-layer decomposition
[Equation]
Using Lanckriet et al.
(2001, Equation (6)), the inner maximum value is given by
[Equation]
Combining the last two equalities, we can express the constraint in problem (10) as
[Equation]
which upon rearranging
[Equation] becomes [Equation] Using the same argument as in Lanckriet et al.
(see equation (4) and the discussions following it in Lanckriet et al.
, we could show that the optimal θ = ( w, b ) must classify (cid:98) µ y correctly, i.e. ,
[Equation]
which yields that [Equation] As a consequence, problem (10) is equivalent to [Equation]
Using that τ ϕy ( w ) = [Equation] w [Equation]
Σ y w and that α (cid:55)→
[Equation]
α 1 − α is monotone increasing, the above problem is further equivalent to
[Equation]
[Equation]
From the constraints, we get
[Equation]
So, we can eliminate the variable b and reduce problem (12) to
[Equation]
The inequality constraint in problem (14) is equivalent to
[Equation]
Thus, we can eliminate the variable κ and rewrite problem (14) as
[Equation]
Using the deﬁnition of τ ϕy ( w ) , we can see that the above problem is homogeneous in w , which implies that we could further re-write it as [Equation] Finally, note that from (13) and (15), at optimality, we have [Equation]
and
[Equation]
This completes the proof.
Proof of Proposition 3.5.
First, following exactly the same arguments as in the proof of Proposition 3.4, we see that problem (6) is equivalent to
[Equation]
In the proof of Proposition 3.
[Equation]
we handle the maximum in the constraint by decomposing it into two layers of maximization problems (see (11)).
However, because of the Gaussian assumption, in this case, we have
[Equation]
For each ﬁxed Σ y
[Equation]
using elementary probability theory [Equation] we could calculate the Gaussian probability explicitly:
[Equation]
where Φ is the cumulative distribution function of the standard Gaussian random variable.
Therefore, problem (16) can be re-written as
[Equation]
Using the monotonicity of Φ , the constraints in problem (17) become [Equation] Using that τ ϕy ( w ) = [Equation]
y
[Equation]
S d
[Equation]
[Equation]
y
[Equation]
≤
[Equation]
y [Equation] w (cid:62) Σ y w and that Φ − 1 ( α ) is monotone increasing, problem (17) is further equivalent to
[Equation] which is the same problem as problem (12) in the proof of Proposition 3.4. Hence, problem (6) shares the same optimal solution as problem (3). This completes the proof.
B. P ROOFS OF S ECTION
We ﬁrst prove Proposition 4.3 to lay the foundation for the proof of Theorem 4.2.
Proof of Proposition 4.3.
By Nguyen et al.
(2021, Proposition 2.8), we have
[Equation]
Using the Sherman-Morrison formula (Bernstein, 2009, Corollary 2.8.8), we ﬁnd
[Equation]
Notice that the constraint γI (cid:31) ww (cid:62) is equivalent to γ > (cid:107) w (cid:107) 22 by Schur complement.
Thus, we have
[Equation]
The optimal γ can be found by using calculus, which is given by
[Equation]
with the corresponding optimal value
[Equation]
We thus have the necessary result.
We now prove Theorem 4.2.
Proof of Theorem 4.2.
Using the Bures divergence B , the optimization problem
[Equation]
becomes problem (7) by exploiting the analytical form of τ B y ( w ) in Proposition 4.3.
By invoking
Proposition 3.4, we obtain the postulated results on the optimal solution θ B for the case of the Bures divergence.
Proof of Proposition 4.4.
Note that problem (7) has a unique solution because the objective function is strictly convex and coercive.
Moreover, the optimal solution of (7) coincides with the optimal solution w (cid:63) ( λ ) of the following second-order cone program
[Equation]
where λ = 1 /ρ .
By a compactiﬁcation of W and applying Berge’s maximum theorem (Berge, 1963, pp. 115-116), the function w (cid:63) ( λ ) is continuous on a non-negative compact range of λ , and converges to w (cid:63) (0) as λ → 0 .
The optimal solution w (cid:63) (0) coincides with the solution of [Equation]
[Equation]
which is the Euclidean projection of the origin onto the hyperplane W .
An elementary computation conﬁrms that
[Equation]
Letting w B ∞ = w (cid:63) (0) completes the proof.
B. P ROOFS OF S ECTION
We ﬁrst provide the proof of Proposition 5.3.
Proof of Proposition 5.3.
Notice that
[Equation]
Using the transformation Z y ← (cid:98) Σ − 12 y Σ y (cid:98) Σ − 12 y , we have
[Equation]
with v = (cid:98) Σ 12 w .
We now proceed to show that the above optimization problem admits the maximizer
[Equation]
where U is an d × ( d − 1) orthonormal matrix whose columns are orthogonal to v .
First, by Nguyen et al.
(2019, Lemma C.1), the feasible region is compact.
Since the objective function v (cid:62) Z Y v is continuous in Z y , an optimal solution Z (cid:63)y exists.
Next, we ﬁrst claim that the constraint holds with equality at optimality.
Suppose that (cid:107) log Z (cid:63)y (cid:107) F < ρ y .
Then, for some small δ > 0 , the matrix
Z (cid:63)y + δ vv (cid:62) is feasible due to the continuity of the constraint function (cid:107) log Z y (cid:107) F and has a strictly better objective value than the optimal solution Z (cid:63)y .
This violates the optimality of Z (cid:63)y .
Hence, (cid:107) log Z (cid:63)y (cid:107) F = ρ y for any optimal solution Z (cid:63)y , and the problem is equivalent to
[Equation]
where O ( d ) is the set of d × d orthogonal matrices.
For any orthogonal matrix Q , the objective function
[Equation]
the right-hand side of which can be attained by setting
[Equation]
Therefore, our problem is further reduced to
[Equation]
It is then easy to see that at optimality, the optimal λ ∈ R d ++ must satisfy λ 2 = · · · = λ d = 1 and (log λ 1 ) 2 =
ρ 2 y .
Since λ 1 ≥ λ 2 = 1 , we have log λ 1 =
ρ y and hence λ 1 = exp( ρ y ) .
In other words,
[Equation]
The corresponding optimal value is
[Equation]
This completes the proof.
We are now ready to prove Theorem 5.2.
Proof of Theorem 5.2.
Using the Fisher-Rao divergence F , the optimization problem
[Equation]
becomes problem (8) by exploiting the analytical form of τ F y ( w ) in Proposition 5.3.
By invoking
Proposition 3.4, we obtain the postulated results on the optimal solution θ F for the case of the
Fisher-Rao divergence.
Proof of Proposition 5.4.
Note that problem (8) has a unique solution because the objective function is strictly convex and coercive.
Also, the optimal solution of (8) coincides with the solution w (cid:63) ( λ ) of
[Equation]
where λ = exp (cid:0) ρ − y − ρ y(cid:1) .
By a compactiﬁcation of W and applying Berge’s maximum theorem (Berge, 1963, pp. 115-116), the function w (cid:63) ( λ ) is continuous on a non-negative compact range of λ , and converges to w (cid:63) (0) as λ → 0 .
The optimal solution w (cid:63) (0) coincides with the solution of
[Equation]
Because the square-root function is monotonically increasing, w (cid:63) (0) also solves
[Equation]
which is a convex, quadratic program with a single linear constraint.
If a is deﬁned as in the statement, then a convex optimization argument implies
[Equation]
which completes the proof.
C L OG D ET MPM
In this appendix, we consider when ϕ is the Log-Determinant (LogDet) divergence.
The LogDet divergence is formally deﬁned as follows.
Deﬁnition C.
(LogDet divergence) .
Given two positive deﬁnite matrices Σ , (cid:98) Σ ∈ S d ++ , the logdeterminant divergence between them is
[Equation]
It can be shown that D is a divergence because it is non-negative, and it vanishes to zero if and only if Σ = (cid:98) Σ .
However, D is not symmetric, and in general we have D (Σ (cid:107) (cid:98) Σ) =
D ( (cid:98) Σ (cid:107) Σ) .
The LogDet divergence D is related to the relative entropy: it is equal to the Kullback-Leibler divergence between two Gaussian distributions with the same mean vector and covariance matrices Σ and (cid:98) Σ .
We now provide the form of the LogDet MPM problem.
Theorem C.2 (LogDet MPM) .
Suppose that ϕ ≡ D .
Let w D be the optimal solution of the following second-order cone problem
[Equation]
where c y = − W − 1 ( − exp( − ρ y − 1)) and W − 1 is the Lambert-W function for the branch − 1 .
Let κ D and b D be calculated as
[Equation]
then θ D = ( w D , b D ) is the optimal solution of the distributionally robust MPM problem (3) .
Theorem C.2 shows that the LogDet divergence induces a similar reweighting scheme as the FisherRao MPM.
The asymptotic analysis of the LogDet MPM follows similarly from the Fisher-Rao MPM and is omitted.
The proof of Theorem C.2 follows trivially from the below result, which provides the analytical form of τ D y ( w ) .
Proposition C.3 (LogDet divergence) .
Suppose that ϕ ≡ D , then for any y ∈ Y , we have
[Equation]
where W − 1 is the Lambert-W function for the branch − 1 .
Proof of Proposition C.3.
By Le et al.
(2021, Proposition 3.4), we have
[Equation]
Using the matrix determinant formula (Bernstein, 2009), we have
[Equation]
Notice that the constraint γ (cid:98) Σ − 1 y (cid:31) ww
[Equation]
is translated into γ > w (cid:62) (cid:98) Σ y w .
Thus, we have
[Equation]
The ﬁrst-order optimality condition for γ is
[Equation]
and the optimal solution for γ is
[Equation]
Replacing the value of γ (cid:63) into the objective function leads to the necessary result.