D ISTRIBUTIONALLY R OBUST F AIR P RINCIPAL C OMPONENTS VIA G EODESIC D ESCENTS
A BSTRACT
Principal component analysis is a simple yet useful dimensionality reduction technique in modern machine learning pipelines.
In consequential domains such as college admission, healthcare and credit approval, it is imperative to take into account emerging criteria such as the fairness and the robustness of the learned projection.
In this paper, we propose a distributionally robust optimization problem for principal component analysis which internalizes a fairness criterion in the objective function.
The learned projection thus balances the trade-off between the total reconstruction error and the reconstruction error gap between subgroups, taken in the min-max sense over all distributions in a moment-based ambiguity set.
The resulting optimization problem over the Stiefel manifold can be efﬁciently solved by a Riemannian subgradient descent algorithm with a sub-linear convergence rate.
Our experimental results on real-world datasets show the merits of our proposed method over state-of-the-art baselines.
I NTRODUCTION
Machine learning models are ubiquitous in our daily lives and supporting the decision-making process in diverse domains.
With their ﬂourishing applications, there also surface numerous concerns regarding the fairness of the models’ outputs (Mehrabi et al., 2021).
Indeed, these models are prone to biases due to various reasons (Barocas et al., 2018).
First, the collected training data is likely to include some demographic disparities due to the bias in the data acquisition process (e g., conducting surveys on a speciﬁc region instead of uniformly distributed places) or the imbalance of observed events at a speciﬁc period of time.
Second, because machine learning methods only care about data statistics and are objective driven, groups that are under-represented in the data can be neglected in exchange for a better objective value.
Finally, even human feedback to the predictive models can also be biased, e.g., click counts are human feedback to recommendation systems but they are highly correlated with the menu list suggested previously by a potentially biased system.
Real-world examples of machine learning models that amplify biases and hence potentially cause unfairness are commonplace, ranging from recidivism prediction giving higher false positive rates for African-American 1 to facial recognition systems having large error rate for women 2 .
To tackle the issue, various fairness criteria for supervised learning have been proposed in the literature, which encourage the (conditional) independence of the model’s predictions on a particular sensitive attribute (Dwork et al., 2012; Hardt et al., 2016b; Kusner et al., 2017; Chouldechova, 2017;
Verma & Rubin, 2018; Berk et al., 2021).
Strategies to mitigate algorithmic bias are also investigated for all stages of the machine learning pipelines (Berk et al., 2021).
For the pre-processing steps, (Kamiran & Calders, 2012) proposed reweighting or resampling techniques to achieve statistical parity between subgroups; in the training steps, fairness can be encouraged by adding constraints (Donini et al., 2018) or regularizing the original objective function (Kamishima et al., 2012; Zemel et al., 2013); and in the post-processing steps, adjusting classiﬁcation threshold by examining black-box models over a holdout dataset can be used (Hardt et al., 2016b; Wei et al., 2019).
Since biases may already exist in the raw data, it is reasonable to demand machine learning pipelines to combat biases as early as possible.
We focus in this paper on the Principal Component Analysis (PCA), which is a fundamental dimensionality reduction technique in the early stage of the a r X i v : .
03071v [ c s . L G ] F e b pipelines (Pearson, 1901; Hotelling, 1933).
PCA ﬁnds a linear transformation that embeds the original data into a lower-dimensional subspace that maximizes the variance of the projected data.
Thus, PCA may amplify biases if the data variability is different between the majority and the minority subgroups, see an example in Figure 1.
A naive approach to promote fairness is to train one independent transformation for each subgroup.
However, this requires knowing the sensitive attribute of each sample, which would raise disparity concerns.
On the contrary, using a single transformation for all subgroups is “group-blinded” and faces no discrimination problem (Lipton et al., 2018).
Learning a fair PCA has attracted attention from many ﬁelds from machine learning, statistics to signal processing.
Samadi et al.
and Zalcberg & Wiesel (2021) propose to ﬁnd the principal components that minimize the maximum subgroup reconstruction error; the min-max formulations can be relaxed and solved as semideﬁnite programs.
Olfat & Aswani (2019) propose to learn a transformation that minimizes the possibility of predicting the sensitive attribute from the projected data.
Apart from being a dimensionality reduction technique, PCA can also be thought of as a representation learning toolkit.
Viewed in this way, we can also consider a more general family of fair representation learning methods that can be applied before any further analysis steps.
There are a number of works develop towards this idea (Kamiran & Calders, 2012; Zemel et al., 2013;
Calmon et al., 2017; Feldman et al., 2015; Beutel et al., 2017; Madras et al., 2018; Zhang et al., 2018; Tantipongpipat et al., 2019), which apply a multitude of fairness criteria.
In addition, we also focus on the robustness criteria for the linear transformation.
Recently, it has been observed that machine learning models are susceptible to small perturbations of the data (Goodfellow et al., 2014; Madry et al., 2017; Carlini & Wagner, 2017).
These observations have fuelled many defenses using adversarial training (Akhtar & Mian, 2018;
Chakraborty et al., 2018) and distributionally robust optimization (Rahimian & Mehrotra, 2019; Kuhn et al., 2019; Blanchet et al., 2021).
Contributions.
This paper blends the ideas from the ﬁeld of fairness in artiﬁcal intelligence and distributionally robust optimization.
Our contributions can be described as follows.
• We propose the fair principal components which balance between the total reconstruction error and the absolute gap of reconstruction error between subgroups.
Moreover, we also add a layer of robustness to the principal components by considering a min-max formulation that hedges against all perturbations of the empirical distribution in a moment-based ambiguity set.
• We provide the reformulation of the distributionally robust fair PCA problem as a ﬁnitedimensional optimization problem over the Stiefel manifold.
We provide a Riemannian gradient descent algorithm and show that it has a sub-linear convergence rate.
Figure 1 illustrates the qualitative comparison between (fair) PCA methods and our proposed method on a 2-dimensional toy example.
The majority group (blue dots) spreads on the horizontal axis, while the minority group (yellow triangles) spreads on the slanted vertical axis.
The nominal PCA (red) captures the majority direction to minimize the total error, while the fair PCA of Samadi et al.
(2018) returns the diagonal direction to minimize the maximum subgroup error.
Our fair PCA can probe the full spectrum in between these two extremes by sweeping through our penalization parameters appropriately.
If we do not penalize the error gap between subgroups, we recover the PCA method; if we penalize heavily, we recover the fair PCA of Samadi et al.
Extensive numerical results on real datasets are provided in
Section 5.
Proofs are relegated to the appendix.
F AIR P RINCIPAL C OMPONENT A NALYSIS 2. P RINCIPAL C OMPONENT A NALYSIS
We ﬁrst brieﬂy revisit the classical PCA.
Suppose that we are given a collection of N i.i.d.
samples { ˆ x i } Ni =1 generated by some underlying distribution P .
For simplicity, we assume that both the empirical and population mean are zero vectors.
The goal of PCA is to ﬁnd a k -dimensional linear subspace of R d that explains as much variance contained in the data { ˆ x i } Ni =1 as possible, where k < d is a given integer.
More precisely, we parametrize k -dimensional linear subspaces by orthonormal matrices, i.e., matrices whose columns are orthogonal and have unit Euclidean norm.
Given any such matrix V , the associated k -dimensional subspace is the one spanned by the columns of V .
The projection matrix onto the subspace is V V (cid:62) , and hence the variance of the projected data is given by tr (cid:0) V V (cid:62) ΞΞ (cid:62) (cid:1) ,
where Ξ = [ˆ x 1 , · · · , ˆ x N ] ∈ R d × N is the data matrix.
By a slight abuse of terminology, sometimes we refer to V as the projection matrix.
The problem of PCA then reads
[Equation]
For any vector X ∈ R d and orthonormal matrix V , denote by (cid:96) ( V, X ) the reconstruction error, i.e.,
[Equation]
) X.
The problem of PCA can alternatively be formulated as a stochastic optimization problem
[Equation]
where ˆ P is the empirical distribution associated with the samples { ˆ x i } Ni =1 and X ∼ ˆ P .
It is wellknown that PCA admits an analytical solution.
In particular, the optimal solution to problem (2) (and also problem (1)) is given by any orthonormal matrix whose columns are the eigenvectors associated with the k largest eigenvalues of the sample covariance matrix ΞΞ (cid:62) .
2. F AIR P RINCIPAL C OMPONENT A NALYSIS
In the fair PCA setting, we are also given a discrete sensitive attribute A ∈ A , where A may represent features such as race, gender or education.
We consider binary attribute A and let A = { 0 , 1 } .
A straightforward idea to deﬁne fairness is to require the (strict) balance of a certain objective between the two groups.
For example, this is the strategy in Hardt et al.
(2016a) for developing fair supervised learning algorithms.
A natural objective to balance in the PCA context is the reconstruction error.
Deﬁnition 2.1 (Fair projection) .
Let Q be an arbitrary distribution of ( X, A ) .
A projection matrix
V ∈ R d × k is fair relative to Q if the conditional expected reconstruction error is equal between subgroups, i.e., E Q [ (cid:96) ( V, X )
| A = a ] = E Q [ (cid:96) ( V, X ) | A = a (cid:48) ] for any ( a, a (cid:48) ) ∈ A × A .
Unfortunately, Deﬁnition 2.1 is too stringent: for a general probability distribution Q , it is possible that there exists no fair projection matrix V .
Proposition 2.2 (Impossibility result) .
For any distribution Q on X ×A , there exists a fair projection matrix V ∈ R d × k relative to Q if and only if rank( E Q [ XX (cid:62) | A = 0] − E Q [ XX (cid:62) |
A = 1]) ≤ k .
One way to circumvent the impossibility result is to relax the requirement of strict balance to approximate balance.
In other words, an inequality constraint of the following form is imposed:
[Equation]
where (cid:15) > 0 is some prescribed fairness threshold.
This approach has been adopted in other fair machine learning settings, see Donini et al.
(2018) and Agarwal et al.
for example.
In this paper, instead of imposing the fairness requirement as a constraint, we penalize the unfairness in the objective function.
Speciﬁcally, for any projection matrix V , we deﬁne the unfairness as the absolute difference between the conditional loss between two subgroups:
[Equation]
We thus consider the following fairness-aware PCA problem
[Equation]
where λ ≥ 0 is a penalty parameter to encourage fairness.
Note that for fair PCA, the dataset is { (ˆ x i , ˆ a i ) } N i =1 and hence the empirical distribution ˆ P is given by ˆ P = 1 N (cid:80) Ni
=1 δ (ˆ x i , ˆ a i ) .
D ISTRIBUTIONALLY R OBUST F AIR PCA
The weakness of empirical distribution-based stochastic optimization has been well-documented, see (Smith & Winkler, 2006; Homem-de Mello & Bayraksan, 2014).
In particular, due to overﬁtting, the out-of-sample performance of the decision, prediction, or estimation obtained from such a stochastic optimization model is unsatisfactory, especially in the low sample size regime.
Ideally, we could improve the performance by using the underlying distribution P instead of the empirical distribution ˆ P .
But the underlying distribution P is unavailable in most practical situations, if not all.
Distributional robustiﬁcation is an emerging approach to handle this issue and has been shown to deliver promising out-of-sample performance in many applications (Delage & Ye, 2010; Namkoong & Duchi, 2017; Kuhn et al., 2019; Rahimian & Mehrotra, 2019).
Motivated by the success of distributional robustiﬁcation, especially in machine learning Nguyen et al. ; Taskesen et al. (2021), we propose a robustiﬁed version of model (3), called the distributionally robust fairness-aware PCA:
[Equation]
(4) where B (ˆ P ) is a set of probability distributions similar to the empirical distribution ˆ P in a certain sense, called the ambiguity set.
The empirical distribution ˆ P is also called the nominal distribution.
Many different ambiguity sets have been developed and studied in the optimization literature, see Rahimian & Mehrotra (2019) for an extensive overview.
3. T HE W ASSERSTEIN - TYPE A MBIGUITY S ET
To present our ambiguity set and main results, we need to introduce some deﬁnitions and notations.
Deﬁnition 3.1 (Wasserstein-type divergence) .
The divergence W between two probability distributions Q 1 ∼ ( µ 1 , Σ 1 ) ∈ R d × S d + and Q 2 ∼ ( µ 2 , Σ 2 ) ∈ R
d × S d + is deﬁned as
[Equation]
The divergence W coincides with the squared type- 2 Wasserstein distance between two Gaussian distributions N ( µ 1 , Σ 1 ) and N ( µ 2 , Σ 2 ) (Givens & Shortt, 1984).
One can readily show that W is non-negative, and it vanishes if and only if ( µ 1 , Σ 1 ) =
( µ 2 , Σ 2 ) , which implies that Qand Q 2 have the same ﬁrst- and second-moments.
Recently, distributional robustiﬁcation with
Wasserstein-type ambiguity sets has been applied widely to various problems including domain adaption (Taskesen et al., 2021), risk measurement (Nguyen et al., 2021b) and statistical estimation (Nguyen et al., 2021a).
The Wasserstein-type divergence in Deﬁnition 3.1 is also related to the theory of optimal transport with its applications in robust decision making (Mohajerin Esfahani &
Kuhn, 2018; Blanchet & Murthy, 2019;
Yue et al., 2021) and potential applications in fair machine learning (Taskesen et al., 2020; Si et al., 2021; Wang et al., 2021).
Recall that the nominal distribution is ˆ P = 1 N (cid:80)
Ni =1 δ (ˆ x i , ˆ a i ) .
For any a ∈ A , its conditional distribution given A = a is given by
[Equation]
where I
[Equation]
We also use (ˆ µ a , ˆΣ a ) to denote the empirical mean vector and covariance matrix of X given A = a : ˆ µ a = E
ˆ P a [ X ] =
E ˆ P [ X |
A = a ] and ˆΣ a + ˆ µ a ˆ µ (cid:62) a
=
E
ˆ P a [ XX (cid:62) ] = E ˆ P [ XX (cid:62) |
A = a ] . For any a ∈ A , the empirical marginal distribution of A is denoted by ˆ p a = |I a | /N .
Finally, for any set S , we use P ( S ) to denote the set of all probability distributions supported on S .
For any integer k , the k -by- k identity matrix is denoted I k .
We then deﬁne our ambiguity set as
[Equation] 
[Equation]
: ∃ Q
[Equation]
such that:
[Equation]
where Q a is the conditional distribution of X
| A = a .
Intuitively, each Q ∈ B (ˆ P ) is a joint distribution of the random vector ( X, A ) , formed by taking a mixture of conditional distributions Q a with mixture weight ˆ p a .
Each conditional distribution Q a is constrained in an ε a -neighborhood of the nominal conditional distribution ˆ P a with respect to the W divergence.
Because the loss function (cid:96) is a quadratic function of X , the (conditional) expected losses only involve the ﬁrst two moments of X , and thus prescribing the ambiguity set using W would sufﬁce for the purpose of robustiﬁcation.
3. R EFORMULATION
We now present the reformulation of problem (4) under the ambiguity set B (ˆ P ) .
Theorem 3.2 (Reformulation) .
Suppose that for any a ∈ A , either of the following two conditions holds:
(i) Marginal probability bounds: 0 ≤ λ ≤ ˆ p a , (ii) Eigenvalue bounds: the empirical second moment matrix ˆ M a = 1 N a (cid:80) i ∈I a ˆ x i ˆ x (cid:62) i satisﬁes (cid:80) d − k j =1 σ j
( ˆ M a ) ≥ ε a , where σ j ( ˆ M a ) is the j -th smallest eigenvalues of ˆ M a .
Then problem (4) is equivalent to
[Equation]
(6a) where for each ( a, a (cid:48) )
∈ { (0 , 1) , (1 , 0) } , the function J a is deﬁned as
[Equation]
[Equation]
6b) and the parameters κ ∈ R , θ ∈ R , ϑ ∈ R and C ∈ S
d
+ are deﬁned as
[Equation]
We now brieﬂy explain the steps that lead to the results in Theorem 3.2.
Letting
[Equation]
then by expanding the term U
( V
[Equation]
Q ) using its deﬁnition
[Equation]
problem (4) becomes
[Equation]
Leveraging the deﬁnition the ambiguity set B (ˆ P ) , for any pair ( a, a (cid:48) )
∈ { (0 , 1) , (1 , 0) }
[Equation]
we can decompose J a into two separate supremum problems as follows
[Equation]
The next proposition asserts that each individual supremum in the above expression admits an analytical expression.
Proposition 3.3 (Reformulation) .
Fix a ∈ A .
For any υ ∈ R , ε a ∈ R + , it holds that
[Equation]
=  
[Equation]
if
[Equation] [Equation]
if υ < 0 and (cid:10) I d − V V (cid:62) , ˆ M a (cid:11) ≥ ε a , if υ < 0 and (cid:10) I d − V V (cid:62) , ˆ M a (cid:11) <
ε a .
The proof of Theorem 3.2 now follows by applying Proposition 3.3 to each term in J a , and balance the parameters to obtain (6c).
A detailed proof is relegated to the appendix.
In the next section, we study an efﬁcient algorithm to solve problem (6a).
Remark 3.4 (Recovery of the nominal PCA) .
If λ = 0 and ε a = 0 ∀ a ∈ A , our formulation (4) becomes the standard PCA problem (2).
In this case, our robust fair principal components reduce to the standard principal components.
On the contrary, existing fair PCA methods such as Samadi et al.
and Olfat & Aswani (2019) cannot recover the standard principal components.
R IEMANNIAN G RADIENT D ESCENT A LGORITHM
The distributionally robust fairness-aware PCA problem (4) is originally an inﬁnite-dimensional min-max problem.
Indeed, the inner maximization problem in (4) optimizes over the space of probability measures.
Thanks to Theorem 3.2, it is reduced to the simpler ﬁnite-dimensional minimax problem (6a), where the inner problem is only a maximization over two points.
Problem (6a) is, however, still challenging as it is a non-convex optimization problem over a non-convex feasible region deﬁned by the orthogonality constraint V (cid:62)
V = I d .
The purpose of this section is to devise an efﬁcient algorithm for solving problem (6a) to local optimality based on Riemannian optimization.
4. R EPARAMETRIZATION
As mentioned above, the non-convexity of problem (6a) comes from both the objective function and the feasible region.
It turns out that we can get rid of the non-convexity of the objective function via a simple change of variables.
To see that, we let U ∈ R d × ( d − k ) be an orthonormal matrix complement to V , that is, U and V satisfy UU (cid:62) + V V (cid:62)
=
I d .
Thus, we can express the objective function J via
J ( V ) = F
[Equation]
, where for ( a, a (cid:48) ) ∈ { (0 , 1) , (1 , 0) } , the function F a is deﬁned as
[Equation]
Moreover, letting M (cid:44) { U ∈ R d × ( d − k ) : U (cid:62) U
= I
d − k } , we can re-express problem (6a) as
[Equation]
The set M of problem (7) is a Riemannian manifold, called the Stiefel manifold (Absil et al., 2007, Section 3.3.2).
It is then natural to solve (7) using a Riemannian optimization algorithms (Absil et al., 2007).
In fact, problem (6a) itself (before the change of variables) can also be cast as a Riemannian optimization problem over another Stiefel manifold.
The change of variables above might seem unnecessary.
Nonetheless, the upshot of problem (7) is that the objective function F is convex (in the traditional sense).
This faciliates the application of the theoretical and algorithmic framework developed in Li et al.
(2021) for (weakly) convex optimization over the Stiefel manifolds.
4. T HE R IEMANNIAN S UBGRADIENT
Note that the objective function F is non-smooth since it is deﬁned as the maximum of two functions F 0 and F 1 .
To apply the framework in Li et al.
(2021), we need to compute the Riemannian subgradient of the objective function F .
Since the Stiefel manifold M is an embedded manifold in
Euclidean space, the Riemannian subgradient of F at any point U ∈ M is given by the orthogonal projection of the usual Euclidean subgradient onto the tangent space of the manifold M at the point
U , see Absil et al.
(2007, Section 3.6.1) for example.
Lemma 4.1.
For any point U ∈ M , let 3 a U ∈ arg max a ∈{ 0 , 1 } F a ( U ) and
a (cid:48) U = 1 − a U .
Then, a
Riemannian subgradient of the objective function F at the point U is given by grad
[Equation]
4. R ETRACTIONS
Another important instrument required by the framework in Li et al.
(2021) is a retraction of the
Stiefel manifold M .
At each iteration, the point U − γ ∆ obtained by moving from the current iterate U in the opposite direction of the Riemannian gradient ∆ may not lie on the manifold in general, where γ > 0 is the stepsize.
In Riemannian optimization, this is circumvented by the concept of retraction.
Given a point U ∈ M on the manifold, the Riemannian gradient ∆ ∈
T U M (which must lie in the tangent space T U M ) and a stepsize γ , the retraction map Rtr deﬁnes a point Rtr U ( − γ ∆) which is guaranteed to lie on the manifold M .
Roughly speaking, the retraction
Rtr U ( · ) approximates the geodesic curve through U along the input tangential direction.
For a formal deﬁnition of retractions, we refer the readers to (Absil et al., 2007, Section 4.1).
In this paper, we focus on the following two commonly used retractions for Stiefel manifolds.
The ﬁrst one is the QR decomposition-based retraction using the Q-factor qf ( · ) in the QR decomposition:
[Equation]
The second one is the polar decomposition-based retraction
[Equation]
4. A LGORITHM AND C ONVERGENCE G UARANTEES
Associated with any choice of retraction Rtr is a concrete instantiation of the Riemannian subgradient descent algorithm for our problem (7), which is presented in Algorithm 1 with speciﬁc choice of the stepsizes γ t motivated by the theoretical results of (Li et al., 2021).
U 0 , a number of iterations τ and a retraction Rtr : ( U, ∆) (cid:55)→ Rtr U (∆) .
2: for t = 0 , 1 , . . .
, τ − 1 , do 3:
Find a t (cid:44) arg max a ∈{ 0 , 1 } { F a ( U t ) } .
Compute the Riemannian subgradient ∆ t = grad F ( U t ) using the formula
[Equation]
t =
( I
[Equation]
U t U t (cid:62)
[Equation]
Set U t +1 =
Rtr U t ( − γ t ∆ t ) , where the step-size γ t ≡ 1 √ τ +1 is constant.
6: end for 7: Output: U τ .
We now study the convergence guarantee of Algorithm 1.
The following lemma shows that the objective function F is Lipschitz continuous (with respect to the Riemannian metric on the Stiefel manifold M ) with an explicit Lipschitz constant L .
Lemma 4.2 (Lipschitz continuity) .
The function F is L -Lipschitz continuous on M , where L >is given by
[Equation]
We now proceed to show that Algorithm 1 enjoys a sub-linear convergence rate.
To state the result, we deﬁne the Moreau envelope
[Equation]
where (cid:107) · (cid:107) F denotes the Frobenius norm of a matrix.
Also, to measure the progress of the algorithm, we need to introduce the proximal mapping on the Stiefel manifold (Li et al., 2021):
[Equation]
From Li et al.
(2021, Equation (22)), we have that
[Equation]
Therefore, the number gap µ ( U ) is a good candidate to quantify the progress of optimization algorithms for solving problem (7).
Theorem 4.3 (Convergence guarantee) .
Let { U t } t =1 ,...,τ be the sequence of iterates generated by
Algorithm 1.
Suppose that µ = 1 / 4 L , where L is the Lipschitz constant of F in (9).
Then, we have
[Equation]
N UMERICAL E XPERIMENTS
We compare our proposed method, denoted RFPCA , against two state-of-the-art methods for fair
PCA: 1) FairPCA Samadi et al.
(2018) 4 , and 2) CFPCA Olfat & Aswani (2019) 5 with both cases: only mean constraint, and both mean and covariance constraints.
We consider a wide variety of datasets with ranging sample sizes and number of features.
Further details about the datatasets can be found in Appendix C.
The code for all experiments is available in supplementary materials.
We include here some details about the hyper-parameters that we search in the cross-validation steps.
• RFPCA .
We notice that the neighborhood size ε a should be inversely proportional to the size of subgroup a .
Indeed, a subgroup with large sample size is likely to have more reliable estimate of the moment information.
Then we parameterize the neighborhood size ε a by a common scalar α , and we have ε a = α/ √ N a , where N a is the number of samples in group a .
We search α ∈ { 0 . 05 , 0 . 1 , 0 . 15 } and λ ∈ { 0 ., 0 .
For better convergence quality, we set the number of iteration for our subgradient descent algorithm to τ = 1000 and also repeat the Riemannian descent for 20 randomly generated initial point U 0 .
• FairPCA .
According to Samadi et al.
, we only need tens of iterations for the multiplicative weight algorithm to provide good-quality solution; however, to ensure a fair comparison, we set the number of iterations to 1000 for the convergence guarantee.
We search the learning rate η of the algorithm from set of 17 values evenly spaced in [0 . 25 , 4 . 25] and { 0 . 1 } .
• CFPCA .
Following Olfat & Aswani (2019), for the mean-constrained version of CFPCA , we search δ from { 0 ., 0 . 1 , 0 . 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } , and for both the mean and covariance constrained version, we ﬁx δ = 0 while searching µ in { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 05 , 0 . 5 } .
Trade-offs.
First, we examine the trade-off between the total reconstruction error and the gap between the subgroup error.
In this experiment, we only compare our model with FairPCA and
CFPCA mean-constraint version.
We plot a pareto curve for each of them over the two criteria with different hyper-parameters (hyper-parameters test range are mentioned above).
The whole datasets are used for training and evaluation.
The results averaged over 5 runs are shown in Figure 2.
In testing methods with different principal components, we ﬁrst split each dataset into training set and test set with equal size ( 50% each), the projection matrix of each method is learned from training set and tested over both sets.
In this case, we only compare our method with traditional PCA and
FairPCA method.
We ﬁx one set hyper-parameters for each method.
For FairPCA , we set η = 0 .and
for RFPCA we set α = 0 .
15 , λ = 0 .
5 , others hyper-parameters are kept as discussed before.
The results are averaged over 5 different splits.
Figure 3 shows the consistence of our method performing fair projections over different values of k .
Our method (cross) exhibits smaller gap of subgroup errors.
More results and discussions on the effect of ε can be found in Appendix D.2.
Cross-validations.
Next, we report the performance of all methods based on three criteria: absolute difference between average reconstruction error between groups (ABDiff.), average reconstruction error of all data (ARE.), and the fairness criterion deﬁned by Olfat & Aswani (2019) with respect to a linear SVM’s classiﬁer family ( (cid:52)F Lin ).
6 Due to the space constraint, we only include the ﬁrst two criteria in the main text, see Appendix 4 for full results.
To emphasize the generalization capacity of each algorithm, we split each dataset into a training set and a test set with ratio of 30% − 70% respectively, and only extract top three principal components from the training set.
We ﬁnd the best hyper-parameters by 3 -fold cross validation, and prioritize the one giving minimum value of the summation (ABDiff.
+ ARE.).
The results are averaged over 10 different training-testing splits.
We report the performance on both training set (In-sample data) and test set (Out-of-sample data).
The details results for Out-of-sample data is given in Table 1, more details about settings and performance can be found at Appendix D.
Results.
Our proposed RFPCA method outperforms on 11 out of 15 datasets in terms of the subgroup error gap ABDiff, and 9 out of 15 with the totall error ARE.
criterion.
There are 5 datasets that
RFPCA gives the best results for both criteria, and for the remaining datasets, RFPCA has small performance gaps compared with the best method.
[Table]
Taiwan Credit
[Table]
A P ROOFS A. P ROOFS OF S ECTION
Proof of Proposition 2.2.
Let S =
E Q [ XX (cid:62) |
A = 0] − E Q [ XX (cid:62) |
A = 1] .
We ﬁrst prove the “only if” direction.
Suppose that there exists a fair projection matrix V ∈ M k relative to Q .
Let U ∈ M d − k be a complement matrix of V .
Then
Deﬁnition 2.1 can be rewritten as
[Equation]
which implies that the null space of S has a dimension at least d − k .
By the rank-nullity duality, we have rank( S ) ≤ k .
Next, we prove the “if” direction.
Suppose that rank( S ) ≤ k .
Then, the matrix S has at least d − k (repeated) zero eigenvalues.
Let U ∈ M d − k be an orthonormal matrix whose columns are any d − k eigenvectors corresponding to the zero eigenvalues of S and V ∈
M k be a complement matrix of
U .
Then,
[Equation]
Therefore, V is a fair projection matrix relative to Q .
This completes the proof.
A. P ROOF OF S ECTION
Proofs of Proposition 3.3.
By exploiting the deﬁnition of the loss function (cid:96) , we ﬁnd
[Equation]
where the last equality follows from Nguyen (2019, Lemma 3.22).
By the Woodbury matrix inversion, we have
[Equation]
Moreover
[Equation]
using the Schur complement
[Equation]
the semideﬁnite constraint is equivalent to
[Equation]
which implies that at optimality [Equation] we have
[Equation]
At the same time [Equation] the constraint γI (cid:31) υ
( I − V V (cid:62) ) is equivalent to γ > υ .
Combining all previous equations, we have
[Equation]
The dual optimal solution γ (cid:63) is given by
[Equation]
if υ < 0 and (cid:10) I d
[Equation]
V V (cid:62) [Equation] ˆ M [Equation]
(cid:11) [Equation] if υ < 0 and (cid:10) I d [Equation]
V V (cid:62) [Equation] ˆ M a (cid:11) <
ε a .
Note that γ (cid:63) ≥ max { 0 , υ } in all the cases.
Therefore, we have
[Equation]
[Equation]
This completes the proof.
We are now ready to prove Theorem 3.2.
Proof of Theorem 3.2.
By expanding the absolute value, problem (4) is equivalent to
[Equation]
where for each ( a, a (cid:48) )
∈ { (0 , 1) , (1 , 0) } , we can re-express J a as
[Equation]
Using Proposition 3.3 to reformulate the two individual supremum problems
[Equation]
we have
[Equation]
By deﬁning the necessary parameters κ , θ , ϑ and C as in the statement of the theorem, we arrive at the postulated result.
A. P ROOFS OF S ECTION
Proof of Lemma 4.1.
Let a U ∈ arg max a ∈{ 0 , 1 } F a ( U ) and
a (cid:48) U = 1 − a U .
Then, an Euclidean subgradient of F is given by
[Equation]
The tangent space of the Stiefel manifold M at U is given by
[Equation]
whose orthogonal projection (Absil et al., 2007, Example 3.6.2) can be computed explicitly via [Equation] Therefore
[Equation]
a Riemannian subgradient of F at any point U ∈ M is given by grad [Equation] ( U [Equation]
Proj T
[Equation]
In the last line
[Equation]
we have used the fact that, if D = SU for some symmetric matrix S , then
[Equation]
This completes the proof.
The proof of Lemma 4.2 relies on the following preliminary result.
Lemma A.1.
Let M ∈ R
( d − k ) × ( d − k ) be a positive deﬁnite matrix.
Then,
[Equation]
and
[Equation]
where σ max ( M ) and σ min ( M ) denote the maximum and minimum eigenvalues of the matrix M
[Equation]
Proof of Lemma A.1.
For inequality (10),
[Equation]
For inequality (11), we ﬁrst note that the function x (cid:55)→ √ x is 1 / (2 √ x min ) -Lipschitz on [ x min , + ∞ ) and that
[Equation]
M ) ∀ U ∈ M .
Therefore,
[Equation]
where the last inequality follows from (10).
This completes the proof.
We are now ready to prove Lemma [Equation]
[Equation]
Proof of Lemma 4.2.
Let U , U (cid:48)
∈
M be two arbitrary points.
We have
[Equation]
where the last inequality follows from the deﬁnition of F a and Lemma A.1.
This completes the proof.
Proof of Theorem 4.3.
The proof follows from the fact that F is convex on the Euclidean space
R d × ( d − k ) , Lemma 4.2 and Li et al.
(2021, Theorem 2) (and the remarks following it).
B E XTENSION TO N ON - BINARY S ENSITIVE A TTRIBUTES
The main paper focuses on the case of a binary sensitive attribute with A = { 0 , 1 } .
In this appendix, we extend our approach to the case when the sensitive attribute is non-binary.
Concretely, we suppose that the sensitive attribute A can take on any of the m possible values from 1 to m .
In other words, the attribute space now becomes A = { 1 , . . .
, m } .
Deﬁnition B.1 (Generalized unfairness measure) .
The generalized unfairness measure is deﬁned as the maximum pairwise unfairness measure, that is,
[Equation]
Notice that if A = { 0 , 1 } , then U max ≡ U recovers the unfairness measure for binary sensitive attribute deﬁned in Section 2.2.
We now consider the following generalized fairness-aware PCA problem
[Equation]
Here we recall that the ambiguity set B (ˆ P ) is deﬁned in (5).
The next theorem provides the reformulation of (12).
Theorem B.2 (Reformulation of non-binary fairness-aware PCA) .
Suppose that for any a ∈ A , either of the following two conditions holds:
(i) [-
-]Marginal probability bounds: 0 ≤ λ ≤ ˆ p a , (ii) Eigenvalue bounds: the empirical second moment matrix ˆ
[Equation] i [Equation]
i [Equation] i satisﬁes (cid:80) d − k j =1 σ j ( ˆ M a ) ≥ ε a , where σ j ( ˆ M a ) is the j -th smallest eigenvalues of ˆ M a .
Then problem (12) is equivalent to
[Equation]
(cid:40)(cid:88)
[Equation]
where the parameter c a,a (cid:48) ,b admits values c a,a (cid:48) ,b =   ˆ p a + λ if b = a, | ˆ p a (cid:48) − λ | if b = a (cid:48) , ˆ p b otherwise.
Proof of Theorem B.2.
For simplicity, we let E ( V, Q , b ) =
E Q
[ (cid:96) ( V, X ) | A = b ] .
Then, the objective function of problem (12) can be re-written as
[Equation]
Q , b ) + λ max a (cid:54) = a (cid:48) { E ( V, Q , a ) − E ( V, Q , a (cid:48) ) } = max a (cid:54) = a (cid:48)   (cid:88) b (cid:54) =
a,a (cid:48) sup W ( Q b , ˆ P b ) ≤ ε b ˆ p b E ( V, Q b , b ) + sup W ( Q a , ˆ P a ) ≤ ε a (ˆ p a + λ ) E ( V, Q a , a ) + sup W ( Q a (cid:48) , ˆ P a (cid:48) ) ≤ ε a (cid:48)
(ˆ p a (cid:48) − λ )
E ( V, Q a (cid:48) , a (cid:48) )  
= max a (cid:54) = a (cid:48) (cid:40)
(cid:88) b (cid:54) =
a,a (cid:48)
{+ˆ p b (cid:18)(cid:113) (cid:104) I d − V V (cid:62) , ˆ M b (cid:105) + √ ε b (cid:19) 
+}
+ (ˆ p a + λ ) (cid:18)(cid:113) (cid:104) I
d − V V (cid:62) , ˆ M a (cid:105) + √ ε a (cid:19)+ (ˆ p a (cid:48) − λ ) (cid:18)(cid:113) (cid:104)
I d − V V (cid:62) ,
ˆ M a (cid:48) (cid:105) + sgn
(ˆ p a (cid:48) − λ ) √ ε a (cid:48) (cid:19) 2 (cid:41) = max a (cid:54) = a (cid:48) (cid:40) (cid:88)
b ∈A ˆ p b (cid:16) (cid:104) I
d − V V (cid:62) , ˆ M b (cid:105) + ε b (cid:17) + (cid:88) b
∈A 2 c a,a (cid:48) ,b (cid:113) ε b (cid:104) I
d − V V (cid:62) , ˆ M b (cid:105) + λ (cid:16) (cid:104) I d − V V (cid:62) , ˆ M a − ˆ M a (cid:48) (cid:105) + ε a − ε a (cid:48) (cid:17) (cid:41) , where the ﬁrst equality follows from the deﬁnition of U max ( V, Q ) and E ( V, Q , b ) , the second from the deﬁnition (5) of the ambiguity set B (ˆ P ) , the third from Proposition 3.3 and the fourth from the deﬁnition of c a,a (cid:48) ,b .
Noting that the ﬁrst sum in the above maximization is independent of a and a (cid:48) , the proof is completed.
Theorem 12 indicates that if the sensitive attribute admits ﬁnite values, then the distributionally robust fairness-aware PCA problem using an U max unfairness measure can be reformulated as an optimization problem over the Stiefel manifold, where the objective function is a pointwise maximization of ﬁnite number of individual functions.
It is also easy to see that each individual function can be reparametrized using U , and the Riemannian gradient descent algorithm in Section 4 can be adapted to solve for the optimal solution.
The details on the algorithm are omitted.
C I NFORMATION ON D ATASETS
We summarize here the number of observations, dimensions [Table] and the sensitive attribute of the data sets.
For further information about the data sets and pre-processing steps, please refer to Samadi et al.
(2018) for Default Credit and Labeled Faces in the Wild (LFW) data sets [Table] and Olfat & Aswani (2019) for others.
For each data set, we further remove columns with too small standard deviation ( ≤ 1 e − 5 ) as they do not signiﬁcantly affect the results, and ones with too large standard deviation ( ≥ 1000 ) which we consider as unreliable features.
[Table]
D A DDITIONAL R ESULTS D. D ETAIL P ERFORMANCES
Table 3 shows the performances of four examined methods with two criteria ABDiff.
and ARE.
It is clear that our method achieves the best results over all 14 datasets w.r.t.
ABDiff.
, and 7 datasets on ARE., which is equal to the number of datasets FairPCA out-perform others.
Table 4 complements Table 1 from the main text
[Table]
from which we can see that two versions of CFPCA out-perform others over all datasets w.r [Table]
t. (cid:52)F Lin , which is the criteria they optimize for.
[Table]
Adjustment for the LFW dataset.
To demonstrate the efﬁcacy of our method on high-dimensional data sets [Table] we also do experiments on a subset of 2000 faces for each of male and female group ( [Table]
in total) from LFW dataset
[Table]
7 all images are rescaled to resolution 24 × 24 (dimensions d = 576 ).
The experiment follows the same procedure in Section [Table] with reducing the number of iterations to 500 for both RFPCA and FairPCA and 2 -fold cross validation, the results are averaged over 10 train-test simulations.
Due to the high dimension of the input, the implementation of Olfat & Aswani (2019) fails to return any result.
D. V ISUALIZATION D.2. E FFECTS OF THE A MBIGUITY S ET R ADIUS
We examine the change of the model’s performance with respect to the change of the radius of the ambiguity sets.
To generate the toy data (also used for Figure 1), we use two 2 -dimensional Gaussian distributions to represent two groups of a sensitive attribute,
A = 0 and A = 1 , or groups 0 andfor simplicity.
The two distributions both have the mean at (0 , 0) and covariance matrices for group 0 and 1 are (cid:18) 4 .  0 .(cid:19) and (cid:18) 0 .
4 3 .(cid:19) , respectively.
For the test set, the number of samples is 8000 for group 0 and 4000 for group 1 , while for the training set, we have 200 for group 0 and 100 for group 1 .
We average the results oversimulations, for each simulation, the test data is ﬁxed, the training data is randomly generated with the number of samples mentioned above.
The projections are learned on training data and measured on test data by the summation of ARE.
and ABDiff.
We ﬁxed λ = 0 .
1 , which is not too small for achieving fair projections, and not too large to clearly observe the effects of ε , and we also ﬁxed ε 0 for better visualization.
Note that we still compute ε 1 = α/ √ N 1 in which, α is tested withvalues evenly spaced in [0 , 10] .
The experiment results are visualized in Figure 4.
The result suggests that increasing the ambiguity set radius can improve the overall model’s performance.
This justiﬁes the beneﬁt of adding distributional robustness to the fairness-aware PCA model.
After a saturation point, a too large radius can lessen the role of empirical data, and the model prioritizes a more extreme distribution that is far from the target distribution, which causes the reduction in the model’s performance on target data.
[Figure]
Figures 5 and 6 plot the Pareto frontier for two datasets (Biodeg and German Credit) with 3 principal components. One can observe that RFPCA produces points that dominate other methods based on the trade-off between ARE. and ABDiff. D.2. P ERFORMANCE WITH DIFFERENT PRINCIPAL COMPONENTS
We collect here the reconstruction errors for different numbers of principal components.