S TOCHASTIC C ANONICAL C ORRELATION A NALYSIS : A R IEMANNIAN A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
We present an efﬁcient stochastic algorithm (RSG+) for canonical correlation analysis (CCA) derived via a differential geometric perspective of the underlying optimization task.
We show that exploiting the Riemannian structure of the problem reveals natural strategies for modiﬁed forms of manifold stochastic gradient descent schemes that have been variously used in the literature for numerical optimization on manifolds.
Our developments complement existing methods for this problem which either require O ( d 3 ) time complexity per iteration with O ( 1 √ t ) convergence rate (where d is the dimensionality) or only extract the top 1 component with O ( 1 t ) convergence rate.
In contrast, our algorithm achieves O ( d 2 k ) runtime complexity per iteration for extracting top k canonical components with O ( 1 t ) convergence rate.
We present our theoretical analysis as well as experiments describing the empirical behavior of our algorithm, including a potential application of this idea for training fair models where the label of protected attribute is missing or otherwise unavailable.
I NTRODUCTION
Canonical correlation analysis (CCA) is a popular method for evaluating correlations between two sets of variables.
It is commonly used in unsupervised multi-view learning, where the multiple views of the data may correspond to image, text, audio and so on Rupnik & Shawe-Taylor (2010);
Chaudhuri et al.
; Luo et al. (2015).
Classical CCA formulations have also been extended to leverage advances in representation learning, for example, Andrew et al.
(2013) showed how the
CCA can be interfaced with deep neural networks enabling modern use cases.
Many results over the last few years have used CCA or its variants for problems including measuring representational similarity in deep neural networks Morcos et al.
(2018), speech recognition Couture et al.
(2019), etc.
The goal in CCA is to ﬁnd linear combinations within two random variables X and Y which have maximum correlation with each other.
Formally, the CCA problem is deﬁned in the following way.
Given a pair of random variables, a d x -variate random variable X and a d y -variate random variable Y , with unknown joint probability distribution, ﬁnd the projection matrices U ∈ R d x × k and
V ∈ R d y × k , with k ≤ min { d x , d y } , such that the correlation is maximized:
[Equation]
Here, X, Y are samples of X and Y respectively.
The objective function in (1) is the expected crosscorrelation in the projected space and the constraints specify that different canonical components should be decorrelated.
Let C X =
E X
[ X T X ] and C Y =
E Y
[ Y T Y ] be the covariance matrices, and C XY =
E ( X , Y )
[ X T Y ] denote cross-covariance.
Let us deﬁne the whitened covariance T :=
C − 1 / 2 X
C XY C − 1 / 2 Y and Φ k (and Ψ k ) contains the top- k left (and right) singular vectors of T .
It is known Golub & Zha (1992) that the optimum of (1) is achieved at U ∗ =
C − 1 / 2 X Φ k ,
[Equation]
∗ =
[Equation]
Ψ
[Equation]
In practice, we may be given two views of N samples as X ∈ R N × d x and Y ∈ R N × d y .
A natural approach to solving CCA is based on on the following sequence of steps.
We ﬁrst compute the empirical covariance and cross-covariance matrices, namely, (cid:101) C X = 1 / N X T X , (cid:101) C Y = 1 / N Y T Y and (cid:101)
C XY = 1 / N X T Y .
We then calculate the empirical whitened cross-covariance matrix (cid:101) T , ﬁnally, compute U ∗ ,
Runtime and memory considerations.
The above procedure is simple but is only feasible when the data matrices are small.
But in most modern applications, not only are the datasets large but also the dimension d (let d = max { d x , d y } ) of each sample can be quite high, especially if representations are being learned using deep neural network models.
As a result, the computational footprint of the foregoing algorithm can be quite high.
This has motivated the study of stochastic optimization routines for solving CCA.
Observe that in contrast to the typical settings where stochastic optimization schemes are most effective, the CCA objective does not decompose over samples in the dataset.
Many efﬁcient strategies have been proposed in the literature: for example, Ge et al.
; Wang et al.
(2016) present Empirical Risk Minimization (ERM) models which optimize the empirical objective.
More recently, Gao et al.
; Bhatia et al.
Arora et al.
(2017) describe proposals that optimize the population objective.
To summarize the approaches succinctly, if we are satisﬁed with identifying the top 1 component of CCA, effective schemes are available by utilizing either extensions of the Oja’s rule Oja (1982) to generalized eigenvalue problem Bhatia et al.
(2018) or the alternating SVRG algorithm Gao et al.
Otherwise, a stochastic approach must make use of an explicit whitening operation which involves a cost of d 3 for each iteration Arora et al.
Observation.
Most approaches either directly optimize (1) or instead a reparametrized or regularized form Ge et al.
(2016); Allen-Zhu & Li (2016);
Arora et al.
Often, the search space for U and V corresponds to the entire R d × k (ignoring the constraints for the moment).
But if the formulation could be cast in a form which involved approximately writing U and V as a product of several matrices with nicer properties, we may obtain specialized routines which are tailored to exploit those properties.
Such a reformulation is not difﬁcult to derive – where the matrices used to express U and V can be identiﬁed as objects that live in well studied geometric spaces.
Then, utilizing the geometry of the space and borrowing relevant tools from differential geometry leads to an efﬁcient approximate algorithm for top-
k CCA which optimizes the population objective in a streaming fashion.
Contributions.
(a) First, we re-parameterize the top- k CCA problem as an optimization problem on speciﬁc matrix manifolds, and show that it is equivalent to the original formulation in equation 1.
(b) Informed by the geometry of the manifold, we derive stochastic gradient descent algorithms for solving the re-parameterized problem with O ( d 2 k ) cost per iteration and provide convergence rate guarantees.
(c)
This analysis provides a direct mechanism to obtain an upper bound on the number of iterations needed to guarantee an (cid:15) error w.r.t.
the population objective for the CCA problem.
(d) The algorithm works in a streaming manner so it easily scales to large datasets and we do not need to assume access to the full dataset at the outset.
(e) We present empirical evidence for both the standard CCA model and the DeepCCA setting Andrew et al.
(2013), describing advantages and limitations.
S TOCHASTIC CCA: R EFORMULATION , A LGORITHM AND A NALYSIS
The formulation of Stochastic CCA and the subsequent optimization scheme will seek to utilize the geometry of the feasible set for computational gains. Speciﬁcally, we will use the following manifolds (please see Absil et al. (2007) for more details): (a) Stiefel: St ( p, n ) .
The manifold consists of n × p , with p < n , column orthonormal matrices, i
e., St ( p, n ) = (cid:8) X ∈ R n × p | X T X = I p (cid:9) . (b) Grassmanian: Gr ( p, n ) . The manifold consists of p -dimensional subspaces in R n , with p < n .
(c) Rotations: SO ( n ) . the manifold/group consists of n × n special orthogonal matrices, i.e., SO ( n ) =
(cid:8) [Equation]
We summarize certain geometric properties/operations for these manifolds in the Appendix but have been leveraged in recent works for other problems also Li et al. Rezende et al. Let us recall the objective function for CCA as given in (1).
We denote X ∈ R N × d x as the matrix consisting of the samples { x i } drawn from a zero mean random variable X ∼ X and Y ∈ R N × d y denotes the matrix consisting of samples { y i } drawn from a zero mean random variable Y ∼ Y .
For notational and formulation simplicity, we assume that d x = d y =
d in the remainder of the paper although the results hold for general d x and d y .
Let C X , C Y be the covariance matrices of X , Y .
Also, let C XY be the cross-correlation matrix between X and Y .
Then, we can write the CCA objective as
[Equation]
subject to U T C X U = I
[Equation]
C [Equation] k
[Equation] [Equation] Here, U ∈ R d × k ( V ∈ R d × k ) is the matrix consisting of { u j } ( { v j } ) , where ( { u j } , { v j } ) are the canonical directions.
The constraints in equation 2 are called whitening constraints .
Let us deﬁne matrices (cid:101) U, (cid:101) V ∈ R d × k which lie on the Stiefel manifold, St ( k, d ) .
Also, let S u , S v ∈
R k × k denote upper triangular matrices and Q u , Q v ∈ SO ( k ) .
We can rewrite the above equation and the constraint as follows.
A Reformulation for CCA
[Equation]
subject to [Equation] St [Equation]
[Equation]
Q [Equation] Q [Equation]
∈ [Equation] k ); S u , S v is upper triangular Here, we will maximize (3a) with respect to (cid:101) U , (cid:101) V , S u , S v , Q u , and Q v satisfying equation 3b.
Main adjustment from (2) to (3): In (2), while U and V should decorrelate C X and C Y respectively, the optimization/search is unrestricted and treats them as arbitrary matrices. In contrast, equationadditionally decomposes U and V as U = (cid:101) UQ u S u and V = (cid:101) V Q v S v with the components as structured matrices. Hence, the optimization is regularized .
The above adjustment raises two questions: (i) does there exist a non-empty feasible solution set for (3)? (ii) if a solution to (3) can be found (which we will describe later), how “good” is this solution for the CCA objective problem, i.e., for (2)? Existence of a feasible solution: We need to evaluate if the constraints in (3b) can be satisﬁed at all. Observe that by using (cid:101) U to be the top- k principal directions of X , S u to be the 1 / √ top-k eigen values of C X and Q u to be any orthogonal matrix, we can easily satisfy the “whitening constraint” and hence (cid:101) UQ u S u is a feasible solution of U in (3) and similarly for V .
From this starting point, we can optimize the objective while ensuring that we maintain feasibility. Is the solution for equation 3 a good approximation for equation 2? : We can show that under some assumptions, the estimator for canonical correlation, i.e., solution of equation 3, is consistent, i.e., solves equation 2. We will state this formally shortly.
Before characterizing the properties of a solution for equation 3, we ﬁrst provide some additional intuition behind equation 3 and describe how it helps computationally.
Intuition behind the decomposition U = ˜
UQ u S u : A key observation is the following. Recall that by performing principal component analysis (PCA), the resultant projection matrix will exactly satisfy the decorrelation condition needed for the “whitening constraint” in equation 2 (projection matrix consists of the eigen-vectors of X T X ). A natural question to ask is: Can we utilize streaming PCA algorithm to help us obtain an efﬁcient streaming CCA algorithm? Let us assume that our estimate for canonical correlation directions, i.e., solutions of equation 3, lies in the principal subspace calculated above.
If so, we can use the decomposition U = ˜ UA u (analogously for V ), where ˜ U contains the principal directions, i.e., ∈ St ( k, d ) and A u is a full rank k × k matrix containing the coefﬁcients of the span. But maintaining the full rank constraint during optimization is hard, so we further decompose A u into A u = Q u S u with Q u ∈ SO ( k ) ; S u is upper triangular.
Additionally, we ensure the diagonal of S u to be non-zero to maintain full-rank of S u . During optimization, we can maintain the non-zero diagonal entries by optimizing the log of the diagonal entries instead.
Why equation 3 helps? First, we note that CCA seeks to maximize the total correlation under the constraint that different components are decorrelated. The difﬁcult part in the optimization is to ensure decorrelation, which leads to a higher complexity in existing streaming CCA algorithms.
On the contrary, in equation 3, we separate equation 2 into ﬁnding the PCs and ﬁnding the linear coefﬁcients for the span of principal directions. Then, by utilizing an efﬁcient streaming PCA algorithm, a lower complexity can be achieved.
We will defer describing the speciﬁc details of the optimization itself until the next sub-section.
First, we will show formally why substituting equation 2 with equation 3a–equation 3b is sensible under some assumptions.
2. H OW TO USE THE REFORMULATION IN EQUATION 3?
We ﬁrst start by stating some mild assumptions needed for the analysis.
Assumptions: (a) The random variables X ∼ N ( 0 , Σ x ) and Y ∼ N ( 0 , Σ y ) with Σ x (cid:22) cI d and Σ y (cid:22) cI d for some c > 0 .
(b) The samples X and Y drawn from X and Y respectively have zero mean.
(c) For a given k ≤ d , Σ x and Σ y have non-zero top-
k eigen values.
A high-level solution to optimize (cid:101) F in equation 3: Recall the following scheme which we brieﬂy summarized earlier.
(a) Initialize (cid:101) U, (cid:101) V ∈ St ( k, d ) as the top- k eigen vectors of C X =
( 1 / N ) X T X and C Y =
( 1 / N ) Y T Y respectively; Initialize Q u and Q v to be random SO ( k ) matrices; (b) Set S u and S v to be diagonal matrices with the diagonal entries to be the square root of the inverse of the top- k eigen values (to satisfy upper-triangular property);
Observe that with this initialization, the constraints in equation 3b are satisﬁed.
With a feasible solution for U and V in hand, we may optimize equation 3a while satisfying equation 3b.
The speciﬁc details of how this is done is not critical at this point as long as we assume that a suitable numerical optimization scheme exists and can be implemented.
With the component matrices, we can construct the solution as U =
(cid:101) UQ u S u and V = (cid:101) V Q v S v .
Why the solution makes sense?
We now show how the presented solution, assuming access to an effective numerical procedure, approximates the CCA problem presented in equation 2.
We formally state the result in the following theorem with a sketch of proof (appendix includes the full proof) by ﬁrst stating the following proposition and a deﬁnition.
Deﬁnition 1.
A random variable X is called sub-Gaussian if the norm given by (cid:107) X (cid:107) (cid:63) :=
inf { d ≥ 0 | E X [exp ( trace ( X T X ) / d 2 )] ≤ 2 } is ﬁnite.
Let U ∈ R
d[-(cid:21) 
 j E X-]
× k , then X U is sub-Gaussian Vershynin (2017).
Proposition 1 (Reiß et al.
Let X be a random variable which follows a sub-Gaussian distribution.
Let (cid:98) X be the approximation of X ∈ R N × d (samples drawn from X ) with the top- k principal vectors.
Let (cid:101) C X be the covariance of (cid:98) X .
Also, assume that λ i is the i th eigen value of
C X for i = 1 , · · · , d − 1 and λ i ≥ λ i +1 for all i .
Then, the PCA reconstruction error, denoted by
E k = (cid:107) X − (cid:98) X (cid:107) (in the Frobenius norm sense) can be upper bounded as follows
[Equation]
where ∆ = C X − (cid:101) C X .
The aforementioned proposition suggests that the error between the data matrix X and the reconstructed data matrix (cid:98) X using the top- k principal vectors is bounded.
Recall from (2) and (3) that the value of the CCA objective is denoted by F and (cid:101) F .
The following theorem states that we can bound the error, E = (cid:107) F − (cid:101) F (cid:107) (proof is in the Appendix).
The proof includes upper-bounding E by the reconstruction error of the data projected on the principal directions using Prop. 1.
Theorem 1.
Using the hypothesis and assumptions above, the approximation error
E = (cid:107) F − (cid:101) F (cid:107) is bounded and goes to zero while the whitening constraints in equation 3b are satisﬁed.
Now, the only unresolved issue is an optimization scheme for equation 3a that keeps the constraints in equation 3b satisﬁed by leveraging the geometry of the feasible set
. 2. H OW TO NUMERICALLY OPTIMIZE (3 A ) SATISFYING CONSTRAINTS IN (3 B )?
Overview.
We now describe how to maximize the formulation in equation 3a–equation 3b with respect to (cid:101) U , (cid:101) V , Q u , Q v , S u and S v .
We will ﬁrst compute top-
k principal vectors to get (cid:101) U and (cid:101) V .
Then, we will use a gradient update rule to solve for Q u , Q v , S u and S v to improve the objective.
Since all these matrices are “structured”, care must be taken to ensure that the matrices remain on their respective manifolds – which is where the geometry of the manifolds will offer desirable properties.
We re-purpose a Riemannian stochastic gradient descent (RSGD) to do this, so call our algorithm RSG+ .
Of course, more sophisticated Riemannian optimization techniques can be substituted in.
For instance, different Riemannian optimization methods are available in Absil et al.
(2007) and optimization schemes for many manifolds are offered in PyManOpt Boumal et al.
The algorithm block is presented in Algorithm 1 (a direct implementable block for the algorithm including the expression for gradients is presented in the Appendix A.3).
Let (cid:101) F pri = trace (cid:0) U T C X U ) (cid:1) + trace (cid:0) V T C Y V ) (cid:1) be the contribution from the principal directions which we used to ensure the “whitening constraint”.
Let (cid:101) F can = trace (cid:0) U T C XY V (cid:1) be the contribution from the canonical correlation directions. The algorithm consists of four main blocks denoted by different colors, namely (a) the Red block deals with gradient calculation of the objective function where we calculate the top- k principal vectors (denoted by (cid:101) F pri ) with respect to (cid:101) U , (cid:101) V ; (b) the Green block describes calculation of the gradient corresponding to the canonical directions (denoted by (cid:101) F can ) with respect to (cid:101) U , (cid:101) V , S u , S v , Q u and Q v ; (c) the Gray block combines the gradient computation from both (cid:101) F pri and (cid:101) F can with respect to unknowns (cid:101) U , (cid:101) V , S u , S v , Q u and Q v ; and ﬁnally (d) the Blue block performs a batch update of the canonical directions (cid:101) F can using Riemannian gradient updates.
Gradient calculations.
The gradient update for (cid:101) U, (cid:101) V is divided into two parts (a) The (Red block) gradient updates the “principal” directions (denoted by ∇ (cid:101) U (cid:101) F pri and ∇ (cid:101) V (cid:101) F pri ), which is speciﬁcally designed to satisfy the whitening constraint .
Since this requires updating the principal subspaces, so, the gradient descent needs to proceed on the manifold of subspaces, i.e., on the Grassmannian. (b) The (green block) gradient from the objective function in equation 3, is denoted by ∇ (cid:101) U (cid:101) F can and ∇ (cid:101) V (cid:101) F can . In order to ensure that the Riemannian gradient update for (cid:101) U and (cid:101) V stays on the manifold St ( k, d ) , we need to make sure that the gradients, i.e., ∇ (cid:101) U (cid:101) F can and ∇ (cid:101) V (cid:101) F can lies in the tangent space of St ( k, d ) .
In order to do that, we need to ﬁrst calculate the Euclidean gradient and then project on to the tangent space of St ( k, d ) .
The gradient updates for Q u , Q v , S u , S v are given in the green block, denoted by ∇ Q u (cid:101) F can , ∇ Q v (cid:101) F can , ∇ S u (cid:101) F can and ∇ S v (cid:101) F can .
Note that unlike the previous step, this gradient only has components from canonical correlation computation.
As before, this step requires ﬁrst computing the Euclidean gradient and then projecting on to the tangent space of the underlying Riemannian manifolds involved, i.e.,
SO ( k ) and the space of upper triangular matrices. Finally, we get the gradient to update the canonical directions by combining the gradients which is shown in gray block.
With these gradients we can perform a batch update as shown in the blue block.
Using convergence results presented next in Propositions 2–3, this scheme can be shown (under some assumptions) to approximately optimize the CCA objective in equation 2.
We can now move to the convergence properties of the algorithm.
We present two results stating the asymptotic proof of convergence for top- k principal vectors and canonical directions in the algorithm.
Proposition 2 (Chakraborty et al.
(Asymptotically)
If the samples, X , are drawn from a Gaussian distribution, then the gradient update rule presented in Step 5 in Algorithm 1 returns an orthonormal basis – the top- k principal vectors of the covariance matrix C X .
Proposition 3.
(Bonnabel (2013)) Consider a connected Riemannian manifold M with injectivity radius bounded from below by I > 0 .
Assume that the sequence of step sizes ( γ l ) satisfy the condition (a) (cid:80) γ 2 l < ∞ (b) (cid:80) γ l = ∞ .
Suppose { A l } lie in a compact set K ⊂ M .
We also suppose that ∃ D > 0 such that, g A l (cid:16) ∇ A l (cid:101) F, ∇ A l (cid:101) F (cid:17) ≤ D .
Then ∇ A l (cid:101) F → 0 and l → ∞ .
Notice that in our problem, the manifold M can be Gr ( p, n ) , St ( p, n ) or SO ( p ) .
Hence all the assumptions in Proposition 3 are satisﬁed if we guarantee the step sizes satisfy the aforementioned condition.
One example of the step sizes that satisﬁes the property is γ l = 1 l +1 .
2. C ONVERGENCE RATE AND COMPLEXITY OF THE RSG+ ALGORITHM
In this section, we describe the convergence rate and complexity of the algorithm proposed in Algorithm 1.
Observe that the key component of Algorithm 1 is a Riemannian gradient update.
Let A using the update
A t +1 =
Exp A t (cid:16) − γ t ∇ A t (cid:101) F (cid:17) , where γ t is the step size at time step t .
Also assume { A t } ⊂ M for a Riemannian manifold M .
The following proposition states that under certain assumptions, the
Riemannian gradient update has a convergence rate of O (cid:0) 1 t (cid:1) .
Proposition 4.
(Nemirovski et al.
; Bécigneul & Ganea (2018)) Let { A t } lie inside a geodesic ball of radius less than the minimum of the injectivity radius and the strong convexity radius of M .
Assume M to be a geodesically complete Riemannian manifold with sectional curvature lower bounded by κ ≤ 0 .
Moreover, assume that the step size { γ t } diverges and the squared step size converges.
Then, the Riemannian gradient descent update given by A t +1 =
Exp A t (cid:16) − γ t ∇ A t (cid:101) F (cid:17) with a bounded ∇ A t (cid:101) F , i.e., (cid:107)∇ A t (cid:101) F (cid:107) ≤ C < ∞ for some C ≥ 0 , converges in the rate of O (cid:0) 1 t (cid:1) .
All Riemannian manifolds we used, i.e., Gr ( k, d ) , St ( k, d ) and SO ( k ) are geodesically complete, and these manifolds have non-negative sectional curvatures, i.e., lower bounded by κ = 0 .
Now, as long as the Riemannian updates lie inside the geodesic ball of radius less than the minimum of injectivity and convexity radius, the convergence rate for RGD applies in our setting.
Running time.
To evaluate time complexity, we must look at the main compute-heavy steps needed.
The basic modules are Exp and Exp − 1 maps for St ( k, d ) , Gr ( k, d ) and SO ( k ) manifolds (see Table 4 in the appendix).
Observe that the complexity of these modules is inﬂuenced by the complexity of svd needed for the Exp map for the St and Gr manifolds.
Our algorithm involves structured matrices of size d × k and k × k , so any matrix operation should not exceed a cost of O (max( d 2 k, k 3 )) , since in general d (cid:29) k . Speciﬁcally, the most expensive calculation is SVD of matrices of size d × k , which is O ( d 2 k ) , see Golub & Reinsch (1971). All other calculations are dominated by this term.
E XPERIMENTS
We ﬁrst evaluate RSG+ for extracting top- k canonical components on three benchmark datasets and show that it performs favorably compared with Arora et al.
Then, we show that RSG+ can also ﬁts into feature learning in DeepCCA Andrew et al.
and can scale to large feature dimensions where the non-stochastic method fails to.
Finally we show that RSG+ can be used to improve fairness of deep neural networks without needing labels of protected attributes during training. [Table]
); 3. Orthogonalize each block and let the orthogonalized block be denoted by ˆ Z xl ( ˆ Z yl ); 4.
Let the subspace spanned by each ˆ Z xl (and ˆ Z yl ) be ˆ Z xl ∈ Gr ( k,d x ) (and ˆ Z
yl ∈
Gr ( k,d y ) ); 5. Update ∇ (cid:101) U (cid:101) F pri and ∇ (cid:101) V (cid:101) F pri based on ˆ Z xl and ˆ Z yl respectively. [Table] from
[Table]
(cid:101)
[Table]
(cid:101)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
3. CCA ON F IXED D ATASETS
Datasets and baseline.
We conduct experiments on three benchmark datasets (MNIST LeCun et al.
(2010), Mediamill Snoek et al.
and CIFAR-10 Krizhevsky (2009)) to evaluate the performance of RSG+ to extract top- k canonical components.
To our best knowledge, Arora et al.
(2017) is the only previous work which stochastically optimizes the population objective in a streaming fashion and can extract top- k components, so we compare our RSG+ with the matrix stochastic gradient (MSG) method proposed in Arora et al.
(There are two methods proposed in Arora et al.
and we choose MSG because it performs better in the experiments of Arora et al.
The details about the three datasets and how we process them are as follows:
MNIST LeCun et al.
: MNIST contains grey-scale images of size × 28 .
We use its full training set containing 60 K images.
Every image is split into left/right half, which are used as the two views.
Mediamill Snoek et al.
(2006): Mediamill contains around 25 . 8 K paired features of videos and corresponding commentary of dimension 120 , 101 respectively.
CIFAR-10 Krizhevsky (2009): CIFAR-10 contains 60 K 32 × 32 color images.
Like MNIST, we split the images into left/right half and use them as two views.
Evaluation metric.
We choose to use Proportion of Correlations Captured (PCC) which is widely used Ma et al.
; Ge et al.
(2016), partly due to its efﬁciency, especially for relatively large datasets.
Let ˆ U ∈ R d x × k , ˆ V ∈ R d y × k denote the estimated subspaces returned by RSG+, and
U ∗ ∈ R d x × k , V ∗ ∈ R d y × k denote the true canonical subspaces (all for top- k ).
The PCC is deﬁned as PCC =
[Equation] [Equation]
( XU ∗ ,Y V ∗ ) , where TCC is the sum of canonical correlations between two matrices.
Performance.
See A.4 for the implementation deails.
The performance in terms of PCC as a function of # of seen samples (coming in a streaming way) are shown in Fig.
1, and the runtime is reported in A.5 .
Our RSG+ captures more correlation than MSG Arora et al.
(2017) while being 5 − 10 times faster.
One case where our RSG+ underperform Arora et al.
(2017) is when the top- k eigenvalues are dominated by the top- l eigenvalues with l < k (Fig.
1b): on Mediamill dataset, the top-4 eigenvalues of the covariance matrix in view 1 are: 8 . 61 , 2 . 99 , 1 . 15 , 0 . 37 .
The ﬁrst eigenvalue is dominantly large compared with the rest and our RSG+ performs better for k = 1 and worse than Arora et al.
(2017) for k = 2 , 4 .
We also plot the runtime of RSG+ under different data dimension (set d x = d y = d ) and number of total samples sampled from joint gaussian distribution in A.5. We implemented the method from Yger et al. and conduct experiments on the three datasets above. The results are shown in Table 1.
We tune the step size between [0 . 0001 , 0 . 1] and β = 0 .as used in their paper.
On MNIST and MEDIAMILL, the method performs comparably with ours except k = 4 case on MNIST where it does not converge well.
Since this algorithms also has a d 3 complexity, the runtime is 100 × more than ours on MNIST and 20 × more on Mediamill. On CIFAR10, we fail to ﬁnd a suitable step size for convergence. 3. CCA FOR D EEP F EATURE L EARNING
Background and motivation.
A deep neural network (DNN) extension of CCA was proposed by
Andrew et al.
and has become popular in the multi-view representation learning tasks.
The idea is to learn a deep neural network as the mapping from original data space to a latent space where the canonical correlations are maximized.
We refer the reader to Andrew et al.
for details of the task.
Since deep neural networks are usually trained using SGD on mini-batches, this requires [Figure] getting estimate of CCA objective at every iteration in a streaming fashion, thus our RSG+ can be a natural choice here.
We conduct experiments on a noisy version of MNIST dataset to evaluate RSG+
[Table]
Dataset.
We follow Wang et al.
(2015a) to construct a noisy version of MNIST:
View 1 is a randomly sampled image which is ﬁrst rescaled to [0 , 1] and then rotated by a random angle from [ − π 4 , π 4 ] .
View 2 is randomly sampled from the same class as view 1 .
Then we add independent uniform noise from [0 , 1] to each pixel.
Finally the image is truncated into [0 , 1] to form the view [Table]
[Table]
[Table]
We use a simple 2 -layer MLP with ReLU nonlinearity, where the hidden dimension in the middle isand the output feature dimension is d ∈ { 100 , 500 , 1000 } .
After the network is trained on the CCA objective, we use a linear Support
Vector Machine (SVM) to measure classiﬁcation accuracy on output latent features.
Andrew et al.
(2013) uses the closed form CCA objective on the current batch directly, which costs O ( d 3 ) memory and time for every iteration.
Performance.
Table 2 shows that we get similar performance when d = 100 and can scale to large latent dimensions d = 1000 while the batch method Andrew et al.
(2013) encounters numerical difﬁculty on our GPU resources and Pytorch Paszke et al.
(2019) platform in performing eigendecomposition of d × d matrix when d = 500 , and becomes difﬁcult if d is larger than 1000 .
3. CCA FOR F AIRNESS
[Table]
Background and motivation.
Fairness is becoming an important issue to consider in the design of learning algorithms.
A common strategy to make an algorithm fair is to remove the inﬂuence of one/more protected attributes when training the models, see Lokhande et al.
Most methods assume that the labels of protected attributes are known during training but this may not always be possible.
CCA enables considering a slightly different setting, where we may not have per-sample protected attributes which may be sensitive or hard to obtain for third-parties Price & Cohen (2019).
On the other hand, we assume that a model trained to predict the protected attribute labels has been trained and is provided.
For example, if the protected attribute is gender, we only assume that a well trained classiﬁer which is trained to predict gender from the samples is available rather than sample-wise gender values themselves.
We next demonstrate that fairness of the model, using standard measures, can be improved via constraints on correlation values from CCA.
Dataset.
CelebA
Wang et al.
(2015b) consists of 200 K celebrity face images from the internet.
There are up to 40 labels, each of which is binary-valued.
Here, we follow Lokhande et al.
(2020) to focus on the attactiveness attribute (which we want to train a classiﬁer to predict) and the gender is treated as “protected” since it may lead to an unfair classiﬁer according to Lokhande et al.
Method.
Our strategy is inspired by Morcos et al.
(2018) which showed that canonical correlations can reveal the similarity in neural networks: when two networks (same architecture) are trained using different labels/schemes for example, canonical correlations can indicate how similar their features are.
Our observation is the following.
Consider a classiﬁer that is trained on gender (the protected attribute), and another classiﬁer that is trained on attractiveness , if the features extracted by the latter model share a high similarity with the one trained to predict gender, then it is more likely that the latter model is inﬂuenced by features in the image pertinent to gender, which will lead to an unfairly biased trained model.
We show that by imposing a loss on the canonical correlation between the network being trained (but we lack per-sample protected attribute information) and a well trained classiﬁer pre-trained on the protected attributes, we can get a more fair model.
This may enable training fairer models in settings which would otherwise be difﬁcult.
Implementation details.
To simulate the case where we only have a pretrained network on protected attributes, we train a Resnet- 18 He et al.
(2016) on gender attribute,
and when we train the classiﬁer to predict attractiveness , we add a loss using the canonical correlations between these two networks on intermediate layers: L total =
L cross-entropy + L CCA where the ﬁrst term is the standard cross entropy term and the second term is the canonical correlation.
See A.7 for more details of training/evaluation.
Results.
We choose two commonly used error metrics for fairness: difference in Equality of Opportunity Hardt et al.
(2016) (DEO), and difference in Demographic Parity Yao & Huang (2017) (DDP).
See appendix A.6 for more detailed explaination of the two metrics.
We conduct experiments by applying the canonical correlation loss on three different layers in Resnet-18.
In Table 3, we can see that applying canonical correlation loss generally improves the DEO and DDP metrics (lower is better) over the standard model (trained using cross entropy loss only).
Speciﬁcally, applying the loss on early layers like conv 0 and conv 1 gets better performance than applying at a relatively late layer like conv 2 .
Another promising aspect of our approach is that is can easily handle the case where the protected attribute is a continuous variable (as long as a well trained regression network on the protected attribute is given) while other methods like Lokhande et al.
; Zhang et al.
(2018) need to ﬁrst discretize the variable and then enforce constraints which can be much more involved.
R ELATED W ORK
Stochastic CCA: There has been much interest in designing scalable and provable algorithms for
CCA: Ma et al.
(2015) proposed the ﬁrst stochastic algorithm for CCA, while only local convergence is proven for non-stochastic version.
Wang et al.
(2016) designed algorithm which uses alternating
SVRG combined with shift-and-invert pre-conditioning, with global convergence.
These stochastic methods, together with Ge et al.
Allen-Zhu & Li (2016), which reduce CCA problem to generalized eigenvalue problem and solve it by performing efﬁcient power method, all belongs to the methods that try to solve empirical CCA problem, it can be seen as ERM approxiamtion of the priginal population objective, which requires solving numerical optimization of the empirical CCA objective on a ﬁxed data set.
These methods usually assume the access to the full dataset in the beginning, which is not very suitable for many practical applications where data tend to come in a streaming way.
Recently, there are increasingly interest in considering population CCA problem
Arora et al.
Gao et al.
The main difﬁculty in population setting is we have limited knowledge about the objective unless we know the distribution of X and Y .
Arora et al.
(2017) handles this problem by deriving an estimation of gradient of population objecitve whose error can be properly bounded so that applying proximal gradient to a convex relexed objective will provably converge.
Gao et al.
(2019) provides tightened analysis of the time complexity of the algorithm in
Wang et al.
and provides sample complexity under certain distribution.
The problem we are trying to solve in this work is the same as that in Arora et al.
; Gao et al.
(2019): to optimize the population objective of CCA in a streaming fashion.
Riemannian Optimization: Riemannian optimization is a generalization of standard Euclidean optimization methods to smooth manifolds, which takes the following form: Given f : M → R , solve min x ∈M f ( x ) , where M is a Riemannian manifold.
One advantage is that it provides a nice way to express many constrained optimization problems as unconstrained problems.
Applications include matrix and tensor factorization Ishteva et al.
(2011), Tan et al.
PCA Edelman et al.
(1998), CCA Yger et al.
(2012), and so on.
Yger et al.
(2012) rewrites CCA formulation as Riemannian optimization on Stiefel manifold.
In our work, we further explore the ability of Riemannian optimization framework, decomposing the linear space spanned by canonical vectors into products of several matrices which lie in several different Riemannian manifolds.
C ONCLUSIONS
In this work, we presented a stochastic approach (RSG+) for the CCA model based on the observation that the solution of CCA can be decomposed into a product of matrices which lie on certain structured spaces.
This affords specialized numerical schemes and makes the optimization more efﬁcient.
The optimization is based on Riemannian stochastic gradient descent and we provide a proof for its O ( 1 t ) convergence rate with O ( d 2 k ) time complexity per iteration.
In experimental evaluations, we ﬁnd that our RSG+ behaves favorably relative to the baseline stochastic CCA method in capturing the correlation in the datasets.
We also show the use of RSG+ in the DeepCCA setting showing feasibility when scaling to large dimensions as well as in an interesting use case in training fair models.
A A PPENDIX A. A BRIEF REVIEW OF RELEVANT DIFFERENTIAL GEOMETRY CONCEPTS
To make the paper self-contained, we brieﬂy review certain differential geometry concepts.
We only include a condensed description – needed for our algorithm and analysis – and refer the interested reader to Boothby (1986) for a comprehensive and rigorous treatment of the topic.
Riemannian Manifold: A Riemannian manifold, M , (of dimension m ) is deﬁned as a (smooth) topological space which is locally diffeomorphic to the
Euclidean space R m .
Additionally, M is equipped with a Riemannian metric g which can be deﬁned as g X : T X M × T X M → R , where T X M is the tangent space at X of M , see Fig. 2.
If X ∈ M , the Riemannian Exponential map at
X , denoted by Exp X : T X M → M is deﬁned as γ (1) where γ : [0 , 1] → M .
We can ﬁnd γ by solving the following differential equation: γ (0) =
X, ( ∀ t 0 ∈ [0 , 1]) dγdt (cid:12)(cid:12)(cid:12) t = t [-
-]0 =
U.
In general Exp X is not invertible but the inverse Exp − 1 X :
U ⊂ M → T X M is deﬁned only if U =
B r ( X ) , where r is called the injectivity radius Boothby (1986) of M .
This concept will be useful to deﬁne the mechanics of gradient descent on the manifold.
In our reformulation, we will shortly make use of the following manifolds, speciﬁcally, when decomposing U and V into a product of several matrices.
(a) St ( p, n ) : the manifold consists of n × p column orthonormal matrices (b) Gr ( p, n ) : the manifold consists of p -dimensional subspaces in R n (c) SO ( n ) , the manifold/group consists of n × n special orthogonal matrices, i.e., space of orthogonal matrices with determinant 1 . Differential Geometry of SO ( n ) :
SO ( n ) is a compact Riemannian manifold, hence by the HopfRinow theorem, it is also a geodesically complete manifold Helgason (2001). Its geometry is well understood and we recall a few relevant concepts here and refer the reader to Helgason (2001) for details. SO ( n ) has a Lie group structure and the corresponding Lie algebra, so ( n ) , is deﬁned as, so ( n ) = { W ∈ R n × n | W T = − W } .
In other words, so ( n ) (the set of Left invariant vector ﬁelds with associated Lie bracket) is the set of n × n anti-symmetric matrices.
The Lie bracket, [ , ] , operator on so ( n ) is deﬁned as the commutator, i.e., for U, V ∈ so ( n ) , [ U, V ] = UV − V U .
Now, we can deﬁne a Riemannian metric on SO ( n ) as follows: (cid:104) U, V (cid:105)
X = trace (cid:16) U T V (cid:17) where, U, V ∈
T X ( SO ( n )) , X ∈ SO ( n ) .
Note that, it can be shown that this is a bi-invariant Riemannian metric.
Under this bi-invariant metric, now we deﬁne the Riemannian exponential and inverse exponential map as follows.
Let, X, Y ∈ SO
( n ) , U ∈ T X ( SO ( n )) .
Then, Exp − 1 X ( Y ) =
X log( X T Y ) and
Exp X ( U ) =
X exp( X T U ) where, exp , log are the matrix exponential and logarithm respectively.
Differential Geometry of the Stiefel manifold: The set of all full column rank ( n × p ) dimensional real matrices form a Stiefel manifold, St ( p, n ) , where n ≥ p .
A compact Stiefel manifold is the set of all column orthonormal real matrices.
When p < n , St ( p, n ) can be identiﬁed with
SO ( n ) /SO ( n − p ) .
Note that, when we consider the quotient space, SO ( n ) /SO
( n − p ) , we assume that SO ( n − p ) (cid:39) F ( SO ( n − p )) is a subgroup of SO ( n ) , where, F : SO ( n − p ) → SO ( n ) deﬁned by X (cid:55)→ (cid:20) I p
X (cid:21) is an isomorphism from SO ( n − p ) to F ( SO ( n − p )) .
Differential Geometry of the Grassmannian Gr ( p, n ) :
The Grassmann manifold (or the Grassmannian) is deﬁned as the set of all p -dimensional linear subspaces in R n and is denoted by Gr ( p, n ) , where p ∈ Z + , n ∈ Z + , n ≥ p .
Grassmannian is a symmetric space and can be identiﬁed with the quotient space SO ( n ) /S
( O ( p ) ×
O ( n − p )) , where S ( O ( p ) ×
O ( n − p )) is the set of all n × n matrices whose top left p × p and bottom right n − p × n − p submatrices are orthogonal and all other entries are 0 , and overall the determinant is 1 .
A point X ∈ Gr ( p, n ) can be speciﬁed by a basis,
X .
We say that X
=
Col ( X ) if X is a basis of X , where Col ( . ) is the column span operator.
It is easy to see that the general linear group GL ( p ) acts isometrically, freely and properly on St ( p, n ) .
Moreover, Gr ( p, n ) can be identiﬁed with the quotient space St ( p, n ) / GL ( p ) .
Hence, the projection map Π : St ( p, n ) → Gr ( p, n ) is a Riemannian submersion , where Π( X ) (cid:44) Col ( X ) .
Moreover, the triplet ( St ( p, n ) , Π , Gr ( p, n )) is a ﬁber bundle.
At every point X ∈ St ( p, n ) , we can deﬁne the vertical space , V X ⊂ T X St ( p, n ) to be Ker (Π ∗ X ) .
Further, given g St , we deﬁne the horizontal space , H X to be the g St -orthogonal complement of
V X .
Now, from the theory of principal bundles, for every vector ﬁeld (cid:101) U on Gr ( p, n ) , we deﬁne the horizontal lift of (cid:101) U to be the unique vector ﬁeld U on St ( p, n ) for which U X ∈ H X and Π ∗
X
U X
= (cid:101)
U Π
( X ) , for all X ∈ St ( p, n ) .
As, Π is a Riemannian submersion, the isomorphism Π ∗ X | H X : H X → T Π( X ) Gr ( p, n ) is an isometry from ( H X , g St X ) to ( T Π( X ) Gr ( p, n ) , g Gr Π( X ) ) .
So, g Gr Π( X ) is deﬁned as:
[Equation]
where, (cid:101) U, (cid:101)
V ∈ T Π( X ) Gr ( p, n ) and Π ∗
X
U X
= (cid:101)
U Π
( X ) , Π ∗ X V X
= (cid:101) V Π
( X ) , U X ∈ H X
and V X ∈ H X .
We covered the exponential map and the Riemannian metric above, and their explicit formulation for manifolds listed above is provided for easy reference in Table 4.
[Equation]
[Table]
[Table]
A. P ROOF OF T HEOREM
We ﬁrst restate the assumptions from section 3.1:
Assumptions: (a) The random variables X ∼ N ( 0 , Σ x ) and Y ∼ N ( 0 , Σ y ) with Σ x (cid:22) cI d and Σ y (cid:22) cI d for some c > 0 . (b) The samples X and Y drawn from X and Y respectively have zero mean. (c) For a given k ≤ d , Σ x and Σ y have non-zero top- k eigen values.
Let F be the trace value solution for Eq.
(2), and (cid:101) F be the trace value solution for Eqs.
equation 3a, equation 3b, we next restate Theorem 1 and give its proof:
Theorem.
Under the assumptions and notations above, the approximation error
E = (cid:107) F − (cid:101) F (cid:107) is bounded and goes to zero while the whitening constraints in equation 3b are satisﬁed.
Proof.
Let Q u , S u , Q v , S v be the solutions for Eqs.
equation 3a and equation 3b, (cid:101) U, (cid:101) V be matrices consisting of top- k eigen vectors of ( 1 / N ) X T X and ( 1 / N ) Y T Y respectively, U, V be solutions for equation 2.
Let (cid:101) X u
=
X (cid:101) UQ u S u and (cid:101) Y v = Y (cid:101)
V Q v S v .
Also let X u =
XU
and Y
v = Y V .
Observe that mean of X u , Y v , (cid:101) X u and (cid:101) Y v are zero.
Moreover the sample covariance of X u and Y v are given by
U T C X U and V T C Y V respectively.
Thus by the constraint in equation 2, X Tu X u = I k and Y Tv Y v =
I k .
Let these covariance matrices be denoted by C ( X u ) and
C ( Y v ) respectively.
Analogously the sample covariance of (cid:101) X u and (cid:101)
Y v are given by S Tu Q Tu (cid:101) U T C X (cid:101) UQ u S u and S Tv Q Tv (cid:101) V T C Y (cid:101) UQ v S v respectively.
Let these covariance matrices be denoted by C ( (cid:101) X u ) and C ( (cid:101)
Y v ) respectively.
Using Def. 1, we know X u , X v , (cid:101) X u and (cid:101) Y v follow sub-Gaussian distributions.
Let F = trace (cid:0)
U T C XY V (cid:1) which can be rewritten as F = trace (cid:0) X Tu Y v (cid:1) .
Moreover, let (cid:101) F = trace (cid:16) S Tu Q Tu (cid:101) U T C XY (cid:101) V Q v S v (cid:17) which similarly can be rewritten by (cid:101) F = trace (cid:16) (cid:101)
X Tu (cid:101) Y v (cid:17) .
Consider the approximation error between the objective functions as E = | F − (cid:101) F | .
We can rewrite
E = | trace (cid:0)
X T u Y v (cid:1) − trace (cid:16) (cid:101) X T u (cid:101) Y v (cid:17) | .
Due to von Neumann’s trace inequality and Cauchy–Schwarz inequality, we have
[Equation]
(using Von Neumann’s trace inequality) [Equation] using Cauchy-Scwartz’s inequality [Equation]
where σ i (A) denote the i th singular value of matrix A and (cid:107) • (cid:107) F denotes the Frobenius norm.
Now, using Proposition 1, we get
[Equation]
where, ∆ x = C ( X u ) − C ( (cid:101) X u ) , ∆ y = C ( Y v ) − C ( (cid:101) Y v ) .
Here λ x s and λ y s are the eigen values of
C ( X u ) and C ( Y v ) respectively.
Now, assume that C ( X u ) =
I k and C ( Y v )
=
I k as X u and Y v are solutions of Eq.
equation 2.
Furthermore assume λ xk − λ xk +1 ≥ Λ and λ yk − λ yk +1 ≥ Λ for some Λ > 0 .
Then, we can rewrite Eq.
equation A.1 as
E ≤ min [Equation] √
[Equation]
And as C ( (cid:101) X u )
→ I k or C ( (cid:101) Y v ) → I k , E → 0 . Observe that, the limiting conditions for C ( (cid:101) X u ) and C ( (cid:101) Y v ) can be satisﬁed by the “whitening” constraint. In other words, as C ( X u ) =
I k and C ( Y v ) = I k , C ( (cid:101) X u ) and C ( (cid:101) Y v ) converge to C ( X u ) and C ( Y v ) , the approximation error goes to zero.
A. RSG+ ALGORITHM
Here we show our algorithm with more details about the gradients in every step in Alg.2.
A. I MPLEMENTATION DETAILS OF CCA ON FIXED DATASET
Implementation details.
On all three benchmark datasets, we only passed the data once for both our RSG+ and MSG Arora et al.
and we use the code from Arora et al.
(2017) to produce
MSG results.
We conducted experiments on different dimensions of target space: k = 1 , 2 , 4 .
The choice of k is motivated by the fact that the spectrum of the datasets decays quickly.
Since our RSG+ processes data in small blocks, we let data come in mini-batches (mini-batch size was set to 100 ).
A. R UNTIME OF RSG+ AND BASELINE METHODS
The runtime comparison of RSG+ and MSG is reported in Table 5.
Our algorithm is 5 – 10 times faster.
We also plot the runtime of our algorithm under different data dimension (set d x = d y = d ) and number of total samples sampled from joint gaussian distribution in Fig. A. E RROR METRICS FOR FAIRNESS Equality of Opportunity (EO) Hardt et al.
A classiﬁer h is said to satisfy EO if the prediction is independent of the protected attribute s (in our experiment s is a binary variable where
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
Here,
[Table] [Table]
[Table]
[Equation] [Table] [Equation] give the Euclidean gradients.
For , the closed form expression of the gradients is,
[Table] [Equation]
[Table]
[Equation]
Gradient to update canonical directions
[Equation]
∇
[Equation]
∇ X (cid:101) F can where, X is a generic entity: X ∈ { Q u , Q v , S u , S v } ;
Batch update of canonical directions
A =
Exp A (cid:16) − γ j ∇ A (cid:101) F (cid:17) where, A is a generic entity: A ∈ { (cid:101) U, (cid:101) V , Q u , Q v , S u , S v } ; 8 end 9 U = (cid:101) UQ u S u and V = (cid:101) V Q v S v ;
[Figure] [Table]
s
= stands for for for label y ∈ { use the difference of false negative rate (conditioned on y = 1 ) across two groups identiﬁed by protected attribute s metric denote
it
Demographic Parity (DP) Yao & Huang (2017) :
A classiﬁer h satisﬁes DP if the likelihodd of making a misclassiﬁcation among the positive predictions of the classiﬁer is independent of the protected attribute s .
We denote the difference of demographic parity between two groups identiﬁed by the protected attribute as DDP.
A. I MPLEMENTATION DETAILS OF FAIRNESS EXPERIMENTS
Implementation details.
The network is trained for 20 epochs with learning rate 0 . 01 and batch size 256 .
We follow Donini et al.
(2018) to use NVP (novel validation procedure) to evaluate our result: ﬁrst we search for hyperparameters that achieves the highest classiﬁcation score and then report the performance of the model which gets minimum fairness error metrics with accuracy within the highest 90% accuracies.
When we apply our RSG+ on certain layers, we ﬁrst use randomized projection to project the feature into 1 k dimension, and then extract top- 10 canonical components for training.
Similar to our previous experiments on DeepCCA, the batch method does not scale to 1 k dimension.
A R ESNET -18 ARCHITECTURE AND POSITION OF C ONV -0,1,2 IN T ABLE
The Resnet-18 contains a ﬁrst convolutional layer followed by normalization, nonlinear activation, and max pooling.
Then it has four residual blocks, followed by average polling and a fully connected layer.
We denote the position after the ﬁrst convolutional layer as conv 0 , the position after the ﬁrst residual block as conv 1 and the position after the second residual block as conv 2 .
We choose early layers since late layers close to the ﬁnal fully connected layer will have feature that is more directly relevant to the classiﬁcation variable ( attractiveness in this case).