A K ERNEL R ANDOM M ATRIX -B ASED A PPROACH FOR S PARSE PCA
A BSTRACT
In this paper, we present a random matrix approach to recover sparse principal components from n p -dimensional vectors.
Speciﬁcally, considering the large dimensional setting where n, p → ∞ with p/n → c ∈ (0 , ∞ ) and under Gaussian vector observations, we study kernel random matrices of the type f ( ˆ C ) , where f is a three-times continuously differentiable function applied entry-wise to the sample covariance matrix ˆ C of the data.
Then, assuming that the principal components are sparse, we show that taking f in such a way that f (cid:48) (0) =
f (cid:48)(cid:48) (0) =
allows for powerful recovery of the principal components, thereby generalizing previous ideas involving more speciﬁc f functions such as the soft-thresholding function.
I NTRODUCTION
Principal component analysis (PCA) is extensively used in data analysis and machine learning applications.
It is a dimension reduction technique that aims to project a given dataset onto principal subspaces spanned by the leading eigenvectors of the sample covariance matrix (Wold et al., 1987), which represent the principal modes of variance.
Basically, the statistical interpretation of PCA lies in the fact that most of the variance in the data is captured by these modes.
Consequently, PCA reduces the dimension of the feature space while keeping most of the information in the data.
It is well-known (Anderson, 1963) that PCA performs efﬁciently in the traditional data setting where the number of features is small and the number of samples is large.
Consider a data matrix Y ∈ R p × n consisting of n centered samples, each sample having p features.
The standard PCA method requires the computation of the sample covariance matrix ˆ C
=
Y Y (cid:124) /n
and estimates the ﬁrst principal components u 1 , u 2 , . . .
( i.e. , the successive dominant eigenvectors of
C = E
[ Y Y (cid:124) /n ] ) by the ordered eigenvectors ˆ u 1 , ˆ u 2 , . . .
of ˆ C .
(Johnstone & Lu, 2009) demonstrated that, in the high dimensional regime where n, p → ∞ with p/n → c > 0 , the principal component ˆ u 1 estimated by standard PCA is inconsistent.
Essentially, if p/n (cid:57) 0 then (cid:107) ˆ u 1 − u 1 (cid:107) 2 (cid:57) 0 in the high-dimensional asymptotic regime.
This phenomenon is well investigated within the ﬁeld of random matrix theory for covariance models of the form ˆ C = 1 n Σ 1 / 2 p XX (cid:124) Σ 1 / 2 p , where Σ p is a positive semi-deﬁnite matrix and X is a p × n matrix with random i.i.d.
entries.
One of the main results from random matrix theory concerns the so-called spiked models, where Σ p is a low-rank perturbation of the identity matrix, namely Σ p
= I p
+ (cid:80) ki
=1 ω
i u
i u (cid:124) i with k ﬁxed with respect to p, n .
(Baik et al., 2005) and (Paul, 2007) notably exhibited a phase transition phenomenon: as p/n → c , if ω
i < √ c the estimated principal component ˆ u i using standard PCA is (almost surely) asymptotically orthogonal to the true principal component u i ( i.e. , ˆ u (cid:124)
i u i → 0 ); on the other hand, if ω i > √ c , lim inf n | ˆ u (cid:124)
i u i | > 0 .
This phase transition phenomenon has attracted recently much attention within the random matrix community (Benaych-Georges & Nadakuditi, 2011;
Capitaine et al., 2009; Féral & Péché, 2007; Knowles & Yin, 2013).
The inconsistency of standard PCA in high dimensions motivated the idea to look for more structural information on the principal components.
In particular, considering that the principal components are sparse in an appropriate basis ( e.g. , in the wavelet domain), a large body of works have emerged and proposed improved PCA approaches that account for sparsity.
One of the most consistent sparse
PCA methods in the literature is the covariance thresholding (CT) algorithm
(Krauthgamer et al., 2015).
Based on the intuition that the small entries of the empirical covariance matrix ˆ C induce noise in its principal components, this method consists in applying the popular soft-thresholding function (with threshold τ > 0 ); soft( · ; τ ) :
t (cid:55)→ sign( t ) · ( | t | − τ ) + , entry-wise to the empirical covariance matrix ˆ C and performing PCA on the resulting matrix.
(Deshpande & Montanari, 2014; 2016) have theoretically demonstrated that the covariance thresholding algorithm recovers the sought-for principal components with high probability under controlled growth rates between p , n and the sparsity level.
In this paper, we show that the soft-thresholding method in fact falls within a broader class of kernel-based 1 PCA algorithms that are particularly suited to sparse PCA recovery.
This method consists in considering the matrix f
(ˆ C ) instead of ˆ C where f is a function applied entry-wise.
By imposing some constraints on f , most importantly that f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 , sparse PCA can be performed with provably high accuracy for sufﬁciently large n, p .
The rest of the paper is organized as follows.
In Section 2, we present the related work on sparse
PCA.
We recall some necessary concentration of measure tools and notions about sparse matrices in Section 3.
Our main theoretical results are then provided in Section 4.
Section 5 discusses the practical aspects and provides experimental results.
Section 6 concludes the article.
Notation: In following, the notation [ n ] denotes the set { 1 , . . . , n } , (cid:98) a (cid:99) denotes the integer part of a .
Given a vector x ∈ R n , the (cid:96) 2 -norm of x is denoted as (cid:107) x (cid:107) 2 = (cid:80) ni
=1 x 2 i .
Given an n × n matrix M , M ij or [ M ] ij denote the entry of the matrix M at line i and column j .
[ M ] · ,j denotes the j -th column vector of M and [ M ] i, · its i -th line vector.
The Frobenius norm (cid:107) M (cid:107) F of the matrix M is deﬁned as (cid:107) M (cid:107) 2 F = (cid:80) ni,j =1 M 2 ij , and the operator norm (cid:107) M (cid:107) op of M is deﬁned as (cid:107) M (cid:107) op = max (cid:107) x (cid:107) =1 (cid:107) Mx (cid:107) .
Finally, (cid:12) denotes the Hadamard product, with [ M (cid:12) N ] ij =
M ij N ij .
R ELATED WORK
The problem of sparse PCA has been tackled with a large range of techniques.
Mainly, three classes of approaches emerge in the literature.
Most popular techniques are optimization-based algorithms (d’Aspremont et al., 2005; Moghaddam et al., 2006; Zass & Shashua, 2007; Zou et al., 2006; Wright et al., 2009), where the idea is to see the problem of sparse PCA through an optimization perspective, and to propose methods to solve the latter by either considering a different formulation – e.g. semi-deﬁnite programming (SDP) or convex relaxations – or adding penalties to the original optimization problem such as a LASSO regularization.
The second class of approaches covers matrix decomposition-based techniques (Asteris et al., 2014;
Papailiopoulos et al., 2013; Shen &
Huang, 2008), where sparse principal components are extracted through solving a low rank matrix approximation problem based on Singular Value Decomposition.
Finally, most consistent sparse PCA methods adopt thresholding-based approaches: initial heuristics used factor rotation techniques and thresholding of eigenvectors to obtain sparsity (Cadima & Jolliffe, 1995).
Based on the well-known power method, (Yuan & Zhang, 2013) introduced an efﬁcient sparse PCA approximation to obtain the exact level of required sparsity, by truncating to zero the principal components iteratively except for their largest entries.
A step further, under a spiked covariance model (see Section 4.1), (Ma et al., 2013) proposed a very efﬁcient iterative thresholding approach for estimating principal subspaces in the sparse setting.
Similarly, assuming a single-spike model, (Krauthgamer et al., 2015) proved that, when the sparsity level s ≥ Ω( √ n ) , a standard SDP approach cannot recover consistently the sparse spike; in particular, the authors presented empirical results suggesting that for s =
O ( √ n ) , recovery is possible by a simple covariance thresholding algorithm.
More recently, (Deshpande &
Montanari, 2016) analyzed and theoretically proved, under a spiked model, that indeed the covariance thresholding algorithm (Krauthgamer et al., 2015) succeeds with high probability under controlled growth rates between p, n and s .
In this work, while restricting ourselves to a setting where p and n grow at a controlled joint rate, we provide an elementary argument, based on a matrix-wise Taylor expansion controlled through a concentration of measure approach, that generalizes the CT method to a large family of kernel-based methods, by means of a kernel random matrix approach (El Karoui, 2010b;a).
Concretely, we study kernel random matrices of the form f
( Y Y (cid:124) /n )
where Y = Σ 1 / 2 p X and X is a random matrix with
N (0 , 1) i.i.d.
entries.
(El Karoui, 2010b) studied kernel matrices of the form f ( Y (cid:124) Y/n ) ( i.e. , the socalled inner-product kernel matrices), which is equivalent to the case Σ p =
I p when considering the form f
( Y Y (cid:124) /n ) .
In particular, we elaborate from El Karoui’s study by Taylor expanding f ( Y Y (cid:124) /n ) in the vicinity of Σ p entry-wise and controlling the resulting matrices via concentration arguments.
P RELIMINARIES
Before introducing our model setting we recall some deﬁnitions and notions of the concentration of measure theory (Ledoux, 2005) that are at the heart of our main results.
Furthermore, we recall a deﬁnition, introduced by (El Karoui, 2008), of sparse matrices in the large-dimensional context that will also be exploited in this paper.
3. C ONCENTRATION OF MEASURE RESULTS
We start by a deﬁnition of the notion of concentration for a real random variable.
Deﬁnition 1 (Concentration of a Random Variable) .
Given a function δ : R + → R + , a random variable Z is said to be δ -concentrated (around its mean) and we write Z ∈ δ , if for all t > 0 , P {| Z − E Z | ≥ t } ≤ δ ( t ) .
In particular, Z is said to be normally (resp., exponentially) concentrated when δ ( t ) =
Ce − ct 2 (resp., δ ( t ) =
Ce − ct ) and we write Z ∈ C N ( c · ) (resp.
, Z ∈ C E ( c · ) ), where
C, c > 0 are some absolute constants.
In particular, δ -concentration remains stable by application of Lipschitz functions:
Proposition 1 (Concentration of Lipschitz Functions) .
Given a λ -Lipschitz function f : R → R and a concentrated random variable Z ∈ δ , we have f ( Z ) ∈ δ ( · /λ ) .
As a consequence, linear combinations of δ -concentrated random variables remain concentrated.
However, products of δ -concentrated random variables are more technical to handle, but we still have the following proposition in the case of normally concentrated random variables and which will be essential in this article.
Proposition 2 (Square of Normally Concentrated Random Variables) .
Given Z ∈ C N ( c · ) , the random variable Z 2 is exp-normally concentrated, precisely
[Equation]
where K C > 0 is a constant depending only on C .
The extension of the notion of concentration to random vectors Z ∈ R p demands that R p
→ R
Lipschitz functions are concentrated random variables.
Deﬁnition 2 (Concentration of a Random Vector) .
Given a function δ : R + → R + and a normal space ( E, (cid:107) .
(cid:107) ) , a random vector Z ∈ E is said to be δ -concentrated if for any 1 -Lipschitz function f : E → R , the random variable f
( Z ) is δ -concentrated.
We note again Z ∈ δ .
In particular, we have the concentration of Gaussian random vectors in the sense of Deﬁnition 2 in the following proposition (Tao, 2012, Theorem 2.1.12).
Proposition 3 (Normal Concentration of Gaussian Random Vectors) .
A Gaussian vector Z ∈ R p , with independent and identically distributed N (0 , 1) entries, is normally concentrated independently on the dimension p .
Furthermore, Z ∈ 2 N ( · / 2) .
Remark 1.
According to Deﬁnition 2, given a Lipschitz application F : R p
→ R q for q ∈ N ∗ ,
Proposition 3 provides the normal concentration of all the random vectors F ( Z ) .
In particular, note that our results are extensible to this family of vectors and random vectors with independent entries.
3. ε - SPARSE MATRICES
When considering a large-dimensional random matrix setting, the notion of sparsity for such matrices is particularly attached to the choice of the matrix norm.
2 (El Karoui, 2008) introduced a deﬁnition ( ε -sparsity) for sparsity of matrices that is compatible with spectral analysis, and speciﬁcally adapted to the operator norm.
The ε -sparsity deﬁnition requires some notions from graph theory that we present in the following: to each p × p symmetric matrix M , we deﬁne its corresponding adjacency matrix as A ( M ) =
{ 1 M ij (cid:54) =0 } pi,j =1 , which corresponds to a graph G p with p vertices.
A walk is said to be closed on this graph if it starts and ﬁnishes at the same vertex and the number of edges traversed by a walk deﬁnes the length of this walk.
Denote C p ( k ) the set of closed walks of length k on G p .
Deﬁnition 3 ( ε -sparse matrices (El Karoui, 2008, Deﬁnition 1)) .
A sequence of covariance matrices { Σ p } ∞ p =1 is said to be ε -sparse if the sequence of their associated graphs {G p } ∞ p =1 satisﬁes, for all k ∈ 2 N , |C p ( k ) | ≤ C k p ε ( k − 1)+1 where ε ∈ [0 , 1] , C k > 0 independent of p and |S| denotes the cardinality of the set
S .
The ε -sparsity is both useful and convenient to out study for the following reasons: 1) it is adapted to the analysis of the operator norm of large sparse matrices (as we give concentration results on the operator norm); 2) it is also more general than other sparsity notions such as in (Bickel & Levina, 2008).
In the latter, the authors developed a natural permutation-invariant notion of sparsity which is more speciﬁc than Deﬁnition 3 as pointed out in the introduction of their article.
Furthermore, note that both sparsity notions (Deﬁnition 3 and the one in (Bickel & Levina, 2008)) provide equivalent bounds for ε < 12 and when considering the large dimensional p ∼ n setting (see subsection 2.4 in (Bickel & Levina, 2008)); this is precisely the setting considered in Corollary 2 introduced subsequently ( cf. µ > 0 ).
Remark 2.
As Deﬁnition 3 is based on a graph deﬁned by its corresponding adjacency matrix, we have the following property: given an ε -sparse matrix M and a function f such that f (0) =
and f ( x ) (cid:54) = x (cid:54) =0 0 , the matrix f ( M ) , resulting from the application of f entry-wise to M , remains ε -sparse; this is simply a consequence of A ( M )
=
A ( f ( M )) .
M AIN R ESULTS
In this section, we ﬁrst present the setting of the article.
Then, we provide an asymptotic equivalent to the matrix f
(ˆ C ) .
Finally, we treat as a special case the application of our result to the context of sparse PCA.
4. G ENERAL S ETTING AND M AIN R ESULTS
Consider a data matrix Y ∈ R p × n deﬁned as
[Equation]
[Equation]
where X ∈ R p × n is a random matrix with i.i.d.
N (0 , 1) entries, P = (cid:80)
ki =1 ω
i u i
[Equation]
(cid:124) [Equation] and
U =
[ u 1 , . . .
, u k ] ∈ R p × k is isometric.
Here, k refers to the number of principal components (or eigenvectors)
u 1 , . .
. , u k ∈ R p to be evaluated, with ω 1 > . . .
>
ω k > 0 the corresponding eigenvalues respectively.
We deﬁne the quantity 3 β p ≡ max i (cid:107)
[Σ 1 / 2 p ] · ,i (cid:107) .
Assumptions: There exists B > 0 independent of p, n such that max ij | [Σ p ] ij | < B .
Besides, there exists (cid:15) > 0 such that β p ≤ B (cid:48) n 14 − (cid:15) for all p, n and for some absolute constant B (cid:48) > 0 .
Under these assumptions, our main technical result is as follows:
2 Considering the identity matrix (which is a sparse matrix), (cid:107) I p (cid:107) op = 1 while (cid:107) I p (cid:107) F = √ p → ∞
The role of β p is to ensure the concentration of the quadratic form in equation 4 introduced subsequently.
When Σ p is a sparse matrix, β p plays the same role as the maximum spike strength in the bounds given in (Deshpande & Montanari, 2016) for the CT method.
Theorem 1 (Asymptotic Equivalent) .
For f a three-times continuously differentiable function, deﬁne the matrices F and ˜ F respectively by
[Equation]
Then for η > 0 and for an absolute constant C > 0 , we have with probability at least 1 − η
[Equation]
For a general smooth function f , the kernel random matrix f
(ˆ C ) is particularly difﬁcult to analyze through the usual tools of random matrix theory, such as the moment or Stieltjes transform-based methods (Tao, 2012).
Rather than directly analyzing such a kernel random matrix, Theoremgives an asymptotic equivalent to it, in operator norm, that has mainly two properties.
First, the approximation matrix ˜ F contains “simple” objects that have already been analyzed in random matrix theory – in particular, the term ( XX (cid:124) /n − I p ) in the expression of ˜ F .
Second, the approximation in operator norm implies (by Weyl’s inequality (Eisenstat & Ipsen, 1998, Theorem 4.1))
that, when (cid:107) F − ˜ F (cid:107) op → 0 , F and ˜ F have the same eigenvalues and same “isolated” eigenvectors asymptotically (see Corollary 2 subsequently).
Sketch of Proof of Theorem 1.
The main idea of the proof relies on the following intuition: for large n , the entries of XX (cid:124)
/n
− I p and its successive Hadamard products tend to zero at controllable rate.
The concentration of measure framework then allows for the control of non-linear functions of the entries of XX (cid:124)
/n − I p .
Of utmost importance to this end is the following lemma.
Lemma 1 (A Concentration Result) .
For all i, j ∈ [ p ] , the bilinear form g ij ( X ) ≡ [Σ 1 / 2 p
[Equation]
i, ·
[Equation]
satisﬁes
[Equation]
for some absolute constants c 1 , c 2 , K > 0 .
Proof.
Denoting by v i the i -th column vector of the matrix Σ 1 / 2 p , we have by the polarization identity, for all M Hermitian, v (cid:124) i
Mv j =
[( v i + v j ) (cid:124) M
( v i + v j ) −
v
i − v j )
(cid:124) M ( v i − v j )] .
It thus sufﬁces to prove the result for the quadratic form g ( X ) = v (cid:124) (cid:0)
n XX (cid:124) − I p (cid:1) v where v ∈ R p .
Noticing that v (cid:124) XX (cid:124) v = (cid:107) v (cid:124) X (cid:107) 2 and E (cid:2) 1 n v (cid:124) XX (cid:124) v (cid:3)
= v (cid:124) v , we need to prove the concentration of the random variable (cid:107) v (
cid:124) X (cid:107) 2 .
In fact, since v (cid:124)
X is a Gaussian vector, by Proposition 3, (cid:107) v (cid:124)
X
(cid:107) ∈ 2 N (cid:16) · 2 (cid:107) v (cid:107)(cid:17) by Remark 1 and by Deﬁnition 2 since M (cid:55)→ v (cid:124) M and u (cid:55)→ (cid:107) u (cid:107) are respectively (cid:107) v (cid:107) -Lipschitz and 1 -Lipschitz functions.
We get the ﬁnal result by Proposition 2.
A Taylor expansion of F around f (Σ p ) then leads to controlling the operator norm of f (3) ( ξ n ) (cid:12) [Σ 1 / 2 p ( XX (cid:124) /n
− I p )Σ 1 / 2 p ] (cid:12) 3 for ξ n a matrix with entries in the set [[ Y Y (cid:124) /n ] ij , [Σ p ] ij ] (or [[Σ p ] ij , [ Y Y (cid:124) /n ] ij ] ).
This follows precisely from exploiting Lemma 1 twice, to control the ﬂuctuations of the entries of both ξ n (by the conditions on max ij | [Σ p ] ij | and β p ) and [Σ 1 / 2 p ( XX (cid:124) /n −
I p )Σ 1 / 2 p ] (cid:12) 3 , with the bound provided in the theorem statement, thereby completing the proof.
A detailed proof of Theorem 1 is provided in Section A.2 of the Appendix.
From now on, to simplify our arguments, we make the following assumptions:
Assumptions:
As
[Equation]
Under this setting, we have the following important corollary to Theorem 1.
Corollary 1.
Deﬁne the matrices F and ˜ F as in Theorem 1 and Assumptions A1 and A2 hold.
Then, for η >
[Equation]
where the notation X =
O mη ( n − α ) stands for the fact that P
[Equation]
η for some absolute constant C > 0 and non-negative integer m .
As a consequence of Corollary 1, we have, by the sin(Θ) theorem of (Davis & Kahan, 1970), the corollary below concerning the eigenvectors of the matrices F and ˜ F .
Corollary 2.
Let v 1 , . .
. , v k and ˜ v 1 , . . .
, ˜ v k denote respectively the k principal eigenvectors of F and ˜ F .
Denote by ∆ i = ω
i − ω
i +1 for i ∈
[ k − 1] .
Then for η > 0 , we have
[Equation]
4. S PECIAL C ASE : S PARSE PCA
To get an insight on our coming results, consider the scenario where U contains ﬁnitely many non-zero entries.
In this case, the perturbation matrix P in Eq.
equation 11 contains ﬁnitely many non-zero entries (say s ) on each line and a simple enumeration shows that |C p ( k ) | ≤ p s k − 1 , thus P is 0 -sparse in the sense of Deﬁnition 3.
Similarly, I p is 0 -sparse and by the additive stability 5 of the ε -sparsity notion, Σ p remains 0 -sparse.
More generally, if we assume that it exists ε ∈ [0 , 12 ) such that the population covariance matrix Σ p is ε -sparse, we have the following set of consequences.
By
Corollary 1, choosing f in such a way that f (cid:48) (0) =
f (cid:48)(cid:48) (0)
= 0 ensures that the terms f (cid:48) (Σ p ) (cid:12) . . .
and f (cid:48)(cid:48) (Σ p ) (cid:12) . . .
vanish in the expression of ˜ F .
Indeed, for k ∈ { 1 , 2 }
(i) Only ﬁnitely entries of f ( k ) (Σ p ) do not vanish, precisely by Remark 2, since A ( f ( k ) (Σ p ))
=
A (Σ p ) , 6 the matrix f
( k ) (Σ p ) is also (almost) ε -sparse.
(ii)
The matrix F ( k ) =
(cid:104) Σ 1 / 2 p (cid:0) 1 n XX (cid:124) − I p (cid:1) Σ 1 / 2 p (cid:105) (cid:12) k has entries of order O ( n − k/ 2 ) .
As a result, 7 we have for η > 0 and for all m > 0 , max i,j
[Equation]
F ( k )
[Equation]
Since in addition the operator norm of Σ 1 / 2 p ( XX (cid:124) /n − I p )Σ 1 / 2 p is typically of order O (1) (see e.g. , (Bai & Silverstein, 1998)), it is then easily seen that, for each k ≥ 1 , the operator norm of the
Hadamard product f ( k ) (Σ p ) (cid:12) F ( k ) vanishes (see Lemma 5 in Appendix A).
In particular, note that the non-zero entries of Σ p are controlled through the maximum entry of F ( k ) which is vanishing asymptotically, as mentioned in item (ii) above.
On the opposite f (Σ p ) does not vanish since it has entries bounded away from zero (as long of course as f (cid:54) = 0 ).
We precisely have the following result.
Theorem 2.
Let µ > 0 and suppose Σ p is a 1 2+ µ -sparse matrix.
For f a three-times continuously differentiable function and for η > 0 , we have for all (cid:15) ∈ (
[Equation]
Proof.
See Section A.3 in Appendix A. (See Corollary 1 for the notation O mη ( . ) ).
Remark 3.
Theorem 2 gives a general result concerning the estimation of ε -sparse covariance matrices (more precisely, element-wise functionals of sparse covariance matrices).
In particular, the spiked model in Eq.
equation 11 with U sparse corresponds to the particular case when µ → ∞ ; in this case, for η > 0 and for all (cid:15) ∈ (0 , 1 / 4) , F = f (Σ p )
+ O
[Equation] [Equation]
η
[Equation]
n
12 +
[Equation]
[Figure] [Figure]
gap
[Figure]
D e n s it y
[Figure] [Figure]
Eigenvalues
D e n s it y
[Figure]
) p p/n
[Figure]
and
One may then perform a PCA on F for some function f with f (cid:48) (0) =
f (cid:48)(cid:48) (0) =
(we denote by f 1 , 2 (0) =
these two conditions in the following).
But, while Σ p = I p
+ P is a low rank perturbation of the identity (therefore having only k eigenvalues strictly greater than 1), f (Σ p ) is likely more complex and not a mere low rank deformation of the identity.
Now, if Σ p has all its non-zero entries greater than a certain threshold τ , an appropriate choice for f that avoids the deformation of I p + P is such that f 1 , 2 (0)
=
and f ( t ) =
t for all | t | > τ .
Such a convenient choice is
[Equation]
for some a > 0 .
This function notably satisﬁes f
[Equation]
The ﬁgure above depicts the function f along with its derivatives for a = 1 .
Note that a compromise in the choice of a must be made that both maintains a close approximation of the identity by f on a large range and rather small values of f (cid:48)(cid:48) in the vicinity of zero.
Interestingly, it can be veriﬁed that the extrema of f (cid:48) are independent of a but are found at ± (cid:113) 32 a and thus smaller values of a create sharper f (cid:48) in the vicinity of zero.
Similarly, the extrema of f (cid:48)(cid:48) are found at ± (cid:113)
[Equation]
a , and precisely given by the four values 2 (cid:113)
[Equation] [Equation]
[Equation] [Equation]
6) ∝ √ a .
Thus smaller a induce larger maxima for f (cid:48)(cid:48) but no sharper slope.
E XPERIMENTS
In this section, we provide some experiments in the context of sparse PCA, where we consider the spiked model presented in Section 4.1.
Precise setting given in caption of Figure 1.
The spectrum of the sample covariance matrix (in gray) is quite different from that of Σ p .
One instead observes a “bulk” of eigenvalues spread in the vicinity of 1 .
Furthermore, one observes a gap between the true spike and the estimated spike (in red) through the sample covariance matrix.
This phenomenon is well-understood in random matrix theory.
In particular, the extreme eigenvalue in our setting converges almost surely to the quantity (1 + ω 1 ) (cid:16) 1 + cω 1 (cid:17) , where we recall that c = lim n p/n .
However, thanks to sparsity, the spectrum of F = f ( ˆ C ) closely matches that of Σ p , as suggested by
Theorem 2.
In particular, the extreme eigenvalue, which corresponds to the principal component, is consistently estimated.
Figure 1 (right) depicts the alignment between the estimated principal
[Figure]
Our method retrieves the principal component even when the spike is not visible in the spectrum of ˆ C ; namely beyond the phase transition (cid:112) p/n ≥ ω 1 .
In fact, the standard PCA result is too noisy compared with the one when considering f (ˆ C ) , as depicted in Figure 2.
Further detailed examples are provided in Section A.4 of Appendix A, that conﬁrm the consistency of the proposed method.
In terms of complexity, as our method consists in computing the sparse eigenvectors of a p × p matrix which can be done by power method, the complexity of estimating the principal component is about O ( ps ) where s is the sparsity level.
And regarding the performance w.r.t.
state-of-theart methods, Figure 3 depicts the performances of standard PCA, different state-of-the-art sparse
PCA methods and our method, in terms of total projections score (left) and total projections error (right), for different values of the amplitudes ω i ’s.
We refer, in this ﬁgure, to standard PCA as PCA,
TpowPCA for the method in (Yuan & Zhang, 2013), ITSPCA for the method in (Ma et al., 2013),
CT refers to the method in (Deshpande & Montanari, 2016) and ﬁnally we refer to our method as f -PCA.
The total projections score S and error E are given respectively by S
= 1 k (cid:80) ki
=1 ( u (cid:124)
i ˆ u i )and E = (cid:107) UU (cid:124) − ˆ U
ˆ U (cid:124) (cid:107) F , where U =
[ u 1 , . . .
, u k ] are the ground truth principal components and ˆ U =
[ˆ u 1 , . . .
, ˆ u k ] are the estimated ones.
As suggested theoretically and veriﬁed experimentally, our proposed method strongly attenuates the “noise component” of the sample covariance matrix and thus consistently estimates the principal components.
In particular, in term of total projections score, PCA is the most inconsistent.
In general,
ITSPCA, CT and our method give equivalent results.
The same holds when considering the total projections error as a metric, except that TpowPCA performs inconsistently, compared to PCA, for small values of amplitudes due to the initialization step from the PCA eigenvectors.
[Figure]
The mostly used concurrent methods to PCA in a sparse context are iterative truncated power methods (such as the TPower (Yuan & Zhang, 2013) algorithm or the ITSPCA (Ma et al., 2013) approach).
These algorithms, despite great observed performances, as compared to standard PCA, suffer from two limitations.
First, they are usually initialized from the PCA eigenvectors themselves and may not converge to good estimates.
For weak signals, PCA is so impacted by noise that the mentioned initialization limitation may lead to non convergent or dramatically erroneous outcomes of the method.
The proposed approach deals precisely with this limitation by strongly attenuating the “noise component” of the sample covariance matrix.
In particular, our approach gives equivalent results to the CT method while generalizing it to the class of smooth functions f such that f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 , in the considered regime.
The second limitation concerns the choice of the hyper-parameters; in fact,
TPower and ITSPCA need to set up an arbitrary deterministic threshold value that maintains at each iteration step only most powerful components.
The proposed method as well as CT need also to set up a “soft” parameter ( a and τ respectively).
But, on the basis of (Cheng & Singer, 2013; Kammoun & Couillet, 2017), we believe that our present investigation can be extended to the asymptotically non-trivial setting where ω
i =
O (1 / √ p ) (in which case the dominant eigenmodes scale at a similar rate with residual noise)
; this setting may likely allow to exhibit and estimate optimal hyper-parameter choices.
Notably, this setting has already been used in (Tiomoko Ali et al., 2018) in a different context, for hyper-parameters estimation.
C ONCLUSION
In this paper, we tackled the problem of sparse PCA through a random matrix perspective thereby generalizing recent ideas to a broader kernel-based method.
Our analysis of this problem has yielded insights into how the principal components can be consistently estimated.
Namely, given a spiked covariance model ˆ C and a smooth function f , we gave in this paper sufﬁcient conditions on f to consistently estimate the principal components through the matrix f
(ˆ C ) .
Our methodology can be generalized to other sparse covariance matrix-based contexts, in the same vein as the works in (Bickel & Levina, 2008; El Karoui, 2008).
A P ROOFS AND FURTHER EXPERIMENTS
In this appendix we provide the proofs of the different results presented in the paper and some additional experiments that validate our ﬁndings.
It includes the following items: (i) the proof of
Theorem 1 (Section A.2); (ii) The proof of Theorem 2 concerning the analysis of the sparse case (Section A.3); and ﬁnally (iii) further experiments which conﬁrm the consistency of our method and the necessity of the conditions f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 on the kernel function f , using the signals of
Johnstone et al.
(Johnstone & Lu, 2009)
(Section A.4).
For convenience we make the present appendix self-contained by recalling the preliminaries and the results presented in the main paper.
A. P RELIMINARIES
Proposition 2 (Square of Normally Concentrated Random Variables) .
Given Z ∈ C N ( c · ) , the random variable Z 2 is exp-normally concentrated, precisely
[Equation]
where K C > 0 is a constant depending only on C .
Deﬁnition 2 (Concentration of a Random Vector) .
Given a function δ : R + → R + and a normal space ( E, (cid:107) .
(cid:107) ) , a random vector Z ∈ E is said to be δ -concentrated if for any 1 -Lipschitz function f : E → R , the random variable f
( Z ) is δ -concentrated.
We note again Z ∈ δ .
Proposition 3 (Normal Concentration of Gaussian Random Vectors (Tao, 2012, Theorem 2.1.12)) .
A Gaussian vector Z ∈ R p , with independent and identically distributed N (0 , 1) entries, is normally concentrated independently on the dimension p .
Furthermore, Z ∈ 2 N ( · / 2) .
Remark 1.
According to Deﬁnition 2, given a Lipschitz application F : R p
→ R q for q ∈ N ∗ ,
Theorem 3 provides the normal concentration of all the random vectors F ( Z ) .
Deﬁnition 3 ( ε -sparse matrices (El Karoui, 2008, Deﬁnition 1)) .
A sequence of covariance matrices { Σ p } ∞ p =1 is said to be ε -sparse if the sequence of their associated graphs 8 {G p } ∞ p =1 satisﬁes, for all k ∈
[Equation]
N
[Equation]
(10) where ε ∈ [0 , 1] , C k > 0 independent of p and |S| denotes the cardinality of the set S .
Remark 2.
As Deﬁnition 3 is based on a graph deﬁned by its corresponding adjacency matrix, we have the following property: given an ε -sparse matrix M and a function f such that f (0) =
and f ( x ) (cid:54) = x (cid:54) =0 0 , the matrix f ( M ) , resulting from the application of f entry-wise to M , remains ε -sparse; this is simply a consequence of A ( M )
=
A ( f ( M )) .
A. P ROOF OF T HEOREM
Setting: Consider a data matrix Y ∈ R p × n deﬁned as
[Equation]
where X ∈ R p × n with i.i.d.
N (0 , 1) entries, P =
[Equation]
ki
[Equation] [Equation]
[Equation] [Equation]
R
[Equation]
× k is isometric, k refers to the number of principal components and ω 1 > . . .
>
ω k > 0 .
Let β p ≡ max
i (cid:107) [
[Equation]
[Equation] [Equation]
(cid:107) .
Assumptions: There exists B > 0 independent of p, n such that max ij | [Σ p ] ij | < B .
Besides, there exists (cid:15) > 0 such that β p ≤ B (cid:48) n 14 − (cid:15) for all p, n and for some absolute constant B (cid:48) > 0 .
Under these assumptions, we have
Theorem 1.
For f a three-times continuously differentiable function, deﬁne the matrices F and ˜ F respectively by
[Equation]
Then for η > 0 and for an absolute constant C > 0 , we have with probability at least 1 − η
[Equation]
Proof.
Before starting the proof, we need to introduce the following key lemmas:
Lemma 1 (A Concentration Result) .
For all i, j ∈ [ p ] , the bilinear form g ij ( X ) ≡ [Σ 1 / 2 p ] i, ·
[Equation]
for some absolute constants c 1 , c 2 , K > 0 .
Proof.
Denoting by v i the i -th column vector of the matrix Σ 1 / 2 p , we have by the polarization identity, for all M Hermitian, v (cid:124) i
Mv j =
[( v i + v j ) (cid:124) M
( v i + v j ) −
v
i − v j )
(cid:124) M ( v i − v j )] .
It thus sufﬁces to prove the result for the quadratic form g ( X ) = v (cid:124) (cid:0)
n XX (cid:124) − I p (cid:1) v where v ∈ R p .
Noticing that v
[Equation]
XX (cid:124) v = (cid:107) v (cid:124) X (cid:107) 2 and E (cid:2) 1 n v (cid:124) XX (cid:124) v (cid:3)
= v (cid:124) v , we need to prove the concentration of the random variable (cid:107) v (
cid:124) X (cid:107) 2 .
In fact, since v (cid:124)
X is a Gaussian vector, by Proposition 3, (cid:107) v (cid:124)
X
(cid:107) ∈ 2 N (cid:16)
[Equation]
(cid:107) v (cid:107)(cid:17) by Remark 1 and by Deﬁnition 2 since u (cid:55)→ (cid:107) u (cid:107) and M
(cid:55)→ v (cid:124)
M are respectively 1-Lipschitz and (cid:107) v (cid:107) -Lipschitz functions.
We get the ﬁnal result by Proposition 2.
Lemma 2 (A Moment Result) .
For g ij ( X ) ≡ [Σ 1 / 2 p ] i, ·
[Equation]
1 n XX
[Equation]
− I p (cid:1)
[Σ 1 / 2 p ] · ,j , we have, for all k ∈ N and for some absolute constant C k > 0 ,
[Equation] [Equation]
Proof.
Given a random variable Z , we have
[Equation]
whenever the right hand side is ﬁnite.
Applying this identity to the random variable g ij ( X ) with m = 2 k and exploiting the concentration property in Lemma 1 yields the result.
The proof starts by a Taylor expansion of F ij in the vicinity of [Σ p ] ij , i.e. ,
[Equation]
where σ ij =
[Σ p ] ij , ξ nij ∈ (cid:98)
[ Y Y (cid:124) /n ] ij , σ ij (cid:99) , 9 and F ( k ) is the matrix with entries
[Equation]
We have by Lemma 1 that [ Y Y (cid:124) /n ]
ij concentrates around σ ij , so that ξ nij is bounded by σ ij + ε , for all ε > 0 , with high probability 10 (note that the condition max ij | σ ij | < B ensures that σ ij is bounded and the condition on β p ensures the quasi-exponential concentration of [ Y Y (cid:124) /n ]
ij around σ ij ; see considered Assumptions above), formally
[Equation]
where K (cid:48) > 0 .
And since f (3) is continuous, we deduce that f (3) ( ξ nij ) is in particular bounded by
[Equation]
with probability 1 − p n .
Knowing that the operator norm is bounded by the Frobenius norm, we look for a control of the Frobenius norm of the tailing term.
We have
[Equation]
By Lemma 2, for all k ∈ N
[Equation]
for some absolute constant C k > 0 .
Thus, by Markov’s inequality , we have for all η >
[Equation]
Recalling Eq.
equation 15, we have with probability at least 1 − η
[Equation]
A. P ROOF OF T HEOREM
Assumptions: As n → ∞ ,
[Equation]
With these assumptions, we have the following corollary to Theorem 1.
Corollary 1.
Deﬁne the matrices F and ˜ F as in Theorem 1 and let Assumptions A1 and A2 hold.
Then, for η >
[Equation]
(16) where the notation X
= O mη ( n − α ) stands for the fact that P
[Equation]
≤ η for some absolute constant C > 0 and non-negative integer m .
Theorem 2.
Let µ > 0 and suppose Σ p is a 1 2+ µ -sparse matrix.
For f a three-times continuously differentiable function and for η > 0 , we have for all (cid:15) ∈ (0 ,
[Equation]
Proof.
The proof needs the introduction of the following two lemmas, that can be found in (El Karoui, 2008, Lemma A.1 and A.2) and which are a consequence of the ε -sparsity notion
Lemma 3.
Given an ε -sparse p × p real symmetric matrix M and calling m = max ij | M ij | , we have, for all k ∈ 2 N
[Equation]
Lemma 4.
Given two real symmetric matrices M and N with | M ij | ≤ N ij .
Then, we have (cid:107) M (cid:107) op ≤ (cid:107) N (cid:107) op .
First, we show that when Σ p is ε -sparse, the Hadamard product f ( k ) (Σ p ) (cid:12) F ( k ) is of vanishing operator norm for k ≥ 1 , precisely
Lemma 5.
Let µ > 0 , suppose Σ p is a 1 2+ µ -sparse matrix.
For f a real and differentiable function, k ∈ { 1 , 2 } such that f ( k ) (0) = 0 and for η > 0 , we have for all (cid:15) ∈ (0 , k
(2+
[Equation] [Equation]
Proof.
We start by proving that the matrix F ( k ) has entries of order O ( n − k/ 2 ) .
In fact, we have by
Lemma 2, for all m ∈ N ∗
[Equation]
thus applying Markov’s inequality to the random variable | F ( k ) ij | 2 m yields to the following tail control.
[Equation]
where C is an absolute constant.
Recalling Assumption A1 and by the union bound, we have
[Equation]
which implies for η > 0 and for all m >
[Equation]
Besides, let M be the matrix deﬁned as M ≡ max ij | F ( k ) ij | · f ( k ) (Σ p ) , we have
[Equation]
thus, one has by Lemma
[Equation]
In particular, since f ( k ) (Σ p ) is 1 2+ µ -sparse (by Remark 2), we have by Lemma 3 and by equation 19, for some η >
[Equation]
choosing (cid:15) = 1 2 m < k
(2+ µ ) − 2 2(3+2 µ ) yields the ﬁnal result.
When considering f such that f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 , the result holds by Corollary 1 and Lemma 5.
In fact, the dominant order corresponds to k = 1 in Lemma 5.
Which completes the proof.
A.
F URTHER E XPERIMENTS
A.4.
H IGHER R ANK C ASE
In this section, we provide further experiments by considering a rank three case and by using the “Three Peak”, “Piece Poly” and “Step New” signals of Johnstone et al.
(Johnstone & Lu, 2009), in the “Symmlet 8” wavelet basis, as principal components.
We compare the estimated PCs by our method with the kernel function in equation 8 to the estimated ones through standard PCA and the
CT method (Deshpande & Montanari, 2016).
As shown in Figure 4, the proposed method retrieves consistently the principal components compared to a standard PCA.
In particular, we obtain results that are similar to the ones obtained by the CT method while generalizing it to the class of smooth functions with f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 .
[Figure]
CT 1 ,2 ,f -PCA (ours)
[Figure]
We use p = 2048 , n = 1024 and the spikes strengths are set respectively as ω 1 = 100 , ω 2 = 75 and ω 3 = 50 .
In particular, we note the similarity between the results obtained by our method and CT.
A.4. O THER CHOICES OF THE KERNEL FUNCTION f
In this section, we consider functions of the form f ( t ) =
αt
+ βt 2 + γt where α, β, γ ∈ R are some parameters to ﬁx in order to allow or not the conditions f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 .
In particular, we set different parameters choices for α, β and γ in order to validate these conditions.
Figure 5 depicts different PC recovery using the f -PCA method with the considered class of functions.
As we can observe from this ﬁgure, the “cleanest” signal recovery is obtained when α (cid:54) = 0 , β = 0 , γ =
( i.e. , when f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 ) thereby validating our theoretical conditions on the kernel function f for a consistent sparse PCA recovery.
Note that these conditions are necessary but not sufﬁcient in the sense that f has to be linear for large values of t (In particular, this is the case for the function f given by equation 8).
In fact, the outcome provided by f -PCA for f ( t ) =
αt 3 with α (cid:54) = 0 is not optimal as the obtained signal is deformed (due to the unveriﬁed linearity condition), compared to the GT one.
[Equation] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
3 + βt 2 + γt for different values of the parameters ( α, β, γ ) ∈ R 3 .
We consider the “Three Peak” example of (Johnstone & Lu, 2009) which is sparse in the “Symmlet 8” wavelet basis.
We use p = 2048 , n = 256 and ω 1 = 5 .
In particular, we notice that the “cleanest” signal is obtained when α (cid:54) = 0 , β = 0 , γ = 0 which validate our theoretical conditions f (cid:48) (0) =
f (cid:48)(cid:48) (0) = 0 .