G ENERATIVE P RINCIPAL C OMPONENT A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we study the problem of principal component analysis with generative modeling assumptions, adopting a general model for the observed matrix that encompasses notable special cases, including spiked matrix recovery and phase retrieval.
The key assumption is that the underlying signal lies near the range of an L -Lipschitz continuous generative model with bounded k -dimensional inputs.
We propose a quadratic estimator, and show that it enjoys a statistical rate of order (cid:113)
k log L m on the squared error, where m is the number of samples.
Moreover, we provide a variant of the classic power method, which projects the calculated data onto the range of the generative model during each iteration.
We show that under suitable conditions, this method converges linearly to a point achieving the abovementioned statistical rate.
This rate is conjectured in (Aubin et al., 2019; Cocola et al., 2020) to be the best possible even when we only restrict to the special case of spiked matrix models.
We perform experiments on various image datasets for spiked matrix and phase retrieval models, and illustrate performance gains of our method to the classic power method and the truncated power method devised for sparse principal component analysis.
I NTRODUCTION
Principal component analysis (PCA) is one of the most popular techniques for data processing and dimensionality reduction (Jolliffe, 1986), with an abundance of applications such as image recognition (Hancock et al., 1996), gene expression data analysis (Alter et al., 2000), and clustering (Ding & He, 2004; Liu & Tan, 2019).
PCA seeks to ﬁnd the directions that capture maximal variances in vector-valued data.
In detail, letting x 1 , x 2 , . . .
, x m be m realizations of a random vector x ∈ R n with a population covariance matrix ¯ Σ ∈ R n × n , PCA aims to reconstruct the top r principal eigenvectors of ¯ Σ .
The ﬁrst principal eigenvector can then be computed as follows:
[Equation]
where the empirical covariance matrix is deﬁned as Σ := 1 m (cid:80) mi
=1 ( x i − c )( x i − c ) T , with c :=m (cid:80) mi
=1 x i .
In addition, subsequent principal eigenvectors can be estimated by similar optimization problems subject to being orthogonal to the previous vectors.
PCA is consistent in the conventional setting where the dimension of the data n is relatively small compared to the sample size m (Anderson, 1962), but leads to rather poor estimates in the highdimensional setting where m (cid:28) n .
In particular, it has been shown in various papers that the empirical principal eigenvectors are no longer consistent estimates of their population counterparts (Nadler, 2008;
Johnstone & Lu, 2009; Jung & Marron, 2009; Birnbaum et al., 2013).
In order to tackle the curse of dimensionality, a natural approach is to impose certain structural constraints on the principal eigenvectors, e.g., chosen based on prior knowledge.
One of the most widely used assumptions is that the principal eigenvectors are sparse, and this gives rise to the problem of sparse principal component analysis (SPCA)
(Zou et al., 2006).
In particular, for recovering the top principal eigenvector, the optimization problem of SPCA is given by
[Equation]
where (cid:107) w (cid:107)
= |{ i : w i (cid:54) = 0 }| denotes the number of non-zero entries of w , and s ∈ N represents the sparsity level.
In addition to reducing the effective number of parameters, the sparsity assumption also enhances the interpretability (Zou et al., 2006).
Departing momentarily from the PCA problem, recent years have seen tremendous advances in deep generative models in a wide variety of real-world applications (Foster, 2019).
This has motivated a new perspective of the related problem of compressed sensing (CS), in which the standard sparsity assumption is replaced by a generative modeling assumption.
That is, the underlying signal is assumed to lie near the range of a (deep) generative model (Bora et al., 2017).
The authors of (Bora et al., 2017) characterized the number of samples required to attain an accurate recovery, and also presented numerical results on image datasets showing that compared to sparsity-based methods, generative priors can lead to large reductions (e.g., a factor of 5 to 10 ) in the number of measurements needed to recover the signal up to a given accuracy.
Additional numerical and theoretical results concerning inverse problems using generative models have been provided in (Van Veen et al., 2018; Dhar et al., 2018; Heckel & Hand, 2019; Jalal et al., 2020; Liu & Scarlett, 2020a; Ongie et al., 2020; Whang et al., 2020; Jalal et al., 2021; Nguyen et al., 2021), among others.
In this paper, following the developments in both PCA/SPCA and inverse problems with generative priors, we study the use of generative priors in principal component analysis (GPCA), which gives a generative counterpart of SPCA in (2), formulated as follows:
[Equation]
where G is a (pre-trained) generative model, which we assume has a range contained in the unit sphere of R n .
1 Similarly to SPCA, the motivation for this problem is to incorporate prior knowledge on the vector being recovered (or alternatively, a prior preference), and to permit meaningful recovery and theoretical bounds even in the high-dimensional regime m (cid:28) n .
1. R ELATED W ORK
In this subsection, we summarize some relevant works, which can roughly be divided into (i) the
SPCA problem, and (ii) signal recovery with generative models.
SPCA : It has been proved that the SPCA problem in (2) attains the optimal statistical rate (cid:112) s log n/m (Vu & Lei, 2012), where m is the number of samples, n is the ambient dimension, and s is the sparsity level of the ﬁrst principal eigenvector.
However, due to the combinatorial constraint, the computation of (2) is intractable.
To address this computational issue, in recent years, a huge body of practical approaches for estimating sparse principal eigenvectors have been proposed in the literature, including (d’Aspremont et al., 2007; Vu et al., 2013; Chang et al., 2016; Moghaddam et al., 2006; d’Aspremont et al., 2008; Jolliffe et al., 2003; Zou et al., 2006; Shen & Huang, 2008; Journ ´ ee et al., 2010; Hein & B ¨ uhler, 2010; Kuleshov, 2013; Yuan & Zhang, 2013; Asteris et al., 2011; Papailiopoulos et al., 2013), just to name a few.
Notably, statistical guarantees for several approaches have been provided.
The authors of (Yuan & Zhang, 2013) propose the truncated power method (TPower), which adds a truncation operation to the power method to ensure the desired level of sparsity.
It is shown that this approach attains the optimal statistical rate under appropriate initialization.
While most approaches for SPCA only focus on estimating the ﬁrst principal eigenvector, with a certain deﬂation method (Mackey, 2008) being leveraged to reconstruct the rest, an iterative thresholding approach is proposed in (Ma, 2013), and is shown to attain a near-optimal statistical rate when estimating multiple individual principal eigenvectors.
In addition, the authors of (Cai et al., 2013) propose a regression-type method that gives an optimal principal subspace estimator.
Both works (Ma, 2013; Cai et al., 2013) rely on the assumption of a spiked covariance model to ensure a good initial vector.
To avoid the spiked covariance model assumption, the work (Wang et al., 2014) proposes a two-stage procedure that attains the optimal subspace estimator in polynomial time.
Signal recovery with generative models : Since the seminal work (Bora et al., 2017), there has been a substantial volume of papers studying various inverse problems with generative priors.
One of the problems more closely related to PCA is spectral initialization in phase retrieval, which amounts to solving an eigenvalue problem.
Phase retrieval with generative priors has been studied in (Hand et al., 2018;
Hyder et al., 2019; Jagatap & Hegde, 2019;
Wei et al., 2019; Shamshad & Ahmed,
; Aubin et al., 2020; Liu et al., 2021b).
In particular, the work (Hand et al., 2018) models the underlying signal as being in the range of a fully-connected ReLU neural network with no offsets, and all the weight matrices of the ReLU neural network are assumed to have i.i.d.
zeromean Gaussian entries.
In addition, the neural network needs to be sufﬁciently expansive in the sense that n i ≥ Ω( n i − 1 log n i − 1 ) , where n i is the width of the i -th layer.
Under these assumptions, the authors establish favorable global optimization landscapes for the corresponding objective, and derive near-optimal sample complexity upper bound.
They minimize the objective function directly over the latent space in R k using gradient descent, which may suffer from local minima in general optimization landscapes (Hyder et al., 2019; Shah & Hegde, 2018).
In (Aubin et al., 2020), the assumptions on the neural network are similar to those in (Hand et al., 2018), relaxing to general activation functions (beyond ReLU) and n i ≥ Ω( n i − 1 ) .
The authors focus on the high dimensional regime where n, m, k → ∞ with the ratio m/n being ﬁxed, and assume that the input vector in R k is drawn from a separable distribution.
They derive sharp asymptotics for the information-theoretically optimal performance and for the associated approximate message passing (AMP) algorithm.
Both works (Hand et al., 2018; Aubin et al., 2020) focus on noiseless phase retrieval.
When only making the much milder assumption that the generative model is Lipschitz continuous, with no assumption on expansiveness, Gaussianity, and offsets, similarly to sparse phase retrieval, a spectral initialization step is typically required in order to accurately reconstruct the signal (Netrapalli et al., 2015; Cand ` es et al., 2015).
The authors of (Liu et al., 2021b) propose an optimization problem similar to (3) for the spectral initialization for phase retrieval with generative models.
It was left open in (Liu et al., 2021b) how to solve (or approximate sufﬁciently accurately) the optimization problem in practice.
Understanding the eigenvalues of spiked random matrix models has been a central problem of random matrix theory, and spiked matrices have been widely used in the statistical analysis of SPCA.
Recently, theoretical guarantees concerning spiked matrix models with generative priors have been provided in (Aubin et al., 2019; Cocola et al., 2020).
In particular, in (Aubin et al., 2019), the assumptions are similar to those in (Aubin et al., 2020), except that the neural network is assumed to have exactly one hidden layer.
The Bayes-optimal performance is analyzed, and it is shown that the
AMP algorithm can attain this optimal performance.
In addition, the authors of (Aubin et al., 2019) also propose the linearized approximate message passing (LAMP) algorithm, which is a spectral algorithm speciﬁcally designed for a single-layer feedforward neural network with no bias term.
The authors show its superiority to classical PCA via numerical results on the Fashion-MNIST dataset.
In (Cocola et al., 2020), the same assumptions are made as those in (Hand et al., 2018) on the neural network, and the authors demonstrate the benign global geometry for a nonlinear least squares objective.
Similarly to (Hand et al., 2018), the objective is minimized over R k using a gradient descent algorithm, which can get stuck in local minima for general global geometries.
Compared to the above discussed works that use generative models, we make no assumption on expansiveness, Gaussianity, and offsets for the generative model, and we consider a data model that simultaneously encompasses both spiked matrix and phase retrieval models, among others.
1. C ONTRIBUTIONS
The main contributions of this paper are threefold:
• We study eigenvalue problems with generative priors (including GPCA), and characterize the statistical rate of a quadratic estimator similar to (3) under suitable assumptions.
• We propose a variant of the classic power method, which uses an additional projection operation to ensure that the output of each iteration lies in the range of a generative model.
We refer to our method as projected power method (PPower).
We further show that under appropriate conditions (most notably, assuming exact projections are possible), PPower obtains a solution achieving a statistical rate that is conjectured to be optimal in (Aubin et al., 2019; Cocola et al., 2020) for spiked matrix models.
• For spiked matrix and phase retrieval models, we perform numerical experiments on image datasets, and demonstrate that when the number of samples is relatively small compared to the ambient dimension, PPower leads to signiﬁcantly better performance compared to the classic power method and TPower.
1. N OTATION
We use upper and lower case boldface letters to denote matrices and vectors respectively.
We write [ N ] = { 1 , 2 , · · · , N } for a positive integer N , and we use I N to denote the identity matrix in R N × N .
A generative model is a function G : D → R n , with latent dimension k , ambient dimension n , and input domain D ⊆ R k .
We focus on the setting where k (cid:28) n .
For a set S ⊆ R k and a generative model G : R k
→ R n , we write G ( S ) =
{ G ( z ) : z ∈ S } .
We use (cid:107) X (cid:107) 2 → 2 to denote the spectral norm of a matrix X .
We deﬁne the (cid:96) q -ball B kq ( r ) :=
{ z ∈ R k : (cid:107) z (cid:107) q ≤ r } for q ∈ [0 , + ∞ ] .
S n − 1 := { x ∈ R n : (cid:107) x (cid:107) 2 = 1 } represents the unit sphere in R n .
The symbols C, C (cid:48) , C (cid:48)(cid:48) are absolute constants whose values may differ from line to line.
P ROBLEM S ETUP
In this section, we formally introduce the problem, and overview some important assumptions that we adopt.
Except where stated otherwise, we will focus on the following setting:
• We have a matrix V ∈ R n × n satisfying
[Equation]
where E is a perturbation matrix and ¯ V is assumed to be positive semideﬁnite (PSD).
For PCA and its constrained variants, V and ¯ V can be thought of as the empirical and population covariance matrices respectively.
• We have an L -Lipschitz continuous generative model G : B k 2 ( r ) → R n .
For convenience, similarly to that in (Liu et al., 2020), we assume that Range( G ) ⊆
S n − 1 .
Remark
For a general (unnormalized) L -Lipschitz
continuous generative model G , we can instead consider a corresponding normalized generative model ˜ G : D → S n − 1 as in (Liu et al., 2021b), where D := { z ∈ B k 2 ( r ) :
(cid:107) z (cid:107) 2 > R min } for some R min > 0 .
Then, the
Lipschitz constant of ˜ G becomes L/R min .
For a d -layer neural network, we typically have
L = n Θ( d ) (Bora et al., 2017).
Thus, we can set R min to be as small as 1 /n Θ( d ) without changing the scaling laws, which makes the dependence on R min very mild.
• We aim to solve the following eigenvalue problem with a generative prior:
[Equation]
Note that since Range( G ) ⊆ S n − 1 , we do not need to impose the constraint (cid:107) w (cid:107)
[Equation]
Since V is not restricted to being an empirical covariance matrix, (5) is more general than
GPCA in (3).
However, we slightly abuse terminology and also refer to (5) as GPCA.
• To approximately solve (5), we perform a projected power method (PPower), which gives the following iterative procedure:
[Equation]
where P G ( · ) is the projection function onto G ( B k 2 ( r )) , 4 and the initialization vector w (0) may be chosen either manually or randomly, e.g., uniform over S n − 1 .
Often the initialization vector w (0) plays an important role and we may need a careful design of it.
For example, for phase retrieval with generative models, as mentioned in (Liu et al., 2021b, Section V), we may choose the column corresponding to the largest diagonal entry of V as the starting point.
See also (Yuan & Zhang, 2013, Section 3) for a discussion on the initialization strategy for TPower devised for SPCA.
We present the algorithm corresponding to (6) in Algorithm
Input : V , number of iterations T , pre-trained generative model G , initial vector w (0)
Procedure: Iterate w ( t +1) =
P G (cid:0) Vw ( t ) (cid:1) for t ∈ { 0 , 1 , . .
, T − 1 } , and return w ( T )
Remark 2.
To tackle generalized eigenvalue problems encountered in some speciﬁc applications, there are variants of the projected power method, which combine a certain power iteration with additional operations to ensure sparsity or enforce other constraints.
These applications include but not limited to sparse PCA (Journ´ee et al., 2010; Yuan & Zhang, 2013), phase synchronization (Boumal, 2016;
Liu et al., 2017), the hidden clique problem (Deshpande & Montanari, 2015), the joint alignment problem (Chen & Cand`es, 2018), and coneconstrained PCA (Deshpande et al., 2014; Yi & Neykov, 2020).
For example, under the simple spiked Wigner model (Perry et al., 2018) for the observed data matrix V with the underlying signal being assumed to lie in a convex cone, the authors of (Deshpande et al., 2014) show that cone-constrained PCA can be computed efﬁciently via a generalized projected power method.
In general, the range of a Lipschitz-continuous generative model is non-convex and not a cone.
In addition, we consider a matrix model that is more general than the spiked Wigner model.
Under the assumption that V is PSD, we have the following useful lemma.
The PSD assumption will be satisﬁed, for example, when E = 0 , or when V is a sample covariance matrix.
Lemma 1.
For any x ∈ R n , let Q ( x ) = x T Vx .
Then, if V is PSD, the sequence { Q ( w ( t ) ) } t> 0 for w ( t ) in (6) is monotonically non-decreasing.
Proof.
Since w ( t +1) =
P G (cid:0) Vw ( t ) (cid:1) and w ( t ) ∈
Range( G ) , we have
[Equation]
[Equation] and since (cid:107) w ( t +1) (cid:107) 2 = (cid:107) w ( t ) (cid:107) 2 = 1 , expanding the square gives
[Equation] [Equation]
Then, we obtain
[Equation]
where (11) follows from the assumption that V is PSD, and (12) follows from (8).
S PECIALIZED D ATA M ODELS AND E XAMPLES
In this section, we make more speciﬁc assumptions on V = ¯ V + E , starting with the following.
Assumption 1 (Assumption on ¯ V ) .
Assume that ¯ V is PSD with eigenvalues ¯ λ 1 > ¯ λ 2 ≥ . . .
≥ ¯ λ n ≥ 0 .
We use ¯ x (a unit vector) to represent the eigenvector of ¯ V that corresponds to ¯ λ 1 .
In the following, it is useful to think of ¯ x is being close to the range of the generative model G .
In the special case of (3), letting m be the number of samples, it is natural to derive that the upper bound of (cid:107) E (cid:107) 2 → 2 grows linearly in ( n/m ) b for some positive constant b such as 12 or 1 (with high probability; see, e.g., (Vershynin, 2010, Corollary 5.35)).
In the following, we consider general scenarios with
V depending on m samples (see below for speciﬁc examples).
Similarly to (Yuan & Zhang, 2013), we may consider a restricted version of (cid:107) E (cid:107) 2 → 2 , leading to the following.
Assumption 2 (Assumption on E ) .
Let S 1 , S 2 be two (arbitrary) ﬁnite sets in R n satisfying m = Ω(log( | S 1 | · | S 2 | )) .
Then, we have for all s 1 ∈ S 1 and s 2 ∈ S 2 that
[Equation]
where C is an absolute constant.
In addition, we have (cid:107) E (cid:107) 2 → 2 = O ( n/m ) .
The following examples show that when the number of measurements is sufﬁciently large, the data matrices corresponding to certain spiked matrix and phase retrieval models satisfy the above assumptions with high probability.
Short proofs are given in Appendix A for completeness.
Example 1 (Spiked covariance model) .
In the spiked covariance model (Johnstone & Lu, 2009;
Deshpande & Montanari, 2016), the observed vectors x 1 , x 2 , . .
x m ∈ R n are of the form
[Equation]
where s 1 , . .
, s r ∈ R n are orthonormal vectors that we want to estimate, while z i ∼ N ( 0 , I n ) and u q,i ∼ N (0 , 1) are independent and identically distributed.
In addition, β 1 , . . .
, β r are positive constants that dictate the signal-to-noise ratio (SNR).
To simplify the exposition, we focus on the rank-one case and drop the subscript q ∈ [ r ] .
Let
[Equation]
and ¯ V = E
[ V ] =
β ss T .
Then, ¯ V satisﬁes Assumption 1 with ¯ λ 1 = β > 0 , ¯ λ 2 = . . .
=
¯ λ n = 0 , and ¯ x = s .
In addition, let E =
V − ¯ V .
From the Bernstein-type inequality (Vershynin, 2010, Proposition 5.10) for the sum of sub-exponential random variables, we observe that for any ﬁnite sets S 1 , S 2 ⊂ R n , when m = Ω (cid:0) log( | S 1 | · | S 2 | )) , with probability 1 − e − Ω(log( | S 1 |·| S 2 | )) ,
E satisﬁes (13) in Assumption 2.
Moreover, standard concentration arguments give (cid:107) E (cid:107) 2 → 2 =
O ( n/m ) with probability 1 − e − Ω( n ) .
Remark 3.
We can also consider the simpler spiked Wigner model (Perry et al., 2018; Chung &
Lee, 2019)
where
V = β ss T + 1 √ n H , with the signal s being a unit vector, β > 0 being an SNR parameter, and H ∈ R n × n being a symmetric matrix with entries drawn i.i.d.
(up to symmetry) from
N (0 , 1) .
In this case, when m = n is sufﬁciently large, with high probability, ¯ V :=
E
[ V ] =
β ss T and E :=
V − ¯ V similarly satisfy Assumptions 1 and 2 respectively.
Example 2 (Phase retrieval) .
Let A ∈ R m × n be a matrix having i.i.d.
N (0 , 1) entries, and let a Ti be the i -th row of A .
For some unit vector s , suppose that the observed vector is y =
| As | , where the absolute value is applied element-wise.
We construct the weighted empirical covariance matrix (Zhang et al., 2017; Liu et al., 2021b)
[Equation]
16) where u > l > 1 are positive constants, and for g ∼ N (0 , 1) , γ := E (cid:2) | g | 1 { l< | g | <u } (cid:3) .
Let ¯ V =
E
[ V ] =
β ss T , where β := E (cid:2)(cid:0) | g | 3 − | g | (cid:1) 1 { l< | g | <u } (cid:3) .
Then, ¯ V satisﬁes Assumption 1 with ¯ λ 1 = β > 0 , ¯ λ 2 = . . .
=
¯ λ n = 0 , and ¯ x = s .
In addition, letting E = V − ¯ V , we have similarly to
Example 1 that E satisﬁes Assumption 2 with high probability.
M AIN R ESULTS
The following theorem concerns globally optimal solutions of (5).
The proof is given in Appendix B.
Theorem 1.
Let V = ¯ V + E with Assumptions 1 and 2 being satisﬁed by ¯ V and E respectively, and let x G :=
P G (¯ x ) =
arg min w ∈ Range( G ) (cid:107) w −
¯
x
(cid:107) 2 .
Suppose that ˆ v is a globally optimal solution 5 For the spectral norm of E , one often expects an even tighter bound O ( (cid:112) n/m ) , but we use O ( n/m ) to {+simplify the analysis of our examples. Moreover, at least under the typical scaling where L is polynomial in n (Bora et al., 2017), the upper bound for (cid:107) E (cid:107) 
 → 2 can be easily relaxed to O (cid:0) ( n/m ) b (cid:1) for any positive constant b , without affecting the scaling of our derived statistical rate. To avoid non-essential complications, β is typically assumed to be known (Johnstone & Lu, 2009).+}
to (5) .
Then, for any δ ∈ (0 , 1) , we have
[Equation]
We note that since both ¯ x and − ¯ x are leading eigenvectors of ¯ V , without further assumptions, a quadratic estimator of the form (5) can only recover the signal up to a global sign ﬂip.
From
Theorem 1, we observe that when there is no representation error, i.e., ¯ x ∈ Range( G ) , under the scaling ¯ λ 1 − ¯ λ 2 = Θ(1) , L = n Ω(1) , and δ =
O ( √ m/n ) , the solution ˆ v to the quadratic estimator in (5) satisﬁes
[Equation]
Before providing our main theorem for PPower described in (6), we present the following important lemma, whose proof is presented in Appendix C. To simplify the statement of the lemma, we ﬁx δ to be O (1 /n ) , though a more general form analogous to Theorem 1 is also possible.
Lemma 2.
Let V =
¯ V + E with Assumptions 1 and 2 being satisﬁed by ¯ V and E respectively, and further assume that ¯ x ∈ Range( G ) .
Let ¯ γ =
¯
λ 2 / ¯ λ 1 with ¯ λ 1 = Θ(1) .
Then, for all s ∈ Range( G ) satisfying s T ¯ x > 0 , let ˆ s =
P G ( Vs ) , and we have
[Equation]
Remark 4.
The assumption s T ¯ x > 0 will be generally satisﬁed when the range of G only contains nonnegative vectors.
Such a condition holds for a neural network with the popular ReLU or Sigmoid activation functions.
In addition, as mentioned in some works studying nonnegative SPCA (Zass &
Shashua, 2007; Sigg & Buhmann, 2008; Asteris et al., 2014), for several practical ﬁelds such as economics, bioinformatics, and computer vision, it is natural to assume that the underlying signal has no negative entries.
More generally, the assumption s T ¯ x > 0 can be removed if we additionally have that − ¯ x is also contained in the range of G .
For this case, when s T ¯ x < 0 , we can instead derive an upper bound for (cid:107) ˆ s + ¯ x (cid:107) 2 .
Based on the above lemma, we have the following theorem, whose proof is given in Appendix D.
Theorem 2.
Suppose that the assumptions on the data model V =
¯ V + E are the same as those in
Lemma 2, and assume that there exists a t 0 ∈ N such that ¯ x T w ( t 0 ) ≥ 2¯ γ + ν , where ¯ γ = ¯ λ 2 / ¯ λand ν is a sufﬁciently small positive constant.
Let µ 0 = 2¯ γ/ (cid:0) ¯ x T w ( t 0 ) (cid:1) < 1 .
Finally, suppose that m = Ω( k log( nLr )) with a large enough implied constant.
Then, for any t ≥ t 0 , we have either
[Equation]
or
[Equation]
where C is an absolute constant, and ξ is strictly positive.
If L = n Ω(1) , then (20) reveals that we have established conditions under which PPower in (6) converges linearly to a point achieving the statistical rate of order (cid:113)
k log L m .
Based on the minimax rates for SPCA (Vu & Lei, 2012;
Birnbaum et al., 2013) and the information-theoretic lower bounds for CS with generative models (Liu & Scarlett, 2020b; Kamath et al., 2020), the optimal rate for
GPCA is naturally conjectured to be of the same order (cid:113) k log L m .
We highlight that Theorempartially closes the computational-to-statistical gap (e.g., see (Wang et al., 2016; Hand et al., 2018;
Aubin et al., 2019; Cocola et al., 2020)) for spiked matrix recovery and phase retrieval under a generative prior, though closing it completely would require efﬁciently ﬁnding a good initialization and addressing the assumption of exact projections.
Regarding the assumption ¯ x T w ( t 0 ) ≥ 2¯ γ + ν , we note that if t 0 = 1 , this condition can be viewed as having a good initialization.
For both Examples 1 and 2, we have ¯ γ = 0 .
Thus, for the spiked covariance and phase retrieval models corresponding to these examples, the assumption ¯ x T w ( t 0 ) ≥ 2¯ γ
+ ν reduces to ¯ x T w ( t 0 ) ≥
ν for a sufﬁciently small positive constant ν , which results in a mild assumption.
Such an assumption is also required for the projected power method devised for coneconstrained PCA under the simple spiked Wigner model, with the underlying signal being assumed to lie in a convex cone; see (Deshpande et al., 2014, Theorem 3). However, despite using a similar assumption on the initialization, our proof techniques are signiﬁcantly different from Deshpande et al. ; see Appendix E for discussion.
Perhaps the main caveat to Theorem 2 is that it assumes the projection step can be performed exactly.
However, this is a standard assumption in analyses of projected gradient methods, e.g., see (Shah &
Hegde, 2018), and both gradient-based projection and GAN-based projection have been shown to be highly effective in practice (Shah & Hegde, 2018; Raj et al., 2019).
E XPERIMENTS
In this section, we experimentally study the performance of Algorithm 1 ( PPower ).
We note that these experiments are intended as a simple proof of concept rather than seeking to be comprehensive, as our contributions are primarily theoretical.
We compare with the truncated power method ( TPower ) devised for SPCA proposed in (Yuan & Zhang, 2013, Algorithm 1) and the vanilla power method ( Power ) that performs the iterative procedure w ( t +1) =
( Vw ( t ) ) / (cid:107) Vw ( t ) (cid:107) 2 .
For a fair comparison, for PPower , TPower , and Power , we use the same initial vector, i.e., as mentioned in (Liu et al., 2021b, Section V), we choose the initialization vector w (0) as the column of V that corresponds to its largest diagonal entry.
For all these three algorithms, the total number of iterations
T is set to be 30 .
To compare the performance across algorithms, we use the scale-invariant Cosine
Similarity metric deﬁned as Cossim (cid:0) ¯ x , w ( T ) (cid:1) :
= (cid:104) ¯ x , w ( T ) (cid:105) (cid:107) ¯ x (cid:107) 2 (cid:107) w ( T ) (cid:107) 2 ,
where ¯ x is the ground-truth signal to estimate, and w ( T ) denotes the output vector of the algorithm.
The experiments are performed on the MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017) datasets, with the numerical results for the Fashion-MNIST dataset being presented in
Appendix F. The MNIST dataset consists of 60 , 000 images of handwritten digits.
The size of each image in the MNIST dataset is 28 × 28 , and thus n = 784 .
To reduce the impact of local minima, we perform 10 random restarts, and choose the best among these.
The cosine similarity is averaged over the test images, and also over these 10 random restarts.
The generative model G is set to be a pre-trained variational autoencoder (VAE) model with latent dimension k = 20 .
We use the VAE model trained by the authors of (Bora et al., 2017) directly, for which the encoder and decoder are both fully connected neural networks with two hidden layers, with the architecture being 20 − 500 − 500 − 784 .
The VAE is trained by the Adam optimizer with a mini-batch size of 100 and a learning rate of 0 .
The projection step P G ( · ) is solved by the Adam optimizer with a learning rate of 0 . 03 and 200 steps.
In each iteration of TPower , the calculated entries are truncate to zero except for the largest q entries, where q ∈ N is a tuning parameter.
Since for TPower , q is usually selected as an integer larger than the true sparsity level, and it is unlikely that the image of the MNIST dataset can be well approximated by a k -sparse vector with k = 20 , we choose a relatively large q , namely q = 150 .
Similarly to (Bora et al., 2017) and other related works, we only report the results on a test set that is unseen by the pre-trained VAE model, i.e., the training of G and the PPower computations do not use common data.
All experiments are run using Python 3.6 and Tensorﬂow 1.5.0, with a NVIDIA
GeForce GTX {+
+} Ti 11GB GPU.
Spiked covariance model (Example 1) :
The numerical results are shown in Figures 1 and 2.
We observe from Figure 1 that Power and TPower attain poor reconstructions, and the generative prior based method PPower attains signiﬁcantly better reconstructions.
To illustrate the effect of the sample size m , we ﬁx the SNR parameter β = 1 and vary m in { 100 , 200 , 300 , 400 , 500 } .
In addition, to illustrate the effect of the SNR parameter β , we ﬁx m = 300 , and vary β in { 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 , 2 , 3 , 4 } .
From Figure 2, we observe that for these settings of m and β , PPower always leads to a much higher cosine similarity compared to Power and TPower , which is natural given the more precise modeling assumptions used.
[Figure]
r i g i n a [Figure] P o w [Figure]
r T P o w
[Figure]
[Figure]
Phase retrieval (Example 2) : The results are shown in Figure 2 (Right) and Figure 3. Again, we can observe that PPower signiﬁcantly outperforms Power and TPower .
In particular, for sparse phase retrieval, when performing experiments on image datasets, even for the noiseless setting, solving an eigenvalue problem similar to (5) can typically only serve as a spectral initialization step, with a subsequent iterative algorithm being required to reﬁne the initial guess.
In view of this, it is notable that for phase retrieval with generative priors, PPower can return meaningful reconstructed images for m = 200 , which is small compared to n = 784 .
C ONCLUSION
We have proposed a quadratic estimator for eigenvalue problems with generative models, and we showed that this estimator attains a statistical rate of order (cid:113)
k log L m on the squared error.
We provided a projected power method to efﬁciently solve (modulo the complexity of the projection step) the corresponding optimization problem, and showed that our method converges linearly to a point achieving a statistical rate of order (cid:113)
k log L m under suitable conditions.
O r i g i n a l
P o w e r
T P o w e r
PP o w e r
[Figure]
g
[Figure]
A PPENDIX FOR G ENERATIVE P RINCIPAL C OMPONENT A NALYSIS A P ROOFS FOR S PIKED M ATRIX AND P HASE R ETRIEVAL E XAMPLES
Before proceeding, we present the following standard deﬁnitions.
Deﬁnition 1.
A random variable X is said to be sub-Gaussian if there exists a positive constant C such that ( E [ | X | p ]) 1 /p ≤ C √ p for all p ≥ 1 .
The sub-Gaussian norm of a sub-Gaussian random variable X is deﬁned as (cid:107) X (cid:107) ψ 2 :=
sup p ≥ 1 p − 1 / 2 ( E [ | X | p ]) 1 /p .
Deﬁnition 2.
A random variable X is said to be sub-exponential if there exists a positive constant C such that ( E [ | X | p ]) 1 p ≤ Cp for all p ≥ 1 .
The sub-exponential norm of X is deﬁned as (cid:107) X (cid:107) ψ 1 :=
sup p ≥ 1 p − 1 ( E [ | X | p ]) 1 p .
The following lemma states that the product of two sub-Gaussian random variables is subexponential, regardless of the dependence between them.
Lemma 3.
(Vershynin, 2018, Lemma 2.7.7) Let X and Y be sub-Gaussian random variables (not necessarily independent).
Then XY is sub-exponential, and satisﬁes
[Equation]
The following lemma provides a useful concentration inequality for the sum of independent subexponential random variables.
Lemma 4.
(Vershynin, 2010, Proposition 5.16)
Let X 1 , . . .
X N be independent zero-mean subexponential random variables, and K = max i (cid:107)
X
i (cid:107) ψ 1 .
Then for every α =
[ α 1 , . . .
, α N ] T ∈ R N and (cid:15) ≥ 0 , it holds that
[Equation]
The widely-used notion of an (cid:15) -net is introduced as follows.
Deﬁnition 3.
Let ( X , d ) be a metric space, and ﬁx (cid:15) > 0 .
A subset S ⊆
X is said be an (cid:15) -net of X if, for all x ∈ X , there exists some s ∈ S such that d ( s, x ) ≤ (cid:15) .
The minimal cardinality of an (cid:15) -net of X , if ﬁnite, is denoted C ( X , (cid:15) ) and is called the covering number of X (at scale (cid:15) ).
The following lemma provides a useful upper bound for the covering number of a unit sphere.
Lemma 5.
(Vershynin, 2010, Lemma 5.2)
The unit Euclidean sphere S N − 1 equipped with the
Euclidean metric satisﬁes for every (cid:15) >
that
[Equation]
The following lemma provides an upper bound for the spectral norm of a symmetric matrix.
Lemma 6.
(Vershynin, 2010, Lemma 5.4) Let X be a symmetric N × N matrix, and let C (cid:15) be an (cid:15) -net of S
N − 1 for some (cid:15) ∈ [0 , 1 / 2) .
Then,
[Equation] [Equation]
With the above auxiliary results in place, we provide the relevant proofs for our two example introduced in Section 3.
A. P ROOF FOR THE S PIKED C OVARIANCE M ODEL (E XAMPLE 1)
For r = 1 , we have x i = √
βu
i s + z
i and a direct calculation gives E [ V ] =
¯ V = β ss T .
For any s 1 ∈ S 1 , we have that s T 1 x i = √ βu
i s T s
+ z Ti s 1 is sub-Gaussian, with the sub-Gaussian norm being upper bounded by C ( √ β + 1) (cid:107) s 1 (cid:107) 2 .
Similarly, we have that for any s 2 ∈ S 2 , (cid:107) s T 2 x i (cid:107) ψ 2 ≤
C
[Equation]
√ β + 1)
[Equation]
s
[Equation]
Applying Lemma 3, we deduce that ( s T 1 x i )( s T 2 x i ) is sub-exponential, with the sub-exponential norm being upper bounded by C 2 ( √ β + 1) 2 (cid:107) s 1 (cid:107) 2 (cid:107) s 2 (cid:107) 2 .
In addition, we have
[Equation]
T
[Equation] [Equation]
where E [( x Ti s 1 )( x Ti s 2 )]
= ( s T 1 s 2 ) + β ( s T s 1 )( s T s 2 ) .
Then, from Lemma 4, we obtain that for any t > 0 satisfying m = Ω( t ) , with probability 1 − e − Ω( t ) ,
[Equation]
Taking a union bound over all s 1 ∈ S 1 and s 2 ∈ S 2 , and setting t = log( | S 1 | · | S 2 | ) , we obtain with probability
[Equation]
·| S 2 | )) that (13) holds (with β being a ﬁxed positive constant).
Moreover, from Lemma 4, we also obtain that for any t > 0 satisfying t
= Ω( m ) , and any r ∈ S n − 1 , with probability 1 − e − Ω( t ) ,
[Equation] [Equation]
From Lemma 5, there exists an (1 / 4) -net
C 14 of S n − 1 satisfying log (cid:12)(cid:12) C 14 (cid:12)(cid:12) ≤ n log 9 .
Taking a union bound over all r ∈ C 14 , and setting t =
Cn , we obtain with probability 1 − e − Ω( n ) that
[Equation]
Then, from Lemma 6, we have
[Equation]
A. P ROOF FOR P HASE R ETRIEVAL (E XAMPLE 2)
Let W = 1 m (cid:80) mi
=
y i a i
a Ti 1 { l<y i <u } and ¯ W = β
ss T + γ I n .
It is shown in (Liu et al., 2021b,
Lemma 8) that
[Equation]
which implies
[Equation] [Equation]
Then, for any s 1 ∈ S 1 and s 2 ∈ S 2 , we have
[Equation]
We observe that y i ( a Ti s 1 )( a Ti s 2 ) 1 { l<y i <u } is sub-exponential with the sub-exponential norm being upper bounded by Cu (cid:107) s 1 (cid:107) 2 (cid:107) s 2 (cid:107) 2 .
In addition, from (32), we have E [ y i ( a Ti s 1 )( a Ti s 2 ) 1 { l<y
i <u } ]
=
β ( s T s 1 )( s T s 2 )+ γ ( s T 1 s 2 ) .
Then, from Lemma 4, we obtain that for any t > 0 satisfying m = Ω( t ) , with probability 1 − e − Ω( t ) ,
[Equation]
[Equation]
Taking a union bound over all s 1 ∈ S 1 and s 2 ∈ S 2 , and setting t = log( | S 1 | · | S 2 | ) , we obtain that with probability 1 − e − Ω(log( | S 1 |·| S 2 | )) , (13) holds as desired (with u being a ﬁxed positive constant).
In addition, similarly to (31), we have with probability 1 − e − Ω( n ) that
[Equation]
B P ROOF OF T HEOREM 1 (G UARANTEE ON THE G LOBAL O PTIMUM )
Let the singular value decomposition (SVD) of ¯
V be
[Equation]
where ¯ D = Diag([¯ λ 1 , . .
, ¯ λ n ]) , and ¯ U ∈ R n is an orthogonal matrix with the ﬁrst column being ¯ x .
For i > 1 , let the i -th column of ¯ U be ¯
u i .
Let M be a ( δ/L ) -net of B k 2 ( r ) ; from Lemma 5, we know that there exists such a net with
[Equation]
[Equation]
By the L -Lipschitz continuity of G , we have that G ( M ) is a δ -net of Range( G )
=
G ( B k 2 ( r )) .
We write
[Equation]
where ˜ x ∈ G ( M ) satisﬁes (cid:107) ˆ v − ˜ x (cid:107) 2 ≤ δ .
Then, we have
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
where (44) follows from Assumption 2 and (39)
[Equation]
and we use (cid:0) ¯ x T ˆ v (cid:1) 2 + (cid:80) i>
[Equation]
In addition
[Equation]
from (5) and x G ∈ Range( G )
[Equation]
we have
[Equation] [Equation]
[Equation]
where (52) expands the brackets and uses the fact that (¯ λ 1 , ¯ x ) is an eigenvalue-eigenvector pair, and (53) uses (cid:107) ¯ x (cid:107) 2 = 1 and the PSD property of ¯ V .
Since { x G , G ( M ) } is also a δ -net of Range( G ) , in (53), we also have (cid:12)(cid:12) x
TG Ex G (cid:12)(cid:12) ≤
O (cid:0)(cid:113) k log Lrδ m (cid:1) .
Combining this with (48) and (53), we obtain for some absolute constants C, C (cid:48)
that
[Equation]
which simpliﬁes to
[Equation]
Hence, we obtain
[Equation]
Without loss of generality
[Equation]
we assume ¯ x T ˆ v ≥
(otherwise
[Equation]
we can similarly provide an upper bound for (cid:107) ¯ x + ˆ v (cid:107) 22 )
[Equation]
which yields
[Equation]
where (59) uses ¯ x T ˆ v ∈ [0 , 1] , (60) follows from (57), and (61) follows from the assumption (cid:107) E (cid:107) 2 → 2 = O ( n/m ) .
C P ROOF OF L EMMA 2 (A UXILIARY R ESULT FOR PP OWER A NALYSIS )
By the assumption Range( G ) ⊆
S n − 1 , for any x ∈ R n and a > 0 , we have
[Equation]
Let ¯ η = 1 / ¯ λ 1 > 0 .
Then, we have ˆ s =
P G ( Vs ) =
P G (¯ η Vs ) .
Since ¯ x ∈ Range( G ) , we have
[Equation]
This is equivalent to
[Equation] [Equation]
which gives
[Equation]
Note also that from ¯ V ¯ x = ¯ λ
1 ¯ x , we obtain ¯ x = ¯
η ¯ V ¯ x , which we will use throughout the proof.
For δ > 0 , let M be a ( δ/L ) -net of B k 2 ( r ) ; from Lemma 5, there exists such a net with
[Equation]
By the L -Lipschitz continuity of G , we have that G ( M ) is a δ -net of Range( G )
=
G ( B k 2 ( r )) .
We write
[Equation]
where ˜ s ∈ G ( M ) satisﬁes (cid:107) ˆ v − ˜ x (cid:107) 2 ≤ δ and s 0 ∈ G ( M ) satisﬁes (cid:107) s − s 0 (cid:107) 2 ≤ δ .
Then, we have
[Equation]
where (68) follows from V
= ¯ V + E and ¯ x =
¯
η ¯ V ¯ x .
In the following, we control the two terms in (68) separately.
The term (cid:104) ¯ η ¯ V
( s − ¯ x ) , ˆ s − ¯ x (cid:105) : Let s =
α ¯ x + β t and ˆ s =
ˆ α ¯ x + ˆ β ˆ t , where (cid:107) t (cid:107)
= (cid:107)
ˆ t (cid:107) 2 =and t T ¯ x = ˆ
t T ¯ x = 0 .
Since (cid:107) s (cid:107) 2 = (cid:107) ˆ s (cid:107) 2 = 1 , we have α 2 + β 2 =
ˆ α 2 + ˆ β 2 = 1 .
In addition, we have α = s T ¯ x
and ˆ α
= ˆ s T ¯ x . Recall that in (38), we write the SVD of ¯ V
as ¯ V = ¯ U ¯ D ¯ U T .
Since t T ¯ x = 0 , we can write t as t =
(cid:80) i> 1 h
i ¯ u i .
In addition, since (cid:107) t (cid:107) 2 = 1 , we have (cid:80) i> 1 h 2 i = 1 .
Then, we obtain
[Equation]
Therefore, we obtain
[Equation] [Equation]
[Equation] [Equation]
The term (cid:104) ¯ η Es , ˆ s − ¯ x (cid:105) :
We have |(cid:104) ¯ η Es , ˆ s − ¯ x
(cid:105)| = (cid:104)
[Equation]
where (80) follows from Assumption 2 and (66), and (81) follows from ¯ η = 1 / ¯ λ 1 , along with the fact that we assumed ¯ λ 1 = Θ(1) .
Note that (cid:107) ˆ s − ¯ x (cid:107) 22 =
2(1 − ˆ s T ¯ x ) =
2(1 − ˆ α ) .
Combining (65), (68), (77), and (81), we obtain 2(1 − ˆ α )
[Equation]
(cid:16) (1 − α )(1 − ˆ α ) + ¯ γ (cid:112)
[Equation]
− α
[Equation]
1 − ˆ α
[Equation]
+ O ( δ (cid:107) E (cid:107) 2 →
[Equation]
+
[Equation]
(cid:112) 2(1 − ˆ
[Equation]
+ δ ) .
Using 2(1 − ˆ α ) − 2(1 − α )(1 − ˆ α ) =
2 α (1 − ˆ α ) , √ 1 − α 2 = √ 1 − α √ 1 + α ≤ (cid:112) 2(1 − α ) , and √ 1 − ˆ α 2 ≤ (cid:112) 2(1 − ˆ α ) , from (82), we obtain
[Equation]
[Equation]
Since (cid:107) ˆ s − ¯ x (cid:107) 22 = 2(1 − ˆ α ) and (cid:107) s − ¯ x (cid:107)
= 2(1 − α ) , this is equivalent to α (cid:107)
[Equation]
−
[Equation]
Therefore, we obtain
[Equation] [Equation]
where we use α = s T ¯ x in (86), and we use the assumption (cid:107) E (cid:107) 2 → 2 = O ( n/m ) and set δ = 1 /n in (87).
D P ROOF OF T HEOREM 2 (M AIN T HEOREM FOR PP OWER A NALYSIS )
From Lemma 2, we have
[Equation]
In addition, if (cid:107)
[Equation]
then we obtain from (88)
that
[Equation]
from which the assumed scaling m
= Ω( k log( nLr ))
(with a large enough implied constant) yields
[Equation]
Squaring both sides, expanding, and canceling equal terms, we obtain
[Equation]
and by induction, we obtain the desired result.
E C OMPARISON OF A NALYSIS TO D ESHPANDE ET AL . (2014)
As mentioned in Section 4, our analysis is signiﬁcantly different from that of Deshpande et al. (2014) despite using a similar assumption on the initialization. We highlight the differences in the following:
Perhaps the most signiﬁcant difference is that the proof of (Deshpande et al., 2014, Theorem 3) is highly dependent on the Moreau decomposition, which is only valid for a closed convex cone (see (Deshpande et al., 2014, Deﬁnition 1.
In particular, the Moreau decomposition needs to be used at the beginning of the proof of (Deshpande et al., 2014, Theorem 3), such as
Eqs.
(18) and (19) in the supplementary material therein.
We do not see a way for the proof to proceed without the Moreau decomposition, and our Range( G ) may be completely different from a convex cone.
[Figure]
We highlight that one key observation in our proof of Lemma 2 (and thus Theorem 2) is that for a normalized generative model G , and any x ∈ R n and a > 0 , we have P G ( a x ) =
P G ( x ) (Eq. (62)).
This enables us to derive the important equation ˆ s =
P G ( Vs ) =
P G (¯ η Vs ) .
We are not aware of a similar idea being used in the proof of (Deshpande et al., 2014, Theorem 3).
In the PPower method in (Deshpande et al., 2014), the authors need to add ρ I n with ρ >to the observed data matrix V to improve the convergence.
In particular, they mention in the paragraph before the statement of Theorem 3 that “the memory term ρ v t is necessary for our proof technique to go through”.
In contrast, our proof of Theorem 2 does not require adding such terms, even when our data model is restricted to the spiked Wigner model considered in (Deshpande et al., 2014).
Finally, we emphasize that we consider a matrix model that is signiﬁcantly more general than the spiked Wigner model studied in (Deshpande et al., 2014).
F N UMERICAL R ESULTS FOR THE F ASHION -MNIST D ATASET
The Fashion-MNIST dataset consists of Zalando’s article images with a training set of 60 , 000 examples and a test set of 10 , 000 examples.
The size of each image in the Fashion-MNIST dataset is also 28 × 28 , and thus n = 784 .
The generative model G is set to be a boundary-seeking generative adversarial network (BEGAN).
The BEGAN architecture is summarized as follows.
The generator has latent dimension k =and four layers.
The ﬁrst two are fully connected layers with the architecture 62 − 1024 − 6272 , and with ReLU activation functions.
The output of the second layer, reshaped to 128 × 7 × 7 , is forwarded to a deconvolution layer with kernel size 4 and stride 2 .
The third layer uses ReLU activation and has output size 64 × 14 × 14 , where 64 is the number of channels.
The fourth layer is also a deconvolution layer with kernel size 4 and strides 2 , and it uses ReLU activation and has output size 1 × 28 × 28 , where the number of channels is 1 .
The BEGAN is trained with a mini-batch size of 256 , a learning rate of 0 . 0002 , and 100 epochs.
Other settings are the same as those for the MNIST dataset.
We perform two sets of experiments, considering the spiked covariance and phase retrieval models separately.
The corresponding results are reported in Figures 4, 5, 6, and 7.
From these ﬁgures, again, we can observe clear superiority of PPower to Power and TPower .
We note that for the Fashion-MNIST dataset, some of the images are not sparse in the natural basis, but we observe from Figures 4 and 6 that even for the sparsest images (sandals), PPower also signiﬁcantly outperforms TPower.
[Figure]
C
o s i n e S i m il a r i t y
PowerTPowerPPower 0.
[Figure] C
o s i n e S i m il a r i t y
[Figure]
[Figure]
[Figure]