G ENERATIVE P RINCIPAL C OMPONENT A NALYSIS
A BSTRACT
In this paper, we study the problem of principal component analysis with generative modeling assumptions, adopting a general model for the observed matrix that encompasses notable special cases, including spiked matrix recovery and phase retrieval.
The key assumption is that the ﬁrst principal eigenvector lies near the range of an L -Lipschitz continuous generative model with bounded k -dimensional inputs.
We propose a quadratic estimator, and show that it enjoys a statistical rate of order (cid:113) k log L m , where m is the number of samples.
Moreover, we provide a variant of the classic power method, which projects the calculated data onto the range of the generative model during each iteration.
We show that under suitable conditions, this method converges exponentially fast to a point achieving the above-mentioned statistical rate.
This rate is conjectured in (Aubin et al., 2019;
Cocola et al., 2020) to be the best possible even when we only restrict to the special case of spiked matrix models.
We perform experiments on various image datasets for spiked matrix and phase retrieval models, and illustrate performance gains of our method to the classic power method and the truncated power method devised for sparse principal component analysis.
I NTRODUCTION
Principal component analysis (PCA) is one of the most popular techniques for data processing and dimensionality reduction (Jolliffe, 1986), with an abundance of applications such as image recognition (Hancock et al., 1996), gene expression data analysis (Alter et al., 2000), and clustering (Ding & He, 2004; Liu & Tan, 2019).
PCA seeks to ﬁnd the directions that capture maximal variances in vector-valued data.
In more detail, letting x 1 , x 2 , . . .
, x m be m realizations of a random vector x ∈ R n with a population covariance matrix ¯ Σ ∈ R n × n , PCA aims to reconstruct the top principal eigenvectors of ¯ Σ .
The ﬁrst principal eigenvector can be computed as follows:
[Equation]
where the empirical covariance matrix is deﬁned as Σ := 1 m (cid:80) mi
=1 ( x i − c )( x i − c ) T , with c :=m (cid:80) mi
=1 x i .
In addition, subsequent principal eigenvectors can be estimated by similar optimization problems subject to being orthogonal to the previous vectors.
PCA is consistent in the conventional setting where the dimension of the data n is relatively small compared to the sample size m (Anderson, 1962), but leads to rather poor estimates in the highdimensional setting where m (cid:28) n .
In particular, it has been shown in various papers that the empirical principal eigenvectors are no longer consistent estimates of their population counterparts (Nadler,
; Johnstone & Lu, 2009; Jung & Marron, 2009; Birnbaum et al., 2013).
In order to tackle the curse of dimensionality, a natural approach is to impose certain structural constraints on the principal eigenvectors.
A common assumption is that the principal eigenvectors are sparse, and this gives rise to the problem of sparse principal component analysis (SPCA) (Zou et al., 2006).
In particular, for recovering the top principal eigenvector, the optimization problem of SPCA is given by
[Equation]
where (cid:107) w (cid:107)
= |{ i : w i (cid:54) = 0 }| denotes the number of non-zero entries of w , and s ∈ N represents the sparsity level.
In addition to reducing the effective number of parameters, the sparsity assumption also enhances the interpretability (Zou et al., 2006).
Departing momentarily from the PCA problem, recent years have seen tremendous advances in deep generative models in a wide variety of real-world applications (Foster, 2019).
This has motivated a new perspective of the related problem of compressed sensing (CS), in which the standard sparsity assumption is replaced by a generative modeling assumption.
That is, the underlying signal is assumed to lie near the range of a (deep) generative model (Bora et al., 2017).
The authors of (Bora et al., 2017) characterized the number of samples required to attain an accurate reconstruction, and also presented numerical results on image datasets showing that compared to sparsity-based methods, generative priors can lead to large reductions (e.g., a factor of 5 to 10 ) in the number of measurements needed to recover the signal up to a given accuracy.
Additional numerical and theoretical results concerning inverse problems using generative models have been provided in (Van Veen et al., 2018;
Dhar et al., 2018; Heckel & Hand, 2019; Jalal et al., 2020; Liu & Scarlett, 2020a; Ongie et al., 2020; Whang et al., 2020; Jalal et al., 2021; Nguyen et al., 2021), among others.
In this paper, following the developments in both PCA/SPCA and inverse problems with generative priors, we study the use of generative priors in principal component analysis (GPCA), which gives a generative counterpart of SPCA in (2), formulated as follows:
[Equation]
where G is a (pre-trained) generative model, which we assume has a range contained in the unit sphere of R n .
1 Similarly to SPCA, the motivation for this problem is to incorporate prior knowledge on the vector being recovered (or alternatively, a prior preference), and to permit meaningful recovery and theoretical bounds even in the high-dimensional regime m (cid:28) n .
1. R ELATED W ORK
In this subsection, we summarize some relevant works, which can roughly be divided into (i) the
SPCA problem, and (ii) signal recovery with generative models.
SPCA : It has been proved that the solution of the SPCA problem in (2) attains the optimal statistical rate (cid:112) s log n/m (Vu & Lei, 2012), where m is the number of samples, n is the ambient dimension, and s is the sparsity level of the ﬁrst principal eigenvector.
However, due to the combinatorial constraint, the computation of (2) is intractable.
To address this computational issue, in recent years, an extensive body of practical approaches for estimating sparse principal eigenvectors have been proposed in the literature, including (d’Aspremont et al., 2007; Vu et al., 2013; Chang et al., 2016; Moghaddam et al., 2006; d’Aspremont et al., 2008; Jolliffe et al., 2003; Zou et al., 2006; Shen & Huang, 2008; Journ´ee et al., 2010; Hein & B¨uhler, 2010; Kuleshov, 2013; Yuan & Zhang, 2013;
Asteris et al., 2011; Papailiopoulos et al., 2013), just to name a few.
Notably, statistical guarantees for several approaches have been provided.
The authors of (Yuan & Zhang, 2013) propose the truncated power method (TPower), which adds a truncation operation to the power method to ensure the desired level of sparsity.
It is shown that this approach attains the optimal statistical rate under appropriate initialization.
Most approaches for SPCA only focus on estimating the ﬁrst principal eigenvector, with a certain deﬂation method (Mackey, 2008) being leveraged to reconstruct the rest.
However, there are some exceptions; for instance, an iterative thresholding approach is proposed in (Ma, 2013), and is shown to attain a near-optimal statistical rate when estimating multiple individual principal eigenvectors.
In addition, the authors of (Cai et al., 2013) propose a regression-type method that gives an optimal principal subspace estimator.
Both works (Ma, 2013; Cai et al., 2013) rely on the assumption of a spiked covariance model to ensure a good initial vector.
To avoid the spiked covariance model assumption, the work (Wang et al., 2014) proposes a two-stage procedure that attains the optimal subspace estimator in polynomial time.
Signal recovery with generative models : Since the seminal work (Bora et al., 2017), there has been a substantial volume of papers studying various inverse problems with generative priors.
One of the problems more closely related to PCA is spectral initialization in phase retrieval, which amounts to solving an eigenvalue problem.
Phase retrieval with generative priors has been studied in (Hand et al., 2018;
Hyder et al., 2019; Jagatap & Hegde, 2019;
Wei et al., 2019; Shamshad & Ahmed, 2020; Aubin et al., 2020; Liu et al., 2021b).
In particular, the work (Hand et al., 2018) models the underlying signal as being in the range of a fully-connected ReLU neural network with no offsets, and all the weight matrices of the ReLU neural network are assumed to have i.i.d.
zero-mean
Gaussian entries.
In addition, the neural network needs to be sufﬁciently expansive in the sense that n i ≥ Ω( n i − 1 log n i − 1 ) , where n i is the width of the i -th layer.
Under these assumptions, the authors establish favorable global optimization landscapes for the corresponding objective, and derive a near-optimal sample complexity upper bound.
They minimize the objective function directly over the latent space in R k using gradient descent, which may suffer from local minima in general optimization landscapes (Hyder et al., 2019; Shah & Hegde, 2018).
In (Aubin et al., 2020), the assumptions on the neural network are similar to those in (Hand et al., 2018), relaxing to general activation functions (beyond ReLU) and n i ≥ Ω( n i − 1 ) .
The authors focus on the high dimensional regime where n, m, k → ∞ with the ratio m/n being ﬁxed, and assume that the input vector in R k is drawn from a separable distribution.
They derive sharp asymptotics for the information-theoretically optimal performance and for the associated approximate message passing (AMP) algorithm.
Both works (Hand et al., 2018; Aubin et al., 2020) focus on noiseless phase retrieval.
When only making the much milder assumption that the generative model is Lipschitz continuous, with no assumption on expansiveness, Gaussianity, and offsets, a spectral initialization step (similar to that of sparse phase retrieval) is typically required in order to accurately reconstruct the signal (Netrapalli et al., 2015; Cand ` es et al., 2015).
The authors of (Liu et al., 2021b) propose an optimization problem similar to (3) for the spectral initialization for phase retrieval with generative models.
It was left open in (Liu et al., 2021b) how to solve (or approximate sufﬁciently accurately) the optimization problem in practice.
Understanding the eigenvalues of spiked random matrix models has been a central problem of random matrix theory, and spiked matrices have been widely used in the statistical analysis of SPCA.
Recently, theoretical guarantees concerning spiked matrix models with generative priors have been provided in (Aubin et al., 2019; Cocola et al., 2020).
In particular, in (Aubin et al., 2019), the assumptions are similar to those in (Aubin et al., 2020), except that the neural network is assumed to have exactly one hidden layer.
The Bayes-optimal performance is analyzed, and it is shown that the
AMP algorithm can attain this optimal performance.
In addition, the authors of (Aubin et al., 2019) propose the linearized approximate message passing (LAMP) algorithm, which is a spectral algorithm speciﬁcally designed for single-layer feedforward neural networks with no bias terms.
The authors show its superiority to classical PCA via numerical results on the Fashion-MNIST dataset.
In (Cocola et al., 2020), the same assumptions are made as those in (Hand et al., 2018) on the neural network, and the authors demonstrate the benign global geometry for a nonlinear least squares objective.
Similarly to (Hand et al., 2018), the objective is minimized over R k using a gradient descent algorithm, which can get stuck in local minima for general global geometries.
1. C ONTRIBUTIONS
The main contributions of this paper are as follows:
• We study eigenvalue problems with generative priors (including GPCA), and characterize the statistical rate of a quadratic estimator similar to (3) under suitable assumptions.
• We propose a variant of the classic power method, which uses an additional projection operation to ensure that the output of each iteration lies in the range of a generative model.
We refer to our method as projected power method (PPower).
We further show that under appropriate
conditions (most
notably
, assuming exact projections are possible), PPower
obtains a solution achieving a statistical rate that is conjectured to be optimal in (Aubin et al., 2019; Cocola et al., 2020) for spiked matrix models.
• For the spiked matrix and phase retrieval models, we perform numerical experiments on image datasets, and demonstrate that when the number of samples is relatively small compared to the ambient dimension, PPower leads to signiﬁcantly better performance compared to the classic power method and TPower.
Compared to the above-mentioned works that use generative models, we make no assumption on expansiveness, Gaussianity, and offsets for the generative model, and we consider a data model that simultaneously encompasses both spiked matrix and phase retrieval models, among others.
1. N OTATION
We use upper and lower case boldface letters to denote matrices and vectors respectively.
We write [ N ] = { 1 , 2 , · · · , N } for a positive integer N , and we use I N to denote the identity matrix in R N × N .
A generative model is a function G : D → R n , with latent dimension k , ambient dimension n , and input domain D ⊆ R k .
We focus on the setting where k (cid:28) n .
For a set S ⊆ R k and a generative model G : R k
→ R n , we write G ( S ) =
{ G ( z ) : z ∈ S } .
We use (cid:107) X (cid:107) 2 → 2 to denote the spectral norm of a matrix X .
We deﬁne the (cid:96) q -ball B k q ( r ) :=
{ z ∈ R k : (cid:107) z (cid:107) q ≤ r } for q ∈ [0 , + ∞ ] .
S n − 1 := { x ∈ R n : (cid:107) x (cid:107) 2 = 1 } represents the unit sphere in R n .
The symbols C, C (cid:48) , C (cid:48)(cid:48) are absolute constants whose values may differ from line to line.
P ROBLEM S ETUP
In this section, we formally introduce the problem, and overview some important assumptions that we adopt.
Except where stated otherwise, we will focus on the following setting:
• We have a matrix V ∈ R n × n satisfying
[Equation]
where E is a perturbation matrix, and ¯ V is assumed to be positive semideﬁnite (PSD).
For PCA and its constrained variants, V and ¯ V can be thought of as the empirical and population covariance matrices, respectively.
• We have an L -Lipschitz continuous generative model G : B k 2 ( r ) → R n .
For convenience, similarly to that in (Liu et al., 2020), we assume that Range( G ) ⊆
S n − 1 .
Remark 1.
For a general (unnormalized) L -Lipschitz continuous generative model G , we can instead consider a corresponding normalized generative model ˜ G : D → S n − 1 as in (Liu et al., 2021b), where D := { z ∈ B k 2 ( r ) :
(cid:107) G ( z ) (cid:107) 2 > R min } for some R min > 0 , and ˜
G
( z
=
G
( z ) (cid:107) G ( z
) (cid:107) 2 .
Then, the Lipschitz constant of ˜ G becomes L/R min .
For a d -layer neural network, we typically have L = n Θ( d ) (Bora
et al
Thus, we can set R min to be as small as 1 /n Θ( d ) without changing the scaling laws, which makes the dependence on R min very mild.
• We aim to solve the following eigenvalue problem with a generative prior:
[Equation]
Note
that
since Range( G ) ⊆
S n − 1 , we do not need to impose the constraint (cid:107) w (cid:107) 2 = 1 .
Since
V is not restricted to being an empirical covariance matrix, (5) is more
general than
GPCA in (3).
However, we slightly abuse terminology and also refer to (5) as GPCA.
• To approximately solve (5), we use a projected power method (PPower)
, which is described by the following iterative procedure:
[Equation]
Input : V , number of iterations T , pre-trained generative model G , initial vector w (0)
Procedure: Iterate w ( t +1) =
P G (cid:0) Vw ( t ) (cid:1) for t ∈ { 0 , 1 , . .
, T − 1 } , and return w ( T ) where P G ( · ) is the projection function onto G ( B k 2 ( r )) , 4 and the initialization vector w (0) may be chosen either manually or randomly, e.g., uniform over S n − 1 .
Often the initialization vector w (0) plays an important role and we may need a careful design for it.
For example, for phase retrieval with generative models, as mentioned in (Liu et al., 2021b, Section V), we may choose the column corresponding to the largest diagonal entry of V as the starting point.
See also (Yuan & Zhang, 2013, Section 3) for a discussion on the initialization strategy for TPower devised for SPCA.
We present the algorithm corresponding to (6) in Algorithm 1.
Remark 2.
To tackle generalized eigenvalue problems encountered in some speciﬁc applications, there are variants of the projected power method, which combine a certain power iteration with additional operations to ensure sparsity or enforce other constraints.
These applications include but not limited to sparse PCA (Journ´ee et al., 2010; Yuan & Zhang, 2013), phase synchronization (Boumal, 2016;
Liu et al., 2017), the hidden clique problem (Deshpande & Montanari, 2015), the joint alignment problem (Chen & Cand ` es, 2018), and coneconstrained PCA (Deshpande et al., 2014; Yi & Neykov, 2020).
For example, under the simple spiked Wigner model (Perry et al., 2018) for the observed data matrix V with the underlying signal being assumed to lie in a convex cone, the authors of (Deshpande et al., 2014) show that cone-constrained PCA can be computed efﬁciently via a generalized projected power method.
In general, the range of a Lipschitz-continuous generative model is non-convex and not a cone.
In addition, we consider a matrix model that is more general than the spiked Wigner model.
Although it is not needed for our main results, we ﬁrst state a lemma (proved in Appendix A) that establishes a monotonicity property with minimal assumptions, only requiring that V is PSD; see also Proposition 3 of Yuan & Zhang (2013) for an analog in sparse PCA.
By comparison, our main results in Section 4 will make more assumptions, but will also provide stronger guarantees.
Note that the PSD assumption holds, for example, when E = 0 , or when V is a sample covariance matrix.
Lemma 1.
For any x ∈ R n , let Q ( x ) = x T Vx .
Then, if V is PSD, the sequence { Q ( w ( t ) ) } t> 0 for w ( t ) in (6) is monotonically non-decreasing.
S PECIALIZED D ATA M ODELS AND E XAMPLES
In this section, we make more speciﬁc assumptions on V = ¯ V + E , starting with the following.
Assumption 1 (Assumption on ¯ V ) .
Assume that ¯ V is PSD with eigenvalues ¯ λ 1 > ¯ λ 2 ≥ . . .
≥ ¯ λ n ≥ 0 .
We use ¯ x (a unit vector) to represent the eigenvector of ¯ V that corresponds to ¯ λ 1 .
In the following, it is useful to think of ¯ x is being close to the range of the generative model G .
In the special case of (3), letting m be the number of samples, it is natural to derive that the upper bound of (cid:107) E (cid:107) 2 → 2 grows linearly in ( n/m ) b for some positive constant b such as 12 or 1 (with high probability; see, e.g., (Vershynin, 2010, Corollary 5.35)).
In the following, we consider general scenarios with
V depending on m samples (see below for speciﬁc examples).
Similarly to (Yuan & Zhang, 2013), we may consider a restricted version of (cid:107) E (cid:107) 2 → 2 , leading to the following.
Assumption 2 (Assumption on E ) .
Let S 1 , S 2 be two (arbitrary) ﬁnite sets in R n satisfying m = Ω(log( | S 1 | · | S 2 | )) .
Then, we have for all s 1 ∈ S 1 and s 2 ∈ S 2 that
[Equation]
where C is an absolute constant.
In addition, we have (cid:107) E (cid:107) 2 → 2 = O ( n/m ) .
The following examples show that when the number of measurements is sufﬁciently large, the data matrices corresponding to certain spiked matrix and phase retrieval models satisfy the above assumptions with high probability.
Short proofs are given in Appendix B for completeness.
Example 1 (Spiked covariance model) .
In the spiked covariance model (Johnstone & Lu, 2009;
Deshpande & Montanari, 2016), the observed vectors x 1 , x 2 , . .
x m ∈ R n are of the form
[Equation]
where s 1 , . .
, s r ∈ R n are orthonormal vectors that we want to estimate, while z i ∼ N ( 0 , I n ) and u q,i ∼ N (0 , 1) are independent and identically distributed.
In addition, β 1 , . . .
, β r are positive constants that dictate the signal-to-noise ratio (SNR).
To simplify the exposition, we focus on the rank-one case and drop the subscript q ∈ [ r ] .
Let
[Equation]
and ¯ V = E
[ V ] =
β ss T .
Then, ¯ V satisﬁes Assumption 1 with ¯ λ 1 = β > 0 , ¯ λ 2 = . . .
=
¯ λ n = 0 , and ¯ x = s .
In addition, letting E =
V − ¯ V , the Bernstein-type inequality (Vershynin, 2010,
Proposition 5.10) for the sum of sub-exponential random variables yields that for any ﬁnite sets
S 1 , S 2 ⊂ R n , when m = Ω (cid:0) log( | S 1 | · | S 2 | )) , with probability 1 − e − Ω(log( | S 1 |·| S 2 | )) , E satisﬁes (7) in Assumption 2.
Moreover, standard concentration arguments give (cid:107) E (cid:107) 2 → 2 = O ( n/m ) with probability 1 − e − Ω( n ) .
Remark 3.
We can also consider the simpler spiked Wigner model (Perry et al., 2018; Chung &
Lee, 2019)
where
V = β ss T + 1 √ n H , with the signal s being a unit vector, β > 0 being an SNR parameter, and H ∈ R n × n being a symmetric matrix with entries drawn i.i.d.
(up to symmetry) from
N (0 , 1) .
In this case, when m = n is sufﬁciently large, with high probability, ¯ V :=
E
[ V ] =
β ss T and E :=
V − ¯ V similarly satisfy Assumptions 1 and 2 respectively.
Example 2 (Phase retrieval) .
Let A ∈ R m × n be a matrix having i.i.d.
N (0 , 1) entries, and let a Ti be the i -th row of A .
For some unit vector s , suppose that the observed vector is y =
| As | , where the absolute value is applied element-wise.
We construct the weighted empirical covariance matrix as follows (Zhang et al., 2017; Liu et al., 2021b):
[Equation]
where u > l > 1 are positive constants, and for g ∼ N (0 , 1) , γ :
= E (cid:2) | g | 1 { l< |
[Equation]
<
[Equation]
} (cid:3) .
Let ¯ V =
E
[ V ] =
β ss T , where β := E (cid:2)(cid:0) | g | 3 − | g | (cid:1) 1 { l< | g | <u } (cid:3) .
Then, ¯ V satisﬁes Assumption 1 with ¯ λ 1 = β > 0 , ¯ λ 2 = . . .
=
¯ λ n = 0 , and ¯ x = s .
In addition, letting E = V − ¯ V , we have similarly to
Example 1 that E satisﬁes Assumption 2 with high probability.
M AIN R ESULTS
The following theorem concerns globally optimal solutions of (5).
The proof is given in Appendix D.
Theorem 1.
Let V = ¯ V + E with Assumptions 1 and 2 being satisﬁed by ¯ V and E respectively, and let x G :=
P G (¯ x ) =
arg min w ∈ Range( G ) (cid:107) w −
¯
x
(cid:107) 2 .
Suppose that ˆ v is a globally optimal solution to (5) .
Then, for any δ ∈ (0 , 1) , we have
[Equation]
where (cid:15) n = O (cid:0) (cid:113) k log Lrδ m (cid:1) .
in n (Bora et al., 2017), the upper bound for (cid:107) E (cid:107) 2 → 2 can be easily relaxed to O (cid:0) ( n/m ) b (cid:1) for any positive constant b , without affecting the scaling of our derived statistical rate.
To avoid non-essential complications, β is typically assumed to be known (Johnstone & Lu, 2009).
Without loss of generality, we assume that s is a unit vector.
For a general signal s , we may instead focus on estimating ¯ s = s / (cid:107) s (cid:107) 2 , and simply use 1 m (cid:80) mi =1 y i to approximate (cid:107) s (cid:107) 2 .
We have stated this result as an upper bound on (cid:107) ˆ v ˆ v T − ¯ x ¯ x T (cid:107) F , which intuitively measures the distance between the 1D subspaces spanned by ˆ v and ¯ x .
Note, however, that for any two unit vectors w 1 , w 2 with w T 1 w 2 ≥ 0 , the distances (cid:107) w 1 − w 2 (cid:107) 2 and (cid:107) w 1 w T 1 − w 2 w T 2 (cid:107) F are equivalent up to constant factors, whereas if w T 1 w 2 ≤ 0 , then a similar statement holds for (cid:107) w 1 + w 2 (cid:107) 2 .
See
Appendix C for the precise statements.
In Theorem 1, the ﬁnal term quantiﬁes the effect of representation error.
When there is no such error (i.e., ¯ x ∈ Range( G ) ), under the scaling ¯ λ 1 − ¯ λ 2 = Θ(1) , L = n Ω(1) , and δ =
O (1 /n ) , Theoremsimpliﬁes to (cid:107) ˆ v ˆ v T − ¯ x ¯ x T (cid:107) F = O ( (cid:113)
k log L m ) .
This provides a natural counterpart to the statistical rate of order (cid:113) s log n m for SPCA mentioned in Section 1.1.
Before providing our main theorem for PPower described in (6), we present the following important lemma, whose proof is presented in Appendix E.
To simplify the statement of the lemma, we ﬁx δ to be O (1 /n ) , though a more general form analogous to Theorem 1 is also possible.
Lemma 2.
Let V =
¯ V + E with Assumptions 1 and 2 being satisﬁed by ¯ V and E respectively, and further assume that ¯ x ∈ Range( G ) .
Let ¯ γ =
¯
λ 2 / ¯ λ 1 with ¯ λ 1 = Θ(1) .
Then, for all s ∈ Range( G ) satisfying s T ¯ x > 0 , we have
[Equation]
where C is an absolute constant.
Remark 4.
The assumption s T ¯ x > 0 will be particularly satisﬁed when the range of G only contains nonnegative vectors.
As mentioned in various works studying nonnegative SPCA (Zass &
Shashua, 2007; Sigg & Buhmann, 2008; Asteris et al., 2014), for several practical ﬁelds such as economics, bioinformatics, and computer vision, it is natural to assume that the underlying signal has no negative entries.
More generally, the assumption s T ¯ x > 0 can be removed if we additionally have that − ¯ x is also contained in the range of G .
For this case, when s T ¯ x < 0 , we can instead derive an upper bound for (cid:107) ˆ s + ¯ x (cid:107) 2 .
Based on Lemma 2, we have the following theorem, whose proof is given in Appendix F.
Theorem 2.
Suppose that the assumptions on the data model V =
¯ V + E are the same as those in
Lemma 2, and assume that there exists t 0 ∈ N such that ¯ x T w ( t 0 )
=
2¯ γ
+ ν with 2¯ γ + ν ≤ 1 − τ , where ¯ γ = ¯
λ
2 / ¯ λ 1 ∈ [0 , 1) , and ν, τ are both positive and scale as Θ(1) .
Let µ 0 =
[Equation]
and in addition, suppose that m ≥ C ν,τ · k log( nLr ) with C ν,τ > 0 being large enough.
Then, we have after ∆
= O (cid:0) log (cid:0) m k log( nLr ) (cid:1)(cid:1) iterations of PPower (beyond t 0 )
that
[Equation]
i.e., this equation holds for all t ≥ T 0 :
=
t 0 + ∆ 0 .
Moreover, if ¯ γ = 0 then ∆ 0 ≤ 1 , whereas if ¯ γ = Θ(1) , we have exponentially fast convergence via the following contraction property: There exists a constant ξ ∈ (0 , 1) such that for t ∈ [ t 0 , T 0 ) , it holds that
[Equation]
Regarding the assumption ¯ x T w ( t 0 ) ≥ 2¯ γ + ν , we note that when t 0 = 0 , this condition can be viewed as having a good initialization.
For both Examples 1 and 2, we have ¯ γ = 0 .
Thus, for the spiked covariance and phase retrieval models corresponding to these examples, the assumption ¯ x T w ( t 0 ) ≥ 2¯ γ
+ ν reduces to ¯ x T w ( t 0 ) ≥
ν for a sufﬁciently small positive constant ν , which results in a mild assumption.
Such an assumption is also required for the projected power method devised for cone-constrained PCA under the simple spiked Wigner model, with the underlying signal being assumed to lie in a convex cone; see (Deshpande et al., 2014, Theorem 3).
Despite using a similar assumption on the initialization, our proof techniques are signiﬁcantly different from Deshpande et al.
; see Appendix G for discussion.
When L is polynomial in n , Theorem 2 reveals that we have established conditions under which
PPower in (6) converges exponentially fast to a point achieving the statistical rate of order (cid:113)
k log L m .
Based on the minimax rates for SPCA (Vu & Lei, 2012; Birnbaum et al., 2013) and the informationtheoretic lower bounds for CS with generative models (Liu & Scarlett, 2020b; Kamath et al., 2020), the optimal rate for GPCA is naturally conjectured to be of the same order (cid:113)
k log L m .
We highlight that Theorem 2 partially addresses the computational-to-statistical gap (e.g., see (Wang et al., 2016;
Hand et al., 2018; Aubin et al., 2019; Cocola et al., 2020)) for spiked matrix recovery and phase retrieval under a generative prior, though closing it completely would require efﬁciently ﬁnding a good initialization and addressing the assumption of exact projections.
Perhaps the main caveat to Theorem 2 is that it assumes the projection step can be performed exactly.
However, this is a standard assumption in analyses of projected gradient methods, e.g., see (Shah &
Hegde, 2018), and both gradient-based projection and GAN-based projection have been shown to be highly effective in practice (Shah & Hegde, 2018; Raj et al., 2019).
E XPERIMENTS
In this section, we experimentally study the performance of Algorithm 1 ( PPower ).
We note that these experiments are intended as a simple proof of concept rather than seeking to be comprehensive, as our contributions are primarily theoretical.
We compare with the truncated power method ( TPower ) devised for SPCA proposed in (Yuan & Zhang, 2013, Algorithm 1) and the vanilla power method ( Power ) that performs the iterative procedure w ( t +1) =
( Vw ( t ) ) / (cid:107) Vw ( t ) (cid:107) 2 .
For a fair comparison, for PPower , TPower , and Power , we use the same initial vector.
Speciﬁcally, as mentioned in (Liu et al., 2021b, Section V), we choose the initialization vector w (0) as the column of V that corresponds to its largest diagonal entry.
For all three algorithms, the total number of iterations T is set to be 30 .
To compare the performance across algorithms, we use the scale-invariant
Cosine Similarity metric deﬁned as Cossim (cid:0) ¯ x , w ( T ) (cid:1) :
= (cid:104) ¯ x , w ( T ) (cid:105) (cid:107) ¯ x (cid:107) 2 (cid:107) w ( T ) (cid:107) 2 ,
where ¯ x is the ground-truth signal to estimate, and w ( T ) denotes the output vector of the algorithm.
The experiments are performed on the MNIST (LeCun et al., 1998),
Fashion-MNIST (Xiao et al., 2017) and CelebA (Liu et al., 2015) datasets, with the numerical results for the Fashion-MNIST and CelebA datasets being presented in Appendix H and I.
The MNIST dataset consists of 60 ,images of handwritten digits.
The size of each image is 28 × 28 , and thus n = 784 .
To reduce the impact of local minima, we perform 10 random restarts, and choose the best among these.
The cosine similarity is averaged over the test images, and also over these 10 random restarts.
The generative model G is set to be a pre-trained variational autoencoder (VAE) model with latent dimension k = 20 .
We use the VAE model trained by the authors of (Bora et al., 2017) directly, for which the encoder and decoder are both fully connected neural networks with two hidden layers, with the architecture being 20 − 500 − 500 − 784 .
The VAE is trained by the Adam optimizer with a minibatch size of 100 and a learning rate of 0 .
The projection step P G ( · ) is solved by the Adam optimizer with a learning rate of 0 . 03 and 200 steps.
In each iteration of TPower , the calculated entries are truncated to zero except for the largest q entries, where q ∈ N is a tuning parameter.
Since for TPower , q is usually selected as an integer larger than the true sparsity level, and since it is unlikely that the image of the MNIST dataset can be well approximated by a k -sparse vector with k = 20 , we choose a relatively large q , namely q = 150 .
Similarly to (Bora et al., 2017) and other related works, we only report the results on a test set that is unseen by the pre-trained VAE model, i.e., the training of G and the PPower computations do not use common data.
Spiked covariance model (Example 1) :
The numerical results are shown
in
Figures 1 and 2.
We observe
from Figure 1 that
Power and TPower attain poor reconstructions
, and
the generative prior based method PPower attains signiﬁcantly better reconstructions
To il lustrate the
effect
of
the sample
size m ,
we
ﬁx
the
SNR parameter β = 1 and vary m in {
In addition, to illustrate the
effect
of the SNR parameter β , we ﬁx m = 300 ,
and vary β in { 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 , 2 , 3 , 4 } .
From
Figure
2, we observe that for
these
settings
of m
and β , PPower always leads to
a
much
higher cosine similarity
compared
to Power and TPower
which is natural given the more precise modeling assumptions used.
O r i g i n a
[Figure]
P o w e r
[Figure]
P
[Figure] [Figure]
Phase retrieval (Example 2) :
The results are shown in Figure 2 (Right) and Figure 3.
Again, we can observe that PPower signiﬁcantly outperforms Power and TPower .
In particular, for sparse phase retrieval, when performing experiments on image datasets, even for the noiseless setting, solving an eigenvalue problem similar to (5) can typically only serve as a spectral initialization step, with a subsequent iterative algorithm being required to reﬁne the initial guess.
In view of this, it is notable that for phase retrieval with generative priors, PPower can return meaningful reconstructed images for m = 200 , which is small compared to n = 784 .
C ONCLUSION
We have proposed a quadratic estimator for eigenvalue problems with generative models, and we showed that this estimator attains a statistical rate of order (cid:113)
k log L m .
We provided a projected power method to efﬁciently solve (modulo the complexity of the projection step) the corresponding optimization problem, and showed that our method converges exponentially fast to a point achieving a statistical rate of order (cid:113) k log L m under suitable conditions.
Acknowledgment.
J.S. was supported by the Singapore National Research Foundation (NRF) under grant R-252-000-A74-281, and S.G. was supported in part by the MOE grants R-146-000-250-133, R-146-000-312-114, and MOE-T2EP20121-0013.
O r i g i n a l
P o w e r
T P o w e r
PP o w e r
O r i g i n a l
P o w e r
T P o w e r
PP o w e r (a) m =
[Figure]
A PPENDIX (G ENERATIVE P RINCIPAL C OMPONENT A NALYSIS , L IU /L IU /G HOSH /H AN /S CARLETT , ICLR 2022) A P ROOF OF L EMMA 1 (N ON -D ECREASING P ROPERTY OF Q )
Since w ( t +1) =
P G (cid:0) Vw ( t ) (cid:1) and w ( t ) ∈
Range( G ) , we have (cid:107)
[Equation]
( t )
[Equation]
w ( t +1) (cid:107) 2 ≤ (cid:107)
[Equation]
[Equation]
, (15) and since (cid:107) w ( t +1) (cid:107) 2 = (cid:107) w ( t ) (cid:107) 2 = 1 , expanding the square gives
[Equation]
Then, we obtain
[Equation] [Equation]
[Equation] [Equation]
(20) where (18) follows by writing w ( t +1)
=
w ( t ) + ( w ( t +1) − w ( t ) ) and expanding, (19) follows from the assumption that V is PSD, and (20) follows from (16).
B P ROOFS FOR S PIKED M ATRIX AND P HASE R ETRIEVAL E XAMPLES
Before proceeding, we present the following standard deﬁnitions.
Deﬁnition 1.
A random variable X is said to be sub-Gaussian if there exists a positive constant C such that ( E [ | X | p ]) 1 /p ≤ C √ p for all p ≥ 1 .
The sub-Gaussian norm of a sub-Gaussian random variable X is deﬁned as (cid:107) X (cid:107) ψ 2 :=
sup p ≥ 1 p − 1 / 2 ( E [ | X | p ]) 1 /p .
Deﬁnition 2.
A random variable X is said to be sub-exponential if there exists a positive constant C such that ( E [ | X | p ]) 1 p ≤ Cp for all p ≥ 1 .
The sub-exponential norm of X is deﬁned as (cid:107) X (cid:107) ψ 1 :=
sup p ≥ 1 p − 1 ( E [ | X | p ]) 1 p .
The following lemma states that the product of two sub-Gaussian random variables is subexponential, regardless of the dependence between them.
Lemma 3.
(Vershynin, 2018, Lemma 2.7.7) Let X and Y be sub-Gaussian random variables (not necessarily independent).
Then XY is sub-exponential, and satisﬁes
[Equation]
The following lemma provides a useful concentration inequality for the sum of independent subexponential random variables.
Lemma 4.
(Vershynin, 2010, Proposition 5.16)
Let X 1 , . . .
X N be independent zero-mean subexponential random variables, and K = max i (cid:107)
X
i (cid:107) ψ 1 .
Then for every α =
[ α 1 , . . .
, α N ] T ∈ R N and (cid:15) ≥ 0 , it holds that
[Equation]
where c > 0 is an absolute constant.
In particular, with α = (cid:2) 1 N , . . .
, 1 N (cid:3) T , we have
[Equation]
The widely-used notion of an (cid:15) -net is introduced as follows.
Deﬁnition 3.
Let ( X , d ) be a metric space, and ﬁx (cid:15) > 0 .
A subset S ⊆
X is said be an (cid:15) -net of X if, for all x ∈ X , there exists some s ∈ S such that d ( s, x ) ≤ (cid:15) .
The minimal cardinality of an (cid:15) -net of X , if ﬁnite, is denoted C ( X , (cid:15) ) and is called the covering number of X (at scale (cid:15) ).
The following lemma provides a useful upper bound for the covering number of the unit sphere.
Lemma 5.
(Vershynin, 2010, Lemma 5.2)
The unit Euclidean sphere S N − 1 equipped with the
Euclidean metric satisﬁes for every (cid:15) >
that
[Equation]
The following lemma provides an upper bound for the spectral norm of a symmetric matrix.
Lemma 6.
(Vershynin, 2010, Lemma 5.4) Let X be a symmetric N × N matrix, and let C (cid:15) be an (cid:15) -net of S
N − 1 for some (cid:15) ∈ [0 , 1 / 2) .
Then,
[Equation]
With the above auxiliary results in place, we provide the proofs of Assumption 2 holding for the two examples described in Section 3.
B. S PIKED C OVARIANCE M ODEL (E XAMPLE 1)
As per Assumption 2, ﬁx two ﬁnite signal sets S 1 and S 2 .
For r = 1 , we have x i = √
βu
i s + z
i and a direct calculation gives E [ V ] =
¯ V = β ss T .
Recall also that (cid:107) s (cid:107) 2 = 1 , u i ∼ N (0 , 1) , and z i ∼
N ( 0 , I n ) .
It follows that for any s 1 ∈ S 1 , we have that s T 1 x i = √ βu
i s T s
+ z Ti s 1 is sub-Gaussian, with the sub-Gaussian norm being upper bounded by C ( √ β + 1) (cid:107) s 1 (cid:107) 2 .
Similarly, we have for any s 2 ∈ S 2 that (cid:107) s T 2 x i (cid:107) ψ 2 ≤ C ( √ β + 1) (cid:107) s 2 (cid:107) 2 .
Applying Lemma 3, we deduce that ( s T 1 x i )( s T 2 x i ) is sub-exponential, with the sub-exponential norm being upper bounded by C 2 ( √ β + 1) 2 (cid:107) s 1 (cid:107) 2 (cid:107) s 2 (cid:107) 2 .
In addition, from (9) and ¯ V = β ss T , we have
[Equation] [Equation]
[Equation]
and we observe that E [( x Ti s 1 )( x Ti s 2 )]
= ( s T 1 s 2 )+ β ( s T s 1 )( s T s 2 ) .
Then, from Lemma 4, we obtain that for any t > 0 satisfying m = Ω( t ) , the following holds with probability 1 − e − Ω( t ) (recall that
C may vary from line to line):
[Equation]
where we note that the assumption m = Ω( t ) ensures that the ﬁrst term is dominant in the minimum in (23).
Taking a union bound over all s 1 ∈ S 1 and s 2 ∈ S 2 , and setting t = log( | S 1 | · | S 2 | ) , we obtain with probability 1 − e − Ω(log( | S 1 |·| S 2 | )) that (7) holds (with β being a ﬁxed positive constant).
Next, we bound | r T
Er | for ﬁxed r ∈ S n − 1 , but this time consider t > 0 (different from the above t ) satisfying t
= Ω( m ) .
In this case, we can follow the above analysis (with s 1 and s 2 both replaced by r ), but the assumption t = Ω( m ) means that when applying Lemma 4, the second term in the minimum in (23) is now the dominant one.
As a result, for any t > 0 satisfying t = Ω( m ) , and any r ∈ S n − 1 , we have with probability 1 − e − Ω( t ) that
[Equation] [Equation]
From Lemma 5, there exists an (1 / 4) -net
C 14 of S n − 1 satisfying log (cid:12)(cid:12) C 14 (cid:12)(cid:12) ≤ n log 9 .
Taking a union bound over all r ∈ C 14 , and setting t =
Cn , we obtain with probability 1 − e − Ω( n ) that
[Equation]
Then, from Lemma 6, we have
[Equation]
B. P HASE R ETRIEVAL (E XAMPLE 2)
Let W = 1 m (cid:80) mi
=
y i a i
a Ti 1 { l<y i <u } and ¯ W = β
ss T + γ I n .
It is shown in (Liu et al., 2021b,
Lemma 8) that
[Equation]
33) which implies
[Equation]
Then, for any s 1 ∈ S 1 and s 2 ∈ S 2 , we have
[Equation]
[Equation]
Since each a i has i.i.d.
N (0 , 1) entries, we observe that y i ( a Ti s 1 )( a Ti s 2 ) 1 { l<y i <u } is subexponential with the sub-exponential norm being upper bounded by Cu (cid:107) s 1 (cid:107) 2 (cid:107) s 2 (cid:107) 2 .
In addition, from (33), we have E [ y i ( a Ti s 1 )( a Ti s 2 ) 1 { l<y
i <u } ]
=
β ( s T s 1 )( s T s 2 ) + γ ( s T 1 s 2 ) .
Then, from
Lemma 4, we obtain that for any t > 0 satisfying m = Ω( t ) , with probability 1 − e − Ω( t ) , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)m m (cid:88)
[Equation]
√ t √ m.
Taking a union bound over all s 1 ∈ S 1 and s 2 ∈ S 2 , and setting t = log( | S 1 | · | S 2 | ) , we obtain that with probability 1 − e − Ω(log( | S 1 |·| S 2 | )) , (7) holds as desired (with u being a ﬁxed positive constant).
In addition, similarly to (32), we have with probability 1 − e − Ω( n ) that (cid:107) E (cid:107) 2 → 2 = O (cid:0) nm (cid:1) .
C E QUIVALENCE OF D ISTANCES
The following lemma gives a useful equivalence between two distances.
Lemma 7.
For any pair of unit vectors w 1 , w 2 with w T 1 w 2 ≥ 0 , we have
[Equation]
Moreover, if w T 1 w 2 < 0 , then the same holds with (cid:107) w 1 − w 2 (cid:107) 2 replaced by (cid:107) w 1 + w 2 (cid:107) 2 .
Proof.
When w T 1 w 2 ≥ 0 , we have
[Equation]
where (40) follows by expanding the product and writing tr( w 1 w T 1 w 1 w T 1 )
= tr( w T 1 w 1 w T 1 w 1 ) =
( w T 1 w 1 ) 2 = 1 and handling the other terms similarly, and (41) follows since w T 1 w 2 ∈ (0 , 1) .
In addition, we have
[Equation]
which gives the desired inequality.
The case w T 1 w 2 < 0 is handled similarly
D P ROOF OF T HEOREM 1 (G UARANTEE ON THE G LOBAL O PTIMUM )
Let the singular value decomposition (SVD) of ¯
V be
[Equation]
where ¯ D = Diag([¯ λ 1 , . .
, ¯ λ n ]) , and ¯ U ∈ R n is an orthonormal matrix with the ﬁrst column being ¯ x .
For i > 1 , let the i -th column of ¯ U be ¯
u i .
Then, we have
[Equation] [Equation]
where we use (cid:0) ¯ x T ˆ v (cid:1)
+ (cid:80) i> 1 (cid:0) ¯ u Ti
[Equation]
[Equation]
In addition, for any A ∈ R n × n and any s 1 , s 2 ∈ R n , we have
[Equation]
In particular, when A is symmetric, we obtain
[Equation]
Let M be a ( δ/L ) -net of B k 2 ( r ) ; from (Vershynin, 2010, Lemma 5.2), we know that there exists such a net with
[Equation]
[Equation]
Since G is L -Lipschitz continuous, we have that G ( M ) is a δ -net of Range( G )
=
G ( B k 2 ( r )) .
We write
[Equation]
(55) where ˜ x ∈ G ( M ) satisﬁes (cid:107) ˆ v − ˜ x (cid:107) 2 ≤ δ .
Suppose that ¯ x T ˆ v ≥ 0 ; if this is not the case, we can use analogous steps to obtain an upper bound for (cid:107) ¯ x + ˆ v (cid:107) 2 instead of (cid:107) ¯ x − ˆ v (cid:107) 2 .
We have ¯
[Equation]
[Equation] [Equation]
[Equation] [Equation]
where:
• (57)–(58) follow from (cid:107) ¯ x (cid:107)
= (cid:107)
ˆ v (cid:107) 2 = 1 and hence | ¯ x T ˆ v | ≤ 1 ; • (60) follows since (¯ λ 1 , ¯ x ) are an eigenvalue-eigenvector pair for ¯ V with (cid:107) ¯ x (cid:107) 2 = 1 ; • (61) follows from (50);
• (62) follows from (53) with ¯ V being symmetric and setting s
= ¯ x , s 2 = x G ; • (63) follows from (cid:107) ¯ x + x G (cid:107) 2 ≤ 2 and (cid:107) ¯ V (cid:107) 2 → 2 = ¯ λ 1 ; • (64) follows since ¯ V =
V − E ; • (65) follows since ˆ v is a globally optimal solution to (5) and x G ∈ Range( G ) ; • (66) follows from (52) with s
= ˆ v and s 2 = ˜
x
; • (67) follows from (55) along with (cid:107) ˆ v − ¯ x (cid:107)
≤ δ and (cid:107) ˆ v + ˜ x
(cid:107) 2 ≤ 2 ; • (68) follows from (52); • (69) follows from Assumption 2 (with S
= S 2 being G ( M ) shifted by x G ) and (54); • (70) follows from the triangle inequality; • (71) follows by substituting (cid:107) ˆ v − ˜ x (cid:107) 2 ≤ δ , along with the assumptions (cid:107) E (cid:107) 2 → 2 = O ( n/m ) , m
= Ω (cid:0) k log Lrδ (cid:1) , and (cid:15) n
=
O (cid:0)(cid:113) k log Lrδ m
(cid:1)
From (71), we have the following when ˆ v T ¯ x ≥ 0 : (cid:107) ˆ v − ¯ x (cid:107) 2 =
O (cid:18)(cid:113)
[Equation]
(cid:19) ¯ λ
[Equation]
− ¯ λ
[Equation]
+ O (cid:32)(cid:115)
δn
[Equation]
m ¯ λ
[Equation]
−
[Equation]
(cid:33) + O   (cid:115) (¯ λ
[Equation]
+ (cid:15)
[Equation]
(cid:107) ¯ x − x
[Equation]
(cid:107)
[Equation]
¯ λ
[Equation]
− ¯ λ
[Equation]

 .
As mentioned earlier, if ˆ v T ¯ x < 0 , we have the same upper bound as in (72) for (cid:107) ˆ v +¯ x (cid:107) 2 .
Therefore, we obtain
[Equation]
¯ λ
[Equation]

 , (76) where (73) follows from (40), (75) follows from (cid:107) ˆ v ± ¯ x (cid:107)
= 2(1 ±
¯ x T ˆ v ) , and (76) follows from (72).
E P ROOF OF L EMMA 2 (A UXILIARY R ESULT FOR PP OWER A NALYSIS )
By the assumption Range( G ) ⊆
S n − 1 , for any x ∈ R n and a > 0 , we have
[Equation]
(77) which is seen by noting that when comparing (cid:107) x − a (cid:107) 2 with (cid:107) x − b (cid:107) 2 (in accordance with projection mapping to the closest point), as long as (cid:107) a (cid:107)
= (cid:107) b (cid:107) 2 , the comparison reduces to comparing (cid:104) x , a (cid:105) with (cid:104) x , b (cid:105) , so is invariant to positive scaling of x .
Let ¯ η = 1 / ¯ λ 1 > 0 and ˆ s = P G ( Vs ) .
Then, we have ˆ s =
P G ( Vs ) =
P G (¯ η Vs ) .
Since ¯ x ∈
Range( G ) , we have
[Equation]
This is equivalent to
[Equation]
and expanding the square gives
[Equation]
Note also that from ¯ V ¯ x = ¯ λ
1 ¯ x , we obtain ¯ x = ¯
η ¯ V ¯ x , which we will use throughout the proof.
For δ > 0 , let M be a ( δ/L ) -net of B k 2 ( r ) ; from Lemma 5, there exists such a net with
[Equation]
By the L -Lipschitz continuity of G , we have that G ( M ) is a δ -net of Range( G )
=
G ( B k 2 ( r )) .
We write s
[Equation]
(82) where ˜ s ∈ G ( M ) satisﬁes (cid:107) ˆ s − ˜ s (cid:107) 2 ≤ δ , and s 0 ∈ G ( M ) satisﬁes (cid:107) s − s 0 (cid:107) 2 ≤ δ .
Then, we have
[Equation]
¯
[Equation]
Vs − ¯ x , ˆ s − ¯ x (cid:105) =
(cid:104)
[Equation]
which follows from V
= ¯ V + E and ¯ x =
¯
η ¯ V ¯ x .
In the following, we control the two terms in (83) separately.
The term (cid:104) ¯ η ¯ V
( s − ¯ x ) , ˆ s − ¯ x (cid:105) :
We decompose s = α ¯ x + β t and ˆ s =
ˆ α ¯ x + ˆ β ˆ t , where (cid:107) t (cid:107)
= (cid:107)
ˆ t (cid:107) 2 = 1 and t T ¯ x = ˆ
t T ¯ x = 0 .
Since (cid:107) s (cid:107) 2 = (cid:107) ˆ s (cid:107) 2 = 1 , we have α 2 + β 2 =
ˆ α 2 + ˆ β 2 = 1 .
In addition, we have α = s T ¯ x
and ˆ α
= ˆ s T ¯ x . Recall that in (47), we write the SVD of ¯ V as ¯ V
= ¯
U ¯ D ¯ U T .
Since t T ¯ x = 0 , we can write t as t =
(cid:80) i> 1 h
i ¯ u i .
In addition, since
(cid:107) t (cid:107) 2 = 1 , we have (cid:80) i> 1 h 2 i = 1 .
Hence, by the Cauchy-Schwarz inequality, we have
[Equation]
Therefore, we obtain
[Equation]
(92) where (89) uses η ¯
V ¯ x = ¯ x , and (90) uses (cid:107) ¯ x (cid:107) 2 = 1 and (cid:104)
¯ x , t (cid:105) = 0 .
The term (cid:104) ¯ η Es , ˆ s − ¯ x (cid:105) :
We have |(cid:104) ¯ η Es ,
[Equation] [Equation]
where (95) follows from Assumption 2 and (81), and (96) follows from ¯ η = 1 / ¯ λ 1 , along with the fact that we assumed ¯ λ 1 = Θ(1) .
Note that (cid:107) ¯ x − ˆ s (cid:107)
=
2(1 − ˆ s T ¯ x ) =
2(1 − ˆ α ) .
Hence, and using (80), (83), (92), and (96), we obtain
[Equation]
[Equation]
Using 2(1 − ˆ α ) − 2(1 − α )(1 −
[Equation]
=
[Equation]
, √
[Equation]
−
[Equation]
= √
[Equation]
−
[Equation]
√ 1 + α ≤ (cid:112) 2(1 − α ) , and similarly √ 1 − ˆ α 2 ≤ (cid:112) 2(1 − ˆ α ) , we obtain from (97) that
[Equation]
Since (cid:107) ˆ s − ¯ x (cid:107) 22 = 2(1 − ˆ α ) and (cid:107) s − ¯ x (cid:107)
= 2(1 − α ) , this is equivalent to
[Equation]
This equation is of the form az 2 ≤ bz + c (where z = (cid:107) ˆ s − ¯ x (cid:107) 2 and a = α == s T ¯ x > 0 ), and using a simple application of the quadratic formula, 9 we obtain
[Equation]
where we use the assumption (cid:107) E (cid:107) 2 → 2 = O ( n/m ) and set δ = 1 /n in (101).
F P ROOF OF T HEOREM 2 (M AIN T HEOREM FOR PP OWER )
Suppose for the time being that (13) holds for at least one index t ≥ t 0 (we will later verify that this is the case), and let T 0 ≥ t 0 be the smallest such index.
Thus, we have
[Equation]
Note that according to the theorem statement, 1 − µ 0 is bounded away from zero.
Using (cid:107) w ( T 0 ) (cid:107)
= (cid:107) ¯ x (cid:107) 2 =
and the assumption that m ≥ C ν,τ · k log( nLr ) with C ν,τ > 0 being large enough, we deduce from (102) that (cid:107) w ( T 0 )
− ¯ x (cid:107) 2 is sufﬁciently small such that
[Equation]
Next, using the assumption 2¯ γ + ν ≤ 1 − τ , we write
[Equation]
Then, from Lemma 2, we obtain
[Equation]
(107) where (106) follows from (102)–(103), and (107) follows from (104).
Thus, we have transferred (102) from T 0 to T 0 + 1 , and proceeding by induction, we obtain [Equation]
[Equation]
for al t ≥ T 0 .
Next, we consider t ∈ [ t 0 , T 0 ) .
Again using Lemma 2 (with ˆ s = w ( t 0 +1)
=
P G ( Vw ( t 0 ) ) ), we have
[Equation]
where we recall that µ 0 = 2¯ γ ¯ x T w ( t 0)
=
2¯ γ 2¯ γ + ν < 1 , and note that the denominator in the second term of (109) follows since ¯ x T w ( t 0 )
=
2¯ γ + ν .
Supposing that t 0 < T 0 (otherwise, the above analysis for t ≥ T 0 alone is sufﬁcient), we have that (13) is reversed at t
= t 0 :
[Equation]
This means that we can upper bound the second term in (109) by Cν · (cid:113) k
[Equation]
¯ x (cid:107) 2 , which gives
[Equation]
Squaring both sides, expanding, and canceling the norms (which all equal one), we obtain
[Equation]
and by induction, we obtain that { ¯ x T w ( t ) } t ∈ [ t 0 ,T 0 ) is monotonically increasing.
Recall that we assume that ¯ λ 1 = Θ(1) , and that T 0 =
t 0 + ∆ 0 is the smallest integer such that (13) holds.
To verify that T 0 is ﬁnite and upper bound ∆ 0 , we consider the following three cases:
• µ 0 =
(or equivalently, ¯ γ =
¯
λ
2 = 0 ):
In this case, (109) gives T 0 =
t 0 +1 (or T 0 = t 0 , which we already addressed above).
Thus, we have ∆ 0 ≤ 1 , as stated in the theorem.
• µ
= o (1) (or equivalently, ¯ γ =
o (1) and ¯ λ
2 = o (1) ):
Since { ¯ x T w ( t ) } t ∈ [ t 0 ,T 0 ) is monotonically increasing, for any positive integer ∆ with t 0 +∆ ≤ T 0 , by applying Lemma 2 (or (109)) multiple times, we obtain
[Equation]
Then, if we choose ∆ 0 ∈ N such that
[Equation]
115) we obtain from (114) that
[Equation]
119) where (117) follows from (cid:107) w ( t 0 )
− ¯ x (cid:107) 2 ≤ 2 , (118) follows from (115) and 1 < 1 1 − µ 0 , and (119) follows from µ 0 = 2¯ γ 2¯ γ + ν , which implies µ 0 ν
+
[Equation]
=
[Equation]
Observe that (119) coincides with (13), and since µ 0 = o (1) , we obtain from (115) that ∆
= O (cid:0) log (cid:0) m k log(
nLr )
(cid:1)(cid:1) as desired.
• µ
= Θ(1) (or equivalently, ¯ γ = Θ(1) and ¯ λ 2 = Θ(1) ):
Recall that we only need to focus on the case T 0 > t 0 .
This means that (110) holds, implying that we can upper bound the second term in (109) by (1 − µ 0 )
ν 2¯ γ + ν · (cid:107) w ( t 0 )
− ¯ x (cid:107) 2 , yielding
[Equation] [Equation]
where ξ = µ 0 ν 2¯ γ + ν = µ 0 (1 − µ 0 )
= Θ(1) .
With the distance to ¯ x shrinking by a constant factor in each iteration according to (122), and the initial distance (cid:107) w ( t 0 ) − ¯ x (cid:107) 2 being at most 2 due to the vectors having unit norm, we deduce that ∆
= O (cid:0) log (cid:0) m k log( nLr ) (cid:1)(cid:1)
iterations sufﬁce to ensure that (13) holds for t
= t 0 + ∆ 0 .
G C OMPARISON OF A NALYSIS TO D ESHPANDE ET AL . (2014)
As mentioned in Section 4, our analysis is signiﬁcantly different from that of Deshpande et al.
despite using a similar assumption on the initialization.
We highlight the differences as follows: 1.
Perhaps the most signiﬁcant difference is that the proof of (Deshpande et al., 2014, Theorem 3) is highly dependent on the Moreau decomposition, which is only valid for a closed convex cone (see (Deshpande et al., 2014, Deﬁnition 1.2)).
In particular, the Moreau decomposition needs to be used at the beginning of the proof of (Deshpande et al., 2014, Theorem 3), such as
Eqs.
(18) and (19) in the supplementary material therein.
We do not see a way for the proof to proceed without the Moreau decomposition, and our Range( G ) may be very different from a convex cone.
We highlight that one key observation in our proof of Lemma 2 (and thus Theorem 2) is that for a generative model G with Range( G ) ⊆
S n − 1 , and any x ∈ R n and a > 0 , we have
P G ( a x ) =
P G ( x ) (Eq. (77)).
This enables us to derive the important equation ˆ s =
P G ( Vs ) =
P G (¯ η Vs ) .
We are not aware of a similar idea being used in the proof of (Deshpande et
al., 2014, Theorem 3).
In the PPower method in (Deshpande et al., 2014), the authors need to add ρ I n with ρ >to the observed data matrix V to improve the convergence.
In particular, they mention in the paragraph before the statement of Theorem 3 that “the memory term ρ v t is necessary for our proof technique to go through”.
In contrast, our proof of Theorem 2 does not require adding such terms, even when our data model is restricted to the spiked Wigner model considered in (Deshpande et al., 2014).
We consider a matrix model that is signiﬁcantly more general than the spiked Wigner model studied in (Deshpande et al., 2014).
H N UMERICAL R ESULTS FOR THE F ASHION -MNIST D ATASET
The Fashion-MNIST dataset consists of Zalando’s article images with a training set of 60 , 000 examples and a test set of 10 , 000 examples.
The size of each image in the Fashion-MNIST dataset is also 28 × 28 , and thus n = 784 .
The generative model G is set to be a boundary-seeking generative adversarial network (BEGAN).
The BEGAN architecture is summarized as follows.
The generator has latent dimension k =and four layers.
The ﬁrst two are fully connected layers with the architecture 62 − 1024 − 6272 , and with ReLU activation functions.
The output of the second layer, reshaped to 128 × 7 × 7 , is forwarded to a deconvolution layer with kernel size 4 and stride 2 .
The third layer uses ReLU activations and has output size 64 × 14 × 14 , where 64 is the number of channels.
The fourth layer is a deconvolution layer with kernel size 4 and strides 2 , and it uses ReLU activations and has output size 1 × 28 × 28 , where the number of channels is 1 .
The BEGAN is trained with a mini-batch size of 256 , a learning rate of 0 . 0002 , and 100 epochs.
The other parameters are the same as those for the MNIST dataset.
We perform two sets of experiments, considering the spiked covariance and phase retrieval models separately.
The corresponding results are reported in Figures 4, 5, 6, and 7.
From these ﬁgures, again, we can observe clear superiority of PPower to Power and TPower .
We note that for the Fashion-MNIST dataset, some of the images are not sparse in the natural basis, but we observe from Figures 4 and 6 that even for the sparsest images (sandals), PPower also signiﬁcantly outperforms TPower.
[Figure]
[Figure]
s i n e S
i m i
l a r i t
[Figure] [Figure]
[Figure]
O r i g i n a l
P o w e r
T P o w e r T P o w e r W
PP o w e r
[Figure]
r i g i n a l
P
[Figure]
w e r
T P o w e r
[Figure] [Figure]
I N UMERICAL R ESULTS FOR THE C ELEB A D ATASET
The CelebA dataset consists of more than 200 , 000 face images of celebrities, where each input image is cropped to a 64 × 64 RGB image with n = 64 × 64 × 3 = 12288 .
The generative model
G is set to be a pre-trained Deep Convolutional Generative Adversarial Networks (DCGAN) model with latent dimension k = 100 .
We use the DCGAN model trained by the authors of (Bora et al., 2017) directly.
We select the best estimate among 2 random restarts.
The Adam optimizer withsteps and a learning rate of 0 .
1 is used for the projection operator.
For the images of the CelebA dataset, the corresponding vectors are clearly not sparse in the natural basis.
To make a fairer comparison to the sparsity-based method TPower , we convert the original images to the wavelet basis, and perform TPower on these converted images.
The obtained results of TPower are then converted back to the vectors in the natural basis.
The corresponding method is denoted by TPowerW , with “W” referring to the conversion to images in the wavelet basis.
In each iteration of TPower and TPowerW , the calculated entries are truncated to zero except for the largest q entries.
For CelebA, q is set to be 2000 .
Other parameters are the same as those for the MNIST dataset.
We also perform two sets of experiments, considering the spiked covariance and phase retrieval models separately.
The corresponding results are reported in Figures 8 and 9.
From these ﬁgures, we can observe the superiority of PPower to Power , TPower and TPowerW , whereas TPowerW only marginally improves over TPower .