Sub-exponential time Sum-of-Squares lower bounds for Principal Components Analysis
s
AfﬁliationAddress
Abstract
Principal Components Analysis (PCA) is a dimension-reduction technique widelyused in machine learning and statistics.
However, due to the dependence of theprincipal components on all the dimensions, the components are notoriously hard tointerpret.
Therefore, a variant known as sparse PCA is often preferred.
Sparse PCAlearns principal components of the data but enforces that such components must be sparse.
This has applications in diverse ﬁelds such as computational biology and image processing.
To learn sparse principal components, it’s well known that standard PCA will not work, especially in high dimensions, and thereforealgorithms for sparse PCA are often studied as a separate endeavor.
Variousalgorithms have been proposed for Sparse PCA over the years, but given how fundamental it is for applications in science, the limits of efﬁcient algorithms are only partially understood.
In this work, we study the limits of the powerful
Sum of Squares (SoS) family of algorithms for Sparse PCA.
SoS algorithms have recently revolutionized robust statistics, leading to breakthrough algorithms for long-standing open problems in machine learning, such as optimally learningmixtures of gaussians, robust clustering, robust regression, etc.
Moreover, it isbelieved to be the optimal robust algorithm for many statistical problems.
Therefore,for sparse PCA, it’s plausible that it can beat simpler algorithms such as diagonalthresholding that have been traditionally used.
In this work, we show that this is notthe case, by exhibiting strong tradeoffs between the number of samples required,the sparsity and the ambient dimension, for which SoS algorithms, even if allowedsub-exponential time, will fail to optimally recover the component.
Our resultsare complemented by known algorithms in literature, thereby painting an almostcomplete picture of the behavior of efﬁcient algorithms for sparse PCA.
Since SoSalgorithms encapsulate many algorithmic techniques such as spectral or statisticalquery algorithms, this solidiﬁes the message that known algorithms are optimal forsparse PCA.
Moreover, our techniques are strong enough to obtain similar tradeoffs
for Tensor PCA, another important higher order variant of PCA with applicationsin topic modeling, video processing, etc.
Introduction
Principal components analysis (PCA) [54] is a popular data processing and dimension reductionroutine that is widely used.
It has numerous applications in Machine Learning, Statistics, Engineering,
Biology, etc.
Given a dataset, PCA projects the data to a lower dimensional space spanned bythe principal components.
The intuition is that PCA sheds lower order information such as noisebut importantly preserves much of the intrinsic information present in the data that are needed fordownstream tasks.
However, despite great optimality properties, PCA has its drawbacks.
Firstly, because theprincipal components are linear combinations of all the original variables, it’s notoriously hard tointerpret them [70].
Secondly, it’s well known that PCA does not yield good estimators in highdimensional settings [12, 79, 53].
To address these issues, a variant of PCA known as Sparse PCA is often used.
Sparse PCAsearches for principal components of the data with the added constraint of sparsity.
Concretely,consider given data v 1 , v 2 , . .
. , v m ∈ R d .
In Sparse PCA, we want to ﬁnd the top principal componentof the data under the extra constraint that it has sparsity at most k .
That is, we want to ﬁnd a vectorv ∈ R d that maximizes (cid:80) mi =1 (cid:104) v, v i (cid:105) 2 such that (cid:107) v (cid:107) 0 ≤ k .
Sparse PCA has enjoyed applications in a diverse range of ﬁelds ranging from medicine, computational biology, economics, image and signal processing, ﬁnance and of course, machine learning andstatistics (e.g. [92, 74, 71, 90, 29, 2]).
Moreover, sparse PCA comes with the important beneﬁt thatthe components are easier to interpret.
A notable example of this is to recover topics from documents[30, 77].
Moreover, this has important beneﬁts for algorithmic fairness in machine learning.
A large volume of research has been devoted to study Sparse PCA and its variants.
Algorithmshave been proposed and studied by several works, e.g. [3, 69, 62, 31, 93, 19, 68, 32, 49, 22, 33, 27, 34].
For example, simple variants of PCA such as thresholding on top of standard PCA [53, 27] workwell in certain parameter settings.
This leads to the natural question whether more sophisticatedalgorithms can do better either for these settings or other parameter settings.
On the other hand, there have been works from the inapproximability perspective as well (e.g.[19, 49, 22, 62, 33, 93], see Section 3.1 for a more detailed overview)
In particular, a lot of theseinapproximability results have relied on various other conjectures, due to the difﬁculty of provingunconditional lower bounds.
Despite these prior works, exactly understanding the limits of efﬁcientalgorithms to this problem is still an active research area.
This is natural considering the importanceof sparse PCA and how fundamental it is to a multitude of applications.
In this work, we focus on the powerful Sum-of-Squares (SoS) family of algorithms [89, 75, 78,44] based on semideﬁnite programming relaxations.
SoS algorithms have recently revolutionizedrobust statistics, leading to breakthrough algorithms for long-standing open problems in machinelearning and statistics [15, 67, 46, 59, 40, 61, 13, 14, 88].
Highlights include
- Robustly learning mixtures of high dimensional Gaussians.
This is an extremely important
problem that has been subjected to intense scrutiny, with a long line of work culminating in
- Efﬁcient algorithms for the fundamental problems of regression [59], moment estimation
[61], clustering [13] and subspace recovery [14] in the presence of outliers.
Also known as
robust machine learning, this setting is more akin to real life data which almost always has
outliers or corrupted data.
Moreover, SoS algorithms are believed to be the optimal robust algorithm for many statisticalproblems.
In a different direction, SoS algorithms have led to the design of fast algorithms forproblems
such as tensor decomposition [48, 88].
Put more concretely, SoS algorithms, also known as the SoS hierarchy or the Lasserre hieararchy,offers a series of convex semideﬁnite programming (SDP) based relaxations to optimization problems.
Due to its ability to capture a wide variety of algorithmic techniques, it has become a fundamental toolin algorithms and optimization.
It was and still remains an extremely versatile tool for combinatorialoptimization [43, 8, 45, 83]) but recently, it is being extensively used in Statistics and Machine
Learning (apart from the references above, see also [16, 17, 47, 81]).
Therefore, we ask (also raised by and posed as an open problem in the works [68, 49, 50])
Can Sum-of-Squares algorithms beat known algorithms for Sparse PCA?
In this work, we show that SoS algorithms cannot beat known spectral algorithms, even ifwe allow sub-exponential time!
Therefore, this suggests that currently used algorithms such asthresholding or other spectral algorithms are in a sense optimal for this problem.
To prove our results, we will consider random instances of Sparse PCA and show that they arenaturally hard for SoS.
In particular, we focus on the Wishart random model of Sparse PCA.
This model is a more natural modeling assumption compared to other random models that have beenstudied before, such as the Wigner random model.
Note importantly that our model assumptions only strengthen our results because we are provingimpossibility results.
In other words, if SoS algorithms do not work for this restricted version ofsparse PCA, then it will not work for more general models, e.g. with general covariance or multiplespikes.
We now describe the model.
The Wishart model of Sparse PCA, also known as the Spiked Covariance model, was originallyproposed by [53].
In this model, we observe m vectors v 1 , . .
v m ∈ R d from the distribution
N (0 , I d + λuu T ) where u is a k -sparse unit vector, that is, (cid:107) u (cid:107) 0 ≤ k and we would like to recoverthe principal component u .
Here, the sparsity of a vector is the number of nonzero entries and λ isknown as the signal-to-noise ratio.
As the signal to noise ratio λ gets lower, it becomes harder and maybe even impossible to recoveru since the signature left by u in the data becomes fainter.
But it’s possible that this may be mitigatedif the number of samples m grows.
Therefore, there is a tradeoff between m, n and k at play here.
Algorithms proposed earlier have been able to recover u at various regimes.
For example, if thenumber of samples is really large, namely m (cid:29) max( dλ , dλ 2 ) , then standard PCA will work.
But ifthis is not the case, we may still be able to recover u by assuming that the sparsity is not too largecompared to the number of samples, namely m (cid:29) k 2 λ 2 .
To do this, we use a variant of standard PCAknown as diagonal thresholding.
Similar results have been obtained for various regimes, while someregimes have resisted attack to algorithms.
Our results here complete the picture by showing that in the regimes that have so far resisted attackby efﬁcient algorithms, the powerful Sum of Squares algorithms also cannot recover the principalcomponent.
We now state our theorem informally, with the formal statement in Theorem 3.1.
Theorem 1.1.
For the Wishart model of Sparse PCA, sub-exponential time SoS algorithms fail torecover the principal component when the number of samples m (cid:28) min( dλ 2 , k 2 λ 2 ) .
In particular, this theorem resolves an open problem posed by [68] and [49, 50].
In almost all other regimes, algorithms to recover the principal component u exist.
We give asummary of such algorithms in Section 3, captured succinctly in Fig.
We say almost all otherregimes because there is one interesting regime, namely dλ 2 ≤ m ≤ min( d,k ) λ marked by light greenin Fig.
1, where we can show that information theoretically, we cannot recover u
but it’s possibleto do hypothesis testing of Sparse PCA.
That is, in this regime, we can distinguish purely randomunspiked samples from the spiked samples.
However, we will not be able to recover the principalcomponent even if we use an exponential time bruteforce algorithm.
We use our techniques to also obtain strong results for the related Tensor Principal componentsanalysis (Tensor PCA) problem.
Tensor PCA, originally introduced by [87], is a generalizationof PCA to higher order tensors.
Formally, given an order k tensor of the form λu ⊗ k + B whereu ∈ R n is a unit vector and B ∈ R [ n ] k has independent Gaussian entries, we would like to recoverthe principal component u .
Here, λ is known as the signal-to-noise ratio.
Tensor PCA is a remarkably useful statistical and computational technique to exploit higher ordermoments of the data.
It was originally envisaged to be applied in latent variable modeling and indeed,it has found multiple applications in this context (e.g. [4, 58, 5]).
Here, a tensor containing statisticsof the input data is computed and then it’s decomposed in order to recover the latent variables.
Because of the technique’s versatility, it has gathered a lot of attention in machine learning withapplications in topic modeling, video processing, collaborative ﬁltering, community detection, etc.(see e.g. [51, 6, 87, 4, 5, 36, 66] and references therein.)
For Tensor PCA, similar to sparse PCA, there has been wide interest in the community to studyalgorithms (e.g. [10, 21, 47, 48, 87, 96, 94, 57, 7]) as well as approximability and hardness (e.g.[73, 64, 23, 49], see Section 3.2 for a more detailed overview).
It’s worth noting that many of thesehardness results are conditional, that is, they rely on various conjectures, sometimes stronger than
P (cid:54) = NP.
Moreover, there has been widespread interest from the statistics community as well, e.g.[52, 80, 65, 25, 26], due to fascinating connections to random matrix theory and statistical physics.
In this work, we study the performance of sub-exponential time Sum of Squares algorithms for
Tensor PCA.
Our main result is stated informally below and formally in Theorem 3.2.
[Figure]
Theorem 1.2.
For Tensor PCA, sub-exponential time SoS algorithms fail to recover the principalcomponent when the signal to noise ratio λ (cid:28) n k 4 .
In particular, this resolves an open question posed by the works [47, 21, 49, 50].
Therefore, our main contributions can be summarized as follows
Despite the huge breakthroughs achieved by Sum-of-Squares algorithms in recent works on
high dimensional statistics
, we show barriers to it for the fundamental problems of Sparse
PCA and Tensor PCA.
We achieve optimal tradeoffs compared to known algorithms, thereby painting a full pic-
ture of the computational thresholds of tractable algorithms.
This suggests that existing
algorithms are preferrable for PCA and its variants.
Prior lower bounds for these problems have either focused on weaker classes of algorithms
or were obtained assuming other hardness conjectures, whereas we prove high degree
sub-exponential time SoS lower bounds without relying on any conjectures.
The Sum of Squares (SoS) hierarchy is a powerful class of algorithms that utilizes the power ofsemideﬁnite programming for optimization problems, which has achieved breakthrough algorithmsfor many problems in machine learning and statistics.
In this section, we brieﬂy describe the sum ofsquares hierarchy of algorithms.
For a more detailed treament with an eye towards applications tomachine learning and statistics, see the ICM survey [83] or the monograph [40].
Given an optimization problem given by a program with polynomial constraints, the SoShierarchy of algorithms gives a family of convex relaxations parameterized by an integer known as itsdegree.
As the degree gets higher, the running time to solve the convex relaxation increases but on theother hand, the relaxation gets stronger and hence serves as a better algorithm.
This offers a smoothtradeoff between running time and the quality of approximation.
In general, we can solve degree- d SoS in n O ( d ) time 1 .
Therefore, constant degree SoS corresponds to polynomial time algorithmswhich in general translates to efﬁcient algorithms.
In this work, we focus on and show limitations ofdegree n ε SoS which corresponds to subexponential running time.
Suppose we are given multivariate polynomials p, g 1 , . .
. , g m on n variables x 1 , . . .
, x n (denotedcollectively by x ) taking real values.
Consider the task:maximize p ( x ) such that g 1 ( x ) = 0 , . .
g m ( x ) =
In general, we could also allow inequality constraints, e.g., g i ( x ) ≥ 0 .
In this work, we onlyhave equality constraints but much of the theory generalizes when we have inequality constraintsinstead.
We now formally describe the Sum of Squares hierarchy of algorithms, via the so-called pseudoexpectation operators.
Deﬁnition 2.1 (Pseudo-expectation values) .
Given multivariate polynomial constraints g 1 =0 ,..., g m = 0 on n variables x 1 , . . .
, x n , degree d pseudo-expectation values are a linear map˜ E from polynomials of x 1 , . . .
, x n of degree at most d to R satisfying the following conditions:
˜ E
[1] = 1 ,
˜ E
[ f · g i ] = 0 for every i ∈ [ m ] and polynomial f such that deg( f · g i ) ≤ d .
˜ E
[ f 2 ] ≥ 0 for every polynomial f such that deg( f 2 ) ≤ d .
Any linear map ˜ E satisfying the above properties is known as a degree d pseudoexpectationoperator satisfying the constraints g 1 = 0 , . . .
, g m = 0 .
The intuition behind pseudo-expectation values is that the conditions on the pseudo-expectationvalues are conditions that would be satisﬁed by any actual expectation operator that takes expectedvalues over a distribution of true optimal solutions, so optimizing over pseudo-expectation valuesgives a relaxation of the problem.
Deﬁnition 2.2 (Degree d SoS) .
The degree d SoS relaxation for the polynomial optimization problemmaximize p ( x ) such that g 1 ( x ) = 0 , . . .
, g m ( x ) =is the program that maximizes ˜ E
[ p ( x )] over all degree d pseudoexpectation operators ˜ E satisfyingthe constraints g 1 = 0 , . . .
, g m = 0 .
The main advantage is that the SoS relaxation can be efﬁciently solved via convex programming!
In particular, Item 3 in Deﬁnition 2.1 is equivalent to a matrix being positive semideﬁnite, thereforethe degree d SoS relaxation can be done via semideﬁnite programming [91].
This meta-algorithm isknown as a degree- d SoS algorithm.
This algorithm runs in n O ( d ) time 1 .
Therefore, constant degree SoS can be solved in polynomial time.
In the next section, we apply SoS on PCA and formally stateour results.
2.1 Related algorithmic techniques
Statistical Query algorithms Statistical query algorithms are another popular restricted class ofalgorithms introduced by [56].
In this model, for a given data distribution, we are allowed to queryexpected value of functions.
Concretely, for a dataset D on R n , we have access to it via an oracle thatgiven as query a function f : R n → [ − 1 , 1] returns E x ∼ D f ( x ) up to some additive adversarial error.
SQ algorithms capture a broad class of algorithms in statistics and machine learning and have beenused to study information-computation tradeoffs [86, 38, 28].
There has also been signiﬁcant worktrying to understand the limits of SQ algorithms (e.g. [38, 39, 32]).
Formally, SQ algorithms and
SoS are in general incomparable.
However, the recent work [24] showed that under mild conditions,low-degree polynomial algorithms (deﬁned next) and statistical query algorithms have equivalentpower.
But also, under these conditions, it’s easy to see that SoS is a more powerful algorithmthan low degree algorithms and hence, SoS algorithms are stronger than statistical query algorithms.
Therefore, SoS lower bounds as shown in this work give strictly stronger evidence of hardness than
SQ lower bounds.
Low degree polynomial algorithms
In statistics, a hypothesis testing problem is a problem wherethe input is sampled from one of two distributions and we would like to identify which distribution itwas sampled from.
In this setting, a low degree polynomial algorithm is to compare the expectationof a low-degree polynomial to try and distinguish the two distributions.
This method has been used toconjecture hardness thresholds for various problems [49, 50, 64].
However, under mild conditions,the SoS hierarchy of algorithms is more powerful than low degree polynomial algorithms [49] andtherefore potentially yields better algorithms.
Therefore, the SoS lower bounds shown in this workare stronger than low degree polynomial lower bounds as well.
Lower bounds for Sparse Principal Components Analysis
In this section, we will state our main results for Sparse PCA and Tensor PCA.
3.1 Sparse PCA
We recall the setting of the Wishart model of Sparse PCA:
We are given v 1 , . .
v m ∈ R d sampledfrom N (0 , I d + λuu T ) where u is a k -sparse unit vector and we wish to recover u .
We will further assume that the entries of u are in {− 1 √ k , 0 , 1 √ k } chosen such that the sparsityis k (and hence, the norm is 1 ).
Note importantly that this assumption is only strengthening ourresult: If SoS cannot solve this problem even for this speciﬁc u , it cannot do any better for the generalproblem with arbitrary u .
Let the vectors from the given dataset be v 1 , . .
. , v m .
Let them form the rows of a matrix
S ∈ R m × d .
Let Σ = 1 m (cid:80) mi
=1 v i
v Ti be the sample covariance matrix.
Then the standard PCAobjective is to maximize x T Σ x .
Therefore, the sparse PCA problem can be rephrased asmaximize m · x T Σ x = m (cid:88) i = (cid:104) x, v i (cid:105) 2 such that x 3 i
= x
i for all i ≤ d and d (cid:88) i =
x 2 i = k where the program variables are x 1 , . . .
, x d .
The constraint x 3 i = x
i enforces that the entries of x arein {− 1 , 0 , 1 } and along with these constraints, the last condition (cid:80) di =1 x 2 i =
k enforces k -sparsity.
Then, the vector u can be recovered by setting u = 1 √ k x .
Now, we will consider the series of convex relaxations for Sparse PCA obtained by SoS algorithms.
In particular, we will consider SoS degree of d ε for a small constant ε > 0 .
Note that thiscorresponds to SoS algorithms of subexponential running time in the input size
d O (1) .
Our main result states that for choices of m below a certain threshold, when the vectorsv 1 , . .
. , v m are sampled from the unspiked standard Gaussian N (0 , I d ) , then sub-exponential time
SoS algorithms will have optimal value close to m + mλ .
This is also the optimal value in thecase when the vectors v 1 , . .
, v m are indeed sampled from the spiked Gaussian N (0 , I d + λuu T ) .
Therefore, SoS is unable to distinguish N (0 , I d ) from N (0 , I d + λuu T ) and hence cannot solvesparse PCA.
Formally,
Theorem 3.1.
For all sufﬁciently small constants ε > 0 , suppose m ≤ d 1 − ε λ 2 , m ≤ k 2 − ε λ 2 , and forsome A > 0 , d A ≤ k ≤ d 1 − Aε , √ λ √ k ≤ d − Aε , then for an absolute constant C > 0 , with highprobability over a random m × d input matrix S with Gaussian entries, the sub-exponential time SoSalgorithm of degree d Cε for sparse PCA has optimal value at least m + mλ − o (1) .
In other words, sub-exponential time SoS cannot certify that for a random dataset with Gaussianentries, there is no unit vector u with k nonzero entries and m · u T Σ u ≈ m + mλ .
The proof isdeferred to the appendix.
A few remarks are in order.1.
Note here that m
+ mλ is approximately the value of the SoS program when the inputvectors v 1 , . .
. , v m are indeed sampled from the spiked model N (0 , I d + λuu T ) .
Therefore,sub-exponential time SoS is unable to distinguish a completely random distribution fromthe spiked distribution and hence is unable to solve sparse PCA.2.
The constant A can be thought of as {+≈ 
+} and it appears for technical reasons, to ensurethat we have sufﬁcient decay in our bounds.
In particular, most values of k, λ fall under theconditions of the theorem.
Informally, our main result says that when m (cid:28) min (cid:16) dλ 2 , k 2 λ 2 (cid:17) , then subexponential time SoScannot recover the principal component u .
This is the content of Theorem 1.1.
Prior work on algorithms Due to its widespread importance, a tremendous amount of work hasbeen devoted to obtaining algorithms for sparse PCA, both theoretically and practically, [3, 69, 62,31, 93, 19, 68, 32, 49, 22, 33, 27, 34] to cite a few.
We now place our result in the context of known algorithms for Sparse PCA and explain why itoffers tight tradeoffs between approximability and inapproximability.
Between this work and priorworks, we completely understand the parameter regimes where sparse PCA is easy or conjecturedto be hard up to polylogarithmic factors.
In Fig.
1a and Fig.
1b, we assign the different parameterregimes into the following categories
- Diagonal thresholding:
In this regime, Diagonal thresholding [53, 3] recovers the sparsevector.
Covariance thresholding [62, 31] and SoS algorithms [35] can also be used in thisregime.
The beneﬁts of these alternate algorithms are that covariance thresholding has betterdependence on logarithmic factors and SoS algorithms works in the presence of adversarial errors.
- Vanilla PCA: Vanilla PCA (i e. standard PCA) can recover the vector, i.e. we do not need to
use the fact that the vector is sparse (see e.g. [20, 35]).
- Spectral: An efﬁcient spectral algorithm recovers the sparse vector (see e.g. [35]).
- Can test but not recover: A simple spectral algorithm can solve the hypothesis testing version
of Sparse PCA
but it is information theoretically impossible to recover the sparse vector [35,Appendix E].
- Hard: A regime where it is conjectured to be hard for algorithms to recover the sparse
principal component.
We discuss this in more detail below.
In Fig.
1a and Fig.
1b, the regimes corresponding to Diagonal thresholding, Vanilla PCA and
Spectral are dark green, while the regimes corresponding to Spectral* and Hard are light green andred respectively
Prior work on hardness Prior works have explored statistical query lower bounds [24], basic SDPlower bounds [62], reductions from conjectured hard problems [20, 19, 22, 41, 93], lower boundsvia the low-degree conjecture [33, 35], lower bounds via statistical physics [33, 11], etc.
We notethat similar threshold behaviors as us have been predicted by [35], but importantly, they assume aconjecture known as the low-degree likelihood conjecture.
Similarly, many of these other lowerbounds rely on various conjectures.
To put in context, the low-degree likelihood conjecture is astronger assumption than P (cid:54) = NP.
In contrast, our results are unconditional and do not assume anyconjectures.
Compared to these other lower bounds, there have only been two prior works on lower boundsagainst SoS algorithms [62, 20, 68] which are only for degree 2 and degree 4 SoS. In particular,degree 2 SoS lower bounds have been studied in [62, 20] although they don’t state it this way.
And[68] obtained degree 4 SoS lower bounds
but they were very lossy, i.e. they hold for a strict subsetof the Hard regime m (cid:28) k 2 λ 2 and m (cid:28) dλ 2 .
Moreover, the ideas used in these prior works do notgeneralize for higher degrees.
The lack of other SoS lower bounds can be attributed to the difﬁcultyin proving such lower bounds.
In this paper, we vastly strengthen these known results and showalmost-tight lower bounds for SoS algorithms of degree d ε which correspond to sub-exponentialrunning time d d O ( ε ) .
We note that SoS algorithms get stronger as the degree increases, thereforeour results immediately imply these prior results and even in the special case of degree 4 SoS, weimprove the known lossy bounds.
In summary, Theorem 3.1 subsumes all these earlier known resultsand is a vast improvement over prior known SoS lower bounds which provides compelling evidencefor the hardness of Sparse PCA in this parameter range.
The work [49] also states SoS lower bounds for Sparse PCA
but it differs from our work in threeimportant aspects.
First, they handle the related but qualitatively different Wigner model of Sparse
PCA.
Their techniques fail for the Wishart model of Sparse PCA, which is more natural in practice.
We overcome this shortcoming and work with the Wishart model.
We emphasize that their techniquesare insufﬁcient to handle this generality and overcoming this is far from being a mere technicality.
On the other hand, our techniques can easily recover their results.
Second, while they sketch a highlevel proof overview for their lower bound, they don’t give a proof.
On the other hand, our proofsare fully explicit.
Finally, they assume the input distribution has entries in {± 1 } , that is, they workwith the ± 1 variant of PCA.
On the other hand, we work with the more realistic setting where thedistribution is N (0 , 1) .
Again, our techniques can easily recover their results as well.
3.2 Tensor PCA
We will now state our main result for Tensor PCA.
Let k ≥ 2 be an integer.
We are given an order ktensor A of the form
A = λu ⊗ k + B where u ∈ R n is a unit vector and B ∈ R [ n ] k has independent
Gaussian entries and we would like to recover the principal component u .
Tensor PCA can berephrased by the program
[Equation]
such that
[Equation]
where the program variables are x 1 , .
, x n .
The principal component u will then just be the returnedsolution x .
We will again consider sub-exponential time SoS algorithms, in particular degree n ε SoS,for this problem.
This is sub-exponential time because the input size is n O (1) .
We then show that if the signal to noise ratio λ is below a certain threshold, then sub-exponentialtime SoS for the unspiked input A ∼ N (0 , I [ n ] k ) will have optimal value close to λ , which is also theoptimal value in the spiked case when A = λu ⊗ k + B, B ∼ N (0 , I [ n ] k ) .
In other words, SoS cannotdistinguish the unspiked and spiked distributions and hence cannot recover the principal componentu .
Theorem 3.2.
Let k ≥ 2 be an integer.
For all sufﬁciently small ε > 0 , if λ ≤ n k 4 − ε , for an absoluteconstant C > 0 , with high probability over a random tensor A ∼ N (0 , I [ n ] k ) , the sub-exponentialtime SoS algorithm of degree n Cε for Tensor PCA has optimal value at least λ − o (1) .
Therefore, sub-exponential time SoS cannot certify that for a random tensor A ∼ N (0 , I [ n ] k ) ,there is no unit vector u such that (cid:104) A, u ⊗ . . . ⊗ u (cid:124)
[Equation] (cid:105) ≈ λ .
The proof is deferred to the appendix.
We again remark that when the tensor A is actually sampled from the spiked model
A =λu ⊗ k + B , the optimal value of the SoS program is approximately λ .
Therefore, this shows thatsub-exponential time SoS algorithms cannot solve Tensor PCA.
Informally, the theorem says that when the signal to noise ratio λ (cid:28) n k 4 , SoS algorithms cannotsolve Tensor PCA, as stated in Theorem 1.2.
Prior work Algorithms for Tensor PCA have been studied in the works [10, 21, 47, 48, 87, 96,94, 57, 7].
It was shown in [21] that the degree q SoS algorithm certiﬁes an upper bound of
2 O ( k ) ( n · polylog ( n )) k/ 4 q k/ 4 − 1 /for
the Tensor PCA problem.
When q = n ε this gives an upper bound ofn k 4 − O ( ε ) .
Therefore, our result is tight, giving insight into the computational threshold for Tensor
PCA.
Lower bounds for Tensor PCA have been studied in various forms including statistical querylower bounds [24, 37], reductions from conjectured hard problems [95, 23], lower bounds from thelow-degree conjecture [49, 50, 64], evidence based on the landscape behavior [9, 73], etc.
Comparedto a lot of these works which rely on various conjectures, we remark that our lower bounds areunconditional and do not rely on any conjectures.
In [49], similar to Sparse PCA, they state a similar theorem for a different variant of Tensor
PCA.
However, they do not give a proof whereas we give explicit proofs.
In particular, they statetheir result without proof for the ± 1 variant of Tensor PCA whereas we work with the more realisticsetting where the distribution is N (0 , 1) .
We remark that their techniques do not recover our resultsbut on the other hand, our techniques can recover theirs.
Related work
As stated in their respective sections, there have been some prior works on (degree at most 4 )
SoSlower bounds on Sparse and Tensor PCA and various other lower bounds that have mostly reliedon various hardness conjectures, some of which are stronger than P (cid:54) = NP .
The lack of results onhigher degree SoS, compared to other models, can be attributed to the difﬁculty of proving such lowerbounds, which we undertake in this work.
Sum of Squares lower bounds have been obtained for various problems of interest, such as
Sherrington-Kirkpatrick Hamiltonian [42, 63], Maximum Cut [72], Constraint Satisfaction Problems[60], etc.
The techniques used in this work are closely related to the work [18] which proved Sum of
Squares lower bounds for a problem known as Planted Clique.
Some of the ideas and techniqueswe employ in this work, namely pseudo-calibration and graph matrices have also appeared in otherworks [72, 84, 42, 1, 55, 85].
It’s plausible that our generalized techniques could be applied to otherhigh dimensional statistical problems, which we leave for future work.
Conclusion
In this work, we show sub-exponential time lower bounds for the powerful Sum-of-Squares algorithmsfor Sparse PCA and Tensor PCA.
With the evergrowing research into better algorithms for Sparse
PCA [3, 69, 62, 31, 93, 19, 68, 32, 49, 22, 33, 34] and Tensor PCA [10, 21, 47, 48, 87, 96, 94, 57, 7],combined with the recent breakthrough of Sum of Squares algorithms in statistics [15, 67, 46, 59, 40,61 , 13, 14, 88], it’s therefore an important goal to understand whether Sum of Squares algorithmscan beat state of the art algorithms for these problems.
In this work, we answer this negatively and show that even sub-exponential time SoS algorithmscannot do much better than relatively simpler algorithms.
In particular, we settle open problems raisedby [68, 49, 50, 47, 21].
Our work does not handle exponential time Ω( n ) degree SoS so analyzingthese algorithms is a potential future direction.
Another important direction is to understand the limitsof powerful algorithms such as SoS for other statistical problems of importance, such as mixturemodeling or clustering.
For algorithm designers, our results illustrates the intrinsic difﬁculty of PCAproblems and sheds light on information-computation gaps exhibited by PCA.
For practitioners, thisresult provides strong evidence that existing algorithms work relatively well.
Broader impacts This is a primarily theoretical work that improves the understanding of thelimits of powerful algorithms on Principal Components Analysis (PCA).
A beneﬁt of our researchis that the limits of powerful algorithms could potentially help devise devise better cryptographicschemes, that protect against nefarious attackers.
An indirect positive impact of our research is toadvise practitioners to focus on non-SoS algorithms for problems like sparse PCA, which may haveimportant consequences if the practitioners are from ﬁelds such as medicine.
A negative impact couldbe the amount of resources used in conducting this research.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5 and the Broaderimpacts statement.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the
Broader impacts statement (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3,Theorem 3.1, Theorem 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the supplementarymaterial.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A] Relevant theoreticalworks have been cited.(b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]