Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
Anonymous Author(s)
AfﬁliationAddress email
Abstract
A simple model to study subspace clustering is the high-dimensional k -Gaussianmixture model where the cluster means are sparse vectors.
Here we provide anexact asymptotic characterization of the statistically optimal reconstruction errorin this model in the high-dimensional regime with extensive sparsity, i.e. whenthe fraction of non-zero components of the cluster means ⇢ , as well as the ratio ↵between the number of samples and the dimension are ﬁxed, while the dimensiondiverges.
We identify the information-theoretic threshold below which obtaining a positive correlation with the true cluster means is statistically impossible.
Additionally, we investigate the performance of the approximate message passing (AMP) algorithm analyzed via its state evolution, which is conjectured to beoptimal among polynomial algorithm for this task.
We identify in particular theexistence of a statistical-to-computational gap between the algorithm that requiresa signal-to-noise ratio  alg
k/ p ↵ to perform better than random, and the information theoretic threshold at  it ⇡ p k ⇢ log ⇢ / p ↵ .
Finally, we discuss the caseof sub-extensive sparsity ⇢ by comparing the performance of the AMP with othersparsity-enhancing algorithms, such as sparse-PCA and diagonal thresholding.
Introduction
With the growing size of modern data clustering techniques play an important role in reducing thedimensionality of the features used in modern Machine Learning pipelines.
Indeed, in many tasksof interest ranging from DNA sequence analysis to image classiﬁcation, the relevant features areknown to live in a lower-dimensional space (intrinsic dimension) than their raw acquisition format(extrinsic dimension) [1].
In these cases, identifying these features can help saving computationalresources while signiﬁcantly improving on learning performance.
But given a corrupted embeddingof low-dimensional features in a high-dimensional space, is it always statistically possible to retrievethem?
And if yes - how can reconstruction be achieved efﬁciently in practice?
In this manuscriptwe address these two fundamental questions in a simple model for subspace clustering: a k -cluster
Gaussian mixture model with sparse centroids.
In this model, the low-dimensional hidden featuresare given by the sparse centroids, which are embedded in a higher dimensional space and corruptedby additive Gaussian noise.
We assume that the number of non-zero components of the centroids aswell as the number of samples scales linearly with the dimension of the embedding space.
Givena ﬁnite sample from the mixture, the goal of the statistician is to cluster the data, i.e. estimate thecentroids (or features) as well as possible.
Let x ⌫ 2 R d , ⌫ 2 [ n ] := { 1 , · · · , n } denote n i.i.d.
data points drawn from an isotropic k -cluster 34 Gaussian mixture:
[Equation]
where p c are the class probabilities, C is the index set ( |C| = k ), µ c 2 R d are the s -sparse vectors 36 representing the means of the clusters and  is a measure of the signal-to-noise ratio (SNR) for thismodel.
In the following, we focus in the balanced case for which p c = 1 / k .
We select the subspace ofrelevant features thanks to the introduction of the vector v i 2 R k . The projection of all the clustermeans, on a given dimension i 2 [ d ] , will be completely determined by a linear combination of thecomponents of v i . We consider for this purpose a Gauss-Bernoulli distribution:
[Equation]
where ⇢ :
= s / d is the density of non-zero elements.
For the convenience of the theoretical analysis thatfollows, it will be useful to work with a particular encoding of the class labels C .
For a given sample⌫ 2 [ n ] belonging to the class c 2 C , deﬁne the following label indicator vector u ⌫ c 2 {  1 k , k  1 k } k :
[Equation]
For a given draw ( x ⌫ ) ⌫ 2 [ n ] , deﬁne the matrices X 2 R d ⇥ n with columns given by x ⌫ and U {+
+} R n ⇥ k , V 2 R d ⇥ k with rows given by u ⌫ c and v i .
A crucial point that we shall exploit in this paper is that, with this notation, our model for subspaceclustering can be written as a matrix factorization problem, where the data has been generated as:
[Equation]
with W 2 R d ⇥ n a Gaussian matrix with elements W i ⌫ ⇠ N (0 , 1) .
This formulation is completelyequivalent to the one in eq.
(1), by identifying the cluster means as given component-wise by:µ ( i ) c = v > i u c , i 2 [ d ] . The centroids will have, in expectation, only s non-zero components; asdiscussed in the introduction, the s -sparse class means { µ c } c 2 C can be thought of as low-dimensionalfeatures embedded in a higher-dimensional space R d , that have been corrupted by isotropic additive Gaussian noise with variance ⇠   1 .
Given a ﬁnite draw ( x ⌫ ) ⌫ 2 [ n ] generated from this model withclass means V ? and labels U ? , the goal of the statistician is to perform clustering , i.e. to estimate U ?
from X, which is equivalent to retrieving the class label for each sample in the data.
However,note that there is a clear class symmetry in this problem: the labels can only be estimated up to apermutation ⇡ {+
+} Sym ( C ) reshufﬂing the columns of U ? .
Taking this symmetry into account we canassess the performance of an estimator ˆ U through the averaged symmetrized mean-squared error: [Equation]
[Equation]
where || · || F denotes the matrix Frobenius norm.
In particular, we will be interested in characterizing 60 reconstruction in the proportional high-dimensional limit where the number of samples n , the ambientdimension d and the sparsity level s go to inﬁnity n, d, s ! 1 at ﬁxed density ⇢ =
s / d , samplecomplexity ↵ = n / d , number of clusters k  2 and signal strength  > 0 .
Note that the clustering problem above is closely related to the problem of estimating the class means/ features V ? .
Indeed, written as in eq.
(4) the problem of estimating both the labels and centroids( U ? , V ? ) boils down to a low-rank matrix factorization problem.
In this manuscript, we have chosento frame the discussion in terms of the clustering, but all our results could be presented also in termsof the reconstruction of the class means.
In this manuscript we provide a sharp asymptotic characterization of when reconstruction, as measuredby positive correlation with the ground truth, is possible in high-dimensions for this model, bothstatistically and algorithmically .
In particular, our main contributions are:
• We map the subspace clustering problem to an asymmetric matrix factorization problem.
Wecompute the closed-form asymptotic formula for the performance of the Bayesian-optimal estimatorin the high-dimensional limit, building on a statistical physics inspired approach [2] that has beenrigorously proven in this case [3, 4] .
This allows us to provide a sharp threshold bellow whichreconstruction of the features is statistically impossible as a function of the parameters of the model.• To estimate the algorithmic limitations of reconstruction, we analyse a tailored approximatemessage passing (AMP) algorithm
[2, 5, 6] approximating the posterior marginals in this problem,and derive the associated state evolution equations characterising its asymptotic performance.
Suchalgorithms are optimal among ﬁrst order methods [7, 8]) and therefore their reconstruction thresholdprovides a bound on the algorithmic complexity clustering in our model.• The two results above allow us to paint a full picture of the statistical-to-algorithmic trade-offsin high-dimensional subspace clustering for the sparse k -Gaussian mixture model, and in particularto identify an algorithmically hard region of the sparsity level ( 1  ⇢ ) vs. signal-to-noise ratio (  )plane for which reconstruction is possible statistically but not algorithmically, see Fig.
Further,we provide a detailed analysis in the high sparsity regime ( ⇢ !
+ ) of how the algorithmically hardregion grows as we increase the number of clusters and the sparsity level.
In particular, the informationtheoretical transition arises at  it ⇡ p k ⇢ log ⇢ / p ↵ , and the algorithmic one at
alg
k / p ↵ .•
The analysis for AMP optimality relies on the ﬁnite ⇢ assumption as n, d ! 1 .
We thus alsoinvestigate the complementary case when the number of non-zero components is of the order s .
nand indeed see that sparse principal component analysis (SPCA) and Diagonal thresholding [9]can perform better than random in this region.
We ﬁnd, however, that this requires s  p n (thus⇢
= o (1) ).
We rephrase our ﬁndings in terms of existing literature on the subspace clustering fortwo-classes Gaussian mixtures [10, 11].
Related works: Subspace clustering is a well-studied topic in classical statistics, with a wide rangeof methods used in practice, see [12] for a review.
Closer to this work is the theoretical line ofwork studying the limitations of clustering in high-dimensions.
Baik, Ben Arous and Péché haveshown that PCA for Gaussian mixture clustering fails to correlate with the mixture means belowcertain threshold known as the BBP transition [13].
For dense Gaussian means, the statistical andalgorithmic limitations of clustering were analysed in different regimes of interest.
Our approach tostudy subspace clustering relies on a mapping to a low-rank matrix factorization problem.
Low-rankmatrix factorization has been extensively studied in the literature, and its asymptotic Bayes-optimalperformance was characterized in [2–5, 14–16].
The construction of AMP algorithms and theassociated state evolutions for matrix factorization was done in [2, 5, 6, 17].
In this work, we leveragethese general results on matrix factorization.
The closest to our work is perhaps [2, 18], wherethe authors characterize the asymptotic reconstruction thresholds for the dense case ( ⇢ = 1 ) in theproportional limit where the number of samples and input dimensions diverge at a constant rate.
Non-asymptotic results were also discussed in [19] which considered a modiﬁcation of Lloyd’salgorithm [20] achieving minimax optimal rate and proving a computational lower bound.
To the bestof our knowledge, the case in which the means are sparse has only been analysed in the regime wherethe number of non-zero components is sub-extensive with respect to the input dimension.
Lowerbounds for the statistically optimal recovery threshold in this regime were given in [21, 22], whilecomputational lower bounds are studied in [23, 24].
Clustering of two component Gaussian mixtureshas been studied in order to shed light on comparison between statistical and algorithmical tractabilityin a sparse scenario in [10, 11, 25].
In particular, [10] conjectured and [11] proved algorithmicbounds for this problem.
They claim that even below the BBP threshold they can build an algorithmachieving exponentially small misclustering error, given that (up-to log factors)
we have s .
p n .
We relate our ﬁndings to their work exploring the extreme sparsity regime in detail.
Main theoretical results
In this section, we introduce the two main technical results allowing us to characterize the limitationsof clustering reconstruction (both statistically and algorithmically) for the model introduced above inthe proportional high-dimensional limit.
Statistical reconstruction:
First, note that up to the permutation symmetry, the estimator minimizing the averaged mean-squared error in eq.
(5) admits a closed-form solution given by the marginalsof the posterior distribution:
[Equation]
In the easy region AMP can achieves Bayes see where the posterior distribution for the model deﬁned in eq.
(4) explicitly reads:
[Equation]
[Equation]
and for convenience we deﬁned the vectors u ⌫ , v i 2 R k which are the rows of U , V, and with theprior distribution P u being the uniform distribution over the indicator vectors deﬁned in eq.
(3) and
P v given by the Gauss-Bernouilli distribution deﬁned in eq.
Although in principle it would be possible to compute the minimum mean-squared error (MMSE)given by the Bayes-optimal estimator in eq.
(6) by sampling from the posterior from eq.
(7), thisis impractical when d, n are large.
For instance, simply computing the different integrals involvedin the evidence Z scale exponentially with the dimensions.
The ﬁrst result consists precisely in aclosed-form solution for the asymptotic performance of the Bayes-optimal estimator:
theoretical result
In the proportional high-dimensional limit where n, d, s ! 1 with ﬁxedratios ⇢ = s / d , ↵ = n / d and ﬁxed  , k , the minimum mean-squared error for the reconstruction of
U 2 R n ⇥ k is given by:
[Equation]
where M ?
u 2 R k ⇥ k is the solution of the following minimization problem:
[Equation]
with:
[Equation]
where we introduced u ⇤ ⇠ P u , v ⇤ ⇠ P v , w ⇠ N (0 , I k ) , and we deﬁned Z u/v as:
[Equation]
Result 1 follows from our mapping of the subspace clustering problem introduced in Sec.
2 to alow-rank matrix factorization form eq.
Indeed, this mapping allows us to leverage a closed-formformula characterizing the asymptotic MMSE for low-rank matrix estimation with generic priors thatwas derived heuristically [2, 18] using the replica method from Statistical Physics and was rigorouslyproven in a series of works [3–5, 14] to the context of subspace clustering.
Our contribution residesin making this connection and drawing the consequences for the subspace clustering problem, anon-trivial endeavour given the complexity of the resulting formulas.
Note that the minimizationproblem eq.
(9) is fundamentally different from the one in eq.
Indeed, the ﬁrst involves only lowdimensional variables, and can be easily solved in a computer, while the latter is a high-dimensionalproblem which is computationally intractable for large d, n .
The other parameter M ? v 2 R k ⇥ k solvingeq.
(9) can be used to characterize the MMSE reconstruction error on V.
Algorithmic reconstruction:
While result 1 allow us to sharply characterize when clustering isstatistically possible, it does not provide us a practical way to estimate the true class labels U ?
2 R n ⇥ kfrom the data
X 2 R d ⇥ n .
In order to provide a bound in the algorithmic complexity clustering,we consider an approximate message passing (AMP) algorithm for our problem.
Message passingalgorithms are a class of ﬁrst order algorithms (scaling as O ( nd ) , the dimensions of the input matrix
X)
designed to approximate the marginals of a target posterior distribution, and which have twovery important features.
First, for a large class of random problems (such as the clustering problemstudied here) AMP provides the best known ﬁrst order method in terms of estimation performance[26–30], and has been rigorously proven to be the optimal for certain problems [7, 8].
Second, theasymptotic performance of AMP can be tracked by a set of low-dimensional state evolution equations[31], meaning that its reconstruction performance can be sharply computed without having to run ahigh-dimensional instance of the problem.
For low-rank matrix factorization problems an associated
AMP algorithm can be derived [5, 6, 17, 32, 33].
Therefore, yet again we leverage the mappingof subspace clustering to a low-rank matrix factorization problem to derive the associated AMPalgorithm 1 with denoising functions ⌘ v , ⌘ u :
[Equation]
As mentioned above, one of they key features of Algorithm 1 is that its asymptotic performance canbe tracked exactly by a set of low-dimensional equations.
Main theoretical result
In the proportional high-dimensional limit where n, d, s ! 1 with ﬁxedratios ⇢ = s / d , ↵ = n / d and ﬁxed  , k , the correlation between the ground truth ( U ? , V ? ) and the
AMP estimators (ˆ U t amp , ˆ V t amp ) at iterate t ,
[Equation]
(15) satisfy the following state evolution equations :
[Equation]
Algorithm
Input: Data X 2 R d ⇥ n
Initialize ˆ v t =
i , ˆ u t =0 ⌫ ⇠ N ( 0 k , ✏ I k ) , ˆ  t =0 u, ⌫ = 0 k ⇥ k , ˆ  t =0 v,i =
k ⇥ k .
for t  t max do
A tu =  s ⇣
ˆ U t ⌘ > ˆ U ,
A tv =
s ⇣
ˆ V t ⌘ > ˆ V
B tv =
q  s X
ˆ U t   s n
P ⌫ =1  tu, ⌫ ˆ V t  1 ,
B tu =
q  s X > V   s d
P i =1  tv,i ˆ U t
Take { b tv,i 2 R k } di =1 , { b tu, ⌫ 2 R k } n ⌫ =1 rows of B tv , B tu ˆ v t
+1 i
= ⌘ v ( A tv , b tv,i ) , ˆ u t +1 ⌫ = ⌘ u ( A tu , b tu, ⌫ ) ˆ  t +1 v,i = @
b ⌘ v ( A tv , b tv,i ) , ˆ  t +1 u, ⌫ = @ b ⌘ u ( A tu , b tv, ⌫ )
Here ˆ U t 2 R n ⇥ k , ˆ V t 2 R d ⇥ k , B tu 2 R n ⇥ k , B tv 2 R d ⇥ k , A tu 2 R k ⇥ k , A tv 2 R k ⇥ k end for
Return: Estimators ˆ v amp ,i , ˆ u amp , ⌫ 2 R k , ˆ  u, ⌫ , ˆ  v,i 
 R k ⇥ k
Moreover, the asymptotic performance of is simply given by: [Equation] Result 2 is a consequence from the general theory connecting AMP algorithms to their state evolution(SE) [31]. In the context of low-rank matrix factorization, a derivation of the state evolution equationsabove from Algorithm 1 were ﬁrst provided for the rank-one case in [5, 6] and were extended togeneral rank and denoising functions in [2, 18].
A crucial observation is that Result 2 is intimatelyrelated to Result 1.
Indeed, the state evolution equations (17) coincide exactly with running gradientdescent on the potential deﬁned in eq.
However, while the performance of the statisticallyoptimal estimator ˆ U bo is given by the ﬁxed point with the minimal value of the potential  rs , theperformance of the AMP estimator ˆ U amp is given by the minima achieved by gradient descent: i.e.the one closest to the initialization M t =0 u .
Therefore, studying both the statistical and algorithmiclimitations of clustering in high-dimensions boils down to the study of the minima of the potential rs , see App.
A for further details.
Reconstruction limits for sparse clustering
These results allow us to paint a full picture for when sparse subspace clustering is possible in themodel deﬁned in Sec. 2 as a function of the quantity of data ↵ , the sparsity 1  ⇢ , the number of clustersk and the SNR  .
Figure 1 summarizes the different reconstruction regimes in the ( ⇢ ,  ) plane fora two-clusters problem at ﬁxed sample complexity ↵ =2 , also known as a phase diagram .
Moreoverthe general considerations on the reconstruction limits for sparse clustering, given by analyzing thetwo-clusters problem, are easily generalizable to the general mixture case and not restricted to thatparticular model, see Sec.
For a ﬁxed sparsity 1  ⇢ , we identify the following regions in Fig.
Impossible phase: There is not enough information in the data matrix X handled to the statistician
to assign cluster membership better than chance for any algorithm .
The Bayes-optimal MMSE is not
better than a random guess.
Clustering (reconstruction of U ?
better than chance) is impossible.
Hard phase: The MMSE is non-trivial, and clustering is statistically possible to some extent, but
the best known polynomial time algorithm, AMP, fails to correlate better than chance with the true
cluster assignment U ? .
Any polynomial-time algorithm is conjectured to fail in this region.
Easy phase:
In the
easy phase, not only clustering is statistically possible, but AMP is able toachieve positive correlation with U ? .
One can also investigate when AMP achieves the Bayes-optimal
MMSE (instead of just positive correlation) leading to the same transition (except from a subtlecorrection very close to the tri-critical point, see the discussion in App.
A).
In Fig.
2, we investigate the different phases presented above by varying the sparsity level 1  ⇢ and SNR  at ﬁxed ↵ .
We compare the performance of AMP with popular spectral algorithms: Principal
Component Analysis (PCA) and Sparse Principal Component Analysis (SPCA).
We can initialize AMP and the SE equations in two different ways: we call uninformative initialization a choice forthe ﬁrst iterates of AMP and SE which assumes no knowledge on the ground truth values; conversely with informative initialization we consider that the statistician has some prior knowledge on thethe ground truth signal.
Note that in a real-life scenario the statistician does not have to the truth.
Yet, as a theoretical tool the informative initialization provides important informationabout the algorithmically hard phase, see App A for a discussion.
The initialization strategy weconsidered in the uninformed case for Algorithm 1 is not the only possible choice, there are smarterways of initialising which can lead to a considerable improvement without explicitly assumingany information about the signal, e.g. spectral initialization [34].
Looking at Fig.
2 we see that SE with uninformed initialization tracks AMP.
Morover we note that increasing the sparsity level,i.e. decreasing ⇢ , the problem becomes algorithmically harder.
This is reﬂected in a discontinousjump in the MSE at  alg which becomes larger.
We observe along the same lines a neat advantageby imposing the sparsity constraint in the spectral algorithm (SPCA) with respect to vanilla one(PCA) as the sparsity grows.
We discuss the details on the numerical simulations in App.
D andthe code is available at https://github.com/anonymous.
Stability analysis and algorithmic threshold
In this section we provide a detailed analytical derivation of the threshold  alg characterizing thealgorithmic reconstruction as a function of the number of clusters and the sparsity.
First, note due tothe permutation symmetry of the clusters the overlap matrices admit the following parametrization:
[Equation]
where J k is the k ⇥ k -matrix with all elements equal to one.
This parametrization is preserved underthe SE iterations.
Therefore, inserting it in the SE equations yield equations for ( m tu , m tv ) :
[Equation]
where we introduced the following update functions:
[Equation]
where S k  1 is the surface of the k -dimensional unitary hypersphere.
Recall that ( m tu , m tv ) 2 [0 , 1] 2fully characterize the reconstruction performance, with ( m 1 u , m 1 v )=(0 , 0) corresponding to the performance of a random guess.
Conversely, ( m 1 u , m 1 v )=(1 , ⇢ ) corresponds to perfect reconstructionof the cluster membership and the sparse cluster means.
One can check that the trivial ﬁxed point( m u , m v )=(0 , 0) is always a ﬁxed point of the SE equations.
Moreover, note that its stability is crucially connected to the algorithmic threshold  alg .
Indeed, if the trivial ﬁxed point ( m u , m v )=(0 , 0)is stable, AMP with an uninformed initialization will always converge to this point - and thereforeachieve random guessing performance.
To study its stability, we expand the update functions around( m u , m v )=(0 , 0) up to the second order.
Expressing everything in terms of m u , we obtain:
[Equation]
This immediately tells us that the trivial ﬁxed point becomes unstable at the algorithmic threshold alg
= k / p ↵ .
This transition is a well-known result in random matrix theory and goes under thename of BBP transition [13].
Despite the fact that we expand around the trivial ﬁxed point, theperturbative method also provides information also about the Bayes-optimal performance, thanksto the general properties that the phase diagrams in Bayes-optimal inference problems must respect[35].
Indeed, a sufﬁcient criterion for the presence of an algorithmically hard phase requires at thealgorithmic threshold: F 00 se (0) > 0 .
The local study predicts that the phase diagram will present analgorithmically hard region for k > k hard = 4 + 2 p ↵ .
The criterion nevertheless is not necessaryin this setting, as we immediately see from the phase diagram for the two-components GMM in Fig.
1: we clearly observe an algorithmically hard region for high-sparsity although k < k hard .
Infact, the information-theoretic threshold  it , deﬁned as the value of the SNR at which the problembecomes statistically possible, cannot be found simply thanks to the expansion around the trivialﬁxed point.
A detailed discussion on the thresholds is given in App.
A, while in App.
B we discusshow to compute  it , which we see plotted in Fig.
1 for the two-classes case.
It is interesting to stressthat the linearization of the SE equation around the uninformative ﬁxed point coincide to study thegradient of the potential  rs in eq. (10) around that point.
Large sparsity regime
We characterize in this section the scaling of the thresholds in the very sparse (and most interesting)regime when ⇢ !
+ .
We highlight the main passages of the computation and focus on the principalresults, see Appendix C for a detailed analysis.
The starting point is the following change of variables:
[Equation]
Inserting these expressions into eq.
(20), we obtain simpliﬁed SE equations for the rescaled overlaps( ˜ m u , ˜ m v ) without any residual dependence on ( ⇢ , ↵ ) : [Equation]
[Equation]
where we introduced the auxiliary function T k ( · ) deﬁned as:
[Equation]
where ⇥ ( · ) is the Heaviside theta function.
By considering the large k expansion of T k ( · ) , we canderive the scaling of the IT threshold with ( k, ⇢ , ↵ ) , see Appendix C for more details.
Putting togetherwith our previous result for  alg from Sec. 5, we obtain the following fundamental result:
[Equation]
These equations completely characterize the behavior at large rank & small (but ﬁnite) sparsity.
Thestatistical-to-computational gap, where AMP is not able to exploit the information on the sparsenature of the cluster means, grows with both the sparsity and the rank.
In App.
C, we further show thatthe large rank expression for the IT threshold  it is accurate already at moderate k ⇡ 10 , see Fig. 9.
It is interesting at this point to reframe our results on the high sparsity regime in the context of theexisting literature.
The results for the IT transition for the detection of two-classes sparse GMM were discussed in [22, 23], and we obtain a consistent scaling in the small ⇢ behaviour.
Despite this fact,the algorithmic bounds of different relevant works for the same problem [10, 11, 23] seem, at ﬁrstsight, to not agree with our ﬁndings.
In particular [11] proves rigorously the existence of an algorithmthat achieves minimax rate under the BBP threshold for clustering of two-classes sparse GMM.
Thisapparent inconsistency is related to different sparsity regimes analyzed.
Indeed, here we investigatethe extensive sparsity regime, i.e. ⇢ =
O (1) , while the guarantees for the efﬁcient algorithmsworking under the BBP threshold require a very high sparsity level, i.e. ⇢ =
o (1) .
This differenceis crucial.
In Fig.
3 we illustrate it by considering the performance of two popular algorithmsfor this problem: Diagonal Thresholding (DT) [9] and SPCA.
For extremely large sparsity, i.e.s = O (1) , these algorithms indeed provide estimators with positive correlation with the true classesbelow the BBP threshold!
However, as soon as we increase the density of non-zero components theperformance strongly deteriorates.
In fact, the transition to random chance performance takes placeas the number of non-zero components approach s .
p n , in agreement with the literature [10, 11].
Conclusion
In this work we considered the problem of clustering k homogeneous Gaussian mixtures withsparse means.
Mapping this to a low-rank matrix factorization problem, we have provided anexact asymptotic characterization of the MMSE in the high dimensional limit.
The Bayes-optimalperformance was compared to AMP, the best known polynomial time algorithm for this problem inthe studied regime.
In the large sparsity regime, we uncovered a large statistical-to-computationalgap as the sparsity level grows, and unveiled the existence of a computationally hard phase .
Inparticular, we have shown that the SNR threshold below which recovery is statistically impossibleis given by  it ⇡ p k ⇢ log ⇢ / p ↵ , while the one for which AMP positively correlates with the groundtruth classes is given by  alg  k / p ↵ .
Our result for the existence of an algorithmically hard regionwas compared with the existing literature for this problem, solving an apparent contradiction dueto the scaling assumption of the sparsity level with the dimension of the features.
We corroboratedour ﬁndings with the help of algorithms for subspace clustering such as sparse principal componentanalysis and diagonal thresholding.
The mapping of the subspace clustering problem to a low-rankmatrix factorization one is ﬂexible and extending this work to more general scenarios is deﬁnitelyinteresting.
The main limitation of the current setting is the Gaussian assumption for the noise.
Onenatural idea to go beyond this limitation is to consider other ﬂavours of message passing algorithms,such as vector AMP (VAMP) [36] to study more general rotationally invariant noise distributions.
Afurther prospective direction is to consider inhomegeneous mixture models, i.e. p c 6 = 1 / k in eq.
(1),in order to study the effect of unbalancedness on the statistical-to-computational gap.