Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
Abstract
A simple model to study subspace clustering is the high-dimensional k -Gaussian mixture model where the cluster means are sparse vectors.
Here we provide an exact asymptotic characterization of the statistically optimal reconstruction error in this model in the high-dimensional regime with extensive sparsity, i.e. when the fraction of non-zero components of the cluster means ⇢ , as well as the ratio ↵ between the number of samples and the dimension are ﬁxed, while the dimension diverges.
We identify the information-theoretic threshold below which obtaining a positive correlation with the true cluster means is statistically impossible.
Additionally, we investigate the performance of the approximate message passing (AMP) algorithm analyzed via its state evolution, which is conjectured to be optimal among polynomial algorithm for this task.
We identify in particular the existence of a statistical-to-computational gap between the algorithm that requires a signal-to-noise ratio  alg  k/ p ↵ to perform better than random, and the information theoretic threshold at  it ⇡ p k ⇢ log ⇢ / p ↵ .
Finally, we discuss the case of sub-extensive sparsity ⇢ by comparing the performance of the AMP with other sparsity-enhancing algorithms, such as sparse-PCA and diagonal thresholding.
Introduction
With the growing size of modern data clustering techniques play an important role in reducing the dimensionality of the features used in modern Machine Learning pipelines.
Indeed, in many tasks of interest ranging from DNA sequence analysis to image classiﬁcation, the relevant features are known to live in a lower-dimensional space (intrinsic dimension) than their raw acquisition format (extrinsic dimension)
In these cases, identifying these features can help saving computational resources while signiﬁcantly improving on learning performance.
But given a corrupted embedding of low-dimensional features in a high-dimensional space, is it always statistically possible to retrieve them?
And if yes - how can reconstruction be achieved efﬁciently in practice?
In this manuscript we address these two fundamental questions in a simple model for subspace clustering: a k -cluster
Gaussian mixture model with sparse centroids.
In this model, the low-dimensional hidden features are given by the sparse centroids, which are embedded in a higher dimensional space and corrupted by additive Gaussian noise.
We assume that the number of non-zero components of the centroids as well as the number of samples scales linearly with the dimension of the embedding space.
Given a ﬁnite sample from the mixture, the goal of the statistician is to cluster the data, i.e. estimate the centroids (or features) as well as possible.
Model & setting
Let x ⌫ 2 R d , ⌫ 2 [ n ] := { 1 , · · · , n } denote n i.i.d.
data points drawn from an isotropic k -cluster Gaussian mixture:
[Equation]
where p c are the class probabilities, C is the index set ( |C| = k ), µ c 2 R
d are the s -sparse vectors representing the means of the clusters and  is a measure of the signal-to-noise ratio (SNR) for this model.
In the following, we focus in the balanced case for which p c = 1 / k .
We select the subspace of relevant features thanks to the introduction of the vector v i 2 R k .
The projection of all the cluster means, on a given dimension
i 2 [ d ] , will be completely determined by a linear combination of the components of v i .
We consider for this purpose a Gauss-Bernoulli distribution:
[Equation]
where ⇢ : = s / d is the density of non-zero elements.
For the convenience of the theoretical analysis that follows, it will be useful to work with a particular encoding of the class labels C .
For a given sample ⌫ 2 [ n ] belonging to the class c 2 C , deﬁne the following label indicator vector u ⌫ c 2 {  1 k , k  [Equation] k } k :
[Equation]
For a given draw ( x ⌫ ) ⌫ 2 [ n ] , deﬁne the matrices X 2 R d ⇥ n with columns given by x ⌫ and U
R n ⇥ k , V 2 R d ⇥ k with rows given by u ⌫ c and v i .
A crucial point that we shall exploit in this paper is that, with this notation, our model for subspace clustering can be written as a matrix factorization problem, where the data has been generated as:
[Equation]
with W 2 R d ⇥ n a Gaussian matrix with elements W i ⌫ ⇠ N (0 , 1) .
This formulation is completely equivalent to the one in eq.
(1), by identifying the cluster means as given component-wise by: µ ( i ) c = v > i u c , i 2 [ d ] .
The centroids will have, in expectation, only s non-zero components; as discussed in the introduction, the s -sparse class means { µ c } c 2 C can be thought of as low-dimensional features embedded in a higher-dimensional space R d , that have been corrupted by isotropic additive
Gaussian noise with variance ⇠   1 .
Given a ﬁnite draw ( x ⌫ ) ⌫ 2 [ n ] generated from this model with class means V ? and labels U ? , the goal of the statistician is to perform clustering , i.e. to estimate
U ? from X, which is equivalent to retrieving the class label for each sample in the data.
However, note that there is a clear class symmetry in this problem: the labels can only be estimated up to a permutation ⇡ 2 Sym ( C ) reshufﬂing the columns of U ? .
Taking this symmetry into account we can assess the performance of an estimator ˆ U through the averaged symmetrized mean-squared error:
[Equation]
where || · || F denotes the matrix Frobenius norm.
In particular, we will be interested in characterizing reconstruction in the proportional high-dimensional limit where the number of samples n , the ambient dimension d and the sparsity level s go to inﬁnity n, d, s ! 1 at ﬁxed density ⇢ =
s / d , sample complexity ↵ = n / d , number of clusters k  2 and signal strength  > 0 .
Note that the clustering problem above is closely related to the problem of estimating the class means / features V ? .
Indeed, written as in eq.
(4) the problem of estimating both the labels and centroids ( U ? , V ? ) boils down to a low-rank matrix factorization problem.
In this manuscript, we have chosen to frame the discussion in terms of the clustering, but all our results could be presented also in terms of the reconstruction of the class means.
In this manuscript we provide a sharp asymptotic characterization of when reconstruction, as measured by positive correlation with the ground truth, is possible in high-dimensions for this model, both statistically and algorithmically .
In particular, our main contributions are:
• We map the subspace clustering problem to an asymmetric matrix factorization problem.
We compute the closed-form asymptotic formula for the performance of the Bayesian-optimal estimator in the high-dimensional limit, building on a statistical physics inspired approach [2] that has been rigorously proven in this case [3, 4] .
This allows us to provide a sharp threshold bellow which reconstruction of the features is statistically impossible as a function of the parameters of the model.
• To estimate the algorithmic limitations of reconstruction, we analyse a tailored approximate message passing (AMP) algorithm [2, 5, 6] approximating the posterior marginals in this problem, and derive the associated state evolution equations characterising its asymptotic performance.
Such algorithms are optimal among ﬁrst order methods [7, 8]) and therefore their reconstruction threshold provides a bound on the algorithmic complexity clustering in our model.
• The two results above allow us to paint a full picture of the statistical-to-algorithmic trade-offs in high-dimensional subspace clustering for the sparse k -Gaussian mixture model, and in particular to identify an algorithmically hard region of the sparsity level ( 1  ⇢ ) vs. signal-to-noise ratio (  ) plane for which reconstruction is possible statistically but not algorithmically, see Fig.
Further, we provide a detailed analysis in the high sparsity regime ( {+⇢ ! 
+} + ) of how the algorithmically hard region grows as we increase the number of clusters and the sparsity level.
In particular, the information theoretical transition arises at  it ⇡ p k ⇢ log ⇢ / p ↵ , and the algorithmic one at  alg  k / p ↵ .
• The analysis for AMP optimality relies on the ﬁnite ⇢ assumption as n, d ! 1 .
We thus also investigate the complementary case when the number of non-zero components is of the order s . n and indeed see that sparse principal component analysis (SPCA) and Diagonal thresholding [9] can perform better than random in this region.
We ﬁnd, however, that this requires s  p n (thus ⇢ = o (1) ).
We rephrase our ﬁndings in terms of existing literature on the subspace clustering for two-classes Gaussian mixtures [10, 11].
Related works: Subspace clustering is a well-studied topic in classical statistics, with a wide range of methods used in practice, see [12] for a review.
Closer to this work is the theoretical line of work studying the limitations of clustering in high-dimensions.
Baik, Ben Arous and Péché have shown that PCA for Gaussian mixture clustering fails to correlate with the mixture means below certain threshold known as the BBP transition [13].
For dense Gaussian means, the statistical and algorithmic limitations of clustering were analysed in different regimes of interest.
Our approach to study subspace clustering relies on a mapping to a low-rank matrix factorization problem.
Low-rank matrix factorization has been extensively studied in the literature, and its asymptotic Bayes-optimal performance was characterized in [2–5, 14–16].
The construction of AMP algorithms and the associated state evolutions for matrix factorization was done in [2, 5, 6, 17].
In this work, we leverage these general results on matrix factorization.
The closest to our work is perhaps [2, 18], where the authors characterize the asymptotic reconstruction thresholds for the dense case ( ⇢ = 1 ) in the proportional limit where the number of samples and input dimensions diverge at a constant rate.
Non-asymptotic results were also discussed in [19] which considered a modiﬁcation of Lloyd’s algorithm [20] achieving minimax optimal rate and proving a computational lower bound.
To the best of our knowledge, the case in which the means are sparse has only been analysed in the regime where the number of non-zero components is sub-extensive with respect to the input dimension.
Lower bounds for the statistically optimal recovery threshold in this regime were given in [21, 22], while computational lower bounds are studied in [23, 24].
Clustering of two component Gaussian mixtures has been studied in order to shed light on comparison between statistical and algorithmical tractability in a sparse scenario in [10, 11, 25].
In particular, [10] conjectured and [11] proved algorithmic bounds for this problem.
They claim that even below the BBP threshold they can build an algorithm
We relate our ﬁndings to their work exploring the extreme sparsity regime in detail.
Main theoretical results
In this section, we introduce the two main technical results allowing us to characterize the limitations of clustering reconstruction (both statistically and algorithmically) for the model introduced above in the proportional high-dimensional limit.
Statistical reconstruction:
First, note that up to the permutation symmetry, the estimator minimizing the averaged mean-squared error in eq.
(5) admits a closed-form solution given by the marginals of the posterior distribution:
[Equation]
where the posterior distribution for the model deﬁned in eq.
(4) explicitly reads:
[Equation]
and for convenience we deﬁned the vectors u ⌫ ,
[Equation]
k which are the rows of U , V, and with the prior distribution P u being the uniform distribution over the indicator vectors deﬁned in eq.
(3) and
P v given by the Gauss-Bernouilli distribution deﬁned in eq.
Although in principle it would be possible to compute the minimum mean-squared error (MMSE) given by the Bayes-optimal estimator in eq.
(6) by sampling from the posterior from eq.
(7), this is impractical when d, n are large.
For instance, simply computing the different integrals involved in the evidence Z scale exponentially with the dimensions.
The ﬁrst result consists precisely in a closed-form solution for the asymptotic performance of the Bayes-optimal estimator:
Main theoretical result
In the proportional high-dimensional limit where n, d, s ! 1 with ﬁxed ratios ⇢ = s / d , ↵ = n / d and ﬁxed  , k , the minimum mean-squared error for the reconstruction of
U 2 R n ⇥ k is given by:
[Equation]
where M ? u 2 R k ⇥ k is the solution of the following minimization problem:
[Equation]
9) with:
[Equation]
where we introduced u ⇤ ⇠ P u , v ⇤ ⇠ P v , w ⇠ N (0 , I k ) , and we deﬁned Z u/v as:
[Equation]
Result 1 follows from our mapping of the subspace clustering problem introduced in Sec.
2 to a low-rank matrix factorization form eq.
Indeed, this mapping allows us to leverage a closed-form formula characterizing the asymptotic MMSE for low-rank matrix estimation with generic priors that was derived heuristically [2, 18] using the replica method from Statistical Physics and was rigorously proven in a series of works [3–5, 14] in the context of subspace clustering.
Our contribution resides in making this connection and drawing the consequences for the subspace clustering problem, a non-trivial endeavour given the complexity of the resulting formulas.
Note that the minimization problem eq.
(9) is fundamentally different from the one in eq.
Indeed, the ﬁrst involves only lowdimensional variables, and can be easily solved in a computer, while the latter is a high-dimensional problem which is computationally intractable for large d, n .
The other parameter M ? v 2 R k ⇥ k solving eq.
(9) can be used to characterize the MMSE reconstruction error on V.
Algorithmic reconstruction:
While result 1 allow us to sharply characterize when clustering is statistically possible, it does not provide us a practical way to estimate the true class labels U ? 2 R n ⇥ k from the data X 2 R d ⇥ n .
In order to provide a bound in the algorithmic complexity clustering, we consider an approximate message passing (AMP) algorithm for our problem.
Message passing algorithms are a class of ﬁrst order algorithms (scaling as O ( nd ) , the dimensions of the input matrix
X)
designed to approximate the marginals of a target posterior distribution, and which have two very important features.
First, for a large class of random problems (such as the clustering problem studied here) AMP provides the best known ﬁrst order method in terms of estimation performance [26–30], and has been rigorously proven to be the optimal for certain problems [7, 8].
Second, the asymptotic performance of AMP can be tracked by a set of low-dimensional state evolution equations [31], meaning that its reconstruction performance can be sharply computed without having to run a high-dimensional instance of the problem.
For low-rank matrix factorization problems an associated
AMP algorithm can be derived [5, 6, 17, 32, 33].
Therefore, yet again we leverage the mapping of subspace clustering to a low-rank matrix factorization problem to derive the associated AMP algorithm 1 with denoising functions ⌘ v , ⌘ u :
[Equation]
As mentioned above, one of they key features of Algorithm 1 is that its asymptotic performance can be tracked exactly by a set of low-dimensional equations.
Algorithm
Input: Data X 2 R d ⇥ n
Initialize ˆ v t =
[-i , ˆ u t =
 ν ∼ N ( [Equation]-]
i , ˆ u t =0 ⌫ ⇠ N ( 0 k , ✏ I k ) , ˆ  t =0 u, ⌫ = [-
-]0 k ⇥ k , ˆ  t =0 v,i =
k ⇥ k .
for t  t max do
A tu =  s ⇣ ˆ U t ⌘ > ˆ U ,
A tv =
s ⇣ ˆ V t ⌘ > ˆ V
B tv =
q  s X ˆ U t   s n P ⌫ =1  tu, ⌫ ˆ V t  1 ,
B tu =
q  s X > V   s d
P i =1  tv,i ˆ U t
Take { b tv,i 2 R k } di =1 , { b tu, ⌫ 2 R k } n ⌫ =1 rows of B tv , B tu ˆ v t
+1 i = ⌘ v ( A tv , b tv,i ) , ˆ u t +1 ⌫ = ⌘ u ( A tu , b tu, ⌫ ) ˆ  t +1 v,i = @
b ⌘ v ( A tv , b tv,i ) , ˆ  t +1 u, ⌫ = @ b ⌘ u ( A tu , b tv, ⌫ )
Here ˆ U t 2 R n ⇥ k , ˆ V t 2 R d ⇥ k , B tu 2 R n ⇥ k , B tv 2 R d ⇥ k , A tu 2 R k ⇥ k , A tv 2 R k ⇥ k end for
Return: Estimators ˆ v amp ,i , ˆ u amp , ⌫ 2 R k , ˆ  u, ⌫ , ˆ  v,i {+
+}
R k ⇥ k
Main theoretical result
In the proportional high-dimensional limit where n, d, s ! 1 with ﬁxed ratios ⇢ = s / d , ↵ = n / d and ﬁxed  , k , the correlation between the ground truth ( U ? , V ? ) and the
AMP estimators (ˆ U t amp , ˆ V t amp ) at iterate t ,
[Equation]
(15) satisfy the following state evolution equations :
[Equation]
Moreover, the asymptotic performance of is simply given by:
[Equation]
Result 2 is a consequence from the general theory connecting AMP algorithms to their state evolution (SE) [31].
In the context of low-rank matrix factorization, a derivation of the state evolution equations above from Algorithm 1 were ﬁrst provided for the rank-one case in [5, 6] and were extended to general rank and denoising functions in [2, 18].
A crucial observation is that Result 2 is intimately related to Result 1.
Indeed, the state evolution equations (17) coincide exactly with running gradient descent on the potential deﬁned in eq.
While the performance of the statistically optimal estimator ˆ U bo is given by the ﬁxed point with the minimal value of the potential  rs , the performance of the AMP estimator ˆ U amp is described by the closest minima to the initialization M t =0 u .
Therefore, studying both the statistical and algorithmic limitations of clustering in high-dimensions boils down to the study of the minima of the potential  rs , see App.
A for further details.
The identiﬁcation of a subspace clustering problem with a matrix factorization one allows us to leverage the rich literature from this ﬁeld.
However, analyzing these formulas is far from trivial and consitutes a considerable technical challenge.
The major difﬁculty is to ﬁnd a suitable parametrization of the overlap matrices that reduces the number of parameters to be tracked, we discuss this in detail in Sec. 5.
Moreover, we ﬁnd the scaling ansatz in the high-sparsity limit that closes the equations on amenable quantities in order to ﬁnd analytically the statistical-to-computational gap for the large-rank setting, see Sec. 6.
However, we stress that we did not take all the necessary precautions to claim full rigor. e.g. prove formally that the minimum is unique (although we checked all these both analytically and numerically).
Reconstruction limits for sparse clustering
These results allow us to paint a full picture for when sparse subspace clustering is possible in the model deﬁned in Sec. 2 as a function of the quantity of data ↵ , the sparsity 1  ⇢ , the number of clusters k and the SNR  .
Figure 1 summarizes the different reconstruction regimes in the ( ⇢ ,  ) plane for a two-clusters problem at ﬁxed sample complexity ↵ =2 , also known as a phase diagram .
Moreover the general considerations on the reconstruction limits for sparse clustering, given by analyzing the two-clusters problem, are easily generalizable to the general mixture case and not restricted to that particular model, see Sec. 5.
For a ﬁxed sparsity 1  ⇢ , we identify the following regions in Fig.
Impossible phase: There is not enough information in the data matrix X handled to the statistician to assign cluster membership better than chance for any algorithm .
The Bayes-optimal MMSE is not better than a random guess.
Clustering (reconstruction of U ? better than chance) is impossible.
Hard phase: The MMSE is non-trivial, and clustering is statistically possible to some extent, but the best known polynomial time algorithm, AMP, fails to correlate better than chance with the true cluster assignment U ? .
Any polynomial-time algorithm is conjectured to fail in this region.
Easy phase: In the easy phase, not only clustering is statistically possible, but AMP is able to achieve positive correlation with U ? .
One can also investigate when AMP achieves the Bayes-optimal
MMSE (instead of just positive correlation) leading to the same transition (except from a subtle correction very close to the tri-critical point, see the discussion in App.
A).
In Fig.
2, we investigate the different phases presented above by varying the sparsity level 1  ⇢ and
SNR  at ﬁxed ↵ .
We compare the performance of AMP with popular spectral algorithms: Principal
Component Analysis (PCA) and Sparse Principal Component Analysis (SPCA).
We can initialize
AMP and the SE equations in two different ways: we call uninformative initialization a choice for the ﬁrst iterates of AMP and SE which assumes no knowledge on the ground truth values; conversely with informative initialization we consider that the statistician has some prior knowledge on the the ground truth signal.
Note that in a real-life scenario the statistician does not have access to the ground truth.
Yet, as a theoretical tool the informative initialization provides important information about the algorithmically hard phase, see App A for a discussion.
The initialization strategy we considered in the uninformed case for Algorithm 1 is not the only possible choice, there are smarter ways of initialising which can lead to a considerable improvement without explicitly assuming any information about the signal, e.g. spectral initialization [34].
Looking at Fig.
2 we see that
SE with uninformed initialization tracks AMP.
Morover we note that increasing the sparsity level, i.e. decreasing ⇢ , the problem becomes algorithmically harder.
This is reﬂected in a discontinous jump in the MSE at  alg which becomes larger.
We observe along the same lines a neat advantage by imposing the sparsity constraint in the spectral algorithm (SPCA) with respect to vanilla one (PCA) as the sparsity grows.
We discuss the details on the numerical simulations in App.
D and the code is available at https://github.com/lucpoisson/SubspaceClustering.
Stability analysis and algorithmic threshold
In this section we provide a detailed analytical derivation of the threshold  alg characterizing the algorithmic reconstruction as a function of the number of clusters and the sparsity.
First, note due to the permutation symmetry of the clusters the overlap matrices admit the following parametrization:
[Equation]
[Equation]
where J k is the k ⇥ k -matrix with all elements equal to one.
This parametrization is preserved under the SE iterations.
Therefore, inserting it in the SE equations yield equations for ( m tu , m tv ) :
[Equation] [Equation]
where we introduced the following update functions:
[Equation]
where S k  1 is the surface of the k -dimensional unitary hypersphere.
Recall that ( m tu , m tv ) 2 [0 , 1]fully characterize the reconstruction performance, with ( m 1 u , m 1 v )=(0 , 0) corresponding to the performance of a random guess.
Conversely, ( m 1 u , m 1 v )=(1 , ⇢ ) corresponds to perfect reconstruction of the cluster membership and the the
( m u , m v )=(0 , 0) is always a ﬁxed point of the SE equations.
Moreover, its stability is cru cially connected to the algorithmic threshold  alg .
Indeed, if the trivial ﬁxed point ( m u , m v )=(0 , 0) is stable, AMP with an uninformed initialization will always converge to this point - and therefore achieve random guessing performance.
To study its stability, we expand the update functions around ( m u , m v )=(0 , 0) up to the second order.
Expressing everything in terms of m u , we obtain:
[Equation]
This immediately tells us that the trivial ﬁxed point becomes unstable at the algorithmic threshold  alg = k / p ↵ .
This transition is a well-known result in random matrix theory and goes under the name of BBP transition [13].
Despite the fact that we expand around the trivial ﬁxed point, the perturbative method also provides information also about the Bayes-optimal performance, thanks to the general properties that the phase diagrams in Bayes-optimal inference problems must respect [35].
Indeed, a sufﬁcient criterion for the presence of an algorithmically hard phase requires at the algorithmic threshold: F 00 se (0) > 0 .
The local study predicts that the phase diagram will present an algorithmically hard region for k > k hard = 4+2 p ↵ .
The criterion nevertheless is not necessary in this setting, as we immediately see from the phase diagram for the two-components GMM in Fig.
1: we clearly observe an algorithmically hard region for high-sparsity although k < k hard .
In fact, the information-theoretic threshold  it , deﬁned as the value of the SNR at which the problem becomes statistically possible, cannot be found simply thanks to the expansion around the trivial ﬁxed point.
Indeed the exact computation of  it requires evaluating the potential function  in eq.
(10): one needs to ﬁnd the threshold value of the SNR at which the informative ﬁxed point has a lower value of the free energy with respect to the uninformative ﬁxed point.
The technical discussion on the calculation of  it , which we see plotted in Fig.
1 for the two-classes case, is given in App.
B, while a general overview on the different thresholds is given in App.
A.
It is interesting to stress that the linearization of the SE equation around the trivial ﬁxed point in eq.
(23) coincide to study the gradient of the potential  rs in eq.
(10) around that point, and hence the stability of the trivial ﬁxed point is determined by the potential landscape.
This connection is at the roots of the conjectured link between the hardness of an inference task and AMP: the statistical-to-computational gap opens up as the ﬁxed point with minimal value of the potential in eq.
(10), describing the performance of the Bayes-optimal estimator, is attained far from a locally stable region around the uninformative ﬁxed point, where the AMP algorithm gets stuck.
Large sparsity regime
We characterize in this section the scaling of the thresholds in the very sparse (and most interesting) regime when {+⇢ ! 
+} + .
We highlight the main passages of the computation and focus on the principal results, see Appendix C for a detailed analysis.
The starting point is the following change of variables:
[Equation]
Inserting these expressions into eq.
(20), we obtain simpliﬁed SE equations for the rescaled overlaps ( ˜ m u , ˜ m v ) without any residual dependence on ( ⇢ , ↵ ) :
[Equation]
where we introduced the auxiliary function T k ( · ) deﬁned as:
[Equation]
where ⇥ ( · ) is the Heaviside theta function.
By considering the large k expansion of T k ( · ) , we can derive the scaling of the IT threshold with ( k, ⇢ , ↵ ) , see Appendix C for more details.
Putting together with our previous result for  alg from Sec. 5, we obtain the following fundamental result:
[Equation]
These equations completely characterize the behavior at large rank & small (but ﬁnite) sparsity.
The statistical-to-computational gap, where AMP is not able to exploit the information on the sparse nature of the cluster means, grows with both the sparsity and the rank.
In App.
C, we further show that the large rank expression for the IT threshold  it is accurate already at moderate k ⇡ 10 , see Fig. 9.
It is interesting at this point to reframe our results on the high sparsity regime in the context of the existing literature.
The results for the IT transition for the detection of two-classes sparse GMM were discussed in [22, 23], and we obtain a consistent scaling in the small ⇢ behaviour.
Despite this fact, the algorithmic bounds of different relevant works for the same problem [10, 11, 23] seem, at ﬁrst sight, to not agree with our ﬁndings.
In particular [11] proves rigorously the existence of an algorithm that achieves minimax rate under the BBP threshold for clustering of two-classes sparse GMM.
This apparent inconsistency is related to different sparsity regimes analyzed.
Indeed, here we investigate the extensive sparsity regime, i.e. ⇢ = O (1) , while the guarantees for the efﬁcient algorithms working under the BBP threshold require a very high sparsity level, i.e. ⇢ = o (1) .
This difference is crucial.
In Fig.
3 we illustrate it by considering the performance of two popular algorithms for this problem: Diagonal Thresholding (DT) [9] and SPCA.
For extremely large sparsity, i.e. s = O (1) , these algorithms indeed provide estimators with positive correlation with the true classes below the BBP threshold!
However, as soon as we increase the density of non-zero components the performance strongly deteriorates.
In fact, the transition to random chance performance takes place as the number of non-zero components approach s . p n , in agreement with the literature [10, 11].
Conclusion
In this work we considered the problem of clustering k homogeneous Gaussian mixtures with sparse means.
Mapping this to a low-rank matrix factorization problem, we have provided an exact asymptotic characterization of the MMSE in the high dimensional limit.
The Bayes-optimal performance was compared to AMP, the best known polynomial time algorithm for this problem in the studied regime.
In the large sparsity regime, we uncovered a large statistical-to-computational gap as the sparsity level grows, and unveiled the existence of a computationally hard phase .
In particular, we have shown that the SNR threshold below which recovery is statistically impossible is given by  it ⇡ p  k ⇢ log ⇢ / p ↵ , while the one for which AMP positively correlates with the ground truth classes is given by  alg  k / p ↵ .
Our result for the existence of an algorithmically hard region was compared with the existing literature for this problem, solving an apparent contradiction due to the scaling assumption of the sparsity level with the dimension of the features.
We corroborated our ﬁndings with the help of algorithms for subspace clustering such as sparse principal component analysis and diagonal thresholding.
The mapping of the subspace clustering problem to a low-rank matrix factorization one is ﬂexible and extending this work to more general scenarios is deﬁnitely interesting.
The main limitation of the current setting is the Gaussian assumption for the noise.
One natural idea to go beyond this limitation is to consider other ﬂavours of message passing algorithms, such as vector AMP (VAMP) [36] to study more general rotationally invariant noise distributions.
A further prospective direction is to consider inhomegeneous mixture models, i.e. p c 6 = 1 / k in eq.
(1), in order to study the effect of unbalancedness on the statistical-to-computational gap.
Acknowledgment
We thank Maria Reﬁnetti for discussions.
This work started as a part of the doctoral course Statistical
Physics For Optimization and Learning taught at EPFL in spring 2021.
We acknowledge funding from the ERC under the European Union’s Horizon 2020 Research and Innovation Program Grant
Agreement 714608-SMiLe, and by the Swiss National Science Foundation grant SNFS OperaGOST, 200021 _ 200390 .
[-Therefore, AMP with a uninformed initialization M t =
-]
[-Although reconstruction is statistically possible in this region, AMP with a uninformed initialization M t =
-]
[-Initialize ˆ v t =
-]