Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
Anonymous Author(s)
AfﬁliationAddress email
Abstract
A simple model to study subspace clustering is the high-dimensional k -Gaussianmixture model where the cluster means are sparse vectors.
Here we provide anexact asymptotic characterization of the statistically optimal reconstruction errorin this model in the high-dimensional regime with extensive sparsity, i.e. whenthe fraction of non-zero components of the cluster means ⇢ , as well as the ratio ↵between the number of samples and the dimension are ﬁxed, while the dimensiondiverges.
We identify the information-theoretic threshold below which obtaining a positive correlation with the true cluster means is statistically impossible.
Additionally, we investigate the performance of the approximate message passing (AMP) algorithm analyzed via its state evolution, which is conjectured to beoptimal among polynomial algorithm for this task.
We identify in particular theexistence of a statistical-to-computational gap between the algorithm that requirea signal-to-noise ratio  alg
k/ p ↵ to perform better than random, and the information theoretic threshold at  it ⇡ p k ⇢ log ⇢ / p ↵ .
Finally, we discuss the caseof sub-extensive sparsity ⇢ by comparing the performance of the AMP with othersparsity-enhancing algorithms, such as sparse-PCA and diagonal thresholding.
Introduction
With the growing size of modern data, clustering techniques play an important role in reducing thedimensionality of the features used in modern Machine Learning pipelines.
Indeed, in many tasksof interest ranging from DNA sequence analysis to image classiﬁcation, the relevant features areknown to live in a lower-dimensional space (intrinsic dimension) than their raw acquisition format(extrinsic dimension) [1].
In these cases, identifying these features can help saving computationalresources while signiﬁcantly improving on learning performance.
But given a corrupted embeddingof low-dimensional features in a high-dimensional space, is it always statistically possible to retrievethem?
And if yes - how can reconstruction be achieved efﬁciently in practice?
In this manuscriptwe address these two fundamental questions in a simple model for subspace clustering: a k -cluster
Gaussian mixture model with sparse centroids.
In this model, the low-dimensional hidden featuresare given by the sparse centroids, which are embedded in a higher dimensional space and corruptedby additive Gaussian noise.
We assume that the number of non-zero components of the centroids aswell as the number of samples scales linearly with the dimension of the embedding space.
Givena ﬁnite sample from the mixture, the goal of the statistician is to cluster the data, i.e. estimate thecentroids (or features) as well as possible.
Let x ⌫ 2 R d , ⌫ 2 [ n ] := { 1 , · · · , n } denote n i.i.d.
data points drawn from an isotropic k -cluster 34 Gaussian mixture:
[Equation]
where p c are the class probabilities, C is the index set ( |C| = k ), v c 2 R d are the s -sparse vectors 36 representing the means of the clusters and  is a measure of the signal-to-noise ratio (SNR) for thismodel.
In the following, we focus in the balanced case for which p c = 1 / k .
We consider the means{ v c } c 2 C distributed according to a Gauss-Bernoulli distribution:
[Equation]
where ⇢ :
= s / d is the density of non-zero elements.
For the convenience of the theoretical analysis thatfollows, it will be useful to work with a particular encoding of the class labels C .
For a given sample⌫ 2 [ n ] belonging to the class c 2 C , deﬁne the following label indicator vector u ⌫ c 2 {  1 , k  1 } k :
[Equation]
For a given draw ( x ⌫ ) ⌫ 2 [ n ] , deﬁne the matrices X 2 R d ⇥ n , V 2 R d ⇥ k with columns given by x ⌫and v c and U[-
 R n ⇥ k , V-] 2 R n ⇥ k with rows given by u ⌫ c .
A crucial point that we shall exploit in this paper is that, with this notation, our model for subspaceclustering can be written as a matrix factorization problem, where the data has been generated as:
[Equation]
with W 2 R d ⇥ n a Gaussian matrix with elements W i ⌫ ⇠ N (0 , 1) .
As discussed in the introduction, the s -sparse class means { v c } c 2 C can be thought of as lowdimensional features embedded in a higher-dimensional space R d , that have been corrupted byisotropic additive Gaussian noise with variance ⇠   1 .
Given a ﬁnite draw ( x ⌫ ) ⌫ 2 [ n ] generated fromthis model with class means V ? and labels U ? , the goal of the statistician is to perform clustering , i.e.to estimate U ?
from X, which is equivalent to retrieving the class label for each sample in the data.
However, note that there is a clear class symmetry is this problem: the labels can only be estimated upto a permutation ⇡ [-
-]2 Sym ( C ) reshufﬂing the columns of U ? .
Taking this symmetry into account wecan assess the performance of an estimator ˆ U through the averaged symmetrized mean-squared error:
[Equation]
where || · || F denotes the matrix Frobenius norm.
In particular, we will be interested in characterizing 56 reconstruction in the proportional high-dimensional limit where the number of samples n , the ambientdimension d and the sparsity level s go to inﬁnity n, d, s ! 1 at ﬁxed density ⇢ =
s / d , samplecomplexity ↵ = n / d , number of clusters k  2 and signal strength  > 0 .
Note that the clustering problem above is closely related to the problem of estimating the class means/ features V ? .
Indeed, written as in eq.
(4) the problem of estimating both the labels and centroids( U ? , V ? ) boils down to a low-rank matrix factorization problem.
In this manuscript, we have chosento frame the discussion in terms of the clustering, but all our results could be presented also in termsof the reconstruction of the class means.
In this manuscript we provide a sharp asymptotic characterization of when reconstruction, as measuredby positive correlation with the ground truth, is possible in high-dimensions for this model, bothstatistically and algorithmically .
In particular, our main contributions are:
• We map the subspace clustering problem to an asymmetric matrix factorization problem.
We compute the closed-form asymptotic formula for the performance of the Bayesian-optimal estimator in the high-dimensional limit, building on a statitiscal physics inspired approach [2] that has been rigorously proven in this case [3, 4] .
This allows us to provide a sharp threshold bellow which reconstruction of the features is statistically impossible as a function of the parameters of the model. • To estimate the algorithmic limitations of reconstruction, we analyse a tailored approximatemessage passing (AMP) algorithm
[2, 5, 6] approximating the posterior marginals in this problem,and derive the associated state evolution equations characterising its asymptotic performance.
Suchalgorithms are optimal among ﬁrst order methods [7, 8]) and therefore their reconstruction thresholdprovides a bound on the algorithmic complexity clustering in our model.• The two results above allow us to paint a full picture of the statistical-to-algorithmic trade-offsin high-dimensional subspace clustering for the sparse k -Gaussian mixture model, and in particularto identify an algorithmically hard region of the sparsity level ( 1  ⇢ ) vs. signal-to-noise ratio (  )plane for which reconstruction is possible statistically but not algorithmically, see Fig.
Further,we provide a detailed analysis in the high sparsity regime ( ⇢ !
+ ) of how the algorithmicallyhard region grows as we increase the number of clusters and the sparsity level.
In particular, theinformation theoretical transition arise at  it ⇡ p k ⇢ log ⇢ / p ↵ , and the algorithmic one at
alg
k / p ↵ .•
The analyis for AMP optimality relies on the ﬁnite ⇢ assumption as n, d ! 1 .
We thus alsoinvestigate the complementary case when the number of non-zero components is of the order s .
nand indeed see that sparse principal component analysis (SPCA) and Diagonal thresholding [9]can perform better than random in this region.
We ﬁnd, however, that this requires s  p n (thus⇢
= o (1) ).
We rephrase our ﬁndings in terms of existing literature on the subspace clustering fortwo-classes Gaussian mixtures [10, 11].
Related works: Subspace clustering is a well-studied topic in classical Statistics, with a widerange of methods used in practice, see [12] for a review.
Closer to this work is the theoretical lineof work studying the limitations of clustering in high-dimensions.
Baik, Ben Arous and Péché haveshown that PCA for Gaussian mixture clustering fails to correlate with the mixture means belowcertain threshold known as the BBP transition [13].
For dense Gaussian means, the statistical andalgorithmic limitations of clustering were analysed in different regimes of interest.
Our approach tostudy subspace clustering relies on a mapping to a low-rank matrix factorization problem.
Low-rankmatrix factorization has been extensively studied in the literature, and its asymptotic Bayes-optimalperformance was characterized in [2–5, 14–16].
The construction of AMP algorithms and theassociated state evolutions for matrix factorization was done in [2, 5, 6, 17].
In this work, we leveragethese general results on matrix factorization.
The closest to our work is perhaps [2, 18], wherethe authors characterize the asymptotic reconstruction thresholds for the dense case ( ⇢ = 1 ) in theproportional limit where the number of samples and input dimensions diverge at a constant rate.
Non-asymptotic results were also discussed in [19] which considered a modiﬁcation of Lloyd’salgorithm [20] achieving minimax optimal rate and proving a computational lower bound.
To the bestof our knowledge, the case in which the means are sparse has only been analysed in the regime wherethe number of non-zero components is sub-extensive with respect to the input dimension.
Lowerbounds for the statistically optimal recovery threshold in this regime were given in [21, 22], whilecomputational lower bounds are studied in [23, 24].
Clustering of two component Gaussian mixtureshas been studied in order to shed light on comparison between statistical and algorithmical tractabilityin a sparse scenario in [10, 11, 25].
In particular, [10] conjectured and proved [11] algorithmicbounds for this problem.
They claim that even below the BBP threshold they can build an algorithmachieving exponentially small misclustering error, given that (up-to log factors)
we have s .
p n .
We relate our ﬁndings to their work exploring the extreme sparsity regime in detail.
Main theoretical results
In this section, we introduce the two main technical results allowing us to characterize the limitationsof clustering reconstruction (both statistically and algorithmically) for the model introduced above inthe proportional high-dimensional limit.
Statistical reconstruction:
First, note that up to the permutation symmetry, the estimator minimizing the averaged mean-squared error in eq.
(5) admits a closed-form solution given by the marginalsof the posterior distribution:
[Equation]
(6) where the posterior distribution for the model deﬁned in eq.
(4) explicitly reads:
[Equation]
Although in principle it would be possible to compute the minimum mean-squared error (MMSE)given by the Bayes-optimal estimator in eq.
(6) by sampling from the posterior from eq.
(7) thisis unpractical when d n are large.
For instance simply computing the different integrals involvedin the evidence Z scale exponentially with the dimensions.
The ﬁrst result consists precisely in aclosed-form solution for the asymptotic performance of the Bayes-optimal estimator:
Main theoretical result
In the proportional high-dimensional limit where n, d, s ! 1 with ﬁxedratios ⇢ = s / d , ↵ = n / d and ﬁxed  , k , the minimum mean-squared error for the reconstruction of
U 2 R n ⇥ k is given by:
[Equation]
where M ?
u 2 R k ⇥ k is the solution of the following minimization problem:
[Equation]
with:
[Equation]
and:
[Equation]
Result 1 follows from our mapping of the subspace clustering problem introduced in Sec.
2 to alow-rank matrix factorization form eq.
Indeed, this mapping allow us to leverage a closed-formformula characterizing the asymptotic MMSE for low-rank matrix estimation with generic priors thatwas derived heuristically [2, 18] using the replica method from Statistical Physics and was rigorouslyproven in a series of works [3–5, 14] to the context of subspace clustering.
Our contribution residesin making this connection and drawing the consequences for the subspace clustering problem, anon-trivial endeavour given the complexity of the resulting formulas.
Note that the minimizationproblem eq.
(38) is fundamentally different from the one in eq.
Indeed, the ﬁrst involves onlyscalar variables, and can be easily solved in a computer, while the latter is a high-dimensional problemwhich is computationally intractable for large d, n .
The other parameter M ? v 2 R k ⇥ k solving eq.
(38)can be used to characterize the MMSE reconstruction error on V.
Algorithmic reconstruction:
While result 1 allow us to sharply characterize when clustering isstatistically possible, it does not provide us a practical way to estimate the true class labels U ?
2 R n ⇥ kfrom the data
X 2 R d ⇥ n .
In order to provide a bound in the algorithmic complexity clustering,we consider an approximate message passing (AMP) algorithm for our problem.
Message passingalgorithms are a class of ﬁrst order algorithms (scaling as O ( nd ) , the dimensions of the input matrix
X)
designed to approximate the marginals of a target posterior distribution, and which have twovery important features.
First, for a large class of random problems (such as the clustering problemstudied here) AMP provides the best known ﬁrst order method in terms of estimation performance[26–30], and has been rigorously proven to be the optimal for certain problems [7, 8].
Second, theasymptotic performance of AMP can be tracked by a set of low-dimensional state evolution equations[31], meaning that its reconstruction performance can be sharply computed without having to run ahigh-dimensional instance of the problem.
For low-rank matrix factorization problems an associated
AMP algorithm can be derived [5, 6, 17, 32, 33].
Therefore, yet again we leverage the mappingof subspace clustering to a low-rank matrix factorization problem to derive the associated AMPalgorithm 3 with denoising functions ⌘ v , ⌘ u :
[Equation]
As mentioned above, one of they key features of Algorithm 3 is that its asymptotic performance canbe tracked exactly by a set of low-dimensional equations.
Main theoretical result
In the proportional high-dimensional limit where n, d, s ! 1 with ﬁxedratios ⇢ = s / d , ↵ = n / d and ﬁxed  , k , the correlation between the ground truth ( U ? , V ? ) and the
AMP estimators (ˆ U t amp , ˆ V t amp ) at iterate t ,
[Equation]
[Equation]
satisfy the following state evolution equations :
[Equation]
Moreover, the asymptotic performance of is simply given by:
[Equation]
Result 2 is a consequence from the general theory connecting AMP algorithms to their state evolution(SE) [31].
In the context of low-rank matrix factorization, a derivation of the state evolution equations
Algorithm
Input: Data X 2 R d ⇥ n
Initialize ˆ v t =
i , ˆ u t =0 ⌫ ⇠ N ( 0 k , ✏ I k ) , ˆ  t =0 u, ⌫ = 0 k ⇥ k , ˆ  t =0 v,i =
k ⇥ k .
for t  t max do
A tu =  s ⇣
ˆ U t ⌘ > ˆ U ,
A tv =
s ⇣
ˆ V t ⌘ > ˆ V
B tv =
q  s X
ˆ U t   s n
P ⌫ =1  tu, ⌫ ˆ V t  1 ,
B tu =
q  s X > V   s d
P i =1  tv,i ˆ U t
Take { b tv,i 2 R k } di =1 , { b tu, ⌫ 2 R k } n ⌫ =1 rows of B tv , B tu ˆ v t
+1 i
= ⌘ v ( A tv , b tv,i ) , ˆ u t +1 ⌫ = ⌘ u ( A tu , b tu, ⌫ ) ˆ  t +1 v,i = @
b ⌘ v ( A tv , b tv,i ) , ˆ  t +1 u, ⌫ = @ b ⌘ u ( A tu , b tv, ⌫ )
Here ˆ U t 2 R n ⇥ k , ˆ V t 2 R d ⇥ k , B tu 2 R n ⇥ k , B tv 2 R d ⇥ k , A tu 2 R k ⇥ k , A tv 2 R k ⇥ k end for
Return: Estimators ˆ v amp ,i , ˆ u amp , ⌫ 2 R k , ˆ  u, ⌫ , ˆ  v,i
R k ⇥ k above from Algorithm 3 were ﬁrst provided for the rank-one case in [5, 6] and were extended to general rank and denoising functions in [2, 18].
A crucial observation is that Result 2 is intimately relatedto Result 1.
Indeed, the state evolution equations (17) coincide exactly with running gradient descenton the potential deﬁned in eq.
However, while the performance of the statistically optimal estimator ˆ U bo is given by the global minima of the potential  rs , the performance of the AMP estimatorˆ U amp is given by the minima achieved by gradient descent: i.e. the one closest the initialization M t =0 u .
Therefore, studying both the statistical and algorithmic limitations of clustering in high-dimensionsboils down to the study of the minima of the potential  rs , see App.
A for further details.
Reconstruction limits for sparse clustering
These results allow us to paint a full picture for when sparse subspace clustering is possible in themodel deﬁned in Sec. 2 as a function of the quantity of data ↵ , the sparsity 1  ⇢ , the number of clustersk and the SNR  .
Figure 1 summarizes the different reconstruction regimes in the ( ⇢ ,  ) plane fora two-clusters problem at ﬁxed sample complexity ↵ =2 , also known as a phase diagram .
Moreoverthe general considerations on the reconstruction limits for sparse clustering, given by analyzing thetwo-clusters problem, are easily generalizable to the general mixture case and not restricted to thatparticular model, see Sec.
For a ﬁxed sparsity 1  ⇢ , we identify the following regions in Fig.
Impossible phase: There is not enough information in the data matrix X handled to the statistician
to assign cluster membership better than chance for any algorithm .
The Bayes-optimal MMSE is not
better than a random guess.
Clustering (reconstruction of U ?
better than chance) is impossible.
Hard phase: The MMSE is non-trivial, and clustering is statistically possible to some extent, but
the best known polynomial time algorithm, AMP, fails to correlate better than chance with the true
cluster assignment U ? .
Any polynomial-time algorithm is conjectured to fail in this region.
Easy phase: In the easy phase, not only clustering is statistically possible, but AMP is able
to
achieve positive correlation with U ? .
One can also invextigate when AMP achieves the Bayes-optimal
MMSE (instead of just positive correlation) leading to the same transition (except from a subtle correction very close to the tri-critical point, see the discussion in App.
A).
In Fig.
2, we investigate the different phases presented above by varying the sparsity level 1  ⇢and SNR  at ﬁxed ↵ .
We compare the performance of AMP with popular spectral algorithms:
Principal Component Analysis (PCA) and Sparse Principal Component Analysis (SPCA).
We caninitialize AMP and the SE equations in two different ways: we call uninformative initialization achoice for the ﬁrst iterates of AMP and SE which assumes no knowledge on the ground truth values;conversely with informative initialization we consider that the statistician has some prior knowledgeon the the ground truth signal.
Note that in a real-life scenario the statistician doesn’t have access theground truth.
Yet, as a theoretical tool the informative initialization provides important informationabout the algorithmically hard phase, see App A for a discussion.
The initialization strategy weconsidered in the uninformed case for Algorithm 3 is not the only possible choice, there are smarterways of initialising which can lead to a considerable improvement without explicitly assuming any information about the signal, e.g. spectral initialization [34].
Looking at Fig.
2 we see that SEwith uninformed initialization tracks AMP.
Morover we note that increasing the sparsity level, i.e.decreasing ⇢ , the problem becomes algorithmically harder.
This is reﬂected in a discontinous jumpin the MSE at  alg which becomes larger.
We observe along the same lines a neat advantage byimposing the sparsity constraint in the spectral algorithm (SPCA) with respect to vanilla one (PCA)as the sparsity grows.
We discuss the details on the numerical simulations in App.
D and the code isavailable at https://github.com/anonymous.
Stability analysis and algorithmic threshold
In this section we provide a detailed analytical derivation of the threshold  alg characterizing thealgorithmic reconstruction as a function of the number of clusters and the sparsity.
First, note due tothe permutation symmetry of the clusters the overlap matrices admit the following parametrization:
[Equation]
where J k is the k ⇥ k -matrix with all elements equal to one.
This parametrization is preserved underthe SE iterations.
Therefore, inserting it in the SE equations yield equations for ( m tu , m tv ) :
[Equation]
where we introduced the following update functions:
[Equation]
22) where S k  1 is the surface of the k -dimensional unitary hypersphere.
Recall that ( m tu , m tv ) 2 [0 , 1] 2fully characterize the reconstruction performance, with ( m 1 u , m 1 v )=(0 , 0) corresponding to the performance of a random guess.
Conversely, ( m 1 u , m 1 v )=(1 , ⇢ ) corresponds to perfect reconstructionof the cluster membership and the sparse cluster means.
One can check that the trivial ﬁxed point ( m u , m v )=(0 , 0) is always a ﬁxed point of the SE equations.
Moreover, note that its stability is crucially connected to the algorithmic threshold  alg .
Indeed, if the trivial ﬁxed point ( m u , m v )=(0 , 0)is stable, AMP with an uninformed initialization will always converge to this point - and thereforeachieve random guessing performance.
To study its stability, we expand the update functions around( m u , m v )=(0 , 0) up to the second order.
Expressing everything in terms of m u , we obtain:
[Equation]
This immediately tells us that the trivial ﬁxed point becomes unstable at the algorithmic threshold alg
= k / p ↵ .
This transition is a well-known result in random matrix theory and goes under thename of BBP transition [13].
Despite the fact that we expand around the trivial ﬁxed point, theperturbative method also provides information also about the Bayes-optimal performance, thanksto the general properties that the phase diagrams in Bayes-optimal inference problems must respect[35].
Indeed, a sufﬁcient criterion for the presence of an algorithmically hard phase requires at thealgorithmic threshold: F 00 se (0) > 0 .
The local study predicts that the phase diagram will present analgorithmically hard region for k > k hard = 4 + 2 p ↵ .
The criterion nevertheless is not necessary inthis setting, as we immediately see from the phase diagram for the two-components GMM in Fig.
1:we clearly observe an algorithmically hard region for high-sparsity although k < k hard .
In fact, theinformation-theoretic threshold  it , deﬁned as the value of the SNR at which the problem becomesstatistically possible, cannot be found simply thanks to the expansion around the trivial ﬁxed point.
In App.
B we discuss how to compute  it , which we see plotted in Fig.
1 for the two-classes case.
A detailed discussion on the thresholds is given in App. A.
Large sparsity regime
We characterize in this section the scaling of the thresholds in the very sparse (and most interesting)regime when ⇢ !
+ .
We highlight the main passages of the computation and focus on the principalresults, see Appendix C for a detailed analysis.
The starting point is the following change of variables:
[Equation]
Inserting these expressions into eq.
(20), we obtain simpliﬁed SE equations for the rescaled overlaps( ˜ m u , ˜ m v ) without any residual dependence on ( ⇢ , ↵ ) :
[Equation]
where we introduced the auxiliary function T k ( · ) deﬁned as:
[Equation]
where ⇥ ( · ) is the Heaviside theta function.
By considering the large k expansion of T k ( · ) , we canderive the scaling of the IT threshold with ( k, ⇢ , ↵ ) , see Appendix C for more details.
Putting togetherwith our previous result for  alg from Sec. 5, we obtain the following fundamental result:
[Equation]
[Equation]
These equations completely characterize the behavior at large rank & small (but ﬁnite) sparsity.
Thestatistical-to-computational gap, where AMP is not able to exploit the information on the sparsenature of the cluster means, grows with both the sparsity and the rank.
In App.
C, we further show thatthe large rank expression for the IT threshold  it is accurate already at moderate k ⇡ 10 , see Fig. 8.
It is interesting at this point to reframe our results on the high sparsity regime in the context of theexisting literature.
The results for the IT transition for the detection of two-classes sparse GMM werediscussed in [22, 23], and we obtain a consistent scaling in the small ⇢ behaviour.
Despite of this fact,the algorithmic bounds of different relevant works for the same problem [10, 11, 23] seem, at ﬁrstsight, to not agree with our ﬁndings.
In particular [11] proves rigorously the existence of an algorithmthat achieves minimax rate under the BBP threshold for clustering of two-classes sparse GMM.
Thisapparent inconsistency is related to different sparsity regimes analyzed.
Indeed, here we investigatethe extensive sparsity regime, i.e. ⇢ =
O (1) , while the guarantees for the efﬁcient algorithms working under the BBP threshold require a very high sparsity level, i.e. ⇢ =
o (1) .
This difference is crucial.
In Fig.
3 we illustrate it by considering the performance of two popular algorithms for this problem: Diagonal Thresholding (DT) [9] and SPCA.
For extremely large sparsity, i.e. s = O (1) , thesealgorithms indeed provide estimators with positive correlation with the true classes below the BBPthreshold!
However, as soon as we increase the density of non-zero components the performancestrongly deteriorates.
In fact, the transition to random chance performance takes place as the numberof non-zero components approach s .
p n , in agreement with the literature [10, 11].
Conclusion
In this work we considered the problem of clustering k homogeneous Gaussian mixtures withsparse means.
Mapping this to a low-rank matrix factorization problem, we have provided anexact asymptotic characterization of the MMSE in the high dimensional limit.
The Bayes-optimalperformance was compared to AMP, the best known polynomial time algorithm for this problem inthe studied regime.
In the large sparsity regime, we uncovered a large statistical-to-computational gapas the sparsity level grows, and unveiled the existence of a computationally hard phase .
In particular,we have shown that the SNR threshold below which recovery is statistically impossible is given by it ⇡ p  k ⇢ log ⇢ / p ↵ , while the one for which AMP positively correlates with the ground truth classesis given by  alg  k / p ↵ .
Our result for the existence of an algorithmically hard region was comparedwith the existing literature for this problem, solving an apparent contradiction due to the scalingassumption of the sparsity level with the dimension of the features.
We corroborated our ﬁndingswith the help of algorithms for subspace clustering such as sparse principal component analysis anddiagonal thresholding.