Noisy ℓ 0 -Sparse Subspace Clustering on Dimensionality Reduced Data
Abstract a Sparse subspace clustering methods with sparsity induced by ℓ 0 -norm, such as ℓ 0 -Sparse Subspace Clustering ( ℓ 0 -SSC)
[Yang et al., 2018], are demonstrated to be more effective than its ℓ 1 counterpart such as Sparse Subspace Clustering (SSC)
[Elhamifar and Vidal, 2013].
However, the theoretical analysis of ℓ 0 -SSC is restricted to clean data that lie exactly in subspaces.
Real data often suffer from noise and they may lie close to subspaces.
In this paper, we show that an optimal solution to the optimization problem of noisy ℓ 0 -SSC achieves subspace detection property (SDP), a key element with which data from different subspaces are separated, under deterministic and semi-random model.
Our results provide theoretical guarantee on the correctness of noisy ℓ 0 -SSC in terms of SDP on noisy data for the first time, which reveals the advantage of noisy ℓ 0 -SSC in terms of much less restrictive condition on subspace affinity.
In order to improve the efficiency of noisy ℓ 0 -SSC, we propose Noisy-DR- ℓ 0 -SSC which provably recovers the subspaces on dimensionality reduced data.
Noisy-DR- ℓ 0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy ℓ 0 -SSC on the dimensionality reduced data for improved efficiency.
Experimental results demonstrate the effectiveness of Noisy-DRℓ 0 -SSC.
INTRODUCTION
Clustering is an important unsupervised learning procedure for analyzing a broad class of scientific data.
Highdimensional data, such as facial images and gene expression data, often lie in low-dimensional subspaces in many cases, and clustering in accordance to the underlying subspace structure is particularly important.
Among various subspace clustering algorithms, the ones that employ sparsity prior, such as Sparse Subspace Clustering (SSC) [Elhamifar and Vidal, 2013] and ℓ 0 -Sparse Subspace Clustering ( ℓ 0 -SSC)
[Yang et al., 2018], have been proven to be effective in separating the data in accordance with the subspaces that the data lie in under certain assumptions.
Sparse subspace clustering methods construct the sparse similarity matrix by sparse representation of the data.
Subspace detection property (SDP) defined in Section 2.2 ensures that the similarity between data from different subspaces vanishes in the sparse similarity matrix, and applying spectral clustering [Ng et al., 2001] on such sparse similarity matrix leads to compelling clustering performance.
Elhamifar and
Vidal [Elhamifar and Vidal, 2013] prove that when the subspaces are independent or disjoint, SDP can be satisfied by solving the canonical sparse linear representation problem using data as the dictionary, under certain conditions on the rank, or singular value of the data matrix and the principle angle between the subspaces.
Under the independence assumption on the subspaces, low rank representation [Liu et al., 2010, 2013a] is also proposed to recover the subspace structures.
Relaxing the assumptions on the subspaces to allowing overlapping subspaces, the Greedy Subspace
Clustering [Park et al., 2014] and the Low-Rank Sparse Subspace Clustering [Wang et al., 2013] achieve subspace detection property with high probability.
The geometric analysis in Soltanolkotabi and Candés [2012] shows the theoretical results on subspace recovery by SSC.
In the following text, we use the term SSC or ℓ 1 -SSC exchangeably to indicate the Sparse Subspace Clustering method in Elhamifar and
Vidal [2013].
Real data often suffer from noise.
The correctness of noisy
SSC is analyzed in Wang and Xu [2013] which handles noisy data that lie close to disjoint or overlapping subspaces, and the original optimization problem of noisy SSC is proposed in Elhamifar and Vidal [2013].
While Yang et al. [2016], Yang [2018] prove the correctness of ℓ 0 -SSC or its dimensionality reduced variant on clean data based on a constrained ℓ 0 -minimization problem, it empirically solves an unconstrained ℓ 0 -regularized problem to handle noise in data, and they lack theoretical analysis on the correctness of ℓ 0 -SSC on noisy data.
In this paper, the correctness of noisy ℓ 0 -SSC on noisy data in terms of the subspace detection property is established.
Our analysis is under both deterministic model and semi-random model, which are the models employed in the geometric analysis of SSC [Soltanolkotabi and Candés, 2012].
Our randomized analysis demonstrates the advantage of noisy ℓ 0 -SSC over its ℓ 1 counterpart as more general assumption on data distribution can be adopted.
Moreover, we present Noisy Dimensionality Reduced ℓ 0Sparse Subspace Clustering (Noisy-DR- ℓ 0 -SSC), an efficient version of noisy ℓ 0 -SSC which also enjoys robustness to noise.
Noisy-DR- ℓ 0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy ℓ 0 -SSC on the dimensionality reduced data.
Noisy-DR- ℓ 0 -SSC provably recovers the underlying subspace structure in the original data from the dimensionality reduced data under deterministic model.
Experimental results demonstrate the effectiveness of both noisy ℓ 0 -SSC and Noisy-DR- ℓ 0 -SSC.
1.1 NOTATIONS
We use bold letters for matrices and vectors, and regular lower letter for scalars throughout this paper.
The bold letter with superscript indicates the corresponding column of a matrix, e.g. A i is the i -th column of matrix A , and the bold letter with subscript indicates the corresponding element of a matrix or vector.
∥ · ∥ F and ∥ · ∥ p denote the Frobenius norm and the vector ℓ p -norm or the matrix p -norm, and ∥·∥is the ℓ 0 -norm, that is, the number of nonzero elements of a vector.
diag( · ) indicates the diagonal elements of a matrix.
H T ⊆
R d indicates the subspace spanned by the columns of T , and A I denotes a submatrix of A whose columns correspond to the nonzero elements of I (or with indices in
I without confusion).
σ t ( · ) denotes the t -th largest singular value of a matrix, and σ min ( · ) indicates the smallest singular value of a matrix.
supp( · ) is the support of a vector, P S ′ is the operator of orthogonal projection onto the subspace
S ′ .
[ n ] represents all the natural numbers between 1 and n inclusively.
S d − 1 denotes the unit sphere in R d .
{+Θ( a ) denotes a number such that there exists two constants c 1 and c 
+}
such that Θ( a ) ∈ [ c 1 a, c 2 a ] . 1.2 CONTRIBUTIONS First, the correctness of noisy ℓ 0 -SSC on noisy data in terms of the subspace detection property is established for the first time, which is presented in Section 3 of this paper.
Our analysis is under both deterministic model and semi-random model, which are also the models employed by the geometric analysis of SSC [Soltanolkotabi2012].
Our randomized analysis demonstrates the significant advantage of noisy ℓ 0SSC over its ℓ 1 counterpart and other competing subspace clustering methods in terms of much less restrictive condition on the subspace affinity. Table 1 below demonstrates the conditions under which SDP holds for representative subspace clustering methods under the semi-random model, including Greedy Subspace Clustering (GSC) [Park et al., 2014], ℓ 1 -SSC[C], Noisy SSC[G], Affine Sparse Subspace Clustering (ASSC)[K,L], Noisy ℓ 0 -SSC[I,J].
When the size of data n grows exponentially in terms of the common subspace dimension d 0 (the dimension of every subspace is d 0 ), in particular, n = Θ( e d τ 0 ) for τ ∈ (0 . 5 , 0 . 9) , then all the competing subspace clustering methods other than Noisy ℓ 0 -SSC either do not allow overlapping subspaces, or require the maximum pairwise subspace affinity goes towhen d 0 → ∞ , which means that these methods require all the subspaces to be almost pairwise orthogonal when the common subspace dimension d 0 is very large. Instead, Noisy ℓ 0 -SSC allows that subspace affinity can be lower bounded from 0 , suggesting that Noisy ℓ 0 -SSC is still able to recover subspaces which are not orthogonal when d 0 is very large. In Table 1, it is preferred that a subspace clustering method requires milder conditions, which are allowing overlapping subspaces and larger upper bound for the maximum subspace affinity, denoted by max k,
l ∈
[ K ] aff ( S k , S l ) where {S k } Kk =1 are K subspaces, so that the underlying subspaces can be recovered for overlapping subspaces and for subspaces which are closer to each other (larger subspace affinity).
Here aff denotes subspace affinity, d 0 is the common subspace dimension, δ is a small positive constant (see [Park et al., 2014]).
r 0 > 1 is an upper bound for the support of an optimal solution to the noisy ℓ 0 -SSC problem for all data points.
In this table, n = Θ( e d τ 0 ) for τ ∈ (0 . 5 , 0 . 9) when d 0 → ∞ .
Note that two subspaces are overlapping subspaces if the dimension of their intersection is larger than 1 .
When a subspace clustering method does not allow overlapping subspaces, then no condition on subspace affinity is presented in the subspace clustering literature.
Second, we propose Noisy Dimensionality Reduced ℓ 0Sparse Subspace Clustering (Noisy-DR- ℓ 0 -SSC) to accelerate noisy ℓ 0 -SSC with provable robustness to noise.
NoisyDR- ℓ 0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy ℓ 0 -SSC on the dimensionality reduced data.
Two types of random projections are used in Noisy-DR- ℓ 0 -SSC, which are the random projection induced by randomized low-rank approximation and the sparse random projection by Oblivious
Sparse Norm-Approximating Projection (OSNAP).
It should be emphasized that Yang [2018] also studies dimensionality reduced ℓ 0 -SSC.
However, the analysis of this work is performed on the following constrained ℓ 0 - Please refer to Section 1.2 for notations [Table] minimization problem and only for clean data without noise,
[Equation]
where ˜ X is the dimensionality reduced version of clean data by random projection.
On the other hand, the actual optimization problem [Yang, 2018] solves the unconstrained ℓ 0 -regularized problem,
[Equation]
In contrast, we analyze noisy ℓ 0 -SSC on the unconstrained ℓ 0 -regularized problem with noisy data which reveals the advantage of noisy ℓ 0 SSC over ℓ 1 -SSC.
Our analysis also suggests that a larger λ tends to guarantee the subspace detection property (Remark 3.5), which is verified by experiments.
Throughout this paper, we refer to ℓ 0 -SSC for noisy data with the unconstrained ℓ 0 -regularized problem as noisy ℓ 0 -SSC.
PROBLEM SETUP 2.1 NOTATIONS
We hereby introduce the notations for subspace clustering on noisy data considered in this paper.
The uncorrupted data matrix is denoted by Y =
[ y 1 , . . .
, y n ] ∈ R d × n , where d is the dimensionality and n is the size of the data.
The uncorrupted data Y lie in a union of K distinct subspaces {S k } Kk =1 of dimensions { d k } Kk =1 with d max :=
max k ∈ [ K ] d k and d min :=
min k ∈ [ K ] d k .
The observed noisy data is X =
Y + N , where N =
[ n 1 , . .
. , n n ] ∈ R d × n is the additive noise.
x i =
y i + n i is the noisy data point that is corrupted by the noise n i .
We let Y ( k ) ∈
R
d × n k denote the data belonging to subspace S k with K (cid:80)
[Equation]
and denote the corresponding columns in X by X ( k ) .
Let
U ( k ) ∈ R
d × d k be the orthogonal basis of S k for all k ∈ [ K ] .
The data X are normalized such that each column has unit ℓ 2 -norm in our deterministic analysis.
We consider deterministic noise model where the noise N is fixed and max i ∈ [ n ] ∥ n i ∥ 2 ≤ δ .
Note that our analysis can be extended to a random noise model which is common and also considered by noisy SSC [Wang and Xu, 2013], and the random noise model assumes that columns of N are i.i.d.
samples with maximum ℓ 2 -norm bounded by δ with high probability.
Note that such random noise model does not require spherical symmetric noise as that in Wang and Xu [2013].
2.2 REVISITING SPARSE SUBSPACE CLUSTERING (SSC) AND ℓ 0 -SSC
Sparse Subspace Clustering (SSC) methods, such as [Elhamifar and Vidal, 2011, Soltanolkotabi and Candés, 2012], construct a sparse similarity matrix by sparse representation of the data, and then perform clustering on the sparse similarity matrix. Formally, given observed data X ∈ R d × n , X = [ x 1 , . .
, x n ] , where x i ∈ R d , SSC solves the following optimization problem for each i ∈ [ n ] : [Equation] In order to handle noisy data, noisy SSC [Wang and Xu, 2013] was proposed to solve the ℓ 1 regularized problem: [Equation]
The sparse code β of the data point x i is obtained by solving (3) or (4) for SSC or noisy SSC.
A coefficient matrix
Z ∈ R n × n is then formed by concatenating the sparse codes of all the data points, and the i -th column of Z is the sparse code of x i .
The sparse similarity matrix is then computed by W = | Z | + | Z ⊤ |.
Subspace detection property (SDP, to be formally defined later) ensures that the similarity between data from different subspaces vanishes in the sparse similarity matrix.
If SDP holds, then similarity between data points from different clusters vanish in W .
As a result, performing spectral clustering on W leads to compelling clustering results.
Under the independence assumption on the subspaces, low rank representation [Liu et al., 2013b] is proposed to recover the subspace structures.
Relaxing the assumptions on the subspaces to allowing overlapping subspaces, the Greedy
Subspace Clustering [Park et al., 2014] and the Low-Rank
Sparse Subspace Clustering [Liu et al., 2013b] achieve subspace detection property with high probability.
The geometric analysis in Soltanolkotabi and Candés [2012] shows the theoretical results on subspace recovery by SSC.
In the following text, we use the term SSC or ℓ 1 -SSC exchangeably to indicate the Sparse Subspace Clustering method in Elhamifar and Vidal [2011], Soltanolkotabi and Candés [2012].
ℓ 0 -SSC [Yang et al., 2018] proposes to solve the following ℓ 0 sparse representation problem
[Equation]
and it proves that SDP is satisfied with an globally optimal solution to problem (1).
in Yang et al. [2018], the ℓ 0 regularized sparse approximation problem below is solved so as to handle noisy data for ℓ 0 -SSC, which is the optimization problem of noisy ℓ 0 -SSC:
[Equation]
The optimization problem of noisy ℓ 0 -SSC (2) is separable.
For each i ∈ [ n ] , the optimization problem with respect to the sparse code β of i -th data point is
[Equation]
The sparse similarity matrix W is then computed in the same way as ℓ 1 -SSC by W = | Z | + | Z ⊤ |, and the subspace clustering result of noisy ℓ 0 -SSC is achieved by performing spectral clustering on W .
In the following text, we always use β ∗ to denote an optimal solution to (7), and define r ∗ :=
∥ β ∗ ∥ 0 .
The definition of subspace detection property for noisy ℓ 0SSC and noiseless ℓ 0 -SSC, i.e. ℓ 0 -SSC on noiseless data, is defined in Definition 2.1 below.
Definition 2.1.
(Subspace detection property for noisy and noiseless ℓ 0 -SSC) Let Z ∗ be an optimal solution to (6).
The subspaces {S k } Kk =1 and the data X satisfy the Subspace Detection Property (SDP) for noisy ℓ 0 -SSC if Z i is a nonzero vector, and nonzero elements of Z i correspond to the columns of X from the same subspace as y i for all 1 ≤ i ≤ n .
We say that SDP holds for x i if nonzero elements of Z ∗ i , which is β ∗ for problem (7), correspond to the data that lie in the same subspace as y i , for either noisy ℓ 0 -SSC or noiseless ℓ 0 -SSC.
It should be emphasized that the theoretical analysis of ℓ 0SSC [Yang et al., 2018] is limited to clean data, as a result, it is not clear if SDP can hold for an optimal solution to the noisy ℓ 0 -SSC problem.
In the following section, we provide theoretical analysis of
THEORETICAL ANALYSIS FOR NOISY ℓ 0 -SSC
The theoretical results on the subspace detection property for noisy ℓ 0 -SSC are presented in this section under the deterministic model and the semi-random model.
These models are introduced in the following subsection.
3.1 MODELS
Similar to Soltanolkotabi and Candés [2012], we introduce the deterministic and the semi-random model for the analysis of noisy ℓ 0 -SSC.
• Deterministic Model: the subspaces and the data in each subspace are fixed.
• Semi-Random Model: the subspaces are fixed but the data are independent and identically distributed in each of the subspaces.
The data in the above definitions refer to clean data without noise.
Both the deterministic model and the semirandom model are extensively employed to analyze the subspace detection property in the subspace learning literature [Soltanolkotabi and Candés, 2012, Wang et al., 2013, Wang and Xu, 2013, Yining Wang and Singh, 2016].
3.2 NOISY ℓ 0 -SSC: DETERMINISTIC ANALYSIS
We introduce the definition of general position and external subspace before our analysis on noisy ℓ 0 -SSC.
Definition 3.1.
(General position)
For any 1 ≤ k ≤ K , the data Y ( k ) are in general position if any subset of L ≤ d k data points (columns) of Y ( k ) are linearly independent.
Y are in general position if Y ( k ) are in general position for 1 ≤ k ≤ K .
The assumption of general condition is rather mild.
In fact, if the data points in X ( k ) are independently distributed according to any continuous distribution, then they almost surely in general position.
Let the distance between a point x ∈ R d and a subspace
S ⊆
R d be defined as d ( x , S ) =
inf y ∈S ∥ x − y ∥ 2 , the definition of external subspaces is presented as follows.
Definition 3.2.
(External subspace of limited dimension)
For a point y ∈ Y ( k ) , a subspace H { y ij } Lj =1 spanned by a set of linear independent points { y i j } Lj =1 ⊆
Y is defined to be an external subspace of y
if { y i j } Lj =1 ̸⊆ Y ( k ) and y / ∈ { y i j } Lj =1 .
The set of all external subspaces of y of dimension no greater than r with r ≥ 1 for y is denoted by H y ,r , that is, H y ,r
= { H : H
= H { y ij }
Lj =1 , dim[ H ] =
L, L ≤ r, { y i j } Lj =1 ̸⊆ Y ( k ) , y / ∈ { y i j } Lj =1 } .
The point y is said to be away from its external subspaces of dimension r if min H ∈H y ,r d ( y , H ) > 0 .
All the data points in Y ( k ) are said to be away from the external subspaces if each of them is away from the its associated external spaces.
To present our theoretical results of the correctness of noisy ℓ 0 -SSC, we need the definitions related to the spectrum of
X and Y , which are defined as follows.
In the following analysis, we employ β to denote the sparse code of datum x i so that a simpler notation other than Z i is dedicated to our analysis.
Definition 3.3.
The minimum restricted eigenvalue of the uncorrupted data is defined as
[Equation]
for r ≥ 1 .
In addition, the normalized minimum restricted eigenvalue of the uncorrupted data is defined by
[Equation]
Moreover, the following quantities are defined for our analysis.
We define
[Equation]
where
[Equation]
with δ < ¯ σ ∗ Y , and ¯ σ ∗ Y is defined as
[Equation]
Now we present the main result on the correctness of noisy ℓ 0 -SSC.
Theorem 3.1.
(Subspace detection property holds for noisy ℓ 0 -SSC) Let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (7) for point x
i with ∥ β ∗ ∥
= r ∗ ≥ 1 , and c ∗ := ∥ x i − X β ∗ ∥ 2 .
Suppose Y is in general position, y i ∈
S k for some 1 ≤ k ≤ K , δ < ¯
σ ∗ Y , λ > τ 0 ,
B ( y i , δ + c ∗ + 2 δ √ r ∗ σ ∗ X ) ∩ H = ∅ for any H ∈ H
y i ,r ∗ .
Then the subspace detection property holds for x i with β ∗ .
Here τ 0 , τ 1 , ¯ σ ∗ Y and σ ∗ X are defined in (10), (11) and (12).
Remark 3.2.
When δ = 0 and there is no noise in the data
X , the conditions for the correctness of noisy ℓ 0 -SSC in
Theorem 3.1 almost reduce to that for noiseless ℓ 0 -SSC.
To see this, the conditions are reduced to B ( y i , c ∗ )
∩ H = ∅ , which are exactly the conditions required by noiseless ℓ 0SSC in Lemma 7.1 in the supplementary, namely data are away from the external subspaces by choosing λ → 0 and it follows that c ∗ = 0 .
While Theorem 3.1 establishes geometric conditions under which the subspace detection property holds for noisy ℓ 0SSC, it can be seen that these conditions are often coupled with an optimal solution β ∗ to the noisy ℓ 0 -SSC problem (7).
In the following theorem, the correctness of noisy ℓ 0 -SSC is guaranteed in terms of λ , the weight for the ℓ 0 regularization term in (7), and the geometric conditions independent of an optimal solution to (7).
Let M i > 0 be the minimum distance between y i ∈
S k and its external subspaces when y i is away from its external subspaces of dimension r , that is,
[Equation]
The following two quantities related to the spectrum of clean and noisy data, µ r and σ X ,r , are defined as follows with r > 1 for the analysis in Theorem 3.3.
[Equation]
Theorem 3.3.
(Subspace detection property holds for noisy ℓ 0 -SSC under deterministic model with conditions in terms of λ )
Let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (7) for point x
i with ∥ β ∗ ∥
= r ∗ > 1 , n k ≥ d k +1 for every k ∈
[ K ] , and there exists 1 < r 0 ≤ ⌊ 1 λ ⌋ such that r ∗ ≤ r 0 .
Suppose Y is in general position, y i ∈
S k for some 1 ≤ k ≤ K , δ < ¯
σ ∗ Y , and M i,δ :=
M i [Equation]
Suppose
M
[Equation]
16) and
[Equation]
Then if λ 0 < λ < 1 ,
[Equation] [Equation]
where λ 0 :
=
max { λ 1 , λ 2 } and
[Equation]
the subspace detection property holds for x i with β ∗ .
Here
M i , µ r 0 and σ X ,r 0 are defined in (13), (14) and (15) respectively.
Remark 3.4.
The two conditions (16) and (17) are induced by two conditions, B ( y i , δ + c ∗ + 2 δ √ r ∗ σ ∗ X ) ∩ H = ∅ for any
H ∈ H
y i ,d k and λ > τ 0 respectively, which are required by
Theorem 3.1.
Note that when (16) and (17) hold, λ 1 and λcan always be chosen in accordance with (19) and (20).
Remark 3.5.
It can be observed from condition (18) that noisy ℓ 0 -SSC encourages sparse solution by a relatively large λ so as to guarantee the subspace detection property.
This theoretical finding is consistent with the empirical study shown in the experimental results.
3.3 NOISY ℓ 0 -SSC: RANDOMIZED ANALYSIS
In this subsection, the correctness of noisy ℓ 0 -SSC is analyzed under the semi-random model that the data in subspace S ( k ) are i.i.d.
according to the uniform distribution on the unit sphere, S d k − 1 , of R d k centered at the origin for all k ∈ [ K ] .
This setting is employed extensively in the subspace learning literature [Soltanolkotabi and Candés, 2012,
Wang et al., 2013, Wang and Xu, 2013, Yining Wang and
Singh, 2016].
We then have the major theorem below stating the theoretical guarantee of the subspace detection property of noisy ℓ
-SSC under the semi-random model.
Before stating this theorem, we introduce the following definition of subspace affinity, which is widely used in the analysis of semi-random model in the sparse subspace clustering literature.
Definition 3.4.
(Subspace affinity)
The affinity between two subspaces, S k and S l with k, l ∈ [ K ] , is defined by
[Equation]
where cos θ ( t ) kl is the t -th canonical angle between S k and S l defined in Soltanolkotabi and Candés [2012].
Let U ( k ) and
U ( l ) be the orthonormal basis for S k and S l respectively, then
[Equation]
with the orthogonality constraints u ⊤ u j = 0 and v ⊤ v j =with j = 1 , 2 , . . .
, t − 1 .
It can be verified that aff( S k , S l ) =
∥ U ( k )
⊤ U ( l ) ∥ F .
Theorem 3.6.
(Subspace detection property holds for noisy ℓ 0 -SSC under semi-random model with conditions in terms of λ )
Under the semi-random model, let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (7) for point x
i with ∥ β ∗ ∥
= r ∗ > 1 , n k ≥ d k + 1 for every 1 ≤ k ≤ K , and there exists 1 < r 0 ≤ ⌊ 1 λ ⌋ such that r ∗ ≤ r 0 .
Let σ ′
( d min , 1 λ [Equation] :=
[Equation]
(cid:16)
[Equation]
(cid:17)
[Equation]
c : [Equation] (cid:113)
[Equation]
t
[Equation]
S t
[Equation]
r. For t > 0 such that
[Equation]
suppose
[Equation]
and
[Equation]
where M :=
[Equation]
[Equation]
and d min
[Equation]
Then with probability at least
[Equation]
−
[Equation]
k
[Equation]
n k exp( − d k t 2 2 ) , the subspace detection property holds for x i with β ∗ for all i ∈ [ n ] .
Remark 3.7 (Advantage of Noisy ℓ 0 -SSC in terms of Subspace Affinity) .
It is well known that the difficulty of achieving the subspace detection property increases with larger affinity between subspaces, that is, the subspace are closer to each other.
Our analysis reveals the significant advantage of noisy ℓ 0 -SSC over ℓ 1 -SSC in terms of the maximum subspace affinity.
To the best of our knowledge, the best theoretical result of ℓ 1 -SSC, including its geometrical analysis [Soltanolkotabi and Candés, 2012] and the subsequent works on noisy or dimensionality-reduced data [Wang and
Xu, 2013, Wang et al., 2015], requires that the maximum subspace affinity satisfies
[Equation]
under the setting in Soltanolkotabi and Candés [2012] that d k = d 0 and n k = ρd 0 + 1 for all k ∈ [ K ] , so that n =
K ( ρd 0 + 1) .
When n > exp ( d τ 0 ) for τ ∈ (0 . 5 , 0 . 9) , then (29) requires max t 1 ,t 1 ∈
[ K ] ,t 1 ̸ = t 2 aff( S t 1 , S t 2 ) → 0 when d 0 → ∞ , while the condition of noisy ℓ 0 -SSC, (22), only requires that max t 1 ,t 1 ∈
[ K ] ,t 1 ̸ = t 2 aff( S t 1 , S t 2 ) < 1 √ 5 · 1 r 0 −where r 0 can be set to ⌊ 1 λ ⌋ when d 0 → ∞ .
Such less restive condition on the maximum subspace affinity reveals the theoretical advantage of noisy ℓ 0 -SSC over ℓ 1 -SSC.
NOISY ℓ 0 -SSC ON DIMENSIONALITY REDUCED DATA: NOISY-DR- ℓ 0 -SSC
Albeit the theoretical guarantee and compelling empirical performance of noisy ℓ 0 -SSC to be shown in the experimental results, the computational cost of noisy ℓ 0 -SSC is high with the high dimensionality of the data.
In this section, we propose Noisy Dimensionality Reduced ℓ
-SSC
(Noisy-DRℓ 0 -SSC) which performs noisy ℓ 0 -SSC on dimensionality reduced data.
The theoretical guarantee on the correctness of Noisy-DR- ℓ 0 -SSC under the deterministic model as well as its empirical performance are presented.
4.1 METHOD
Noisy-DR- ℓ 0 -SSC performs subspace clustering by the following two steps: 1) obtain the dimension reduced data ˜
X = PX with a linear transformation P ∈ R p ×
d
( p < d ).
2) perform noisy ℓ 0 -SSC on the compressed data ˜ X : min
[Equation]
If p < d , Noisy-DR- ℓ 0 -SSC operates on the compressed data ˜ X rather than on the original data, so that the efficiency is improved.
We will introduce two different types of random projection for Noisy-DR- ℓ 0 -SSC in the following two subsections.
4.2 RANDOM PROJECT INDUCED BY RANDOMIZED LOW-RANK APPROXIMATION
High-dimensional data often exhibits low-dimensional structures, which often leads to low-rankness of the data matrix.
Intuitively, if the data is low rank, then it could be safe to perform noisy ℓ 0 -SSC on its dimensionality reduced version by the linear projection P , and it is expected that P can preserve the information of the subspaces contained in the original data as much as possible, while effectively removing uninformative dimensions.
To this end, we propose to choose P as a random projection induced by randomized low-rank approximation of the data.
The key idea is to obtain an approximate low-rank decomposition of the data.
Using the random projection induced by such low-rank approximation as the linear transformation P , the clustering correctness hold for Noisy-DR- ℓ 0 -SSC with a high probability.
Randomized algorithms are efficient and they have been extensively studied in the computer science and numerical linear algebra literature.
They have been employed to accelerate various numerical matrix computation and matrix optimization problems, including random projection for matrix decomposition [Frieze et al., 2004, Drineas et al., 2004, Sarlos, 2006, Drineas et al., 2006, 2008, Mahoney and Drineas, 2009, Drineas et al., 2011, Lu et al., 2013].
Formally, a random matrix T ∈ R n × p is generated such that each element T ij is sampled independently according to the
Gaussian distribution N (0 , 1) .
QR decomposition is then performed on XT to obtain the basis of its column space, namely XT = QR where Q ∈ R d × p is an orthogonal matrix of rank p and R ∈ R p × p is an upper triangle matrix.
The columns of Q form the orthogonal basis for the sample matrix XT .
An approximation of X is then obtained by projecting X onto the column space of XT : QQ ⊤ X =
QW = (cid:98) X where W = Q ⊤ X ∈ R p × n .
In this manner, a randomized low-rank decomposition of X is achieved by (cid:98) X = QW .
It is proved that the low rank approximation ¯ X is close to X in terms of the spectral norm [Halko et al., 2011].
We present probabilistic result in Theorem 4.1 on the correctness of
Noisy-DR- ℓ 0 -SSC using the random projection induced by randomized low-rank decomposition of the data X , namely
P = Q ⊤ .
In the sequel, ˜ x = Px for any x ∈ R n .
To guarantee the subspace detection property on the dimensionalityreduced data ˜ X , it is crucial to make sure that the conditions, such as (16) and (17) in Theorem 3.3, still hold after the linear transformation.
Each subspace S k is transformed into ˜ S
k = P ( S k ) with dimension ˜ d k .
We denote by ˜ β ∗ an optimal solution to (30), and define C p,p 0 :
= (cid:0) 1 + 17 (cid:113) {+1 + p 0 p − p 0 (cid:1) σ p 0 +1 + 8 √ p p − p 
 +1 ( (cid:80) j>p 
+}
σ 2 j ) 12 with p 0 ≥ 2 .
We also define the following quantities for the convenience of our analysis, which correspond to M i , ¯ σ Y ,r , σ X ,r and µ r used in the analysis on the original data:
[Equation]
where H ˜ y
i , ˜ d k is all the external subspaces of ˜ y
i with dimension no greater than ˜ d k in the transformed space by
P ,
[Equation]
Theorem 4.1.
(Subspace detection property holds for
Noisy-DR- ℓ 0 -SSC under deterministic model) Let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (7) for point x
i with ∥ β ∗ ∥
= r ∗ > 1 , n k ≥ d k +for every 1 ≤ k ≤ K , and there exists 1 < r 0 ≤ d k such that r ∗ ≤ r
=
⌊ 1 λ ⌋ .
Suppose Y is in general position, δ < min 1 ≤ r<r 0 ¯ σ Y ,r , and ˜ M i,δ :=
˜ M i − δ .
Furthermore, suppose the following conditions hold:
[Equation]
where ˜ d max := max [Equation] d
[Equation]
, σ [Equation] Y := min
[Equation]
min
[Equation]
A
[Equation]
: A ⊆
[Equation]
˜ d k and [Equation]
[Equation]
for all y i ∈ S k and 1 ≤ k ≤ K ,
[Equation]
If ˜
[Equation]
where ˜ λ 0 = max { ˜ λ 1 , ˜ λ 2 } and ˜ λ 1 = inf
[Equation]
[Equation]
then with probability at least 1 − 6 e − p , the subspace detection property holds for ˜ x
i with ˜ β ∗ .
Here ˜ M i , ˜ µ r and ˜ σ ˜ X ,rare defined in (32), (35) and (34) respectively.
4.3 RANDOM PROJECTION BY OBLIVIOUS SPARSE NORM-APPROXIMATING PROJECTION (OSNAP)
In this subsection, we study the case when the linear transformation P for the dimensionality reduced ℓ 0 -SSC problem (30) is a sparse matrix, that is, the Oblivious Sparse NormApproximating Projection (OSNAP) [Nelson and Nguyen, 2013].
We choose P as one the most efficient version of
OSNAP, where each column of P only has 1 nonzero element.
Each column of P is chosen to have exactly one nonzero element in a random location, and each nonzero element is ± 1 uniformly at random.
Such P enjoys ℓ 2 -norm preserving property stated as follows.
Let ε be a positive number such that 0 < ε ≤ 1 .
By Theorem 3 in Nelson and
Nguyen [2013], for any subspace S of dimension d in R n ,
Pr (cid:2) (1 − ε )
∥ v ∥ 2 ≤ ∥
Pv ∥ 2 ≤ (1 + ε ) ∥ v ∥ 2 (cid:3) ≥ 1 − δ ′ for all v ∈ S and δ ′ ∈ (0 , 1) if p ≥ d 2 + d δ ′ (2 ε − ε 2 ) 2 .
We note that while the general Johnson-Lindenstrauss transform possesses the similar ℓ 2 -norm preserving property, the sparse matrix P considerably speeds up the computation of ˜ X = PX .
Using the ℓ 2 -norm preserving property of OSNAP, we have the following theorem about the correctness of Noisy-DRℓ 0 -SSC when P is the said OSNAP where each column of
P only has 1 nonzero element.
Theorem 4.2.
(Subspace detection property holds for
Noisy-DR- ℓ 0 -SSC under deterministic model with P being the OSNAP)
Let nonzero vector β ∗ be the optimal solution to the noisy ℓ 0 -SSC problem (7) for point x
i with ∥ β ∗ ∥
= r ∗ , n k ≥ d k + 1 for every 1 ≤ k ≤ K , and there exists 1 < r 0 ≤ ⌊ 1 λ ⌋ such that 1 < r ∗ ≤ r 0 .
Suppose
Y is in general position, y i ∈
S k for some 1 ≤ k ≤ K ,
[Equation]
min
[Equation] <
[Equation]
σ Y ,r .
Let M i,δ :=
M i [-with ∥ β ∗ ∥ 
 = r ∗ >-]− δ , ε be a positive number such that 0 < ε ≤ 1 .
Suppose
[Equation]
[Equation]
and
[Equation]
[Equation]
Then if ˜ λ
[Equation]
<
[Equation]
[-c ∗ + λ ∥ β ∗ ∥ 
-]
λ < 1 ,
[Equation]
44) where ˜ λ 0 :
= max { λ 1 , λ 2 } and λ 1 := inf { 0 < ˜ λ < 1: (cid:113)
[Equation]
then with probability at least 1 −
Kδ
′ for all δ ′ ∈ (0 , 1 K ) , the subspace detection property holds for ˜ x
i with ˜ β ∗ .
Here ˜ µ rand σ ˜ X ,r 0 are defined in (35) and (34) respectively.
˜ β ∗ is the optimal solution to (30) with P being the OSNAP described in the beginning of this subsection with p ≥ d 2max + d max δ ′ (2 ε − ε 2 ) 2 .
4.4 THE ALGORITHM OF NOISY-DR- ℓ 0 -SSC
We denote by Noisy-DR- ℓ 0 -SSC-LR the Noisy-DR- ℓ 0 -SSC with random projection induced by randomized low-rank approximation in Section 4.2, and denote by Noisy-DR- ℓ 0SSC-OSNAP the Noisy-DR- ℓ 0 -SSC with OSNAP serving as the random projection in Section 4.3.
Noisy-DR- ℓ 0 -SSCLR is described by Algorithm 1.
[-min t : (cid:98) β t ̸ =
-]
The algorithm of NoisyDR- ℓ 0 -SSC-OSNAP is similar to Algorithm 1 except that
OSNAP serves as the random projection P , so its details are omitted here.
Algorithm 2 in Section 7.4 of the supplementary describes how to solve the noisy ℓ 0 -SSC problem (7).
1: Generate a Gaussian random matrix T ∈ R n × p where each element T ij is sampled independently according to the standard Gaussian distribution
, 1) 2: Perform QR decomposition on XT , XT = QR where Q ∈ R d × p 3: Set the linear transformation P = Q ⊤ , and obtain the dimensionality reduced data ˜ X = PX
[Table]
We demonstrate the performance of Noisy-DR- ℓ 0 -SSC-LR and Noisy-DR- ℓ 0 -SSC-OSNAP, with comparison to other competing clustering methods including K-means (KM), Spectral Clustering (SC), noisy SSC, Sparse Manifold Clustering and Embedding (SMCE)
[Elhamifar and Vidal, 2011] and SSC-OMP [Dyer et al., 2013] in this section. We will use Noisy-DR- ℓ 0 -SSC to refer to its two variants.
With the coefficient matrix Z obtained by the optimization of noisy ℓ 0 -SSC or Noisy-DR- ℓ 0 -SSC, a sparse similarity matrix is built by W = | Z | + | Z ⊤ |, and spectral clustering is performed on W to obtain the clustering results. Two measures are used to evaluate the performance of different clustering methods, i.e. the Accuracy (AC) and the Normalized Mutual
Information (NMI) [Zheng et al., 2004].
We use randomized rank- p decomposition of the data matrix in Noisy-DR- ℓ 0 -SSC-LR with p = min { d,n }.
It can be observed that noisy ℓ 0 -SSC and Noisy-DR- ℓ 0 -SSC always achieve better performance than other methods in Table 2, including the noisy SSC on dimensionality reduced data (Noisy DR-SSC)
[Wang et al., 2015].
Note that noisy ℓ 0 -SSC has the same performance as ℓ 0 -SSC
[Yang et al., 2018]. Throughout all the experiments we find that the best clustering accuracy is achieved whenever λ is chosen by 0 . 5 < λ < 0 .
95 , justifying our theoretical finding claimed in Remark 3.5 and (18) in Theorem 3.3. For all the methods that involve random projection, we conduct the experiments for 30 times and report the average performance. Note that the cluster accuracy of SSC-OMP on the extended Yale-B data set is reported according to You et al. We randomly sample 1000 images from each class of the MNIST data set so as to collect a total number of 10000 images on which clustering is performed, and the average performance of 10 random sampling is reported for this data set.
The time complexity of noisy ℓ 0 -SSC and the two variants of Noisy-DR- ℓ 0 -SSC are analyzed in Section 7.3 of the supplementary. The actual running time of both algorithms confirms such time complexity, and we observe that NoisyDR- ℓ 0 -SSC-LR is always 8 . 7 times faster than noisy ℓ 0 -SSC with the same number of iterations, and the acceleration is boosted to 9 .
6 times by Noisy-DR- ℓ 0 -SSC-OSNAP due to the sparse random projection.
5.1 JUSTIFICATION OF THEORETICAL ANALYSIS FOR NOISY ℓ 0 -SSC
We further demonstrates the practical implication of our theoretical Analysis for noisy ℓ 0 -SSC. As mentioned in Remark 3.5, a relatively large λ tends to preserve the subspace detection property.
This theoretical finding is consistent with the empirical study shown in this subsection.
We add Gaussian noise of zero mean and different choices of variance σto the extended Yale-B data set. In Section 7.5 of the supplementary, Figure 3a to Figure 3f illustrate SDP violation with respect to λ for different noise levels with σ 2 ranging over 10 , 20 , 30 , 40 , 50 , 60 . The SDP violation is defined in Wang and Xu [2013] which is the percentage of pairs of data points which are mistakenly put in the same subspace by the similarity matrix W , namely the percentage of pairs ( x i , x j ) with nonzero W ij while they are in fact not in the same subspace. It can be observed that increasing λ effectively reduces SDP violation for noisy ℓ 0 -SSC, NoisyDR- ℓ 0 -SSC-LR and Noisy-DR- ℓ 0 -OSNAP, confirming our theoretical prediction.
CONCLUSION
In this paper, we prove that noisy ℓ 0 -SSC recovers subspaces from noisy data through ℓ 0 -induced sparsity.
We then propose Noisy-DR- ℓ 0 -SSC to improve the efficiency of noisy ℓ 0 -SSC, which performs noisy ℓ 0 -SSC on dimensionality reduced data and still provably recovers the underlying subspaces with effectiveness verified by experiments.