Noisy ℓ 0 -Sparse Subspace Clustering on Dimensionality Reduced Data
Abstract a Sparse subspace clustering methods with sparsity induced by ℓ 0 -norm, such as ℓ 0 -Sparse Subspace
Clustering ( ℓ 0 -SSC)
[Yang et al., 2018], are demonstrated to be more effective than its ℓ 1 counterpart such as Sparse Subspace Clustering (SSC)
[Elhamifar and Vidal, 2013].
However, the theoretical analysis of ℓ 0 -SSC is restricted to clean data that lie exactly in subspaces.
Real data often suffer from noise and they may lie close to subspaces.
In this paper, we show that an optimal solution to the optimization problem of noisy ℓ 0 -SSC achieves subspace detection property (SDP), a key element with which data from different subspaces are separated, under deterministic and semi-random model.
Our results provide theoretical guarantee on the correctness of noisy ℓ 0 -SSC in terms of SDP on noisy data for the first time, which reveals the advantage of noisy ℓ 0 -SSC in terms of much less restrictive condition on subspace affinity.
In order to improve the efficiency of noisy ℓ 0 -SSC, we propose Noisy-DRℓ 0 -SSC which provably recovers the subspaces on dimensionality reduced data.
Noisy-DR- ℓ 0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy ℓ 0 -SSC on the projected data for improved efficiency.
Experimental results demonstrate the effectiveness of Noisy-DR- ℓ 0 -SSC.
INTRODUCTION
Clustering is an important unsupervised learning procedure for analyzing a broad class of scientific data.
Highdimensional data, such as facial images and gene expression data, often lie in low-dimensional subspaces in many cases, and clustering in accordance to the underlying subspace structure is particularly important.
Among various subspace clustering algorithms, the ones that employ sparsity prior, such as Sparse Subspace Clustering (SSC) [Elhamifar and Vidal, 2013] and ℓ 0 -Sparse Subspace Clustering ( ℓ 0 -SSC)
[Yang et al., 2018], have been proven to be effective in separating the data in accordance with the subspaces that the data lie in under certain assumptions.
Furthermore,
Sparse Additive Subspace Clustering [Yuan and Li, 2014] considers a nonlinear transformation of each data point such that the transformed point can be linearly represented by data in the same subspace as that point, extending the usual linear representation by SSC.
Sparse subspace clustering methods construct the sparse similarity matrix by sparse representation of the data.
Subspace detection property (SDP) defined in Section 2 ensures that the similarity between data from different subspaces vanishes in the sparse similarity matrix, and applying spectral clustering [Ng et al., 2001] on such sparse similarity matrix leads to compelling clustering performance.
Elhamifar and Vidal [Elhamifar and Vidal, 2013] prove that when the subspaces are independent or disjoint, SDP can be satisuyfied by solving the canonical sparse linear representation problem using data as the dictionary, under certain conditions on the rank, or singular value of the data matrix and the principle angle between the subspaces.
Under the independence assumption on the subspaces, low rank representation [Liu et al., 2013, Liu and Li, 2014, 2016] is also proposed to recover the subspace structures.
Relaxing the assumptions on the subspaces to allowing overlapping subspaces, the Greedy Subspace Clustering [Park et al., 2014] and the Low-Rank Sparse Subspace Clustering [Wang et al., 2013] achieve subspace detection property with high probability.
The geometric analysis in Soltanolkotabi and Candés [2012] shows the theoretical results on subspace recovery by SSC.
In the following, we use the term SSC or ℓ 1 -SSC exchangeably to indicate the Sparse Subspace Clustering method in Elhamifar and Vidal [2013].
Real data often suffer from noise.
The correctness of noisy
SSC is analyzed in Wang and Xu [2013] which handles noisy data that lie close to disjoint or overlapping subspaces, Please refer to Section 1.2 for the definition of notations. [Table] and the original optimization problem of noisy SSC is proposed in Elhamifar and Vidal [2013].
While Yang et al.
[2016], Yang [2018] prove the correctness of ℓ 0 -SSC or its dimensionality reduced variant on clean data based on a constrained ℓ 0 -minimization problem, it empirically solves an unconstrained ℓ 0 -regularized problem to handle noise in data, and they lack theoretical analysis on the correctness of ℓ 0 -SSC on noisy data.
fhandles noisy data that lie close to disjoint or overlapping subspaces.
While ℓ 0 -SSC [Yang et al., 2018] has guaranteed clustering correctness via subspace detection property under much milder assumptions than previous subspace clustering methods including SSC, it assumes that the observed data lie in exactly in the subspaces and does not handle noisy data.
In this paper, we present noisy ℓ 0 -SSC, which enhances ℓ 0 -SSC by theoretical guarantee on the correctness of clustering on noisy data.
It should be emphasized that while ℓ 0 -SSC on clean data [Yang et al., 2018] empirically adopts a form of optimization problem robust to noise, it lacks theoretical analysis on the correctness of ℓ 0 -SSC on noisy data.
In this paper, the correctness of noisy ℓ 0 -SSC on noisy data in terms of the subspace detection property is established.
Our analysis is under both deterministic model and semi-random model, which are the models employed in the geometric analysis of SSC [Soltanolkotabi and Candés, 2012].
Our randomized analysis demonstrates the advantage of noisy ℓ 0 -SSC over its ℓ 1 counterpart as more general assumption on data distribution can be adopted.
Moreover, we present Noisy Dimensionality Reduced ℓ 0 -Sparse Subspace Clustering (Noisy-DR- ℓ 0 -SSC), an efficient version of noisy ℓ 0 -SSC which also enjoys robustness to noise.
Noisy-DRℓ 0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy ℓ 0 -SSC on the dimensionality reduced data.
Noisy-DR- ℓ 0 -SSC provably recovers the underlying subspace structure in the original data from the projected data under deterministic model.
Experimental results show the effectiveness of both noisy ℓ 0 -SSC and Noisy-DR- ℓ 0 -SSC.
1.1 NOTATIONS
We use bold letters for matrices and vectors, and regular lower letter for scalars throughout this paper.
The bold letter with superscript indicates the corresponding column of a matrix, e.g. A i is the i -th column of matrix A , and the bold letter with subscript indicates the corresponding element of a matrix or vector.
∥ · ∥ F and ∥ · ∥ p denote the Frobenius norm and the vector ℓ p -norm or the matrix p -norm, and ∥·∥is the ℓ 0 -norm, that is, the number of nonzero elements of a vector.
diag( · ) indicates the diagonal elements of a matrix.
H T ⊆
R d indicates the subspace spanned by the columns of T , and A I denotes a submatrix of A whose columns correspond to the nonzero elements of I (or with indices in
I without confusion).
σ t ( · ) denotes the t -th largest singular value of a matrix, and σ min ( · ) indicates the smallest singular value of a matrix.
supp( · ) is the support of a vector, P S ′ is the operator of orthogonal projection onto the subspace
S ′ .
[ n ] represents all the natural numbers between 1 and n inclusively.
S d − 1 denotes the unit sphere in R d .
Θ( a ) denotes a number such that there exists two constants c 1 and c
such that Θ( a ) ∈
[ c 1 a, c 2 a ] .
1.2 CONTRIBUTIONS
First, the correctness of noisy ℓ 0 -SSC on noisy data in terms of the subspace detection property is established for the first time, which is presented in Section 3 of this paper.
Our analysis is under both deterministic model and semi-random model, which are also the models employed by the geometric analysis of SSC [Soltanolkotabi2012].
Our randomized analysis demonstrates the significant advantage of noisy ℓ 0 -SSC over its ℓ 1 counterpart and other competing subspace clustering methods in terms of much less restrictive condition on the subspace affinity.
Table 1 below demonstrates the conditions under which SDP holds for representative subspace clustering methods under the semi-random model, including Greedy Subspace Clustering (GSC)
[Park et al., 2014], ℓ 1 -SSC [Elhamifar and Vidal, 2011], Noisy SSC [Wang and Xu, 2013], Affine Sparse Subspace Clustering (ASSC)
[Li et al., 2018, You et al., 2019], Noisy ℓ 0 -SSC [Yang et al., 2018].
When the size of data n grows exponentially in terms of the common subspace dimension d 0 (the dimension of every subspace is d 0 ), in particular, n = Θ( e d τ 0 ) for τ ∈ (0 . 5 , 0 . 9) , then all the competing subspace clustering methods other than Noisy ℓ 0 -SSC either do not allow overlapping subspaces, or require the maximum pairwise subspace affinity goes to 0 when d 0 → ∞ , which means that these methods require all the subspaces to be almost pairwise orthogonal when the common subspace dimension d 0 is very large.
Instead, Noisy ℓ 0 -SSC allows subspace affinity to be lower bounded from 0 , suggesting that Noisy ℓ 0 -SSC is still able to recover subspaces which are not orthogonal when d 0 is very large.
σ ′ min in Table 1 is defined in Theorem 3.6.
In Table 1, it is preferred that a subspace clustering method requires milder conditions, which are allowing overlapping subspaces and larger upper bound for the maximum subspace affinity, denoted by max k,
l ∈
[ K ] aff ( S k , S l ) where {S k } Kk =1 are K subspaces, so that the underlying subspaces can be recovered for overlapping subspaces and for subspaces which are closer to each other (larger subspace affinity).
Here aff denotes subspace affinity, d 0 is the common subspace dimension, δ is a small positive constant (see [Park et al., 2014]).
r 0 > 1 is an upper bound for the support of an optimal solution to the noisy ℓ 0 -SSC problem for all data points.
In this table, n = Θ( e d τ 0 ) for τ ∈ (0 . 5 , 0 . 9) when d 0 → ∞ .
Note that two subspaces are overlapping subspaces if the dimension of their intersection is larger than 1 .
When a subspace clustering method does not allow overlapping subspaces, then no condition on subspace affinity is presented in the subspace clustering literature.
Second, we propose Noisy Dimensionality Reduced ℓ 0Sparse Subspace Clustering (Noisy-DR- ℓ 0 -SSC) to accelerate noisy ℓ 0 -SSC with provable robustness to noise.
NoisyDR- ℓ 0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy ℓ 0 -SSC on the dimensionality reduced data.
Two types of random projections are used in Noisy-DR- ℓ 0 -SSC, which are the random projection induced by randomized low-rank approximation and the sparse random projection, particular "Count-Sketch (SC) Projections”.
It should be emphasized that Yang [2018] also studies dimensionality reduced ℓ 0 -SSC.
However, the analysis of this work is performed on the following constrained ℓ 0minimization problem and only for clean data without noise,
On the other hand, the actual optimization problem [Yang, 2018] solves a unconstrained ℓ 0 -regularized problem.
In contrast, we analyze noisy ℓ 0 -SSC on the unconstrained ℓ 0 -regularized problem with noisy data which reveals the advantage of noisy ℓ 0 SSC over ℓ 1 -SSC.
Our analysis also suggests that a larger λ tends to guarantee the subspace detection property (Remark 3.5), verified by experiments.
Throughout the paper, we refer to ℓ 0 -SSC for noisy data with the unconstrained ℓ 0 -regularized problem as noisy ℓ 0 -SSC.
PROBLEM SETUP
Sparse Subspace Clustering (SSC) methods, such as [Elhamifar and Vidal, 2011, Soltanolkotabi and Candés, 2012,
Wang and Xu, 2013, Yuan and Li, 2014], construct a sparse similarity matrix by sparse representation of the data, and then perform clustering on the sparse similarity matrix.
We hereby introduce the notations for subspace clustering on noisy data considered in this paper.
The uncorrupted data matrix is denoted by Y =
[ y 1 , . . .
, y n ] ∈ R d × n , where d is the dimensionality and n is the size of the data.
The uncorrupted data Y lie in a union of K distinct subspaces {S k } Kk =1 of dimensions { d k } Kk =1 with d max :=
max k ∈ [ K ] d k and d min :=
min k ∈ [ K ] d k .
The observed noisy data is X =
Y + N , where N =
[ n 1 , . .
. , n n ] ∈ R d × n is the additive noise.
x i =
y i + n i is the noisy data point that is corrupted by the noise n i .
We let Y ( k ) ∈
R
d × n k denote the data belonging to subspace S k with K (cid:80) k
=1 n k = n , and denote the corresponding columns in X by X ( k ) .
Let
U ( k ) ∈ R
d × d k be the orthogonal basis of S k for all k ∈ [ K ] .
The data X are normalized such that each column has unit ℓ 2 -norm in our deterministic analysis.
We consider deterministic noise model where the noise N is fixed and max i ∈ [ n
[Equation]
Formally, given observed data X ∈
R d × n , X = [ x 1 , . .
, x n ] , where x i ∈ R d , SSC solves the following optimization problem for each i ∈ [ n ] :
[Equation]
In order to handle noisy data, noisy SSC [Wang and Xu, 2013] was proposed to solve the ℓ 1 regularized problem:
[Equation]
The sparse code β of the data point x i is obtained by solving (1) or (2) for SSC or noisy SSC.
A coefficient matrix
Z ∈ R n × n is then formed by concatenating the sparse codes of all the data points, and the i -th column of Z is the sparse code of x i .
The sparse similarity matrix is then computed by [Equation]
Subspace detection property (SDP, formally defined later) ensures that the similarity between data from different subspaces vanishes in the sparse similarity matrix.
If SDP holds, then similarity between data points from different clusters vanish in W .
As a result, performing spectral clustering on W leads to compelling clustering results.
Under the independence assumption on the subspaces, low rank representation [Liu et al., 2013] is proposed to recover the subspace structures.
Relaxing the assumptions on the subspaces to allowing overlapping subspaces, the Greedy
Subspace Clustering [Park et al., 2014] and the Low-Rank
Sparse Subspace Clustering [Liu et al., 2013] achieve subspace detection property with high probability.
The geometric analysis in Soltanolkotabi and Candés [2012] shows the theoretical results on subspace recovery by SSC.
In the following text, we use SSC or ℓ 1 -SSC exchangeably to indicate the Sparse Subspace Clustering method in Soltanolkotabi and Candés [2012], Elhamifar and Vidal [2013].
ℓ 0 -SSC [Yang et al., 2018] proposes to solve the following ℓ 0 sparse representation problem
[Equation]
and it proves that SDP is satisfied with an globally optimal solution to problem (3).
In Yang et al.
[2018], the ℓ 0 regularized sparse approximation problem below is solved so as to handle noisy data for ℓ 0 -SSC, which is the optimization problem of noisy ℓ 0 -SSC:
[Equation]
The optimization problem of noisy ℓ 0 -SSC (4) is separable.
For each i ∈ [ n ] , the optimization problem with respect to the sparse code β of i -th data point is
[Equation]
The sparse similarity matrix W is then computed in the same way as ℓ 1 -SSC by W = | Z | + | Z ⊤ |, and the subspace clustering result of noisy ℓ 0 -SSC is achieved by performing spectral clustering on W .
In the following text, we always use β ∗ to denote an optimal solution to (5), and define r ∗ :=
∥ β ∗ ∥ 0 .
The definition of subspace detection property for noisy ℓ 0SSC and noiseless ℓ 0 -SSC, i.e. ℓ 0 -SSC on noiseless data, is defined in Definition 2.1 below.
Definition 2.1.
(Subspace detection property for noisy and noiseless ℓ 0 -SSC) Let Z ∗ be an optimal solution to (4).
The subspaces {S k } Kk =1 and the data X satisfy the Subspace Detection Property (SDP) for noisy ℓ 0 -SSC if Z i is a nonzero vector, and nonzero elements of Z i correspond to the columns of X from the same subspace as y i for all 1 ≤ i ≤ n .
We say that SDP holds for x i if nonzero elements of Z ∗ i , which is β ∗ for problem (5), correspond to the data that lie in the same subspace as y i , for either noisy ℓ 0 -SSC or noiseless ℓ 0 -SSC.
ANALYSIS FOR NOISY ℓ 0 -SSC
Similar to Soltanolkotabi and Candés [2012], we introduce the deterministic and the semi-random model for the analysis of noisy ℓ 0 -SSC.
• Deterministic Model: the subspaces and the data in each subspace are fixed.
• Semi-Random Model: the subspaces are fixed but the data are independent and identically distributed in each of the subspaces.
The data in the above definitions refer to clean data without noise.
Both the deterministic model and the semirandom model are extensively employed to analyze the subspace detection property in the subspace learning literature [Soltanolkotabi and Candés, 2012, Wang et al., 2013,
Wang and Xu, 2013, Acharyya and Ghosh, 2015].
3.1 NOISY ℓ 0 -SSC: DETERMINISTIC ANALYSIS
We first introduce the definition of general position and external subspace before our analysis on noisy ℓ 0 -SSC.
Definition 3.1.
(General position)
For any 1 ≤ k ≤ K , the data Y ( k ) are in general position if any subset of L ≤ d k data points (columns) of Y ( k ) are linearly independent.
Y are in general position if Y ( k ) are in general position for 1 ≤ k ≤ K .
The assumption of general condition is rather mild.
In fact, if the data points in X ( k ) are independently distributed according to any continuous distribution, then they almost surely in general position.
Let the distance between a point x ∈ R d and a subspace
S ⊆
R d be defined as d ( x , S ) =
inf y ∈S ∥ x − y ∥ 2 , the definition of external subspaces is presented as follows.
Definition 3.2.
(External subspace of limited dimension)
For a point y ∈ Y ( k ) , a subspace H { y ij } Lj =1 spanned by a set of linear independent points { y i j } Lj =1 ⊆
Y is defined to be an external subspace of y
if { y i j } L j =1 ̸⊆ Y ( k ) and y / ∈ { y i j } Lj =1 .
The set of all external subspaces of y of dimension no greater than r with r ≥ 1 for y is denoted by H y ,r , that is, H y ,r
= { H : H
= H { y ij }
Lj =1 , dim[ H ] =
L, L ≤ r, { y i j } Lj =1 ̸⊆ Y ( k ) , y / ∈ { y i j } Lj =1 } .
The point y is said to be away from its external subspaces of dimension r if min H ∈H y ,r d ( y , H ) > 0 .
All the data points in Y ( k ) are said to be away from the external subspaces if each of them is away from the its associated external spaces.
We also need the definitions related to the spectrum of X and
Y , which are defined as follows.
In the following analysis, we employ β to denote the sparse code of datum x i so that a simpler notation other than Z i is dedicated to our analysis.
Definition 3.3.
The minimum restricted eigenvalue of the uncorrupted data is defined as
[Equation]
for r ≥ 1 .
In addition, the normalized minimum restricted eigenvalue of the uncorrupted data is defined by
[Equation]
Moreover, the following quantities are defined for our analysis.
We define
[Equation]
where
[Equation]
with δ < ¯ σ ∗ Y , and ¯ σ ∗ Y is defined as
[Equation]
Now we present our main result on noisy ℓ 0 -SSC.
Theorem 3.1.
(Subspace detection property holds for noisy ℓ 0 -SSC) Let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (5) for point x
i with ∥ β ∗ ∥
= r ∗ ≥ 1 , and c ∗ := ∥ x i − X β ∗ ∥ 2 .
Suppose Y is in general position, y i ∈
S k for some 1 ≤ k ≤ K , δ < ¯
σ ∗ Y , λ > τ 0 ,
B ( y i , δ + c ∗ + 2 δ √ r ∗ σ ∗ X ) ∩ H = ∅ for any H ∈ H
y i ,r ∗ .
Then the subspace detection property holds for x i with β ∗ .
Here τ 0 , τ 1 , ¯ σ ∗ Y and σ ∗ X are defined in (6), (7) and (8).
Remark 3.2.
When δ = 0 and there is no noise in the data
X , the conditions for the correctness of noisy ℓ 0 -SSC in
Theorem 3.1 almost reduce to that for noiseless ℓ 0 -SSC.
To see this, the conditions are reduced to B ( y i , c ∗ )
∩ H = ∅ , which are exactly the conditions required by noiseless ℓ 0SSC in Lemma 1.1 in the supplementary, namely data are away from the external subspaces by choosing λ → 0 and it follows that c ∗ = 0 .
While Theorem 3.1 establishes geometric conditions under which the subspace detection property holds for noisy ℓ 0SSC, it can be seen that these conditions are often coupled with an optimal solution β ∗ to the noisy ℓ 0 -SSC problem (5).
In the following theorem, the correctness of noisy ℓ 0 -SSC is guaranteed in terms of λ , the weight for the ℓ 0 regularization term in (5), and the geometric conditions independent of an optimal solution to (5).
Let M i > 0 be the minimum distance between y i ∈
S k and its external subspaces when y i is away from its external subspaces of dimension r , that is,
[Equation]
The following two quantities related to the spectrum of clean and noisy data, µ r and σ X ,r , are defined as follows with r > 1 for the analysis in Theorem 3.3.
[Equation]
Theorem 3.3.
(Subspace detection property holds for noisy ℓ 0 -SSC under deterministic model with conditions in terms of λ )
Let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (5) for point x
i with ∥ β ∗ ∥
= r ∗ > 1 , n k ≥ d k +1 for every k ∈
[ K ] , and there exists 1 < r 0 ≤ ⌊ 1 λ ⌋ such that r ∗ ≤ r 0 .
Suppose Y is in general position, y i ∈
S k for some 1 ≤ k ≤ K , δ < ¯
σ ∗ Y , and M i,δ :=
M i − δ .
Suppose
M i,
[Equation]
12) and
[Equation]
Then if λ 0 < λ < 1 ,
[Equation]
14) where λ 0 :=
max { λ 1 , λ 2 } and
[Equation]
[Equation]
the subspace detection property holds for x i with β ∗ .
Here
M i , µ r 0 , σ X ,r 0 are defined in (9), (10), (11) respectively.
Remark 3.4.
The two conditions (12) and (13) are induced by two conditions, B ( y i , δ + c ∗ + 2 δ √ r ∗ σ ∗ X ) ∩ H = ∅ for any
H ∈ H
y i ,d k and λ > τ 0 respectively, which are required by
Theorem 3.1.
Note that when (12) and (13) hold, λ 1 and λcan always be chosen in accordance with (15) and (16).
Remark 3.5.
It can be observed from condition (14) that noisy ℓ 0 -SSC encourages sparse solution by a relatively large λ so as to guarantee the subspace detection property.
This theoretical finding is consistent with the empirical study shown in the experimental results.
3.2 NOISY ℓ 0 -SSC: RANDOMIZED ANALYSIS
The correctness of noisy ℓ 0 -SSC is analyzed under the semirandom model that the data in subspace S ( k ) are i.i.d.
according to the uniform distribution on the unit sphere, S d k − 1 , of R d k centered at the origin for all k ∈ [ K ] .
This setting is employed extensively in the subspace learning literature [Soltanolkotabi and Candés, 2012, Wang et al., 2013,
Wang and Xu, 2013, Acharyya and Ghosh, 2015].
We then have the major theorem below stating the theoretical guarantee of the subspace detection property of noisy ℓ
-SSC under the semi-random model.
Before stating this theorem, we introduce the following definition of subspace affinity, which is widely used in the analysis of semi-random model in the sparse subspace clustering literature.
Definition 3.4.
(Subspace affinity)
The affinity between two subspaces, S k and S l with k, l ∈ [ K ] , is defined by
[Equation]
where cos θ ( t ) kl is the t -th canonical angle between S k and S l defined in Soltanolkotabi and Candés [2012].
Let U ( k ) and
U ( l ) be the orthonormal basis for S k and S l respectively, then
[Equation]
v t ∥ u t ∥ 2 ∥ v t ∥ 2 , with orthogonality: u ⊤ u j = 0 , v ⊤ v j = 0 , j = 1 , . . .
, t − 1 .
It can be verified that aff( S k , S l ) =
∥ U ( k )
⊤ U ( l ) ∥ F .
Theorem 3.6.
(Subspace detection property holds for noisy ℓ 0 -SSC under semi-random model with conditions in terms of λ )
Under the semi-random model, let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (5) for point x
i with ∥ β ∗ ∥
= r ∗ > 1 , n k ≥ d k +for every 1 ≤ k ≤ K , and there exists 1 < r 0 ≤ ⌊ 1 λ ⌋ such that r ∗ ≤ r 0 .
Suppose c 1 > 0 is an arbitrary small constant, ε 0 , ε 1 > 0 be small constants, and d k is large enough such that d k ≥ ⌊ 1 λ ⌋ , 2 d − 0 .
05 k
+ 2 d − 0 .
1 k ≤ ε 0 and (cid:113)
[Equation]
+ (cid:113) 2 λd k log en k r 0 ≤ ε 1 hold for all k ∈
[ K ] .
Define
[Equation]
[Equation]
For t > 0 such that 1 d max −
[Equation]
(cid:113) 1 − 1 d max
[Equation]
2 > 0 , suppose
[Equation] [Equation]
where M := sup 0 ≤ t< 1 2 t 3 arccos t π < 1 , λ ′:= max { λ ′ 1 , λ ′ 2 } , λ ′ 1 := inf { 0 < λ < 1:
[Equation]
When the conditions in Lemma 1.5 of the supplementary hold for all k ∈ [ K ] and every point y ∈
Y ( k ) , then with probability at least
[Equation]
k =
[Equation]
n k exp( − d k t 2 2 ) , the subspace detection property holds for x i with β ∗ for all i ∈ [ n ] .
Remark 3.7 (Advantage of Noisy ℓ 0 -SSC in terms of Subspace Affinity) .
It is well known that the difficulty of achieving the subspace detection property increases with larger affinity between subspaces, that is, the subspace are closer to each other.
Our analysis reveals the significant advantage of noisy ℓ 0 -SSC over ℓ 1 -SSC in terms of the maximum subspace affinity.
To the best of our knowledge, the best theoretical result of ℓ 1 -SSC, including its geometrical analysis [Soltanolkotabi and Candés, 2012] and the subsequent works on noisy or dimensionality-reduced data [Wang and
Xu, 2013, Wang et al., 2015], requires that the maximum subspace affinity satisfies
[Equation]
under the setting in Soltanolkotabi and Candés [2012] that d k = d 0 and n k = ρd 0 + 1 for all k ∈ [ K ] , so that n =
K ( ρd 0 + 1) .
When n > exp ( d τ 0 ) for τ ∈ (0 . 5 , 0 . 9) , then (23) requires max t 1 ,t 1 ∈
[ K ] ,t 1 ̸ = t 2 aff( S t 1 , S t 2 ) → 0 when d 0 → ∞ , while the condition of noisy ℓ 0 -SSC, (18), only requires that max t 1 ,t 1 ∈
[ K ] ,t 1 ̸ = t 2 aff( S t 1 , S t 2 ) < σ ′ 2min r 0 − 1 when d 0 is sufficiently large.
Such less restive condition on the maximum subspace affinity reveals the theoretical advantage of noisy ℓ 0 -SSC over ℓ 1 -SSC.
NOISY ℓ 0 -SSC ON DIMENSIONALITY REDUCED DATA: NOISY-DR- ℓ 0 -SSC
Albeit the theoretical guarantee and compelling empirical performance of noisy ℓ 0 -SSC to be shown in the experimental results, the computational cost of noisy ℓ 0 -SSC is high with the high dimensionality of the data.
In this section, we propose Noisy Dimensionality Reduced ℓ
-SSC
(Noisy-DRℓ 0 -SSC) which performs noisy ℓ 0 -SSC on dimensionality reduced data.
The theoretical guarantee on the correctness of Noisy-DR- ℓ 0 -SSC under the deterministic model as well as its empirical performance are presented.
4.1 METHOD
Noisy-DR- ℓ 0 -SSC performs subspace clustering by the following two steps: 1) obtain the dimension reduced data ˜
X = PX with a linear transformation P ∈ R p ×
d
( p < d ).
2) perform noisy ℓ 0 -SSC on the compressed data ˜ X :
[Equation]
If p < d , Noisy-DR- ℓ 0 -SSC operates on the compressed data ˜ X rather than on the original data, so that the efficiency is improved.
We introduce two types of random projection for Noisy-DR- ℓ 0 -SSC in the following two subsections.
4.2 RANDOMIZED LOW-RANK APPROXIMATION
High-dimensional data often exhibits low-dimensional structures, which often leads to low-rankness of the data matrix.
Intuitively, if the data is low rank, then it could be safe to perform noisy ℓ 0 -SSC on its dimensionality reduced version by the linear projection P , and it is expected that P can preserve the information of the subspaces contained in the original data as much as possible, while effectively removing uninformative dimensions.
To this end, we propose to choose P as a random projection induced by randomized low-rank approximation of the data.
The key idea is to obtain an approximate low-rank decomposition of the data.
Using the random projection induced by such low-rank approximation as the linear transformation P , the clustering correctness hold for Noisy-DR- ℓ 0 -SSC with a high probability.
The merit of random projection (RP) is highlighted by the celebrated Johnson-Lindenstrauss Lemma [Johnson and Lindenstrauss, 1984]. In the past 20 years or more, RP has been used extensively in dimension reduction, approximate near neighbor search, compressed sensing, computational biology, etc [Dasgupta, 2000, Bingham and Mannila, 2001, Buhler, 2001, Achlioptas, 2003, Fern and Brodley, 2003,
Datar et al., 2004, Candès et al., 2006, Donoho, 2006, Freund et al., 2007, Li, 2007, 2017, 2019]. In particular, RP has been employed to accelerate numerical matrix computation and matrix optimization problems, including matrix decomposition [Frieze et al., 2004, Drineas et al., 2004, Sarlós, 2006, Drineas et al., 2006, 2008, Mahoney and Drineas, 2009, Drineas et al., 2011, Lu et al., 2013].
Formally, a random matrix T ∈ R n × p is generated such that each element T ij is sampled independently according to the
Gaussian distribution N (0 , 1) .
QR decomposition is then performed on XT to obtain the basis of its column space, namely XT = QR where Q ∈ R d × p is an orthogonal matrix of rank p and R ∈ R p × p is an upper triangle matrix.
The columns of Q form the orthogonal basis for the sample matrix XT .
An approximation of X is then obtained by projecting X onto the column space of XT : QQ ⊤ X =
QW = (cid:98) X where W = Q ⊤ X ∈ R p × n .
In this manner, a randomized low-rank decomposition of X is achieved by [Equation]
It is proved that the low rank approximation ¯ X is close to X in spectral norm [Halko et al., 2011].
We present probabilistic result in Theorem 4.1 on the correctness of Noisy-DR- ℓ 0SSC using the random projection induced by randomized low-rank decomposition of the data X , namely P = Q ⊤ .
In the sequel, ˜ x = Px for any x ∈ R n .
To guarantee the subspace detection property on the dimensionality-reduced data ˜ X , it is crucial to ensure that the conditions, such as (12) (13) in Theorem 3.3, still hold after linear transformation.
Each subspace S k is transformed into ˜ S
k = P ( S k ) with dimension ˜ d k .
We denote by ˜ β ∗ an optimal solution to (24), and define C p,p 0 : = (cid:0) [Equation]
17 (cid:113) [Equation] p p [-− p 
-][Equation]
p 0 +1 ( (cid:80) j>p
σ 2 j ) 12 with p 0 ≥ 2 .
We also define the following quantities for the convenience of our analysis, which correspond to M i , ¯ σ Y ,r , σ X ,r and µ r used in the analysis on the original data:
[Equation]
where H ˜ y
i , ˜ d k is all the external subspaces of ˜ y
i with dimension no greater than ˜ d k in the transformed space by P ,
[Equation] [Equation] [Equation]
[Equation]
Theorem 4.1.
(Subspace detection property holds for
Noisy-DR- ℓ 0 -SSC under deterministic model) Let nonzero vector β ∗ be an optimal solution to the noisy ℓ 0 -SSC problem (5) for point x
i with ∥ β ∗ ∥
= r ∗ > 1 , n k ≥ d k +for every 1 ≤ k ≤ K , and there exists 1 < r 0 ≤ d k such that r ∗ ≤ r [-
 =-]0 ≤ ⌊ 1 λ ⌋ .
Suppose Y is in general position, δ < min 1 ≤ r<r 0 ¯ σ Y ,r , and ˜ M i,δ :=
˜ M i − δ .
Furthermore, suppose the following conditions hold: (i) C p,p
[Equation]
max
[Equation]
[Equation]
,K
[Equation]
[Equation]
) Y , where ˜ d max := max k ˜ d k , σ ( k ) Y := min { σ min ( A ): A ⊆
Y ( k ) , A ∈ R d × n ′ , n ′ ≤ ˜ d k } , (ii δ (1 + 2 √ r 0 )
< min 1 ≤ r<r 0 ¯ σ Y ,r − C p,p 0 , ( iii) min 1 ≤ r ≤ ˜ d k σ Y ,r > C p,p 0 − 2 δ (cid:112) ˜
d k and
M i − C p,p 0 (1 + min 1 ≤ r ≤ ˜ d k σ Y ,r − C p,p 0 − 2 δ (cid:112) ˜ d k )
> δ + 2 δ σ X ,r 0 − C p,p 0 , for all y i ∈
S k and 1 ≤ k ≤ K , (iv) min 1 ≤ r<r 0 ¯ σ Y ,r 0 > C p,p 0 − 2 δ √ r 0 − δ and δ min 1 ≤ r<r 0 ¯ σ Y ,r 0 − C p,p 0 − 2 δ √ r 0 − δ < 1 − 2 δ σ X ,r 0 − C p,p 0 .
If ˜ λ 0 < ˜ λ < 1 , where ˜ λ 0 = max { ˜ λ 1 , ˜ λ
2 } and ˜ λ 1 = inf { 0 < ˜ λ < 1: (cid:113) 1 −
[Equation]
then with probability at least 1 − 6 e − p , the subspace detection property holds for ˜ x
i with ˜ β ∗ .
Here ˜ M i , ˜ µ r and ˜ σ ˜ X ,rare defined in (25), (28) and (27) respectively.
4.3 VERY SPARSE RANDOM PROJECTIONS
In this subsection, we study the case when the linear transformation P for the dimensionality reduced ℓ 0 -SSC problem (24) is a sparse matrix [Charikar et al., 2004, Cormode and Muthukrishnan, 2005, Li, 2007, Weinberger et al., 2009, Gilbert and Indyk, 2010, Li et al., 2011, Nelson and Nguyen, 2013, Li and Zhao, 2022].
In particular, we choose P such that each column of P only has 1 nonzero element, in a fashion known as “count-sketch”
[Charikar et al., 2004].
Weinberger et al.
[2009] applied count-sketch as a dimension reduction tool for machine learning.
The work of [Li et al., 2011], in addition to developing hash learning algorithm based on minwise hashing, also provided the thorough theoretical analysis for count-sketch in the context of estimating inner products. The conclusion from Li et al.
[2011] is that, to estimate inner products, we should use count-sketch (or very sparse random projections [Li, 2007]) instead of the original (dense) random projections, because count-sketch is not only computationally much more efficient but also (slightly) more accurate, as far as the task of similarity estimation is concerned.
Using those nice theoretical properties of count-sketch projections, we have the following theorem about the correctness of Noisy-DR- ℓ 0 -SSC when P has only 1 nonzero element in each column. For breveity, we name such a projection matrix to be “CSP".
Theorem 4.2.
(Subspace detection property holds for
Noisy-DR- ℓ 0 -SSC under deterministic model with P being the CSP) Let nonzero vector β ∗ be the optimal solution to the noisy ℓ 0 -SSC problem (5) for point x i with ∥ β ∗ ∥
= r ∗ , n k ≥ d k + 1 for every 1 ≤ k ≤ K , and there exists 1 < r 0 ≤ ⌊ 1 λ ⌋ such that 1 < r ∗ ≤ r 0 .
Suppose
Y is in general position, y i ∈
S k for some 1 ≤ k ≤ K , δ < min 1 ≤ r<r 0 ¯ σ Y ,r .
Let M i,δ :=
M i − δ , ε be a positive number such that 0 < ε ≤ 1 .
Suppose
[Equation]
Then if ˜ λ 0 < ˜ λ < 1 , where ˜ λ 0 := max { λ 1 , λ 2 } and λ 1 := inf { 0 < ˜ λ < 1: (cid:113)
[Equation]
then with probability at least 1 −
Kδ
′ for all δ ′ ∈ (0 , 1 K ) , the subspace detection property holds for ˜ x
i with ˜ β ∗ .
Here ˜ µ rand σ ˜ X ,r 0 are defined in (28) and (27) respectively.
˜ β ∗ is the optimal solution to (24) with P being the CSP described in the beginning of this subsection with p ≥ d 2max + d max δ ′ (2 ε − ε 2 ) 2 .
4.4 THE ALGORITHM OF NOISY-DR- ℓ 0 -SSC
We denote by Noisy-DR- ℓ 0 -SSC-LR the Noisy-DR- ℓ 0 -SSC with random projection induced by randomized low-rank approximation in Section 4.2, and denote by Noisy-DR- ℓ 0SSC-CSP the Noisy-DR- ℓ 0 -SSC with CSP serving as the random projection in Section 4.3.
Noisy-DR- ℓ 0 -SSC-LR is described by Algorithm 1.
The algorithm of Noisy-DR- ℓ 0 -SSC-CSP is similar to Algorithmexcept that CSP serves as the random projection P .
Algorithm 1 in Section 4 of the supplementary describes how to solve the noisy ℓ 0 -SSC problem (5).
EXPERIMENTS
We demonstrate the performance of Noisy-DR- ℓ 0 -SSC-LR and Noisy-DR- ℓ 0 -SSC-CSP, with comparison to other competing clustering methods including K-means (KM), Spectral Clustering (SC), noisy SSC, Sparse Manifold Clustering and Embedding (SMCE)
[Elhamifar and Vidal, 2011] and SSC-OMP
[Dyer et al., 2013] in this section.
We will use Noisy-DR- ℓ 0 -SSC to refer to its two variants.
With the coefficient matrix Z obtained by the optimization of noisy ℓ 0 -SSC or Noisy-DR- ℓ 0 -SSC, a sparse similarity matrix is built by W = | Z | + | Z ⊤ |, and spectral clustering is performed on W to obtain the clustering results.
Two measures are used to evaluate the performance of different clustering methods, i.e. the Accuracy (AC) and the Normalized Mutual
Information (NMI)
[Zheng et al., 2004].
We use randomized rank- p decomposition of the data matrix in Noisy-DR- ℓ 0 -SSC-LR with p = min { d,n }.
It can be observed that noisy ℓ 0 -SSC and Noisy-DR- ℓ 0 -SSC always achieve better performance than other methods in
Table 2, including the noisy SSC on dimensionality reduced data (Noisy DR-SSC)
[Wang et al., 2015].
Note that noisy ℓ 0 -SSC has the same performance as ℓ 0 -SSC
[Yang et al., 2018].
Throughout all the experiments we find that the best clustering accuracy is achieved whenever λ is chosen by 0 .
5 < λ < 0 .
95 , justifying our theoretical finding claimed in Remark 3.5 and (14) in Theorem 3.3.
For all the methods that involve random projection, we conduct the experiments for 30 times and report the average performance.
Note that [Table] the cluster accuracy of SSC-OMP on the extended Yale-B data set is reported according to You et al.
We randomly sample 1000 images from each class of the MNIST data set so as to collect a total number of 10000 images on which clustering is performed, and the average performance of 10 random sampling is reported for this data set.
The time complexity of noisy ℓ 0 -SSC and the two variants of Noisy-DR- ℓ 0 -SSC are analyzed in Section 3.
The actual running time of both algorithms confirms such time complexity, and we observe that Noisy-DR- ℓ 0 -SSC-LR is always 8 .times faster than noisy ℓ 0 -SSC with the same number of iterations, and the acceleration is boosted to 9 .
6 times by Noisy-DR- ℓ 0 -SSC-CSP due to sparse random projections.
We further demonstrate the practical implication of our theoretical analysis for noisy ℓ 0 -SSC.
As mentioned in Remark 3.5, a relatively large λ tends to preserve the subspace detection property.
This theoretical finding is consistent with the empirical study shown in this subsection.
We add Gaussian noise of zero mean and different choices of variance σto the extended Yale-B data set.
In Section 5 of the supplementary, Figure 2a to Figure 2f illustrate SDP violation with respect to λ for different noise levels with σ 2 ranging over 10 , 20 , 30 , 40 , 50 , 60 .
The SDP violation is defined in Wang and Xu [2013] which is the percentage of pairs of data points which are mistakenly put in the same subspace by the similarity matrix W , namely the percentage of pairs ( x i , x j ) with nonzero W ij while they are in fact not in the same subspace.
We observe that increasing λ effectively reduces SDP violation for noisy ℓ 0 -SSC, Noisy-DR- ℓ 0 -SSC-LR and Noisy-DR- ℓ 0 -CSP, confirming our theoretical prediction
CONCLUSION
In this paper we prove that noisy ℓ 0 -SSC recovers subspaces from noisy data through ℓ 0 -induced sparsity.
Our results for the first time reveal the theoretical advantage of noisy ℓ 0 -SSC over its ℓ 1 counterpart and other competing subspace clustering methods in terms of much less restrictive condition on the subspace affinity, when the size of data grows exponentially in the subspace dimension.
We then propose Noisy-DR- ℓ 0 -SSC to improve the efficiency of noisy ℓ 0 -SSC, which performs noisy ℓ 0 -SSC on dimensionality reduced data and still provably recovers the underlying subspaces. Experiments evidence the findings of our theoretical results in the robustness of noisy ℓ 0 -SSC against noise as well as the effectiveness of Noisy-DR- ℓ 0 -SSC.