A Simple Approach to Automated Spectral Clustering
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The performance of spectral clustering heavily relies on the quality of afﬁnitymatrix.
A variety of afﬁnity-matrix-construction (AMC) methods have been proposed but they have hyperparameters to determine beforehand, which requiresstrong experience and lead to difﬁculty in real applications especially when theinter-cluster similarity is high and/or the dataset is large.
In addition, we oftenneed to choose different AMC methods for different datasets, which still dependson experience.
To solve these two challenging problems, in this paper, we presenta simple yet effective method for automated spectral clustering.
First, we proposeto ﬁnd the most reliable afﬁnity matrix via grid search or Bayesian optimizationamong a set of candidates given by different AMC methods with different hyperparameters, where the reliability is quantiﬁed by the relative-eigen-gap of graph
Laplacian introduced in this paper.
Second, we propose a fast and accurate AMCmethod based on least squares representation and thresholding and prove its effectiveness theoretically.
Finally, we provide a large-scale extension for the automated spectral clustering, of which the time complexity is linear with the numberof data points.
Extensive experiments of natural image clustering show that ourmethod is more versatile, accurate, and efﬁcient than baseline methods.
Introduction
Clustering is an important approach to data mining and knowledge discovery.
Particularly, spectralclustering [Weiss, 1999; Shi and Malik, 2000; Ng et al. , 2002; Von Luxburg, 2007] has superiorperformance than k-means clustering [Steinhaus and others, 1956], hierarchical clustering [Johnson,1967], DBSCAN [Ester et al. , 1996] and mixtures of probabilistic principal component analyzers[Tipping and Bishop, 1999] in many applications.
Roughly speaking, spectral clustering consistsof two steps: 1) construct an afﬁnity matrix in which each element denotes the similarity betweentwo data points; 2) perform normalized cut [Shi and Malik, 2000] on the graph corresponding to theafﬁnity matrix.
K-nearest neighbors (K-NN) and Gaussian kernel k ( x , y ) = exp( −(cid:107) x
− y (cid:107) 2 / (2 ς 2 ))are two popular methods to construct afﬁnity matrices, where k and ς are hyperparameters.
As the performance of spectral clustering heavily relies on the quality of afﬁnity matrix, in recentyears, a variety of methods have been proposed to construct or learn afﬁnity matrices for spectralclustering.
Many of them are in the framework of self-expressive [Roweis and Saul, 2000; Elhamifarand Vidal, 2013] model, i.e., minimize C 12 (cid:107)
X − XC (cid:107) 2 F + λ R ( C ) .
Here the columns of X ∈
R m × n are the data points drawn from a union of subspaces.
C ∈ R n × n is a coefﬁcient matrix.
R ( C ) denotes a regularization operator on C .
λ is a hyperparameter to be determined in advance.
Elhamifar and Vidal [2013] proposed to use R ( C )
= (cid:107) C (cid:107) 1 := (cid:80) ni
=1 (cid:80) nj =1 | c ij | under a constraintdiag ( C ) = 0 .
In [Elhamifar and Vidal, 2013], the afﬁnity matrix for spectral clustering is given by
A = | C | + | C | (cid:62) .
The method is called Sparse Subspace Clustering (SSC).
Some theoretical resultsof SSC can be found in [Wang and Xu, 2013; Soltanolkotabi et al. , 2014].
Following the self-expressive framework, Liu et al.
[2013] let R ( C )
=
(cid:107) C (cid:107) ∗ (nuclear norm of C )and proposed a Low-Rank Representation (LRR) method for subspace clustering.
Lu et al.
[2012]and Pan Ji et al.
[2014] used the least squares representation (LSR) model for subspace clustering.
A few variants of LRR and SSC can be found in [Patel and Vidal, 2014; Li and Vidal, 2015; Patel etal. , 2015; Shen and Li, 2016; Li and Vidal, 2016;
Lu et al. , 2018].
Recently, deep learning methodswere also used to learn afﬁnity matrices for spectral clustering [Ji et al. , 2017; Zhang et al. , 2019b,a]and have achieved state-of-the-art performance on many benchmark datasets.
One common limitation of these spectral or subspace clustering methods is that they have at leastone hyperparameter to determine.
In the codes of SSC 1 and its variants provided by their authors,there is usually one more thresholding parameter for afﬁnity matrix, which affects the clusteringaccuracy a lot.
In the deep learning based methods, we need to determine the network structures andregularization parameters, which is much more difﬁcult.
Since clustering is an unsupervised learningproblem, the hyperparameters cannot be tuned by cross validation that is widely used in supervisedlearning.
Thus we have to tune the hyperparameters in spectral clustering by experience, whichis difﬁcult when the dataset is quite different from those in our experience and/or the inter-classsimilarity is high compared to the intra-cluster similarity.
Note that SSC, LRR, and their kernel ordeep learning extensions have quadratic or even cubic time complexity (per iteration), which furtherincreases the difﬁculty of hyperparameter selection in clustering large datasets, though there havebeen a few work improving the computational efﬁciency [Peng et al. , 2013; Cai and Chen, 2014;
Wang et al. , 2014; Peng et al. , 2015; You et al. , 2016a,b; Li and Zhao, 2017; You et al. , 2018;
Matsushima and Brbic, 2019; Li et al. , 2020; Chen et al. , 2020; Kang et al. , 2021].
On the otherhand, different datasets often require different AMC methods, which is hard to tackle by experience.
This paper aims at model and hyperparameter selection for spectral clustering and wants to improvethe convenience, accuracy, and efﬁciency of spectral clustering.
Our contributions are as follows.
• We propose a relative-eigen-gap based automated spectral clustering (AutoSC) method.
It ﬁnds the Laplacian matrix with largest relative-eigen-gap among a set of candidatesconstructed by different models with different hyperparameters.• We also implement the AutoSC method via Bayesian optimization.
The method can selectthe possibly best model and optimize the hyperparameters automatically.
Note that any
AMC methods (e.g. SSC) can be included into the framework of AutoSC.
• To improve the accuracy and efﬁciency of AutoSC, we propose a new AMC method based
on least squares representation and thresholding and prove its effectiveness theoretically.
• We provide an extension for AutoSC to cluster large-scale datasets.
Experiments on seven benchmark image datasets demonstrate the effectiveness of our method.
Particularly, our method is able to outperform state-of-the-art methods of large-scale clustering.
Related work
Exploiting eigenvalue information for clustering As the number of zero eigenvalues of a Laplacian matrix is equal to the number of connected components of the graph [Von Luxburg, 2007], afew researchers took advantage of eigenvalue information in spectral clustering [Meila et al. , 2005;
Meila and Shortreed, 2006; Ji et al. , 2015; Hu et al. , 2017; Lu et al. , 2018].
For instance, Ji et al.[2015] utilized eigen-gap to determine the rank of the Shape Interaction Matrix.
But the method requires to determine another hyperparameter γ beforehand and needs to perform spectral clusteringfor multiple times.
The methods of [Meila et al. , 2005; Hu et al. , 2017; Lu et al. , 2018] are based oniterative optimization (need to perform eigenvalue decomposition at every iteration) and hence arenot effective in handling large-scale datasets.
In addition, the BDR method of [Lu et al. , 2018] hastwo hyperparameters ( λ , γ ) to determine by experience, although it outperformed SSC and LRR onsome datasets.
A comparison is shown in Figure 1.
Automated machine
learning Automated model and hyperparameter selection for supervisedlearning has been extensively studied Hutter et al.
In contrast, the study for unsupervised learning is very limited.
The reason is that in unsupervised learning there is no ground truth or reliable metric to evaluate the performance of algorithms.
Concurrently to our work, Poulakis [2020]also attempted to do automated clustering.
Speciﬁcally, Poulakis [2020] proposed to use metalearning to select clustering algorithm and use a heuristic combination of some clustering validitymetrics such as Silhouette coefﬁcient Liu et al. [2010] and S Dbw [Halkidi and Vazirgiannis, 2001]as an objective to maximize via grid search or Bayesian optimization [Jones et al. , 1998].
One problem is that these metrics are mainly based on Euclidean distance or densities and hence may notbe suitable to evaluate the clustering performance of non-distance or non-density based clusteringalgorithms.
Another one is that there is no uniﬁed metric to compare different clustering algorithms.
Automated Spectral Clustering (AutoSC)3.1 Preliminary Knowledge
Let A ∈ R n × n be an afﬁnity matrix constructed from a given a data matrix X ∈ R m × n .
Thecorresponding graph is denoted by G =
( V, E ) , where V = { v 1 , . .
. , v n } is the vertex set and E ={ e 1 , . .
, e l } is the edge set.
The degree matrix of a graph G is deﬁned as D = diag
( A 1 ) , where1 = [1 , . . .
, 1] (cid:62) .
Our goal is to partition the vertices into k disjoint nonempty subsets C 1 , . . .
, C k .
Let C
= { C 1 , . .
, C k } .
It is expected to ﬁnd a partition C that minimizes the following metric:
Deﬁnition 3.1 (MNCut) .
The multiway normalized cut (MNCut)
[Meila, 2001] is deﬁned as
MNCut ( C
[Equation]
= k (cid:88) i = (cid:88)
j (cid:54)
=
i
Cut (
[Equation]
C j
[Equation]
V
[Equation]
where Cut ( C i , C j ) =
(cid:80) u
∈ C
i (cid:80) v ∈ C
j A uv and V ol ( C i ) denotes the sum of vertex degrees of C i .
The normalized graph Laplacian matrix is deﬁned as
[Equation]
where I is an identity matrix.
The normalized graph Laplacian is often more effective than unnormalized one in spectral clustering (some theoretical justiﬁcation was given by Von Luxburg
Von Luxburg [2007]).
Let σ i ( L ) be the i -th smallest eigenvalue of L .
The following claim showsthe connection between MNCut ( C ) and L .
Claim 3.2.
The sum of the k smallest singular values of L quantiﬁes the potential connectivityamong C 1 , . . .
, C k : MNCut ( C ) ≥ (cid:80) ki
=1 σ i ( L ) .
The claim can be easily proved by using Lemma 4 of Meila [2001].
We defer all proof ofthis paper to the appendices.
Because the multiplicity k of the eigenvalue 0 of L equals to thenumber of connected components in G Von Luxburg [2007], we expect to construct an afﬁnity matrix A from X such that L has k zero eigenvalues.
Thus the optimal partition means
MNCut ( C ) =
(cid:80) ki
=1 σ i ( L ) = 0 .
3.2 Relative Eigen-Gap Guided Search
In practice, we may construct an A such that (cid:80) ki =1 σ i ( L ) is as small as possible because guaranteeing zero eigenvalues is difﬁcult.
But this is not enough because L may have k + 1 or more verysmall or even zero eigenvalues.
The second smallest eigenvalue of the Laplacian matrix of a graph
G is called the algebraic connectivity of G (denoted by ac ( G ) ) Fiedler [1973].
We have ac ( G ) = 0if and only if G is not connected.
When G has k disjointed components, there are k algebraicconnectivities, denoted by ac ( C 1 ) , . .
, ac ( C k ) .
Based on this, we have
Claim 3.3.
The k + 1 th smallest eigenvalue of L quantiﬁes the least potential connectivity of partitions C 1 , . . .
, C k of C :
[Equation]
In other words, σ k +1 ( L ) measures the difﬁculty in segmenting each of C i into two subsets.
Hence,when σ k +1 ( L ) is large, the partitions C 1 , . .
, C k are stable.
Based on Claim 3.2 and Claim 3.3, we may construct an A that has small (cid:80) ki =1 σ i ( L ) and large σ k +1 ( L ) simultaneously, by solving [Equation] subject to L =
I
[Equation]
where f θ : R m × n → R n × n is a function with parameter θ , e.g. A =
[exp( −(cid:107) x
i − x j (cid:107) 2 / (2 ς 2 ))] − I .
It is difﬁcult to solve (4) because of the composition of f θ , symmetric normalized Laplacian, andeigenvalue decomposition.
On the other hand, in (4), we have to choose f in advance, which requiresdomain expert or strong experience because different dataset usually needs different f .
Note that different f can result in very different distribution of eigenvalues and the small eigenvaluesare sensitive to f , θ , and noise.
Hence the objective in (4) is not effective to compared different fand θ .
In this paper, we deﬁne a new metric relative-eigen-gap as follows
[Equation]
where ε is a small constant (e.g. 10 − 6 ) to avoid zero denominator.
reg ( L ) is not sensitive to thescale of the small eigenvalues.
Therefore, instead of (4), we propose to solve
[Equation]
where F is a set of pre-deﬁned functions and Θ is a set of hyperparameters.
In fact, (6) is equivalentto choosing one A (or L ) from a set of candidates constructed by different f with different θ , ofwhich the relative-eigen-gap is largest.
The best θ can be found using grid search (or even randomsearch).
For convenience, we call the method AutoSC-GD.
Table 1 shows a few examples of f andits parameters.
One may use a weighted sum of afﬁnity matrices given by different f , like Huang etal.
[2012], which however will introduce more hyperparameters.
[Table]
The following theorem 2 shows the connection between reg ( L ) and the stability of the clustering C .
Theorem 3.4.
Let C and C (cid:48) be two partitions of the vertices of G , where |C| =
|C (cid:48) | =
k .
Deﬁne the distance between C and C (cid:48) as dist ( C , C (cid:48) )
= 1 − 1 k (cid:80)
[Table]
Suppose ηkε ≥ (cid:80) ki =
σ i
( L ) ≥ kε and reg ( L ) > ( k − 1)
η/ 2 .
Let δ = max (cid:0) MNCut ( C ) −(cid:80)
k
i =1 σ i ( L ) , MNCut ( C (cid:48) ) − (cid:80) [-ki =
-]k
i =1 σ i ( L ) (cid:1) .
Thendist ( C , C (cid:48) [Equation] <
[Equation]
It indicates that when reg ( L ) is large and δ is small, the partitions C and C (cid:48) are close to each other.
Thus, the clustering has high stability.
When (cid:80) ki =
σ i ( L ) =
kε , we have dist ( C , C (cid:48) )
< 6 δ σ k +1 ( L ) − kε ,which means the larger σ k +
( L ) the more stable clustering.
Compare reg ( L ) with [Meila and Shortreed, 2006]
It is worth noting that Meila and Shortreed[2006] proposed to minimize J ( L ) :
= 1 k (cid:80) ki
=1 σ i ( L )+ α ( σ k
( L ) − σ k +1 ( L )) 2 to ﬁnd a good afﬁnity matrix for spectral clustering.
Although reg ( L ) and J ( L ) seem similar, they are essentiallydifferent.
As mentioned before, different AMC methods may lead to different scales for the smalleigenvalues, which makes it difﬁcult to compare different AMC methods using J ( L ) .
In addition,
J ( L ) has a hyperparameter α to determine beforehand, which violates our goal of searching modelsand hyperparameters.
A comparative study ( α ≤ 0 ) is in Section 4.1.
3.3 AutoSC via Bayesian Optimization
Bayesian optimization (BO) Jones et al.
[1998] has become a promising tool for hyperparameter optimization of supervised machine learning algorithms [Snoek et al. , 2012; Klein et al. , 2017].
Givena black-box function g : X → R , BO aims to ﬁnd an x ∗ ∈
X that globally minimizes g and usuallyhas three steps.
The ﬁrst step is ﬁnding the most promising point x t +1 ∈ argmax x a p ( g ) ( x ) by numerical optimization, where a p ( g ) :
X → R is an acquisition function (e.g. Expected Improvement)relying on an prior p ( g ) (e.g. Gaussian processes [Williams and Rasmussen, 2006]).
The secondstep is evaluating the expensive and possibly noisy function y t +1 ∼ g
( x ) + N (0 , σ 2 ) and add thenew sample ( x t +1 , y t +1 ) to observation set D t
= { ( x t +1 , y t +1 ) , . . .
, ( x t , y t ) } .
The last step isupdating p ( g ) and a p ( g ) using D t +1 .
As an alternative to the grid search for (6), we can maximize reg ( L ) via BO.
Suppose we have a set of different AMC models, i.e., F = { f 1 , f 2 , . . .
, f M } .
For i = 1 , 2 , . . .
, M , let [Equation]
where θ ( i ) denotes the hyperparameters in f i and X denotes the dataset.
Then we use BO to ﬁnd
[Equation]
(8) where S ( i ) denotes the set of constraints.
Now we get the best model with its best hyperparametersf
[Equation]
( θ ( (cid:63) ) ∗ | X ) , where (cid:63) = argmin
[Equation]
For convenience, we denote the method by AutoSC-BO.
More details about the implementation of
AutoSC-BO are in Appendix D. Note that F can include any AMC methods such as those in Table1 and even DSC
[Ji et al. , 2017]
(see Appendix E.5).
3.4 Discussion on AMC Methods for AutoSC and LSR with Thresholding
In AutoSC, the size of searching space is |F| × (cid:81)
j | θ j | , which should be large enough to includeeffective models and their hyperparameters.
A large number of works have shown that the selfexpressive models [Elhamifar and Vidal, 2013; Liu et al. , 2013; Lu et al. , 2018; Ji et al. , 2017] oftenoutperform other AMC models such as Gaussian kernel.
However, the self-expressive model based
AMC methods often require iterative optimization and has at least quadratic time complexity periteration, which leads to huge time cost in AutoSC.
Although LSR [Lu et al. , 2012] has closed-formsolution, the clustering accuracy is not satisfactory
[Lu et al. , 2018].
In this work, we will showthat LSR with a simple post-processing can be a good AMC method and is able to outperform SSC,
LRR, and BDR [Lu et al. , 2018].
Speciﬁcally, the LSR model is given as
[Equation]
of which the closed-form solution is C
= ( X (cid:62) X + λ I ) − 1 X (cid:62) X .
Let diag ( C ) = 0 and C ← | C | ,the afﬁnity matrix can be constructed as A
=
( C + C (cid:62) ) / 2 .
One problem is that the off-diagonalelements of A are dense (leading to a connected graph), which can result in low clustering accuracy.
Therefore, we propose to truncate C by keeping only the largest τ elements of each column of C .
Nevertheless, it is not easy to determine τ beforehand.
When τ is too small, the correspondinggraph will have k + 1 or more connected components. When τ is too large, the corresponding graphwill have k − 1 or less connected components.
However, τ can be automatically determined by our AutoSC-GD and AutoSC-BO.
In the case that the data have some low-dimensional nonlinear structures, the similarity betweenpair-wise columns of X cannot be well recognized by the linear regression (10).
Therefore, we alsoconsider the following nonlinear regression model
[Equation]
where φ denotes a nonlinear feature map performed on each column of the matrix, i.e. φ ( X ) =[
φ ( x 1 ) , . .
, φ ( x n )] .
In (11), letting φ be some feature map induced by a kernel function k ( · , · ) (e.g.polynomial kernel k ( x i , x j ) = ( x (cid:62) i x j + b ) q and Gaussian kernel), we get the kernel LSR (KLSR):
[Equation]
[Equation]
where K = φ ( X ) (cid:62) φ ( X ) and [ K ] ij = k
( x i , x j ) .
The closed-form solution is C
=
( K + λ I ) − 1 K .
The post-processing is the same as that for the solution of LSR.
We introduce the following property (a necessary condition of successful subspace clustering)
whichis similar to the one used in [Wang and Xu, 2013;
Soltanolkotabi et al. , 2014].
Deﬁnition 3.5 (Subspace Detection Property) .
A symmetric afﬁnity matrix A obtained from X hassubspace detection property if for all i , the nonzero elements of a i correspond to the columns of Xin the same subspace as x i .
For convenience, let π ( i ) be the index of the subspace x i belongs to and C j be the index set of thecolumns of X in subspace j .
We consider the following deterministic model.
Deﬁnition 3.6 (Deterministic Model) .
The columns of X ∈ R m × n are drawn from a union of kdifferent subspaces and are further corrupted by noise, where dim ( S 1 ∪ · · · ∪ S k ) =
d < m ≤ n .
Let X = U Σ V (cid:62) be the SVD of X , where Σ = diag ( σ 1 , . . .
, σ n ) and σ 1 ≥ σ 2 ≥ · · · σ n .
Letγ =
σ d +1 /σ
d .
Denote v i
= ( v i 1 , . . .
, v in ) the i -th row of V and let ¯ v i =
( v i 1 , . . .
, v id ) .
Supposethe following conditions hold 3 : 1) for every i ∈ [ n ] , the ¯ τ -th largest element of {| ¯ v (cid:62) i ¯ v j | : j ∈ C π ( i ) }is greater than α ; 2) max i ∈ [ n ] max j ∈ [ n ] \ C π ( i ) | ¯ v (cid:62) i ¯ v j | ≤ β ; 3) max i,j,l | v il v jl | ≤ µ .
Then the following theorem veriﬁes the effectiveness of (10) followed by the truncation (thresholding) operation in subspace detection.
Theorem 3.7.
Suppose X is given by Deﬁnition 3.6 and C is given by (10) with
[Equation]
where ρ = 2 µmγ 2 − ∆(1 + γ 2 ) and ∆ = α − β .
Then the C truncated by τ ≤ ¯ τ has the subspacedetection property.
In Theorem 3.7, the width of the range of λ is w =
[Equation]
We see that alarger σ d , ∆ , or smaller γ , d leads to a wider range of λ , which corresponds to a simper clusteringproblem.
When ρ 2 ≤ 4(2 µd − ∆)(2 µm − 2 µd − ∆) , λ does not exist.
Theorem 3.7 can be extendedto the kernel case (12) without the restriction of d < m even when the columns of X are drawn froma union of nonlinear low-dimensional manifolds.
See Deﬁnition C.1, Deﬁnition C.2, and Theorem
C.3 in Appendix C of the supplementary material.
Based on Theorem 3.7 and Theorem C.3, thefollow proposition indicates that AutoSC can cluster the data correctly.
Proposition 3.8.
Suppose the afﬁnity matrix A given by AutoSC has the subspace or manifolddetection property (deﬁned in Appendix C of the supplementary material) and reg ( L )
=
σ k +1 (cid:15) > 0 .
Then each component of G consists of all columns of X in the same subspace or manifold.
Now we see that LSR and KLSR with thresholding can provide effective self-expressive afﬁnitymatrices for AutoSC without performing iterative optimization. On the other hand, the relativeeigen-gap is able to compare LSR with KLSR, compare different kernels, and evaluate λ , τ , andkernel parameters. Note that if we use SSC and KSSC instead of LSR and KLSR, AutoSC will bevery time consuming. If we use LSR and KLSR without thresholding, AutoSC may not providehigh clustering accuracy.
We hope that AutoSC is not only automatic but also accurate and efﬁcient. 3.5 AutoSC+NSE for Large-Scale Data Since the time and space complexity of AutoSC are quadratic with n , it cannot be directly appliedto large-scale datasets. To solve the problem, we propose to perform Algorithm 1 on a set of slandmarks of the data (denoted by ˆ X ) to get a ˆ Z .
The landmarks can be generated by k-means orrandomly.
Then we regard ˆ Z as a feature matrix and learn a map g : R m → R k from ˆ X to ˆ Z . According to the universal approximation theorem [Sonoda and Murata, 2017] of neural networks,we approximate g by a two-layer neural network and solveminimize [Equation] where W 1 ∈ R d × m , W 2 ∈ R k × d , b 1 ∈ R d , and b 2 ∈ R k .
Since A is sparse, k is often less thanm , and a neural network is used, we call (14) Neural Sparse
Embedding (NSE).
We use mini-batch
Adam [Kingma and Ba, 2014] to solve NSE.
It is worth noting that NSE is different from [Li et al. ,2020].
In [Li et al. , 2020], the regression is for an afﬁnity matrix, which led to high computationalcost.
The network learned from (14) is applied to X to extract a k -dimensional feature matrix Z :
[Equation]
Finally, we perform k-means on Z to get the clusters.
The procedures are summarized into the
Algorithm 2 of the supplementary material.
Note that the time complexity of AutoSC+NSE is
O ( dmn + ˜ ds 2 ) , where ˜ d depends on the speciﬁc AMC method.
When s (cid:28) n , the time complexityof AutoSC+NSE is linear with the number of data points n .
Some theoretical result for NSE can befound in Section C.2 of the supplementary material (Proposition C.4).
Experiments
We test our AutoSC on Extended Yale B Face [Kuang-Chih et al. , 2005], ORL Face [Samaria and
Harter, 1994], COIL20 [Nene et al. , 1996], AR Face [Mart´ınez and Kak, 2001], MNIST [LeCunet al. , 1998], Fashion-MNIST [Xiao et al. , 2017], GTSRB [Stallkamp et al. , 2012], subsets andextracted features of MNIST and Fashion-MNIST.
The descriptions for the datasets are in Section
E.1 of the supplementary material.
4.1 Intuitive validation of AutoSC
Performance of Relative-Eigen-Gap First, we use LSR and KLSR to show the effectiveness ofthe proposed reg ( L ) .
Figure 1(i) shows an intuitive example of the performance of LSR and KLSRin clustering a subset of the Extended Yale B database, where for (12) we use Gaussian kernel withς = 1 n 2 (cid:80) ij (cid:107) x
i − x j (cid:107) .
We see that: 1) in LSR and KLSR, for a ﬁxed λ (or τ ), the τ (or λ ) withlarger reg ( L ) provides higher clustering accuracy; 2) for a ﬁxed λ and a ﬁxed τ , if LSR has a largerreg ( L ) , its clustering accuracy is higher than that of KLSR, and vice versa.
We conclude that alarger reg ( L ) indeed leads to a higher clustering accuracy, which is consistent with our theoreticalanalysis in Section 3.2.
Now we show the superiority of LSR and KLSR compared to a few important AMC methods.
Figure1(ii) shows the clustering accuracy of SSC [Elhamifar and Vidal, 2013], LRR [Liu et al. , 2013], and BDR-B [Lu et al. , 2018] with different hyperparameters and our method Auto-GD with LSR and
KLSR (detailed by Algorithm 1 in the supplementary material) on the Extended Yale Face B subset.
SSC and BDR-B are sensitive to the value of λ , especially for the relatively difﬁcult task, say Figure1(ii-b).
LRR is not sensitive to the value of λ but its accuracy is low.
LSR and KLSR are moreaccurate and efﬁcient than other methods.
The performance of AutoSC-BO
We show the performance of AutoSC-BO with many AMCmethods such as SSC [Elhamifar and Vidal, 2013] and LSR.
Taking the KLSR model (12) withpolynomial kernel as an example, the parameters are θ =
( λ, b, q, τ ) (cid:62) and the constraints are givenby S
= { λ ∈ R : λ min ≤ λ ≤ λ max ; b ∈ R : b min ≤ b ≤ b max ; q ∈ Z + : q min ≤ q ≤ q max ; τ ∈ 276 Z + : τ min ≤ τ ≤ τ max } .
More details are in Appendix E.5 of the supplement.
Shown in Table 2,larger reg corresponds to higher clustering accuracy and KLSR with polynomial kernel (the optimalq is 1) performs best.
Figure 2 shows the performance of KSSC [Patel and Vidal, 2014] and KLSRin each iteration of AutoSC-BO.
[Table]
Relative-Eigen-Gap versus Eigen-Gap We compare the proposed relative-eigen-gap (reg ( L ) )with eigen-gap (denoted by eg ( L ) ), and the regularizer J ( L , α ) proposed by [Meila and Shortreed,2006] with different α . Note that in AutoSC, we need to minimize J ( L , α ) instead.
The clusteringresults of AutoSC-GD are reported in Table 3 (details about the datasets are in Section 4).
We seethat our reg ( L ) outperforms eg ( L ) and J ( L , α ) in all cases except AR.
[Table]
4.2 Comparative studies of AutoSC and baselines
First we compare AutoSC with SSC, LRR [Liu et al. , 2013], LSR [Lu et al. , 2012],
EDSC Pan Ji etal.
[2014], KSSC, SSC-OMP
[You et al. , 2016b], BDR-Z Lu et al.
[Table]
and BDR-B [Lu et al. ,2018] on four smaller datasets.
The clustering accuracy and time cost on four (more the results arein the supplementary material) are reported in Table 4.
AutoSC-GD and AutoSC-BO outperformedother methods signiﬁcantly.
SSC-OMP and AutoSC-GD are more efﬁcient than other methods.
The time cost of AutoSC-BO is much higher than that of AutoSC-GD because we optimize allhyperparameters including λ , τ , and kernel parameters (e.g. b, q ).
[Table]
We compare our method with LSR, LSC-K [Chen and Cai, 2011], SSC-OMP
[You et al. , 2016b],and S 5 C [Matsushima and Brbic, 2019], and S 3 COMP-C
[Chen et al. , 2020] on the larger datasets.
The parameter settings and more results are in the supplement.
Table 5 shows the clustering accuracyand standard deviation of 10 repeated trials on the raw-pixel data of MNIST and Fashion-MNIST.
Our methods have the highest clustering accuracy in every case.
It is worth mentioning that if NSE isablated, the accuracy of AutoSC-GD on MNIST-10k, -20k, and -30k are 0.9772, 0.9783 and 0.9864respectively, higher than that of AutoSC-GD+NSE, though the time costs increased.
It is worth mentioning that, to the best of our knowledge, the deep clustering method of [Zhang etal. , 2019a] has SOTA performance Yale B (acc=0.98) and ORL (acc=0.89), the method proposed by[Zhang et al. , 2019b] has SOTA performance on Fashion-MNIST (acc=0.72), the method proposedby [Mahon and Lukasiewicz, 2021] has SOTA performance on MNIST (acc=0.99), and our methodhas SOTA performance on GTSRB.
Nevertheless, we focus on automated spectral clustering.
[Table] [Table]
Conclusion
We have proposed an automated spectral clustering method.
Extensive experiments showed the effectiveness and superiority of our methods over baseline methods.
The efﬁciency improvement isfrom the closed-form solutions of the least squares regressions.
The accuracy improvement is fromthe effectiveness of LSR and KLSR with thresholding and the automation of model and hyperparameter selection.
Note that one limitation of this work is that we only considered automated spectralclustering while there are a few other clustering methods not relying on afﬁnity matrix.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁ- [Table]
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] ) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]