Randomized Sketches for Clustering: Fast and Optimal Kernel k -Means
Abstract
Kernel k -means is arguably one of the most common approaches to clustering.
In this paper, we investigate the efﬁciency of kernel k -means combined with randomized sketches in terms of both statistical analysis and computational requirements.
More precisely, we propose a uniﬁed randomized sketches framework to kernel k -means and investigate its excess risk bounds, obtaining the state-of-the-art risk bound with only a fraction of computations.
Indeed, we prove that it sufﬁces to choose the sketch dimension Ω( √ n ) to obtain the same accuracy of exact kernel k -means with greatly reducing the computational costs, for sub-Gaussian sketches, the randomized orthogonal system (ROS) sketches, and Nyström kernel k -means, where n is the number of samples.
To the best of our knowledge, this is the ﬁrst result of this kind for unsupervised learning.
Finally, the numerical experiments on simulated data and real-world datasets validate our theoretical analysis.
Introduction
Kernel learning is an important ﬁeld of machine learning Yin et al.
(2020b,a, 2021, 2022).
Kernel k -means is one of the fundamental approaches in unsupervised learning and has been widely used in numerous applications Zhang & Rudnicky (2002); Dhillon et al.
; Chitta et al.
; Li &
Liu (2021), whose basic idea is to classify similar samples into the same cluster, and there is a large difference between samples in different clusters.
The statistical properties of kernel k -means have been studied for decades, but they may appear to be not sufﬁcient.
Consistency of the empirical minimizer of the clustering risk was shown in Abaya &
Wise (1984); Pollard (1981, 1982).
Rates of convergence and nonasymptotic performance bounds have been considered by Antos (2005); Antos et al.
; Bartlett et al.
; Linder (2000, 2002).
The existing excess risk bounds are mostly dependent upon the dimension of the hypothesis space.
For example, in Bartlett et al.
(1998), the clustering risk upper bound is O ( (cid:112) kd/n ) , where n is the number of samples, k is the number of clusters, and d is the dimension of the hypothesis space.
Note that the hypothesis space of kernel k -means is typically an inﬁnite-dimensional Hilbert space and the upper bound become useless when d is very large.
Subsequently, some researchers deduced dimension-independent upper bounds for kernel k -means Koltchinskii (2006);
Biau et al.
; Maurer & Pontil (2010); Canas et al.
; Levrard et al.
; Fefferman et al.
Calandriello & Rosasco (2018); Liu (2021).
However, the existing excess clustering risk bounds either have a slow convergence rate O ( k/ √ n ) Biau et al.
; Calandriello & Rosasco (2018) or require pretty strong assumptions on the underlying distribution or large approximate dimensions m to get the faster convergence rate.
Speciﬁcally, in Calandriello & Rosasco (2018), if the approximate dimension reaches Ω( √ n ) , the clustering risk upper bound is O ( k/ √ n ) , which is proportional to k .
Based on its method, Liu Yong Liu (2021) further improves the convergence rate to O ( (cid:112) k/n ) , but the corresponding approximate dimension is increased to Ω( √ nk ) .
Meanwhile, in order to reduce the approximate dimension to Ω( √ n ) with the convergence rate unchanged, this paper Liu (2021) requires a stronger assumption of algebraically decreasing eigenvalues of the kernel matrix.
From the perspective of computational requirements, kernel k -means requires manipulating and storing an empirical kernel matrix, which is unfeasible for large-scale problems.
Exploring approximate kernel k -means algorithms to scale to large-scale application scenarios has become a subject of recent works, see for example Nyström approximations Williams & Seeger (2001); Fowlkes et al.
Pourkamali-Anaraki et al.
; Calandriello & Rosasco (2018); Wang et al.
; Liu (2021), randomized sketches Biau et al.
; Wang et al.
(2019), random features Rahimi & Recht (2008); Chitta et al.
; Pham & Pagh (2013); Atarashi et al.
, incremental clustering
Can (1993);
Bradley et al.
(2000), and reference therein.
This paper focuses on the excess risk bound and computational requirements for kernel k -means.
Although there are many studies on the approximate kernel k -means, these approximate works pay little attention to the excess risk of clusters with the exception of Biau et al.
; Calandriello & Rosasco (2018); Liu (2021).
For example, the works in Wang et al.
(2019) establish the 1 + ε relative-error bound for randomized sketches kernel k -means instead of excess risk bound.
Therefore, in this paper, we mainly introduce the most related approximate kernel k -means with excess risk guarantees.
In Biau et al.
(2008), they employ the randomized sketches method to project the data in Hilbert space so as to approximate kernel k -means.
However, the data in Hilbert space are implicit and inﬁnite-dimensional, and its sketch matrix is dense and unstructured.
In Calandriello & Rosasco (2018), the excess risk upper bound is
O ( k/ √ n ) when the approximate dimension reaches Ω( √ n ) .
The upper bound of clustering risk in
Biau et al.
and Calandriello & Rosasco (2018) does not reach the optimal O ( (cid:112) k/n ) Bartlett et al.
In Liu (2021), the approximate Nyström kernel k -means obtains the risk upper bound
O ( (cid:112) k/n ) with the approximate dimension Ω( √ nk ) .
Although this paper Liu (2021) further reduces the approximate dimension to Ω( √ n ) by introducing a stronger assumption, this is not universal.
In addition, the computational requirements in Biau et al.
; Calandriello & Rosasco (2018); Liu (2021) are still high.
Motivated by these issues, in this paper, we focus on improving the statistical analysis and computational approximations of kernel k -means.
We propose a randomized sketches framework to kernel k -means and construct three novel and speciﬁc examples: sub-Gaussian sketches, the randomized orthogonal system (ROS) sketches, and Nyström kernel k -means.
Theoretical analysis shows that the proposed three randomized sketches methods obtain the optimal excess clustering risk upper bound O ( (cid:112) k/n ) with the sketch dimension (i.e. approximate dimension) of Ω( √ n ) (see Theorem 2).
To the best of our knowledge, this is the ﬁrst optimal excess risk bound with the least approximate dimension and no strong assumptions for general approximate kernel k -means.
From a computational point of view, the proposed methods lead to massive improvements reducing the time complexity from O ( n 2 kt ) to at least O ( n √ n + n √ nkt ) and the memory complexity from O ( n 2 ) to O ( n √ n ) , where t is the number of iterations.
Moreover, we further derive the similarity bound of approximate solutions in the general case, which can be effectively calculated by k -means++ (see Theorem 3).
Experimental results verify and illustrate our theoretical analysis.
The rest of the paper is organized as follows.
Section 2 is the background of kernel k -means.
Section 3 describes the proposed randomized sketches kernel k -means framework and provides three novel examples.
In section 4, we mainly show excess risk bounds of the proposed randomized sketches kernel k -means and the further theoretical analysis in the general case of k -means++.
Sections 5 and 6 are the experiments and conclusions.
Background 2.1 Notation
Given a sampling distribution µ on an arbitrary input space X and n samples S
= { x i } ni
=1 ∈ X drawn i.i.d.
from µ , we denote with µ n ( S ) =
1 n (cid:80) ni
=
I { x i ∈
X} the empirical distribution, where I ( · ) is the indicator function.
In this paper, we use the feature map ϕ ( · ) :
X → H to map X into a Reproducing Kernel Hilbert Space (RKHS)
H Schölkopf et al.
; Scholkopf & Smola (2018), and assume that H is separable, such that for any x ∈ X , we have Φ x = ϕ ( x ) .
Let κ : X × X
→ R be a mercer kernel.
We denote the inner product of H by (cid:104)· , ·(cid:105) , the associated norm by (cid:107) · (cid:107) , the Cartesian product of H by H k = ⊗ ki =1 H , and with K the kernel matrix, where
K ij = κ ( x i , x j ) = (cid:104) Φ i , Φ j (cid:105) =
Φ Ti Φ j .
This paper assumes that (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X .
2.2 Kernel k -Means
Let C =
[ c 1 , . .
c k ] be a collection of k centroids from H .
We divide the given dataset into k disjoint clusters, each characterized by its centroid c j .
The Voronoi cell associated with a centroid c j is deﬁned as Calandriello & Rosasco (2018)
[Equation]
(1) where [ k ] = 1 , 2 , . . .
, k .
That is, the point Φ i belongs to the j -th cluster if c j is its closest centroid.
Now we formalize the criterion used to measure the clustering quality.
The empirical squared norm criterion is deﬁned as
[Equation]
and the expected squared norm criterion is deﬁned as
[Equation] [Equation]
The empirical risk minimizer (ERM) is deﬁned as
[Equation]
The sub-script n in C n indicates that it minimizes W ( C , µ n ) for n samples in S .
In this paper, we bound the excess clustering risk E ( C n ) of the empirical risk minimizer Calandriello & Rosasco (2018):
[Equation]
where W ∗ ( µ ) :=
inf C ∈H k W ( C , µ ) is the optimal clustering risk.
In the following, we will ignore the subscript S ∼ µ if the input dataset S is clear.
From a computational perspective, one cannot compute C n directly, since the points Φ i in H cannot be explicitly represented.
However, due to the properties of the squared norm criterion and the kernel trick, one can reformulate the objective W ( · , µ n ) of kernel k -means.
Proposition 1 ( Proposition 2 of Calandriello & Rosasco (2018) ) .
Let K nn ∈ R n × n be the empirical kernel matrix, and k i its i -th columns.
Then
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) s
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
This approach constructs an n -dimensional embedding k i for each point i , namely the i -th columns of the kernel matrix K nn , which can be explicitly computed, and perfectly preserves W ( · , µ n ) and its minimizer C n .
However, it requires O ( n 2 ) time and space to construct and store the kernel matrix
K , which is not scalable to large-scale scenarios.
2.3 The Existing Excess Risk Bounds of Kernel k -Means
Here we provide the existing upper bound and lower bound of kernel k -means.
According to Bartlett et al.
(1998), we know that there exists a collection of centroids C l ∈ H k and (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , such that
[Equation]
where d is the dimension of Φ x .
In general, d is very large or even inﬁnite.
Therefore, the lower bound of kernel k -means is Ω (cid:18)(cid:113) kn
(cid:19) .
The following is the upper bound of kernel k -means.
Theorem 1 ( Theorem 1 in Liu (2021) ) .
If (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , then for any δ ∈ (0 , 1) , with probability at least 1 − δ , we have,
[Equation]
Note that, p = O ( u ) means that there exists an constant c such that p ≤ cu .
˜ O ( · ) means to hide the logarithmic terms.
This upper bound matches the theoretical lower bound Ω (cid:18) (cid:113)
kn
[Equation]
, and therefore shows that the ERM C n achieve an excess risk (nearly) optimal in n .
The Proposed Algorithms
Kernel k -means is one of the most popular clustering methods Yin et al.
(2020c).
However, it is nonscalable to large scenarios due to computing the exact embedding k i .
To reduce the computational requirements, we propose novel approximate embeddings by using randomized sketches.
In this section, we propose a uniﬁed randomized sketches kernel k -means.
In addition, three speciﬁc examples of randomized sketches algorithms and the corresponding complexity analysis are provided.
1 Framework of Randomized Sketches Kernel k -Means
We consider an approximation based on reducing the original column k i ∈ R n to an m -dimensional subspace of R n , where m (cid:28) n is the sketch dimension.
More precisely, the proposed approximation is deﬁned via a sketch matrix R ∈ R m × n , such that the m -dimensional subspace is generated by the span of R .
Therefore, the proposed randomized sketches method can be described as:
[Equation]
where K = K nn and Q ∈ R m × n is a sampling matrix.
The rows of Q are composed of m rows sampled uniformly from the n × n identity matrix without replacement.
The matrix S ∈ R m × m is constructed in three ways, which will be introduced in detail in the following section.
Then the uniﬁed randomized sketches kernel k -means can be written as (similarly to Proposition 1):
[Equation]
where ˜ k
i is the column of the approximate kernel matrix ˜ K in Eq.(9) and ¯C n,m =
[¯ c 1 , . .
, ¯ c k ] is the empirical clustering centers associated with the m -dimensional ˜ k 1 , . . . , ˜ k n .
Each ¯ c j is the mean of those ˜ k
i ’s in the Voronoi cell ˜ C j .
Deﬁne the clustering centers by
[Equation]
where ˜ C n,m =
[˜ c 1 , . .
˜ c j , . . .
, ˜ c k ] and I {·} is the indicator function.
I { ˜ k i ∈ ˜ C j } = 1 if ˜ k i ∈ ˜ C
j and
I { ˜ k i ∈ ˜ C j } = 0 otherwise.
The n -dimensional embeddings k i are replaced with the lower m -dimensional enbeddings ˜ k
i .
We can perform any k -means algorithms over { ˜ k 1 , . . . ,
˜ k n } and compute the clustering centers in Eq.(11).
The detail of the proposed randomized sketches kernel k -means is shown in Algorithm 1.
The proposed algorithm is mainly divided into two parts.
The ﬁrst part is from step 1 to step 4, which is mainly to construct the sketch matrix R = SQ and obtain the variant kernel matrix ˜ K = SQK .
The second part is from step 5 to step 6, mainly performing k -means over the columns of ˜ K and obtaining centroids.
In step 1, one samples m data points from S according to the sampling matrix Q .
Then, computing the variant kernel matrix ˆ K ∈ R m × n by m sampling data points and all n data points.
From a mathematical point of view, this step can be expressed as ˆ K = QK .
In step 3, we construct a matrix S , whose speciﬁc expression will be given in Section 3.2.
This paper provides three different examples of S , which brings different effects in the approximate kernel k -mean algorithms.
In step 5, take the columns ˜ k
i of ˜ K
= S
ˆ K generated in step 4 as the processing objects and execute k -means algorithm on them.
Finally, compute the centroids ˜ C n,m in Eq.(11).
3.2 Examples of Randomized Sketches Kernel k -Means
Here, we introduce three examples of randomized sketches kernel k -means, which are constructed by three different matrices S in Eq.(9).
In addition, the detailed complexity analysis of the corresponding three approximate kernel k -means is provided.
Example 1: Sub-Gaussian Sketches
Kernel k -Means
The ﬁrst example of approximate kernel k -means is called sub-Gaussian sketches kernel k -means, whose matrix S ∈ R m × m in Eq.(9) is described by a hash function.
Let σ be a hash function and σ ( i ) ∈ { +1 , − 1 } is 2-wise independent hash function.
The entries S i,j = σ ( i ) / √ m with a probability of 1 √ n and S i,j = 0 with a probability of 1 − 1 √ n .
Complexity analysis: In the terms of time, we ﬁrst sample the data by Q , then generate the variant kernel matrix ˆK .
Therefore, the time cost of computing S ˆK should be O ( nm 2 ) .
However, due to the sparsity of the sub-Gaussian matrix S , we only need to compute the non-zero elements instead of the total elements, which can further reduce the computational requirements of S ˆK from O ( nm 2 ) to
O ( √ nm 2 ) .
In the iteration operation of performing k -means algorithm over the columns ˜
k
i of ˜ K , one needs O ( nmkt ) time.
Combining the above, the total time cost of sub-Gaussian sketches kernel k -means is O ( √ nm 2 + nmkt ) .
In terms of space, due to the operation of sampling, the key of the space cost is changed to ˆ K and ˜ K instead of K .
Therefore, the space complexity of the proposed sub-Gaussian sketches kernel k -means is O ( nm ) .
Example 2: ROS Sketches
Kernel k -Means
The second example of the random sketches kernel k -means is based on the randomized orthogonal system (ROS) sketches.
The corresponding matrix
S ∈ R m × m in Eq.(9) can be deﬁned as below:
[Equation]
(12) where D ∈ R m × m is a random diagonal matrix whose entries are i.i.d.
Rademacher variables.
A ∈ R m × m is an orthogonal matrix with uniformly bounded entries, for example the Hadamard matrix Wallis (1976) and the discrete Fourier transform matrix.
We use the Hadamard matrix in this paper.
The Hadamard matrix is deﬁned recursively as: A m = (cid:20) A m/
A m/ 2 A m/− A m/(cid:21) with
A
[Equation]
(cid:20)
[Equation]
, and A = 1 √ m A m .
Due to the constructed property of the Hadamard matrix A , we can use FFT (Fast Fourier Transform algorithm) to compute the matrix-vector product, such as Au for any u ∈ R m , whose time complexity is O ( m log m ) instead of O ( m 2 ) .
Therefore, in step 4 of Algorithm 1, the computation of S ˆ K can be realized by the fast FFT, which is another way to further reduce the time cost, in addition to the sparsity mentioned above.
Complexity analysis: In terms of time cost, due to the use of the Hadamard matrix, we can compute
S ˆ K by FFT, whose time cost is O ( nm log m ) .
In the iteration operation of k -means algorithm over the columns ˜ k
i , the time cost is O ( nmkt ) .
Therefore, the total time cost of ROS sketches kernel k -means is O ( nm log m + nmkt ) .
In terms of space, the key to the space cost is to store the matrices ˆ K and ˜ K , whose space requirements is O ( nm ) .
Therefore, the space complexity of the proposed
ROS sketches kernel k -means is O ( nm ) .
Example 3:
Nyström Kernel k -Means
The third example of the approximate kernel k -means is
Nyström kernel k -means, whose matrix S ∈ R m × m in Eq.(9) can be deﬁned as: S
=
I , where I is an identify matrix.
That is, we only use the sample matrix Q in Eq.(9).
Therefore, the proposed
Nyström kernel k -means can be converted into:
[Equation]
where ˜Φ i = Φ Tm Φ i , ˜ c
j = Φ Tm c j , Φ m =
[Φ π (1) , . . . , Φ π ( m ) ] , π ( i ) ∈
[1 , n ] , and the dictionary (i.e., subset) { Φ π ( i ) } mi =1 is m points Φ j sampled from { Φ j } nj =1 through the sampling matrix Q .
Note that, the proposed Nyström kernel k -means can also be understood as a variant ROS, based on the identity matrix as an orthonormal matrix and not using the Rademacher randomization.
Complexity analysis: In terms of time cost, the matrix S is a scaled identity matrix so that the computation of step 3 and step 4 in Algorithm 1 is not needed.
Therefore, the time complexity of
Nyström kernel k -means is decided by the iteration operation of k -means algorithm over the columns ˜ k
i , which is O ( nmkt ) .
In terms of space, the key is the matrix ˜ K .
Therefore, the space complexity of the proposed Nyström kernel k -means is O ( nm ) .
In algorithm, the function of Q is to reduce the scale of data.
The function of S is to fuse data features.
In complexity, the proposed randomized sketches can reduce the time and space complexity.
We sample data points according to Q , then generate the variant kernel matrix, instead of generating and processing the kernel matrix directly, which can greatly reduce the time and space complexity.
In addition, our matrices S are structured (in ROS) or sparse (in sub-Gaussian and Nyström), which can speed up kernel k -means by FFT or sparsity.
In theoretical analysis, we obtain the optimal excess risk bound with a small sketch dimension based on the proposed randomized sketches, which can further reduce the time and space complexity.
Overall, the proposed randomized sketches can greatly reduce the time and space complexity with the optimal excess risk bound.
Theoretical Analysis
In this section, we exploit the excess risk bound of the proposed randomized sketches kernel k -means.
Theoretical analysis shows that we can improve the computational requirements of kernel k -means using sub-Gaussian, ROS, and Nyström, while maintaining optimal generalization guarantees.
Theorem 2.
If (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , ε ∈ (0 , 1) , δ ∈ (0 , 1) , and, in either one of the three cases of sub-Gaussian, ROS, and Nyström, the sketch dimension is m = Ω (cid:16) 4log n − 2log
δ ε − log(1+ ε ) (cid:17) , then, with probability at least 1 − δ , we have
[Equation]
Remark 1.
From a statistical point of view, let ε = 1 / √ n , Theorem 2 shows that when the sketch dimension is m = Ω( √ n ) , the proposed randomized sketches (sub-Gaussian, ROS, and Nyström) kernel k -means achieve the same excess risk bound ˜ O
(cid:16)(cid:112) k/n (cid:17) as the exact kernel k -means.
Remark 2.
From a computational point of view, we can construct the √ n -dimension randomized sketches simply, which can greatly reduces the total required space from O ( n 2 ) to O ( n √ n ) and the total required time from O ( n 2 kt ) to O ( n √ n + n √ nkt ) at least, with the optimal excess risk bound.
Remark 3.
In Calandriello & Rosasco (2018), when the approximate dimension m is Ω( √ n ) , the excess risk bound can reach ˜ O ( k/ √ n ) , which is linearly dependent on k and fail to reach the optimal bound.
The corresponding space complexity and time complexity are O ( n √ n ) and O (cid:0) nkt √ n + n 2 (cid:1) , respectively.
Compared to it, our proposed methods obtain the better excess risk bound and reduce the time complexity from O (cid:0) nkt √ n + n 2 (cid:1) to O ( nkt √ n + n √ n ) at least.
Subsequently, Liu Yong
Liu (2021) further improves the excess risk bound of the method in Calandriello & Rosasco (2018) to ˜ O
(cid:16)(cid:112) k/n (cid:17) , but the corresponding approximate dimension m is increased to Ω( √ nk ) .
Meanwhile, its space complexity and time complexity increase to O (cid:16) n √ nk (cid:17) and O (cid:16) nkt √ nk + n 2 k (cid:17) .
Compared to it, the proposed methods reduce the time complexity from O (cid:16) nkt √ nk + n 2 k (cid:17) to
O ( nkt √ n + n √ n ) at least and reduce the space complexity from O (cid:16) n √ nk (cid:17) to O ( n √ n ) while maintaining the optimal excess risk bound ˜ O
(cid:16)(cid:112) k/n (cid:17) and the smaller m = Ω( √ n ) .
To the best of our knowledge, the proposed methods are the ﬁrst time that they are always possible to maintain the optimal excess risk bound ˜ O
(cid:16)(cid:112) k/n (cid:17) in unsupervised non-parametric problem with smaller m = Ω ( √ n ) , while greatly reducing the time and space requirements.
In Table 1, we show the detail space complexity, time complexity, excess risk bounds, and m of the approximate kernel k -means.
4.1 Further Results: k -Means++
We adopt the improved kernel k -means++ sampling Lattanzi & Sohler (2019), which has a local search strategy, for the proposed randomized sketches kernel k -mean.
Here is its theoretical analysis.
Lemma 1 ( Lattanzi & Sohler (2019) ) .
If C + n is obtained by the improved k -means++ algorithm with a local search strategy Lattanzi & Sohler (2019), then E J
[Equation]
, where (cid:36) is a constant and J is the randomness derived from the k -means++ initialization.
Note that, this is a multiplicative error bound on the empirical risk.
Theorem 3.
Let C + n,m be obtained by the improved k -means++ algorithm with a local search strategy
Lattanzi & Sohler (2019).
If (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , ε ∈ (0 , 1) , δ ∈ (0 , 1) , and, in either one of the three cases of sub-Gaussian, ROS, and Nyström, the sketch dimension is m = Ω (cid:16) 4log
[Equation]
, then, with probability at least 1 − δ , we have
[Equation]
where J is the randomness derived from the k -means++ initialization.
Approach
[Table]
NytrömCalandriello &
[Figure]
[Equation]
[Figure]
[Table]
[Figure]
[Table]
NytrömLiu (2021)
[Table]
[Figure]
[Table]
[Figure]
[Table]
Sub-Gaussian Sketches ( This Paper )
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
[Figure]
[Table]
[Figure]
ROS Sketches ( This Paper )
[Figure]
[Table]
Nyström ( This Paper )
[Figure]
[Table]
[Figure]
Theorem 3 shows that, if the optimal clustering risk W ∗ ( µ ) is smaller than ˜ O
( (cid:112) k/n ) , the risk of
W ( C + n,m , µ ) can reach ˜ O
( (cid:112) k/n ) .
Note that ε is small, i.e. ε = 1 / √ n .
Experiments
In this section, we evaluate experimentally our theoretical analysis on both simulated data and real-world data for the proposed methods.
The server is 32 cores (2.40GHz) and 32 GB of RAM.
The compared methods are the exact kernel k -means, Gaussian Biau et al.
Nyström (Liu) Liu (2021), ROS, Sub-Gaussian, and Nyström (Ours).
For the sake of distinguishing, Nyström (Liu) is
Nyström Liu (2021) in this paper.
Each experiment is repeated 5 times.
5.1 Numerical Experiments on Simulated Data
We conduct the experiments to validate our theoretical analysis of the proposed randomized sketches kernel k -means on simulated data.
Now we generate the simulated data.
Let c ∗ i ∈ R 8 , i = [1 , k ] , be the clustering centers, where the values of the dimensions are 1 or − 1 with the probability of 1/2.
The data in i th clustering follows the normal distribution with mean c ∗ i and variance 1.
The number of data in each clustering is the same.
We use the Gaussian kernel κ ( x , x (cid:48) )
=
exp (cid:0) −(cid:107) x − x (cid:48) (cid:107) 2 / 8 (cid:1) .
Generating 10,000 samples for training and 10,000 samples for testing.
The number of training samples in each clustering is 10000 /k .
The accuracy of kernel k -means He & Zhang (2018) on the test set can be written as (cid:80) ¨
ni =1 υ
(ˆ y,map ( y ))
¨ n , where y is the solution returned by the (approximate)
[Table]
kernel k -means using Lloyd’s algorithm Lloyd (1982), ˆ y is the real label, and ¨ n is the number of data in the test set.
If p = q , υ ( p, q ) = 1 , otherwise υ ( p, q ) = 0 .
map ( · ) represents the best mapping to match ˆ y and y .
The higher the accuracy, the better the method.
The test accuracy and training time of the approximate kernel k -means with different m are given in Figure 1 , which can be summarized as follows: (1) There exists a lower bound of the approximate dimensions m = √ n = 100 .
When this lower bound is reached, the accuracy of the proposed methods tends to be stable.
This is consistent with our theoretical analysis in Theorem 2.
The accuracy of the proposed methods keeps the similar accuracy to the exact kernel k -means.
(3) We take the logarithm of the running time (in seconds) in Figure 1.
Our methods (ROS, Sub-Gaussian, Nyström) have obvious advantages over other methods in running time.
This veriﬁes our complexity analysis.
5.2 Numerical Experiments on Real-World Scenarios
In this subsection, we perform the experiments on the 9 real datasets: dna, segment, mushrooms, pendigits, protein, a8a, w7a, connect-4, and covtype, which are from LIBSVM website 2 .
70 percent of the data in each dataset is used for training experiments, and the rest is used for testing.
m
= 150 .
The Gaussian kernel is exp (cid:0) −(cid:107) x − x (cid:48)
(cid:107) 2 /σ 2 (cid:1) , where σ = (cid:113) (cid:80) ij (cid:107) x
i − x j (cid:107) 2 n .
The detail of the datasets and experimental results are shown in Table 2.
From the above results, we can ﬁnd that these methods give a similar accuracy as the exact kernel k -means.
The proposed methods outperform Nytröm (Liu) and Gaussian in time cost, which matches our theoretical analysis.
If the training time exceeds 90 seconds or the memory is insufﬁcient, the experiment will be stopped.
In the large covtype dataset, kernel k -means, Gaussian, and Nytröm (Liu) cannot achieve the experimental results, but our proposed methods can obtain small training time and good accuracy.
Those verify the smaller computational requirements of the proposed methods.
Conclusions
We propose a uniﬁed randomized sketches framework to kernel k -means and provide three speciﬁc examples of sub-Gaussian sketches, the randomized orthogonal system (ROS) sketches, and Nyström kernel k -means.
Theoretical analysis show that the proposed methods obtain the state-of-the-art risk bound and greatly reduce the computational requirements with sketch dimension Ω( √ n ) .
To the best of our knowledge, this is the ﬁrst optimal excess risk bound with the least approximate dimension and no strong assumptions for general approximate kernel k -means.
Moreover, we further derive the similarity optimal bound of approximate solutions in the general case, which can be effectively calculated by k -means++.
The extensive experiments illustrate our theoretical analysis.
Acknowledgments and Disclosure of Funding
We appreciate all the anonymous reviewers, ACs, and PCs for their invaluable and constructive comments.
This work is supported in part by the Special Research Assistant project of CAS (No.E0YY221-2020000702), the National Natural Science Foundation of China (No.62106259,
No.62076234), Beijing Outstanding Young Scientist Program (NO.BJJWZYJH012019100020098), and Beijing Natural Science Foundation (No. 4222029). Thank Intelligent Social Governance Platform and Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” initiative.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide Pseudo code, data, and instructions.
Lucky to be accepted, the code will be provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]