Randomized Sketches for Clustering: Fast and Optimal Kernel k -Means
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Kernel k -means is arguably one of the most common approaches to clustering.
Inthis paper, we investigate the efﬁciency of kernel k -means combined with randomized sketches in terms of both statistical analysis and computational requirements.
More precisely, we propose a uniﬁed randomized sketches framework to kernelk -means and investigate its excess risk bounds, obtaining the state-of-the-art riskbound with only a fraction of computations.
Indeed, we prove that it sufﬁces tochoose the sketch dimension Ω( √ n ) to obtain the same accuracy of exact kernelk-means with greatly reducing the computational costs, for sub-Gaussian sketches,the randomized orthogonal system (ROS) sketches, and Nyström kernel k -means,where n is the number of samples.
To the best of our knowledge, this is the ﬁrstresult of this kind for unsupervised learning.
Finally, the numerical experiments onsimulated data and real-world datasets validate our theoretical analysis.
Introduction
Kernel k -means is one of the fundamental approaches in unsupervised learning and has been widelyused in numerous applications Zhang & Rudnicky (2002); Dhillon et al.
; Chitta et al.
Li & Liu (2021), whose basic idea is to classify similar samples into the same cluster and there is alarge difference between samples in different clusters.
The statistical properties of kernel k -means have been studied for decades, but they may appear to benot sufﬁcient.
Consistency of the empirical minimizer of the clustering risk was shown in Abaya &
Wise (1984); Pollard (1981, 1982).
Rates of convergence and nonasymptotic performance boundshave been considered by Antos (2005); Antos et al.
; Bartlett et al.
; Linder (2000,2002).
The existing excess risk bounds are mostly dependent upon the dimension of the hypothesisspace.
For example, in Bartlett et al.
(1998), the clustering risk upper bound is O ( (cid:112) kd/n ) , wheren is the number of samples, k is the number of clusters, and d is the dimension of the hypothesisspace.
Note that the hypothesis space of kernel k -means is typically an inﬁnite-dimensional Hilbertspace and the upper bound become useless when d is very large.
Subsequently, some researchersdeduced dimension-independent upper bounds for kernel k -means Koltchinskii (2006); Biau et al.(2008); Maurer & Pontil (2010); Canas et al.
; Levrard et al.
; Fefferman et al.
Calandriello & Rosasco (2018); Liu (2021).
However, the existing excess clustering risk boundseither have a slow convergence rate O ( k/ √ n )
Biau et al.
(2008); Calandriello & Rosasco (2018) orrequire pretty strong assumptions on the underlying distribution or large approximate dimensions mto get the faster convergence rate.
Speciﬁcally, in Calandriello & Rosasco (2018), if the approximatedimension reaches Ω( √ n ) , the clustering risk upper bound is O ( k/ √ n ) , which is proportional to k .
Based on its method, Liu Yong Liu (2021) further improves the convergence rate to O ( (cid:112) k/n ) , butthe corresponding approximate dimension is increased to Ω( √ nk ) .
Meanwhile, in order to reduce the approximate dimension to Ω( √ n ) with the convergence rate unchanged, this paper Liu (2021)requires a stronger assumption of algebraically decreasing eigenvalues of the kernel matrix.
From the perspective of computational requirements, kernel k -means requires manipulating and storing an empirical kernel matrix, which is unfeasible for large-scale problems.
Exploring approximatekernel k -means algorithms to scale to large-scale application scenarios has become a subject of recentworks, see for example Nyström approximations Williams & Seeger (2001); Fowlkes et al. Pourkamali-Anaraki et al.
; Calandriello & Rosasco (2018); Wang et al.
; Liu (2021),randomized sketches Biau et al. (2008); Wang et al. (2019), random features Rahimi & Recht (2008);
Chitta et al.
; Pham & Pagh (2013); Atarashi et al.
(2019), incremental clustering Can (1993);
Bradley et al.
(2000), and reference therein.
However, with the exception of Biau et al.
Calandriello & Rosasco (2018); Liu (2021), these approximate works pay little attention to the excessrisk of clusters.
In Biau et al.
(2008), they employ the randomized sketches method to project the datain Hilbert space so as to approximate kernel k -means.
However, the data in Hilbert space are implicitand inﬁnite-dimensional. So this method is not practical.
In Calandriello & Rosasco (2018), theexcess risk upper bound is O ( k/ √ n ) when the approximate dimension reaches Ω( √ n ) .
The upperbound of clustering risk in Biau et al.
and Calandriello & Rosasco (2018) does not reach theoptimal O ( (cid:112) k/n ) Bartlett et al.
In Liu (2021), the approximate Nyström kernel k -meansobtains the risk upper bound O ( (cid:112) k/n ) with the approximate dimension Ω( √ nk ) .
Although thispaper Liu (2021) further reduces the approximate dimension to Ω( √ n ) by introducing a strongerassumption, this is not universal.
In addition, the computational requirements in Biau et al.
Calandriello & Rosasco (2018); Liu (2021) are still high.
Motivated by these issues, in this paper, we focus on improving the statistical analysis and computational approximations of kernel k -means.
We propose a randomized sketches framework to kernelk -means and construct three novel and speciﬁc examples: sub-Gaussian sketches, the randomizedorthogonal system (ROS) sketches, and Nyström kernel k -means.
Theoretical analysis shows thatthe proposed three randomized sketches methods obtain the optimal excess clustering risk upperbound O ( (cid:112) k/n ) with the sketch dimension (i.e. approximate dimension) of Ω( √ n ) (see Theorem2).
To the best of our knowledge, this is the ﬁrst optimal excess risk bound with the least approximatedimension and no strong assumptions for general approximate kernel k -means.
From a computationalpoint of view, the proposed methods lead to massive improvements reducing the time complexityfrom O ( n 2 kt ) to at least O ( n √ n + n √ nkt ) and the memory complexity from O ( n 2 ) to O ( n √ n ) ,where t is the number of iterations.
Moreover, we further derive the similarity bound of approximatesolutions in the general case, which can be effectively calculated by k -means++ (see Theorem 3).
Experimental results verify and illustrate our theoretical analysis.
The rest of the paper is organized as follows.
In section 2, we introduce the background of kernelk -means.
Section 3 describes the proposed randomized sketches kernel k -means framework andprovides three novel examples of randomized sketches kernel k -means.
This section also provides adetailed complexity analysis of each method.
In section 4, we mainly show excess risk bounds of theproposed randomized sketches kernel k -means and the further theoretical analysis in the general caseof k -means++.
Through the detailed comparison with the related works, we know that the proposedmethods maintains the optimal state in terms of time complexity, space complexity, excess risk bound,and approximate dimensions m .
The experimental results on simulated data and real-world datasetsare shown in section 5, which verify our theoretical analysis.
Background2.1 Notation
Given a sampling distribution µ on an arbitrary input space X and n samples S
= { x i } ni =1 ∈ Xdrawn i.i.d.
from µ , we denote with µ n ( S ) =
1 n (cid:80) ni
=
I { x i ∈
X} the empirical distribution, where 82 I ( · ) is the indicator function.
In this paper, we use the feature map ϕ ( · ) :
X → H to map Xinto a Reproducing Kernel Hilbert Space (RKHS)
H Schölkopf et al.
; Scholkopf & Smola(2018), and assume that H is separable, such that for any x ∈ X , we have Φ x = ϕ ( x ) .
Letκ :
X × X → R be a mercer kernel.
We denote the inner product of H by (cid:104)· , ·(cid:105) , the associatednorm by (cid:107) · (cid:107) , the Cartesian product of H by H k = ⊗ ki =1 H , and with K the kernel matrix, where
K ij = κ ( x i , x j ) = (cid:104) Φ i , Φ j (cid:105) =
Φ Ti Φ j .
This paper assumes that (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X .
2.2 Kernel k -Means
Let C =
[ c 1 , . .
c k ] be a collection of k centroids from H .
We divide the given dataset into kdisjoint clusters, each characterized by its centroid c j .
The Voronoi cell associated with a centroid c jis deﬁned as Calandriello & Rosasco (2018)
[Equation]
where [ k ] = 1 , 2 , . .
k .
That is, the point Φ i belongs to the j -th cluster if c j is its closest centroid.
Now we formalize the criterion used to measure the clustering quality.
The empirical squared normcriterion is deﬁned as
[Equation]
and the expected squared norm criterion is deﬁned as
[Equation]
The empirical risk minimizer (ERM) is deﬁned as
[Equation]
The sub-script n in C n indicates that it minimizes W ( C , µ n ) for n samples in S .
In this paper, we bound the excess clustering risk E ( C n ) of the empirical risk minimizer Calandriello& Rosasco (2018):
[Equation]
(5) where W ∗ ( µ ) :=
inf C ∈H k W ( C , µ ) is the optimal clustering risk.
In the following, we will ignorethe subscript S ∼ µ if the input dataset S is clear.
From a computational perspective, one cannot compute C n directly, since the points Φ i in H cannotbe explicitly represented.
However, due to the properties of the squared norm criterion and the kerneltrick, one can reformulate the objective W ( · , µ n ) of kernel k -means.
Proposition 1 ( Proposition 2 of Calandriello & Rosasco (2018) ) .
Let K nn ∈ R n × n be the empirical kernel matrix, and k i its i -th columns.
Then
[Equation]
This approach constructs an n -dimensional embedding k i for each point i , namely the i -th columnsof the kernel matrix K nn , which can be explicitly computed, and perfectly preserves W ( · , µ n ) andits minimizer C n .
However, it requires O ( n 2 ) time and space to construct and store the kernel matrix
K , which is not scalable to large-scale scenarios.
2.3 The Existing Excess Risk Bounds of Kernel k -Means
Here we provide the existing upper bound and lower bound of kernel k -means.
According to Bartlett et al.
(1998), we know that there exists a collection of centroids C l ∈ H k and(cid:107)
Φ x (cid:107) ≤ 1 for any x ∈ X , such that
[Equation]
where d is the dimension of Φ x .
In general, d is very large or even inﬁnite.
Therefore, the lowerbound of kernel k -means is Ω (cid:18)(cid:113) kn
(cid:19) .
The following is the upper bound of kernel k -means.
Theorem 1 ( Theorem 1 in Liu (2021) ) .
If (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , then for any δ ∈ (0 , 1)
, withprobability at least 1 − δ , we have,
[Equation]
Note that, p = O ( u ) means that there exists an constant c such that p ≤ cu .
˜ O ( · ) means to hide thelogarithmic terms.
This upper bound matches the theoretical lower bound Ω (cid:18)(cid:113) kn (cid:19) , and thereforeshows that the ERM C n achieve an excess risk (nearly) optimal in n .
The Proposed Algorithms
Kernel k -means is one of the most popular clustering methods.
However, it is non-scalable to largescenarios due to computing the exact embedding k i .
To reduce the computational requirements, wepropose novel approximate embeddings by using randomized sketches.
In this section, we proposea uniﬁed randomized sketches kernel k -means.
In addition, three speciﬁc examples of randomizedsketches algorithms and the corresponding complexity analysis are provided.
3.1 Framework of Randomized Sketches Kernel k -Means
We consider an approximation based on reducing the original column k i ∈ R n to an m -dimensionalsubspace of R n , where m (cid:28) n is the sketch dimension.
More precisely, the proposed approximationis deﬁned via a sketch matrix R ∈ R m × n , such that the m -dimensional subspace is generated by thespan of R .
Therefore, the proposed randomized sketches method can be described as:
[Equation]
where K = K nn and Q ∈ R m × n is a sampling matrix.
The rows of Q are composed of m rowssampled uniformly from the n × n identity matrix without replacement.
The matrix S ∈ R m × m isconstructed in three ways, which will be introduced in detail in the following section.
Then the uniﬁed randomized sketches kernel k -means can be written as (similarly to Proposition 1):
[Equation]
where ˜ k
i is the column of the approximate kernel matrix ˜ K in Eq.(9) and ¯ C n,m =
[¯ c 1 , . .
, ¯ c k ] is theempirical clustering centers associated with the m -dimensional ˜ k 1 , . . . ,
˜ k n .
Each ¯ c j is the mean ofthose ˜ k
i ’s in the Voronoi cell ˜ C j .
Deﬁne the clustering centers by
[Equation]
where ˜ C n,m =
[˜ c 1 , . .
˜ c j , . . .
, ˜ c k ] and I {·} is the indicator function.
I { ˜ k i ∈ ˜ C j } = 1 if ˜ k i ∈ ˜ C
j and
I { ˜ k i ∈ ˜ C j } = 0 otherwise.
The n -dimensional embeddings k i are replaced with the lower m -dimensional enbeddings ˜ k
i .
We canperform any k -means algorithms over { ˜ k 1 , . . . , ˜ k n } and compute the clustering centers in Eq.(11).
The detail of the proposed randomized sketches kernel k -means is shown in Algorithm 1.
In step 1,one samples m data points from S according to the sampling matrix Q .
Then, computing the variant kernel matrix ˆ K ∈ R m × n by m sampling data points and all n data points.
From a mathematicalpoint of view, this step can be expressed as ˆ K = QK .
In step 3, we construct a matrix S , whosespeciﬁc expression will be given in Section 3.2.
This paper provides three different examples of S , which brings different effects in the approximate kernel k -mean algorithms.
In step 5, take thecolumns ˜ k i of ˜ K
= S
ˆ K generated in step 4 as the processing objects and execute k -means algorithmon them.
Finally, compute the centroids ˜ C n,m in Eq.(11).
3.2 Examples of Randomized Sketches Kernel k -Means
Here, we introduce three examples of randomized sketches kernel k -means, which are constructed bythree different matrices S in Eq.(9).
In addition, the detailed complexity analysis of the correspondingthree approximate kernel k -means is provided.
Example 1: Sub-Gaussian Sketches
Kernel k -Means
The ﬁrst example of approximate kernelk -means is called sub-Gaussian sketches kernel k -means, whose matrix S ∈ R m × m in Eq.(9) isdescribed by a hash function.
Let σ be a hash function and σ ( i ) ∈ { +1 , − 1 } is 2-wise independenthash function.
The entries S i,j = σ ( i ) / √ m with a probability of 1 √ n and S i,j = 0 with a probabilityof 1 − 1 √ n .
Complexity analysis: In the terms of time, we ﬁrst sample the data by Q , then generate the variantkernel matrix ˆK .
Therefore, the time cost of computing S ˆK should be O ( nm 2 ) .
However, due to thesparsity of the sub-Gaussian matrix S , we only need to compute the non-zero elements instead of thetotal elements, which can further reduce the computational requirements of S ˆK from O ( nm 2 ) to
O ( √ nm 2 ) .
In the iteration operation of performing k -means algorithm over the columns ˜
k
i of ˜ K ,one needs O ( nmkt ) time.
Combining the above, the total time cost of sub-Gaussian sketches kernelk -means is O ( √ nm 2 + nmkt ) .
In terms of space, due to the operation of sampling, the key of thespace cost is changed to ˆ K and ˜ K instead of K .
Therefore, the space complexity of the proposedsub-Gaussian sketches kernel k -means is O ( nm ) .
Example 2: ROS Sketches
Kernel k -Means
The second example of the random sketches kernelk -means is based on the randomized orthogonal system (ROS) sketches.
The corresponding matrix
S ∈ R m × m in Eq.(9) can be deﬁned as below:
[Equation] [Equation]
where D ∈ R m × m is a random diagonal matrix whose entries are i.i.d.
Rademacher variables.
A ∈ R m × m is an orthogonal matrix with uniformly bounded entries, for example the Hadamard matrix Wallis (1976) and the discrete Fourier transform matrix.
We use the Hadamard matrix in this paper.
The Hadamard matrix is deﬁned recursively as:
[Equation]
with A 2 = (cid:20) 1 −(cid:21) , and A = 1 √ m A m .
Due to the constructed property of the Hadamard matrix A , we can use FFT (Fast Fourier Transformalgorithm) to compute the matrix-vector product, such as Au for any u ∈ R m , whose time complexityis O ( m log m ) instead of O ( m 2 ) .
Therefore, in step 4 of Algorithm 1, the computation of S
ˆ K canbe realized by the fast FFT, which is another way to further reduce the time cost, in addition to thesparsity mentioned above.
Complexity analysis: In terms of time cost, due to the use of the Hadamard matrix, we can compute
S ˆ K by FFT, whose time cost is O ( nm log m ) .
In the iteration operation of k -means algorithm overthe columns ˜ k
i , the time cost is O ( nmkt ) .
Therefore, the total time cost of ROS sketches kernelk -means is O ( nm log m + nmkt ) .
In terms of space, the key to the space cost is to store the matricesˆ K and ˜ K , whose space requirements is O ( nm ) .
Therefore, the space complexity of the proposed
ROS sketches kernel k -means is O ( nm ) .
Example 3:
Nyström Kernel k -Means
The third example of the approximate kernel k -means is
Nyström kernel k -means, whose matrix S ∈ R m × m in Eq.(9) can be deﬁned as:
S =
I , (13) where I is an identify matrix.
That is, we only use the sample matrix Q in Eq.(9).
Therefore, theproposed Nyström kernel k -means can be converted into:
[Equation]
˜ c j (cid:13)(cid:13)(cid:13) 2 , (14) where ˜Φ i = Φ Tm Φ i , ˜ c
j = Φ Tm c j , Φ m =
[Φ π (1) , . . . , Φ π ( m ) ] , π ( i ) ∈
[1 , n ] , and the dictionary (i.e.,subset) { Φ π ( i ) } mi =1 is m points Φ j sampled from { Φ j } nj =1 through the sampling matrix Q .
Note that, the proposed Nyström kernel k -means can also be understood as a variant ROS, based onthe identity matrix as an orthonormal matrix and not using the Rademacher randomization.
Complexity analysis: In terms of time cost, the matrix S is a scaled identity matrix so that thecomputation of step 3 and step 4 in Algorithm 1 is not needed.
Therefore, the time complexity of
Nyström kernel k -means is decided by the iteration operation of k -means algorithm over the columns˜ k i , which is O ( nmkt ) .
In terms of space, the key is the matrix ˜ K .
Therefore, the space complexityof the proposed Nyström kernel k -means is O ( nm ) .
Note that, computing the (approximate) kernel matrix K ( ˆ K = QK ) is a one-off cost.
Once thecolumns ˆ k i of ˆ K are computed they can be stored and manipulated in nm time and space at most,with at least n/m improvement over the n 2 time and space required by the exact columns k
i of K .
Theoretical Analysis
In this section, we exploit the excess risk bound of the proposed randomized sketches kernel k -means.
Theoretical analysis shows that we can improve the computational requirements of kernel k -meansusing sub-Gaussian, ROS, and Nyström, while maintaining optimal generalization guarantees.
Theorem 2.
If (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , ε ∈ (0 , 1) , δ ∈ (0 , 1) , and, in either one of the three casesof sub-Gaussian, ROS, and Nyström, the sketch dimension is
[Equation]
, then, with probability at least 1 − δ , we have
[Equation]
Remark 1.
From a statistical point of view, let ε = 1 / √ n , Theorem 2 shows that when the sketchdimension is m = Ω( √ n ) , the proposed randomized sketches (sub-Gaussian, ROS, and Nyström)kernel k -means achieve the same excess risk bound ˜ O (cid:18)(cid:113) kn (cid:19) as the exact kernel k -means.
[Table]
Sub-Gaussian Sketches ( This Paper )
[Table]
ROS Sketches ( This Paper )
[Table] ( This Paper ) [Table]
Remark 2.
From a computational point of view [Table] we can construct the √ n -dimension randomizedsketches simply, which can greatly reduces the total required space from O ( n 2 ) to O ( n √ n ) and thetotal required time from O ( n 2 kt ) to O ( n √ n + n √ nkt ) at least, with the optimal excess risk bound.
Remark 3.
In Calandriello & Rosasco (2018), when the approximate dimension m is Ω( √ n ) , theexcess risk bound can reach ˜ O (cid:16) k √ n (cid:17) , which is linearly dependent on k and fail to reach the optimalbound.
The corresponding space complexity and time complexity are O ( n √ n ) and O (cid:0) nkt √ n + n 2 (cid:1) ,respectively.
Compared to it, our proposed methods obtain the better excess risk bound and reducethe time complexity from O (cid:0) nkt √ n + n 2 (cid:1) to O ( nkt √ n + n √ n ) at least.
Subsequently, Liu Yong
Liu (2021) further improves the excess risk bound of the method in Calandriello & Rosasco (2018)to ˜ O (cid:18)(cid:113) kn (cid:19) , but the corresponding approximate dimension m is increased to Ω( √ nk ) .
Meanwhile, its space complexity and time complexity increase to O (cid:16) n √ nk (cid:17) and O (cid:16) nkt √ nk + n 2 k (cid:17) .
Compared to it, the proposed methods reduce the time complexity from O (cid:16) nkt √ nk + n 2 k (cid:17) to
O ( nkt √ n + n √ n ) at least and reduce the space complexity from O (cid:16) n √ nk (cid:17) to O ( n √ n ) whilemaintaining the optimal excess risk bound ˜ O (cid:18)(cid:113) kn (cid:19) and the smaller m = Ω( √ n ) .
To the best ofour knowledge, the proposed methods are the ﬁrst time that they are always possible to maintainthe optimal excess risk bound ˜ O (cid:18)(cid:113) kn (cid:19) in unsupervised non-parametric problem with smallerm =
Ω ( √ n ) , while greatly reducing the time and space requirements.
In Table 1, we show thedetail space complexity, time complexity, excess risk bounds, and m of the main approximate kernelk-means.
4.1 Further Results: k -Means++
We adopt the improved kernel k -means++ sampling Lattanzi & Sohler (2019), which has a localsearch strategy, for the proposed randomized sketches kernel k -mean.
Here is its theoretical analysis.
Lemma 1 ( Lattanzi & Sohler (2019) ) .
If C + n is obtained by the improved k -means++ algorithmwith a local search strategy Lattanzi & Sohler (2019), then
[Equation]
16) where (cid:36) is a constant and J is the randomness derived from the k -means++ initialization.
Note that, this is a multiplicative error bound on the empirical risk.
[Figure]
Theorem 3.
Let C + n,m be obtained by the improved k -means++ algorithm with a local searchstrategy Lattanzi & Sohler (2019).
If (cid:107) Φ x (cid:107) ≤ 1 for any x ∈ X , ε ∈ (0 , 1) , δ ∈ (0 , 1) , and, in eitherone of the three cases of sub-Gaussian, ROS, and Nyström, the sketch dimension is
[Equation]
then, with probability at least 1 − δ , we have
[Equation]
where J is the randomness derived from the k -means++ initialization.
Theorem 3 shows that, if the optimal clustering risk W ∗ ( µ ) is smaller than ˜ O
( (cid:112) k/n ) , the risk of
W ( C + n,m , µ ) can reach ˜ O
( (cid:112) k/n ) .
Note that ε is small, i.e. ε = 1 / √ n .
Experiments
In this section, we evaluate experimentally our theoretical analysis on both simulated data and realword data for the proposed sub-Gaussian sketches kernel k -means, ROS sketches kernel k -means,and Nyström kernel k -means.
The server is 32 cores (2.40GHz) and 32 GB of RAM.
The comparedmethods are the exact kernel k -means,
Nyström (Liu) Liu (2021), ROS, Sub-Gaussian, and Nyström(Ours).
For the sake of distinguishing, Nyström (Liu) is Nyström Liu (2021) in this paper.
Eachexperiment is repeated 5 times.
5.1 Numerical Experiments on Simulated Data
We conduct the experiments to validate our theoretical analysis of the proposed randomized sketcheskernel k -means on simulated data.
Now we generate the simulated data.
Let c ∗ i ∈ R 8 , i = [1 , k ] , be the clustering centers, where the values of the dimensions are 1 or − 1 with the probability of 1/2.
The data in i th clustering follows the normal distribution with mean c ∗ i and variance 1.
The number of data in each clustering is the same.
We use the Gaussian kernel
[Equation]
Generating 10,000 samples for training and 10,000 samples for testing.
The number of training samples in each clustering is 10000 /k .
The accuracy of kernel k-means He & Zhang (2018) on the
[Table]
[Table] [Table]
test set can be written as [Equation] where y is the solution returned by the (approximate) kernel k-means using Lloyd’s algorithm Lloyd(1982), ˆ y is the real label, and ¨ n is the number of data in the test set.
If p = q , υ ( p, q ) = 1 , otherwiseυ ( p, q ) = 0 . map ( · ) represents the best mapping to match ˆ y and y .
The higher the accuracy, thebetter the method.
The test accuracy and training running time of the approximate kernel k -meanswith different m are given in Figure 1 , which can be summarized as follows: (1) There exists alower bound of the approximate dimensions m = √ n = 100 .
When this lower bound is reached, theaccuracy of the proposed methods tends to be stable.
This is consistent with our theoretical analysisin Theorem 2.
The accuracy of the proposed methods keeps the similar accuracy to the exactkernel k-means.
(3) We take the logarithm of the running time (in seconds) in Figure 1.
Our methods(ROS, Sub-Gaussian, Nyström) have obvious advantages over other methods in running time.
Thisveriﬁes our complexity analysis.
5.2 Numerical Experiments on Real-World Scenarios
In this subsection, we perform the experiments on the 8 real datasets: dna, segment, mushrooms,pendigits, protein, a8a, w7a, and connect-4, which are from LIBSVM website 1 .
The detail of thedatasets is shown in Table 2.
70 percent of the data in each dataset is used for training experiments,and the rest is used for testing.
The Gaussian kernel is exp (cid:16) − (cid:107) x − x (cid:48)
(cid:107) 2 σ(cid:17) , where σ = (cid:113) (cid:80) ij (cid:107) x
i − x j (cid:107) 2 n .
The experimental results are shown in Table 3.
From the above results on real datasets, we can ﬁnd that the proposed kernel k-means methods give asimilar accuracy as the exact kernel k -means and outperform it in time cost, which also matches ourtheoretical analysis.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide
Pseudo code, data, and instructions.
Lucky to be accepted, the code will be provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation
[N/A]