Original: C ORESETS FOR K ERNEL C LUSTERING
Revised: C ORESETS FOR K ERNEL C LUSTERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We devise the ﬁrst coreset for kernel k -M EANS , and use it to obtain new, more efﬁcient, algorithms.
Revised: We devise coresets for kernel k -M EANS with a general kernel, and use them to obtain new, more efﬁcient, algorithms.

Original: Kernel k -M EANS has superior clustering capability compared to classical k -M EANS  particularly when clusters are separable non-linearly, but it also introduces signiﬁcant computational challenges.
Revised: Kernel k -M EANS has superior clustering capability compared to classical k -M EANS , particularly when clusters are separable non-linearly, but it also introduces signiﬁcant computational challenges.

Original: We address this computational issue by constructing a coreset, which is a reduced dataset that accurately preserves the clustering costs.
Revised: We address this computational issue by constructing a coreset, which is a reduced dataset that accurately preserves the clustering costs.

Original: Our main result is the ﬁrst coreset for kernel k -M EANS , whose sizeis independent of  the number of input points n , and moreover is constructed in time near-linear in n .
Revised: Our main result is the ﬁrst coreset for kernel k -M EANS with a general kernel, that has size independent of n , the number of input points; moreover, our coreset can be constructed in time near-linear in n .

Original: This result immediately implies new algorithms for kernel k -M EANS , such as a (1 + (cid:15) )
Revised: This result immediately implies new algorithms for kernel k -M EANS , such as a (1 + (cid:15) )

Original: -approximation in time near-linear in n , and a streaming algorithm using space and update time poly( k(cid:15)
Revised: -approximation in time near-linear in n , and a streaming algorithm using space and update time poly( k(cid:15)

Original: − 1 log n ) .
Revised: − 1 log n ) .

Original: We validate our coreset on various datasets with different kernels.
Revised: We validate our coreset on various datasets with different kernels.

Original: Our coreset performs consistently well, achieving small errors while using very few points.
Revised: Our coreset performs consistently well, achieving small errors while using very few points.

Original: We show that our coresets can speed up kernel k -M EANS ++ (the kernelized version of the widely used k -M EANS ++ algorithm), and we further use this faster kernel k -M EANS ++ for spectral clustering.
Revised: We show that our coresets can speed up kernel k -M EANS ++ (the kernelized version of the widely used k -M EANS ++ algorithm), and we further use this faster kernel k -M EANS ++ for spectral clustering.

Original: In both applications, we achieve up to 1000x speedup while the error is comparable to baselines that do not use coresets.
Revised: In both applications, we achieve up to 1000x speedup while the error is comparable to baselines that do not use coresets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We design the ﬁrst coresets, and consequently new efﬁcient algorithms, for kernel k -M EANS and related problems, like its generalization kernel ( k, z ) -C LUSTERING .
Revised: We design the ﬁrst coresets, and consequently new efﬁcient algorithms, for kernel k -M EANS and related problems, like its generalization kernel ( k, z ) -C LUSTERING , under general kernels.

Original: The k -M EANS problem has proved to be fundamental for unsupervised learning in numerous application domains.
Revised: The kM EANS problem has proved to be fundamental for unsupervised learning in numerous application domains.

Original: Vanilla kM EANS fails to capture sophisticated cluster structures, e.g., when the clusters are separable nonlinearly, but this can be tackled by applying kernel methods (Sch ¨ olkopf et al., 1998; Girolami, 2002).
Revised: Vanilla k -M EANS fails to capture sophisticated cluster structures, e.g., when the clusters are separable non-linearly, but this can be tackled by applying kernel methods (Sch ¨ olkopf et al., 1998; Girolami, 2002).

Original: This has led to kernel k -M EANS , where data points are ﬁrst mapped to a high-dimensional feature space (possibly implicitly via a kernel function), and then clustered in this richer space using a classical k -M EANS .
Revised: This has led to kernel k -M EANS , where data points are ﬁrst mapped to a high-dimensional feature space (possibly implicitly via a kernel function), and then clustered in this richer space using a classical k -M EANS .

Original: Formally, a kernel for a dataset X is a function K : X × X
Revised: Formally, a kernel for a dataset X is a function K : X × X

Original: → R + (intended to measure similarity between elements in X ) that can be realized by inner products, i.e., there exist a Hilbert space H and a map ϕ : X → H (called feature space and feature map) such that
Revised: → R + (intended to measure similarity between elements in X ) that can be realized by inner products, i.e., there exist a Hilbert space H and a map ϕ : X → H (called feature space and feature map) such that

Original: [Equation]
Revised: [Equation]

Original: In kernel k -M EANS , the input is a dataset X with weight function w X : X → R + and a kernel function K : X × X
Revised: In kernel k -M EANS , the input is a dataset X with weight function w X : X → R + and a kernel function K : X × X

Original: → R + as above, and the goal is to ﬁnd a k -point center set C ⊆ H that minimizes the objective
Revised: → R + as above, and the goal is to ﬁnd a k -point center set C ⊆ H that minimizes the objective

Original: [Equation]
Revised: [Equation]

Original: (An equivalent formulation asks for a k -partitioning of X , keeping C implicit.)
Revised: (An equivalent formulation asks for a k -partitioning of X , keeping C implicit.)

Original: This kernel version has superior clustering capability compared to classical k -M EANS (Zhang &
Revised: This kernel version has superior clustering capability compared to classical k -M EANS (Zhang &

Original: Rudnicky, 2002; Kim et al., 2005), and has proved useful in different application domains, such as pattern recognition (Shawe-Taylor & Cristianini, 2004), natural language processing (Andrews &
Revised: Rudnicky, 2002; Kim et al., 2005), and has proved useful in different application domains, such as pattern recognition (Shawe-Taylor & Cristianini, 2004), natural language processing (Andrews &

Original: Fox, 2007), biology (G¨onen & Margolin, 2014) and social networks (van Laarhoven & Marchiori, 2016).
Revised: Fox, 2007), biology (G¨onen & Margolin, 2014) and social networks (van Laarhoven & Marchiori, 2016).

Original: In fact, kernel k -M EANS is useful also for solving other clustering problems, such as normalized cut and spectral clustering (Dhillon et al., 2004; Ding et al., 2005).
Revised: In fact, kernel k -M EANS is useful also for solving other clustering problems, such as normalized cut and spectral clustering (Dhillon et al., 2004; Ding et al., 2005).

Original: Computational challenges.
Revised: Computational challenges.

Original: As observed in previous work (Girolami, 2002; Dhillon et al., 2004), the kernel trick can be applied to rewrite kernel k -M EANS using access only to the kernel K ( · , · ) and without computing the very high-dimensional map ϕ explicitly.
Revised: As observed in previous work (Girolami, 2002; Dhillon et al., 2004), the kernel trick can be applied to rewrite kernel k -M EANS using access only to the kernel K ( · , · ) and without computing the very high-dimensional map ϕ explicitly.

Original: However, this approach has outstanding computational challenges (compared to classical k -M EANS ), essentially because of the kernel trick.
Revised: However, this approach has outstanding computational challenges (compared to classical k -M EANS ), essentially because of the kernel trick.

Original: Consider the special case where k = 1 and the input is n unweighted points (i.e., 1 -M EAN clustering).
Revised: Consider the special case where k = 1 and the input is n unweighted points (i.e., 1 -M EAN clustering).

Original: It is well known that the optimal center c (cid:63) has a closed form c (cid:63) :
Revised: It is well known that the optimal center c (cid:63) has a closed form c (cid:63) :

Original: = 1 n (cid:80) x ∈ X ϕ ( x ) .
Revised: = 1 n (cid:80) x ∈ X ϕ ( x ) .

Original: But the kernel trick requires Ω( n 2 ) accesses to K to evaluate cost ϕ
Revised: But the kernel trick requires Ω( n 2 ) accesses to K to evaluate cost ϕ

Original: ( X, c (cid:63) ) , 1 while in the classical setting such evaluation needs only O ( n ) distance computations.
Revised: ( X, c (cid:63) ) , 1 while in the classical setting such evaluation needs only O ( n ) distance computations.

Original: This Ω( n 2 ) barrier can be bypassed by allowing (1 + (cid:15) ) -approximation.
Revised: This Ω( n 2 ) barrier can be bypassed by allowing (1 + (cid:15) ) -approximation.

Original: In particular, let S be a uniform sample of poly( (cid:15) − 1 ) points from X , and let ˆ c := 1 | S | (cid:80) x ∈ S ϕ ( x ) be its 1 -M EAN ; then with high probability, cost ϕ ( X, ˆ c ) ≤ (1 + (cid:15) ) cost ϕ
Revised: In particular, let S be a uniform sample of poly( (cid:15) − 1 ) points from X , and let ˆ c := 1 | S | (cid:80) x ∈ S ϕ ( x ) be its 1 -M EAN ; then with high probability, cost ϕ ( X, ˆ c ) ≤ (1 + (cid:15) ) cost ϕ

Original: ( X, c (cid:63) ) and evaluating cost ϕ ( X, ˆ c ) takes only poly( (cid:15) − 1 ) n time.
Revised: ( X, c (cid:63) ) and evaluating cost ϕ ( X, ˆ c ) takes only poly( (cid:15) − 1 ) n time.

Original: However, this uniform-sampling approach does not generally work for k ≥ 2 , because if the optimal clustering is highly imbalanced, a uniform sample is unlikely to include any point from a small cluster.
Revised: However, this uniform-sampling approach does not generally work for k ≥ 2 , because if the optimal clustering is highly imbalanced, a uniform sample is unlikely to include any point from a small cluster.

Original: Alternative approaches, such as dimension reduction, were also proposed to obtain efﬁcient algorithms for kernel k -M EANS , but they too do not fully resolve the computational issue.
Revised: Alternative approaches, such as dimension reduction, were also proposed to obtain efﬁcient algorithms for kernel k -M EANS , but they too do not fully resolve the computational issue.

Original: We elaborate on these approaches in Section 1.2.
Revised: We elaborate on these approaches in Section 1.2.

Original: Our approach.
Revised: Our approach.

Original: To tackle this computational challenge, we adapt the notion of a coreset (HarPeled & Mazumdar, 2004) to kernel k -M EANS .
Revised: To tackle this computational challenge, we adapt the notion of a coreset (HarPeled & Mazumdar, 2004) to kernel k -M EANS .

Original: Informally, a coreset is a tiny reweighted subset of the original dataset on which the clustering cost is preserved within (1 ± (cid:15) ) factor for all candidate centers C ⊆ H .
Revised: Informally, a coreset is a tiny reweighted subset of the original dataset on which the clustering cost is preserved within (1 ± (cid:15) ) factor for all candidate centers C ⊆ H .

Original: This notion has proved very successful for classical k -M EANS , e.g., to design efﬁcient near-linear algorithms.
Revised: This notion has proved very successful for classical k -M EANS , e.g., to design efﬁcient near-linear algorithms.

Original: In our context of kernel k -M EANS , a coreset of size s for an input of size n = | X | has a huge advantage that its k optimal center points can all be represented as linear combinations of only s points in the feature space.
Revised: In our context of kernel k -M EANS , a coreset of size s for an input of size n = | X | has a huge advantage that its k optimal center points can all be represented as linear combinations of only s points in the feature space.

Original: Given these k optimal centers (as linear combinations), evaluating the distance between a point ϕ ( x ) and such a center takes merely O ( s 2 ) time, instead of O ( n 2 ) , and consequently the objective can be (1+ (cid:15) ) -approximated in time
Revised: Given these k optimal centers (as linear combinations), evaluating the distance between a point ϕ ( x ) and such a center takes merely O ( s 2 ) time, instead of O ( n 2 ) , and consequently the objective can be (1+ (cid:15) ) -approximated in time

Original: O ( s 2 kn ) .
Revised: O ( s 2 kn ) .

Original: Moreover, it sufﬁces to use k centers (again as linear combinations) that are (1 + (cid:15) ) -approximately
Revised: Moreover, it sufﬁces to use k centers (again as linear combinations) that are (1 + (cid:15) ) -approximately

Original: optimal for the coreset S .
Revised: optimal for the coreset S .

Original: In addition, coresets are very useful in dealing with massive datasets, since an ofﬂine construction of coresets usually generalizes to the streaming setting (Har-Peled & Mazumdar, 2004), distributed computing (Balcan et al., 2013) and dynamic algorithms (Henzinger & Kale, 2020) via the merge-and-reduce method (Har-Peled & Mazumdar, 2004), and existing (ofﬂine) algorithms can be efﬁciently applied to the coreset, instead of to the original dataset, with minor or no modiﬁcations.
Revised: In addition, coresets are very useful in dealing with massive datasets, since an ofﬂine construction of coresets usually generalizes to the streaming setting (Har-Peled & Mazumdar, 2004), distributed computing (Balcan et al., 2013) and dynamic algorithms (Henzinger & Kale, 2020) via the merge-and-reduce method (Har-Peled & Mazumdar, 2004), and existing (ofﬂine) algorithms can be efﬁciently applied to the coreset, instead of to the original dataset, with minor or no modiﬁcations.

Original: 1. O UR R ESULTS
Revised: 1. O UR R ESULTS

Original: Our main result is the ﬁrst coreset for kernel k -M EANS , whose sizeis independent of the input size n = | X | , and moreover it is constructed in near-linear time for small k .
Revised: Our main result is the ﬁrst coreset for kernel k -M EANS with a general kernel, that has size independent of the input size n = | X | ; moreover, our coreset can be constructed in near-linear time for small k .

Original: (In fact, it generalizes to kernel ( k, z ) -C
Revised: (In fact, it generalizes to kernel ( k, z ) -C

Original: LUSTERING , see Section 2 for deﬁnitions.)
Revised: LUSTERING , see Section 2 for deﬁnitions.)

Original: Formally, an (cid:15) -coreset for kernel kM EANS with respect to weighted dataset X and kernel function K : X × X
Revised: Formally, an (cid:15) -coreset for kernel k -M EANS with respect to weighted dataset X and kernel function K : X × X

Original: → R + is a weighted subset S ⊆ X , such that for every feature space H and feature map ϕ that realize K , as deﬁned in (1),
Revised: → R + is a weighted subset S ⊆ X , such that for every feature space H and feature map ϕ that realize K , as deﬁned in (1),

Original: [Equation]
Revised: [Equation]

Revised: Previously, only a weak coreset was known for kernel k -M EANS (Feldman et al., 2007), meaning that the objective is preserved only for certain candidate centers (whereas (3) guarantees this for all centers), and that coreset works only for certain kernels (ﬁnite-dimensional).

Revised: While we employ a similar approach, the technical differences make our bottom-line result much stronger.

Original: Throughout, we assume an oracle access to K takes unit time, and therefore our stated running times also bound the number of accesses to K .
Revised: Throughout, we assume an oracle access to K takes unit time, and therefore our stated running times also bound the number of accesses to K .

Original: We denote ˜ O
Revised: We denote ˜ O

Original: ( f ) =
Revised: ( f ) =

Original: O ( f  polylog f ) to suppress logarithmic factors.
Revised: O ( f · polylog f ) to suppress logarithmic factors.

Original: Theorem 1.1 (Informal version of Theorem 3.1) .
Revised: Theorem 1.1 (Informal version of Theorem 3.1) .

Original: Given n -point weighted dataset X , oracle access to a kernel K : X × X → R + , integer k ≥ 1 and 0 < (cid:15) < 1 , one can construct in time
Revised: Given n -point weighted dataset X , oracle access to a kernel K : X × X → R + , integer k ≥ 1 and 0 < (cid:15) < 1 , one can construct in time

Original: ˜ O
Revised: ˜ O

Original: ( nk ) ,
Revised: ( nk ) ,

Original: a reweighted subset S ⊆ X of size | S | = poly( k(cid:15) − 1 ) , that with high probability is an (cid:15) -coreset for kernel k -M EANS with respect to X and K .
Revised: a reweighted subset S ⊆ X of size | S | = poly( k(cid:15) − 1 ) , that with high probability is an (cid:15) -coreset for kernel k -M EANS with respect to X and K .

Original: We can employ our coreset to devise a (1 + (cid:15) )
Revised: We can employ our coreset to devise a (1 + (cid:15) )

Original: -approximation algorithm for kernel k -M EANS , that runs in time that is near-linear in n and parameterized by k .
Revised: -approximation algorithm for kernel k -M EANS , that runs in time that is near-linear in n and parameterized by k .

Original: This is stated in Corollary 1.2, whose proof follows by solving k -M EANS on S optimally, using straightforward enumeration over all kpartitions of S .
Revised: This is stated in Corollary 1.2, whose proof follows by solving k -M EANS on S optimally, using straightforward enumeration over all kpartitions of S .

Original: To the best of our knowledge, such a fast (1+ (cid:15) ) -approximation for kernel k -M EANS was not known even for k = 2 ; for example, uniform sampling would fail in cases where the optimal clustering is very imbalanced, as mentioned earlier.
Revised: To the best of our knowledge, such a fast (1+ (cid:15) ) -approximation for kernel k -M EANS was not known even for k = 2 ; for example, uniform sampling would fail in cases where the optimal clustering is very imbalanced, as mentioned earlier.

Original: Corollary 1.2 (FPT-PTAS) .
Revised: Corollary 1.2 (FPT-PTAS) .

Original: Given n -point weighted dataset X , oracle access to a kernel K :
Revised: Given n -point weighted dataset X , oracle access to a kernel K :

Original: X × X
Revised: X × X

Original: → R + , integer k ≥ 1 and 0 < (cid:15) < 1 , one can compute in time
Revised: → R + , integer k ≥ 1 and 0 < (cid:15) < 1 , one can compute in time

Original: O ( nk + k poly( k(cid:15) − 1 ) ) , a center set C of k points, each represented as a linear combination of at most poly( k(cid:15) − 1 ) points from ϕ ( X ) , such that with high probability C is a (1 + (cid:15) ) -approximation for kernel k -M EANS on
Revised: O ( nk + k poly( k(cid:15) − 1 ) ) , a center set C of k points, each represented as a linear combination of at most poly( k(cid:15) − 1 ) points from ϕ ( X ) , such that with high probability C is a (1 + (cid:15) ) -approximation for kernel k -M EANS on

Original: X and K .
Revised: X and K .

Original: In particular, given such C , one can ﬁnd for each x ∈ X its closest center in C in time poly( k(cid:15) − 1 ) .
Revised: In particular, given such C , one can ﬁnd for each x ∈ X its closest center in C in time poly( k(cid:15) − 1 ) .

Original: In fact, for the purpose of ﬁnding near-optimal solutions, it already sufﬁces to preserve the cost for centers coming from span( ϕ ( X ))
Revised: In fact, for the purpose of ﬁnding near-optimal solutions, it already sufﬁces to preserve the cost for centers coming from span( ϕ ( X ))

Original: (see Fact 2.1) which is an n -dimensional subspace.
Revised: (see Fact 2.1) which is an n -dimensional subspace.

Original: However, our deﬁnition of coreset in (3) is much stronger, in that the objective is preserved even for centers coming from a possibly inﬁnite-dimensional feature space.
Revised: However, our deﬁnition of coreset in (3) is much stronger, in that the objective is preserved even for centers coming from a possibly inﬁnite-dimensional feature space.

Original: This stronger guarantee ensures that the coreset is composable, and thus the standard merge-and-reduce method can be applied.
Revised: This stronger guarantee ensures that the coreset is composable, and thus the standard merge-and-reduce method can be applied.

Original: In particular, our coreset implies the ﬁrst streaming algorithm for kernel k -M EANS .
Revised: In particular, our coreset implies the ﬁrst streaming algorithm for kernel k -M EANS .

Original: Corollary 1.3 (Streaming kernel k -M EANS ) .
Revised: Corollary 1.3 (Streaming kernel k -M EANS ) .

Original: There is a streaming algorithm that given a dataset
Revised: There is a streaming algorithm that given a dataset

Original: X presented as a stream of n points, and oracle access to a kernel K : X × X → R + , constructs a reweighted subset S ⊆ X of poly( k(cid:15) − 1 ) points using poly( k(cid:15) − 1 log n ) words of space and update time, such that with high probability S is an (cid:15) -coreset for k -M EANS with respect to X and K .
Revised: X presented as a stream of n points, and oracle access to a kernel K : X × X → R + , constructs a reweighted subset S ⊆ X of poly( k(cid:15) − 1 ) points using poly( k(cid:15) − 1 log n ) words of space and update time, such that with high probability S is an (cid:15) -coreset for k -M EANS with respect to X and K .

Original: Experiments and other applications.
Revised: Experiments and other applications.

Original: We validate the efﬁciency and accuracy of our coresets on various data sets with polynomial and Gaussian kernels, which are frequently-used kernels.
Revised: We validate the efﬁciency and accuracy of our coresets on various data sets with polynomial and Gaussian kernels, which are frequently-used kernels.

Original: For every dataset, kernel, and coreset-size that we test, our coreset performs consistently better than uniform sampling which serves as a baseline.
Revised: For every dataset, kernel, and coreset-size that we test, our coreset performs consistently better than uniform sampling which serves as a baseline.

Original: In fact, our coreset achieves less than 10% error using only about 1000 points for every dataset.
Revised: In fact, our coreset achieves less than 10% error using only about 1000 points for every dataset.

Original: We also showcase signiﬁcant speedup to several applications that can be obtained using our coresets.
Revised: We also showcase signiﬁcant speedup to several applications that can be obtained using our coresets.

Original: Speciﬁcally, we adapt the widely used k -M EANS ++ (Arthur & Vassilvitskii, 2007) to the kernel setting, and we compare the running time and accuracy of this kernelized k -M EANS ++ with and without coresets.
Revised: Speciﬁcally, we adapt the widely used k -M EANS ++ (Arthur & Vassilvitskii, 2007) to the kernel setting, and we compare the running time and accuracy of this kernelized k -M EANS ++ with and without coresets.

Original: On a dataset of size 10 5 , we observe more than 1000 x speedup of k -M EANS ++ when using coresets, while achieving a very similar error.
Revised: On a dataset of size 10 5 , we observe more than 1000 x speedup of k -M EANS ++ when using coresets, while achieving a very similar error.

Original: Furthermore, this new efﬁcient version of kernelized k -M EANS ++ (based on coresets) is applied to solve spectral clustering, using the connection discoverd by Dhillon et al.
Revised: Furthermore, this new efﬁcient version of kernelized k -M EANS ++ (based on coresets) is applied to solve spectral clustering, using the connection discoverd by Dhillon et al.

Original: Compared to the implementation provided by Scikitlearn (Pedregosa et al., 2011), our algorithm often achieves a better result and uses signiﬁcantly less time.
Revised: Compared to the implementation provided by Scikitlearn (Pedregosa et al., 2011), our algorithm often achieves a better result and uses signiﬁcantly less time.

Original: Hence, our coreset-based approach can potentially become the leading method for solving spectral clustering in practice.
Revised: Hence, our coreset-based approach can potentially become the leading method for solving spectral clustering in practice.

Original: 1. C OMPARISON TO P REVIOUS A PPROACHES
Revised: 1. C OMPARISON TO P REVIOUS A PPROACHES

Original: The computational issue of kernel k -M EANS is an important research topic and has attracted significant attention.
Revised: The computational issue of kernel k -M EANS is an important research topic and has attracted significant attention.

Original: In the following, we compare our result with previous work that is representative of different possible approaches for the problem.
Revised: In the following, we compare our result with previous work that is representative of different possible approaches for the problem.

Original: Uniform sampling of data points is a commonly used technique, which ﬁts well in kernel clustering, because samples can be drawn without any access to the kernel.
Revised: Uniform sampling of data points is a commonly used technique, which ﬁts well in kernel clustering, because samples can be drawn without any access to the kernel.

Original: While the coresets that we use rely on sampling, we employ importance sampling, which is non-uniform by deﬁnition.
Revised: While the coresets that we use rely on sampling, we employ importance sampling, which is non-uniform by deﬁnition.

Original: Chitta et al.
Revised: Chitta et al.

Original: (2011) employs uniform sampling for kernel k -M EANS , but instead of solving kernel k -M EANS on a sample of data points directly (as we do), their method works in iterations, similarly to Lloyd’s algorithm, that ﬁnd a center set that is a linear combination of the sample.
Revised: (2011) employs uniform sampling for kernel k -M EANS , but instead of solving kernel k -M EANS on a sample of data points directly (as we do), their method works in iterations, similarly to Lloyd’s algorithm, that ﬁnd a center set that is a linear combination of the sample.

Original: However, it has no worst-case guarantees on the error or on the running time, which could be much larger than ˜ O
Revised: However, it has no worst-case guarantees on the error or on the running time, which could be much larger than ˜ O

Original: ( nk ) .
Revised: ( nk ) .

Original: Lastly, this method does not generalize easily to other sublinear settings, such as streaming (as our Corollary 1.3).
Revised: Lastly, this method does not generalize easily to other sublinear settings, such as streaming (as our Corollary 1.3).

Original: Ren & Du (2020) analyze uniform sampling for k -M EANS in a Euclidean space, which could be the kernel’s feature space.
Revised: Ren & Du (2020) analyze uniform sampling for k -M EANS in a Euclidean space, which could be the kernel’s feature space.

Original: In their analysis, the number of samples (and thus running time) crucially depends on the diameter of the dataset and on the optimal objective value, and is thus not bounded in the worse-case.
Revised: In their analysis, the number of samples (and thus running time) crucially depends on the diameter of the dataset and on the optimal objective value, and is thus not bounded in the worse-case.

Original: This analysis builds on several earlier papers, for example Czumaj & Sohler (2007) achieve bounds of similar ﬂavor for k -M EANS in general metric spaces.
Revised: This analysis builds on several earlier papers, for example Czumaj & Sohler (2007) achieve bounds of similar ﬂavor for k -M EANS in general metric spaces.

Original: Another common approach to speed up kernel k -M EANS is to approximate the kernel K using dimension-reduction techniques.
Revised: Another common approach to speed up kernel k -M EANS is to approximate the kernel K using dimension-reduction techniques.

Original: In a seminal paper, Rahimi & Recht (2007) proposed a method that efﬁciently computes a low (namely, O (log n ) ) dimensional feature map ˜ ϕ that approximates ϕ (without computing ϕ explicitly), and this ˜ ϕ can be used in downstream applications.
Revised: In a seminal paper, Rahimi & Recht (2007) proposed a method that efﬁciently computes a low (namely, O (log n ) ) dimensional feature map ˜ ϕ that approximates ϕ (without computing ϕ explicitly), and this ˜ ϕ can be used in downstream applications.

Original: Their method is based on designing random Fourier features, and works for a family of kernels that includes the
Revised: Their method is based on designing random Fourier features, and works for a family of kernels that includes the

Original: Gaussian one, but not general kernels.
Revised: Gaussian one, but not general kernels.

Original: This method was subsequently tailored to kernel k -M EANS by Chitta et al.
Revised: This method was subsequently tailored to kernel k -M EANS by Chitta et al.

Original: (2012), and another followup work by Chen & Phillips (2017)
Revised: (2012), and another followup work by Chen & Phillips (2017)

Original: established worst-case bounds for kernel k -M EANS with Gaussian kernels.
Revised: established worst-case bounds for kernel k -M EANS with Gaussian kernels.

Original: Despite these promising advances, we are not aware of any work based on dimension reduction that can handle a general kernel function (as in our approach).
Revised: Despite these promising advances, we are not aware of any work based on dimension reduction that can handle a general kernel function (as in our approach).

Original: In a sense, these dimension-reduction techniques are “orthogonal” to our data-reduction approach, and the two techniques can possibly be combined to yield even better results.
Revised: In a sense, these dimension-reduction techniques are “orthogonal” to our data-reduction approach, and the two techniques can possibly be combined to yield even better results.

Original: An alternative dimension-reduction approach is low-rank approximation of the kernel matrix K .
Revised: An alternative dimension-reduction approach is low-rank approximation of the kernel matrix K .

Original: Recent work by Musco & Musco (2017) and by Wang et al.
Revised: Recent work by Musco & Musco (2017) and by Wang et al.

Original: (2019) presents algorithms based on
Revised: (2019) presents algorithms based on

Original: Nystr¨om approximation to compute a low-rank (namely, O ( k/(cid:15) ) ) approximation (cid:101) K to the kernel matrix K in time near-linear in n , such that the optimal kernel k -M EANS on (cid:101) K achieves (1 + (cid:15) ) - approximation to that on the original kernel K .
Revised: Nystr ¨ om approximation to compute a low-rank (namely, O ( k/(cid:15) ) ) approximation (cid:101) K to the kernel matrix K in time near-linear in n , such that the optimal kernel k -M EANS on (cid:101) K achieves (1 + (cid:15) ) - approximation to that on the original kernel K .

Original: However, the low-rank (cid:101) K does not immediately imply efﬁcientapproximation algorithms for kernel k -M EANS (for instance, Lloyd’s algorithm still requires Θ( n 2 ) queries to (cid:101) K per iteration), while our coreset can be readily combined with off-theshelf clustering algorithms.
Revised: However, the low-rank (cid:101) K does not immediately imply efﬁcient algorithms for kernel k -M EANS (for instance, Lloyd’s algorithm still requires Θ( n 2 ) queries to (cid:101) K per iteration), while our coreset can be readily combined with off-the-shelf clustering algorithms.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Notation.
Revised: Notation.

Original: A weighted set U is a ﬁnite set U associated with a weight function w U : U → R + .
Revised: A weighted set U is a ﬁnite set U associated with a weight function w U : U → R + .

Original: For such U , let (cid:107) U (cid:107) 0 be the number of distinct elements in it.
Revised: For such U , let (cid:107) U (cid:107) 0 be the number of distinct elements in it.

Original: For a weight function as above and a subset S ⊆ U , deﬁne w U ( S ) :
Revised: For a weight function as above and a subset S ⊆ U , deﬁne w U ( S ) :

Original: = (cid:80) u ∈ S w U ( u ) .
Revised: = (cid:80) u ∈ S w U ( u ) .

Original: For any other map f : U → V (not a weight function), we follow the standard deﬁnition f ( S ) :
Revised: For any other map f : U → V (not a weight function), we follow the standard deﬁnition f ( S ) :

Original: = { f ( x ) : x ∈ S } .
Revised: = { f ( x ) : x ∈ S } .

Original: For an integer t ≥ 1 , let [ t ] := { 1 , . . .
Revised: For an integer t ≥ 1 , let [ t ] := { 1 , . . .

Original: , t } .
Revised: , t } .

Original: For a real number x and an integer i ≥ 1 , let log ( i ) x be the i -th iterated log of x , i.e., log (1) x
Revised: For a real number x and an integer i ≥ 1 , let log ( i ) x be the i -th iterated log of x , i.e., log (1) x

Original: = log x
Revised: = log x

Original: and for i ≥ 2 let log ( i ) x = log(log
Revised: and for i ≥ 2 let log ( i ) x = log(log

Original: ( i − 1) x ) .
Revised: ( i − 1) x ) .

Original: Kernel functions.
Revised: Kernel functions.

Original: Let X be a set of n points.
Revised: Let X be a set of n points.

Original: A function K : X × X → R + is a kernel function if the n × n matrix M such that M ij =
Revised: A function K : X × X → R + is a kernel function if the n × n matrix M such that M ij =

Original: K ( x i , x j ) (where x i , x j ∈ X ) is positive semi-deﬁnite.
Revised: K ( x i , x j ) (where x i , x j ∈ X ) is positive semi-deﬁnite.

Original: Since M is positive semi-deﬁnite, there exists a map ϕ from X to some Hilbert space H ,
Revised: Since M is positive semi-deﬁnite, there exists a map ϕ from X to some Hilbert space H ,

Original: such that all x, y ∈ X satisfy K ( x, y ) =
Revised: such that all x, y ∈ X satisfy K ( x, y ) =

Original: (cid:104) ϕ
Revised: (cid:104) ϕ

Original: ( x ) , ϕ ( y ) (cid:105) .
Revised: ( x ) , ϕ ( y ) (cid:105) .

Original: This above (existence of a map ϕ of into H ) can be extended to inﬁnite X , e.g., X = R d , by Mercer’s Theorem.
Revised: This above (existence of a map ϕ of into H ) can be extended to inﬁnite X , e.g., X = R d , by Mercer’s Theorem.

Original: The distance between x (cid:48) , y (cid:48) ∈ H is deﬁned as dist( x (cid:48) , y (cid:48) ) :
Revised: The distance between x (cid:48) , y (cid:48) ∈ H is deﬁned as dist( x (cid:48) , y (cid:48) ) :

Original: = (cid:107) x (cid:48) − y (cid:48) (cid:107) =
Revised: = (cid:107) x (cid:48) − y (cid:48) (cid:107) =

Original: (cid:112) (cid:104) x
Revised: (cid:112) [Equation]

Original: (cid:48) − y (cid:48) , x (cid:48) − y (cid:48) (cid:105) .

Original: Hence, the distance dist( ϕ ( x ) , ϕ ( y )) for x, y ∈ X can be represented using K as
Revised: Hence, the distance dist( ϕ ( x ) , ϕ ( y )) for x, y ∈ X can be represented using K as

Original: [Equation]
Revised: [Equation]

Original: We refer to a survey by Ghojogh et al.
Revised: We refer to a survey by Ghojogh et al.

Original: for a more comprehensive introduction to kernel functions.
Revised: for a more comprehensive introduction to kernel functions.

Original: Kernel ( k, z ) -C LUSTERING .
Revised: Kernel ( k, z ) -C LUSTERING .

Original: In the the kernel ( k, z ) -C LUSTERING problem, the input is a weighted data set X of n objects, a kernel function K : X × X
Revised: In the the kernel ( k, z ) -C LUSTERING problem, the input is a weighted data set X of n objects, a kernel function K : X × X

Original: → R + , an integer k ≥ 1 , and z > 0 .
Revised: → R + , an integer k ≥ 1 , and z > 0 .

Original: The goal is to ﬁnd a k -point center set C ⊆ H that minimizes the objective 
Revised: The goal is to ﬁnd a k -point center set C ⊆ H that minimizes the objective cost

Original: [Equation]
Revised: [Equation]

Original: where H is an induced Hilbert space of K and ϕ :
Revised: where H is an induced Hilbert space of K and ϕ :

Original: X → H is its feature map, and dist( ϕ ( x ) , C ) :
Revised: X → H is its feature map, and dist( ϕ ( x ) , C ) :

Original: = min c ∈H dist( ϕ ( x ) , c ) = min c ∈H (cid:107) ϕ
Revised: = min c ∈H dist( ϕ ( x ) , c ) = min c ∈H (cid:107) ϕ

Original: ( x ) − c (cid:107) .
Revised: ( x ) − c (cid:107) .

Original: The case z = 2 clearly coincides with kernel kM EANS whose objective is (2).
Revised: The case z = 2 clearly coincides with kernel kM EANS whose objective is (2).

Original: The (non-kernel) ( k, z ) -C LUSTERING problem may be viewed as kernel ( k, z ) -C LUSTERING with kernel K ( x, y )
Revised: The (non-kernel) ( k, z ) -C LUSTERING problem may be viewed as kernel ( k, z ) -C LUSTERING with kernel K ( x, y )

Original: = (cid:104) x, y (cid:105) and identity feature map ϕ ( x ) = x .
Revised: = (cid:104) x, y (cid:105) and identity feature map ϕ ( x ) = x .

Original: While the feature map ϕ might not be unique, we show below that this kernel ( k, z ) -C
Revised: While the feature map ϕ might not be unique, we show below that this kernel ( k, z ) -C

Original: LUSTERING is well deﬁned, in the sense that the optimal value is independent of ϕ .
Revised: LUSTERING is well deﬁned, in the sense that the optimal value is independent of ϕ .

Original: The following two facts are standard and easy to prove.
Revised: The following two facts are standard and easy to prove.

Original: Fact 2.1.
Revised: Fact 2.1.

Original: For every map ϕ into H , there is an optimal solution C (cid:63) in which every center point c ∈ C (cid:63) lies inside span( ϕ ( X )) , and is thus a linear combination of ϕ ( X ) .
Revised: For every map ϕ into H , there is an optimal solution C (cid:63) in which every center point c ∈ C (cid:63) lies inside span( ϕ ( X )) , and is thus a linear combination of ϕ ( X ) .

Original: Corollary 2.2.
Revised: Corollary 2.2.

Original: The optimal value of (4) can be represented as a function of kernel values K ( x, y ) , and is thus invariant of ϕ .
Revised: The optimal value of (4) can be represented as a function of kernel values K ( x, y ) , and is thus invariant of ϕ .

Original: (cid:15) -Coresets for kernel ( k, z ) -C LUSTERING .
Revised: (cid:15) -Coresets for kernel ( k, z ) -C LUSTERING .

Original: For 0 < (cid:15) < 1 , an (cid:15) -coreset for kernel ( k, z ) -
Revised: For 0 < (cid:15) < 1 , an (cid:15) -coreset for kernel ( k, z ) -

Original: C LUSTERING on a weighted dataset X and a kernel function K is a reweighted subset S ⊆ X , such that for every Hilbert space H and map ϕ : X → H satisfying (1), we have
Revised: C LUSTERING on a weighted dataset X and a kernel function K is a reweighted subset S ⊆ X , such that for every Hilbert space H and map ϕ : X → H satisfying (1), we have

Original: [Equation]
Revised: [Equation]

Original: The case z = 2 clearly coincides with (3).
Revised: The case z = 2 clearly coincides with (3).

Original:  C ORESETS FOR K ERNEL ( k, z ) -C LUSTERING
Revised:  C ORESETS FOR K ERNEL ( k, z ) -C LUSTERING

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Given n -point weighted dataset X , oracle access to a kernel K : X × X → R + , z ≥ 1 , integer k ≥ 1 , and 0 < (cid:15) < 1 , one can construct in time ˜ O
Revised: Given n -point weighted dataset X , oracle access to a kernel K : X × X → R + , z ≥ 1 , integer k ≥ 1 , and 0 < (cid:15) < 1 , one can construct in time ˜ O

Original: ( nk ) ,
Revised: ( nk ) ,

Original: a reweighted subset S ⊆ X of size (cid:107) S (cid:107) 

Revised: a reweighted subset S ⊆ X of size (cid:107) S (cid:107)

Original: = 2 O ( z ) · poly( k(cid:15) − 1 ) , that with high constant probability is an (cid:15) -coreset for kernel ( k, z )
Revised: = 2 O ( z ) · poly( k(cid:15) − 1 ) , that with high constant probability is an (cid:15) -coreset for kernel ( k, z )

Original: -C LUSTERING with respect to X and K .
Revised: -C LUSTERING with respect to X and K .

Original: At a high level, our prove this theorem by employing recent constructions of coresets for ( k, z ) -
Revised: At a high level, our prove this theorem by employing recent constructions of coresets for ( k, z ) -

Original: C LUSTERING in Euclidean spaces, in which the coreset size is independent of the Euclidean dimension (Sohler & Woodruff, 2018; Feldman et al., 2020; Huang & Vishnoi, 2020; Braverman et al., 2021).
Revised: C LUSTERING in Euclidean spaces, in which the coreset size is independent of the Euclidean dimension (Sohler & Woodruff, 2018; Feldman et al., 2020; Huang & Vishnoi, 2020; Braverman et al., 2021).

Original: However, these coresets are designed for ﬁnite-dimensional Euclidean spaces, and are thus not directly applicable to our feature space H , which might have inﬁnite dimension.
Revised: However, these coresets are designed for ﬁnite-dimensional Euclidean spaces, and are thus not directly applicable to our feature space H , which might have inﬁnite dimension.

Original: To employ these coreset constructions, we show in Lemma 3.2 that the data points in the feature space H embed into an ( n + 1) -dimensional
Revised: To employ these coreset constructions, we show in Lemma 3.2 that the data points in the feature space H embed into an ( n + 1) -dimensional

Original: (Euclidean) space, without any distortion to distances between data points and centers.
Revised: (Euclidean) space, without any distortion to distances between data points and centers.

Original: This observation is similar to one previously made by Sohler &
Revised: This observation is similar to one previously made by Sohler &

Original: Woodruff (2018) for a different purpose.
Revised: Woodruff (2018) for a different purpose.

Original: Due to this embedding, it sufﬁces to construct a coreset for the limited setting where centers come only from an ( n + 1) -dimensional space (Corollary 3.3).
Revised: Due to this embedding, it sufﬁces to construct a coreset for the limited setting where centers come only from an ( n + 1) -dimensional space (Corollary 3.3).

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: Let H be a Hilbert space and let X ⊆
Revised: Let H be a Hilbert space and let X ⊆

Original: H be a subset of n points.
Revised: H be a subset of n points.

Original: Then there exists a map f : H → R n +1 such that
Revised: Then there exists a map f : H → R n +1 such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let S = span( X ) .
Revised: Let S = span( X ) .

Original: Then every point c ∈ H can be written (uniquely) as c = c (cid:107) + c ⊥ , where c (cid:107) ∈ S and c ⊥ is orthogonal to S .
Revised: Then every point c ∈ H can be written (uniquely) as c = c (cid:107) + c ⊥ , where c (cid:107) ∈ S and c ⊥ is orthogonal to S .

Original: Thus, (cid:107) c (cid:107) 

Revised: Thus, (cid:107) c (cid:107)

Original: = (cid:107) c (cid:107) (cid:107) 

Revised: = (cid:107) c (cid:107) (cid:107)

Original: + (cid:107) c ⊥ (cid:107) 2 .
Revised: + (cid:107) c ⊥ (cid:107) 2 .

Original: Note that for all x ∈ X , we have x ⊥ = 0 .
Revised: Note that for all x ∈ X , we have x ⊥ = 0 .

Original: Now, for every c ∈ H , let f ( c ) :
Revised: Now, for every c ∈ H , let f ( c ) :

Original: = ( c (cid:107) ; (cid:107) c ⊥ (cid:107) ) , where we interpret x (cid:107) as an n -dimensional vector.
Revised: = ( c (cid:107) ; (cid:107) c ⊥ (cid:107) ) , where we interpret x (cid:107) as an n -dimensional vector.

Original: Then for all x ∈ X and c ∈ H ,
Revised: Then for all x ∈ X and c ∈ H ,

Original: [Equation]
Revised: [Equation]

Original: ) − f ( c ) (cid:107) 2 .

Original: The claim follows.
Revised: The claim follows.

Original: Corollary 3.3.
Revised: Corollary 3.3.

Original: Consider n -point weighted dataset X , kernel function K : X × X → R + , z ≥ 1 , integer k ≥ 1 , and 0 < (cid:15) < 1 .
Revised: Consider n -point weighted dataset X , kernel function K : X × X → R + , z ≥ 1 , integer k ≥ 1 , and 0 < (cid:15) < 1 .

Original: Suppose that a reweighted subset S ⊆ X satisﬁes that for every ϕ : X → R n +1 such that for all x, y ∈ X , (cid:104) ϕ
Revised: Suppose that a reweighted subset S ⊆ X satisﬁes that for every ϕ : X → R n +1 such that for all x, y ∈ X , (cid:104) ϕ

Original: ( x ) , ϕ ( y ) (cid:105) =
Revised: ( x ) , ϕ ( y ) (cid:105) =

Original: K ( x, y ) , the following holds
Revised: K ( x, y ) , the following holds

Original: [Equation]
Revised: [Equation]

Original: Then S is an (cid:15) -coreset for kernel ( k, z )
Revised: Then S is an (cid:15) -coreset for kernel ( k, z )

Original: -C LUSTERING with respect to X and kernel K .
Revised: -C LUSTERING with respect to X and kernel K .

Original: Proof.
Revised: Proof.

Original: To verify that S is a coreset with respect to X and K , consider some feature space H (cid:48) and feature map ϕ (cid:48) be induced by K .
Revised: To verify that S is a coreset with respect to X and K , consider some feature space H (cid:48) and feature map ϕ (cid:48) be induced by K .

Original: Apply Lemma 3.2 to obtain f : H (cid:48)
Revised: Apply Lemma 3.2 to obtain f : H (cid:48)

Original: → R n +1 , then for all C ⊆ H (cid:48) , | C |
Revised: → R n +1 , then for all C ⊆ H (cid:48) , | C |

Original: =
Revised: =

Original: k , we have ∀ [Equation] X [Equation]
Revised: k , we have ∀ Q ⊆ X, cost ϕ (cid:48)

Original: [Equation]
Revised: z ( Q, C ) = cost f ◦ ϕ (cid:48) z ( Q, f ( C )) , and using the promise about S with ϕ = f ◦ ϕ (cid:48) , cost ϕ (cid:48) z ( S, C ) = cost ϕz

Original: and using the promise about S with ϕ = f ◦ ϕ (cid:48),
Revised: ( S, f ( C )) ∈ (1 ± (cid:15) ) · cost ϕ ( X, f ( C )) = (1 ± (cid:15) ) · cost ϕ (cid:48)

Original: [Equation]
Revised: ( X, C ) .

Original: Thus, S is indeed a coreset with respect to X and K .
Revised: Thus, S is indeed a coreset with respect to X and K .

Revised: 1: let X 0 ← X , i ←2: repeat 3: let i ← i + 1 and (cid:15) i ← (cid:15)/ (log ( i ) (cid:107) X (cid:107) 0 ) 1 /4:

Revised: X i ← I MPORTANCE -S

Revised: AMPLING ( X i − 1 , (cid:15) i ) 5: until (cid:107) X i (cid:107) 0 does not decrease compared with (cid:107) X i − 1 (cid:107)6:

Revised: return X

Original:  Another issue is that some of the existing algorithms, such as Sohler & Woodruff (2018); Feldman et al.
Revised: i Another issue is that some of the existing algorithms, such as Sohler & Woodruff (2018); Feldman et al.

Original: (2020), require the input to be an explicit representations of points in ϕ ( X ) , which is very expensive to compute.
Revised: (2020), require the input to be an explicit representations of points in ϕ ( X ) , which is very expensive to compute.

Original: Fortunately, the importane-sampling based algorithms of Huang & Vishnoi (2020) and Braverman et al.
Revised: Fortunately, the importane-sampling-based algorithms of Huang & Vishnoi (2020) and Braverman et al.

Original: (2021) are oblivious to the representation of ϕ , and only rely on a distance oracle that evaluates (cid:107)
Revised: (2021) are oblivious to the representation of ϕ , and only rely on a distance oracle that evaluates (cid:107)

Original: ϕ
Revised: ϕ

Original: ( x ) − ϕ ( y ) (cid:107) =
Revised: ( x ) − ϕ ( y ) (cid:107) =

Original: (cid:112) K ( x, x ) + K
Revised: (cid:112) K ( x, x ) + K

Original: ( y, y )
Revised: ( y, y )

Original: − 2 K ( x, y ) for data points x, y ∈ X .
Revised: − 2 K ( x, y ) for data points x, y ∈ X .

Original: Now, by Corollary 3.3, executing these algorithms without any modiﬁcations (except for plugging in the distance oracle deﬁned by kernel K ) yields the desired coreset for kernel ( k, z ) -
Revised: Now, by Corollary 3.3, executing these algorithms without any modiﬁcations (except for plugging in the distance oracle deﬁned by kernel K ) yields the desired coreset for kernel ( k, z ) -

Original: C LUSTERING .
Revised: C LUSTERING .

Original: We choose to use the coreset construction of Braverman et al.
Revised: We choose to use the coreset construction of Braverman et al.

Original: (2021), which is arguably simpler.
Revised: (2021), which is arguably simpler.

Original: We now recall its statement for completeness.
Revised: We now recall its statement for completeness.

Original: Theorem 3.4 (Braverman et al.
Revised: Theorem 3.4 (Braverman et al.

Original: Given n -point weighted dataset X ⊂ R m for some integer m , together with z ≥ 1 , integer k ≥ 1 , and 0 < (cid:15) < 1 , one can construct in time
Revised: Given n -point weighted dataset X ⊂ R m for some integer m , together with z ≥ 1 , integer k ≥ 1 , and 0 < (cid:15) < 1 , one can construct in time

Original: ˜ O
Revised: ˜ O

Original: ( nk ) , a reweighted subset S ⊆ X of size (cid:107) S (cid:107) 

Revised: ( nk ) , a reweighted subset S ⊆ X of size (cid:107) S (cid:107)

Original: = ˜
Revised: = ˜

Original: O ( (cid:15) − 4 2 2 z k 2 ) , that with high constant probability is an (cid:15) -coreset for ( k, z ) -C LUSTERING with respect to X .
Revised: O ( (cid:15) − 4 2 2 z k 2 ) , that with high constant probability is an (cid:15) -coreset for ( k, z ) -C LUSTERING with respect to X .

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: It follows immediately by combining Corollary 3.3 and Theorem 3.4.
Revised: It follows immediately by combining Corollary 3.3 and Theorem 3.4.

Original:  D ESCRIPTION OF C ORESET C ONSTRUCTION A LGORITHMS
Revised: 3. D ESCRIPTION OF C ORESET C ONSTRUCTION A LGORITHMS

Original: Next, we present our full algorithm in Algorithm 1 (which depends on the subroutines deﬁned in
Revised: Next, we present our full algorithm in Algorithm 1 (which depends on the subroutines deﬁned in

Original: Algorithm 2 and 3).
Revised: Algorithm 2 and 3).

Original: While it essentially tailors previous work to our kernel clustering setting, we provide full details for completeness.
Revised: While it essentially tailors previous work to our kernel clustering setting, we provide full details for completeness.

Original: The following notation is needed.
Revised: The following notation is needed.

Original: For a subset C ⊆ H and data point x ∈ X , deﬁne NN C ( x ) :
Revised: For a subset C ⊆ H and data point x ∈ X , deﬁne NN C ( x ) :

Original: = arg min { dist( ϕ ( x ) , y ) : y ∈ C } as the nearest neighbor of x in C with respect to the distances in the feature space (breaking ties arbitrarily).
Revised: = arg min { dist( ϕ ( x ) , y ) : y ∈ C } as the nearest neighbor of x in C with respect to the distances in the feature space (breaking ties arbitrarily).

Original: Thus NN C ( · ) deﬁnes a | C | -partition of X , and the cluster that x belongs to (with respect to C ) is denoted C ( x ) :
Revised: Thus NN C ( · ) deﬁnes a | C | -partition of X , and the cluster that x belongs to (with respect to C ) is denoted C ( x ) :

Original: = { x (cid:48) ∈
Revised: = { x (cid:48) ∈

Original: X : NN C ( x (cid:48) )
Revised: X : NN C ( x (cid:48) )

Original: =
Revised: =

Original: NN C ( x ) } .
Revised: NN C ( x ) } .

Original: Algorithm 1 is the main procedure for constructing the coreset, and its loop iteratively executes another importance-sampling-based coreset construction (Algorithm 2).
Revised: Algorithm 1 is the main procedure for constructing the coreset, and its loop iteratively executes another importance-sampling-based coreset construction (Algorithm 2).

Original: Informally, each invocation of I MPORTANCE -S
Revised: Informally, each invocation of I MPORTANCE -S

Original: AMPLING constructs a coreset
Revised: AMPLING constructs a coreset

Original: X i from the current coreset
Revised: X i from the current coreset

Original: X i − 1 , to reduce the number of distinct elements in X i − 1 to roughly log (cid:107)
Revised: X i − 1 , to reduce the number of distinct elements in X i − 1 to roughly log (cid:107)

Original: X
Revised: X

Original: i − 1 (cid:107) 0 .
Revised: i − 1 (cid:107) 0 .

Original: The procedure ends when such size reduction cannot be done any more, at which point the size of the coreset reaches the bound in
Revised: The procedure ends when such size reduction cannot be done any more, at which point the size of the coreset reaches the bound in

Original: Theorem 3.4, which is independent of n .
Revised: Theorem 3.4, which is independent of n .

Original: K 1: let X 0 ← X , i ←2: repeat 3: let i ← i +4: let (cid:15) i ← (cid:15)/ (log ( i ) (cid:107) X (cid:107) 0 )

Original: 1 /5: let X i ← I MPORTANCE -S

Original: AMPLING ( X i − 1 , (cid:15) i ) 6: until (cid:107) X i (cid:107) 0 does not decrease compared with (cid:107) X i − 1 (cid:107)7: return X

Original: i In fact, subroutine I MPORTANCE -S
Revised: In fact, subroutine I MPORTANCE -S

Original: AMPLING already constructs a coreset, albeit it is of a worse size that depends on log (cid:107) X (cid:107) 0 .
Revised: AMPLING already constructs a coreset, albeit it is of a worse size that depends on log (cid:107) X (cid:107) 0 .

Original: This subroutine is based on the well-known importance-sampling approach that was proposed and improved in a series of works (cf.
Revised: This subroutine is based on the well-known importance sampling approach that was proposed and improved in a series of works (cf.

Original: Langberg & Schulman (2010);
Revised: Langberg & Schulman (2010);

Original: Feldman & Langberg (2011); Feldman et al.
Revised: Feldman & Langberg (2011); Feldman et al.

Original: Its ﬁrst step is to compute an importance score σ x for every data point x ∈ X , and then draw independent samples from X with probability proportional to σ x .
Revised: Its ﬁrst step is to compute an importance score σ x for every data point x ∈ X (lines 1–2), and then draw independent samples from X with probability proportional to σ x (lines 3–4).

Revised: The ﬁnal coreset is formed by reweighting the sampled points (lines 5–6).

Revised: Roughly speaking, the importance score σ x measures the relative contribution of x to the objective function in the worst-case, which here means the maximum over all choices of the center set.

Original: The importance score σ x is computed using an O (log k ) -approximate solution for kernel ( k, z ) 
Revised: It can be computed from an O (log k ) -approximate solution for kernel ( k, z ) -

Original: -C LUSTERING on X , which is generated by the D z -sampling subroutine (Algorithm 3), a natural generalization of the D 2 -sampling introduced by Arthur & Vassilvitskii (2007) .
Revised: C LUSTERING on X , which is generated by the D z -sampling subroutine (Algorithm 3), a natural generalization of the D 2 -sampling introduced by Arthur & Vassilvitskii (2007).

Revised: We stress that our algorithm description uses the feature vectors ϕ ( x ) for clarity of exposition.

Original: We stress that all these procedures can be easily implemented using the kernel trick, and the total time (and number of accesses to the kernel function K ) is only
Revised: These vectors do not have to be provided explicitly, because only distances between them are required, and thus all steps can be easily implemented using the kernel trick, and the total time (and number of accesses to the kernel function K ) is only

Original: ˜ O
Revised: ˜ O

Original: ( nk ) .
Revised: ( nk ) .

Original: 1: let C (cid:63) ← D
Revised: 1: let C (cid:63) ← D

Original: z -S AMPLING ( X ) 2: for each x ∈ X , let σ x ← w X ( x ) · (cid:16) (dist( x,C (cid:63) ))
Revised: z -S AMPLING ( X ) 2: for each x ∈ X , let σ x ← w X ( x ) · (cid:16) (dist( x,C (cid:63) ))

Original: z cost ϕz
Revised: z cost ϕz

Original: ( X,C (cid:63) )
Revised: ( X,C (cid:63) )

Original: + 1 w X ( C (cid:63) ( x )) (cid:17) 3: for each x ∈ X , let p x ← σ x (cid:80)
Revised: + 1 w X ( C (cid:63) ( x )) (cid:17) 3: for each x ∈ X , let p x ← σ x (cid:80)

Original: y ∈
Revised: y ∈

Original: X σ y 4: draw N =
Revised: X σ y 4: draw N =

Original: O ( (cid:15) − 4 2 

Revised: O ( (cid:15) − 4 2

Original: z zk 2 log 2 k log (cid:107) X (cid:107) 0 )
Revised: z zk 2 log 2 k log (cid:107) X (cid:107) 0 )

Original: i.i.d.
Revised: i.i.d.

Original: samples from X , using probabilities ( p x ) x ∈ X 5: let D be the sampled set, and for each x ∈ D let w D ( x ) ← w X ( x ) p x N 6: return weighted set Dpoint do 3:
Revised: samples from X , using probabilities ( p x ) x ∈ X 5: let D be the sampled set, and for each x ∈ D let w D ( x ) ← w X ( x ) p x N 6: return weighted set D

Original:  ( x ) ,C ) 
Revised: (cid:46) the feature vectors ϕ ( · ) are mentioned for clarity and not needed for implementation 1: let x be a uniform random point from X , and initialize C ← { ϕ ( x ) } 2: for i = 1 , . . . , k − 1 do 3: draw one sample x ∈ X , using probabilities w X ( x ) · (dist( ϕ ( x ) ,C )) z cost ϕz

Original: ,C ) 4:  C ← C ∪ { ϕ ( x ) } 5: end for 6: return C
Revised: ( X,C ) 4: let C ← C ∪ { ϕ ( x ) } 5: end for 6: return C

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We validate the empirical performance of our coreset for kernel k -M EANS against various datasets, and show that our coresets can signiﬁcantly speedup a kernelized version of the widely used k -M EANS ++ (Arthur & Vassilvitskii, 2007).
Revised: We validate the empirical performance of our coreset for kernel k -M EANS against various datasets, and show that our coresets can signiﬁcantly speedup a kernelized version of the widely used k -M EANS ++ (Arthur & Vassilvitskii, 2007).

Original: In addition, we apply this new coreset-based kernelized k -M EANS ++ to spectral clustering (via a reduction devised by Dhillon et al.
Revised: In addition, we apply this new coreset-based kernelized k -M EANS ++ to spectral clustering (via a reduction devised by Dhillon et al.

Original: , showing that it outperforms the well-known Scikit-learn solver in both running time and objective value.
Revised: , showing that it outperforms the well-known Scikit-learn solver in both running time and objective value.

Original: Experimental setup.
Revised: Experimental setup.

Original: Our experiments are conducted on standard clustering datasets that consist of vectors in R d , and we use the RBF kernel (radial basis function kernel, also known as
Revised: Our experiments are conducted on standard clustering datasets that consist of vectors in R d , and we use the RBF kernel (radial basis function kernel, also known as

Original: Gaussian kernel) and polynomial kernels as kernel functions.
Revised: Gaussian kernel) and polynomial kernels as kernel functions.

Original: An RBF kernel K G is of the form
Revised: An RBF kernel K G is of the form

Original: K G ( x, y ) := exp (cid:16) − (cid:107) x − y (cid:107) 2 σ 2 (cid:17) , where σ > 0 is a parameter, and a polynomial kernel K P is of the form K P ( x, y ) :=
Revised: K G ( x, y ) := exp (cid:16) − (cid:107) x − y (cid:107) 2 σ 2 (cid:17) , where σ > 0 is a parameter, and a polynomial kernel K P is of the form K P ( x, y ) :=

Original: ( (cid:104) x, y (cid:105)
Revised: ( (cid:104) x, y (cid:105)

Original: + c ) d where c and d are parameters.
Revised: + c ) d where c and d are parameters.

Original: Table 1 summarizes the speciﬁcations of datasets and our choice of the parameters for the kernel function.
Revised: Table 1 summarizes the speciﬁcations of datasets and our choice of the parameters for the kernel function.

Original: We note that the parameters are dataset-dependent, and that for Twitter and Census1990 dataset we subsample to 10 5 points since otherwise it takes too long to run for some of our inefﬁcient baselines.
Revised: We note that the parameters are dataset-dependent, and that for Twitter and Census1990 dataset we subsample to 10 5 points since otherwise it takes too long to run for some of our inefﬁcient baselines.

Original: Unless otherwise speciﬁed, we use a typical value k = 5 for the number of clusters.
Revised: Unless otherwise speciﬁed, we use a typical value k = 5 for the number of clusters.

Original: All experiments are conducted on a PC with Intel Core i7 CPU and 16 GB memory, and algorithms are implemented using C++.[Table] σ = [Table]
Revised: All experiments are conducted on a PC with Intel Core i7 CPU and 16 GB memory, and algorithms are implemented using C++.

Original: =

Original: [Table]

Original: =

Original: [Table]

Original: σ =c = 0 , d =

Original: [Table]

Original: σ =c = 0 , d =

Original: [Table]

Original: ) σ =c = 0 , d =

Original: 4. S IZE AND E MPIRICAL E RROR T RADEOFF
Revised: 4. S IZE AND E MPIRICAL E RROR T RADEOFF

Original: Our ﬁrst experiment evaluates the empirical error versus coreset size.
Revised: Our ﬁrst experiment evaluates the empirical error versus coreset size.

Original: In our coreset implementation, we simplify the construction in Algorithm 1 by running the importance sampling step only once instead of running it iteratively, and it turns out this simpliﬁcation still achieves excellent performance.
Revised: In our coreset implementation, we simplify the construction in Algorithm 1 by running the importance sampling step only once instead of running it iteratively, and it turns out this simpliﬁcation still achieves excellent performance.

Original: As in many previous implementations, instead of setting (cid:15) and solving for the number of samples
Revised: As in many previous implementations, instead of setting (cid:15) and solving for the number of samples

Original: N in the I MPORTANCE -S
Revised: N in the I MPORTANCE -S

Original: AMPLING procedure (Algorithm 2), we simply set N as a parameter to directly control the coreset size.
Revised: AMPLING procedure (Algorithm 2), we simply set N as a parameter to directly control the coreset size.

Original: We construct the coreset with this N and evaluate its error by drawing 500 random center sets C (each consisting of k points) from the data set, and evaluate the maximum empirical error, deﬁned as
Revised: We construct the coreset with this N and evaluate its error by drawing 500 random center sets C (each consisting of k points) from the data set, and evaluate the maximum empirical error, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: This empirical error is measured similarly to the deﬁnition of coreset, except that it is performed on a sample of center sets.
Revised: This empirical error is measured similarly to the deﬁnition of coreset, except that it is performed on a sample of center sets.

Original: To make the measurement stable, the empirical error is evaluated independently 100 times and the average is reported. 
Revised: To make the measurement stable, the empirical error is evaluated independently 100 times and the average is reported. [Figure] kernel param. poly.

Original: The tradeoff of the coreset size versus empirical error is shown in Figure 1, where we also compare with a baseline that constructs the coreset using uniform sampling.{+kernel param. Twitter (Chan et al., 2018) σ =c = 0 , d = Census
+}
Revised: {+kernel param. Twitter (Chan et al., 2018) σ =c = 0 , d = Census
+}

Revised: (Meek et al., 1990)

Revised: σ =c = 0 , d =

Revised: Adult (Dua & Graff, 2017) σ =c = 0 , d =

Revised: Bank (Moro et al., 2014)

Revised: [Figure]

Revised: The tradeoff of the coreset size versus empirical error is shown in Figure 1, where we also compare with a baseline that constructs the coreset using uniform sampling.

Original: These experiments show that our coreset performs consistently well on all the datasets and kernels.
Revised: These experiments show that our coreset performs consistently well on all the datasets and kernels.

Original: Furthermore, our coreset admits a similar error curve regardless of dataset and kernel function – for example, one can get within 10% error using a coreset of only 1000 points, – which is perfectly justiﬁed by our theory that the size of (cid:15) -coreset only depends on (cid:15) and k .
Revised: Furthermore, our coreset admits a similar error curve regardless of dataset and kernel function – for example, one can get within 10% error using a coreset of only 1000 points, – which is perfectly justiﬁed by our theory that the size of (cid:15) -coreset only depends on (cid:15) and k .

Original: Comparing with the uniform-sampling baseline, our coreset generally has superior performance, especially when the coreset size is small.
Revised: Comparing with the uniform-sampling baseline, our coreset generally has superior performance, especially when the coreset size is small.

Original: We also observe that the uniform sampling suffers a larger variance compared with our coreset.[Figure]
Revised: We also observe that the uniform sampling suffers a larger variance compared with our coreset.

Original: Census1990 dataset

Original: 4. S PEEDING U P K ERNELIZED k -M EANS ++
Revised: 4. S PEEDING U P K ERNELIZED k -M EANS ++

Original: k -M EANS ++ (Arthur & Vassilvitskii, 2007) is a widely-used algorithm for k -M EANS , and it could easily be adopted to solve kernel k -M EANS by using the kernel trick, however as mentioned earlier this would take Ω( n 2 ) time.
Revised: k -M EANS ++ (Arthur & Vassilvitskii, 2007) is a widely-used algorithm for k -M EANS , and it could easily be adopted to solve kernel k -M EANS by using the kernel trick, however as mentioned earlier this would take Ω( n 2 ) time.

Original: We use our coreset to speedup this kernelized k -M EANS ++ algorithm, by ﬁrst computing the coreset and then running kernelized k -M EANS ++ on the coreest; this yields an implementation of kernelized k -M EANS ++ whose running time is near-linear (in n ).
Revised: We use our coreset to speedup this kernelized k -M EANS ++ algorithm, by ﬁrst computing the coreset and then running kernelized k -M EANS ++ on the coreest; this yields an implementation of kernelized k -M EANS ++ whose running time is near-linear (in n ).

Original: In Figure 2, we demonstrate the running time and the error achieved by kernelized k -M EANS ++ with and without coresets, experimented with varying coreset sizes.
Revised: In Figure 2, we demonstrate the running time and the error achieved by kernelized k -M EANS ++ with and without coresets, experimented with varying coreset sizes.

Original: We measure the relative error of our coreset-based kernelized k -M EANS ++ by comparing the objective value it achieves with that of vanilla (i.e., without coreset) kernelized k -M EANS ++.
Revised: We measure the relative error of our coreset-based kernelized k -M EANS ++ by comparing the objective value it achieves with that of vanilla (i.e., without coreset) kernelized k -M EANS ++.

Original: These experiments show that the error decreases signiﬁcantly as the coreset size increases, and it stabilizes around size N = 100 , achieving merely < 5% error.
Revised: These experiments show that the error decreases signiﬁcantly as the coreset size increases, and it stabilizes around size N = 100 , achieving merely < 5% error.

Original: Naturally, the running time of our coreset-based approach increases with the coreset size, but even the slowest one is still several orders of magnitude faster than vanilla kernelized k -M EANS ++.
Revised: Naturally, the running time of our coreset-based approach increases with the coreset size, but even the slowest one is still several orders of magnitude faster than vanilla kernelized k -M EANS ++.

Original: 4. S PEEDING U P S PECTRAL C LUSTERING
Revised: 4. S PEEDING U P S PECTRAL C LUSTERING

Original: In the spectral clustering problem, the input is a set of n objects X and an n × n similarity matrix
Revised: In the spectral clustering problem, the input is a set of n objects X and an n × n similarity matrix

Original: A that measures the similarity between every pair of elements in X , and the goal is to ﬁnd a k -
Revised: A that measures the similarity between every pair of elements in X , and the goal is to ﬁnd a k -

Original: [Figure]
Revised: [Figure]

Original:  i v
Revised: unn i n g t i m

Original: [Figure]
Revised: [Figure]

Original: B a s el o g o r i t h m i
Revised: [Figure]

Original:  c r
Revised: i c r

Original: unn
Revised: unn

Original: i n g t i m e ( s )
Revised: i n g t i m e ( s )

Original: RBF kernel 
Revised: Polynomial kernel [Figure] RBF

Original: [Figure]
Revised: [Figure]

Original: B a s el o g o r i t 
Revised: B a s e - 2 l o g o r i t h

Original: h m i
Revised: m i

Original: c r
Revised: c r

Original: unn
Revised: unn

Original: i n g t i m e (  s )
Revised: i n g t i m e ( m s )

Original: Polynomial kernel partition of X such that a certain objective function with respect to A is minimized.
Revised: Polynomial kernel partition of X such that a certain objective function with respect to A is minimized.

Original: Dhillon et al.
Revised: Dhillon et al.

Original: (2004) shows a way to write spectral clustering as a (weighted) kernel k -M EANS problem, which is eventually used to produce a spectral clustering.
Revised: (2004) shows a way to write spectral clustering as a (weighted) kernel k -M EANS problem, which is eventually used to produce a spectral clustering.

Original: Speciﬁcally, let D be an n × n diagonal matrix such that D ii
Revised: Speciﬁcally, let D be an n × n diagonal matrix such that D ii

Original: =
Revised: =

Original: (cid:80) j ∈ [ n ] A ij .
Revised: (cid:80) j ∈ [ n ] A ij .

Original: Then according to Dhillon et al.
Revised: Then according to Dhillon et al.

Original: (2004), spectral clustering can be written as a weighted kernel k -M EANS problem with weights w i :=
Revised: (2004), spectral clustering can be written as a weighted kernel k -M EANS problem with weights w i :=

Original: D ii and kernel function K :=
Revised: D ii and kernel function K :=

Original: D − 1 AD − 1 , provided that A is positive semideﬁnite (which could be viewed as a kernel).
Revised: D − 1 AD − 1 , provided that A is positive semideﬁnite (which could be viewed as a kernel).

Original: We use this reduction, and plug in the abovementioned coreset-based kernelized k -M EANS ++ as the solver for kernel k -M EANS .
Revised: We use this reduction, and plug in the abovementioned coreset-based kernelized k -M EANS ++ as the solver for kernel k -M EANS .

Original: We experiment on the subsampled Twitter dataset with varying number of points, and we use the polynomial and RBF kernels as the similarity matrix A .
Revised: We experiment on the subsampled Twitter dataset with varying number of points, and we use the polynomial and RBF kernels as the similarity matrix A .

Original: However, we would need Θ( n 2 ) time if we evaluate D ii naively.
Revised: However, we would need Θ( n 2 ) time if we evaluate D ii naively.

Original: To resolve this issue, we draw a uniform sample S from [ n ] , and use ˆ D ii := n | S | (cid:80)
Revised: To resolve this issue, we draw a uniform sample S from [ n ] , and use ˆ D ii := n | S | (cid:80)

Original: j ∈ S
Revised: j ∈ S

Original: A ij as an estimate for D ii .
Revised: A ij as an estimate for D ii .

Original: The accuracy of ˆ D is justiﬁed by previous work on kernel density estimation (Joshi et al., 2011, Theorem 5.2), and for our application we simply set | S | = 1000 , which achieves good accuracy.
Revised: The accuracy of ˆ D is justiﬁed by previous work on kernel density estimation (Joshi et al., 2011, Theorem 5.2), and for our application we simply set | S | = 1000 , which achieves good accuracy.

Original: We compare our implementation with the spectral clustering solver from the well-known Scikitlearn library (Pedregosa et al., 2011) as a baseline.
Revised: We compare our implementation with the spectral clustering solver from the well-known Scikitlearn library (Pedregosa et al., 2011) as a baseline.

Original: The experimental results, reported in Figure 3, show that our approach has more than 1000x of speedup already for moderately large datasets ( n = 40000 ).
Revised: The experimental results, reported in Figure 3, show that our approach has more than 1000x of speedup already for moderately large datasets ( n = 40000 ).

Original: This difference might be partially caused by efﬁciency issues of the Python language used for the Scikit-learn implementation (recall that our implementation is in C++), but we actually see that the asymptotic growth of our algorithm’s running time is also much better than that of the Scikitlearn baseline.
Revised: This difference might be partially caused by efﬁciency issues of the Python language used for the Scikit-learn implementation (recall that our implementation is in C++), but we actually see that the asymptotic growth of our algorithm’s running time is also much better than that of the Scikitlearn baseline.

Original: This suggests that our improvement in running time is fundamental, and not only due to the programming language.
Revised: This suggests that our improvement in running time is fundamental, and not only due to the programming language.

Original: We also observe that our approach yields better objective values than
Revised: We also observe that our approach yields better objective values than

Original: Scikit-learn.
Revised: Scikit-learn.

Original: One possible reason is that Scikit-learn might be conservative in using more iterations to gain better accuracy, because of the expensive computational cost that we do not suffer.[Figure]
Revised: One possible reason is that Scikit-learn might be conservative in using more iterations to gain better accuracy, because of the expensive computational cost that we do not suffer.