Original: Stability and Generalization of Kernel Clustering: From Single Kernel to Multiple Kernel
Revised: Stability and Generalization of Kernel Clustering: From Single Kernel to Multiple Kernel

Original: Abstract
Revised: Abstract

Original: Multiple kernel clustering (MKC) is an important research topic which has beenwidely studied for decades.
Revised: Multiple kernel clustering (MKC) is an important research topic which has beenwidely studied for decades.

Original: However, current methods still face two problems:inefficient when handling out-of-sample data points and lack of theoretical studyof the stability and generalization of clustering.
Revised: However, current methods still face two problems:inefficient when handling out-of-sample data points and lack of theoretical studyof the stability and generalization of clustering.

Original: In this paper, we propose a novelmethod that is able to efficiently compute the embedding of out-of-sample data witha solid generalization guarantee.
Revised: In this paper, we propose a novelmethod that is able to efficiently compute the embedding of out-of-sample data witha solid generalization guarantee.

Original: Specifically, we approximate the eigen functionsof the integral operator associated with the linear combination of base kernelfunctions to construct low-dimensional embeddings of out-of-sample points forefficient multiple kernel clustering.
Revised: Specifically, we approximate the eigen functionsof the integral operator associated with the linear combination of base kernelfunctions to construct low-dimensional embeddings of out-of-sample points forefficient multiple kernel clustering.

Original: In addition, we, for the first time, theoreticallystudy the stability of clustering algorithms and prove that the single-view version ofthe proposed method has uniform stability as O (cid:0)
Revised: In addition, we, for the first time, theoreticallystudy the stability of clustering algorithms and prove that the single-view version ofthe proposed method has uniform stability as O (cid:0)

Original: Kn − 3 / 2 (cid:1) and establish an upperbound of excess risk as (cid:101) O (cid:0)
Revised: Kn − 3 / 2 (cid:1) and establish an upperbound of excess risk as (cid:101) O (cid:0)

Original: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where K is the cluster number andn is the number of samples.
Revised: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where K is the cluster number andn is the number of samples.

Original: We then extend the theoretical results to multiplekernel scenario and find that the stability of MKC depends on kernel weights.
Revised: We then extend the theoretical results to multiplekernel scenario and find that the stability of MKC depends on kernel weights.

Original: As anexample, we apply our method to a novel MKC algorithm termed SimpleMKKMand derive the upper bound of its excess clustering risk, which is tighter thanthe current results.
Revised: As anexample, we apply our method to a novel MKC algorithm termed SimpleMKKMand derive the upper bound of its excess clustering risk, which is tighter thanthe current results.

Original: Extensive experimental results validate the effectiveness andefficiency of the proposed method.
Revised: Extensive experimental results validate the effectiveness andefficiency of the proposed method.

Original:  Introduction
Revised:  Introduction

Original: Multiple kernel clustering (MKC) [29] aims to learn an optimal kernel from a set of pre-specifiedkernel matrices for high-quality partition.
Revised: Multiple kernel clustering (MKC) [33] aims to learn an optimal kernel from a set of pre-specifiedkernel matrices for high-quality partition.

Original: These methods usually assume that the optimal kernelmatrix is a linear combination of the pre-specified base kernels.
Revised: These methods usually assume that the optimal kernelmatrix is a linear combination of the pre-specified base kernels.

Original: By jointly optimizing kernel weightsand a clustering indicator matrix, MKC improves the clustering performance of single-view kernelk -means (KKM).
Revised: By jointly optimizing kernel weightsand a clustering indicator matrix, MKC improves the clustering performance of single-view kernelk -means (KKM).

Original: In recent years, MKC algorithms have been extensively studied.
Revised: In recent years, MKC algorithms have been extensively studied.

Original: Among theseworks, Lu et al.
Revised: Among theseworks, Lu et al.

Original: [20] propose to change the MKC framework into the form of kernel alignment.
Revised: [23] propose to change the MKC framework into the form of kernel alignment.

Original: Gönen et al.
Revised: Gönen et al.

Original: [8] propose to employ a localized method to obtain more sample-specific informationfrom the data.
Revised: [9] propose to employ a localized method to obtain more sample-specific informationfrom the data.

Original: Recently, Liu et al.
Revised: Recently, Liu et al.

Original: [19] propose a parameter-free MKC algorithm that adopts anovel min-max optimization framework and achieves state-of-the-art performance on a wide rangeof applications.
Revised: [22] propose a parameter-free MKC algorithm that adopts anovel min-max optimization framework and achieves state-of-the-art performance on a wide rangeof applications.

Original: Although impressive improvements have been made by the existing methods, therestill lacks an effective and computational efficient mechanism to handle out-of-sample problems.
Revised: Although impressive improvements have been made by the existing methods, therestill lacks an effective and computational efficient mechanism to handle out-of-sample problems.

Original: This makes them inefficient in both space and computational consumption for large-scale unseensamples.
Revised: This makes them inefficient in both space and computational consumption for large-scale unseensamples.

Original: More specifically, since all existing methods need to conduct eigen-decomposition over thematrix consisting of all visible samples as an intermediate operation, the pre-extracted informationis blocked for the new coming samples.
Revised: More specifically, since all existing methods need to conduct eigen-decomposition over thematrix consisting of all visible samples as an intermediate operation, the pre-extracted informationis blocked for the new coming samples.

Original: This makes the algorithms computationally expensive tohandle large-scale datasets.
Revised: This makes the algorithms computationally expensive tohandle large-scale datasets.

Original: To improve  the ability of MKC in handling out-of-sample data points, we propose a general methodwith a solid generalization guarantee.
Revised: To improve kernel clustering scalability, methods such as Nyström [30] approximation and random Fourier feature (RFF) [3] are proposed. These two methods acquire the non-linear feature of samplesin real space through the approximating the kernel matrix. However, Nyström method can’t beimplemented on out-of-sample points directly.

Revised: RFF can fill this gap, but the dimension of the learned embedding is comparably large such that the subsequent clustering process is time consuming.

Revised: Moreseriously, because it’s difficult to bound the difference of the kernel weights before and after theapproximation, these two methods are rarely implemented on MKC algorithms.

Revised: In this paper, wepropose a general method with a solid generalization guarantee.

Original: More precisely, we find that the main complexity is causedby recomputing eigenvectors for the embedding of new coming samples.
Revised: More precisely, we find that the maincomplexity is caused by recomputing eigenvectors for the embedding of new coming samples.

Original: To this end, throughthe study of the integral operator associated with the kernel functions, we propose to perform an eigen-decomposition approximation [24] for MKC.
Revised: Tothis end, through the study of the integral operator associated with the kernel functions, we propose toperform an eigen-decomposition approximation [27] for MKC.

Original: Specifically, we learn eigen functions from theempirical optimal kernel generated from base kernels and then use these eigen functions to calculatethe approximation embedding for the out-of-sample data points.
Revised: Specifically, we learn eigen functionsfrom the empirical optimal kernel generated from base kernels and then use these eigen functions tocalculate the approximation embedding for the out-of-sample data points.

Original: Then, a standard k -means algorithmcan be performed on the learned embedding for clustering.
Revised: Then, a standard k -meansalgorithm can be performed on the learned embedding for clustering.

Original: To theoretically analyse our algorithm, we propose a novel stability evaluation criterion for multiplekernel clustering and establish the generalization bound accordingly.
Revised: To theoretically analyse our algorithm, we propose a novel stability evaluation criterion for multiplekernel clustering and establish the generalization bound accordingly.

Original: Algorithmic stability [2] is a keyproperty to estimate the generalization bound of learning algorithms.
Revised: Algorithmic stability [2] is a keyproperty to estimate the generalization bound of learning algorithms.

Original: The stability-based technique[5, 12] is widely used in the analysis of learning algorithms’ generalization ability.
Revised: The stability-based technique[6, 14] is widely used in the analysis of learning algorithms’ generalization ability.

Original: However, existingliterature lacks research on the stability of clustering algorithms.
Revised: However, existingliterature lacks research on the stability of clustering algorithms.

Original: Different from supervised learningwhich possesses labels as a deterministic standard, the risk function of clustering tasks is highlycorrelated with uncertain clustering centroids.
Revised: Different from supervised learningwhich possesses labels as a deterministic standard, the risk function of clustering tasks is highlycorrelated with uncertain clustering centroids.

Original: Therefore, it is hard to analyse the algorithm stabilityfor clustering.
Revised: Therefore, it is hard to analyse the algorithm stabilityfor clustering.

Original: To address the issue, we start from single-view kernel k -means with a pairwiselearning framework.
Revised: To address the issue, we start from single-view kernel k -means with a pairwiselearning framework.

Original: We show that the proposed method has uniform stability as O (cid:0)
Revised: We show that the proposed method has uniform stability as O (cid:0)

Original: Kn − 3 / 2 (cid:1) ,where K is the number of clusters and n is the number of samples.
Revised: Kn − 3 / 2 (cid:1) ,where K is the number of clusters and n is the number of samples.

Original: Consequently, the excessrisk can be upper bounded by (cid:101) O (cid:0)
Revised: Consequently, the excessrisk can be upper bounded by (cid:101) O (cid:0)

Original: Kn − 3 / 2 + n − 1 / 2 (cid:1) 1 .
Revised: Kn − 3 / 2 + n − 1 / 2 (cid:1) 1 .

Original: We then extend the generalization boundanalysis from single kernel clustering to multiple kernel clustering and prove that the stability of
Revised: We then extend the generalization boundanalysis from single kernel clustering to multiple kernel clustering and prove that the stability of

Original: MKC is related to the learned kernel weights.
Revised: MKC is related to the learned kernel weights.

Original: As an instantiation, we use the above results toanalyze the simple multiple kernel k -means (SimpleMKKM)[19] and derive a generalization boundas (cid:101) O (cid:0) ( m + 1)
Revised: As an instantiation, we use the above results toanalyze the simple multiple kernel k -means (SimpleMKKM)[22] and derive a generalization boundas (cid:101) O (cid:0) ( m + 1)

Original: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where m is the number of kernels.
Revised: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where m is the number of kernels.

Original: As far as we know, this isthe tightest bound for MKC algorithms.
Revised: As far as we know, this isthe tightest bound for MKC algorithms.

Original: Finally, we conduct numerical experiments to verify theeffectiveness and efficiency of the proposed method.
Revised: Finally, we conduct numerical experiments to verify theeffectiveness and efficiency of the proposed method.

Original: Our contributions are in three-fold and can be summarized as follows:
Revised: Our contributions are in three-fold and can be summarized as follows:

Original:  We propose a method to enable MKC algorithms to handle out-of-sample problems in an efficientway.2)
Revised: 1) We propose an efficient method to enable MKC algorithms to handle out-of-sample problems.2)

Original: We successfully apply algorithmic stability to derive the generalization bound of the clusteringalgorithm for the first time.
Revised: We successfully apply algorithmic stability to derive the generalization bound of the clusteringalgorithm for the first time.

Original: We study the stability of SimpleMKKM with our method and derive itsexcess risk bound.3) By comprehensive numerical experiments, the experimental results show that the proposedalgorithm is effective and efficient.
Revised: We study the stability of SimpleMKKM with our method and derive itsexcess risk bound.3) By comprehensive numerical experiments, the experimental results show that the proposedalgorithm is effective and efficient.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: Section 2 introduces the notations used in this paper.
Revised: Section 2 introduces the notations used in this paper.

Original: Section 3 discusses the related work.
Revised: Section 3 discusses the related work.

Original: Section 4 describes the proposed method.
Revised: Section 4 describes the proposed method.

Original: Section 5 analysesthe stability and generalization of KKM on single kernel, and then extends the proposed method tomultiple kernel clustering.
Revised: Section 5 analysesthe stability and generalization of KKM on single kernel, and then extends the proposed method tomultiple kernel clustering.

Original: The experimental results are reported in Section 6.
Revised: The experimental results are reported in Section 6.

Original: Finally, the paper isconcluded in Section 7.
Revised: Finally, the paper isconcluded in Section 7.

Original: All proofs are placed in the appendix due to the limited space.
Revised: All proofs are placed in the appendix due to the limited space.

Original:  Notations
Revised:  Notations

Original: To avoid confusion with mathematical notations, the uppercase and lowercase letters in bold denotematrices and vectors, respectively.
Revised: To avoid confusion with mathematical notations, the uppercase and lowercase letters in bold denotematrices and vectors, respectively.

Original: For example, A is a matrix, and a is a vector.
Revised: For example, A is a matrix, and a is a vector.

Original: The component ofthem is denoted by A ij or a i .
Revised: The component ofthem is denoted by A ij or a i .

Original: Let us denote with X the sample space and with P the correspondingdata distribution.
Revised: Let us denote with X the sample space and with P the correspondingdata distribution.

Original: The training dataset S =
Revised: The training dataset S =

Original: { x i } ni =1 ⊂ X is drawn i.i.d.
Revised: { x i } ni =1 ⊂ X is drawn i.i.d.

Original: from P .
Revised: from P .

Original: The empiricaldistribution P n is defined as P n ( x ) = 1 n if x ∈ S , otherwise 0.
Revised: The empiricaldistribution P n is defined as P n ( x ) = 1 n if x ∈ S , otherwise 0.

Original: We assume
Revised: We assume

Original: that kernel functionk ( · , · ) : X × X → R is a positive-definite and conjugate symmetric function.
Revised: that kernel functionk ( · , · ) : X × X → R is a positive-definite and conjugate symmetric function.

Original: From [1], we knowthat there exist a Hilbert space H and a feature mapping ϕ ( · ) :
Revised: From [1], we knowthat there exist a Hilbert space H and a feature mapping ϕ ( · ) :

Original: X → H such that ∀ x , x ′ ∈ X ,k ( x , x ′ )
Revised: X → H such that ∀ x , x ′ ∈ X ,k ( x , x ′ )

Original: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ H .
Revised: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ H .

Original: The kernel function k ( · , · ) used in this paper satisfies that ∀ x , x ′ ∈ X ,k
Revised: The kernel function k ( · , · ) used in this paper satisfies that ∀ x , x ′ ∈ X ,k

Original: ( x , x ) =
Revised: ( x , x ) =

Original: k ( x ′ , x ′ ) .
Revised: k ( x ′ , x ′ ) .

Original: We assume that for any x ∈ X , ∥ ϕ ( x ) ∥ ≤ 1 .
Revised: We assume that for any x ∈ X , ∥ ϕ ( x ) ∥ ≤ 1 .

Original: Let us denote with I s the s × sidentity matrix, and with ∥ · ∥ ∞ the infinity norm.
Revised: Let us denote with I s the s × sidentity matrix, and with ∥ · ∥ ∞ the infinity norm.

Original: For the convenience, the probability measure ρafter differential d is omitted, i.e. , d x is short for dρ ( x ) .
Revised: For the convenience, the probability measure ρafter differential d is omitted, i.e. , d x is short for dρ ( x ) .

Original:  Related Work
Revised:  Related Work

Original: In this section, we give a brief introduction to pairwise learning and algorithmic stability which aretwo main contents related to our proofs.
Revised: In this section, we give a brief introduction to pairwise learning and algorithmic stability which aretwo main contents related to our proofs.

Original: Pairwise Learning .
Revised: Pairwise Learning .

Original: Pairwise learning defines loss function with respect to sample pairs.
Revised: Pairwise learning defines loss function with respect to sample pairs.

Original: It hasbeen successfully applied to metric learning[27], bipartite ranking [26], AUC maximization [22],etc.
Revised: It hasbeen successfully applied to metric learning[31], bipartite ranking [29], AUC maximization [25],etc.

Original: Assume that { x 1 , · · · , x n } are n samples drawn independently from some probability measure.
Revised: Assume that { x 1 , · · · , x n } are n samples drawn independently from some probability measure.

Original: Suppose that the learning model is parameterized by w .
Revised: Suppose that the learning model is parameterized by w .

Original: The empirical behavior of the learning modelcan then be evaluated by the empirical risk as:
Revised: The empirical behavior of the learning modelcan then be evaluated by the empirical risk as:

Original: [Equation]
Revised: [Equation]

Original: where l denotes the loss function.
Revised: where l denotes the loss function.

Original: Accordingly, the expected risk can be defined as:
Revised: Accordingly, the expected risk can be defined as:

Original: [Equation]
Revised: [Equation]

Original: Among the researches of learning theory, Lei et al.
Revised: Among the researches of learning theory, Lei et al.

Original: [13] establish a sharper generalization bound ofpairwise learning, Li et al.
Revised: [15] establish a sharper generalization bound ofpairwise learning, Li et al.

Original: [15] utilize a pairwise learning framework to verify the generalizationability of clustering algorithms.
Revised: [17] utilize a pairwise learning framework to verify the generalizationability of clustering algorithms.

Original: In [14], Lei et al.
Revised: In [16], Lei et al.

Original: study the generalization of stochastic gradientdescent for pairwise learning.
Revised: study the generalization of stochastic gradientdescent for pairwise learning.

Original: In [3, 15], the authors learn the excess risk bound of clustering in theform of pairwise learning.
Revised: In [4, 17], the authors learn the excess risk bound of clustering in theform of pairwise learning.

Original: As mentioned above, in the centroid-based clustering algorithms, it isgenerally hard to investigate the stability of the samples at the boundary of two centroids.
Revised: As mentioned above, in the centroid-based clustering algorithms, it isgenerally hard to investigate the stability of the samples at the boundary of two centroids.

Original: To tacklethis problem, we adopt a pairwise learning framework in our analysis.
Revised: To tacklethis problem, we adopt a pairwise learning framework in our analysis.

Original: Algorithmic Stability .
Revised: Algorithmic Stability .

Original: The analysis of algorithmic stability is a powerful technique for deriving thegeneralization bound of stable learning algorithms.
Revised: The analysis of algorithmic stability is a powerful technique for deriving thegeneralization bound of stable learning algorithms.

Original: In the existing literature, uniform stability [2] isthe most widely used one but needs the strongest assumptions.
Revised: In the existing literature, uniform stability [2] isthe most widely used one but needs the strongest assumptions.

Original: A variety of stability notions havebeen proposed for different algorithms to widen the application range of stability.
Revised: A variety of stability notions havebeen proposed for different algorithms to widen the application range of stability.

Original: Kutin et al.
Revised: Kutin et al.

Original: [12]define strong and weak stability for the algorithms which are not uniformly stable.
Revised: [14]define strong and weak stability for the algorithms which are not uniformly stable.

Original: Locally elasticstability is proposed in [5] to capture the sensitivity of the loss function in the population sense butnot in the worst sense.
Revised: Locally elasticstability is proposed in [6] to capture the sensitivity of the loss function in the population sense butnot in the worst sense.

Original: Lei et al.
Revised: Lei et al.

Original: [13] provide a notation of uniform stability for pairwise learning.
Revised: [15] provide a notation of uniform stability for pairwise learning.

Original: Despite the fact that algorithmic stability has been successfully used in analyzing numerous learningalgorithms, no work applies it to clustering algorithms.
Revised: Despite the fact that algorithmic stability has been successfully used in analyzing numerous learningalgorithms, no work applies it to clustering algorithms.

Original: In this paper, we show that the clusteringalgorithm we studied is uniformly stable.
Revised: In this paper, we show that the clusteringalgorithm we studied is uniformly stable.

Original:  Framework
Revised:  Framework

Original: In this section, we first give a brief introduction to kernel k -means and multiple kernel clustering,then introduce the proposed approximation method.
Revised: In this section, we first give a brief introduction to kernel k -means and multiple kernel clustering,then introduce the proposed approximation method.

Original: 4.1 Kernel k -means and Multiple Kernel Clustering
Revised: 4.1 Kernel k -means and Multiple Kernel Clustering

Original: Given a training set S = { x i } ni =1 and a feature map ϕ ( · ) , kernel clustering algorithm aims tominimize the following objective w.r.t.
Revised: Given a training set S = { x i } ni =1 and a feature map ϕ ( · ) , kernel clustering algorithm aims tominimize the following objective w.r.t.

Original: clustering centroids:
Revised: clustering centroids:

Original: [Equation]
Revised: [Equation]

Original: where C = { c k } Kk =1 denotes the centroids of K clusters.
Revised: where C = { c k } Kk =1 denotes the centroids of K clusters.

Original: A clustering indicator matrix H ∈ R n × K is defined as follows: if x i ∈ C k , h ik = 1 / (cid:112) |C k | , otherwiseh ik = 0 , where |C k | denotes the number of samples belong to the k -th cluster.
Revised: A clustering indicator matrix H ∈ R n × K is defined as follows: if x i ∈ C k , h ik = 1 / (cid:112) |C k | , otherwiseh ik = 0 , where |C k | denotes the number of samples belong to the k -th cluster.

Original: Kernel k -means.
Revised: Kernel k -means.

Original: If ϕ ( · ) is defined by some single kernel function k ( · , · ) , then Eq.(1) can be rewrittenas:
Revised: If ϕ ( · ) is defined by some single kernel function k ( · , · ) , then Eq.(1) can be rewrittenas:

Original: [Equation]
Revised: [Equation]

Original: where K n is the empirical kernel matrix computed by the kernel function k and dataset S , i.e. ,
Revised: where K n is the empirical kernel matrix computed by the kernel function k and dataset S , i.e. ,

Original: K ij = k
Revised: K ij = k

Original: ( x i , x j ) .
Revised: ( x i , x j ) .

Original: The discrete constraint of H in Eq.(2) can be relaxed as: 1)
Revised: The discrete constraint of H in Eq.(2) can be relaxed as: 1)

Original: h
Revised: h

Original: ik can take any real number; 2) theorthogonal constraint still holds, i.e. , H ⊤ H
Revised: ik can take any real number; 2) theorthogonal constraint still holds, i.e. , H ⊤ H

Original: = I K .
Revised: = I K .

Original: Consequently, the solution to Eq.(2) are the K eigenvectors corresponding to the K largest eigenvalues of kernel matrix K n .
Revised: Consequently, the solution to Eq.(2) are the Keigenvectors corresponding to the K largest eigenvalues of kernel matrix K n .

Original: Therefore, H can beregarded as a new embedding for the training dataset, and standard k -means can then be performedon H for the final clustering.
Revised: Therefore, H can beregarded as a new embedding for the training dataset, and standard k -means can then be performedon H for the final clustering.

Original: To avoid confusion, in this paper, kernel k -means refers to the relaxedversion as mentioned above.
Revised: To avoid confusion, in this paper, kernel k -means refers to the relaxedversion as mentioned above.

Original: Multiple Kernel Clustering .
Revised: Multiple Kernel Clustering .

Original: In the multiple kernel setting, a sample has m ( m ≥ 2) featuremappings { ϕ p ( x ) } mp =1 associated with m base kernel functions { k p } mp =1 .
Revised: In the multiple kernel setting, a sample has m ( m ≥ 2) featuremappings { ϕ p ( x ) } mp =1 associated with m base kernel functions { k p } mp =1 .

Original: As the setting of multiplekernel learning [7], multiple kernel clustering (MKC) usually lets each kernel multiply by a weightα p ( p ∈ [ m ]) .
Revised: As the setting of multiplekernel learning [8], multiple kernel clustering (MKC) usually lets each kernel multiply by a weightα p ( p ∈ [ m ]) .

Original: These weights satisfy (cid:80) mp
Revised: These weights satisfy (cid:80) mp

Original: =1 α p = 1 and α p ∈ [0 , 1] .
Revised: =1 α p = 1 and α p ∈ [0 , 1] .

Original: Thus, the feature mapping of
Revised: Thus, the feature mapping of

Original: MKC can be represented as:
Revised: MKC can be represented as:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(2), Eq.(1) in multiple kernel setting can be reformed as:
Revised: Similar to Eq.(2), Eq.(1) in multiple kernel setting can be reformed as:

Original: [Equation]
Revised: [Equation]

Original: where K α = (cid:80) mp
Revised: where K α = (cid:80) mp

Original: =1 α 2 p K p .
Revised: =1 α 2 p K p .

Original: MKC aims to learn a set of optimal combination weights.
Revised: MKC aims to learn a set of optimal combination weights.

Original: Then, therelaxed kernel k -means is performed on K α for a unified embedding H .
Revised: Then, therelaxed kernel k -means is performed on K α for a unified embedding H .

Original: Notice that K α is the kernelmatrix of the training dataset.
Revised: Notice that K α is the kernelmatrix of the training dataset.

Original: To obtain the embedding of an out-of-sample point, we have to performeigen-decomposition on the kernel matrix constructed by the training dataset and the new comingdata point.
Revised: To obtain the embedding of an out-of-sample point, we have to performeigen-decomposition on the kernel matrix constructed by the training dataset and the new comingdata point.

Original: Obviously, this process is inefficient.
Revised: Obviously, this process is inefficient.

Original: 4.2 The Proposed Approximation Method
Revised: 4.2 The Proposed Approximation Method

Original: In our approach, we do not approximate the kernel matrix but the eigen functions of the integraloperator associated with the kernel function.
Revised: In our approach, we do not approximate the kernel matrix but the eigen functions of the integraloperator associated with the kernel function.

Original: First, we give a brief analysis of the spectrum ofthe integral operator.
Revised: First, we give a brief analysis of the spectrum ofthe integral operator.

Original: We start from the risk function of a sample and the expected risk which arerespectively defined as:
Revised: We start from the risk function of a sample and the expected risk which arerespectively defined as:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(2), we reform the risk function of MKC for any two samples x , y ∈ X as follows:
Revised: Similar to Eq.(2), we reform the risk function of MKC for any two samples x , y ∈ X as follows:

Original: [Equation]
Revised: [Equation]

Original: where k α ( · , · ) = (cid:80) mp
Revised: where k α ( · , · ) = (cid:80) mp

Original: =1 α 2 p k p ( · , · ) denotes the linear combination of base kernel functions, and
Revised: =1 α 2 p k p ( · , · ) denotes the linear combination of base kernel functions, and

Original: H = { h k ( x ) } Kk =1 denotes the clustering indicator functions which are defined as:
Revised: H = { h k ( x ) } Kk =1 denotes the clustering indicator functions which are defined as:

Original: [Equation]
Revised: [Equation]

Original: The expected clustering risk is then defined as:
Revised: The expected clustering risk is then defined as:

Original: [Equation]
Revised: [Equation]

Original: The following proposition shows that the expected risk of the proposed pairwise learning frameworkequals that of the original kernel clustering algorithm with the same cluster centroids.
Revised: The following proposition shows that the expected risk of the proposed pairwise learning frameworkequals that of the original kernel clustering algorithm with the same cluster centroids.

Original: Thus, we canlearn the excess risk bound with the proposed framework defined by Eq.(5) instead of the originalone defined by Eq.(4).
Revised: Thus, we canlearn the excess risk bound with the proposed framework defined by Eq.(5) instead of the originalone defined by Eq.(4).

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: The expected clustering risk defined by Eq.
Revised: The expected clustering risk defined by Eq.

Original: (7) can be reformed as:
Revised: (7) can be reformed as:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: Algorithm
Revised: Algorithm

Original: Out sample example x ; kernel weights { p } mp =1 , clustering indicator matrix H α  the largest K eigenvalues λ k, α } Kk =1 learned by some MKC on some training set; m kernel functions
Revised: Out sample example x ; kernel weights { p } mp =1 , clustering indicator matrix H α and the largest K eigenvalues λ k, α } Kk =1 learned by some MKC on some training set; m kernel functions

Original: Obviously, { h k ( x ) } Kk =1 are all unit functions and orthogonal to each other in the squared integrablespace L 2 ( X , ρ ) .
Revised: Obviously, { h k ( x ) } Kk =1 are all unit functions and orthogonal to each other in the squared integrablespace L 2 ( X , ρ ) .

Original: Similar to Eq.(2), the discrete constraint of these functions can also be relaxed, andthe relaxed version is investigated in this paper.
Revised: Similar to Eq.(2), the discrete constraint of these functions can also be relaxed, andthe relaxed version is investigated in this paper.

Original: Denote that H ∗ =
Revised: Denote that H ∗ =

Original: { h ∗ k ( x ) } Kk =1 are the K eigenfunctions corresponding to the K largest eigenvalues of the integral operator L k :
Revised: { h ∗ k ( x ) } Kk =1 are the K eigenfunctions corresponding to the K largest eigenvalues of the integral operator L k :

Original: L 2 ( X , ρ )
Revised: L 2 ( X , ρ )

Original: →
Revised: →

Original: L 2 ( X , ρ ) ,
Revised: L 2 ( X , ρ ) ,

Original: [Equation]
Revised: [Equation]

Original: Obviously, H ∗ is the minimizer of the expected clustering risk defined by Eq.(7).
Revised: Obviously, H ∗ is the minimizer of the expected clustering risk defined by Eq.(7).

Original: However, H ∗cannot be obtained because the data distribution is unknown.
Revised: However, H ∗cannot be obtained because the data distribution is unknown.

Original: Rosasco et al.
Revised: Rosasco et al.

Original: [24] analyse the spectralproperties of the integral operator defined by kernel function and its empirical version, i.e., kernelmatrix.
Revised: [27] analyse the spectralproperties of the integral operator defined by kernel function and its empirical version, i.e., kernelmatrix.

Original: Inspired by their method, we aim to construct clustering indicator functions from the empiricalkernel matrix that can approximate H ∗ .
Revised: Inspired by their method, we aim to construct clustering indicator functions from the empiricalkernel matrix that can approximate H ∗ .

Original: Suppose that the combination weights learned by some
Revised: Suppose that the combination weights learned by some

Original: MKC algorithms on S are { α p } mp =1 , the embedding of 1 n K α is H α and the largest K eigenvaluesare { λ k, α } Kk =1 .
Revised: MKC algorithms on S are { α p } mp =1 , the embedding of 1 n K α is H α and the largest K eigenvaluesare { λ k, α } Kk =1 .

Original: We construct K functions as follows:
Revised: We construct K functions as follows:

Original: [Equation]
Revised: [Equation]

Original: where k ranges in [ K ] and h ik, α is the ik -th element of H α .
Revised: where k ranges in [ K ] and h ik, α is the ik -th element of H α .

Original: Then, the approximation embedding foran out-of-sample point x can be constructed by a K -dimensional vector [˜ h 1 , α ( x ) , ..., ˜ h K, α ( x )] .
Revised: Then, the approximation embedding foran out-of-sample point x can be constructed by a K -dimensional vector [˜ h 1 , α ( x ) , ..., ˜ h K, α ( x )] .

Original: Theabove procedures are listed in Algorithm 1, and we term this algorithm as multiple kernel clusteringwith approximation embedding (MKC-AE).
Revised: Theabove procedures are listed in Algorithm 1, and we term this algorithm as multiple kernel clusteringwith approximation embedding (MKC-AE).

Original: This method has two noticeable advantages: 1) It can beapplied to all MKC algorithms based on embedding learning.
Revised: This method has two noticeable advantages: 1) It can beapplied to all MKC algorithms based on embedding learning.

Original: It has a theoretical generalizationguarantee which will be studied in the following section.
Revised: It has a theoretical generalizationguarantee which will be studied in the following section.

Original: The application on large-scale datasets.
Revised: The application on large-scale datasets.

Original: The proposed MKC-AE method can be adopted to perform
Revised: The proposed MKC-AE method can be adopted to perform

Original: MKC on large-scale datasets.
Revised: MKC on large-scale datasets.

Original: Let us denote with S N a large-scale dataset consisting of N samples.
Revised: Let us denote with S N a large-scale dataset consisting of N samples.

Original: We first select n samples as landmarks and perform some MKC algorithm on these landmarks.
Revised: We first select n samples as landmarks and perform some MKC algorithm on these landmarks.

Original: The remaining N − n samples can be regarded as the "out-of-sample" examples, and the approximation embedding of these samples can be obtained by following process.
Revised: The remaining N − n samples can be regarded as the "out-of-sample" examples, and the approximation embedding of these samples can be obtained by following process.

Original: Assume that the optimal kernel matrix of some MKC algorithm on n landmarks is 1 n K n, α .
Revised: Assume that the optimal kernel matrix of some MKC algorithm on n landmarks is 1 n K n, α .

Original: The matrix constructed by the K largest eigenvectors of 1 n K n, α is denoted as H n ∈ R n × K , and the corresponding eigenvalues matrix is
Revised: The matrix constructed by the K largest eigenvectors of 1 n K n, α is denoted as H n ∈ R n × K , and the corresponding eigenvalues matrix is

Original: D n ∈ R K × K .
Revised: D n ∈ R K × K .

Original: Assume that the partial kernel similarity constructed by N samples and n landmarks are { P l } ml =1 ⊂ R N × n , i.e. , [ P l ] ij = k l ( x i , x j ) , ∀ l ∈ [ m ] , i ∈
Revised: Assume that the partial kernel similarity constructed by N samples and n landmarks are { P l } ml =1 ⊂ R N × n , i.e. , [ P l ] ij = k l ( x i , x j ) , ∀ l ∈ [ m ] , i ∈

Original: [ N ] , j ∈ [ n ] .
Revised: [ N ] , j ∈ [ n ] .

Original: According to Eq.(8), the approximation embedding of S N can be computed as:
Revised: According to Eq.(8), the approximation embedding of S N can be computed as:

Original: [Equation]
Revised: [Equation]

Original: 1 n .
Revised: 1 n .

Original: Notice that the main complexity of computing the approximation embedding is O ( nmN + nKN ) ,which is caused by the summation of { P l } ml =1 and subsequent matrix multiplication.
Revised: Notice that the main complexity of computing the approximation embedding is O ( nmN + nKN ) ,which is caused by the summation of { P l } ml =1 and subsequent matrix multiplication.

Original: We then give anexample to show how the proposed MKC-AE method can accelerate MKC.
Revised: We then give anexample to show how the proposed MKC-AE method can accelerate MKC.

Original: The time complexityof the MKC algorithm termed SimpleMKKM proposed in [19] is basically cubic with the numberof training samples.
Revised: The time complexityof the MKC algorithm termed SimpleMKKM proposed in [22] is basically cubic with the numberof training samples.

Original: Performing SimpleMKKM on all N samples will cost O (cid:0) N 3 (cid:1) time, while theproposd MKC-AE method reduces the complexity to O (cid:0) n 3 + nmN + nKN (cid:1) .
Revised: Performing SimpleMKKM on all N samples will cost O (cid:0) N 3 (cid:1) time, while theproposd MKC-AE method reduces the complexity to O (cid:0) n 3 + nmN + nKN (cid:1) .

Original: When n ≪ N , thecomplexity is linear with the number of samples N .
Revised: When n ≪ N , thecomplexity is linear with the number of samples N .

Original: Thus, the proposed algorithm can enable MKCalgorithms to be performed on large-scale datasets.
Revised: Thus, the proposed algorithm can enable MKCalgorithms to be performed on large-scale datasets.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: As the foundation for the theoretical analysis of multiple kernel clustering, we first analyse the stabilityof kernel k -means (KKM) with our approximation method and then establish the generalizationbound.
Revised: As the foundation for the theoretical analysis of multiple kernel clustering, we first analyse the stabilityof kernel k -means (KKM) with our approximation method and then establish the generalizationbound.

Original: 5.1 Stability and Generalization of Clustering on Single Kernel
Revised: 5.1 Stability and Generalization of Clustering on Single Kernel

Original: Notice that there are two intractable problems in the analysis for the stability of kernel clustering:1)
Revised: Notice that there are two intractable problems in the analysis for the stability of kernel clustering:1)

Original: The clustering risk is concerned with centroids.
Revised: The clustering risk is concerned with centroids.

Original: However, the random initialization of clusteringcentroids makes the stability of kernel clustering hard to be analyzed; 2) To compute the clusteringrisk of a sample, it needs to find the nearest centroid.
Revised: However, the random initialization of clusteringcentroids makes the stability of kernel clustering hard to be analyzed; 2) To compute the clusteringrisk of a sample, it needs to find the nearest centroid.

Original: As a result, it is difficult to estimate the stabilityof the samples located at the boundary of two centroids.
Revised: As a result, it is difficult to estimate the stabilityof the samples located at the boundary of two centroids.

Original: In this paper, since the proposed methodadopts a pairwise risk function, the above two issues can be avoided.
Revised: In this paper, since the proposed methodadopts a pairwise risk function, the above two issues can be avoided.

Original: The detailed deduce process isas follows.
Revised: The detailed deduce process isas follows.

Original: In the single kernel setting, let us denote with 1 n K n the (normalized) empirical kernel matrixconstructed by the training set, and suppose that the K largest eigenvalues and the correspondingeigenvectors of 1 n K n are { λ k } Kk =1 and { h k } Kk =1 , respectively.
Revised: In the single kernel setting, let us denote with 1 n K n the (normalized) empirical kernel matrixconstructed by the training set, and suppose that the K largest eigenvalues and the correspondingeigenvectors of 1 n K n are { λ k } Kk =1 and { h k } Kk =1 , respectively.

Original: Then, the approximated clusteringindicator functions ˜ H
Revised: Then, the approximated clusteringindicator functions ˜ H

Original: =
Revised: =

Original: { ˜ h k } K k =1 are given by:
Revised: { ˜ h k } K k =1 are given by:

Original: [Equation]
Revised: [Equation]

Original: where h ik is the i -th component of h k .
Revised: where h ik is the i -th component of h k .

Original: We aim to bound the excess clustering risk as:
Revised: We aim to bound the excess clustering risk as:

Original: [Equation]
Revised: [Equation]

Original: By the following proposition, we know the second item C ≤ 0 in Eq.(10).
Revised: By the following proposition, we know the second item C ≤ 0 in Eq.(10).

Original: Proposition 5.1.
Revised: Proposition 5.1.

Original: In Eq.
Revised: In Eq.

Original: (10) , E S R ( ˜ H, P n )
Revised: (10) , E S R ( ˜ H, P n )

Original: ≤ R
Revised: ≤ R

Original: ( H ∗ , P ) .
Revised: ( H ∗ , P ) .

Original: To give the upper bound of B , we first prove that the approximation method given by Eq.(9) hasuniform stability O (cid:16) Kn √ n (cid:17) with respect to the risk function defined by Eq.(5) .
Revised: To give the upper bound of B , we first prove that the approximation method given by Eq.(9) hasuniform stability O (cid:16) Kn √ n (cid:17) with respect to the risk function defined by Eq.(5) .

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: Denote that two training datasets S, S i have n samples and differ by only one sample 2 .
Revised: Denote that two training datasets S, S i have n samples and differ by only one sample 2 .

Original: Suppose that the empirical kernel matrix has properties as: 1) there exist two constant c, c 1 such thatthe K -th eigenvalue is larger than 1 /c ; and 2) the gaps of first K + 1 eigenvalues are larger than c 1 ,i.e., ∀ k ∈
Revised: Suppose that the empirical kernel matrix has properties as: 1) there exist two constant c, c 1 such thatthe K -th eigenvalue is larger than 1 /c ; and 2) the gaps of first K + 1 eigenvalues are larger than c 1 ,i.e., ∀ k ∈

Original: [ K ] , λ k − λ k +1 ≥ c 1 ≥ 4 √ n .
Revised: [ K ] , λ k − λ k +1 ≥ c 1 ≥ 4 √ n .

Original: The approximation clustering indicator functions of S, S iobtained by Eq.
Revised: The approximation clustering indicator functions of S, S iobtained by Eq.

Original: (9) are denoted as ˜ H and ˜ H i , respectively.
Revised: (9) are denoted as ˜ H and ˜ H i , respectively.

Original: Then, for any x , y ∈ X , there exists aconstant c 0 > 0 such that
Revised: Then, for any x , y ∈ X , there exists aconstant c 0 > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Remark 1 .
Revised: Remark 1 .

Original: Suppose that the K -th eigenvalue of the integral operator associated with kernel functionk ( · , · ) is λ K .
Revised: Suppose that the K -th eigenvalue of the integral operator associated with kernel functionk ( · , · ) is λ K .

Original: Then, by Proposition 10 in [24], the K -th eigenvalue of the corresponding empiricalkernel matrix λ K converges to λ K when n approaches to infinity.
Revised: Then, by Proposition 10 in [27], the K -th eigenvalue of the corresponding empiricalkernel matrix λ K converges to λ K when n approaches to infinity.

Original: When kernel function is fixed,λ K can be considered as a constant.
Revised: When kernel function is fixed,λ K can be considered as a constant.

Original: Consider that a sequence of the K -th eigenvalue λ nK withdifferent sample number n .
Revised: Consider that a sequence of the K -th eigenvalue λ nK withdifferent sample number n .

Original: Let ϵ = λ K 2 , then there exists a positive integer N such that ∀ n ≥ N ,| λ nK − λ K | ≤ λ K 2 .
Revised: Let ϵ = λ K 2 , then there exists a positive integer N such that ∀ n ≥ N ,| λ nK − λ K | ≤ λ K 2 .

Original: We can let 1 /c
Revised: We can let 1 /c

Original: = min { λ 1 k , · · · , λ NK , λ K 2 } , then the assumption that λ K ≥ 1 /cholds.
Revised: = min { λ 1 k , · · · , λ NK , λ K 2 } , then the assumption that λ K ≥ 1 /cholds.

Original: Similarly, the assumption about eigen gap of kernel matrix is related to the eigen gap of theintegral operator corresponding to kernel function, thus appropriate kernel function can make theassumption hold.
Revised: Similarly, the assumption about eigen gap of kernel matrix is related to the eigen gap of theintegral operator corresponding to kernel function, thus appropriate kernel function can make theassumption hold.

Original: We conduct experiments to verify that the kernel functions we adopted satisfy theassumptions in Theorem 5.2.
Revised: We conduct experiments to verify that the kernel functions we adopted satisfy theassumptions in Theorem 5.2.

Original: The detailed experimental results are reported in Section B.4.
Revised: The detailed experimental results are reported in Section B.4.

Original: The following theorem [13] is about the generalization risk bound of the stable pairwise learning
Revised: The following theorem [15] is about the generalization risk bound of the stable pairwise learning

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: [13] Assume that algorithm A has γ -stability.
Revised: [15] Assume that algorithm A has γ -stability.

Original: If there exists a constant M > 0such that | E S
Revised: If there exists a constant M > 0such that | E S

Original: [ l ( x , y , A ( S ))] | < M holds for all x , y ∈ X .
Revised: [ l ( x , y , A ( S ))] | < M holds for all x , y ∈ X .

Original: Then for all δ ∈ (0 , 1 /e ) , the followinginequality holds with probability 1 − δ :
Revised: Then for all δ ∈ (0 , 1 /e ) , the followinginequality holds with probability 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that the proposed risk function defined by Eq.(5) can be upper bounded bysome constant M .
Revised: It is easy to check that the proposed risk function defined by Eq.(5) can be upper bounded bysome constant M .

Original: Then, combining Theorem 5.2 and Theorem 5.3, the proposed method has ageneralization risk bound B as ˜ O (cid:16) Kn √ n + 1 √ n (cid:17) .
Revised: Then, combining Theorem 5.2 and Theorem 5.3, the proposed method has ageneralization risk bound B as ˜ O (cid:16) Kn √ n + 1 √ n (cid:17) .

Original: As seen, the item ˜ O (cid:16) 1 √ n (cid:17) masters the wholebound.
Revised: As seen, the item ˜ O (cid:16) 1 √ n (cid:17) masters the wholebound.

Original: Notice that this is the first bound for clustering deduced by algorithmic stability, and we canuse it to learn the stability and generalization of multiple kernel clustering algorithms.
Revised: Notice that this is the first bound for clustering deduced by algorithmic stability, and we canuse it to learn the stability and generalization of multiple kernel clustering algorithms.

Original: In the future,we will improve this bound by further investigating the generalization theorem about the proposedpairwise learning framework.
Revised: In the future,we will improve this bound by further investigating the generalization theorem about the proposedpairwise learning framework.

Original: 5.2 Stability and Generalization of Clustering on Multiple Kernel
Revised: 5.2 Stability and Generalization of Clustering on Multiple Kernel

Original: To study the generalization ability of the proposed MKC-AE method on multiple kernel, we firstreveal the relation between the stability of MKC-AE and the learned kernel weights.
Revised: To study the generalization ability of the proposed MKC-AE method on multiple kernel, we firstreveal the relation between the stability of MKC-AE and the learned kernel weights.

Original: Theorem 5.4.
Revised: Theorem 5.4.

Original: Denote that two training datasets S, S i have n samples and differ by only one sample.
Revised: Denote that two training datasets S, S i have n samples and differ by only one sample.

Original: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same propertiesin Theorem 5.2.
Revised: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same propertiesin Theorem 5.2.

Original: If ∥ α − α i ∥ ∞ ≤ η , there exists a constant c 0 such that
Revised: If ∥ α − α i ∥ ∞ ≤ η , there exists a constant c 0 such that

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Remark 2 .
Revised: Remark 2 .

Original: Theorem 5.4 shows that if the kernel weights of an MKC algorithm are stable, then itsapproximation method proposed in Algorithm 1 has uniform stability.
Revised: Theorem 5.4 shows that if the kernel weights of an MKC algorithm are stable, then itsapproximation method proposed in Algorithm 1 has uniform stability.

Original: Specifically, if η in Theorem5.4 can be bounded by O (1 / √ n ) , it can be checked that the risk function has the stability in order of
Revised: Specifically, if η in Theorem5.4 can be bounded by O (1 / √ n ) , it can be checked that the risk function has the stability in order of

Original: O (cid:16) ( m +1) K n √ n (cid:17) .
Revised: O (cid:16) ( m +1) K n √ n (cid:17) .

Original: In this regard, for a tighter risk bound, the kernel weights of MKC algorithms shouldbe more stable.
Revised: In this regard, for a tighter risk bound, the kernel weights of MKC algorithms shouldbe more stable.

Original: This can be a design principle for MKC algorithms.
Revised: This can be a design principle for MKC algorithms.

Original: As an instantiation, we use the above theoretical results to analyse simple multiple kernel k -means(SimpleMKKM)
Revised: As an instantiation, we use the above theoretical results to analyse simple multiple kernel k -means(SimpleMKKM)

Original: [19], which is a state-of-the-art MKC algorithm without any hyper-parameters.
Revised: [22], which is a state-of-the-art MKC algorithm without any hyper-parameters.

Original: Wefirst give a brief introduction and then derive the generalization bound of SimpleMKKM with theproposed approximation method.
Revised: Wefirst give a brief introduction and then derive the generalization bound of SimpleMKKM with theproposed approximation method.

Original: In the task of unsupervised learning, finding the optimal hyper-parameters is an open question.
Revised: In the task of unsupervised learning, finding the optimal hyper-parameters is an open question.

Original: SimpleMKKM is proposed as a hyper-parameter-free MKC algorithm and achieves desirable clusteringperformance.
Revised: SimpleMKKM is proposed as a hyper-parameter-free MKC algorithm and achieves desirable clusteringperformance.

Original: The variants of SimpleMKKM [17, 18] further boost the clustering performance andwiden its application.
Revised: The variants of SimpleMKKM [19, 21] further boost the clustering performance andwiden its application.

Original: Specifically, SimpleMKKM aims to solve the following kernel alignment-basedoptimization problem:
Revised: Specifically, SimpleMKKM aims to solve the following kernel alignment-basedoptimization problem:

Original: [Equation]
Revised: [Equation]

Original: where F ( α ) =
Revised: where F ( α ) =

Original: max H 1 n Tr (cid:0) K α HH ⊤ (cid:1) , s.t.
Revised: max H 1 n Tr (cid:0) K α HH ⊤ (cid:1) , s.t.

Original: H ⊤ H
Revised: H ⊤ H

Original: = I K , and △ = { α ∈ R m | (cid:80) mp
Revised: = I K , and △ = { α ∈ R m | (cid:80) mp

Original: =1 α p =1 , α p ≥ 0 , ∀ p ∈
Revised: =1 α p =1 , α p ≥ 0 , ∀ p ∈

Original: [ m ] } .
Revised: [ m ] } .

Original: F ( α ) in Eq.(12) is proven differentiable, and can be optimized by a reducedgradient algorithm [23].
Revised: F ( α ) in Eq.(12) is proven differentiable, and can be optimized by a reducedgradient algorithm [26].

Original: One can refer to Section A.5 in the appendix for detailed optimizationprocedure.
Revised: One can refer to Section A.5 in the appendix for detailed optimizationprocedure.

Original: The following theorem indicates that the learned kernel weights of SimpleMKKM are stable with theinput samples.
Revised: The following theorem indicates that the learned kernel weights of SimpleMKKM are stable with theinput samples.

Original: Theorem 5.5.
Revised: Theorem 5.5.

Original: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.
Revised: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.

Original: Denote that the kernel weights obtained by performing SimpleMKKM on S, S i are respectively α , β , and the number of iterations is T .
Revised: Denote that the kernel weights obtained by performing SimpleMKKM on S, S i are respectively α , β , and the number of iterations is T .

Original: If in each iteration, the index of largest element of α is the same as β , then there exists a constant c 0 ≥ 1 such that
Revised: If in each iteration, the index of largest element of α is the same as β , then there exists a constant c 0 ≥ 1 such that

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: By Theorem 5.4 and Theorem 5.5, we have the following corollary.
Revised: By Theorem 5.4 and Theorem 5.5, we have the following corollary.

Original: We know that SimpleMKKMwith the proposed method has uniform stability, thus the generalization bound can be established.
Revised: We know that SimpleMKKMwith the proposed method has uniform stability, thus the generalization bound can be established.

Original: Corollary 5.6.
Revised: Corollary 5.6.

Original: Assume that for any kernel weights α , and the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.
Revised: Assume that for any kernel weights α , and the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.

Original: Then, there exists a constant c 0 such that
Revised: Then, there exists a constant c 0 such that

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Remark 3 .
Revised: Remark 3 .

Original: From the empirical results of Section B.3 in the appendix, SimpleMKKM alwaysconverges within several iterations, and thus T can be regarded as a constant.
Revised: From the empirical results of Section B.3 in the appendix, SimpleMKKM alwaysconverges within several iterations, and thus T can be regarded as a constant.

Original: Above all, according to
Revised: Above all, according to

Original: Theorem 5.3, we know that the excess risk of SimpleMKKM with the approximation method can bebounded by ˜ O (cid:16) ( m +1)
Revised: Theorem 5.3, we know that the excess risk of SimpleMKKM with the approximation method can bebounded by ˜ O (cid:16) ( m +1)

Original: K n √ n + 1 √ n (cid:17) .
Revised: K n √ n + 1 √ n (cid:17) .

Original: Our bound is tighter than the existing results O ( mK √ n ) proposed in[19].
Revised: Our bound is tighter than the existing results O ( mK √ n ) proposed in[22].

Original: As far as we know, this is the first time to study the generalization ability of MKC with stability,and the provided bound is the tightest among existing results of MKC algorithms.
Revised: As far as we know, this is the first time to study the generalization ability of MKC with stability,and the provided bound is the tightest among existing results of MKC algorithms.

Original:  Experiments
Revised:  Experiments

Original: Besides the theoretical analysis, we further validate the effectiveness of the proposed method experimentally.
Revised: Besides the theoretical analysis, we further validate the effectiveness of the proposed method experimentally.

Original: To illustrate that the proposed method can handle out-of-sample problems properly,we first apply the proposed method to SimpleMKKM [19] and compare the clustering accuracy andefficiency of the proposed method and the original SimpleMKKM [19].
Revised: To illustrate that the proposed method can handle out-of-sample problems properly,we first apply the proposed method to SimpleMKKM [22] and compare the clustering accuracy andefficiency of the proposed method and the original SimpleMKKM [22].

Original: To evaluate the scalabilityof our method, we again apply it to SimpleMKKM and compare it against several state-of-the-artefficient MKC algorithms on four large-scale datasets.
Revised: To evaluate the scalabilityof our method, we again apply it to SimpleMKKM and compare it against several state-of-the-artefficient MKC algorithms on four large-scale datasets.

Original: The analysis of convergence is reported in
Revised: The analysis of convergence is reported in

Original: Section B.3 of the appendix.
Revised: Section B.3 of the appendix.

Original: 6.1 The Comparison with SimpleMKKM
Revised: 6.1 The Comparison with SimpleMKKM

Original: We follow the settings in SimpleMKKM [19] and conduct experiments on 5 benchmark datasets,including Flo17 , Flo102 , DIGIT , Cal102 and Reuters to evaluate the effectiveness of the proposedalgorithm over the out-of-sample problem.
Revised: We follow the settings in SimpleMKKM [22] and conduct experiments on 5 benchmark datasets,including Flo17 , Flo102 , DIGIT , Cal102 and Reuters to evaluate the effectiveness of the proposedalgorithm over the out-of-sample problem.

Original: The details of the above datasets are listed in Section B.1of the appendix.
Revised: The details of the above datasets are listed in Section B.1of the appendix.

Original: We first uniformly select n samples at random from the original dataset, and theremaining N − n samples are regarded as out-of-sample points.
Revised: We first uniformly select n samples at random from the original dataset, and theremaining N − n samples are regarded as out-of-sample points.

Original: We perform SimpleMKKM on the nsamples for the related parameters and compute the approximated embedding by Eq.(8).
Revised: We perform SimpleMKKM on the nsamples for the related parameters and compute the approximated embedding by Eq.(8).

Original: We record theclustering performance of the approximated embedding.
Revised: We record theclustering performance of the approximated embedding.

Original: For comparison, the results of SimpleMKKMon all N samples are also recorded.
Revised: For comparison, the results of SimpleMKKMon all N samples are also recorded.

Original: In our experiments, n is ranged in [ K, 200 , 300 , · · · , 1000] ,where K is the cluster number.
Revised: In our experiments, n is ranged in [ K, 200 , 300 , · · · , 1000] ,where K is the cluster number.

Original: To avoid the influence of randomness, we run the experiments 20 timeswith the same setting.
Revised: To avoid the influence of randomness, we run the experiments 20 timeswith the same setting.

Original: All experiments is performed on a desktop with Intel(R)
Revised: All experiments is performed on a desktop with Intel(R)

Original: Core(TM)-i7-7820X
Revised: Core(TM)-i7-7820X

Original: CPU and 64G RAM.
Revised: CPU and 64G RAM.

Original: We record the clustering performance of the proposed method (denoted as proposed) and SimpleMKKM (denoted as original) in terms of accuracy (ACC), normalized mutual information (NMI),purity, and execution time in Table 1.
Revised: We record the clustering performance of the proposed method (denoted as proposed) and SimpleMKKM (denoted as original) in terms of accuracy (ACC), normalized mutual information (NMI),purity, and execution time in Table 1.

Original: The best performance of the proposed method and the corresponding running time are reported among different landmarks.
Revised: The best performance of the proposed method and the corresponding running time are reported among different landmarks.

Original: As observed in Table 1, the proposedmethod is comparable with SimpleMKKM in terms of the clustering performance evaluation indexes.
Revised: As observed in Table 1, the proposedmethod is comparable with SimpleMKKM in terms of the clustering performance evaluation indexes.

Original: This verifies that the approximated embedding obtained by our method is as effective as the exactembedding.
Revised: This verifies that the approximated embedding obtained by our method is as effective as the exactembedding.

Original: On the other hand, the processing time of the proposed method is much less than thatof SimpleMKKM.
Revised: On the other hand, the processing time of the proposed method is much less than thatof SimpleMKKM.

Original: We can also see that the proposed method achieves a more evident accelerationeffect with an increasing number of samples.
Revised: We can also see that the proposed method achieves a more evident accelerationeffect with an increasing number of samples.

Original: For example, in Flo17 , the proposed method works2.35 times faster than SimpleMKKM, while in Reuters , the proposed method is around 238.5 timesfaster than SimpleMKKM.
Revised: For example, in Flo17 , the proposed method works2.35 times faster than SimpleMKKM, while in Reuters , the proposed method is around 238.5 timesfaster than SimpleMKKM.

Original: We also study the clustering performance of the proposed method with different numbers of landmarks.
Revised: We also study the clustering performance of the proposed method with different numbers of landmarks.

Original: Due to limited space, please refer to Section B.2 for details.
Revised: Due to limited space, please refer to Section B.2 for details.

Original: As mentioned in Section 4, the proposed method can enable MKC to be applied to large-scale datasets.
Revised: As mentioned in Section 4, the proposed method can enable MKC to be applied to large-scale datasets.

Original: To verify the efficiency, we evaluate the performance of the proposed algorithm on 4 large-scale
Revised: To verify the efficiency, we evaluate the performance of the proposed algorithm on 4 large-scale

Original: [Table]
Revised: [Table]

Original: datasets, including NUSWIDE , AwA and MNIST and YtVideo .
Revised: datasets, including NUSWIDE , AwA and MNIST and YtVideo .

Original: The sample numbers of these datasets
Revised: The sample numbers of these datasets

Original: [Table]
Revised: [Table]

Original: are all larger than 30,000, and the maximal one is 101,499.
Revised: are all larger than 30,000, and the maximal one is 101,499.

Original: The detailed information of these datasetsis reported in Section B.1 of the appendix.
Revised: The detailed information of these datasetsis reported in Section B.1 of the appendix.

Original: For comparison, we conduct experiments on several SOTA large-scale multi-view clustering algorithms, including (1) scalable and parameter-free multi-view graph clustering (SFMC)
Revised: For comparison, we conduct experiments on several SOTA large-scale multi-view clustering algorithms, including (1) scalable and parameter-free multi-view graph clustering (SFMC)

Original: [16]which is a graph-based multi-view clustering algorithm with adaptively selected anchors; (2) binarymulti-view clustering (BMVC)
Revised: [18]which is a graph-based multi-view clustering algorithm with adaptively selected anchors; (2) binarymulti-view clustering (BMVC)

Original: [28] which encodes each graphical view into a binary form for lowercomputational complexity; (3) large-scale multi-view subspace clustering (LMVSC) [11] whichconstructs a matrix with selected anchors to reduce the redundant computation of subspace clustering.
Revised: [32] which encodes each graphical view into a binary form for lowercomputational complexity; (3) large-scale multi-view subspace clustering (LMVSC) [13] whichconstructs a matrix with selected anchors to reduce the redundant computation of subspace clustering.

Original: For the comparison algorithms, we select the best hyper-parameters by grid searching as suggested inthe comparison methods.
Revised: For the comparison algorithms, we select the best hyper-parameters by grid searching as suggested inthe comparison methods.

Original: For sufficient landmarks, we set the number of landmarks of the proposedmethod as 1000 .
Revised: For sufficient landmarks, we set the number of landmarks of the proposedmethod as 1000 .

Original: [Table]
Revised: [Table]

Original: We can see from Table 2 that the proposed method outperforms all comparison algorithms in
Revised: We can see from Table 2 that the proposed method outperforms all comparison algorithms in

Original: NUSWIDE , AwA and MNIST .
Revised: NUSWIDE , AwA and MNIST .

Original: For YtVideo data, it is a vast dataset
Revised: For YtVideo data, it is a vast dataset

Original: [Table]
Revised: [Table]

Original: but our algorithm still works andachieves comparable performance.
Revised: but our algorithm still works andachieves comparable performance.

Original: Moreover, the proposed method achieves the fastest processingspeed among all comparison methods.
Revised: Moreover, the proposed method achieves the fastest processingspeed among all comparison methods.

Original: Above all, it can be verified that the proposed algorithm isefficient and effective on large-scale datasets.
Revised: Above all, it can be verified that the proposed algorithm isefficient and effective on large-scale datasets.

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this paper, we propose a simple yet effective approximation embedding method for multiple kernelclustering.
Revised: In this paper, we propose a simple yet effective approximation embedding method for multiple kernelclustering.

Original: It can be applied to accelerate the existing MKC algorithms to efficiently obtain theembedding of out-of-sample points, therefore is suitable for clustering large-scale dataset.
Revised: It can be applied to accelerate the existing MKC algorithms to efficiently obtain theembedding of out-of-sample points, therefore is suitable for clustering large-scale dataset.

Original: We, forthe first time, theoretically study the stability of clustering algorithms and establish the generalizationbound of the proposed method.
Revised: We, forthe first time, theoretically study the stability of clustering algorithms and establish the generalizationbound of the proposed method.

Original: We first give the learning bound of relaxed KKM with algorithmicstability which is then extended for the analysis of MKC.
Revised: We first give the learning bound of relaxed KKM with algorithmicstability which is then extended for the analysis of MKC.

Original: Finally, the experimental results verify theeffectiveness and efficiency of the proposed method.
Revised: Finally, the experimental results verify theeffectiveness and efficiency of the proposed method.

Original: Although we give some theoretical results about the generalization of clustering algorithm by stability,some assumptions still limit the application range of our theorems, e.g. , the assumptions abouteigenvalues.
Revised: Although we give some theoretical results about the generalization of clustering algorithm by stability,some assumptions still limit the application range of our theorems, e.g. , the assumptions abouteigenvalues.

Original: In the future, we aim to 1) establish the stability and generalization bounds for awider range of MKC algorithms based on milder assumptions; 2) determine the optimal number oflandmarks through theoretical analysis.
Revised: In the future, we aim to 1) establish the stability and generalization bounds for awider range of MKC algorithms based on milder assumptions; 2) determine the optimal number oflandmarks through theoretical analysis.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)
Revised: [N/A]

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 2for general technical conditions not given in statements of results.(b)
Revised: [Yes] See Section 2for general technical conditions not given in statements of results.

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Proofs are given inthe supplementary material3.
Revised: [Yes] Proofs are given inthe supplementary material

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A](c)
Revised: [N/A]

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]4.
Revised: [N/A]4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]