Original: Stability and Generalization of Kernel Clustering: From Single Kernel to Multiple Kernel
Revised: Stability and Generalization of Kernel Clustering: From Single Kernel to Multiple Kernel

Original: Abstract
Revised: Abstract

Original: Multiple kernel clustering (MKC) is an important research topic that has been widely studied for decades.
Revised: Multiple kernel clustering (MKC) is an important research topic that has been widely studied for decades.

Original: However, current methods still face two problems: inefficient when handling out-of-sample data points and lack of theoretical study of the stability and generalization of clustering.
Revised: However, current methods still face two problems: inefficient when handling out-of-sample data points and lack of theoretical study of the stability and generalization of clustering.

Original: In this paper, we propose a novel method that can efficiently compute the embedding of out-of-sample data with a solid generalization guarantee.
Revised: In this paper, we propose a novel method that can efficiently compute the embedding of out-of-sample data with a solid generalization guarantee.

Original: Specifically, we approximate the eigen functions of the integral operator associated with the linear combination of base kernel functions to construct low-dimensional embeddings of out-of-sample points for efficient multiple kernel clustering.
Revised: Specifically, we approximate the eigen functions of the integral operator associated with the linear combination of base kernel functions to construct low-dimensional embeddings of out-of-sample points for efficient multiple kernel clustering.

Original: In addition, we, for the first time, theoretically study the stability of clustering algorithms and prove that the single-view version of the proposed method has uniform stability as O (cid:0)
Revised: In addition, we, for the first time, theoretically study the stability of clustering algorithms and prove that the single-view version of the proposed method has uniform stability as O (cid:0)

Original: Kn − 3 / 2 (cid:1) and establish an upper bound of excess risk as (cid:101) O (cid:0)
Revised: Kn − 3 / 2 (cid:1) and establish an upper bound of excess risk as (cid:101) O (cid:0)

Original: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where K is the cluster number and n is the number of samples.
Revised: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where K is the cluster number and n is the number of samples.

Original: We then extend the theoretical results to multiple kernel scenarios and find that the stability of MKC depends on kernel weights.
Revised: We then extend the theoretical results to multiple kernel scenarios and find that the stability of MKC depends on kernel weights.

Original: As an example, we apply our method to a novel MKC algorithm termed SimpleMKKM and derive the upper bound of its excess clustering risk, which is tighter than the current results.
Revised: As an example, we apply our method to a novel MKC algorithm termed SimpleMKKM and derive the upper bound of its excess clustering risk, which is tighter than the current results.

Original: Extensive experimental results validate the effectiveness and efficiency of the proposed method.
Revised: Extensive experimental results validate the effectiveness and efficiency of the proposed method.

Original:  Introduction
Revised:  Introduction

Original: Multiple kernel clustering (MKC) [33] aims to learn an optimal kernel from a set of pre-specified kernel matrices for high-quality partition.
Revised: Multiple kernel clustering (MKC) [33] aims to learn an optimal kernel from a set of pre-specified kernel matrices for high-quality partition.

Original: These methods usually assume that the optimal kernel matrix is a linear combination of the pre-specified base kernels.
Revised: These methods usually assume that the optimal kernel matrix is a linear combination of the pre-specified base kernels.

Original: By jointly optimizing kernel weights and a clustering indicator matrix, MKC improves the clustering performance of single-view kernel k -means (KKM).
Revised: By jointly optimizing kernel weights and a clustering indicator matrix, MKC improves the clustering performance of single-view kernel k -means (KKM).

Original: In recent years, MKC algorithms have been extensively studied.
Revised: In recent years, MKC algorithms have been extensively studied.

Original: Among these works, Lu et al.
Revised: Among these works, Lu et al.

Original: [23] propose to change the MKC framework into the form of kernel alignment.
Revised: [23] propose to change the MKC framework into the form of kernel alignment.

Original: Gönen et al.
Revised: Gönen et al.

Original: [9] propose to employ a localized method to obtain more sample-specific information from the data.
Revised: [9] propose to employ a localized method to obtain more sample-specific information from the data.

Original: Recently, Liu et al.
Revised: Recently, Liu et al.

Original: [22] propose a parameter-free MKC algorithm that adopts a novel min-max optimization framework and achieves state-of-the-art performance on a wide range of applications.
Revised: [20] propose a parameter-free MKC algorithm that adopts a novel min-max optimization framework and achieves state-of-the-art performance on a wide range of applications.

Original: Although impressive improvements have been made by the existing methods, there still lacks an effective and computationally efficient mechanism to handle out-of-sample problems.
Revised: Although impressive improvements have been made by the existing methods, there still lacks an effective and computationally efficient mechanism to handle out-of-sample problems.

Original: This makes them inefficient in both space and computational consumption for large-scale unseen samples.
Revised: This makes them inefficient in both space and computational consumption for large-scale unseen samples.

Original: More specifically, since all existing methods need to conduct eigen-decomposition over the matrix consisting of all visible samples as an intermediate operation, the pre-extracted information is blocked for the new coming samples.
Revised: More specifically, since all existing methods need to conduct eigen-decomposition over the matrix consisting of all visible samples as an intermediate operation, the pre-extracted information is blocked for the new coming samples.

Original: This makes the algorithms computationally expensive to handle large-scale datasets.
Revised: This makes the algorithms computationally expensive to handle large-scale datasets.

Original: To improve kernel clustering scalability, methods such as Nyström [30] approximation and random
Revised: To improve kernel clustering scalability, methods such as Nyström [30] approximation and random

Original: Fourier feature (RFF) [3] are proposed.
Revised: Fourier feature (RFF) [3] are proposed.

Original: These two methods acquire the non-linear feature of samples in real space by approximating the kernel matrix.
Revised: These two methods acquire the non-linear feature of samples in real space by approximating the kernel matrix.

Original: However, the Nyström method can’t be implemented directly on out-of-sample points of MKC directly.
Revised: However, the Nyström method can’t be implemented directly on out-of-sample points of MKC directly.

Original: The shortcoming of RFF is also obvious, i.e., the dimension of the learned embedding is comparably large such that the subsequent clustering process is time-consuming.
Revised: The shortcoming of RFF is also obvious, i.e., the dimension of the learned embedding is comparably large such that the subsequent clustering process is time-consuming.

Original: More seriously, because it’s difficult to bound the difference of the kernel weights before and after the approximation, these two methods are rarely implemented on MKC algorithms.
Revised: More seriously, because it’s difficult to bound the difference of the kernel weights before and after the approximation, these two methods are rarely implemented on MKC algorithms.

Original: In this paper, we propose a general method with a solid generalization guarantee.
Revised: In this paper, we propose a general method with a solid generalization guarantee.

Original: More precisely, we find that the main complexity is caused by recomputing eigenvectors for the embedding of new coming samples.
Revised: More precisely, we find that the main complexity is caused by recomputing eigenvectors for the embedding of new coming samples.

Original: To this end, through the study of the integral operator associated with the kernel functions, we propose to perform an eigen-decomposition approximation [27] for MKC.
Revised: To this end, through the study of the integral operator associated with the kernel functions, we propose to perform an eigen-decomposition approximation [27] for MKC.

Original: Specifically, we learn eigen functions from the empirical optimal kernel generated from base kernels and then use these eigen functions to calculate the approximation embedding for the out-of-sample data points.
Revised: Specifically, we learn eigen functions from the empirical optimal kernel generated from base kernels and then use these eigen functions to calculate the approximation embedding for the out-of-sample data points.

Original: Then, a standard k -means algorithm can be performed on the learned embedding for clustering.
Revised: Then, a standard k -means algorithm can be performed on the learned embedding for clustering.

Original: To theoretically analyze our algorithm, we propose a novel stability evaluation criterion for multiple kernel clustering and establish the generalization bound accordingly.
Revised: To theoretically analyze our algorithm, we propose a novel stability evaluation criterion for multiple kernel clustering and establish the generalization bound accordingly.

Original: Algorithmic stability [2] is a key property to estimate the generalization bound of learning algorithms.
Revised: Algorithmic stability [2] is a key property to estimate the generalization bound of learning algorithms.

Original: The stability-based technique [6, 14] is widely used in analyzing the generalization ability of learning algorithms.
Revised: The stability-based technique [6, 14] is widely used in analyzing the generalization ability of learning algorithms.

Original: However, existing literature lacks research on the stability of clustering algorithms.
Revised: However, existing literature lacks research on the stability of clustering algorithms.

Original: Unlike supervised learning, which possesses labels as a deterministic standard, the risk function of clustering tasks is highly correlated with uncertain clustering centroids.
Revised: Unlike supervised learning, which possesses labels as a deterministic standard, the risk function of clustering tasks is highly correlated with uncertain clustering centroids.

Original: Therefore, it is hard to analyze the algorithm stability for clustering.
Revised: Therefore, it is hard to analyze the algorithm stability for clustering.

Original: To address the issue, we start from single-view kernel k -means with a pairwise learning framework.
Revised: To address the issue, we start from single-view kernel k -means with a pairwise learning framework.

Original: We show that the proposed method has uniform stability as O (cid:0)
Revised: We show that the proposed method has uniform stability as O (cid:0)

Original: Kn − 3 / 2 (cid:1) , where K is the number of clusters and n is the number of samples.
Revised: Kn − 3 / 2 (cid:1) , where K is the number of clusters and n is the number of samples.

Original: Consequently, the excess risk can be upper bounded by (cid:101) O (cid:0)
Revised: Consequently, the excess risk can be upper bounded by (cid:101) O (cid:0)

Original: Kn − 3 / 2 + n − 1 / 2 (cid:1) 2 .
Revised: Kn − 3 / 2 + n − 1 / 2 (cid:1) 2 .

Original: We then extend the generalization bound analysis from single kernel clustering to multiple kernel clustering and prove that the stability of
Revised: We then extend the generalization bound analysis from single kernel clustering to multiple kernel clustering and prove that the stability of

Original: MKC is related to the learned kernel weights.
Revised: MKC is related to the learned kernel weights.

Original: As an instantiation, we use the above results to analyze the simple multiple kernel k -means (SimpleMKKM)[22] and derive a generalization bound as (cid:101) O (cid:0) ( m + 1)
Revised: As an instantiation, we use the above results to analyze the simple multiple kernel k -means (SimpleMKKM)[20] and derive a generalization bound as (cid:101) O (cid:0) ( m + 1)

Original: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where m is the number of kernels.
Revised: Kn − 3 / 2 + n − 1 / 2 (cid:1) , where m is the number of kernels.

Original: As far as we know, this is the tightest bound for MKC algorithms.
Revised: As far as we know, this is the tightest bound for MKC algorithms.

Original: Finally, we conduct numerical experiments to verify the effectiveness and efficiency of the proposed method.
Revised: Finally, we conduct numerical experiments to verify the effectiveness and efficiency of the proposed method.

Original: Our contributions are three-fold and can be summarized as follows:
Revised: Our contributions are three-fold and can be summarized as follows:

Original: 1) We propose an efficient method to enable MKC algorithms to handle out-of-sample problems.
Revised: 1) We propose an efficient method to enable MKC algorithms to handle out-of-sample problems.

Original: We successfully apply algorithmic stability to derive the generalization bound of the clustering algorithm for the first time.
Revised: We successfully apply algorithmic stability to derive the generalization bound of the clustering algorithm for the first time.

Original: We study the stability of SimpleMKKM with our method and derive its excess risk bound.
Revised: We study the stability of SimpleMKKM with our method and derive its excess risk bound.

Original: By comprehensive numerical experiments, the experimental results show that the proposed algorithm is effective and efficient.
Revised: By comprehensive numerical experiments, the experimental results show that the proposed algorithm is effective and efficient.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: Section 2 introduces the notations used in this paper.
Revised: Section 2 introduces the notations used in this paper.

Original: Section 3 discusses the related work.
Revised: Section 3 discusses the related work.

Original: Section 4 describes the proposed method.
Revised: Section 4 describes the proposed method.

Original: Section 5 analyzes the stability and generalization of KKM on the single kernel and then extends the proposed method to multiple kernel clustering.
Revised: Section 5 analyzes the stability and generalization of KKM on the single kernel and then extends the proposed method to multiple kernel clustering.

Original: The experimental results are reported in Section 6.
Revised: The experimental results are reported in Section 6.

Original: Finally, the paper is concluded in Section 7.
Revised: Finally, the paper is concluded in Section 7.

Original: All proofs are placed in the appendix due to the limited space.
Revised: All proofs are placed in the appendix due to the limited space.

Original:  Notations
Revised:  Notations

Original: To avoid confusion with mathematical notations, the uppercase and lowercase letters in bold denote matrices and vectors, respectively.
Revised: To avoid confusion with mathematical notations, the uppercase and lowercase letters in bold denote matrices and vectors, respectively.

Original: For example, A is a matrix, and a is a vector.
Revised: For example, A is a matrix, and a is a vector.

Original: The component of them is denoted by A ij or a i .
Revised: The component of them is denoted by A ij or a i .

Original: Let us denote with X the sample space and with P the corresponding data distribution.
Revised: Let us denote with X the sample space and with P the corresponding data distribution.

Original: The training dataset S =
Revised: The training dataset S =

Original: { x i } ni =1 ⊂ X is drawn i.i.d.
Revised: { x i } ni =1 ⊂ X is drawn i.i.d.

Original: from P .
Revised: from P .

Original: The empirical distribution P n is defined as P n ( x ) = 1 n if x ∈ S , otherwise 0.
Revised: The empirical distribution P n is defined as P n ( x ) = 1 n if x ∈ S , otherwise 0.

Original: We assume that kernel function k ( · , · ) :
Revised: We assume that kernel function k ( · , · ) :

Original: X × X
Revised: X × X

Original: → R is a positive-definite and conjugate symmetric function.
Revised: → R is a positive-definite and conjugate symmetric function.

Original: From [1], we know that there exist a Hilbert space H and a feature mapping ϕ ( · ) :
Revised: From [1], we know that there exist a Hilbert space H and a feature mapping ϕ ( · ) :

Original: X → H such that ∀ x , x ′ ∈ X , k ( x , x ′ )
Revised: X → H such that ∀ x , x ′ ∈ X , k ( x , x ′ )

Original: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ H .
Revised: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ H .

Original: The kernel function k ( · , · ) used in this paper satisfies that ∀ x , x ′ ∈ X , k
Revised: The kernel function k ( · , · ) used in this paper satisfies that ∀ x , x ′ ∈ X , k

Original: ( x , x ) =
Revised: ( x , x ) =

Original: k ( x ′ , x ′ ) .
Revised: k ( x ′ , x ′ ) .

Original: We assume that for any x ∈ X , ∥ ϕ ( x ) ∥ ≤ 1 .
Revised: We assume that for any x ∈ X , ∥ ϕ ( x ) ∥ ≤ 1 .

Original: Let us denote with I s the s × s identity matrix, and with ∥ · ∥ ∞ the infinity norm.
Revised: Let us denote with I s the s × s identity matrix, and with ∥ · ∥ ∞ the infinity norm.

Original: For the convenience, the probability measure ρ after differential d is omitted, i.e. , d x is short for dρ ( x ) .
Revised: For the convenience, the probability measure ρ after differential d is omitted, i.e. , d x is short for dρ ( x ) .

Original:  Related Work
Revised:  Related Work

Original: In this section, we give a brief introduction to pairwise learning and algorithmic stability which are two main contents related to our proofs.
Revised: In this section, we give a brief introduction to pairwise learning and algorithmic stability which are two main contents related to our proofs.

Original: Pairwise Learning .
Revised: Pairwise Learning .

Original: Pairwise learning defines loss function with respect to sample pairs.
Revised: Pairwise learning defines loss function with respect to sample pairs.

Original: It has been successfully applied to metric learning[31], bipartite ranking [29], AUC maximization [25], and etc.
Revised: It has been successfully applied to metric learning[31], bipartite ranking [29], AUC maximization [25], and etc.

Original: Assume that { x 1 , · · · , x n } are n samples drawn independently from some probability measure.
Revised: Assume that { x 1 , · · · , x n } are n samples drawn independently from some probability measure.

Original: Suppose that the learning model is parameterized by w .
Revised: Suppose that the learning model is parameterized by w .

Original: The empirical behavior of the learning model can then be evaluated by the empirical risk as:
Revised: The empirical behavior of the learning model can then be evaluated by the empirical risk as:

Original: [Equation]
Revised: [Equation]

Original: where l denotes the loss function.
Revised: where l denotes the loss function.

Original: Accordingly, the expected risk can be defined as:
Revised: Accordingly, the expected risk can be defined as:

Original: [Equation]
Revised: [Equation]

Original: Among the researches of learning theory, Lei et al.
Revised: Among the researches of learning theory, Lei et al.

Original: [15] establish a sharper generalization bound of pairwise learning, Li et al.
Revised: [15] establish a sharper generalization bound of pairwise learning, Li et al.

Original: [17] utilize a pairwise learning framework to verify the generalization ability of clustering algorithms.
Revised: [17] utilize a pairwise learning framework to verify the generalization ability of clustering algorithms.

Original: In [16], Lei et al.
Revised: In [16], Lei et al.

Original: study the generalization of stochastic gradient descent for pairwise learning.
Revised: study the generalization of stochastic gradient descent for pairwise learning.

Original: In [4, 17], the authors learn the excess risk bound of clustering in the form of pairwise learning.
Revised: In [4, 17], the authors learn the excess risk bound of clustering in the form of pairwise learning.

Original: As mentioned above, in the centroid-based clustering algorithms, it is generally hard to investigate the stability of the samples at the boundary of two centroids.
Revised: As mentioned above, in the centroid-based clustering algorithms, it is generally hard to investigate the stability of the samples at the boundary of two centroids.

Original: To tackle this problem, we adopt a pairwise learning framework in our analysis.
Revised: To tackle this problem, we adopt a pairwise learning framework in our analysis.

Original: Algorithmic Stability .
Revised: Algorithmic Stability .

Original: The analysis of algorithmic stability is a powerful technique for deriving the generalization bound of stable learning algorithms.
Revised: The analysis of algorithmic stability is a powerful technique for deriving the generalization bound of stable learning algorithms.

Original: In the existing literature, uniform stability [2] is the most widely used one but needs the strongest assumptions.
Revised: In the existing literature, uniform stability [2] is the most widely used one but needs the strongest assumptions.

Original: A variety of stability notions have been proposed for different algorithms to widen the application range of stability.
Revised: A variety of stability notions have been proposed for different algorithms to widen the application range of stability.

Original: Kutin et al.
Revised: Kutin et al.

Original: [14] define strong and weak stability for the algorithms which are not uniformly stable.
Revised: [14] define strong and weak stability for the algorithms which are not uniformly stable.

Original: Locally elastic stability is proposed in [6] to capture the sensitivity of the loss function in the population sense but not in the worst sense.
Revised: Locally elastic stability is proposed in [6] to capture the sensitivity of the loss function in the population sense but not in the worst sense.

Original: Lei et al.
Revised: Lei et al.

Original: [15] provide a notation of uniform stability for pairwise learning.
Revised: [15] provide a notation of uniform stability for pairwise learning.

Original: Despite the fact that algorithmic stability has been successfully used in analyzing numerous learning algorithms, no work applies it to clustering algorithms.
Revised: Despite the fact that algorithmic stability has been successfully used in analyzing numerous learning algorithms, no work applies it to clustering algorithms.

Original: In this paper, we show that the clustering algorithm we studied is uniformly stable.
Revised: In this paper, we show that the clustering algorithm we studied is uniformly stable.

Original:  Framework
Revised:  Framework

Original: In this section, we first give a brief introduction to kernel k -means and multiple kernel clustering, then introduce the proposed approximation method.
Revised: In this section, we first give a brief introduction to kernel k -means and multiple kernel clustering, then introduce the proposed approximation method.

Original: 4.1 Kernel k -means and Multiple Kernel Clustering
Revised: 4.1 Kernel k -means and Multiple Kernel Clustering

Original: Given a training set S = { x i } ni =1 and a feature map ϕ ( · ) , kernel clustering algorithm aims to minimize the following objective w.r.t.
Revised: Given a training set S = { x i } ni =1 and a feature map ϕ ( · ) , kernel clustering algorithm aims to minimize the following objective w.r.t.

Original: clustering centroids:
Revised: clustering centroids:

Original: [Equation]
Revised: [Equation]

Original: where C = { c k } Kk =1 denotes the centroids of K clusters.
Revised: where C = { c k } Kk =1 denotes the centroids of K clusters.

Original: A clustering indicator matrix H ∈ R n × K is defined as follows: if x i ∈ C k , h ik = 1 / (cid:112) |C k | , otherwise h ik = 0 , where |C k | denotes the number of samples belong to the k -th cluster.
Revised: A clustering indicator matrix H ∈ R n × K is defined as follows: if x i ∈ C k , h ik = 1 / (cid:112) |C k | , otherwise h ik = 0 , where |C k | denotes the number of samples belong to the k -th cluster.

Original: Kernel k -means.
Revised: Kernel k -means.

Original: If ϕ ( · ) is defined by some single kernel function k ( · , · ) , then Eq.(1) can be rewritten as:
Revised: If ϕ ( · ) is defined by some single kernel function k ( · , · ) , then Eq.(1) can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where K n is the empirical kernel matrix computed by the kernel function k and dataset S , i.e. ,
Revised: where K n is the empirical kernel matrix computed by the kernel function k and dataset S , i.e. ,

Original: K ij = k
Revised: K ij = k

Original: ( x i , x j ) .
Revised: ( x i , x j ) .

Original: The discrete constraint of H in Eq.(2) can be relaxed as: 1)
Revised: The discrete constraint of H in Eq.(2) can be relaxed as: 1)

Original: h
Revised: h

Original: ik can take any real number; 2) the orthogonal constraint still holds, i.e. , H ⊤ H
Revised: ik can take any real number; 2) the orthogonal constraint still holds, i.e. , H ⊤ H

Original: = I K .
Revised: = I K .

Original: Consequently, the solution to Eq.(2) are the K eigenvectors corresponding to the K largest eigenvalues of kernel matrix K n .
Revised: Consequently, the solution to Eq.(2) are the K eigenvectors corresponding to the K largest eigenvalues of kernel matrix K n .

Original: Therefore, H can be regarded as a new embedding for the training dataset, and standard k -means can then be performed on H for the final clustering.
Revised: Therefore, H can be regarded as a new embedding for the training dataset, and standard k -means can then be performed on H for the final clustering.

Original: To avoid confusion, in this paper, kernel k -means refers to the relaxed version as mentioned above.
Revised: To avoid confusion, in this paper, kernel k -means refers to the relaxed version as mentioned above.

Original: Multiple Kernel Clustering .
Revised: Multiple Kernel Clustering .

Original: In the multiple kernel setting, a sample has m ( m ≥ 2) feature mappings { ϕ p ( x ) } m p =1 associated with m base kernel functions { k p } m p =1 .
Revised: In the multiple kernel setting, a sample has m ( m ≥ 2) feature mappings { ϕ p ( x ) } m p =1 associated with m base kernel functions { k p } m p =1 .

Original: As the setting of multiple kernel learning [8], multiple kernel clustering (MKC) usually lets each kernel multiply by a weight α p ( p ∈ [ m ]) .
Revised: As the setting of multiple kernel learning [8], multiple kernel clustering (MKC) usually lets each kernel multiply by a weight α p ( p ∈ [ m ]) .

Original: These weights satisfy (cid:80) mp
Revised: These weights satisfy (cid:80) mp

Original: =1 α p = 1 and α p ∈ [0 , 1] .
Revised: =1 α p = 1 and α p ∈ [0 , 1] .

Original: Thus, the feature mapping of
Revised: Thus, the feature mapping of

Original: MKC can be represented as:
Revised: MKC can be represented as:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(2), Eq.(1) in multiple kernel setting can be reformed as:
Revised: Similar to Eq.(2), Eq.(1) in multiple kernel setting can be reformed as:

Original: [Equation]
Revised: [Equation]

Original: where K α = (cid:80) mp
Revised: where K α = (cid:80) mp

Original: =1 α 2 p K p .
Revised: =1 α 2 p K p .

Original: MKC aims to learn a set of optimal combination weights.
Revised: MKC aims to learn a set of optimal combination weights.

Original: Then, the relaxed kernel k -means is performed on K α for a unified embedding H .
Revised: Then, the relaxed kernel k -means is performed on K α for a unified embedding H .

Original: Notice that K α is the kernel matrix of the training dataset.
Revised: Notice that K α is the kernel matrix of the training dataset.

Original: To obtain the embedding of an out-of-sample point, we have to perform eigen-decomposition on the kernel matrix constructed by the training dataset and the new coming data point.
Revised: To obtain the embedding of an out-of-sample point, we have to perform eigen-decomposition on the kernel matrix constructed by the training dataset and the new coming data point.

Original: Obviously, this process is inefficient.
Revised: Obviously, this process is inefficient.

Original: 4.2 The Proposed Approximation Method
Revised: 4.2 The Proposed Approximation Method

Original: In our approach, we do not approximate the kernel matrix but the eigen functions of the integral operator associated with the kernel function.
Revised: In our approach, we do not approximate the kernel matrix but the eigen functions of the integral operator associated with the kernel function.

Original: First, we give a brief analysis of the spectrum of the integral operator.
Revised: First, we give a brief analysis of the spectrum of the integral operator.

Original: We start from the risk function of a sample and the expected risk, which are respectively defined as:
Revised: We start from the risk function of a sample and the expected risk, which are respectively defined as:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(2), we reform the risk function of MKC for any two samples x , y ∈ X as follows:
Revised: Similar to Eq.(2), we reform the risk function of MKC for any two samples x , y ∈ X as follows:

Original: [Equation]
Revised: [Equation]

Original: where k α ( · , · ) = (cid:80) mp
Revised: where k α ( · , · ) = (cid:80) mp

Original: =1 α 2 p k p ( · , · ) denotes the linear combination of base kernel functions, and
Revised: =1 α 2 p k p ( · , · ) denotes the linear combination of base kernel functions, and

Original: H = { h k ( x ) } Kk =1 denotes the clustering indicator functions which are defined as:
Revised: H = { h k ( x ) } Kk =1 denotes the clustering indicator functions which are defined as:

Original: [Equation]
Revised: [Equation]

Original: The expected clustering risk is then defined as:
Revised: The expected clustering risk is then defined as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Require: Out-of-sample example x ; kernel weights { α p } mp =1 , clustering indicator matrix H α and the largest K eigenvalues { λ k, α } Kk =1 learned by some MKC on some training set; m kernel functions { k p } mp =1 ; cluster number K .
Revised: Require: Out-of-sample example x ; kernel weights { α p } mp =1 , clustering indicator matrix H α and the largest K eigenvalues { λ k, α } Kk =1 learned by some MKC on some training set; m kernel functions { k p } mp =1 ; cluster number K .

Original: A K dimension approximation embedding for x .
Revised: A K dimension approximation embedding for x .

Original: The following proposition shows that the expected risk of the proposed pairwise learning framework equals that of the original kernel clustering algorithm with the same cluster centroids.
Revised: The following proposition shows that the expected risk of the proposed pairwise learning framework equals that of the original kernel clustering algorithm with the same cluster centroids.

Original: Thus, we can learn the excess risk bound with the proposed framework defined by Eq.(5) instead of the original one defined by Eq.(4).
Revised: Thus, we can learn the excess risk bound with the proposed framework defined by Eq.(5) instead of the original one defined by Eq.(4).

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: The expected clustering risk defined by Eq.
Revised: The expected clustering risk defined by Eq.

Original: (7) can be reformed as:
Revised: (7) can be reformed as:

Original: [Equation]
Revised: [Equation]

Original: Obviously, { h k ( x ) } Kk =1 are all unit functions and orthogonal to each other in the squared integrable space L 2 ( X , ρ ) .
Revised: Obviously, { h k ( x ) } Kk =1 are all unit functions and orthogonal to each other in the squared integrable space L 2 ( X , ρ ) .

Original: Similar to Eq.(2), the discrete constraint of these functions can also be relaxed, and the relaxed version is investigated in this paper.
Revised: Similar to Eq.(2), the discrete constraint of these functions can also be relaxed, and the relaxed version is investigated in this paper.

Original: Denote that H ∗ =
Revised: Denote that H ∗ =

Original: { h ∗ k ( x ) } Kk =1 are the K eigen functions corresponding to the K largest eigenvalues of the integral operator L k :
Revised: { h ∗ k ( x ) } Kk =1 are the K eigen functions corresponding to the K largest eigenvalues of the integral operator L k :

Original: L 2 ( X , ρ )
Revised: L 2 ( X , ρ )

Original: →
Revised: →

Original: L 2 ( X , ρ ) ,
Revised: L 2 ( X , ρ ) ,

Original: [Equation]
Revised: [Equation]

Original: Obviously, H ∗ is the minimizer of the expected clustering risk defined by Eq.(7).
Revised: Obviously, H ∗ is the minimizer of the expected clustering risk defined by Eq.(7).

Original: However, H ∗ cannot be obtained because the data distribution is unknown.
Revised: However, H ∗ cannot be obtained because the data distribution is unknown.

Original: Rosasco et al.
Revised: Rosasco et al.

Original: [27] analyze the spectral properties of the integral operator defined by kernel function and its empirical version, i.e., kernel matrix.
Revised: [27] analyze the spectral properties of the integral operator defined by kernel function and its empirical version, i.e., kernel matrix.

Original: Inspired by their method, we aim to construct clustering indicator functions from the empirical kernel matrix that can approximate H ∗ .
Revised: Inspired by their method, we aim to construct clustering indicator functions from the empirical kernel matrix that can approximate H ∗ .

Original: Suppose that the combination weights learned by some
Revised: Suppose that the combination weights learned by some

Original: MKC algorithms on S are { α p } mp =1 , the embedding of 1 n K α is H α and the largest K eigenvalues are { λ k, α } Kk =1 .
Revised: MKC algorithms on S are { α p } mp =1 , the embedding of 1 n K α is H α and the largest K eigenvalues are { λ k, α } Kk =1 .

Original: We construct K functions as follows:
Revised: We construct K functions as follows:

Original: [Equation]
Revised: [Equation]

Original: where k ranges in [ K ] and h ik, α is the ik -th element of H α .
Revised: where k ranges in [ K ] and h ik, α is the ik -th element of H α .

Original: Then, the approximation embedding for an out-of-sample point x can be constructed by a K -dimensional vector [˜ h 1 , α ( x ) , ..., ˜ h K, α ( x )] .
Revised: Then, the approximation embedding for an out-of-sample point x can be constructed by a K -dimensional vector [˜ h 1 , α ( x ) , ..., ˜ h K, α ( x )] .

Original: The above procedures are listed in Algorithm 1, and we term this algorithm as multiple kernel clustering with approximation embedding (MKC-AE).
Revised: The above procedures are listed in Algorithm 1, and we term this algorithm as multiple kernel clustering with approximation embedding (MKC-AE).

Original: This method has two noticeable advantages: 1) It can be applied to all MKC algorithms based on embedding learning.
Revised: This method has two noticeable advantages: 1) It can be applied to all MKC algorithms based on embedding learning.

Original: It has a theoretical generalization guarantee which will be studied in the following section.
Revised: It has a theoretical generalization guarantee which will be studied in the following section.

Original: The application on large-scale datasets.
Revised: The application on large-scale datasets.

Original: The proposed MKC-AE method can be adapted to perform
Revised: The proposed MKC-AE method can be adapted to perform

Original: MKC on large-scale datasets.
Revised: MKC on large-scale datasets.

Original: Let us denote with S N a large-scale dataset consisting of N samples.
Revised: Let us denote with S N a large-scale dataset consisting of N samples.

Original: We first select n samples as landmarks and perform some MKC algorithm on these landmarks.
Revised: We first select n samples as landmarks and perform some MKC algorithm on these landmarks.

Original: The remaining N − n samples can be regarded as "out-of-sample" examples, and the approximation embedding of these samples can be obtained by the following process.
Revised: The remaining N − n samples can be regarded as "out-of-sample" examples, and the approximation embedding of these samples can be obtained by the following process.

Original: Assume that the optimal kernel matrix of some MKC algorithm on n landmarks is 1 n K n, α .
Revised: Assume that the optimal kernel matrix of some MKC algorithm on n landmarks is 1 n K n, α .

Original: The matrix constructed by the
Revised: The matrix constructed by the

Original: K largest eigenvectors of 1 n K n, α is denoted as H n ∈ R n × K , and the corresponding eigenvalues matrix is D n ∈ R K × K .
Revised: K largest eigenvectors of 1 n K n, α is denoted as H n ∈ R n × K , and the corresponding eigenvalues matrix is D n ∈ R K × K .

Original: Assume that the partial kernel similarity constructed by N samples and n landmarks are { P l } ml =1 ⊂ R N × n , i.e. , [ P l ] ij = k l ( x i , x j ) , ∀ l ∈ [ m ] , i ∈
Revised: Assume that the partial kernel similarity constructed by N samples and n landmarks are { P l } ml =1 ⊂ R N × n , i.e. , [ P l ] ij = k l ( x i , x j ) , ∀ l ∈ [ m ] , i ∈

Original: [ N ] , j ∈ [ n ] .
Revised: [ N ] , j ∈ [ n ] .

Original: According to
Revised: According to

Original: Eq.(8), the approximation embedding of S N can be computed as:
Revised: Eq.(8), the approximation embedding of S N can be computed as:

Original: [Equation]
Revised: [Equation]

Original: Notice that the main complexity of computing the approximation embedding is O ( nmN + nKN ) , which is caused by the summation of { P l } ml =1 and subsequent matrix multiplication.
Revised: Notice that the main complexity of computing the approximation embedding is O ( nmN + nKN ) , which is caused by the summation of { P l } ml =1 and subsequent matrix multiplication.

Original: We then give an example to show how the proposed MKC-AE method can accelerate MKC.
Revised: We then give an example to show how the proposed MKC-AE method can accelerate MKC.

Original: The time complexity of the MKC algorithm termed SimpleMKKM proposed in [22] is basically cubic with the number of training samples.
Revised: The time complexity of the MKC algorithm termed SimpleMKKM proposed in [20] is basically cubic with the number of training samples.

Original: Performing SimpleMKKM on all N samples will cost O (cid:0) N 3 (cid:1) time, while the proposed MKC-AE method reduces the complexity to O (cid:0) n 3 + nmN + nKN (cid:1) .
Revised: Performing SimpleMKKM on all N samples will cost O (cid:0) N 3 (cid:1) time, while the proposed MKC-AE method reduces the complexity to O (cid:0) n 3 + nmN + nKN (cid:1) .

Original: When n ≪ N , the complexity is linear with the number of samples N .
Revised: When n ≪ N , the complexity is linear with the number of samples N .

Original: Thus, the proposed algorithm can enable MKC algorithms to be performed on large-scale datasets.
Revised: Thus, the proposed algorithm can enable MKC algorithms to be performed on large-scale datasets.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: As the foundation for the theoretical analysis of multiple kernel clustering, we first analyze the stability of kernel k -means (KKM) with our approximation method and then establish the generalization bound.
Revised: As the foundation for the theoretical analysis of multiple kernel clustering, we first analyze the stability of kernel k -means (KKM) with our approximation method and then establish the generalization bound.

Original: 5.1 Stability and Generalization of Clustering on Single Kernel
Revised: 5.1 Stability and Generalization of Clustering on Single Kernel

Original: Notice that there are two intractable problems in the analysis for the stability of kernel clustering: 1)
Revised: Notice that there are two intractable problems in the analysis for the stability of kernel clustering: 1)

Original: The clustering risk is concerned with centroids.
Revised: The clustering risk is concerned with centroids.

Original: However, the random initialization of clustering centroids makes the stability of kernel clustering hard to be analyzed; 2) To compute the clustering risk of a sample, it needs to find the nearest centroid.
Revised: However, the random initialization of clustering centroids makes the stability of kernel clustering hard to be analyzed; 2) To compute the clustering risk of a sample, it needs to find the nearest centroid.

Original: As a result, it is difficult to estimate the stability of the samples located at the boundary of two centroids.
Revised: As a result, it is difficult to estimate the stability of the samples located at the boundary of two centroids.

Original: In this paper, since the proposed method adopts a pairwise risk function, the above two issues can be avoided.
Revised: In this paper, since the proposed method adopts a pairwise risk function, the above two issues can be avoided.

Original: The detailed deduce process is as follows.
Revised: The detailed deduce process is as follows.

Original: In the single kernel setting, let us denote with 1 n K n the (normalized) empirical kernel matrix constructed by the training set, and suppose that the K largest eigenvalues and the corresponding eigenvectors of 1 n K n are { λ k } Kk =1 and { h k } Kk =1 , respectively.
Revised: In the single kernel setting, let us denote with 1 n K n the (normalized) empirical kernel matrix constructed by the training set, and suppose that the K largest eigenvalues and the corresponding eigenvectors of 1 n K n are { λ k } Kk =1 and { h k } Kk =1 , respectively.

Original: According to Eq.
Revised: According to Eq.

Original: (8), the approximated clustering indicator functions ˜ H
Revised: (8), the approximated clustering indicator functions ˜ H

Original: =
Revised: =

Original: { ˜ h k } Kk =1 are given by:
Revised: { ˜ h k } Kk =1 are given by:

Original: [Equation]
Revised: [Equation]

Original: where h ik is the i -th component of h k .
Revised: where h ik is the i -th component of h k .

Original: We aim to bound the excess clustering risk as:
Revised: We aim to bound the excess clustering risk as:

Original: [Equation]
Revised: [Equation]

Original: By the following proposition, we know the second item C ≤ 0 in Eq.(10).
Revised: By the following proposition, we know the second item C ≤ 0 in Eq.(10).

Original: Proposition 5.1.
Revised: Proposition 5.1.

Original: In Eq.
Revised: In Eq.

Original: (10) , E S R ( ˜ H, P n )
Revised: (10) , E S R ( ˜ H, P n )

Original: ≤ R
Revised: ≤ R

Original: ( H ∗ , P ) .
Revised: ( H ∗ , P ) .

Original: To give the upper bound of B , we first prove that the approximation method given by Eq.(9) has uniform stability O (cid:16) Kn √ n (cid:17) with respect to the risk function defined by Eq.(5) .
Revised: To give the upper bound of B , we first prove that the approximation method given by Eq.(9) has uniform stability O (cid:16) Kn √ n (cid:17) with respect to the risk function defined by Eq.(5) .

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: Denote that two training datasets S, S i have n samples and differ by only one sample 3 .
Revised: Denote that two training datasets S, S i have n samples and differ by only one sample 3 .

Original: Suppose that the empirical kernel matrix has properties as: 1) there exist two constant c, c 1 such that the K -th eigenvalue is larger than 1 /c ; and 2) the gaps of first K + 1 eigenvalues are larger than c 1 , i.e., ∀ k ∈
Revised: Suppose that the empirical kernel matrix has properties as: 1) there exist two constant c, c 1 such that the K -th eigenvalue is larger than 1 /c ; and 2) the gaps of first K + 1 eigenvalues are larger than c 1 , i.e., ∀ k ∈

Original: [ K ] , λ k − λ k +1 ≥ c 1 ≥ 4 √ n .
Revised: [ K ] , λ k − λ k +1 ≥ c 1 ≥ 4 √ n .

Original: The approximation clustering indicator functions of S, S i obtained by Eq.
Revised: The approximation clustering indicator functions of S, S i obtained by Eq.

Original: (9) are denoted as ˜ H and ˜ H i , respectively.
Revised: (9) are denoted as ˜ H and ˜ H i , respectively.

Original: Then, for any x , y ∈ X , there exists a constant c 0 > 0 such that
Revised: Then, for any x , y ∈ X , there exists a constant c 0 > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Remark 1 .
Revised: Remark 1 .

Original: Suppose that the K -th eigenvalue of the integral operator associated with kernel function kernel matrix λ K converges to λ K when n approaches infinity.
Revised: Suppose that the K -th eigenvalue of the integral operator associated with kernel function kernel matrix λ K converges to λ K when n approaches infinity.

Original: When the kernel function is fixed, λ K can be considered as a constant.
Revised: When the kernel function is fixed, λ K can be considered as a constant.

Original: Consider that a sequence of the K -th eigenvalue λ nK with different sample number n .
Revised: Consider that a sequence of the K -th eigenvalue λ nK with different sample number n .

Original: Let ϵ = λ K 2 , then there exists a positive integer N such that ∀ n ≥ N , | λ nK − λ K | ≤ λ K 2 .
Revised: Let ϵ = λ K 2 , then there exists a positive integer N such that ∀ n ≥ N , | λ nK − λ K | ≤ λ K 2 .

Original: We can let 1 /c
Revised: We can let 1 /c

Original: = min { λ 1 k , · · · , λ NK , λ K 2 } , then the assumption that λ K ≥ 1 /c holds.
Revised: = min { λ 1 k , · · · , λ NK , λ K 2 } , then the assumption that λ K ≥ 1 /c holds.

Original: Similarly, the assumption about eigen gap of the kernel matrix is related to the eigen gap of the integral operator corresponding to the kernel function, thus appropriate kernel function can make the assumption hold.
Revised: Similarly, the assumption about eigen gap of the kernel matrix is related to the eigen gap of the integral operator corresponding to the kernel function, thus appropriate kernel function can make the assumption hold.

Original: We conduct experiments to verify that the kernel functions we adopted satisfy the assumptions in Theorem 5.2.
Revised: We conduct experiments to verify that the kernel functions we adopted satisfy the assumptions in Theorem 5.2.

Original: The detailed experimental results are reported in Section B.4.
Revised: The detailed experimental results are reported in Section B.4.

Original: The following theorem [15] is about the generalization risk bound of the stable pairwise learning algorithm.
Revised: The following theorem [15] is about the generalization risk bound of the stable pairwise learning algorithm.

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: [15] Assume that algorithm A has γ -stability.
Revised: [15] Assume that algorithm A has γ -stability.

Original: If there exists a constant M >such that | E S
Revised: If there exists a constant M >such that | E S

Original: [ l ( x , y , A ( S ))] | < M holds for all x , y ∈ X .
Revised: [ l ( x , y , A ( S ))] | < M holds for all x , y ∈ X .

Original: Then for all δ ∈ (0 , 1 /e ) , the following inequality holds with probability 1 − δ :
Revised: Then for all δ ∈ (0 , 1 /e ) , the following inequality holds with probability 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that the proposed risk function defined by Eq.(5) can be upper bounded by some constant M .
Revised: It is easy to check that the proposed risk function defined by Eq.(5) can be upper bounded by some constant M .

Original: Then, combining Theorem 5.2 and Theorem 5.3, the proposed method has a generalization risk bound B as ˜ O (cid:16) Kn √ n + 1 √ n (cid:17) .
Revised: Then, combining Theorem 5.2 and Theorem 5.3, the proposed method has a generalization risk bound B as ˜ O (cid:16) Kn √ n + 1 √ n (cid:17) .

Original: As seen, the item ˜ O (cid:16) 1 √ n (cid:17) masters the whole bound.
Revised: As seen, the item ˜ O (cid:16) 1 √ n (cid:17) masters the whole bound.

Original: Notice that this is the first bound for clustering deduced by algorithmic stability, and we can use it to learn the stability and generalization of multiple kernel clustering algorithms.
Revised: Notice that this is the first bound for clustering deduced by algorithmic stability, and we can use it to learn the stability and generalization of multiple kernel clustering algorithms.

Original: In the future, we will improve this bound by further investigating the generalization theorem about the proposed pairwise learning framework.
Revised: In the future, we will improve this bound by further investigating the generalization theorem about the proposed pairwise learning framework.

Original: 5.2 Stability and Generalization of Clustering on Multiple Kernel
Revised: 5.2 Stability and Generalization of Clustering on Multiple Kernel

Original: To study the generalization ability of the proposed MKC-AE method on multiple kernel, we first reveal the relation between the stability of MKC-AE and the learned kernel weights.
Revised: To study the generalization ability of the proposed MKC-AE method on multiple kernel, we first reveal the relation between the stability of MKC-AE and the learned kernel weights.

Original: Theorem 5.4.
Revised: Theorem 5.4.

Original: Denote that two training datasets S, S i have n samples and differ by only one sample.
Revised: Denote that two training datasets S, S i have n samples and differ by only one sample.

Original: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.
Revised: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.

Original: If ∥ α − α i ∥ ∞ ≤ η , there exists a constant c 0 such that
Revised: If ∥ α − α i ∥ ∞ ≤ η , there exists a constant c 0 such that

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Remark 2 .
Revised: Remark 2 .

Original: Theorem 5.4 shows that if the kernel weights of an MKC algorithm are stable, then its approximation method proposed in Algorithm 1 has uniform stability.
Revised: Theorem 5.4 shows that if the kernel weights of an MKC algorithm are stable, then its approximation method proposed in Algorithm 1 has uniform stability.

Original: Specifically, if η in Theorem 5.4 can be bounded by O (1 / √ n ) , it can be checked that the risk function has the stability in order of
Revised: Specifically, if η in Theorem 5.4 can be bounded by O (1 / √ n ) , it can be checked that the risk function has the stability in order of

Original: O (cid:16) ( m +1) K n √ n (cid:17) .
Revised: O (cid:16) ( m +1) K n √ n (cid:17) .

Original: In this regard, for a tighter risk bound, the kernel weights of MKC algorithms should be more stable.
Revised: In this regard, for a tighter risk bound, the kernel weights of MKC algorithms should be more stable.

Original: This can be a design principle for MKC algorithms.
Revised: This can be a design principle for MKC algorithms.

Original: As an instantiation, we use the above theoretical results to analyze simple multiple kernel k -means (SimpleMKKM) [22], which is a state-of-the-art MKC algorithm without any hyper-parameters.
Revised: As an instantiation, we use the above theoretical results to analyze simple multiple kernel k -means (SimpleMKKM) [20], which is a state-of-the-art MKC algorithm without any hyper-parameters.

Original: We first give a brief introduction and then derive the generalization bound of SimpleMKKM with the proposed approximation method.
Revised: We first give a brief introduction and then derive the generalization bound of SimpleMKKM with the proposed approximation method.

Original: In the task of unsupervised learning, finding the optimal hyper-parameters is an open question.
Revised: In the task of unsupervised learning, finding the optimal hyper-parameters is an open question.

Original: SimpleMKKM is proposed as a hyper-parameter-free MKC algorithm and achieves desirable clustering performance.
Revised: SimpleMKKM is proposed as a hyper-parameter-free MKC algorithm and achieves desirable clustering performance.

Original: The variants of SimpleMKKM [19, 21] further boost the clustering performance and widen its application.
Revised: The variants of SimpleMKKM [19, 22] further boost the clustering performance and widen its application.

Original: Specifically, SimpleMKKM aims to solve the following kernel alignment-based optimization problem:
Revised: Specifically, SimpleMKKM aims to solve the following kernel alignment-based optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where F ( α ) =
Revised: where F ( α ) =

Original: max H 1 n Tr (cid:0) K α HH ⊤ (cid:1) , s.t.
Revised: max H 1 n Tr (cid:0) K α HH ⊤ (cid:1) , s.t.

Original: H ⊤ H
Revised: H ⊤ H

Original: = I K , and △ = { α ∈ R m | (cid:80) mp
Revised: = I K , and △ = { α ∈ R m | (cid:80) mp

Original: =1 α p = 1 , α p ≥ 0 , ∀ p ∈
Revised: =1 α p = 1 , α p ≥ 0 , ∀ p ∈

Original: [ m ] } .
Revised: [ m ] } .

Original: F ( α ) in Eq.(12) is proven differentiable, and can be optimized by a reduced gradient algorithm [26].
Revised: F ( α ) in Eq.(12) is proven differentiable, and can be optimized by a reduced gradient algorithm [26].

Original: One can refer to Section A.5 in the appendix for a detailed optimization procedure.
Revised: One can refer to Section A.5 in the appendix for a detailed optimization procedure.

Original: The following theorem indicates that the learned kernel weights of SimpleMKKM are stable with the input samples.
Revised: The following theorem indicates that the learned kernel weights of SimpleMKKM are stable with the input samples.

Original: Theorem 5.5.
Revised: Theorem 5.5.

Original: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.
Revised: Assume that for any kernel weights α , the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.

Original: Denote that the kernel weights obtained by performing SimpleMKKM on S, S i are respectively α , β , and the number of iterations is T .
Revised: Denote that the kernel weights obtained by performing SimpleMKKM on S, S i are respectively α , β , and the number of iterations is T .

Original: If in each iteration, the index of the largest element of α is the same as β , then there exists a constant c 0 ≥ 1 such that
Revised: If in each iteration, the index of the largest element of α is the same as β , then there exists a constant c 0 ≥ 1 such that

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: By Theorem 5.4 and Theorem 5.5, we have the following corollary.
Revised: By Theorem 5.4 and Theorem 5.5, we have the following corollary.

Original: We know that SimpleMKKM with the proposed method has uniform stability, thus the generalization bound can be established.
Revised: We know that SimpleMKKM with the proposed method has uniform stability, thus the generalization bound can be established.

Original: Corollary 5.6.
Revised: Corollary 5.6.

Original: Assume that for any kernel weights α , and the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.
Revised: Assume that for any kernel weights α , and the empirical kernel matrix 1 n K α satisfies the same properties in Theorem 5.2.

Original: Then, there exists a constant c 0 such that
Revised: Then, there exists a constant c 0 such that

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Remark 3 .
Revised: Remark 3 .

Original: From the empirical results of Section B.3 in the appendix, SimpleMKKM always converges within several iterations, and thus T can be regarded as a constant.
Revised: From the empirical results of Section B.3 in the appendix, SimpleMKKM always converges within several iterations, and thus T can be regarded as a constant.

Original: Above all, according to
Revised: Above all, according to

Original: Theorem 5.3, we know that the excess risk of SimpleMKKM with the approximation method can be bounded by ˜ O (cid:16) ( m +1)
Revised: Theorem 5.3, we know that the excess risk of SimpleMKKM with the approximation method can be bounded by ˜ O (cid:16) ( m +1)

Original: K n √ n + 1 √ n (cid:17) .
Revised: K n √ n + 1 √ n (cid:17) .

Original: Our bound is tighter than the existing results O ( mK √ n ) proposed in [22].
Revised: Our bound is tighter than the existing results O ( mK √ n ) proposed in [20].

Original: As far as we know, this is the first time to study the generalization ability of MKC with stability, and the provided bound is the tightest among existing results of MKC algorithms.
Revised: As far as we know, this is the first time to study the generalization ability of MKC with stability, and the provided bound is the tightest among existing results of MKC algorithms.

Original:  Experiments
Revised:  Experiments

Original: Besides the theoretical analysis, we further validate the effectiveness of the proposed method experimentally.
Revised: Besides the theoretical analysis, we further validate the effectiveness of the proposed method experimentally.

Original: To illustrate that the proposed method can handle out-of-sample problems properly, we first apply the proposed method to SimpleMKKM [22] and compare the clustering accuracy and efficiency of the proposed method and the original SimpleMKKM [22].
Revised: To illustrate that the proposed method can handle out-of-sample problems properly, we first apply the proposed method to SimpleMKKM [20] and compare the clustering accuracy and efficiency of the proposed method and the original SimpleMKKM [20].

Original: To evaluate the scalability of our method, we again apply it to SimpleMKKM and compare it against several state-of-the-art efficient MKC algorithms on four large-scale datasets.
Revised: To evaluate the scalability of our method, we again apply it to SimpleMKKM and compare it against several state-of-the-art efficient MKC algorithms on four large-scale datasets.

Original: The analysis of convergence is reported in
Revised: The analysis of convergence is reported in

Original: Section B.3 of the appendix.
Revised: Section B.3 of the appendix.

Original: 6.1 The Comparison with SimpleMKKM
Revised: 6.1 The Comparison with SimpleMKKM

Original: We follow the settings in SimpleMKKM [22] and conduct experiments on 5 benchmark datasets, including Flo17 , Flo102 , DIGIT , Cal102 and Reuters to evaluate the effectiveness of the proposed algorithm over the out-of-sample problem.
Revised: We follow the settings in SimpleMKKM [20] and conduct experiments on 5 benchmark datasets, including Flo17 , Flo102 , DIGIT , Cal102 and Reuters to evaluate the effectiveness of the proposed algorithm over the out-of-sample problem.

Original: The details of the above datasets are listed in Section B. of the appendix.
Revised: The details of the above datasets are listed in Section B. of the appendix.

Original: We first uniformly select n samples at random from the original dataset, and the remaining N − n samples are regarded as out-of-sample points.
Revised: We first uniformly select n samples at random from the original dataset, and the remaining N − n samples are regarded as out-of-sample points.

Original: We perform SimpleMKKM on the n samples for the related parameters and compute the approximated embedding by Eq.(8).
Revised: We perform SimpleMKKM on the n samples for the related parameters and compute the approximated embedding by Eq.(8).

Original: We record the clustering performance of the approximated embedding.
Revised: We record the clustering performance of the approximated embedding.

Original: For comparison, the results of SimpleMKKM on all N samples are also recorded.
Revised: For comparison, the results of SimpleMKKM on all N samples are also recorded.

Original: In our experiments, n is ranged in [ K, 200 , 300 , · · · , 1000] , where K is the cluster number.
Revised: In our experiments, n is ranged in [ K, 200 , 300 , · · · , 1000] , where K is the cluster number.

Original: To avoid the influence of randomness, we run the experiments 20 times with the same setting.
Revised: To avoid the influence of randomness, we run the experiments 20 times with the same setting.

Original: All experiments are performed on a desktop with Intel(R)
Revised: All experiments are performed on a desktop with Intel(R)

Original: Core(TM)-i7-7820X
Revised: Core(TM)-i7-7820X

Original: CPU and 64G RAM.
Revised: CPU and 64G RAM.

Original: We record the clustering performance of the proposed method (denoted as proposed) and SimpleMKKM (denoted as original) in terms of accuracy (ACC), normalized mutual information (NMI), purity, and execution time in Table 1.
Revised: We record the clustering performance of the proposed method (denoted as proposed) and SimpleMKKM (denoted as original) in terms of accuracy (ACC), normalized mutual information (NMI), purity, and execution time in Table 1.

Original: The best performance of the proposed method and the corresponding running time are reported among different landmarks.
Revised: The best performance of the proposed method and the corresponding running time are reported among different landmarks.

Original: As observed in Table 1, the proposed method is comparable with SimpleMKKM in terms of the clustering performance evaluation indexes.
Revised: As observed in Table 1, the proposed method is comparable with SimpleMKKM in terms of the clustering performance evaluation indexes.

Original: This verifies that the approximated embedding obtained by our method is as effective as the exact embedding.
Revised: This verifies that the approximated embedding obtained by our method is as effective as the exact embedding.

Original: On the other hand, the processing time of the proposed method is much less than that of SimpleMKKM.
Revised: On the other hand, the processing time of the proposed method is much less than that of SimpleMKKM.

Original: We can also see that the proposed method achieves a more evident acceleration effect with an increasing number of samples.
Revised: We can also see that the proposed method achieves a more evident acceleration effect with an increasing number of samples.

Original: For example, in Flo17 , the proposed method works 2.35 times faster than SimpleMKKM, while in Reuters , the proposed method is around 238.5 times faster than SimpleMKKM.
Revised: For example, in Flo17 , the proposed method works 2.35 times faster than SimpleMKKM, while in Reuters , the proposed method is around 238.5 times faster than SimpleMKKM.

Original: [Table]
Revised: [Table]

Original: We also study the clustering performance of the proposed method with different numbers of landmarks.
Revised: We also study the clustering performance of the proposed method with different numbers of landmarks.

Original: Due to limited space, please refer to Section B.2 for details.
Revised: Due to limited space, please refer to Section B.2 for details.

Original: 6.2 Experiments on Large-scale Datasets
Revised: 6.2 Experiments on Large-scale Datasets

Original: As mentioned in Section 4, the proposed method can enable MKC to be applied to large-scale datasets.
Revised: As mentioned in Section 4, the proposed method can enable MKC to be applied to large-scale datasets.

Original: To verify the efficiency, we evaluate the performance of the proposed algorithm on 4 large-scale datasets, including NUSWIDE , AwA and MNIST and YtVideo .
Revised: To verify the efficiency, we evaluate the performance of the proposed algorithm on 4 large-scale datasets, including NUSWIDE , AwA and MNIST and YtVideo .

Original: The sample numbers of these datasets are all larger than 30,000, and the maximal one is 101,499.
Revised: The sample numbers of these datasets are all larger than 30,000, and the maximal one is 101,499.

Original: The detailed information of these datasets is reported in Section B.1 of the appendix.
Revised: The detailed information of these datasets is reported in Section B.1 of the appendix.

Original: For comparison, we conduct experiments on several SOTA large-scale multi-view clustering algorithms, including (1) scalable and parameter-free multi-view graph clustering (SFMC) [18] which is a graph-based multi-view clustering algorithm with adaptively selected anchors; (2) binary multi-view clustering (BMVC) [32] which encodes each graphical view into a binary form for lower computational complexity; (3) large-scale multi-view subspace clustering (LMVSC) [13] which constructs a matrix with selected anchors to reduce the redundant computation of subspace clustering.
Revised: For comparison, we conduct experiments on several SOTA large-scale multi-view clustering algorithms, including (1) scalable and parameter-free multi-view graph clustering (SFMC) [18] which is a graph-based multi-view clustering algorithm with adaptively selected anchors; (2) binary multi-view clustering (BMVC) [32] which encodes each graphical view into a binary form for lower computational complexity; (3) large-scale multi-view subspace clustering (LMVSC) [13] which constructs a matrix with selected anchors to reduce the redundant computation of subspace clustering.

Original: For the comparison algorithms, we select the best hyper-parameters by grid searching as suggested in the comparison methods.
Revised: For the comparison algorithms, we select the best hyper-parameters by grid searching as suggested in the comparison methods.

Original: For sufficient landmarks, we set the number of landmarks of the proposed method as 1000 .
Revised: For sufficient landmarks, we set the number of landmarks of the proposed method as 1000 .

Original: [Table]
Revised: [Table]

Original: We can see from Table 2 that the proposed method outperforms all comparison algorithms in
Revised: We can see from Table 2 that the proposed method outperforms all comparison algorithms in

Original: NUSWIDE , AwA and MNIST .
Revised: NUSWIDE , AwA and MNIST .

Original: For YtVideo data, it is a vast dataset, but our algorithm still works and achieves comparable performance.
Revised: For YtVideo data, it is a vast dataset, but our algorithm still works and achieves comparable performance.

Original: Moreover, the proposed method achieves the fastest processing speed among all comparison methods.
Revised: Moreover, the proposed method achieves the fastest processing speed among all comparison methods.

Original: Above all, it can be verified that the proposed algorithm is efficient and effective on large-scale datasets.
Revised: Above all, it can be verified that the proposed algorithm is efficient and effective on large-scale datasets.

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this paper, we propose a simple yet effective approximation embedding method for multiple kernel clustering.
Revised: In this paper, we propose a simple yet effective approximation embedding method for multiple kernel clustering.

Original: It can be applied to accelerate the existing MKC algorithms to efficiently obtain the embedding of out-of-sample points, therefore is suitable for clustering large-scale datasets.
Revised: It can be applied to accelerate the existing MKC algorithms to efficiently obtain the embedding of out-of-sample points, therefore is suitable for clustering large-scale datasets.

Original: We, for the first time, theoretically study the stability of clustering algorithms and establish the generalization bound of the proposed method.
Revised: We, for the first time, theoretically study the stability of clustering algorithms and establish the generalization bound of the proposed method.

Original: We first give the learning bound of relaxed KKM with algorithmic stability which is then extended for the analysis of MKC.
Revised: We first give the learning bound of relaxed KKM with algorithmic stability which is then extended for the analysis of MKC.

Original: Finally, the experimental results verify the effectiveness and efficiency of the proposed method.
Revised: Finally, the experimental results verify the effectiveness and efficiency of the proposed method.

Original: Although we give some theoretical results about the generalization of clustering algorithms by stability, some assumptions still limit the application range of our theorems, e.g. , the assumptions about eigenvalues.
Revised: Although we give some theoretical results about the generalization of clustering algorithms by stability, some assumptions still limit the application range of our theorems, e.g. , the assumptions about eigenvalues.

Original: In the future, we aim to 1) establish the stability and generalization bounds for a wider range of MKC algorithms based on milder assumptions; 2) determine the optimal number of landmarks through theoretical analysis.
Revised: In the future, we aim to 1) establish the stability and generalization bounds for a wider range of MKC algorithms based on milder assumptions; 2) determine the optimal number of landmarks through theoretical analysis.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by the National Key R&D Program of China 2020AAA0107100 and the National Natural Science Foundation of China (project no. 61773392, 61872377, 61922088,  and 61976196).
Revised: This work was supported by the National Key R&D Program of China 2020AAA0107100 and the National Natural Science Foundation of China (project no. 61773392, 61872377, 61922088, 61976196, and 62006237).