Original: Personalized Online Federated Learning with Multiple Kernels
Revised: Personalized Online Federated Learning with Multiple Kernels

Original: Abstract
Revised: Abstract

Original: Multi-kernel learning (MKL) exhibits well-documented performance in online non-linear function approximation.
Revised: Multi-kernel learning (MKL) exhibits well-documented performance in online non-linear function approximation.

Original: Federated learning enables a group of learners (called clients) to train an MKL model on the data distributed among clients to perform online non-linear function approximation.
Revised: Federated learning enables a group of learners (called clients) to train an MKL model on the data distributed among clients to perform online non-linear function approximation.

Original: There are some challenges in online federated MKL that need to be addressed: i) Communication efficiency especially when a large number of kernels are considered ii) Heterogeneous data distribution among clients.
Revised: There are some challenges in online federated MKL that need to be addressed: i) Communication efficiency especially when a large number of kernels are considered ii) Heterogeneous data distribution among clients.

Original: The present paper develops an algorithmic framework to enable clients to communicate with the server to send their updates with affordable communication cost while clients employ a large dictionary of kernels.
Revised: The present paper develops an algorithmic framework to enable clients to communicate with the server to send their updates with affordable communication cost while clients employ a large dictionary of kernels.

Original: Utilizing random feature (RF) approximation, the present paper proposes scalable online federated MKL algorithm.
Revised: Utilizing random feature (RF) approximation, the present paper proposes scalable online federated MKL algorithm.

Original: We prove that using the proposed online federated MKL algorithm, each client enjoys sub-linear regret with respect to the RF approximation of its best kernel in hindsight, which indicates that the proposed algorithm can effectively deal with heterogeneity of the data distributed among clients.
Revised: We prove that using the proposed online federated MKL algorithm, each client enjoys sub-linear regret with respect to the RF approximation of its best kernel in hindsight, which indicates that the proposed algorithm can effectively deal with heterogeneity of the data distributed among clients.

Original: Experimental results on real datasets showcase the advantages of the proposed algorithm compared with other online federated kernel learning ones.
Revised: Experimental results on real datasets showcase the advantages of the proposed algorithm compared with other online federated kernel learning ones.

Original:  Introduction
Revised:  Introduction

Original: Kernel learning exhibits well-documented performance in function approximation tasks, while providing theoretical guarantees associated with different performance metrics, see e g. [48, 21, 38].
Revised: Kernel learning exhibits well-documented performance in function approximation tasks, while providing theoretical guarantees associated with different performance metrics, see e.g. [48, 21, 38].

Original: In some cases, a group of learners aims at collaborating to perform function approximation without revealing their data.
Revised: In some cases, a group of learners aims at collaborating to perform function approximation without revealing their data.

Original: To this end, federated learning has been emerged as a crucial learning paradigm by enabling a group of learners called clients to collaborate with each other by communicating with a central server to train a centralized model [33, 27, 12, 23].
Revised: To this end, federated learning has been emerged as a crucial learning paradigm by enabling a group of learners called clients to collaborate with each other by communicating with a central server to train a centralized model [33, 27, 12, 23].

Original: Through this process, clients send model parameters and updates to the server without revealing their data.
Revised: Through this process, clients send model parameters and updates to the server without revealing their data.

Original: Upon receiving updates from clients, the server updates the model.
Revised: Upon receiving updates from clients, the server updates the model.

Original: Therefore, federated learning enables clients to perform kernel learning for function approximation.
Revised: Therefore, federated learning enables clients to perform kernel learning for function approximation.

Original: In this context, a server and clients collaborate with each other to learn the optimal kernel.
Revised: In this context, a server and clients collaborate with each other to learn the optimal kernel.

Original: Furthermore, in some practical cases, clients may need to perform the function approximation in an online fashion while they are collaborating with the server to learn the kernel.
Revised: Furthermore, in some practical cases, clients may need to perform the function approximation in an online fashion while they are collaborating with the server to learn the kernel.

Original: For example, consider the case where clients may not have enough memory to store data in batch.
Revised: For example, consider the case where clients may not have enough memory to store data in batch.

Original: In addition, data samples may arrive in a sequential manner such that clients are not able to perform the function approximation in batch form.
Revised: In addition, data samples may arrive in a sequential manner such that clients are not able to perform the function approximation in batch form.

Original: There are major challenges in performing online kernel learning in federated fashion that need to be addressed:
Revised: There are major challenges in performing online kernel learning in federated fashion that need to be addressed:

Original: Communication Efficiency: Communication efficiency arises as a bottleneck in federated learning (see e g. [
Revised: Communication Efficiency: Communication efficiency arises as a bottleneck in federated learning (see e.g. [25, 37, 19, 16]).

Original: Specifically, limited clients -to- server communication bandwidth restricts the number of parameters that can be sent from clients to the 
Revised: Specifically, limited clients-to-server communication bandwidth restricts the number of parameters that can be sent from clients to the server.

Original: server

Original: Heterogeneous Data: The distribution of data observed by a client might be different from others(see e
Revised: Heterogeneous Data: The distribution of data observed by a client might be different from others

Original: g. [44, 29, 6]).
Revised: (see e.g. [44, 29, 6]).

Original: Thus, the optimal kernel that is aimed to be learned is different across clients.
Revised: Thus, the optimal kernel that is aimed to be learned is different across clients.

Original: Computational Complexity: Clients should be able to perform function approximation fast enough in order to make a decision in real-time.
Revised: Computational Complexity: Clients should be able to perform function approximation fast enough in order to make a decision in real-time.

Original: Therefore, the computational complexity of kernel learning methods should be affordable for clients.
Revised: Therefore, the computational complexity of kernel learning methods should be affordable for clients.

Original: Conventional online kernel learning approaches (see e.g. [21, 39]) suffer from ‘curse of dimensionality’ [3] in the sense that the number of parameters that should be learned increases with the number of observed data.
Revised: Conventional online kernel learning approaches (see e.g. [21, 39]) suffer from ‘curse of dimensionality’ [3] in the sense that the number of parameters that should be learned increases with the number of observed data.

Original: This can make employing conventional online kernel learning approaches infeasible to perform online federated kernel learning since clients may be required to send a large number of parameter updates to the server while the available clients-to-server communication bandwidth is not enough for sending such information.
Revised: This can make employing conventional online kernel learning approaches infeasible to perform online federated kernel learning since clients may be required to send a large number of parameter updates to the server while the available clients-to-server communication bandwidth is not enough for sending such information.

Original: Approximating kernels by finite-dimensional feature representations (e.g. Nyström method [49] and random feature method of [36]) makes online kernel learning approaches scalable in the sense that the learner can choose the number of parameters that should be learned, independent of the number of observed data samples (see e.g. [30, 4, 52]).
Revised: Approximating kernels by finite-dimensional feature representations (e.g. Nyström method [49] and random feature method of [36]) makes online kernel learning approaches scalable in the sense that the learner can choose the number of parameters that should be learned, independent of the number of observed data samples (see e.g. [30, 4, 52]).

Original: Employing finite-dimensional feature representations of kernels to perform online federated kernel learning, clients can choose the number of parameters that they should send to the server.
Revised: Employing finite-dimensional feature representations of kernels to perform online federated kernel learning, clients can choose the number of parameters that they should send to the server.

Original: Therefore, finite-dimensional kernel approximation can better cope with limited clients-to-server communication bandwidth compared to conventional kernel learning approaches.
Revised: Therefore, finite-dimensional kernel approximation can better cope with limited clients-to-server communication bandwidth compared to conventional kernel learning approaches.

Original: Random feature (RF) approximation [36] has been exploited to perform online federated kernel learning when a single pre-selected kernel function is employed [26, 17].
Revised: Random feature (RF) approximation [36] has been exploited to perform online federated kernel learning when a single pre-selected kernel function is employed [26, 17].

Original: The choice of the kernel function greatly affects the performance of function approximation when it comes to exploiting only a single kernel function.
Revised: The choice of the kernel function greatly affects the performance of function approximation when it comes to exploiting only a single kernel function.

Original: Employing multiple kernels instead of a single pre-selected one, can lead to obtaining more accurate function approximation since multi-kernel learning (MKL) can learn combination of kernels [24].
Revised: Employing multiple kernels instead of a single pre-selected one, can lead to obtaining more accurate function approximation since multi-kernel learning (MKL) can learn combination of kernels [24].

Original: Online federated MKL algorithms with theoretical guarantees called vM-KOFL and eM-KOFL have been proposed in [22].
Revised: Online federated MKL algorithms with theoretical guarantees called vM-KOFL and eM-KOFL have been proposed in [22].

Original: However, eM-KOFL and vM-KOFL do not provide personalized MKL models for clients since they learn the same combination of kernels for all clients.
Revised: However, eM-KOFL and vM-KOFL do not provide personalized MKL models for clients since they learn the same combination of kernels for all clients.

Original: The present paper proposes a novel p ersonalized o nline f ederated MKL algorithm called POF-MKL that provides a personalized MKL model for each client while it is ensured that the available clientsto-server communication bandwidth can afford communication cost of sending clients’ updates to the server.
Revised: The present paper proposes a novel p ersonalized o nline f ederated MKL algorithm called POF-MKL that provides a personalized MKL model for each client while it is ensured that the available clientsto-server communication bandwidth can afford communication cost of sending clients’ updates to the server.

Original: In order to alleviate the communication cost of MKL, the propsoed POF-MKL employs RF approximation of kernels and at each time instant, each client chooses a subset of kernels to send their updates to the server instead of sending the updates of all kernels.
Revised: In order to alleviate the communication cost of MKL, the propsoed POF-MKL employs RF approximation of kernels and at each time instant, each client chooses a subset of kernels to send their updates to the server instead of sending the updates of all kernels.

Original: The number of kernels in the chosen subset is selected such that the required bandwidth to send all clients’ updates does not exceed the available clients-to-server communication bandwidth.
Revised: The number of kernels in the chosen subset is selected such that the required bandwidth to send all clients’ updates does not exceed the available clients-to-server communication bandwidth.

Original: Therefore, clients can send their updates to the server independent of the number of kernels in the dictionary and as a result a comparatively large dictionary of kernels can be considered to perform function approximation.
Revised: Therefore, clients can send their updates to the server independent of the number of kernels in the dictionary and as a result a comparatively large dictionary of kernels can be considered to perform function approximation.

Original: Contributions of the present paper can be summarized as follows:
Revised: Contributions of the present paper can be summarized as follows:

Original: c1.
Revised: c1.

Original: Leveraging the proposed POF-MKL, clients can update a subset of kernels’ parameters which alleviates computational complexity and communication cost of sending updates to the server; c2.
Revised: Leveraging the proposed POF-MKL, clients can update a subset of kernels’ parameters which alleviates computational complexity and communication cost of sending updates to the server; c2.

Original: Through theoretical analysis, it is proved that using the proposed POF-MKL, each client achieves sub-linear regret with respect to RF approximation of the best kernel in hindsight associated with the corresponding client data samples (c
Revised: Through theoretical analysis, it is proved that using the proposed POF-MKL, each client achieves sub-linear regret with respect to RF approximation of the best kernel in hindsight associated with the corresponding client data samples (c

Original: f. Theorem 1).
Revised: f. Theorem 1).

Original: Moreover, it is guaranteed that the server achieves sub-linear regret with respect to the best function approximator (c
Revised: Moreover, it is guaranteed that the server achieves sub-linear regret with respect to the best function approximator (c

Original: f. Theorem 2); c3.
Revised: f. Theorem 2); c3.

Original: Experiments on real datasets showcase the effectiveness of the proposed POF-MKL compared to existing online federated kernel learning algorithms.
Revised: Experiments on real datasets showcase the effectiveness of the proposed POF-MKL compared to existing online federated kernel learning algorithms.

Original:  Problem Statement and Preliminaries
Revised:  Problem Statement and Preliminaries

Original: Let there be a set of K clients performing function approximation task in an online fashion.
Revised: Let there be a set of K clients performing function approximation task in an online fashion.

Original: The k -th client’s goal is to learn the function f using the stream of data samples { ( x k,t , y k,t ) }
Revised: The k -th client’s goal is to learn the function f using the stream of data samples { ( x k,t , y k,t ) }

Original: Tt =1 such that x k,t ∈ R
Revised: Tt =1 such that x k,t ∈ R

Original: d is the data sample observed by the k -th client at time t and y k,t is the label associated with x k,t .
Revised: d is the data sample observed by the k -th client at time t and y k,t is the label associated with x k,t .

Original: In the kernel learning context, the function f is assumed to belong to a reproducing kernel
Revised: In the kernel learning context, the function f is assumed to belong to a reproducing kernel

Original: Hilbert space (RKHS).
Revised: Hilbert space (RKHS).

Original: The present paper studies the personalized federated supervised function approximation problem
Revised: The present paper studies the personalized federated supervised function approximation problem

Original: [Equation]
Revised: [Equation]

Original: where H represents the RKHS the function f belongs to and L ( · , · ) denotes the loss function which can be defined as
Revised: where H represents the RKHS the function f belongs to and L ( · , · ) denotes the loss function which can be defined as

Original: [Equation]
Revised: [Equation]

Original: where C ( · , · ) is the cost function (e.g. least squares function for regression task), λ denotes the regularization coefficient and Ω ( · ) represents a regularizer function to prevent over-fitting and control the model complexity.
Revised: where C ( · , · ) is the cost function (e.g. least squares function for regression task), λ denotes the regularization coefficient and Ω ( · ) represents a regularizer function to prevent over-fitting and control the model complexity.

Original: Let Θ be the global parameters of the function f which are learned through collaboration of clients with the server while w k be the personalized parameter of the function f learned locally by the k -th client.
Revised: Let Θ be the global parameters of the function f which are learned through collaboration of clients with the server while w k be the personalized parameter of the function f learned locally by the k -th client.

Original: Thus, the goal is that the cumulative difference between f ( x k,t ; Θ , w k ) and y k,t over time is minimized.
Revised: Thus, the goal is that the cumulative difference between f ( x k,t ; Θ , w k ) and y k,t over time is minimized.

Original: At each time instant t , upon observing the data sample x k,t , the k -th client make the prediction f
Revised: At each time instant t , upon observing the data sample x k,t , the k -th client make the prediction f

Original: ( x k,t ; Θ , w k ) and then observes the true label y k,t .
Revised: ( x k,t ; Θ , w k ) and then observes the true label y k,t .

Original: Therefore, the function approximation problem that the server aims at solving can be expressed as min Θ (cid:80) Tt = [Equation]
Revised: Therefore, the function approximation problem that the server aims at solving can be expressed as min Θ (cid:80) Tt =1 (cid:80)

Original: Kk =1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .
Revised: Kk =1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .

Original: Moreover, finding the local parameters w k by the k -th client can be expressed as the optimization problem min w k (cid:80)
Revised: Moreover, finding the local parameters w k by the k -th client can be expressed as the optimization problem min w k (cid:80)

Original: Tt =
Revised: Tt =

Original: 1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .
Revised: 1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .

Original: In order to perform the function approximation task in an online fashion, the k -th client needs to perform the task with the values of Θ and w k at time t denoted by Θ t and w k,t , respectively.
Revised: In order to perform the function approximation task in an online fashion, the k -th client needs to perform the task with the values of Θ and w k at time t denoted by Θ t and w k,t , respectively.

Original: Thus, the values of function parameters Θ and w k should be updated ‘on the fly’.
Revised: Thus, the values of function parameters Θ and w k should be updated ‘on the fly’.

Original: Since the function f ( · ; · , · ) belongs to a reproducing kernel Hilbert space (RKHS), based on the representer theorem [48], given data samples, the optimal solution for (1) can be obtained as
Revised: Since the function f ( · ; · , · ) belongs to a reproducing kernel Hilbert space (RKHS), based on the representer theorem [48], given data samples, the optimal solution for (1) can be obtained as

Original: [Equation]
Revised: [Equation]

Original: where κ ( · , · ) denotes symmetric positive definite kernel function such that κ ( x , x ′ ) measures the similarity between x and x ′ .
Revised: where κ ( · , · ) denotes symmetric positive definite kernel function such that κ ( x , x ′ ) measures the similarity between x and x ′ .

Original: And α k,t is an unknown coefficient associated with κ ( x , x k,t ) which is required to be estimated.
Revised: And α k,t is an unknown coefficient associated with κ ( x , x k,t ) which is required to be estimated.

Original: In this case, ˆ f ( · ) in (3) belongs to the RKHS H :
Revised: In this case, ˆ f ( · ) in (3) belongs to the RKHS H :

Original: =
Revised: =

Original: { f ( · ) | f ( x ) = (cid:80) ∞ t =1 (cid:80)
Revised: { f ( · ) | f ( x ) = (cid:80) ∞ t =1 (cid:80)

Original: Kk =1 α k,t κ ( x , x k,t ) } such that RKHS norm is defined as ∥ f ∥ 2 H :
Revised: Kk =1 α k,t κ ( x , x k,t ) } such that RKHS norm is defined as ∥ f ∥ 2 H :

Original: =
Revised: =

Original: (cid:80) t (cid:80) t ′ α t α t ′ κ ( x t , x t ′ ) .
Revised: (cid:80) t (cid:80) t ′ α t α t ′ κ ( x t , x t ′ ) .

Original: Furthermore, from (3), it can be inferred that Θ t =
Revised: Furthermore, from (3), it can be inferred that Θ t =

Original: [ α 1 , 1 , . . .
Revised: [ α 1 , 1 , . . .

Original: , α K, 1 , . . .
Revised: , α K, 1 , . . .

Original: , α 1 ,t , . .
Revised: , α 1 ,t , . .

Original: . , α K,t ] .
Revised: . , α K,t ] .

Original: Therefore, the number of coefficients { α k,τ } tτ =1 , ∀ k that should be estimated to obtain ˆ f ( · ) increases over time.
Revised: Therefore, the number of coefficients { α k,τ } tτ =1 , ∀ k that should be estimated to obtain ˆ f ( · ) increases over time.

Original: This is known as curse of dimensionality [48] since the computational complexity of function approximation increases with time.
Revised: This is known as curse of dimensionality [48] since the computational complexity of function approximation increases with time.

Original: This brings challenge for federated implementation of function approximation since dimension of updates that should be sent to the server by each client grows over time and when T is large, the available communication bandwidth may not be enough for clients to send their updates.
Revised: This brings challenge for federated implementation of function approximation since dimension of updates that should be sent to the server by each client grows over time and when T is large, the available communication bandwidth may not be enough for clients to send their updates.

Original: In order to deal with the increasing number of unknown coefficients, one can employ random Fourier approximation [36].
Revised: In order to deal with the increasing number of unknown coefficients, one can employ random Fourier approximation [36].

Original: Assume that κ ( · ) is a shift-invariant kernel meaning that κ ( x , x ′ )
Revised: Assume that κ ( · ) is a shift-invariant kernel meaning that κ ( x , x ′ )

Original: = κ ( x − x ′ ) .
Revised: = κ ( x − x ′ ) .

Original: Let π κ ( ρ ) denotes the Fourier transform of κ ( · ) .
Revised: Let π κ ( ρ ) denotes the Fourier transform of κ ( · ) .

Original: If the kernel function κ ( · ) is normalized such that κ ( 0 ) = 1 , then π κ ( ρ ) can be viewed as a probability density function (PDF) (see e.g. [36]).
Revised: If the kernel function κ ( · ) is normalized such that κ ( 0 ) = 1 , then π κ ( ρ ) can be viewed as a probability density function (PDF) (see e.g. [36]).

Original: Let ρ 1 , . .
Revised: Let ρ 1 , . .

Original: , ρ D be a set of D independent and identically distributed (i.i.d) vectors drawn from π κ ( · ) .
Revised: , ρ D be a set of D independent and identically distributed (i.i.d) vectors drawn from π κ ( · ) .

Original: Let the vector z ( x ) be defined as
Revised: Let the vector z ( x ) be defined as

Original: [Equation]
Revised: [Equation]

Original: Then, ˆ κ r ( x − x ′ )
Revised: Then, ˆ κ r ( x − x ′ )

Original: =
Revised: =

Original: z ( x ) ⊤ z ( x ′ ) constitutes an unbiased estimator of κ ( x − x ′ ) and the random feature (RF) approximation of ˆ f ( x ) in (3) can be obtained as
Revised: z ( x ) ⊤ z ( x ′ ) constitutes an unbiased estimator of κ ( x − x ′ ) and the random feature (RF) approximation of ˆ f ( x ) in (3) can be obtained as

Original: [Equation]
Revised: [Equation]

Original: where in this case θ = (cid:80) Tt =1 (cid:80)
Revised: where in this case θ = (cid:80) Tt = [Equation]

Original: Kk =1 α k,t z ( x k,t ) .
Revised: Kk =1 α k,t z ( x k,t ) .

Original: According to (4), z ( x k,t ) is a 2 D vector and as a result it can be concluded that θ is a 2 D vector as well.
Revised: According to (4), z ( x k,t ) is a 2 D vector and as a result it can be concluded that θ is a 2 D vector as well.

Original: Therefore, using RF approximation, the vector θ should be estimated whose dimension does not grow over time.
Revised: Therefore, using RF approximation, the vector θ should be estimated whose dimension does not grow over time.

Original: The performance of a kernel learning algorithm depends on the choice of the kernel.
Revised: The performance of a kernel learning algorithm depends on the choice of the kernel.

Original: Thus, performing the function approximation using a pre-selected kernel requires prior information which may not be available.
Revised: Thus, performing the function approximation using a pre-selected kernel requires prior information which may not be available.

Original: To cope with this, employing a dictionary of kernels in lieu of a pre-selected single kernel has been proposed in the literature (see e.g. [45, 24, 31]).
Revised: To cope with this, employing a dictionary of kernels in lieu of a pre-selected single kernel has been proposed in the literature (see e.g. [45, 24, 31]).

Original: Specifically, the kernel is learned as a combination of kernels in the dictionary.
Revised: Specifically, the kernel is learned as a combination of kernels in the dictionary.

Original: Let κ 1 ( · ) , . .
Revised: Let κ 1 ( · ) , . .

Original: , κ N ( · ) be a set of
Revised: , κ N ( · ) be a set of

Original: N kernels where κ i ( · ) denotes the i -th kernel.
Revised: N kernels where κ i ( · ) denotes the i -th kernel.

Original: The function ¯ κ ( · ) belongs to the convex hull
Revised: The function ¯ κ ( · ) belongs to the convex hull

Original: K :=
Revised: K :=

Original: { ¯ κ ( x ) = (cid:80)
Revised: { ¯ κ ( x ) = (cid:80)

Original: Ni
Revised: Ni

Original: =1 β
Revised: =1 β

Original: i κ i ( x ) , β i ≥ 0 , ∀ i, (cid:80)
Revised: i κ i ( x ) , β i ≥ 0 , ∀ i, (cid:80)

Original: Ni =
Revised: Ni =

Original: 1 β i = 1 } is a kernel [41].
Revised: 1 β i = 1 } is a kernel [41].

Original: Therefore, in online multi-kernel learning, the goal is to learn the convex combination of kernels in the dictionary to minimize the cumulative regret with respect to the best function approximator in hindsight.
Revised: Therefore, in online multi-kernel learning, the goal is to learn the convex combination of kernels in the dictionary to minimize the cumulative regret with respect to the best function approximator in hindsight.

Original: The cumulative regret is defined as the cumulative difference between loss of the online multi-kernel learning algorithm and that of the best function approximator in hindsight.
Revised: The cumulative regret is defined as the cumulative difference between loss of the online multi-kernel learning algorithm and that of the best function approximator in hindsight.

Original: Furthermore, for a dataset { ( x t , y t ) } Tt =1 , the best function approximator is f ∗ ( · ) ∈ arg min
Revised: Furthermore, for a dataset { ( x t , y t ) } Tt =1 , the best function approximator is f ∗ ( · ) ∈ arg min

Original: f ∗ i ,i ∈
Revised: f ∗ i ,i ∈

Original: [ N ] (cid:80) Tt =1 L ( f ∗ i ( x t ) , y t )
Revised: [ N ] (cid:80) Tt =1 L ( f ∗ i ( x t ) , y t )

Original: where f ∗ i ( ·
Revised: where f ∗ i ( ·

Original: ) ∈ arg min f ∈ H
Revised: ) ∈ arg min f ∈ H

Original: i (cid:80) Tt =
Revised: i (cid:80) Tt =

Original: 1 L ( f ( x t ) , y t ) such that H
Revised: 1 L ( f ( x t ) , y t ) such that H

Original: i is an RKHS induced by κ i ( · ) and [ N ] := { 1 , . . .
Revised: i is an RKHS induced by κ i ( · ) and [ N ] := { 1 , . . .

Original: , N } .
Revised: , N } .

Original: Enabled by random feature approximation, centralized and scalable online multikernel learning algorithms have been proposed in literature (see e.g. [40, 43, 15]).
Revised: Enabled by random feature approximation, centralized and scalable online multikernel learning algorithms have been proposed in literature (see e.g. [40, 43, 15]).

Original: The present paper proposes an algorithmic framework for personalized online federated MKL using RF approximation of kernels in the dictionary.
Revised: The present paper proposes an algorithmic framework for personalized online federated MKL using RF approximation of kernels in the dictionary.

Original:  Personalized Online Federated Multi-Kernel Learning
Revised:  Personalized Online Federated Multi-Kernel Learning

Original: The present section proposes an algorithmic framework for online federated multi-kernel learning which can deal with heterogeneous data among clients.
Revised: The present section proposes an algorithmic framework for online federated multi-kernel learning which can deal with heterogeneous data among clients.

Original: To perform function approximation, RF approximations of kernel functions are employed.
Revised: To perform function approximation, RF approximations of kernel functions are employed.

Original: For the i -th kernel κ i , vectors ρ i, 1 , . .
Revised: For the i -th kernel κ i , vectors ρ i, 1 , . .

Original: , ρ i,D are drawn i.i.d from π κ i ( · ) to construct the random feature
Revised: , ρ i,D are drawn i.i.d from π κ i ( · ) to construct the random feature

Original: vector z i ( x ) =√ D
Revised: vector z i ( x ) =√ D

Original: [sin( ρ ⊤ i, 1 x ) , . .
Revised: [sin( ρ ⊤ i, 1 x ) , . .

Original: , sin( ρ ⊤ i,D x ) , cos( ρ ⊤ i, 1 x ) , . . .
Revised: , sin( ρ ⊤ i,D x ) , cos( ρ ⊤ i, 1 x ) , . . .

Original: , cos( ρ ⊤ i,D x )] .
Revised: , cos( ρ ⊤ i,D x )] .

Original: Then, at time instant t , the random feature approximation associated with κ i ( · ) can be obtained as ˆ f RF ,it ( x ) =
Revised: Then, at time instant t , the random feature approximation associated with κ i ( · ) can be obtained as ˆ f RF ,it ( x ) =

Original: θ ⊤ i,t z i ( x ) where θ i,t is the global function parameter associated the i -th kernel at time t .
Revised: θ ⊤ i,t z i ( x ) where θ i,t is the global function parameter associated the i -th kernel at time t .

Original: 3.1 Algorithm
Revised: 3.1 Algorithm

Original: At each time instant t , the server transmits global function parameters θ i,t , ∀ i ∈
Revised: At each time instant t , the server transmits global function parameters θ i,t , ∀ i ∈

Original: [ N ] to all clients.
Revised: [ N ] to all clients.

Original: The k -th client, assigns the weight w ik,t to the i -th kernel which indicates the confidence of the k -th client at time t in the function approximation given by the i -th kernel.
Revised: The k -th client, assigns the weight w ik,t to the i -th kernel which indicates the confidence of the k -th client at time t in the function approximation given by the i -th kernel.

Original: Upon receiving new data sample x k,t , the k -th client performs the function approximation combining kernels’ RF approximations as
Revised: Upon receiving new data sample x k,t , the k -th client performs the function approximation combining kernels’ RF approximations as

Original: [Equation]
Revised: [Equation]

Original: where ˆ Θ t = [ θ 1 ,t , . .
Revised: where ˆ Θ t = [ θ 1 ,t , . .

Original: , θ N,t ] , w k,t
Revised: , θ N,t ] , w k,t

Original: =
Revised: =

Original: [ w 1 k,t , . . .
Revised: [ w 1 k,t , . . .

Original: , w Nk,t ] and W k,t = (cid:80) Ni =1 w ik,t .
Revised: , w Nk,t ] and W k,t = (cid:80) Ni =1 w ik,t .

Original: As it can be inferred from (6), each client constructs its own personalized combination of kernels.
Revised: As it can be inferred from (6), each client constructs its own personalized combination of kernels.

Original: Upon observing the true label y k,t , the k -th client calculates the losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .
Revised: Upon observing the true label y k,t , the k -th client calculates the losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .

Original: Then, the k -th client leverages calculated losses to locally update both global and local parameters.
Revised: Then, the k -th client leverages calculated losses to locally update both global and local parameters.

Original: Let θ ik,t + and w ik,t +1 denote the k -th client’s local updates of θ i,t and w ik,t , respectively.
Revised: Let θ ik,t + and w ik,t +1 denote the k -th client’s local updates of θ i,t and w ik,t , respectively.

Original: Specifically, the k -th client utilizes multiplicative update rule to update w ik,t as
Revised: Specifically, the k -th client utilizes multiplicative update rule to update w ik,t as

Original: [Equation]
Revised: [Equation]

Original: where η k is the learning rate of the k -th client.
Revised: where η k is the learning rate of the k -th client.

Original: Note that the k -th client ( ∀ k ∈
Revised: Note that the k -th client ( ∀ k ∈

Original: [ K ] ) does not send its updated local parameter w k,t +1 to the server.
Revised: [ K ] ) does not send its updated local parameter w k,t +1 to the server.

Original: Clients send their locally updated global parameters to the server (i.e. θ ik,t +1 ).
Revised: Clients send their locally updated global parameters to the server (i.e. θ ik,t +1 ).

Original: Aggregating local updates, the server updates global parameters to ˆ Θ t +1 .
Revised: Aggregating local updates, the server updates global parameters to ˆ Θ t +1 .

Original: If all clients send updates associated with all kernels (i.e. θ ik,t +1 , ∀ i ∈ [ N ] ), this requires sending 2 ND parameters by each client at each time instant.
Revised: If all clients send updates associated with all kernels (i.e. θ ik,t +1 , ∀ i ∈ [ N ] ), this requires sending 2 ND parameters by each client at each time instant.

Original: When the number of both clients and kernels are large, the available client-to-server communication bandwidth may not be enough to afford sending 2 NDK parameters per time instant even for small values of D .
Revised: When the number of both clients and kernels are large, the available client-to-server communication bandwidth may not be enough to afford sending 2 NDK parameters per time instant even for small values of D .

Original: Note that reducing N and D degrade the performance of online federated MKL.
Revised: Note that reducing N and D degrade the performance of online federated MKL.

Original: Reducing N (the number of kernels), decreases the flexibility of clients to construct their ideal kernel using convex combination of kernels in the dictionary.
Revised: Reducing N (the number of kernels), decreases the flexibility of clients to construct their ideal kernel using convex combination of kernels in the dictionary.

Original: Reducing D can degrade the accuracy of RF approximation.
Revised: Reducing D can degrade the accuracy of RF approximation.

Original: The present paper proposes an algorithmic framework to enable clients to perform online function approximation with sufficiently large dictionary of kernels while the available clients-to-server communication bandwidth can afford sending updates from clients to the server when a desirable value for the number of random features D is chosen.
Revised: The present paper proposes an algorithmic framework to enable clients to perform online function approximation with sufficiently large dictionary of kernels while the available clients-to-server communication bandwidth can afford sending updates from clients to the server when a desirable value for the number of random features D is chosen.

Original: To this end, at each time instant, each client randomly chooses a subset of M kernels among all N kernels in the dictionary.

Original: Then, each client updates and sends the global parameters of the chosen M kernels to the server instead of updating and sending global parameters of all kernels.

Original: To choose a subset of M kernels, each clientsplits
Revised: To this end, at each time instant, each client

Original: Algorithm
Revised: Algorithm

Original: Input: Weights w ik,t , ∀ i ∈ [ N ] , parameter M and exploration rate 0 < ξ k ≤ 1 .
Revised: Input: Weights w ik,t , ∀ i ∈ [ N ] , parameter M and exploration rate 0 < ξ k ≤ 1 .

Original: Sort the kernels in descending order with respect to weights { w ik,t } Ni =1 .
Revised: Sort the kernels in descending order with respect to weights { w ik,t } Ni =1 .

Original: Obtain the index sequence s 1 , . . .
Revised: Obtain the index sequence s 1 , . . .

Original: , s N such that w s b k,t ≤ w s a k,t
Revised: , s N such that w s b k,t ≤ w s a k,t

Original: if b > a , ∀ a, b ∈ [ N ] .
Revised: if b > a , ∀ a, b ∈ [ N ] .

Original: Open bin B 1 and initialize j = 1 .
Revised: Open bin B 1 and initialize j = 1 .

Original: for all i ∈ [ N ] , the k -th client do if the bin B j includes less than M kernels then
Revised: for all i ∈ [ N ] , the k -th client do if the bin B j includes less than M kernels then

Original: Adds the s i -th kernel to B j . else
Revised: Adds the s i -th kernel to B j . else

Original: Opens new bin B j +1 , adds the s i -th kernel to B j +1 and updates j ← j + 1 .
Revised: Opens new bin B j +1 , adds the s i -th kernel to B j +1 and updates j ← j + 1 .

Original: end if end for
Revised: end if end for

Original: Draw an index I k,t via PMF q k,t in (8).
Revised: Draw an index I k,t via PMF q k,t in (8).

Original: Output: S k,t : indices set of kernels in the selected bin B I k,tkernels into some bins and draws randomly one of  the bins at each time instant.
Revised: Output: S k,t : indices set of kernels in the selected bin B I k,t randomly chooses a subset of M kernels among all N kernels in the dictionary.

Revised: Then, each client updates and sends the global parameters of the chosen M kernels to the server instead of updating and sending global parameters of all kernels.

Revised: To choose a subset of M kernels, each client splits kernels into some bins and draws randomly one of the bins at each time instant.

Original: Each bin contains at most M kernels and each client updates and sends global parameters associated with kernels in the chosen bin.
Revised: Each bin contains at most M kernels and each client updates and sends global parameters associated with kernels in the chosen bin.

Original: In order to distribute kernels among bins, at first the k -th client sorts kernels in descending order according to kernels’ weights { w ik,t } Ni =1 .
Revised: In order to distribute kernels among bins, at first the k -th client sorts kernels in descending order according to kernels’ weights { w ik,t } Ni =1 .

Original: Let B j represents the j -th bin of kernels.
Revised: Let B j represents the j -th bin of kernels.

Original: The k -th client adds kernels from sorted list one by one to B j until either all kernels are assigned to a bin or the number of kernels in B j reaches M .
Revised: The k -th client adds kernels from sorted list one by one to B j until either all kernels are assigned to a bin or the number of kernels in B j reaches M .

Original: When there are some kernels that are not assigned to any bins while there are M kernels in B j , the k -th client opens the bin B j +1 and adds kernels to this bin.
Revised: When there are some kernels that are not assigned to any bins while there are M kernels in B j , the k -th client opens the bin B j +1 and adds kernels to this bin.

Original: This continues until all kernels are assigned to a bin.
Revised: This continues until all kernels are assigned to a bin.

Original: As it can be inferred from the procedure of distributing kernels into bins, the number of bins at every client is m = (cid:6) NM (cid:7) .
Revised: As it can be inferred from the procedure of distributing kernels into bins, the number of bins at every client is m = (cid:6) NM (cid:7) .

Original: Furthermore, it can be concluded that B 1 includes M kernels with the largest weights while the bin B m includes
Revised: Furthermore, it can be concluded that B 1 includes M kernels with the largest weights while the bin B m includes

Original: N − ( m − 1) M kernels with lowest weights.
Revised: N − ( m − 1) M kernels with lowest weights.

Original: The k -th client assigns the weight u jk,t at time t to B j defined as u jk,t = (cid:80)
Revised: The k -th client assigns the weight u jk,t at time t to B j defined as u jk,t = (cid:80)

Original: κ i ∈B j w ik,t .
Revised: κ i ∈B j w ik,t .

Original: The k -th client draws one of the bins according to the probability mass function (PMF) q k,t defined as
Revised: The k -th client draws one of the bins according to the probability mass function (PMF) q k,t defined as

Original: [Equation]
Revised: [Equation]

Original:  where U k,t = (cid:80) mj
Revised: 8) where U k,t = (cid:80) mj

Original: =1 u jk,t and 0 < ξ k ≤ 1 is an exploration rate determined by the k -th client.
Revised: =1 u jk,t and 0 < ξ k ≤ 1 is an exploration rate determined by the k -th client.

Original: Let
Revised: Let

Original: I k,t be the index of the chosen bin by the k -th client at time t .
Revised: I k,t be the index of the chosen bin by the k -th client at time t .

Original: The PMF in (8) constitutes trade-off between exploitation and exploration.
Revised: The PMF in (8) constitutes trade-off between exploitation and exploration.

Original: According to the first term in the right hand side of (8), it is more probable that the k -th client draws a bin which includes kernels with larger weights w ik,t .
Revised: According to the first term in the right hand side of (8), it is more probable that the k -th client draws a bin which includes kernels with larger weights w ik,t .

Original: Hence, it is more probable that the k -th client collaborates in updating the global parameters of a kernel with larger weight w ik,t .
Revised: Hence, it is more probable that the k -th client collaborates in updating the global parameters of a kernel with larger weight w ik,t .

Original: Let S k,t denotes the set which includes the indices of kernels in the chosen bin at time t .
Revised: Let S k,t denotes the set which includes the indices of kernels in the chosen bin at time t .

Original: The Algorithm 1 summarizes the procedure that the k -th client determines the set S k,t .
Revised: The Algorithm 1 summarizes the procedure that the k -th client determines the set S k,t .

Original: According to Algorithm 1, kernel subset selection is personalized since each client chooses its own subset of kernels to update their parameters.
Revised: According to Algorithm 1, kernel subset selection is personalized since each client chooses its own subset of kernels to update their parameters.

Original: Let p ik,t denotes the probability that i ∈
Revised: Let p ik,t denotes the probability that i ∈

Original: S k,t .
Revised: S k,t .

Original: Then p ik,t = q b i k,t where b i is the index of the bin which includes the i -th kernel.
Revised: Then p ik,t = q b i k,t where b i is the index of the bin which includes the i -th kernel.

Original: The k -th client updates global parameters locally as follows
Revised: The k -th client updates global parameters locally as follows

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where 1 i ∈S k,t denotes an indicator function and it is 1 when i ∈ S k,t .
Revised: where 1 i ∈S k,t denotes an indicator function and it is 1 when i ∈ S k,t .

Original: The update rule in (9) implies that when i / ∈ S k,t , the k -th client does not update θ i,t (i.e. θ ik,t +

Revised: The update rule in (9) implies that when i / ∈ S k,t , the k -th client does not update θ i,t (i.e. θ ik,t +

Original: = θ i,t ).
Revised: = θ i,t ).

Original: Therefore, the k -th client sends θ ik,t +1 to the server only if i ∈ S k,t .
Revised: Therefore, the k -th client sends θ ik,t +1 to the server only if i ∈ S k,t .

Original: Therefore, at each time instant, each client needs to send at most 2 MD parameters to the server.
Revised: Therefore, at each time instant, each client needs to send at most 2 MD parameters to the server.

Original: Let C i,t be a set of client indices such that k ∈ C i,t if the k -th client sends θ ik,t +1 to the server.
Revised: Let C i,t be a set of client indices such that k ∈ C i,t if the k -th client sends θ ik,t +1 to the server.

Original: Upon aggregating updates from clients, the server updates θ i,t asθ i,
Revised: Upon aggregating updates from clients, the server updates θ i,t as

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]

Original: Algorithm 2 summarizes the proposed personalized online federated multi-kernel learning algorithm called POF-MKL.

Original: It is useful to note that using our proposed POF-MKL, the server cannot find the

Original: Input: Kernels κ i , i = 1 , ..., N , learning rate η > 

Revised: Input: Kernels κ i , i = 1 , ..., N , learning rate η >

Original: and the number of random features D .
Revised: and the number of random features D .

Original: Initialize: θ i, 1 = 0 , w ik, 1 = 1 , ∀ i ∈
Revised: Initialize: θ i, 1 = 0 , w ik, 1 = 1 , ∀ i ∈

Original: [ N ] , ∀ k ∈
Revised: [ N ] , ∀ k ∈

Original: [ K ] .
Revised: [ K ] .

Original: for t = 1 , . . .
Revised: for t = 1 , . . .

Original: , T do
Revised: , T do

Original: The server transmits the global parameters ˆ Θ t =
Revised: The server transmits the global parameters ˆ Θ t =

Original: [ θ 1 ,t , . .
Revised: [ θ 1 ,t , . .

Original: , θ N,t ] to all clients.
Revised: , θ N,t ] to all clients.

Original: for all k ∈ [ K ] , the k th client do
Revised: for all k ∈ [ K ] , the k th client do

Original: Receive one datum x k,t .
Revised: Receive one datum x k,t .

Original: Predicts ˆ f ( x k,t ; ˆ Θ t , w k,t ) via (6).
Revised: Predicts ˆ f ( x k,t ; ˆ Θ t , w k,t ) via (6).

Original: Calculates losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .
Revised: Calculates losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .

Original: Updates w ik,t+1 , ∀ i∈
Revised: Updates w ik,t i

Original: [ N] via (7).
Revised: N

Original: Selects a subset of kernel indices Sk,t using Algorithm 1.
Revised: Selects a subset of kernel indices S using Algorithm 1.

Original: Updatesθ ik,t +1 , ∀ i ∈ S k,t via (9) and sends θ ik,t +1 , ∀ i ∈ S k,t to the server.
Revised: Updates

Original: end for
Revised: end for

Original: The server updates θ i,t +1 , ∀ i∈
Revised: The server updates θ i,t 1 , ∀ i

Original: [ N ] via (10).
Revised: [ N ] via (10).

Original: end forgradients
Revised: end for

Revised: Algorithm 2 summarizes the proposed personalized online federated multi-kernel learning algorithm called POF-MKL.

Revised: It is useful to note that using our proposed POF-MKL, the server cannot find the gradients ∇L ( θ ⊤ i,

Original: ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) from updates received from clients.
Revised: t z i ( x k,t ) , y k,t ) from updates received from clients.

Original: Instead, the server can find ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p
Revised: Instead, the server can find ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p

Original: ik,t where p ik,t is a time-varying value determined locally by the k -th client.
Revised: ik,t where p ik,t is a time-varying value determined locally by the k -th client.

Original: This can promote the privacy of the proposed POF-MKL since exchanging the gradients can be hazardous to the privacy of federated learning (see e.g. [53, 14]).
Revised: This can promote the privacy of the proposed POF-MKL since exchanging the gradients can be hazardous to the privacy of federated learning (see e.g. [53, 14]).

Original: Complexity.
Revised: Complexity.

Original: Each client needs to store d -dimensional D random feature vectors for each kernel.
Revised: Each client needs to store d -dimensional D random feature vectors for each kernel.

Original: Therefore, the memory requirement of each client to implement function approximation using POFMKL is O ( dND ) .
Revised: Therefore, the memory requirement of each client to implement function approximation using POFMKL is O ( dND ) .

Original: Using POF-MKL, at each time instant, each client needs to perform O ( dND ) operations including inner products and summations.
Revised: Using POF-MKL, at each time instant, each client needs to perform O ( dND ) operations including inner products and summations.

Original: Furthermore, when ξ k < 1 , in order to choose a subset of kernels, the k -th client needs to sort kernels which imposes worst case computational complexity of O ( N log N ) .
Revised: Furthermore, when ξ k < 1 , in order to choose a subset of kernels, the k -th client needs to sort kernels which imposes worst case computational complexity of O ( N log N ) .

Original: However, when ξ k = 1 , according to PMF in (8), the k -th client chooses one bin uniformly at random and as a result in this case the k -th client does not need to sort kernels.
Revised: However, when ξ k = 1 , according to PMF in (8), the k -th client chooses one bin uniformly at random and as a result in this case the k -th client does not need to sort kernels.

Original: Therefore, setting ξ k < 1 , the computational complexity for the k -th client is O ( dND + N log N ) while setting ξ k = 1 , the computational complexity of the k -th client at each time instant is O ( dND ) .
Revised: Therefore, setting ξ k < 1 , the computational complexity for the k -th client is O ( dND + N log N ) while setting ξ k = 1 , the computational complexity of the k -th client at each time instant is O ( dND ) .

Original: 3.2 Regret Analysis
Revised: 3.2 Regret Analysis

Original: The present section analyzes the regret of the proposed POF-MKL.
Revised: The present section analyzes the regret of the proposed POF-MKL.

Original: Specifically, two types of regret
Revised: Specifically, two types of regret

Original: R k,T and R s,T are considered for the k -th client and the server, respectively.
Revised: R k,T and R s,T are considered for the k -th client and the server, respectively.

Original: The performance of the k -th client utilizing POF-MKL is analyzed in terms of regret defined as
Revised: The performance of the k -th client utilizing POF-MKL is analyzed in terms of regret defined as

Original: [Equation]
Revised: [Equation]

Original: where R k,T measures the cumulative difference between the loss of the k -th client and the loss of the RF approximation of the kernel with minimum loss among all kernels’ RF approximations.
Revised: where R k,T measures the cumulative difference between the loss of the k -th client and the loss of the RF approximation of the kernel with minimum loss among all kernels’ RF approximations.

Original: Let α ∗ ik,t ,
Revised: Let α ∗ ik,t ,

Original: ∀ t ∈
Revised: ∀ t ∈

Original: [ T ] , ∀ k ∈
Revised: [ T ] , ∀ k ∈

Original: [ K ] represents the optimal coefficients associated with the i -th kernel such that f ∗ i ( x )
Revised: [ K ] represents the optimal coefficients associated with the i -th kernel such that f ∗ i ( x )

Original: = (cid:80) Tt =1 (cid:80)
Revised: = (cid:80) Tt =1 (cid:80)

Original: Kk
Revised: Kk

Original: =1 α ∗ ik,t κ i ( x , x k,t ) .
Revised: =1 α ∗ ik,t κ i ( x , x k,t ) .

Original: Then the best function approximator is defined as f ∗ ( · ) ∈ arg min f ∗ i ,i ∈
Revised: Then the best function approximator is defined as f ∗ ( · ) ∈ arg min f ∗ i ,i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] [Equation] L [Equation]

Original: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) .
Revised: x k [Equation] t ) , y k,t ) .

Original: Furthermore, the regret of the server is defined as the cumulative difference between the loss of POF-MKL and that of the best function approximator over all data samples distributed among clients which can be expressed as
Revised: Furthermore, the regret of the server is defined as the cumulative difference between the loss of POF-MKL and that of the best function approximator over all data samples distributed among clients which can be expressed as

Original: [Equation]
Revised: [Equation]

Original: In order to analyze the regret of POF-MKL, suppose that the following assumptions hold true:
Revised: In order to analyze the regret of POF-MKL, suppose that the following assumptions hold true:

Original: (as1) L ( θ ⊤ i,t z
Revised: (as1) L ( θ ⊤ i,t z

Original: i ( x k,t ) , y k,t ) , ∀ k ∈
Revised: i ( x k,t ) , y k,t ) , ∀ k ∈

Original: [ K ] is convex with respect to θ i,t at each time instant t .
Revised: [ K ] is convex with respect to θ i,t at each time instant t .

Original: (as2)
Revised: (as2)

Original: For θ in a bounded set satisfying ∥ θ ∥ ≤ C , the loss function and its gradient are boundedas 0 ≤ L ( θ ⊤ z
Revised: For θ in a bounded set satisfying ∥ θ ∥ ≤ C , the loss function and its gradient are bounded

Original:  i ( x k,t ) , y k,t ) ≤ 1 and ∥∇L ( θ ⊤ z i ( x k,t ) , y k,t ) ∥ ≤ L .
Revised: as 0 ≤ L ( θ ⊤ z i ( x k,t ) , y k,t ) ≤ 1 and ∥∇L ( θ ⊤ z i ( x k,t ) , y k,t ) ∥ ≤ L .

Original: Moreover, each data sample is bounded as ∥ x k,t ∥ ≤ 1 , ∀ k ∈
Revised: Moreover, each data sample is bounded as ∥ x k,t ∥ ≤ 1 , ∀ k ∈

Original: [ K ] , ∀ t ∈ [ T ] .
Revised: [ K ] , ∀ t ∈ [ T ] .

Original: (as3) Kernels κ i ( · ), ∀ i ∈
Revised: (as3) Kernels κ i ( · ) ∀ i ∈

Original: [ N ] are shift-invariant with κ i ( 0 )
Revised: [ N ] are shift-invariant with κ i ( 0 )

Original: = 1 , ∀ i ∈
Revised: = 1 , ∀ i ∈

Original: [ N ].
Revised: [ N ]

Original: The following theorem investigates the regret of the k -th client according to the k -th client data.
Revised: The following theorem investigates the regret of the k -th client according to the k -th client data.

Original: The proof of the following Theorem can be found in Appendix A.
Revised: The proof of the following Theorem can be found in Appendix A.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under (as1)–(as3), the regret of the k -th client with respect to the best kernel satisfiesT
Revised: Under (as1)–(as3), the regret of the k -th client with respect to the best kernel satisfies

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 shows that by setting η k
Revised: Theorem 1 shows that by setting η k

Original: = O (cid:16) 1 √ T (cid:17) , the k -th client achieves sub-linear regret of O ( √ T ) .
Revised: = O (cid:16) 1 √ T (cid:17) , the k -th client achieves sub-linear regret of O ( √ T ) .

Original: Furthermore, Theorem 1 shows that POF-MKL can deal with heterogeneous data among clients since the regret of each client defined in (11) is calculated with respect to the corresponding client data.
Revised: Furthermore, Theorem 1 shows that POF-MKL can deal with heterogeneous data among clients since the regret of each client defined in (11) is calculated with respect to the corresponding client data.

Original: The following theorem studies the regret of the server with respect to the best function approximator.
Revised: The following theorem studies the regret of the server with respect to the best function approximator.

Original: The proof can be found in Appendix B.
Revised: The proof can be found in Appendix B.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let i ∗ :=
Revised: Let i ∗ :=

Original: arg min i ∈
Revised: arg min i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) and σ i be the second Fourier moment of the i -th kernel.
Revised: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) and σ i be the second Fourier moment of the i -th kernel.

Original: Under (as1)–(as3), the regret of the server with respect to the best function approximator satisfies
Revised: Under (as1)–(as3), the regret of the server with respect to the best function approximator satisfies

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) where C :=
Revised: k = L ( ˆ f ( x k,t ; ˆ Θ t , w k,t ) , y k,t ) − T (cid:88) t = K (cid:88) k

Revised: =

Revised: L ( f ∗ ( x k,t ) , y k,t ) ≤ KC

Revised: [Equation]

Revised: T (cid:88) t =

Revised: K (cid:88) k

Revised: =

Revised: L

Revised: [Equation]

Revised: K (cid:88) k

Revised: =

Revised: [Equation]

Revised: (cid:18)

Revised: [Equation]

Original: max i ∈
Revised: N [Equation] + [Equation]

Revised: (cid:19)

Revised: + ϵLKTC (14) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) where C := max i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk
Revised: Kk

Original: =1 α ∗ ik,t .
Revised: =1 α ∗ ik,t .

Original: As it can be inferred from (14), the regret of the server with respect to the best function approximator depends on 1 p i ∗ k,t .
Revised: As it can be inferred from (14), the regret of the server with respect to the best function approximator depends on 1 p i ∗ k,t .

Original: From (8) and the fact that p ik,t = q b i k,t , it can be concluded that p i ∗ k,t > ξ k m .
Revised: From (8) and the fact that p ik,t = q b i k,t , it can be concluded that p i ∗ k,t > ξ k m .

Original: Thus, setting ξ k = O (1) , then p ik,t > O ( MN ) .
Revised: Thus, setting ξ k = O (1) , then p ik,t > O ( MN ) .

Original: The regret bound in (14) shows that setting η = O (cid:16)(cid:113) MNT (cid:17) and ϵ = η k = 1 √ T , ∀ k ∈
Revised: The regret bound in (14) shows that setting η = O (cid:16)(cid:113) MNT (cid:17) and ϵ = η k = 1 √ T , ∀ k ∈

Original: [ K ] , the server obtains regret of O (cid:16)(cid:113) NM T (cid:17) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) .
Revised: [ K ] , the server obtains regret of O (cid:16)(cid:113) NM T (cid:17) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) .

Original: This shows that increasing M tighten the regret bound and increasing
Revised: This shows that increasing M tighten the regret bound and increasing

Original: D increases the probability that the regret bound in (14) holds true.
Revised: D increases the probability that the regret bound in (14) holds true.

Original: However, using POF-MKL, each client needs to transmit MD parameters at each time instant.
Revised: However, using POF-MKL, each client needs to transmit MD parameters at each time instant.

Original: Since both M and D are determined by the algorithm POF-MKL, this shows that POF-MKL can provide flexibility to tighten regret bound while the available clients-to-server communication bandwidth can afford transmission of clients’ updates to the server.
Revised: Since both M and D are determined by the algorithm POF-MKL, this shows that POF-MKL can provide flexibility to tighten regret bound while the available clients-to-server communication bandwidth can afford transmission of clients’ updates to the server.

Original: It is useful to mention that choosing larger value for ξ k increases the lower bound of p i ∗ k,t and as a result the optimal choice for ξ k in terms of regret is ξ k = 1 .
Revised: It is useful to mention that choosing larger value for ξ k increases the lower bound of p i ∗ k,t and as a result the optimal choice for ξ k in terms of regret is ξ k = 1 .

Original: However, choosing smaller values for ξ k makes the value of p ik,t more dependent on weights { w ik,t } Kk =
Revised: However, choosing smaller values for ξ k makes the value of p ik,t more dependent on weights { w ik,t } Kk =

Original: (c.f. (8)).
Revised: (c.f. (8)).

Original: Therefore, choosing smaller values for ξ k makes p ik,t less predictable.
Revised: Therefore, choosing smaller values for ξ k makes p ik,t less predictable.

Original: This makes estimating ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) given ∇L ( θ ⊤ i,
Revised: This makes estimating ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) given ∇L ( θ ⊤ i,

Original: t z i ( x k,t ) , y k,t ) /p
Revised: t z i ( x k,t ) , y k,t ) /p

Original: ik,t more difficult which leads to better protection of privacy.
Revised: ik,t more difficult which leads to better protection of privacy.

Original: Comparison with personalized federated learning.
Revised: Comparison with personalized federated learning.

Original: In order to deal with heterogeneous data among clients, personalized federated learning has been studied extensively in the literature (see [44, 8, 9, 12, 20, 10, 29, 28, 42, 6, 1, 32, 50, 2, 46, 5]).
Revised: In order to deal with heterogeneous data among clients, personalized federated learning has been studied extensively in the literature (see [44, 8, 9, 12, 20, 10, 29, 28, 42, 6, 1, 32, 50, 2, 46, 5]).

Original: Utilizing model-agnostic meta-learning [13], personalized federated learning algorithms have been proposed in [12, 1].
Revised: Utilizing model-agnostic meta-learning [13], personalized federated learning algorithms have been proposed in [12, 1].

Original: In [42, 5], personalized federated learning algorithms have been designed by learning hyper-networks [18].
Revised: In [42, 5], personalized federated learning algorithms have been designed by learning hyper-networks [18].

Original: In [32], a personalized model is a linear combination of a set of shared component models such that each client constructs its personalized mixture of models.
Revised: In [32], a personalized model is a linear combination of a set of shared component models such that each client constructs its personalized mixture of models.

Original: However, in aforementioned personalized federated learning works, clients are assumed to store a dataset to perform local updates with.
Revised: However, in aforementioned personalized federated learning works, clients are assumed to store a dataset to perform local updates with.

Original: Therefore, when clients are not able to store data in batch and they have to make a decision upon receiving a new data sample, aforementioned works in personalized federated learning cannot guarantee sub-linear regret for clients.
Revised: Therefore, when clients are not able to store data in batch and they have to make a decision upon receiving a new data sample, aforementioned works in personalized federated learning cannot guarantee sub-linear regret for clients.

Original: However, according to Theorems 1 and 2, POF-MKL provides sub-linear regret for clients when clients cannot store data in batch and make decision in an online fashion.
Revised: However, according to Theorems 1 and 2, POF-MKL provides sub-linear regret for clients when clients cannot store data in batch and make decision in an online fashion.

Original: Comparison with online federated learning [34].
Revised: Comparison with online federated learning [34].

Original: Fed-OMD algorithm has been proposed in [34] which enables clients to perform their learning task in an online and federated fashion while it is proved that Fed-OMD enjoys sub-linear regret when the loss function is convex with respect to parameters required to be learnt at each time instant.
Revised: Fed-OMD algorithm has been proposed in [34] which enables clients to perform their learning task in an online and federated fashion while it is proved that Fed-OMD enjoys sub-linear regret when the loss function is convex with respect to parameters required to be learnt at each time instant.

Original: The proposed POF-MKL differs from Fed-OMD in the sense that Fed-OMD cannot guarantee sub-linear regret when it comes to performing the online learning task with RF approximations of multiple kernels since the loss function L ( (cid:80) Ni =1 w i,t θ ⊤ i,t z i ( x ) , y ) is not convex with respect to both θ i,t and w i,t .
Revised: The proposed POF-MKL differs from Fed-OMD in the sense that Fed-OMD cannot guarantee sub-linear regret when it comes to performing the online learning task with RF approximations of multiple kernels since the loss function L ( (cid:80) Ni =1 w i,t θ ⊤ i,t z i ( x ) , y ) is not convex with respect to both θ i,t and w i,t .

Original: However, according to Theorems 1 and 2, the proposed
Revised: However, according to Theorems 1 and 2, the proposed

Original: POF-MKL guarantees sub-linear regret.
Revised: POF-MKL guarantees sub-linear regret.

Original: Comparison with [22].
Revised: Comparison with [22].

Original: Online federated MKL algorithms called vM-KOFL and eM-KOFL have been presented in [22].
Revised: Online federated MKL algorithms called vM-KOFL and eM-KOFL have been presented in [22].

Original: Both POF-MKL and algorithms in [22] exploit random feature approximation to alleviate computational complexity of online kernel learning.
Revised: Both POF-MKL and algorithms in [22] exploit random feature approximation to alleviate computational complexity of online kernel learning.

Original: Furthermore, both POF-MKL and algorithms in [22] learn a linear combination of kernels.
Revised: Furthermore, both POF-MKL and algorithms in [22] learn a linear combination of kernels.

Original: The proposed POF-MKL has the following advantages and innovations compared to vM-KOFL and eM-KOFL: i)
Revised: The proposed POF-MKL has the following advantages and innovations compared to vM-KOFL and eM-KOFL: i)

Original: The proposed POF-MKL allows clients to learn their own personalized combination of kernels (c.f. (6)).
Revised: The proposed POF-MKL allows clients to learn their own personalized combination of kernels (c.f. (6)).

Original: As it is proved in
Revised: As it is proved in

Original: Theorem 1, the proposed POF-MKL can deal with heterogeneous data among clients in the sense that using POF-MKL each client guarantees sub-linear regret with respect to the best kernel RF approximation according to the corresponding client data.
Revised: Theorem 1, the proposed POF-MKL can deal with heterogeneous data among clients in the sense that using POF-MKL each client guarantees sub-linear regret with respect to the best kernel RF approximation according to the corresponding client data.

Original: However, both vM-KOFL and eM-KOFL are not able to provide such guarantee.
Revised: However, both vM-KOFL and eM-KOFL are not able to provide such guarantee.

Original: ii) Using vM-KOFL, each client needs to send ( N + 1) D parameters to the server.
Revised: ii) Using vM-KOFL, each client needs to send ( N + 1) D parameters to the server.

Original: However, using the proposed POF-MKL, each client needs to send MD parameters to the server such that M ≤ N is determined by POF-MKL and can be chosen to be much smaller than N . iii)
Revised: However, using the proposed POF-MKL, each client needs to send MD parameters to the server such that M ≤ N is determined by POF-MKL and can be chosen to be much smaller than N . iii)

Original: In eM-KOFL, the server chooses a kernel at each time instant and clients send their local updates associated with the chosen kernel by the server.
Revised: In eM-KOFL, the server chooses a kernel at each time instant and clients send their local updates associated with the chosen kernel by the server.

Original: The proposed POF-MKL provides more flexibility compared to eM-KOFL in the sense that using POF-MKL each client can send local updates of M ≥ 1 kernels to the server.
Revised: The proposed POF-MKL provides more flexibility compared to eM-KOFL in the sense that using POF-MKL each client can send local updates of M ≥ 1 kernels to the server.

Original: And each client chooses its own subset of kernels to send their updates to the server.
Revised: And each client chooses its own subset of kernels to send their updates to the server.

Original: Therefore, even though POF-MKL sets M to 1 , it is possible that at a time instant the server receives updates associated with all kernels in the dictionary.
Revised: Therefore, even though POF-MKL sets M to 1 , it is possible that at a time instant the server receives updates associated with all kernels in the dictionary.

Original: It is useful to mention that using eM-KOFL the cumulative regret of all clients is sub-linear with respect to the best kernel RF approximation with probability 1 − δ
Revised: It is useful to mention that using eM-KOFL the cumulative regret of all clients is sub-linear with respect to the best kernel RF approximation with probability 1 − δ

Original: where 0 < δ ≤ 1 .
Revised: where 0 < δ ≤ 1 .

Original: However, utilizing the update rule in (9), using the proposed POF-MKL, each client obtains sub-linear regret with respect to RF approximation of its best kernel with probability 1.
Revised: However, utilizing the update rule in (9), using the proposed POF-MKL, each client obtains sub-linear regret with respect to RF approximation of its best kernel with probability 1.

Original:  Experiments
Revised:  Experiments

Original: We tested the performance of the proposed POF-MKL for online regression task through a set of experiments.
Revised: We tested the performance of the proposed POF-MKL for online regression task through a set of experiments.

Original: The performance of POF-MKL is compared with the baselines PerFedAvg [12], OFSKL [34], OFMKL-Avg [34], vM-KOFL [22] and eM-KOFL [22].
Revised: The performance of POF-MKL is compared with the baselines PerFedAvg [12], OFSKL [34], OFMKL-Avg [34], vM-KOFL [22] and eM-KOFL [22].

Original: PerFedAvg refers to the personalized federated averaging algorithm in [12].
Revised: PerFedAvg refers to the personalized federated averaging algorithm in [12].

Original: In the experiments, PerFedAvg employs a fully connected feedforward neural network model.
Revised: In the experiments, PerFedAvg employs a fully connected feedforward neural network model.

Original: More information about the implementation of PerFedAvg can be found in Appendix C. OFSKL and OFMKL-Avg are two variations of Fed-OMD [34].
Revised: More information about the implementation of PerFedAvg can be found in Appendix C. OFSKL and OFMKL-Avg are two variations of Fed-OMD [34].

Original: OFSKL leverages Fed-OMD [34] when a single radial basis function (RBF) with bandwidth of 10 is employed to perform the learning task.
Revised: OFSKL leverages Fed-OMD [34] when a single radial basis function (RBF) with bandwidth of 10 is employed to perform the learning task.

Original: In OFMKL-Avg, kernels are learned independently from each other using Fed-OMD [34] and the prediction is the average of approximations given by kernels.
Revised: In OFMKL-Avg, kernels are learned independently from each other using Fed-OMD [34] and the prediction is the average of approximations given by kernels.

Original: Moreover, vM-KOFL and eM-KOFL are online federated MKL algorithms of [22] such that vM-KOFL requires transmission of all kernel updates at every time instant while eM-KOFL requires transmission of a kernel update at each time instant.
Revised: Moreover, vM-KOFL and eM-KOFL are online federated MKL algorithms of [22] such that vM-KOFL requires transmission of all kernel updates at every time instant while eM-KOFL requires transmission of a kernel update at each time instant.

Original: In the experiments, each client observes 500 samples until the end of the learning task meaning that T = 500 .
Revised: In the experiments, each client observes 500 samples until the end of the learning task meaning that T = 500 .

Original: The performance of the proposed POF-MKL and other baselines are tested on the following real datasets downloaded from UCI machine learning repository [11]: Naval [7], UJI [47], Air [51] and WEC [35].
Revised: The performance of the proposed POF-MKL and other baselines are tested on the following real datasets downloaded from UCI machine learning repository [11]: Naval [7], UJI [47], Air [51] and WEC [35].

Original: More detailed information about datasets can be found in Appendix C. Data samples of Naval and UJI datasets are distributed i.i.d among clients.
Revised: More detailed information about datasets can be found in Appendix C. Data samples of Naval and UJI datasets are distributed i.i.d among clients.

Original: Data samples in Air and WEC datasets are distributed non-i.i.d among clients.
Revised: Data samples in Air and WEC datasets are distributed non-i.i.d among clients.

Original: More inforamtion about distributing data samples among clients can be found in Appendix C.
Revised: More inforamtion about distributing data samples among clients can be found in Appendix C.

Original: The number of clients for Naval,
Revised: The number of clients for Naval,

Original: UJI, Air and WEC datasets are 23 , 42 , 240 and 560 , respectively.
Revised: UJI, Air and WEC datasets are 23 , 42 , 240 and 560 , respectively.

Original: The dictionary of kernels consists of 51 RBFs with different bandwidth such that the bandwidth of the i -th kernel is σ i =
Revised: The dictionary of kernels consists of 51 RBFs with different bandwidth such that the bandwidth of the i -th kernel is σ i =

Original: i − 52 25 .
Revised: i − 52 25 .

Original: We consider the case where the clients-to-server communication bandwidth is limited such that at each time instant, the maximum number of parameters that a client is allowed to transmit to the server is 1000 .
Revised: We consider the case where the clients-to-server communication bandwidth is limited such that at each time instant, the maximum number of parameters that a client is allowed to transmit to the server is 1000 .

Original: Furthermore, the memory and computational capability of clients are limited such that the maximum value can be picked for the number of random features D is 100 .
Revised: Furthermore, the memory and computational capability of clients are limited such that the maximum value can be picked for the number of random features D is 100 .

Original: The experiments were carried for 20 different sets of random feature vectors.
Revised: The experiments were carried for 20 different sets of random feature vectors.

Original: The performance of algorithms is measured[Figure] using average of mean squared error (MSE) defined as
Revised: The performance of algorithms is measured using average of mean squared error (MSE) defined as

Original: [Equation]
Revised: [Equation]

Original: where ˆ y jk,t denotes the prediction of the k -th client at time instant t corresponding to the j -th set of random feature vectors.
Revised: where ˆ y jk,t denotes the prediction of the k -th client at time instant t corresponding to the j -th set of random feature vectors.

Original: Learning rates are set to η = η k = 1 √ T , ∀ k .
Revised: Learning rates are set to η = η k = 1 √ T , ∀ k .

Original: Also, exploration rates are set to ξ k = 1 , ∀ k .
Revised: Also, exploration rates are set to ξ k = 1 , ∀ k .

Original: The performance of POF-MKL with different ξ k is studied in Appendix C. Codes are available at https://github.com/pouyamghari/POF-MKL .
Revised: The performance of POF-MKL with different ξ k is studied in Appendix C. Codes are available at https://github.com/pouyamghari/POF-MKL .

Original: Table 1 presents the MSE and run time performance of online federated kernel learning algorithms on real datasets.
Revised: Table 1 presents the MSE and run time performance of online federated kernel learning algorithms on real datasets.

Original: Run time refers to average total run time of clients to perform online learning task on the entire data samples that they observe.
Revised: Run time refers to average total run time of clients to perform online learning task on the entire data samples that they observe.

Original: In Table 1, M refers to the number of kernels whose updates are sent to the server after prediction at each time instant.
Revised: In Table 1, M refers to the number of kernels whose updates are sent to the server after prediction at each time instant.

Original: And, D is the number of random features.
Revised: And, D is the number of random features.

Original: Comparing MSE of POF-MKL with that of OFMKL-Avg, it can be concluded that learning the weights to combine kernels provides higher accuracy than averaging kernels’ predictions.
Revised: Comparing MSE of POF-MKL with that of OFMKL-Avg, it can be concluded that learning the weights to combine kernels provides higher accuracy than averaging kernels’ predictions.

Original: Tableshows that POF-MKL with M = 1 provides lower MSE than eM-KOFL.
Revised: Tableshows that POF-MKL with M = 1 provides lower MSE than eM-KOFL.

Original: Using eM-KOFL, at each time instant, the server receives updates belong to only one kernel.
Revised: Using eM-KOFL, at each time instant, the server receives updates belong to only one kernel.

Original: However, using POF-MKL with
Revised: However, using POF-MKL with

Original: M = 1 , each client sends an update belongs to a kernel which is selected by the client.
Revised: M = 1 , each client sends an update belongs to a kernel which is selected by the client.

Original: Therefore, the server receives updates associated with different kernels even though M = 1 .
Revised: Therefore, the server receives updates associated with different kernels even though M = 1 .

Original: Therefore, experimental results show the effectiveness of the personalized kernel selection provided by POF-MKL.
Revised: Therefore, experimental results show the effectiveness of the personalized kernel selection provided by POF-MKL.

Original: It can be observed that POF-MKL with M = 25 obtains lower MSE than those of POF-MKL with M =and vM-KOFL.
Revised: It can be observed that POF-MKL with M = 25 obtains lower MSE than those of POF-MKL with M =and vM-KOFL.

Original: Since each client is allowed to send at most 1000 parameters per time instant, if clients send updates of all kernels at every time instant as this is the case in vM-KOFL, D cannot be chosen to be greater than 9 .
Revised: Since each client is allowed to send at most 1000 parameters per time instant, if clients send updates of all kernels at every time instant as this is the case in vM-KOFL, D cannot be chosen to be greater than 9 .

Original: However, setting M = 25 , POF-MKL can set D = 20 which can improve the accuracy of online regression task compared to the case where D = 9 .
Revised: However, setting M = 25 , POF-MKL can set D = 20 which can improve the accuracy of online regression task compared to the case where D = 9 .

Original: Note that according to Theorem 2, increase in D increases the probability that the server achieves sub-linear regret with respect to the best function approximator.
Revised: Note that according to Theorem 2, increase in D increases the probability that the server achieves sub-linear regret with respect to the best function approximator.

Original: Furthermore, POF-MKL with M = 51 achieves lower MSE than vM-KOFL even if data samples are distributed i.i.d among clients.
Revised: Furthermore, POF-MKL with M = 51 achieves lower MSE than vM-KOFL even if data samples are distributed i.i.d among clients.

Original: This shows that the proposed POF-MKL can better cope with heterogeneous data among clients which is in agreement with theoretical results in Theorem 1.
Revised: This shows that the proposed POF-MKL can better cope with heterogeneous data among clients which is in agreement with theoretical results in Theorem 1.

Original: In fact, the optimal combination of kernels can be different across clients.
Revised: In fact, the optimal combination of kernels can be different across clients.

Original: Using POF-MKL, each client constructs its own personalized combination of kernels which results in lower MSE compared to vM-KOFL.
Revised: Using POF-MKL, each client constructs its own personalized combination of kernels which results in lower MSE compared to vM-KOFL.

Original: The proposed POF-MKL with M =and M = 25 runs faster than eM-KOFL.
Revised: The proposed POF-MKL with M =and M = 25 runs faster than eM-KOFL.

Original: In fact, using POF-MKL, clients only need to update parameters associated with M kernels while employing vM-KOFL and eM-KOFL, clients have to update parameters of all kernels.
Revised: In fact, using POF-MKL, clients only need to update parameters associated with M kernels while employing vM-KOFL and eM-KOFL, clients have to update parameters of all kernels.

Original: Moreover, POF-MKL obtains lower MSE than PerFedAvg.
Revised: Moreover, POF-MKL obtains lower MSE than PerFedAvg.

Original: Note that since clients are not able to store data in batch, at each time instant clients update PerFedAvg’s model using only the newly observed data sample.
Revised: Note that since clients are not able to store data in batch, at each time instant clients update PerFedAvg’s model using only the newly observed data sample.

Original: Therefore, convergence of PerFedAvg is not guaranteed.
Revised: Therefore, convergence of PerFedAvg is not guaranteed.

Original: Experimental results show that POF-MKL achieves higher accuracy than PerFedAvg in online regression task when it is not possible for clients to store data in batch.
Revised: Experimental results show that POF-MKL achieves higher accuracy than PerFedAvg in online regression task when it is not possible for clients to store data in batch.

Original: Since OFSKL employs only a pre-selected single kernel, OFSKL runs faster than POF-MKL.
Revised: Since OFSKL employs only a pre-selected single kernel, OFSKL runs faster than POF-MKL.

Original: However, utilizing multiple kernels enables POF-MKL to obtain lower MSE than that of OFSKL.
Revised: However, utilizing multiple kernels enables POF-MKL to obtain lower MSE than that of OFSKL.

Original: In fact, using POF-MKL clients learn a linear combination of kernels which is proved to enjoy sub-linear regret with respect to the best kernel in hindsight while employing OFSKL clients have to make predictions using a pre-selected kernel.
Revised: In fact, using POF-MKL clients learn a linear combination of kernels which is proved to enjoy sub-linear regret with respect to the best kernel in hindsight while employing OFSKL clients have to make predictions using a pre-selected kernel.

Original: Furthermore, Figure 1 illustrates the average regret of clients when clients employ vM-KOFL and the proposed POF-MKL with different M parameters.
Revised: Furthermore, Figure 1 illustrates the average regret of clients when clients employ vM-KOFL and the proposed POF-MKL with different M parameters.

Original: From Figure 1, it can be observed that the proposed POF-MKL achieves sub-linear regret.[Table]
Revised: From Figure 1, it can be observed that the proposed POF-MKL achieves sub-linear regret.

Original:  Conclusions
Revised:  Conclusions

Original: The present paper proposed a personalized online federated MKL algorithm called POF-MKL based on RF approximation.
Revised: The present paper proposed a personalized online federated MKL algorithm called POF-MKL based on RF approximation.

Original: Employing the proposed POF-MKL, each client updates the parameters of a subset of kernels which alleviates the computational complexity of the client as well as communication cost of sending updated parameters of kernels.
Revised: Employing the proposed POF-MKL, each client updates the parameters of a subset of kernels which alleviates the computational complexity of the client as well as communication cost of sending updated parameters of kernels.

Original: Theoretical analysis proved that using POF-MKL, each client achieves sub-linear regret with respect to the RF approximation of its best kernel in hindsight which indicates that POF-MKL can deal heterogeneous data among clients. 
Revised: Theoretical analysis proved that using POF-MKL, each client achieves sub-linear regret with respect to the RF approximation of its best kernel in hindsight which indicates that POF-MKL can deal heterogeneous data among clients. While each [Table] [Figure]

Original: While each client updates a subset of kernels, it was proved that the server achieves sub-linear regret with respect to the best function approximator.
Revised: [Table] [Figure] client updates a subset of kernels, it was proved that the server achieves sub-linear regret with respect to the best function approximator.

Original: Experiments on real datasets showcased the advantages of
Revised: Experiments on real datasets showcased the advantages of

Original: POF-MKL compared with other online federated kernel learning algorithms.
Revised: POF-MKL compared with other online federated kernel learning algorithms.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work is supported by NSF ECCS 2207457.
Revised: This work is supported by NSF ECCS 2207457.

Original: PI Yanning Shen also receives support from
Revised: PI Yanning Shen also receives support from

Original: Microsoft Academic Grant Award for AI Research.
Revised: Microsoft Academic Grant Award for AI Research.

Original: Contact: Yanning Shen (yannings@uci.edu)
Revised: Contact: Yanning Shen (yannings@uci.edu).