Original: Personalized Online FederatedMulti-Kernel Learning 
Revised: Personalized Online Federated Learning with Multiple Kernels

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Multi-kernel learning (MKL) exhibits well-documented performance in onlinenon-linear function approximation.
Revised: Multi-kernel learning (MKL) exhibits well-documented performance in online non-linear function approximation.

Original: Federated learning enables a group of learners(called clients) to train an MKL model on the data distributed among clients toperform online non-linear function approximation.
Revised: Federated learning enables a group of learners (called clients) to train an MKL model on the data distributed among clients to perform online non-linear function approximation.

Original: There are some challenges inonline federated MKL that need to be addressed: i) Communication efficiencyespecially when a large number of kernels are considered ii) Heterogeneous datadistribution among clients.
Revised: There are some challenges in online federated MKL that need to be addressed: i) Communication efficiency especially when a large number of kernels are considered ii) Heterogeneous data distribution among clients.

Original: The present paper develops an algorithmic framework toenable clients to communicate with the server to send their updates with affordablecommunication cost while clients employ a large dictionary of kernels.
Revised: The present paper develops an algorithmic framework to enable clients to communicate with the server to send their updates with affordable communication cost while clients employ a large dictionary of kernels.

Original: Utilizingrandom feature (RF) approximation, the present paper proposes scalable onlinefederated MKL algorithm.
Revised: Utilizing random feature (RF) approximation, the present paper proposes scalable online federated MKL algorithm.

Original: We prove that using the proposed online federated MKLalgorithm, each client enjoys sub-linear regret with respect to the RF approximationof its best kernel in hindsight, which indicates that the proposed algorithm caneffectively deal with heterogeneity of the data distributed among clients.
Revised: We prove that using the proposed online federated MKL algorithm, each client enjoys sub-linear regret with respect to the RF approximation of its best kernel in hindsight, which indicates that the proposed algorithm can effectively deal with heterogeneity of the data distributed among clients.

Original: Experimental results on real datasets showcase the advantages of the proposed algorithmcompared with other online federated kernel learning ones.
Revised: Experimental results on real datasets showcase the advantages of the proposed algorithm compared with other online federated kernel learning ones.

Original:  Introduction
Revised:  Introduction

Original: Kernel learning exhibits well-documented performance in function approximation tasks, whileproviding theoretical guarantees associated with different performance metrics, see e.g. [47, 20, 37].
Revised: Kernel learning exhibits well-documented performance in function approximation tasks, while providing theoretical guarantees associated with different performance metrics, see e g. [48, 21, 38].

Original: In some cases, a group of learners aims at collaborating to perform function approximation withoutrevealing their data.
Revised: In some cases, a group of learners aims at collaborating to perform function approximation without revealing their data.

Original: To this end, federated learning has been emerged as a crucial learning paradigmby enabling a group of learners called clients to collaborate with each other by communicating with acentral server to train a centralized model [32, 26, 12, 22].
Revised: To this end, federated learning has been emerged as a crucial learning paradigm by enabling a group of learners called clients to collaborate with each other by communicating with a central server to train a centralized model [33, 27, 12, 23].

Original: Through this process, clients send modelparameters and updates to the server without revealing their data.
Revised: Through this process, clients send model parameters and updates to the server without revealing their data.

Original: Upon receiving updates fromclients, the server updates the model.
Revised: Upon receiving updates from clients, the server updates the model.

Original: Therefore, federated learning enables clients to perform kernellearning for function approximation.
Revised: Therefore, federated learning enables clients to perform kernel learning for function approximation.

Original: In this context, a server and clients collaborate with each otherto learn the optimal kernel.
Revised: In this context, a server and clients collaborate with each other to learn the optimal kernel.

Original: Furthermore, in some practical cases, clients may need to perform thefunction approximation in an online fashion while they are collaborating with the server to learn the kernel.
Revised: Furthermore, in some practical cases, clients may need to perform the function approximation in an online fashion while they are collaborating with the server to learn the kernel.

Original: For example, consider the case where clients may not have enough memory to store data inbatch.
Revised: For example, consider the case where clients may not have enough memory to store data in batch.

Original: In addition, data samples may arrive in a sequential manner such that clients are not able to perform the function approximation in batch form.
Revised: In addition, data samples may arrive in a sequential manner such that clients are not able to perform the function approximation in batch form.

Original: There are major challenges in performing online kernel learning in federated fashion that need to be addressed:
Revised: There are major challenges in performing online kernel learning in federated fashion that need to be addressed:

Original: Communication Efficiency: Communication efficiency arises as a bottleneck in federated learning 
Revised: Communication Efficiency: Communication efficiency arises as a bottleneck in federated learning (see e

Original: (see e.g. [24, 36, 18])
Revised: g. [

Original: Specifically, limited clients-to-server communication bandwidthrestricts the
Revised: Specifically, limited clients -to- server communication bandwidth

Original:  number  can be sent from 
Revised: restricts the number of parameters that can be sent from clients to the

Original: the server
Revised: server

Original: Heterogeneous Data: The distribution of data observed by a client might be different from others(see e.g. [43, 28, 6]).
Revised: Heterogeneous Data: The distribution of data observed by a client might be different from others (see e g. [44, 29, 6]).

Original: Thus, the optimal kernel that is aimed to be learned is different across clients.
Revised: Thus, the optimal kernel that is aimed to be learned is different across clients.

Original: Computational Complexity: Clients should be able to perform function approximation fast enoughin order to make a decision in real-time.
Revised: Computational Complexity: Clients should be able to perform function approximation fast enough in order to make a decision in real-time.

Original: Therefore, the computational complexity of kernel learningmethods should be affordable for clients.
Revised: Therefore, the computational complexity of kernel learning methods should be affordable for clients.

Original: Conventional online kernel learning approaches (see e.g. [20, 38]) suffer from ‘curse of dimensionality’ [3] in the sense that the number of parameters that should be learned increases with the number ofobserved data.
Revised: Conventional online kernel learning approaches (see e.g. [21, 39]) suffer from ‘curse of dimensionality’ [3] in the sense that the number of parameters that should be learned increases with the number of observed data.

Original: This can make employing conventional online kernel learning approaches infeasibleto perform online federated kernel learning since clients may be required to send a large numberof parameter updates to the server while the available clients-to-server communication bandwidthis not enough for sending such information.
Revised: This can make employing conventional online kernel learning approaches infeasible to perform online federated kernel learning since clients may be required to send a large number of parameter updates to the server while the available clients-to-server communication bandwidth is not enough for sending such information.

Original: Approximating kernels by finite-dimensional featurerepresentations (e.g. Nyström method [48] and random feature method of [35]) makes online kernellearning approaches scalable in the sense that the learner can choose the number of parametersthat should be learned, independent of the number of observed data samples (see e.g. [29, 4, 51]).
Revised: Approximating kernels by finite-dimensional feature representations (e.g. Nyström method [49] and random feature method of [36]) makes online kernel learning approaches scalable in the sense that the learner can choose the number of parameters that should be learned, independent of the number of observed data samples (see e.g. [30, 4, 52]).

Original: Employing finite-dimensional feature representations of kernels to perform online federated kernellearning, clients can choose the number of parameters that they should send to the server.
Revised: Employing finite-dimensional feature representations of kernels to perform online federated kernel learning, clients can choose the number of parameters that they should send to the server.

Original: Therefore,finite-dimensional kernel approximation can better cope with limited clients-to-server communicationbandwidth compared to conventional kernel learning approaches.
Revised: Therefore, finite-dimensional kernel approximation can better cope with limited clients-to-server communication bandwidth compared to conventional kernel learning approaches.

Original: Random feature (RF) approximation [35] has been exploited to perform online federated kernel learning when a single pre-selectedkernel function is employed [25, 16].
Revised: Random feature (RF) approximation [36] has been exploited to perform online federated kernel learning when a single pre-selected kernel function is employed [26, 17].

Original: The choice of the kernel function greatly affects the performanceof function approximation when it comes to exploiting only a single kernel function.
Revised: The choice of the kernel function greatly affects the performance of function approximation when it comes to exploiting only a single kernel function.

Original: Employingmultiple kernels instead of a single pre-selected one, can lead to obtaining more accurate functionapproximation since multi-kernel learning (MKL) can learn combination of kernels [23].
Revised: Employing multiple kernels instead of a single pre-selected one, can lead to obtaining more accurate function approximation since multi-kernel learning (MKL) can learn combination of kernels [24].

Original: Onlinefederated MKL algorithms with theoretical guarantees called vM-KOFL and eM-KOFL have beenproposed in [21].
Revised: Online federated MKL algorithms with theoretical guarantees called vM-KOFL and eM-KOFL have been proposed in [22].

Original: However, eM-KOFL and vM-KOFL do not provide personalized MKL models forclients since they learn the same combination of kernels for all clients.
Revised: However, eM-KOFL and vM-KOFL do not provide personalized MKL models for clients since they learn the same combination of kernels for all clients.

Original: The present paper proposes a novel p ersonalized o nline f ederated MKL algorithm called POF-MKLthat provides a personalized MKL model for each client while it is ensured that the available clientsto-server communication bandwidth can afford communication cost of sending clients’ updates to theserver.
Revised: The present paper proposes a novel p ersonalized o nline f ederated MKL algorithm called POF-MKL that provides a personalized MKL model for each client while it is ensured that the available clientsto-server communication bandwidth can afford communication cost of sending clients’ updates to the server.

Original: In order to alleviate the communication cost of MKL, the propsoed POF-MKL employs RFapproximation of kernels and at each time instant, each client chooses a subset of kernels to sendtheir updates to the server instead of sending the updates of all kernels.
Revised: In order to alleviate the communication cost of MKL, the propsoed POF-MKL employs RF approximation of kernels and at each time instant, each client chooses a subset of kernels to send their updates to the server instead of sending the updates of all kernels.

Original: The number of kernels in thechosen subset is selected such that the required bandwidth to send all clients’ updates does not exceedthe available clients-to-server communication bandwidth.
Revised: The number of kernels in the chosen subset is selected such that the required bandwidth to send all clients’ updates does not exceed the available clients-to-server communication bandwidth.

Original: Therefore, clients can send their updatesto the server independent of the number of kernels in the dictionary and as a result a comparativelylarge dictionary of kernels can be considered to perform function approximation.
Revised: Therefore, clients can send their updates to the server independent of the number of kernels in the dictionary and as a result a comparatively large dictionary of kernels can be considered to perform function approximation.

Original: Contributions ofthe present paper can be summarized as follows:
Revised: Contributions of the present paper can be summarized as follows:

Original: c1.
Revised: c1.

Original: Leveraging the proposed POF-MKL, clients can update a subset of kernels’ parameters whichalleviates computational complexity and communication cost of sending updates to the server;c2.
Revised: Leveraging the proposed POF-MKL, clients can update a subset of kernels’ parameters which alleviates computational complexity and communication cost of sending updates to the server; c2.

Original: Through theoretical analysis, it is proved that using the proposed POF-MKL, each client achievessub-linear regret with respect to RF approximation of the best kernel in hindsight associated withthe corresponding client data samples (c.f. Theorem 1).
Revised: Through theoretical analysis, it is proved that using the proposed POF-MKL, each client achieves sub-linear regret with respect to RF approximation of the best kernel in hindsight associated with the corresponding client data samples (c f. Theorem 1).

Original: Moreover, it is guaranteed that the serverachieves sub-linear regret with respect to the best function approximator (c.f. Theorem 2);c3.
Revised: Moreover, it is guaranteed that the server achieves sub-linear regret with respect to the best function approximator (c f. Theorem 2); c3.

Original: Experiments on real datasets showcase the effectiveness of the proposed POF-MKL compared toexisting online federated kernel learning algorithms.
Revised: Experiments on real datasets showcase the effectiveness of the proposed POF-MKL compared to existing online federated kernel learning algorithms.

Original:  Problem Statement and Preliminaries
Revised:  Problem Statement and Preliminaries

Original: Let there be a set of K clients performing function approximation task in an online fashion.
Revised: Let there be a set of K clients performing function approximation task in an online fashion.

Original: The k -thclient’s goal is to learn the function f using the stream of data samples { ( x k,t , y k,t ) }
Revised: The k -th client’s goal is to learn the function f using the stream of data samples { ( x k,t , y k,t ) }

Original: Tt =1 such thatx k,t ∈ R
Revised: Tt =1 such that x k,t ∈ R

Original: d is the data sample observed by the k -th client at time t and y k,t is the label associatedwith x k,t .
Revised: d is the data sample observed by the k -th client at time t and y k,t is the label associated with x k,t .

Original: In the kernel learning context, the function f is assumed to belong to a reproducing kernel
Revised: In the kernel learning context, the function f is assumed to belong to a reproducing kernel

Original: Hilbert space (RKHS).
Revised: Hilbert space (RKHS).

Original: The present paper studies the personalized federated supervised function approximation problem
Revised: The present paper studies the personalized federated supervised function approximation problem

Original: [Equation]
Revised: [Equation]

Original: where H represents the RKHS the function f belongs to and L ( · , · ) denotes the loss function whichcan be defined as
Revised: where H represents the RKHS the function f belongs to and L ( · , · ) denotes the loss function which can be defined as

Original: [Equation]
Revised: [Equation]

Original: where C ( · , · ) denotes the cost function (e.g. least squares function for regression task). Also, λdenotes the regularization coefficient and Ω ( · ) represents a regularizer function to prevent over-fittingand control the model complexity.
Revised: where C ( · , · ) is the cost function (e.g. least squares function for regression task), λ denotes the regularization coefficient and Ω ( · ) represents a regularizer function to prevent over-fitting and control the model complexity.

Original: Let Θ be the global parameters of the function f which arelearned through collaboration of clients with the server while w k be the personalized parameter of thefunction f learned locally by the k -th client.
Revised: Let Θ be the global parameters of the function f which are learned through collaboration of clients with the server while w k be the personalized parameter of the function f learned locally by the k -th client.

Original: Thus, the goal is that the cumulative difference betweenf ( x k,t ; Θ , w k ) and y k,t over time is minimized.
Revised: Thus, the goal is that the cumulative difference between f ( x k,t ; Θ , w k ) and y k,t over time is minimized.

Original: At each time instant t , upon observing the datasample x k,t , the k -th client make the prediction f
Revised: At each time instant t , upon observing the data sample x k,t , the k -th client make the prediction f

Original: ( x k,t ; Θ , w k ) and then observes the true label y k,t .
Revised: ( x k,t ; Θ , w k ) and then observes the true label y k,t .

Original: Therefore, the function approximation problem that the server aims at solving can be expressed asmin Θ (cid:80) 
Revised: Therefore, the function approximation problem that the server aims at solving can be expressed as min Θ (cid:80) Tt =

Original: Tt =1 (cid:80)
Revised: [Equation]

Original: Kk =1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .
Revised: Kk =1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .

Original: Moreover, finding the local parameters w k by the k -thclient can be expressed as the optimization problem min w k (cid:80)
Revised: Moreover, finding the local parameters w k by the k -th client can be expressed as the optimization problem min w k (cid:80)

Original: Tt =
Revised: Tt =

Original: 1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .
Revised: 1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .

Original: In orderto perform the function approximation task in an online fashion, the k -th client needs to perform thetask with the values of Θ and w k at time t denoted by Θ t and w k,t , respectively.
Revised: In order to perform the function approximation task in an online fashion, the k -th client needs to perform the task with the values of Θ and w k at time t denoted by Θ t and w k,t , respectively.

Original: Thus, the values offunction parameters Θ and w k should be updated ‘on the fly’.
Revised: Thus, the values of function parameters Θ and w k should be updated ‘on the fly’.

Original: Since the function f ( · ; · , · ) belongsto a reproducing kernel Hilbert space (RKHS), based on the representer theorem [47], given datasamples, the optimal solution for (1) can be obtained as
Revised: Since the function f ( · ; · , · ) belongs to a reproducing kernel Hilbert space (RKHS), based on the representer theorem [48], given data samples, the optimal solution for (1) can be obtained as

Original: [Equation]
Revised: [Equation]

Original: where κ ( · , · ) denotes symmetric positive definite kernel function such that κ ( x , x ′ ) measures thesimilarity between x and x ′ .
Revised: where κ ( · , · ) denotes symmetric positive definite kernel function such that κ ( x , x ′ ) measures the similarity between x and x ′ .

Original: And α k,t is an unknown coefficient associated with κ ( x , x k,t ) whichis required to be estimated.
Revised: And α k,t is an unknown coefficient associated with κ ( x , x k,t ) which is required to be estimated.

Original: In this case, ˆ f ( · ) in (3) belongs to the RKHS H :
Revised: In this case, ˆ f ( · ) in (3) belongs to the RKHS H :

Original: =
Revised: =

Original: { f ( · ) | f ( x ) =107 (cid:80) ∞ t =
Revised: { f ( · ) | f ( x ) = (cid:80) ∞ t =1 (cid:80)

Original: [Equation] (cid:80) Kk =1 α k,t κ ( x , x k,t ) } such that RKHS norm is defined as ∥ f ∥ 2 H :
Revised: Kk =1 α k,t κ ( x , x k,t ) } such that RKHS norm is defined as ∥ f ∥ 2 H :

Original: = (cid:80) t (cid:80) t ′ α t α t ′ κ ( x t , x t ′ ) .
Revised: = (cid:80) t (cid:80) t ′ α t α t ′ κ ( x t , x t ′ ) .

Original: Furthermore, from (3), it can be inferred that Θ t =
Revised: Furthermore, from (3), it can be inferred that Θ t =

Original: [ α 1 , 1 , . . .
Revised: [ α 1 , 1 , . . .

Original: , α K, 1 , . . .
Revised: , α K, 1 , . . .

Original: , α 1 ,t , . .
Revised: , α 1 ,t , . .

Original: . , α K,t ] .
Revised: . , α K,t ] .

Original: Therefore,the number of coefficients { α k,τ } tτ =1 , ∀ k that should be estimated to obtain ˆ f ( · ) increases overtime.
Revised: Therefore, the number of coefficients { α k,τ } tτ =1 , ∀ k that should be estimated to obtain ˆ f ( · ) increases over time.

Original: This is known as curse of dimensionality [47] since the computational complexity of functionapproximation increases with time.
Revised: This is known as curse of dimensionality [48] since the computational complexity of function approximation increases with time.

Original: This brings challenge for federated implementation of functionapproximation since dimension of updates that should be sent to the server by each client grows overtime and when T is large, the available communication bandwidth may not be enough for clients tosend their updates.
Revised: This brings challenge for federated implementation of function approximation since dimension of updates that should be sent to the server by each client grows over time and when T is large, the available communication bandwidth may not be enough for clients to send their updates.

Original: In order to deal with the increasing number of unknown coefficients, one can employ random Fourierapproximation [35].
Revised: In order to deal with the increasing number of unknown coefficients, one can employ random Fourier approximation [36].

Original: Assume that κ ( · ) is a shift-invariant kernel meaning that κ ( x , x ′ )
Revised: Assume that κ ( · ) is a shift-invariant kernel meaning that κ ( x , x ′ )

Original: = κ ( x − x ′ ) .
Revised: = κ ( x − x ′ ) .

Original: Let π κ ( ρ ) denotes the Fourier transform of κ ( · ) .
Revised: Let π κ ( ρ ) denotes the Fourier transform of κ ( · ) .

Original: If the kernel function κ ( · ) is normalized such thatκ ( 0 ) = 1 , then π κ ( ρ ) can be viewed as a probability density function (PDF) (see e.g. [35]).
Revised: If the kernel function κ ( · ) is normalized such that κ ( 0 ) = 1 , then π κ ( ρ ) can be viewed as a probability density function (PDF) (see e.g. [36]).

Original: Letρ 1 , . .
Revised: Let ρ 1 , . .

Original: , ρ D be a set of D independent and identically distributed (i.i.d) vectors drawn from π κ ( · ) .
Revised: , ρ D be a set of D independent and identically distributed (i.i.d) vectors drawn from π κ ( · ) .

Original: Let the vector z ( x ) be defined as
Revised: Let the vector z ( x ) be defined as

Original: [Equation]
Revised: [Equation]

Original: Then, ˆ κ r ( x − x ′ )
Revised: Then, ˆ κ r ( x − x ′ )

Original: =
Revised: =

Original: z ( x ) ⊤ z ( x ′ ) constitutes an unbiased estimator of κ ( x − x ′ ) and the randomfeature (RF) approximation of ˆ f ( x ) in (3) can be obtained as[Equation]
Revised: z ( x ) ⊤ z ( x ′ ) constitutes an unbiased estimator of κ ( x − x ′ ) and the random feature (RF) approximation of ˆ f ( x ) in (3) can be obtained as

Original: [Equation]
Revised: [Equation]

Original: where in this case θ = (cid:80) Tt =1 (cid:80)
Revised: where in this case θ = (cid:80) Tt =1 (cid:80)

Original: Kk =1 α k,t z ( x k,t ) .
Revised: Kk =1 α k,t z ( x k,t ) .

Original: According to (4), z ( x k,t ) is a 2 D vector and asa result it can be concluded that θ is a 2 D vector as well.
Revised: According to (4), z ( x k,t ) is a 2 D vector and as a result it can be concluded that θ is a 2 D vector as well.

Original: Therefore, using RF approximation, thevector θ should be estimated whose dimension does not grow over time.
Revised: Therefore, using RF approximation, the vector θ should be estimated whose dimension does not grow over time.

Original: The performance of a kernel learning algorithm depends on the choice of the kernel.
Revised: The performance of a kernel learning algorithm depends on the choice of the kernel.

Original: Thus, performing the function approximation using a pre-selected kernel requires prior information whichmay not be available.
Revised: Thus, performing the function approximation using a pre-selected kernel requires prior information which may not be available.

Original: To cope with this, employing a dictionary of kernels in lieu of a pre-selectedsingle kernel has been proposed in the literature (see e.g. [44, 23, 30]).
Revised: To cope with this, employing a dictionary of kernels in lieu of a pre-selected single kernel has been proposed in the literature (see e.g. [45, 24, 31]).

Original: Specifically, the kernel is learned as a combination of kernels in the dictionary.
Revised: Specifically, the kernel is learned as a combination of kernels in the dictionary.

Original: Let κ 1 ( · ) , . .
Revised: Let κ 1 ( · ) , . .

Original: , κ N ( · ) be a set of
Revised: , κ N ( · ) be a set of

Original: N kernels where κ i ( · ) denotes the i -th kernel.
Revised: N kernels where κ i ( · ) denotes the i -th kernel.

Original: The function ¯ κ ( · ) belongs to the convex hull
Revised: The function ¯ κ ( · ) belongs to the convex hull

Original: K :=
Revised: K :=

Original: { ¯ κ ( x ) = (cid:80)
Revised: { ¯ κ ( x ) = (cid:80)

Original: Ni
Revised: Ni

Original: =1 β
Revised: =1 β

Original: i κ i ( x ) , β i ≥ 0 , ∀ i, (cid:80)
Revised: i κ i ( x ) , β i ≥ 0 , ∀ i, (cid:80)

Original: Ni =
Revised: Ni =

Original: 1 β i = 1 } is a kernel [40].
Revised: 1 β i = 1 } is a kernel [41].

Original: Therefore, in onlinemulti-kernel learning, the goal is to learn the convex combination of kernels in the dictionary tominimize the cumulative regret with respect to the best function approximator in hindsight.
Revised: Therefore, in online multi-kernel learning, the goal is to learn the convex combination of kernels in the dictionary to minimize the cumulative regret with respect to the best function approximator in hindsight.

Original: Thecumulative regret is defined as the cumulative difference between loss of the online multi-kernellearning algorithm and that of the best function approximator in hindsight.
Revised: The cumulative regret is defined as the cumulative difference between loss of the online multi-kernel learning algorithm and that of the best function approximator in hindsight.

Original: Furthermore, for adataset { ( x t , y t ) } Tt =1 , the best function approximator is f ∗ ( · ) ∈ arg min
Revised: Furthermore, for a dataset { ( x t , y t ) } Tt =1 , the best function approximator is f ∗ ( · ) ∈ arg min

Original: f ∗ i ,i ∈
Revised: f ∗ i ,i ∈

Original: [ N ] (cid:80) Tt =1 L ( f ∗ i ( x t ) , y t )
Revised: [ N ] (cid:80) Tt =1 L ( f ∗ i ( x t ) , y t )

Original: where f ∗ i ( ·
Revised: where f ∗ i ( ·

Original: ) ∈ arg min f ∈ H
Revised: ) ∈ arg min f ∈ H

Original: i (cid:80) Tt =
Revised: i (cid:80) Tt =

Original: 1 L ( f ( x t ) , y t ) such that H
Revised: 1 L ( f ( x t ) , y t ) such that H

Original: i is an RKHS induced by κ i ( · ) and[ N ] := { 1 , . . .
Revised: i is an RKHS induced by κ i ( · ) and [ N ] := { 1 , . . .

Original: , N } .
Revised: , N } .

Original: Enabled by random feature approximation, centralized and scalable online multikernel learning algorithms have been proposed in literature (see e.g. [39, 42, 15]).
Revised: Enabled by random feature approximation, centralized and scalable online multikernel learning algorithms have been proposed in literature (see e.g. [40, 43, 15]).

Original: The present paperproposes an algorithmic framework for personalized online federated MKL using RF approximationof kernels in the dictionary.
Revised: The present paper proposes an algorithmic framework for personalized online federated MKL using RF approximation of kernels in the dictionary.

Original:  Personalized Online Federated Multi-Kernel Learning
Revised:  Personalized Online Federated Multi-Kernel Learning

Original: The present section proposes an algorithmic framework for online federated multi-kernel learning which can deal with heterogeneous data among clients.
Revised: The present section proposes an algorithmic framework for online federated multi-kernel learning which can deal with heterogeneous data among clients.

Original: To perform function approximation, RF approximations of kernel functions are employed.
Revised: To perform function approximation, RF approximations of kernel functions are employed.

Original: For the i -th kernel κ i , vectors ρ i, 1 , . .
Revised: For the i -th kernel κ i , vectors ρ i, 1 , . .

Original: , ρ i,D are drawn i.i.d from π κ i ( · ) to construct the random feature
Revised: , ρ i,D are drawn i.i.d from π κ i ( · ) to construct the random feature

Original: vector z i ( x ) =1 √ D [sin( ρ ⊤ i, 1 x ) , . .
Revised: vector z i ( x ) =√ D [sin( ρ ⊤ i, 1 x ) , . .

Original: , sin( ρ ⊤ i,D x ) , cos( ρ ⊤ i, 1 x ) , . . .
Revised: , sin( ρ ⊤ i,D x ) , cos( ρ ⊤ i, 1 x ) , . . .

Original: , cos( ρ ⊤ i,D x )] .
Revised: , cos( ρ ⊤ i,D x )] .

Original: Then, at time instant t , the randomfeature approximation associated with κ i ( · ) can be obtained as ˆ f RF ,it ( x ) =
Revised: Then, at time instant t , the random feature approximation associated with κ i ( · ) can be obtained as ˆ f RF ,it ( x ) =

Original: θ ⊤ i,t z i ( x ) where θ i,t isthe global function parameter associated the i -th kernel at time t .
Revised: θ ⊤ i,t z i ( x ) where θ i,t is the global function parameter associated the i -th kernel at time t .

Original: 3.1 Algorithm
Revised: 3.1 Algorithm

Original: At each time instant t , the server transmits global function parameters θ i,t , ∀ i ∈
Revised: At each time instant t , the server transmits global function parameters θ i,t , ∀ i ∈

Original: [ N ] to all clients.
Revised: [ N ] to all clients.

Original: Thek -th client, assigns the weight w ik,t to the i -th kernel which indicates the confidence of the k -th clientat time t in the function approximation given by the i -th kernel.
Revised: The k -th client, assigns the weight w ik,t to the i -th kernel which indicates the confidence of the k -th client at time t in the function approximation given by the i -th kernel.

Original: Upon receiving new data samplex k,t , the k -th client performs the function approximation combining kernels’ RF approximations asˆ f ( x k,t ; ˆ Θ t , w k,t ) = N (cid:88) i = w ik,t W k,t θ ⊤ i,t z i ( x k,t ) =
Revised: Upon receiving new data sample x k,t , the k -th client performs the function approximation combining kernels’ RF approximations as

Original: N (cid:88)

Original: i

Original: [Equation]
Revised: [Equation]

Original: (6) where ˆ Θ t = [ θ 1 ,t , . ..
Revised: where ˆ Θ t = [ θ 1 ,t , . .

Original: , θ N,t ] , w k,t
Revised: , θ N,t ] , w k,t

Original: =
Revised: =

Original: [ w 1 k,t , . . .
Revised: [ w 1 k,t , . . .

Original: , w Nk,t ] and W k,t = (cid:80) Ni =1 w ik,t .
Revised: , w Nk,t ] and W k,t = (cid:80) Ni =1 w ik,t .

Original: As it can beinferred from (6), each client constructs its own personalized combination of kernels.
Revised: As it can be inferred from (6), each client constructs its own personalized combination of kernels.

Original: Upon observingthe true label y k,t , the k -th client calculates the losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .
Revised: Upon observing the true label y k,t , the k -th client calculates the losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .

Original: Then, thek -th client leverages calculated losses to locally update both global and local parameters.
Revised: Then, the k -th client leverages calculated losses to locally update both global and local parameters.

Original: Let θ ik,t +1and w ik,t +1 denote the k -th client’s local updates of θ i,t and w ik,t , respectively.
Revised: Let θ ik,t + and w ik,t +1 denote the k -th client’s local updates of θ i,t and w ik,t , respectively.

Original: Specifically, the k -thclient utilizes multiplicative update rule to update w ik,t as
Revised: Specifically, the k -th client utilizes multiplicative update rule to update w ik,t as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where η k is the learning rate of the k -th client.
Revised: where η k is the learning rate of the k -th client.

Original: Note that the k -th client ( ∀ k ∈
Revised: Note that the k -th client ( ∀ k ∈

Original: [ K ] ) does not send itsupdated local parameter w k,t +1 to the server.
Revised: [ K ] ) does not send its updated local parameter w k,t +1 to the server.

Original: Clients send their locally updated global parameters tothe server (i.e. θ ik,t +1 ).
Revised: Clients send their locally updated global parameters to the server (i.e. θ ik,t +1 ).

Original: Aggregating local updates, the server updates global parameters to ˆ Θ t +1 .
Revised: Aggregating local updates, the server updates global parameters to ˆ Θ t +1 .

Revised: If all clients send updates associated with all kernels (i.e. θ ik,t +1 , ∀ i ∈ [ N ] ), this requires sending 2 ND parameters by each client at each time instant.

Revised: When the number of both clients and kernels are large, the available client-to-server communication bandwidth may not be enough to afford sending 2 NDK parameters per time instant even for small values of D .

Revised: Note that reducing N and D degrade the performance of online federated MKL.

Revised: Reducing N (the number of kernels), decreases the flexibility of clients to construct their ideal kernel using convex combination of kernels in the dictionary.

Original: Algorithm
Revised: Reducing D can degrade the accuracy of RF approximation. The present paper proposes an algorithmic framework to enable clients to perform online function approximation with sufficiently large dictionary of kernels while the available clients-to-server communication bandwidth can afford sending updates from clients to the server when a desirable value for the number of random features D is chosen. To this end, at each time instant, each client randomly chooses a subset of M kernels among all N kernels in the dictionary. Then, each client updates and sends the global parameters of the chosen M kernels to the server instead of updating and sending global parameters of all kernels.

Revised: To choose a subset of M kernels, each client splits

Revised: Algorithm

Original: Input: Weights w ik,t , ∀ i ∈ [ N ] , parameter M and exploration rate 0 < ξ k ≤ 1 .
Revised: Input: Weights w ik,t , ∀ i ∈ [ N ] , parameter M and exploration rate 0 < ξ k ≤ 1 .

Original: Sort the kernels in descending order with respect to weights { w ik,t } Ni =1 .
Revised: Sort the kernels in descending order with respect to weights { w ik,t } Ni =1 .

Original: Obtain the index sequence s 1 , . . .
Revised: Obtain the index sequence s 1 , . . .

Original: , s N such that w s b k,t ≤ w s a k,t
Revised: , s N such that w s b k,t ≤ w s a k,t

Original: if b > a , ∀ a, b ∈ [ N ] .
Revised: if b > a , ∀ a, b ∈ [ N ] .

Original: Open bin B 1 and initialize j = 1 .
Revised: Open bin B 1 and initialize j = 1 .

Original: for all i ∈ [ N ] , the k -th client do if the bin B j includes less than M kernels then
Revised: for all i ∈ [ N ] , the k -th client do if the bin B j includes less than M kernels then

Original: Adds the s i -th kernel to B j . else
Revised: Adds the s i -th kernel to B j . else

Original: Opens new bin B j +1 , adds the s i -th kernel to B j +1 and updates j ← j + 1 .
Revised: Opens new bin B j +1 , adds the s i -th kernel to B j +1 and updates j ← j + 1 .

Original: end if end for
Revised: end if end for

Original: Draw an index I k,t via PMF q k,t in (8).
Revised: Draw an index I k,t via PMF q k,t in (8).

Original: Output: S k,t : indices set of kernels in the selected bin B I k,tIf all clients send updates associated with all kernels (i.e. θ ik,t +1 , ∀ i ∈ [ N ] ), this requires sending2 ND parameters by each client at each time instant.When the number of both clients and kernelsare large, the available client-to-server communication bandwidth may not be enough to affordsending 2 NDK parameters per time instant even for small values of D . Note that reducing N and Ddegrade the performance of online federated MKL.
Revised: Output: S k,t : indices set of kernels in the selected bin B I k,t kernels into some bins and draws randomly one of the bins at each time instant.

Original: Reducing N (the number of kernels), decreasesthe flexibility of clients to construct their ideal kernel using convex combination of kernels in thedictionary.

Original: Reducing D can degrade the accuracy of RF approximation.

Original: The present paper proposes an algorithmic framework to enable clients to perform online functionapproximation with sufficiently large dictionary of kernels while the available clients-to-servercommunication bandwidth can afford sending updates from clients to the server when a desirablevalue for the number of random features D is chosen.

Original: To this end, at each time instant, each clientrandomly chooses a subset of M kernels among all N kernels in the dictionary.

Original: Then, each clientupdates and sends the global parameters of the chosen M kernels to the server instead of updatingand sending global parameters of all kernels.

Original: To choose a subset of M kernels, each client splitskernels into some bins and draws randomly one of the bins at each time instant.

Original: Each bin containsat most M kernels and each client updates and sends global parameters associated with kernels inthe chosen bin.
Revised: Each bin contains at most M kernels and each client updates and sends global parameters associated with kernels in the chosen bin.

Original: In order to distribute kernels among bins, at first the k -th client sorts kernels indescending order according to kernels’ weights { w ik,t } Ni =1 .
Revised: In order to distribute kernels among bins, at first the k -th client sorts kernels in descending order according to kernels’ weights { w ik,t } Ni =1 .

Original: Let B j represents the j -th bin of kernels.
Revised: Let B j represents the j -th bin of kernels.

Original: The k -th client adds kernels from sorted list one by one to B j until either all kernels are assigned to abin or the number of kernels in B j reaches M .
Revised: The k -th client adds kernels from sorted list one by one to B j until either all kernels are assigned to a bin or the number of kernels in B j reaches M .

Original: When there are some kernels that are not assigned toany bins while there are M kernels in B j , the k -th client opens the bin B j +1 and adds kernels to thisbin.
Revised: When there are some kernels that are not assigned to any bins while there are M kernels in B j , the k -th client opens the bin B j +1 and adds kernels to this bin.

Original: This continues until all kernels are assigned to a bin.
Revised: This continues until all kernels are assigned to a bin.

Original: As it can be inferred from the procedureof distributing kernels into bins, the number of bins at every client is m = (cid:6) N M (cid:7) .
Revised: As it can be inferred from the procedure of distributing kernels into bins, the number of bins at every client is m = (cid:6) NM (cid:7) .

Original: Furthermore, itcan be concluded that B 1 includes M kernels with the largest weights while the bin B m includes
Revised: Furthermore, it can be concluded that B 1 includes M kernels with the largest weights while the bin B m includes

Original: N − ( m − 1) M kernels with lowest weights.
Revised: N − ( m − 1) M kernels with lowest weights.

Original: The k -th client assigns the weight u jk,t at time t to B jdefined as u jk,t = (cid:80)
Revised: The k -th client assigns the weight u jk,t at time t to B j defined as u jk,t = (cid:80)

Original: κ i ∈B j w ik,t .
Revised: κ i ∈B j w ik,t .

Original: The k -th client draws one of the bins according to the probabilitymass function (PMF) q k,t defined as
Revised: The k -th client draws one of the bins according to the probability mass function (PMF) q k,t defined as

Original: [Equation]
Revised: [Equation]

Original: (8) where U k,t = (cid:80) mj =1 u jk,t and 0 < ξ k ≤ 1 is an exploration rate determined by the k -th client.
Revised: where U k,t = (cid:80) mj =1 u jk,t and 0 < ξ k ≤ 1 is an exploration rate determined by the k -th client.

Original: Let
Revised: Let

Original: I k,t be the index of the chosen bin by the k -th client at time t .
Revised: I k,t be the index of the chosen bin by the k -th client at time t .

Original: The PMF in (8) constitutes trade-offbetween exploitation and exploration.
Revised: The PMF in (8) constitutes trade-off between exploitation and exploration.

Original: According to the first term in the right hand side of (8), itis more probable that the k -th client draws a bin which includes kernels with larger weights w ik,t .
Revised: According to the first term in the right hand side of (8), it is more probable that the k -th client draws a bin which includes kernels with larger weights w ik,t .

Original: Hence, it is more probable that the k -th client collaborates in updating the global parameters of akernel with larger weight w ik,t .
Revised: Hence, it is more probable that the k -th client collaborates in updating the global parameters of a kernel with larger weight w ik,t .

Original: Let S k,t denotes the set which includes the indices of kernels in thechosen bin at time t .
Revised: Let S k,t denotes the set which includes the indices of kernels in the chosen bin at time t .

Original: The Algorithm 1 summarizes the procedure that the k -th client determines theset S k,t .
Revised: The Algorithm 1 summarizes the procedure that the k -th client determines the set S k,t .

Original: According to Algorithm 1, kernel subset selection is personalized since each client choosesits own subset of kernels to update their parameters.
Revised: According to Algorithm 1, kernel subset selection is personalized since each client chooses its own subset of kernels to update their parameters.

Original: Input: Kernels κ i , i  = [-1 , ..., N , learning rate η > 
-]
Revised: Let p ik,t denotes the probability that i ∈ S k,t . Then p ik,t = [-1 , ..., N , learning rate η > 
-]q b i k,t where b i is the index of the bin which includes the i -th kernel. The k -th client updates global parameters locally as follows

Revised: [Equation]

Revised: where 1 i ∈S k,t denotes an indicator function and it is 1 when i ∈ S k,t .

Original: {+The update rule in (9) implies that when i / ∈ S k,t , the k -th client does not update θ i,t (i.e. θ ik,t +
+}
Revised: {+The update rule in (9) implies that when i / ∈ S k,t , the k -th client does not update θ i,t (i.e. θ ik,t +
+}

Revised: = θ i,t ).

Revised: Therefore, the k -th client sends θ ik,t +1 to the server only if i ∈ S k,t .

Revised: Therefore, at each time instant, each client needs to send at most 2 MD parameters to the server.

Revised: Let C i,t be a set of client indices such that k ∈ C i,t if the k -th client sends θ ik,t +1 to the server.

Revised: Upon aggregating updates from clients, the server updates θ i,t as θ i,

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: Algorithm 2 summarizes the proposed personalized online federated multi-kernel learning algorithm called POF-MKL.

Revised: It is useful to note that using our proposed POF-MKL, the server cannot find the

Original: {+Input: Kernels κ i , i = 1 , ..., N , learning rate η > 
+}
Revised: {+Input: Kernels κ i , i = 1 , ..., N , learning rate η > 
+}

Original: and the number of random features D .
Revised: and the number of random features D .

Original: Initialize: θ i, 1 = 0 , w ik, 1 = 1 , ∀ i ∈
Revised: Initialize: θ i, 1 = 0 , w ik, 1 = 1 , ∀ i ∈

Original: [ N ] , ∀ k ∈
Revised: [ N ] , ∀ k ∈

Original: [ K ] .
Revised: [ K ] .

Original: for t = 1 , . . .
Revised: for t = 1 , . . .

Original: , T do
Revised: , T do

Original: The server transmits the global parameters ˆ Θ t =
Revised: The server transmits the global parameters ˆ Θ t =

Original: [ θ 1 ,t , . .
Revised: [ θ 1 ,t , . .

Original: , θ N,t ] to all clients.
Revised: , θ N,t ] to all clients.

Original: for all k ∈ [ K ] , the k th client do
Revised: for all k ∈ [ K ] , the k th client do

Original: Receive one datum x k,t .
Revised: Receive one datum x k,t .

Original: Predicts ˆ f ( x k,t ; ˆ Θ t , w k,t ) via (6).
Revised: Predicts ˆ f ( x k,t ; ˆ Θ t , w k,t ) via (6).

Original: Calculates losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .
Revised: Calculates losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .

Original: Updates 
Revised: Updates w ik,t +1 , ∀ i ∈ [ N ] via (7).

Original: Selects a subset of kernel  Algorithm end for
Revised: Selects a subset of kernel indices S k,t using Algorithm 1.

Original: The server updates θ 
Revised: Updates θ ik,t +1 , ∀ i ∈

Original: N end for Let p ik,t denotes the probability that i ∈ S k,t .
Revised: S k,t via (9) and sends θ ik,t +1 , ∀

Original: Then p ik,t = q b i k,t where b i is the index of the binwhich includes the i -th kernel. The k -th client updates global parameters locally as follows [Equation]
Revised: i ∈

Original: (9) where 1 i ∈S k,t denotes an indicator function and it is 1 when i ∈ S k,t . The update rule in (9) impliesthat when i / ∈ S k,t , the [-k -th client does not update θ i,t (i.e. θ ik,t +
-]
Revised: S k,t to the [-k -th client does not update θ i,t (i.e. θ ik,t +
-]server.

Original: = θ i,t ). Therefore, the k -th clientsends θ ik,t +1 to the server only if i ∈ S k,t . Therefore, at each time instant, each client needs to sendat most 2 MD parameters to the server. Let C i,t be a set of client indices such that k ∈ C i,t if the k -thclient sends θ ik,t +1 to the server.
Revised: end for

Original: Upon aggregating updates from clients, the server updates θ i,t as
Revised: The server updates θ i,t +1 , ∀ i ∈

Original: [Equation]
Revised: [ N ] via (10).

Original: i [Equation] Algorithm 2 summarizes the proposed personalized online federated multi-kernel learning algorithmcalled POF-MKL. It is useful to note that using our proposed POF-MKL, the server cannot find thegradients ∇L ( θ ⊤ i,
Revised: end for gradients

Original: t z i ( x k,t ) , y k,t ) from updates received from clients.
Revised: ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) from updates received from clients.

Original: Instead, the server can find∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p
Revised: Instead, the server can find ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p

Original: ik,t where p ik,t is a time-varying value determined locally by the k -th client.
Revised: ik,t where p ik,t is a time-varying value determined locally by the k -th client.

Original: This can promote the privacy of the proposed POF-MKL since exchanging the gradients can behazardous to the privacy of federated learning (see e.g. [52, 14]).
Revised: This can promote the privacy of the proposed POF-MKL since exchanging the gradients can be hazardous to the privacy of federated learning (see e.g. [53, 14]).

Original: Complexity.
Revised: Complexity.

Original: Each client needs to store d -dimensional D random feature vectors for each kernel.
Revised: Each client needs to store d -dimensional D random feature vectors for each kernel.

Original: Therefore, the memory requirement of each client to implement function approximation using POFMKL is O ( dND ) .
Revised: Therefore, the memory requirement of each client to implement function approximation using POFMKL is O ( dND ) .

Original: Using POF-MKL, at each time instant, each client needs to perform O ( dND )operations including inner products and summations.
Revised: Using POF-MKL, at each time instant, each client needs to perform O ( dND ) operations including inner products and summations.

Original: Furthermore, when ξ k < 1 , in order to choosea subset of kernels, the k -th client needs to sort kernels which imposes worst case computationalcomplexity of O ( N log N ) .
Revised: Furthermore, when ξ k < 1 , in order to choose a subset of kernels, the k -th client needs to sort kernels which imposes worst case computational complexity of O ( N log N ) .

Original: However, when ξ k = 1 , according to PMF in (8), the k -th client choosesone bin uniformly at random and as a result in this case the k -th client does not need to sort kernels.
Revised: However, when ξ k = 1 , according to PMF in (8), the k -th client chooses one bin uniformly at random and as a result in this case the k -th client does not need to sort kernels.

Original: Therefore, setting ξ k < 1 , the computational complexity for the k -th client is O ( dND + N log N )while setting ξ k = 1 , the computational complexity of the k -th client at each time instant is O ( dND ) .
Revised: Therefore, setting ξ k < 1 , the computational complexity for the k -th client is O ( dND + N log N ) while setting ξ k = 1 , the computational complexity of the k -th client at each time instant is O ( dND ) .

Original: 3.2 Regret Analysis
Revised: 3.2 Regret Analysis

Original: The present section analyzes the regret of the proposed POF-MKL.
Revised: The present section analyzes the regret of the proposed POF-MKL.

Original: Specifically, two types of regret
Revised: Specifically, two types of regret

Original: R k,T and R s,T are considered for the k -th client and the server, respectively.
Revised: R k,T and R s,T are considered for the k -th client and the server, respectively.

Original: The performance of thek -th client utilizing POF-MKL is analyzed in terms of regret defined as
Revised: The performance of the k -th client utilizing POF-MKL is analyzed in terms of regret defined as

Original: [Equation]
Revised: [Equation]

Original: where R k,T measures the cumulative difference between the loss of the k -th client and the lossof the RF approximation of the kernel with minimum loss among all kernels’ RF approximations.
Revised: where R k,T measures the cumulative difference between the loss of the k -th client and the loss of the RF approximation of the kernel with minimum loss among all kernels’ RF approximations.

Original: Let α ∗ ik,t ,
Revised: Let α ∗ ik,t ,

Original: ∀ t ∈
Revised: ∀ t ∈

Original: [ T ] , ∀ k ∈
Revised: [ T ] , ∀ k ∈

Original: [ K ] represents the optimal coefficients associated with the i -th kernelsuch that f ∗ i ( x )
Revised: [ K ] represents the optimal coefficients associated with the i -th kernel such that f ∗ i ( x )

Original: = (cid:80) Tt =1 (cid:80)
Revised: = (cid:80) Tt =1 (cid:80)

Original: Kk
Revised: Kk

Original: =1 α ∗ ik,t κ i ( x , x k,t ) .
Revised: =1 α ∗ ik,t κ i ( x , x k,t ) .

Original: Then the best function approximator is definedas f ∗ ( · ) ∈ arg min f ∗ i ,i ∈
Revised: Then the best function approximator is defined as f ∗ ( · ) ∈ arg min f ∗ i ,i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) .
Revised: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) .

Original: Furthermore, the regret of the server isdefined as the cumulative difference between the loss of POF-MKL and that of the best functionapproximator over all data samples distributed among clients which can be expressed as
Revised: Furthermore, the regret of the server is defined as the cumulative difference between the loss of POF-MKL and that of the best function approximator over all data samples distributed among clients which can be expressed as

Original: [Equation]
Revised: [Equation]

Original: In order to analyze the regret of POF-MKL, suppose that the following assumptions hold true:
Revised: In order to analyze the regret of POF-MKL, suppose that the following assumptions hold true:

Original: (as1) L ( θ ⊤ i,t z
Revised: (as1) L ( θ ⊤ i,t z

Original: i ( x k,t ) , y k,t ) , ∀ k ∈
Revised: i ( x k,t ) , y k,t ) , ∀ k ∈

Original: [ K ] is convex with respect to θ i,t at each time instant t .
Revised: [ K ] is convex with respect to θ i,t at each time instant t .

Original: (as2)
Revised: (as2)

Original: For θ in a bounded set satisfying ∥ θ ∥ ≤ C , the loss function and its gradient are bounded 
Revised: For θ in a bounded set satisfying ∥ θ ∥ ≤ C , the loss function and its gradient are bounded as 0 ≤ L ( θ ⊤ z

Original: as 0 ≤ L ( θ ⊤ z i ( x k,t ) , y k,t ) ≤ 1 and ∥∇L ( θ ⊤ z i ( x k,t ) , y k,t ) ∥ ≤ L .
Revised: i ( x k,t ) , y k,t ) ≤ 1 and ∥∇L ( θ ⊤ z i ( x k,t ) , y k,t ) ∥ ≤ L .

Original: Moreover, each data sample is bounded as ∥ x k,t ∥ ≤ 1 , ∀ k ∈
Revised: Moreover, each data sample is bounded as ∥ x k,t ∥ ≤ 1 , ∀ k ∈

Original: [ K ] , ∀ t ∈ [ T ] .
Revised: [ K ] , ∀ t ∈ [ T ] .

Original: (as3) Kernels κ i ( · ) , ∀ i ∈
Revised: (as3) Kernels κ i ( · ) , ∀ i ∈

Original: [ N ] are shift-invariant with κ i ( 0 )
Revised: [ N ] are shift-invariant with κ i ( 0 )

Original: = 1 , ∀ i ∈
Revised: = 1 , ∀ i ∈

Original: [ N ] .
Revised: [ N ] .

Original: The following theorem investigates the regret of the k -th client according to the k -th client data.
Revised: The following theorem investigates the regret of the k -th client according to the k -th client data.

Original: Theproof of the following Theorem can be found in Appendix A.
Revised: The proof of the following Theorem can be found in Appendix A.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under (as1)–(as3), the regret of the k -th client with respect to the best kernel satisfies
Revised: Under (as1)–(as3), the regret of the k -th client with respect to the best kernel satisfies

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 shows that by setting η k
Revised: Theorem 1 shows that by setting η k

Original: = O (cid:16) 1 √ T (cid:17) , the k -th client achieves sub-linear regret of O ( √ T ) .
Revised: = O (cid:16) 1 √ T (cid:17) , the k -th client achieves sub-linear regret of O ( √ T ) .

Original: Furthermore, Theorem 1 shows that POF-MKL can deal with heterogeneous data among clients sincethe regret of each client defined in (11) is calculated with respect to the corresponding client data.
Revised: Furthermore, Theorem 1 shows that POF-MKL can deal with heterogeneous data among clients since the regret of each client defined in (11) is calculated with respect to the corresponding client data.

Original: The following theorem studies the regret of the server with respect to the best function approximator.
Revised: The following theorem studies the regret of the server with respect to the best function approximator.

Original: The proof can be found in Appendix B.
Revised: The proof can be found in Appendix B.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let i ∗ :=
Revised: Let i ∗ :=

Original: arg min i ∈
Revised: arg min i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) and σ i be the second Fouriermoment of the i -th kernel.
Revised: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) and σ i be the second Fourier moment of the i -th kernel.

Original: Under (as1)–(as3), the regret of the server with respect to the best functionapproximator satisfies
Revised: Under (as1)–(as3), the regret of the server with respect to the best function approximator satisfies

Original: T (cid:88) t

Original: [Equation]
Revised: [Equation]

Original: K (cid:88) k [Equation] L ( ˆ f ( x k
Revised: with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) where C :=

Original: [Equation]

Original: t ; ˆ Θ t

Original: [Equation] w k [Equation] t
Revised: max i ∈

Original: [Equation]

Original: t

Original: [Equation]

Original: −

Original: T (cid:88) t

Original: [Equation]

Original: K (cid:88) k

Original: =

Original: L ( f ∗ ( x k,t

Original: [Equation]

Original: (14) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) where C := max i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk
Revised: Kk

Original: =1 α ∗ ik,t .
Revised: =1 α ∗ ik,t .

Original: As it can be inferred from (14), the regret of the server with respect to the best function approximatordepends on 1 p i ∗ k,t .
Revised: As it can be inferred from (14), the regret of the server with respect to the best function approximator depends on 1 p i ∗ k,t .

Original: From (8) and the fact that p ik,t = q b i k,t , it can be concluded that p i ∗ k,t > ξ k m .
Revised: From (8) and the fact that p ik,t = q b i k,t , it can be concluded that p i ∗ k,t > ξ k m .

Original: Thus,setting ξ k = O (1) , then p ik,t > O ( MN ) .
Revised: Thus, setting ξ k = O (1) , then p ik,t > O ( MN ) .

Original: The regret bound in (14) shows that setting η = O (cid:16)(cid:113) MNT (cid:17)and ϵ = η k = 1 √ T , ∀ k ∈
Revised: The regret bound in (14) shows that setting η = O (cid:16)(cid:113) MNT (cid:17) and ϵ = η k = 1 √ T , ∀ k ∈

Original: [ K ] , the server obtains regret of O (cid:16)(cid:113) NM T (cid:17) with probability at least1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) .
Revised: [ K ] , the server obtains regret of O (cid:16)(cid:113) NM T (cid:17) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) .

Original: This shows that increasing M tighten the regret bound and increasing
Revised: This shows that increasing M tighten the regret bound and increasing

Original: D increases the probability that the regret bound in (14) holds true.
Revised: D increases the probability that the regret bound in (14) holds true.

Original: However, using POF-MKL, eachclient needs to transmit MD parameters at each time instant.
Revised: However, using POF-MKL, each client needs to transmit MD parameters at each time instant.

Original: Since both M and D are determined bythe algorithm POF-MKL, this shows that POF-MKL can provide flexibility to tighten regret boundwhile the available clients-to-server communication bandwidth can afford transmission of clients’ updates to the server.
Revised: Since both M and D are determined by the algorithm POF-MKL, this shows that POF-MKL can provide flexibility to tighten regret bound while the available clients-to-server communication bandwidth can afford transmission of clients’ updates to the server.

Original: It is useful to mention that choosing larger value for ξ k increases the lowerbound of p i ∗ k,t and as a result the optimal choice for ξ k in terms of regret is ξ k = 1 .
Revised: It is useful to mention that choosing larger value for ξ k increases the lower bound of p i ∗ k,t and as a result the optimal choice for ξ k in terms of regret is ξ k = 1 .

Original: However,choosing smaller values for ξ k makes the value of p ik,t more dependent on weights { w ik,t } Kk =1(c.f.
Revised: However, choosing smaller values for ξ k makes the value of p ik,t more dependent on weights { w ik,t } Kk = (c.f. (8)).

Original: Therefore, choosing smaller values for ξ k makes p ik,t less predictable.
Revised: Therefore, choosing smaller values for ξ k makes p ik,t less predictable.

Original: This makesestimating ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) given ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p
Revised: This makes estimating ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) given ∇L ( θ ⊤ i, t z i ( x k,t ) , y k,t ) /p

Original: ik,t more difficult which leads tobetter protection of privacy.
Revised: ik,t more difficult which leads to better protection of privacy.

Original: Comparison with personalized federated learning.
Revised: Comparison with personalized federated learning.

Original: In order to deal with heterogeneous data amongclients, personalized federated learning has been studied extensively in  literature (see [43, 8, 9, 12,19, 10,  28, 27, 41, 6, 1, 31, 49, 2, 45, 5]).
Revised: In order to deal with heterogeneous data among clients, personalized federated learning has been studied extensively in the literature (see [44, 8, 9, 12, 20, 10, 29, 28, 42, 6, 1, 32, 50, 2, 46, 5]).

Original: Utilizing model-agnostic meta-learning [13], personalizedfederated learning algorithms have been proposed in [12, 1].
Revised: Utilizing model-agnostic meta-learning [13], personalized federated learning algorithms have been proposed in [12, 1].

Original: In [41, 5], personalized federatedlearning algorithms have been designed by learning hyper-networks [17].
Revised: In [42, 5], personalized federated learning algorithms have been designed by learning hyper-networks [18].

Original: In [31], a personalizedmodel is a linear combination of a set of shared component models such that each client constructsits personalized mixture of models.
Revised: In [32], a personalized model is a linear combination of a set of shared component models such that each client constructs its personalized mixture of models.

Original: However, in aforementioned personalized federated learningworks, clients are assumed to store a dataset to perform local updates with.
Revised: However, in aforementioned personalized federated learning works, clients are assumed to store a dataset to perform local updates with.

Original: Therefore, when clientsare not able to store data in batch and they have to make a decision upon receiving a new datasample, aforementioned works in personalized federated learning cannot guarantee sub-linear regretfor clients.
Revised: Therefore, when clients are not able to store data in batch and they have to make a decision upon receiving a new data sample, aforementioned works in personalized federated learning cannot guarantee sub-linear regret for clients.

Original: However, according to Theorems 1 and 2, POF-MKL provides sub-linear regret for clientswhen clients cannot store data in batch and make decision in an online fashion.
Revised: However, according to Theorems 1 and 2, POF-MKL provides sub-linear regret for clients when clients cannot store data in batch and make decision in an online fashion.

Original: Comparison with online federated learning [33].
Revised: Comparison with online federated learning [34].

Original: Fed-OMD algorithm has been proposed in [33]which enables clients to perform their learning task in an online and federated fashion while it is provedthat Fed-OMD enjoys sub-linear regret when the loss function is convex with respect to parametersrequired to be learnt at each time instant.
Revised: Fed-OMD algorithm has been proposed in [34] which enables clients to perform their learning task in an online and federated fashion while it is proved that Fed-OMD enjoys sub-linear regret when the loss function is convex with respect to parameters required to be learnt at each time instant.

Original: The proposed POF-MKL differs from Fed-OMD in the sensethat Fed-OMD cannot guarantee sub-linear regret when it comes to performing the online learningtask with RF approximations of multiple kernels since the loss function L ( (cid:80) Ni =1 w i,t θ ⊤ i,t z i ( x ) , y ) isnot convex with respect to both θ i,t and w i,t .
Revised: The proposed POF-MKL differs from Fed-OMD in the sense that Fed-OMD cannot guarantee sub-linear regret when it comes to performing the online learning task with RF approximations of multiple kernels since the loss function L ( (cid:80) Ni =1 w i,t θ ⊤ i,t z i ( x ) , y ) is not convex with respect to both θ i,t and w i,t .

Original: However, according to Theorems 1 and 2, the proposed
Revised: However, according to Theorems 1 and 2, the proposed

Original: POF-MKL guarantees sub-linear regret.
Revised: POF-MKL guarantees sub-linear regret.

Original: Comparison with [21].
Revised: Comparison with [22].

Original: Online federated MKL algorithms called vM-KOFL and eM-KOFL havebeen presented in [21].
Revised: Online federated MKL algorithms called vM-KOFL and eM-KOFL have been presented in [22].

Original: Both POF-MKL and algorithms in [21] exploit random feature approximationto alleviate computational complexity of online kernel learning.
Revised: Both POF-MKL and algorithms in [22] exploit random feature approximation to alleviate computational complexity of online kernel learning.

Original: Furthermore, both POF-MKL andalgorithms in [21] learn a linear combination of kernels.
Revised: Furthermore, both POF-MKL and algorithms in [22] learn a linear combination of kernels.

Original: The proposed POF-MKL has the followingadvantages and innovations compared to vM-KOFL and eM-KOFL: i)
Revised: The proposed POF-MKL has the following advantages and innovations compared to vM-KOFL and eM-KOFL: i)

Original: The proposed POF-MKLallows clients to learn their own personalized combination of kernels (c.f. (6)).
Revised: The proposed POF-MKL allows clients to learn their own personalized combination of kernels (c.f. (6)).

Original: As it is proved in
Revised: As it is proved in

Original: Theorem 1, the proposed POF-MKL can deal with heterogeneous data among clients in the sensethat using POF-MKL each client guarantees sub-linear regret with respect to the best kernel RFapproximation according to the corresponding client data.
Revised: Theorem 1, the proposed POF-MKL can deal with heterogeneous data among clients in the sense that using POF-MKL each client guarantees sub-linear regret with respect to the best kernel RF approximation according to the corresponding client data.

Original: However, both vM-KOFL and eM-KOFLare not able to provide such guarantee.
Revised: However, both vM-KOFL and eM-KOFL are not able to provide such guarantee.

Original: ii) Using vM-KOFL, each client needs to send ( N + 1) Dparameters to the server.
Revised: ii) Using vM-KOFL, each client needs to send ( N + 1) D parameters to the server.

Original: However, using the proposed POF-MKL, each client needs to send MDparameters to the server such that M ≤ N is determined by POF-MKL and can be chosen to bemuch smaller than N . iii)
Revised: However, using the proposed POF-MKL, each client needs to send MD parameters to the server such that M ≤ N is determined by POF-MKL and can be chosen to be much smaller than N . iii)

Original: In eM-KOFL, the server chooses a kernel at each time instant and clientssend their local updates associated with the chosen kernel by the server.
Revised: In eM-KOFL, the server chooses a kernel at each time instant and clients send their local updates associated with the chosen kernel by the server.

Original: The proposed POF-MKLprovides more flexibility compared to eM-KOFL in the sense that using POF-MKL each client cansend local updates of M ≥ 1 kernels to the server.
Revised: The proposed POF-MKL provides more flexibility compared to eM-KOFL in the sense that using POF-MKL each client can send local updates of M ≥ 1 kernels to the server.

Original: And each client chooses its own subset of kernelsto send their updates to the server.
Revised: And each client chooses its own subset of kernels to send their updates to the server.

Original: Therefore, even though POF-MKL sets M to 1 , it is possible thatat a time instant the server receives updates associated with all kernels in the dictionary.
Revised: Therefore, even though POF-MKL sets M to 1 , it is possible that at a time instant the server receives updates associated with all kernels in the dictionary.

Original: It is useful tomention that using eM-KOFL the cumulative regret of all clients is sub-linear with respect to the bestkernel RF approximation with probability 1 − δ
Revised: It is useful to mention that using eM-KOFL the cumulative regret of all clients is sub-linear with respect to the best kernel RF approximation with probability 1 − δ

Original: where 0 < δ ≤ 1 .
Revised: where 0 < δ ≤ 1 .

Original: However, utilizing the updaterule in (9), using the proposed POF-MKL, each client obtains sub-linear regret with respect to RFapproximation of its best kernel with probability 1.
Revised: However, utilizing the update rule in (9), using the proposed POF-MKL, each client obtains sub-linear regret with respect to RF approximation of its best kernel with probability 1.

Original:  Experiments
Revised:  Experiments

Original: We tested the performance of the proposed POF-MKL for online regression task through a set ofexperiments.
Revised: We tested the performance of the proposed POF-MKL for online regression task through a set of experiments.

Original: The performance of POF-MKL is compared with the baselines PerFedAvg [12], OFSKL[33], OFMKL-Avg [33], vM-KOFL [21] and eM-KOFL [21].
Revised: The performance of POF-MKL is compared with the baselines PerFedAvg [12], OFSKL [34], OFMKL-Avg [34], vM-KOFL [22] and eM-KOFL [22].

Original: PerFedAvg refers to the personalizedfederated averaging algorithm in [12].
Revised: PerFedAvg refers to the personalized federated averaging algorithm in [12].

Original: In the experiments, PerFedAvg employs a  feedforward neuralnetwork model.
Revised: In the experiments, PerFedAvg employs a fully connected feedforward neural network model.

Original: More information about the implementation of PerFedAvg can be found in Appendix C. OFSKL and OFMKL-Avg are two variations of Fed-OMD [33].
Revised: More information about the implementation of PerFedAvg can be found in Appendix C. OFSKL and OFMKL-Avg are two variations of Fed-OMD [34].

Original: OFSKL leverages Fed-OMD[33] when a single radial basis function (RBF) with bandwidth of 10 is employed to perform thelearning task.
Revised: OFSKL leverages Fed-OMD [34] when a single radial basis function (RBF) with bandwidth of 10 is employed to perform the learning task.

Original: In OFMKL-Avg, kernels are learned independently from each other using Fed-OMD[33] and the prediction is the average of approximations given by kernels.
Revised: In OFMKL-Avg, kernels are learned independently from each other using Fed-OMD [34] and the prediction is the average of approximations given by kernels.

Original: Moreover, vM-KOFL andeM-KOFL are online federated MKL algorithms of [21] such that vM-KOFL requires transmissionof all kernel updates at every time instant while eM-KOFL requires transmission of a kernel update ateach time instant.
Revised: Moreover, vM-KOFL and eM-KOFL are online federated MKL algorithms of [22] such that vM-KOFL requires transmission of all kernel updates at every time instant while eM-KOFL requires transmission of a kernel update at each time instant.

Original: In the experiments, each client observes 500 samples until the end of the learningtask meaning that T = 500 .
Revised: In the experiments, each client observes 500 samples until the end of the learning task meaning that T = 500 .

Original: The performance of the proposed POF-MKL and other baselines aretested on the following real datasets downloaded from UCI machine learning repository [11]: Naval[7], UJI [46], Air [50] and WEC [34].
Revised: The performance of the proposed POF-MKL and other baselines are tested on the following real datasets downloaded from UCI machine learning repository [11]: Naval [7], UJI [47], Air [51] and WEC [35].

Original: More detailed information about datasets can be found in Appendix C. Data samples of Naval and UJI datasets are distributed i.i.d among clients.
Revised: More detailed information about datasets can be found in Appendix C. Data samples of Naval and UJI datasets are distributed i.i.d among clients.

Original: Data samplesin Air and WEC datasets are distributed non-i.i.d among clients.
Revised: Data samples in Air and WEC datasets are distributed non-i.i.d among clients.

Original: More inforamtion about distributingdata samples among clients can be found in Appendix C.
Revised: More inforamtion about distributing data samples among clients can be found in Appendix C.

Original: The number of clients for Naval,UJI,
Revised: The number of clients for Naval,

Original:  Air and WEC datasets are 23 , 42 , 240 and 560 , respectively.
Revised: UJI, Air and WEC datasets are 23 , 42 , 240 and 560 , respectively.

Original: The dictionary of kernels consists of51 RBFs with different bandwidth such that the bandwidth of the i -th kernel is σ i =
Revised: The dictionary of kernels consists of 51 RBFs with different bandwidth such that the bandwidth of the i -th kernel is σ i =

Original: i − 52 25 .
Revised: i − 52 25 .

Original: Weconsider the case where the clients-to-server communication bandwidth is limited such that at eachtime instant, the maximum number of parameters that a client is allowed to transmit to the serveris 1000 .
Revised: We consider the case where the clients-to-server communication bandwidth is limited such that at each time instant, the maximum number of parameters that a client is allowed to transmit to the server is 1000 .

Original: Furthermore, the memory and computational capability of clients are limited such that themaximum value can be picked for the number of random features D is 100 .
Revised: Furthermore, the memory and computational capability of clients are limited such that the maximum value can be picked for the number of random features D is 100 .

Original: The experiments werecarried for 20 different sets of random feature vectors.
Revised: The experiments were carried for 20 different sets of random feature vectors.

Original: The performance of algorithms is measuredusing average of mean squared error (MSE) defined as
Revised: The performance of algorithms is measured [Figure] using average of mean squared error (MSE) defined as

Original: [Equation]
Revised: [Equation]

Original: where ˆ y jk,t denotes the prediction of the k -th client at time instant t corresponding to the j -th set ofrandom feature vectors.
Revised: where ˆ y jk,t denotes the prediction of the k -th client at time instant t corresponding to the j -th set of random feature vectors.

Original: Learning rates are set to η = η k = 1 √ T , ∀ k .
Revised: Learning rates are set to η = η k = 1 √ T , ∀ k .

Original: Also, exploration rates are set toξ k = 1 , ∀ k .
Revised: Also, exploration rates are set to ξ k = 1 , ∀ k .

Original: The performance of POF-MKL with different ξ k is studied in Appendix C. 
Revised: The performance of POF-MKL with different ξ k is studied in Appendix C. Codes are available at https://github.com/pouyamghari/POF-MKL .

Original: Table 1 presents the MSE and run time performance of online federated kernel learning algorithmson real datasets.
Revised: Table 1 presents the MSE and run time performance of online federated kernel learning algorithms on real datasets.

Original: Run time refers to average total run time of clients to perform online learning taskon the entire data samples that they observe.
Revised: Run time refers to average total run time of clients to perform online learning task on the entire data samples that they observe.

Original: In Table 1, M refers to the number of kernels whoseupdates are sent to the server after prediction at each time instant.
Revised: In Table 1, M refers to the number of kernels whose updates are sent to the server after prediction at each time instant.

Original: And, D is the number of randomfeatures.
Revised: And, D is the number of random features.

Original: Comparing MSE of POF-MKL with that of OFMKL-Avg, it can be concluded that learningthe weights to combine kernels provides higher accuracy than averaging kernels’ predictions.
Revised: Comparing MSE of POF-MKL with that of OFMKL-Avg, it can be concluded that learning the weights to combine kernels provides higher accuracy than averaging kernels’ predictions.

Original: Table 1shows that POF-MKL with M = 1 provides lower MSE than eM-KOFL.
Revised: Tableshows that POF-MKL with M = 1 provides lower MSE than eM-KOFL.

Original: Using eM-KOFL, at eachtime instant, the server receives updates belong to only one kernel.
Revised: Using eM-KOFL, at each time instant, the server receives updates belong to only one kernel.

Original: However, using POF-MKL with
Revised: However, using POF-MKL with

Original: M = 1 , each client sends an update belongs to a kernel which is selected by the client.
Revised: M = 1 , each client sends an update belongs to a kernel which is selected by the client.

Original: Therefore, theserver receives updates associated with different kernels even though M = 1 .
Revised: Therefore, the server receives updates associated with different kernels even though M = 1 .

Original: Therefore, experimentalresults show the effectiveness of the personalized kernel selection provided by POF-MKL.
Revised: Therefore, experimental results show the effectiveness of the personalized kernel selection provided by POF-MKL.

Original: It can beobserved that POF-MKL with M = 25 obtains lower MSE than those of POF-MKL with M = 51and vM-KOFL.
Revised: It can be observed that POF-MKL with M = 25 obtains lower MSE than those of POF-MKL with M =and vM-KOFL.

Original: Since each client is allowed to send at most 1000 parameters per time instant, ifclients send updates of all kernels at every time instant as this is the case in vM-KOFL, D cannotbe chosen to be greater than 9 .
Revised: Since each client is allowed to send at most 1000 parameters per time instant, if clients send updates of all kernels at every time instant as this is the case in vM-KOFL, D cannot be chosen to be greater than 9 .

Original: However, setting M = 25 , POF-MKL can set D = [-
 whichcan-] improve the accuracy of online regression task compared to the case where D = 9 .
Revised: However, setting M = 25 , POF-MKL can set D = [-
 whichcan-]20 which can improve the accuracy of online regression task compared to the case where D = 9 .

Original: Note thataccording to Theorem 2, increase in D increases the probability that the server achieves sub-linearregret with respect to the best function approximator.
Revised: Note that according to Theorem 2, increase in D increases the probability that the server achieves sub-linear regret with respect to the best function approximator.

Original: Furthermore, POF-MKL with M = 51 achieveslower MSE than vM-KOFL even if data samples are distributed i.i.d among clients.This shows [Table]
Revised: Furthermore, POF-MKL with M = 51 achieves lower MSE than vM-KOFL even if data samples are distributed i.i.d among clients.

Original: [Figure] [Table] [Figure] that the proposed POF-MKL can better cope with heterogeneous data among clients which is inagreement with theoretical results in Theorem 1.
Revised: This shows that the proposed POF-MKL can better cope with heterogeneous data among clients which is in agreement with theoretical results in Theorem 1.

Original: In fact, the optimal combination of kernels can bedifferent across clients.
Revised: In fact, the optimal combination of kernels can be different across clients.

Original: Using POF-MKL, each client constructs its own personalized combination ofkernels which results in lower MSE compared to vM-KOFL.
Revised: Using POF-MKL, each client constructs its own personalized combination of kernels which results in lower MSE compared to vM-KOFL.

Original: The proposed POF-MKL with M = 1and M = 25 runs faster than eM-KOFL.
Revised: The proposed POF-MKL with M =and M = 25 runs faster than eM-KOFL.

Original: In fact, using POF-MKL, clients only need to updateparameters associated with M kernels while employing vM-KOFL and eM-KOFL, clients have toupdate parameters of all kernels.
Revised: In fact, using POF-MKL, clients only need to update parameters associated with M kernels while employing vM-KOFL and eM-KOFL, clients have to update parameters of all kernels.

Original: Moreover, POF-MKL obtains lower MSE than PerFedAvg.
Revised: Moreover, POF-MKL obtains lower MSE than PerFedAvg.

Original: Notethat since clients are not able to store data in batch, at each time instant clients update PerFedAvg’smodel using only the newly observed data sample.
Revised: Note that since clients are not able to store data in batch, at each time instant clients update PerFedAvg’s model using only the newly observed data sample.

Original: Therefore, convergence of PerFedAvg is notguaranteed.
Revised: Therefore, convergence of PerFedAvg is not guaranteed.

Original: Experimental results show that POF-MKL achieves higher accuracy than PerFedAvg inonline regression task when it is not possible for clients to store data in batch.
Revised: Experimental results show that POF-MKL achieves higher accuracy than PerFedAvg in online regression task when it is not possible for clients to store data in batch.

Original: Since OFSKL employsonly a pre-selected single kernel, OFSKL runs faster than POF-MKL.
Revised: Since OFSKL employs only a pre-selected single kernel, OFSKL runs faster than POF-MKL.

Original: However, utilizing multiplekernels enables POF-MKL to obtain lower MSE than that of OFSKL.
Revised: However, utilizing multiple kernels enables POF-MKL to obtain lower MSE than that of OFSKL.

Original: In fact, using POF-MKL clientslearn a linear combination of kernels which is proved to enjoy sub-linear regret with respect to the bestkernel in hindsight while employing OFSKL clients have to make predictions using a pre-selectedkernel.
Revised: In fact, using POF-MKL clients learn a linear combination of kernels which is proved to enjoy sub-linear regret with respect to the best kernel in hindsight while employing OFSKL clients have to make predictions using a pre-selected kernel.

Original: Furthermore, Figure 1 illustrates the average regret of clients when clients employ vM-KOFLand proposed POF-MKL with different M parameters.
Revised: Furthermore, Figure 1 illustrates the average regret of clients when clients employ vM-KOFL and the proposed POF-MKL with different M parameters.

Original: From Figure 1, it can be observed that theproposed POF-MKL achieves sub-linear regret. 
Revised: From Figure 1, it can be observed that the proposed POF-MKL achieves sub-linear regret. [Table]

Original:  Conclusions
Revised:  Conclusions

Original: The present paper proposed a personalized online federated MKL algorithm called POF-MKL basedon RF approximation.
Revised: The present paper proposed a personalized online federated MKL algorithm called POF-MKL based on RF approximation.

Original: Employing the proposed POF-MKL, each client updates the parameters of asubset of kernels which alleviates the computational complexity of the client as well as communicationcost of sending updated parameters of kernels.
Revised: Employing the proposed POF-MKL, each client updates the parameters of a subset of kernels which alleviates the computational complexity of the client as well as communication cost of sending updated parameters of kernels.

Original: Theoretical analysis proved that using POF-MKL,each client achieves sub-linear regret with respect to the RF approximation of its best kernel inhindsight which indicates that POF-MKL can deal heterogeneous data among clients.
Revised: Theoretical analysis proved that using POF-MKL, each client achieves sub-linear regret with respect to the RF approximation of its best kernel in hindsight which indicates that POF-MKL can deal heterogeneous data among clients.

Original: While eachclient updates a subset of kernels, it was proved that the server achieves sub-linear regret withrespect to the best function approximator.
Revised: While each client updates a subset of kernels, it was proved that the server achieves sub-linear regret with respect to the best function approximator.

Original: Experiments on real datasets showcased the advantages of
Revised: Experiments on real datasets showcased the advantages of

Original: POF-MKL compared with other online federated kernel learning algorithms.
Revised: POF-MKL compared with other online federated kernel learning algorithms.

Revised: Acknowledgement

Revised: This work is supported by NSF ECCS 2207457.

Revised: PI Yanning Shen also receives support from

Revised: Microsoft Academic Grant Award for AI Research.

Revised: Contact: Yanning Shen (yannings@uci.edu)