Original: Personalized Online Federated Multi-Kernel Learning
Revised: Personalized Online Federated Multi-Kernel Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Multi-kernel learning (MKL) exhibits well-documented performance in onlinenon-linear function approximation.
Revised: Multi-kernel learning (MKL) exhibits well-documented performance in onlinenon-linear function approximation.

Original: Federated learning enables a group of learners(called clients) to train an MKL model on the data distributed among clients toperform online non-linear function approximation.
Revised: Federated learning enables a group of learners(called clients) to train an MKL model on the data distributed among clients toperform online non-linear function approximation.

Original: There are some challenges inonline federated MKL that need to be addressed: i) Communication efficiencyespecially when a large number of kernels are considered ii) Heterogeneous datadistribution among clients.
Revised: There are some challenges inonline federated MKL that need to be addressed: i) Communication efficiencyespecially when a large number of kernels are considered ii) Heterogeneous datadistribution among clients.

Original: The present paper develops an algorithmic framework toenable clients to communicate with the server to send their updates with affordablecommunication cost while clients employ a large dictionary of kernels.
Revised: The present paper develops an algorithmic framework toenable clients to communicate with the server to send their updates with affordablecommunication cost while clients employ a large dictionary of kernels.

Original: Utilizingrandom feature (RF) approximation, the present paper proposes scalable onlinefederated MKL algorithm.
Revised: Utilizingrandom feature (RF) approximation, the present paper proposes scalable onlinefederated MKL algorithm.

Original: We prove that using the proposed online federated MKLalgorithm, each client enjoys sub-linear regret with respect to the RF approximationof its best kernel in hindsight, which indicates that the proposed algorithm caneffectively deal with heterogeneity of the data distributed among clients.
Revised: We prove that using the proposed online federated MKLalgorithm, each client enjoys sub-linear regret with respect to the RF approximationof its best kernel in hindsight, which indicates that the proposed algorithm caneffectively deal with heterogeneity of the data distributed among clients.

Original: Experimental results on real datasets showcase the advantages of the proposed algorithmcompared with other online federated kernel learning ones.
Revised: Experimental results on real datasets showcase the advantages of the proposed algorithmcompared with other online federated kernel learning ones.

Original:  Introduction
Revised:  Introduction

Original: Kernel learning exhibits well-documented performance in function approximation tasks, whileproviding theoretical guarantees associated with different performance metrics, see e.g. [47, 20, 37].
Revised: Kernel learning exhibits well-documented performance in function approximation tasks, whileproviding theoretical guarantees associated with different performance metrics, see e.g. [47, 20, 37].

Original: In some cases, a group of learners aims at collaborating to perform function approximation withoutrevealing their data.
Revised: In some cases, a group of learners aims at collaborating to perform function approximation withoutrevealing their data.

Original: To this end, federated learning has been emerged as a crucial learning paradigmby enabling a group of learners called clients to collaborate with each other by communicating with acentral server to train a centralized model [32, 26, 12, 22].
Revised: To this end, federated learning has been emerged as a crucial learning paradigmby enabling a group of learners called clients to collaborate with each other by communicating with acentral server to train a centralized model [32, 26, 12, 22].

Original: Through this process, clients send modelparameters and updates to the server without revealing their data.
Revised: Through this process, clients send modelparameters and updates to the server without revealing their data.

Original: Upon receiving updates fromclients, the server updates the model.
Revised: Upon receiving updates fromclients, the server updates the model.

Original: Therefore, federated learning enables clients to perform kernellearning for function approximation.
Revised: Therefore, federated learning enables clients to perform kernellearning for function approximation.

Original: In this context, a server and clients collaborate with each otherto learn the optimal kernel.
Revised: In this context, a server and clients collaborate with each otherto learn the optimal kernel.

Original: Furthermore, in some practical cases, clients may need to perform thefunction approximation in an online fashion while they are collaborating with the server to learn the kernel.
Revised: Furthermore, in some practical cases, clients may need to perform thefunction approximation in an online fashion while they are collaborating with the server to learn the kernel.

Original: For example, consider the case where clients may not have enough memory to store data inbatch.
Revised: For example, consider the case where clients may not have enough memory to store data inbatch.

Original: In addition, data samples may arrive in a sequential manner such that clients are not able to perform the function approximation in batch form.
Revised: In addition, data samples may arrive in a sequential manner such that clients are not able to perform the function approximation in batch form.

Original: There are major challenges in performing online kernel learning in federated fashion that need to be addressed:
Revised: There are major challenges in performing online kernel learning in federated fashion that need to be addressed:

Original: Communication Efficiency: Communication efficiency arises as a bottleneck in federated learning
Revised: Communication Efficiency: Communication efficiency arises as a bottleneck in federated learning

Original: (see e.g. [24, 36, 18])
Revised: (see e.g. [24, 36, 18])

Original: Specifically, limited clients-to-server communication bandwidth restricts the
Revised: Specifically, limited clients-to-server communication bandwidth restricts the

Original: number
Revised: number

Original: can be sent from
Revised: can be sent from

Original: the server
Revised: the server

Original: Heterogeneous Data: The distribution of data observed by a client might be different from others(see e.g. [43, 28, 6]).
Revised: Heterogeneous Data: The distribution of data observed by a client might be different from others(see e.g. [43, 28, 6]).

Original: Thus, the optimal kernel that is aimed to be learned is different across clients.
Revised: Thus, the optimal kernel that is aimed to be learned is different across clients.

Original: Computational Complexity: Clients should be able to perform function approximation fast enoughin order to make a decision in real-time.
Revised: Computational Complexity: Clients should be able to perform function approximation fast enoughin order to make a decision in real-time.

Original: Therefore, the computational complexity of kernel learningmethods should be affordable for clients.
Revised: Therefore, the computational complexity of kernel learningmethods should be affordable for clients.

Original: Conventional online kernel learning approaches (see e.g. [20, 38]) suffer from ‘curse of dimensionality’ [3] in the sense that the number of parameters that should be learned increases with the number ofobserved data.
Revised: Conventional online kernel learning approaches (see e.g. [20, 38]) suffer from ‘curse of dimensionality’ [3] in the sense that the number of parameters that should be learned increases with the number ofobserved data.

Original: This can make employing conventional online kernel learning approaches infeasibleto perform online federated kernel learning since clients may be required to send a large numberof parameter updates to the server while the available clients-to-server communication bandwidthis not enough for sending such information.
Revised: This can make employing conventional online kernel learning approaches infeasibleto perform online federated kernel learning since clients may be required to send a large numberof parameter updates to the server while the available clients-to-server communication bandwidthis not enough for sending such information.

Original: Approximating kernels by finite-dimensional featurerepresentations (e.g. Nyström method [48] and random feature method of [35]) makes online kernellearning approaches scalable in the sense that the learner can choose the number of parametersthat should be learned, independent of the number of observed data samples (see e.g. [29, 4, 51]).
Revised: Approximating kernels by finite-dimensional featurerepresentations (e.g. Nyström method [48] and random feature method of [35]) makes online kernellearning approaches scalable in the sense that the learner can choose the number of parametersthat should be learned, independent of the number of observed data samples (see e.g. [29, 4, 51]).

Original: Employing finite-dimensional feature representations of kernels to perform online federated kernellearning, clients can choose the number of parameters that they should send to the server.
Revised: Employing finite-dimensional feature representations of kernels to perform online federated kernellearning, clients can choose the number of parameters that they should send to the server.

Original: Therefore,finite-dimensional kernel approximation can better cope with limited clients-to-server communicationbandwidth compared to conventional kernel learning approaches.
Revised: Therefore,finite-dimensional kernel approximation can better cope with limited clients-to-server communicationbandwidth compared to conventional kernel learning approaches.

Original: Random feature (RF) approximation [35] has been exploited to perform online federated kernel learning when a single pre-selectedkernel function is employed [25, 16].
Revised: Random feature (RF) approximation [35] has been exploited to perform online federated kernel learning when a single pre-selectedkernel function is employed [25, 16].

Original: The choice of the kernel function greatly affects the performanceof function approximation when it comes to exploiting only a single kernel function.
Revised: The choice of the kernel function greatly affects the performanceof function approximation when it comes to exploiting only a single kernel function.

Original: Employingmultiple kernels instead of a single pre-selected one, can lead to obtaining more accurate functionapproximation since multi-kernel learning (MKL) can learn combination of kernels [23].
Revised: Employingmultiple kernels instead of a single pre-selected one, can lead to obtaining more accurate functionapproximation since multi-kernel learning (MKL) can learn combination of kernels [23].

Original: Onlinefederated MKL algorithms with theoretical guarantees called vM-KOFL and eM-KOFL have beenproposed in [21].
Revised: Onlinefederated MKL algorithms with theoretical guarantees called vM-KOFL and eM-KOFL have beenproposed in [21].

Original: However, eM-KOFL and vM-KOFL do not provide personalized MKL models forclients since they learn the same combination of kernels for all clients.
Revised: However, eM-KOFL and vM-KOFL do not provide personalized MKL models forclients since they learn the same combination of kernels for all clients.

Original: The present paper proposes a novel p ersonalized o nline f ederated MKL algorithm called POF-MKLthat provides a personalized MKL model for each client while it is ensured that the available clientsto-server communication bandwidth can afford communication cost of sending clients’ updates to theserver.
Revised: The present paper proposes a novel p ersonalized o nline f ederated MKL algorithm called POF-MKLthat provides a personalized MKL model for each client while it is ensured that the available clientsto-server communication bandwidth can afford communication cost of sending clients’ updates to theserver.

Original: In order to alleviate the communication cost of MKL, the propsoed POF-MKL employs RFapproximation of kernels and at each time instant, each client chooses a subset of kernels to sendtheir updates to the server instead of sending the updates of all kernels.
Revised: In order to alleviate the communication cost of MKL, the propsoed POF-MKL employs RFapproximation of kernels and at each time instant, each client chooses a subset of kernels to sendtheir updates to the server instead of sending the updates of all kernels.

Original: The number of kernels in thechosen subset is selected such that the required bandwidth to send all clients’ updates does not exceedthe available clients-to-server communication bandwidth.
Revised: The number of kernels in thechosen subset is selected such that the required bandwidth to send all clients’ updates does not exceedthe available clients-to-server communication bandwidth.

Original: Therefore, clients can send their updatesto the server independent of the number of kernels in the dictionary and as a result a comparativelylarge dictionary of kernels can be considered to perform function approximation.
Revised: Therefore, clients can send their updatesto the server independent of the number of kernels in the dictionary and as a result a comparativelylarge dictionary of kernels can be considered to perform function approximation.

Original: Contributions ofthe present paper can be summarized as follows:
Revised: Contributions ofthe present paper can be summarized as follows:

Original: c1.
Revised: c1.

Original: Leveraging the proposed POF-MKL, clients can update a subset of kernels’ parameters whichalleviates computational complexity and communication cost of sending updates to the server;c2.
Revised: Leveraging the proposed POF-MKL, clients can update a subset of kernels’ parameters whichalleviates computational complexity and communication cost of sending updates to the server;c2.

Original: Through theoretical analysis, it is proved that using the proposed POF-MKL, each client achievessub-linear regret with respect to RF approximation of the best kernel in hindsight associated withthe corresponding client data samples (c.f. Theorem 1).
Revised: Through theoretical analysis, it is proved that using the proposed POF-MKL, each client achievessub-linear regret with respect to RF approximation of the best kernel in hindsight associated withthe corresponding client data samples (c.f. Theorem 1).

Original: Moreover, it is guaranteed that the serverachieves sub-linear regret with respect to the best function approximator (c.f. Theorem 2);c3.
Revised: Moreover, it is guaranteed that the serverachieves sub-linear regret with respect to the best function approximator (c.f. Theorem 2);c3.

Original: Experiments on real datasets showcase the effectiveness of the proposed POF-MKL compared toexisting online federated kernel learning algorithms.
Revised: Experiments on real datasets showcase the effectiveness of the proposed POF-MKL compared toexisting online federated kernel learning algorithms.

Original:  Problem Statement and Preliminaries
Revised:  Problem Statement and Preliminaries

Original: Let there be a set of K clients performing function approximation task in an online fashion.
Revised: Let there be a set of K clients performing function approximation task in an online fashion.

Original: Thek -th client’s goal is to learn the function f( · ; · , · ) using the stream of data samples { ( x k,t , y k,t ) }
Revised: The k -thclient’s goal is to learn the function f using the stream of data samples { ( x k,t , y k,t ) }

Original: Tt =1such that x k,t ∈ R
Revised: Tt =1 such thatx k,t ∈ R

Original: d is the data sample observed by the k -th client at time t and y k,t is the labelassociated with x k,t .
Revised: d is the data sample observed by the k -th client at time t and y k,t is the label associatedwith x k,t .

Original: In the kernel learning context, the function f( · ; · , · ) is assumed to belong toa reproducing kernel Hilbert space (RKHS).
Revised: In the kernel learning context, the function f is assumed to belong to a reproducing kernel Hilbert space (RKHS).

Original: The present paper studies the personalized federated supervised function approximation  where  the goal is to find function parameters Θ and w k suchthat the cumulative difference between f  ( x k,t ; Θ , w k ) and y k,t over time is minimized.
Revised: The present paper studies the personalized federated supervised function approximation problem [Equation] where H represents the RKHS the function f belongs to and L ( · , · ) denotes the loss function whichcan be defined as [Equation]

Revised: where C ( · , · ) denotes the cost function (e.g. least squares function for regression task).

Revised: Also, λdenotes the regularization coefficient and Ω ( · ) represents a regularizer function to prevent over-fittingand control the model complexity.

Original: Note that Θdenotes the globalfunction parameters  which are learned through collaboration of clients with theserver while w k is the personalizedfunction parameter  learned locally by the k -th client.
Revised: Let Θ be the global parameters of the function f which arelearned through collaboration of clients with the server while w k be the personalized parameter of thefunction f learned locally by the k -th client.

Revised: Thus, the goal is that the cumulative difference betweenf ( x k,t ; Θ , w k ) and y k,t over time is minimized.

Original: At each timeinstant t , upon observing the data sample x k,t , the k -th client make the prediction f
Revised: At each time instant t , upon observing the datasample x k,t , the k -th client make the prediction f

Original: ( x k,t ; Θ , w k )and then observes the true label y k,t .
Revised: ( x k,t ; Θ , w k ) and then observes the true label y k,t .

Original: Therefore, the function approximation problem that the serveraims at solving can be expressed as
Revised: Therefore, the function approximation problem that the server aims at solving can be expressed asmin Θ (cid:80)

Original: [Equation]
Revised: Tt =1 (cid:80)

Original: where L ( · , · ) denotes the loss function which can be defined as [Equation] where C ( · , · ) denotes the cost function (e.g. least squares function for regression task). Also, λdenotes the regularization coefficient and Ω ( · ) represents a regularizer function to prevent over-fittingand control the model complexity.
Revised: Kk =1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .

Original: Moreover, finding the local parameters w k by the k -th clientcan be expressed as the optimization problem min w k (cid:80) Tt =
Revised: Moreover, finding the local parameters w k by the k -thclient can be expressed as the optimization problem min w k (cid:80) Tt =

Original: 1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .
Revised: 1 L ( f ( x k,t ; Θ , w k ) , y k,t ) .

Original: In order toperform the function approximation task in an online fashion, the k -th client needs to perform thetask with the values of Θ and w k at time t denoted by Θ t and w k,t , respectively.
Revised: In orderto perform the function approximation task in an online fashion, the k -th client needs to perform thetask with the values of Θ and w k at time t denoted by Θ t and w k,t , respectively.

Original: Thus, the values offunction parameters Θ and w k should be updated ‘on the fly’.
Revised: Thus, the values offunction parameters Θ and w k should be updated ‘on the fly’.

Original: Since the function f ( · ; · , · ) belongsto a reproducing kernel Hilbert space (RKHS), based on the representer theorem [47], given datasamples, the optimal solution for (1) can be obtained as
Revised: Since the function f ( · ; · , · ) belongsto a reproducing kernel Hilbert space (RKHS), based on the representer theorem [47], given datasamples, the optimal solution for (1) can be obtained as

Original: [Equation]
Revised: [Equation]

Original: where κ ( · , · ) denotes symmetric positive definite kernel function such that κ ( x , x ′ ) measures thesimilarity between x and x ′ .
Revised: where κ ( · , · ) denotes symmetric positive definite kernel function such that κ ( x , x ′ ) measures thesimilarity between x and x ′ .

Original: And α k,t is an unknown coefficient associated with κ ( x , x k,t ) whichis required to be estimated.
Revised: And α k,t is an unknown coefficient associated with κ ( x , x k,t ) whichis required to be estimated.

Original: In this case, ˆ f ( · ) in (3) belongs to the RKHS H :
Revised: In this case, ˆ f ( · ) in (3) belongs to the RKHS H :

Original: =
Revised: =

Original: { f ( · ) | f ( x ) = 106 (cid:80) ∞ t =1 (cid:80)
Revised: { f ( · ) | f ( x ) = 107 (cid:80) ∞ t =

Original:  Kk =1 α k,t κ ( x , x k,t ) } such that RKHS norm is defined as ∥ f ∥ 2 H :
Revised: [Equation] (cid:80) Kk =1 α k,t κ ( x , x k,t ) } such that RKHS norm is defined as ∥ f ∥ 2 H :

Original: = (cid:80) t (cid:80) t ′ α t α t ′ κ ( x t , x t ′ ) .
Revised: = (cid:80) t (cid:80) t ′ α t α t ′ κ ( x t , x t ′ ) .

Original: Furthermore, from (3), it can be inferred that Θ t =
Revised: Furthermore, from (3), it can be inferred that Θ t =

Original: [ α 1 , 1 , . . .
Revised: [ α 1 , 1 , . . .

Original: , α K, 1 , . . .
Revised: , α K, 1 , . . .

Original: , α 1 ,t , . .
Revised: , α 1 ,t , . .

Original: . , α K,t ] .
Revised: . , α K,t ] .

Original: Therefore,the number of coefficients { α k,τ } tτ =1 , ∀ k that should be estimated to obtain ˆ f ( · ) increases overtime.
Revised: Therefore,the number of coefficients { α k,τ } tτ =1 , ∀ k that should be estimated to obtain ˆ f ( · ) increases overtime.

Original: This is known as curse of dimensionality [47] since the computational complexity of functionapproximation increases with time.
Revised: This is known as curse of dimensionality [47] since the computational complexity of functionapproximation increases with time.

Original: This brings challenge for federated implementation of functionapproximation since dimension of updates that should be sent to the server by each client grows overtime and when T is large, the available communication bandwidth may not be enough for clients tosend their updates.
Revised: This brings challenge for federated implementation of functionapproximation since dimension of updates that should be sent to the server by each client grows overtime and when T is large, the available communication bandwidth may not be enough for clients tosend their updates.

Original: In order to deal with the increasing number of unknown coefficients, one can employ random Fourierapproximation [35].
Revised: In order to deal with the increasing number of unknown coefficients, one can employ random Fourierapproximation [35].

Original: Assume that κ ( · ) is a shift-invariant kernel meaning that κ ( x , x ′ )
Revised: Assume that κ ( · ) is a shift-invariant kernel meaning that κ ( x , x ′ )

Original: = κ ( x − x ′ ) .
Revised: = κ ( x − x ′ ) .

Original: Let π κ ( ρ ) denotes the Fourier transform of κ ( · ) .
Revised: Let π κ ( ρ ) denotes the Fourier transform of κ ( · ) .

Original: If the kernel function κ ( · ) is normalized such thatκ ( 0 ) = 1 , then π κ ( ρ ) can be viewed as a probability density function (PDF) (see e.g. [35]).
Revised: If the kernel function κ ( · ) is normalized such thatκ ( 0 ) = 1 , then π κ ( ρ ) can be viewed as a probability density function (PDF) (see e.g. [35]).

Original: Letρ 1 , . .
Revised: Letρ 1 , . .

Original: , ρ D be a set of D independent and identically distributed (i.i.d) vectors drawn from π κ ( · ) .
Revised: , ρ D be a set of D independent and identically distributed (i.i.d) vectors drawn from π κ ( · ) .

Original: Let the vector z ( x ) be defined as
Revised: Let the vector z ( x ) be defined as

Original: [Equation]
Revised: [Equation]

Original: Then, ˆ κ r ( x − x ′ )
Revised: Then, ˆ κ r ( x − x ′ )

Original: =
Revised: =

Original: z ( x ) ⊤ z ( x ′ ) constitutes an unbiased estimator of κ ( x − x ′ ) and the randomfeature (RF) approximation of ˆ f ( x ) in (3) can be obtained as 
Revised: z ( x ) ⊤ z ( x ′ ) constitutes an unbiased estimator of κ ( x − x ′ ) and the randomfeature (RF) approximation of ˆ f ( x ) in (3) can be obtained as [Equation]

Original: [Equation]
Revised: [Equation]

Original: where in this case θ = (cid:80) Tt =1 (cid:80)
Revised: where in this case θ = (cid:80) Tt =1 (cid:80)

Original: Kk =1 α k,t z ( x k,t ) .
Revised: Kk =1 α k,t z ( x k,t ) .

Original: According to (4), z ( x k,t ) is a 2 D vector and asa result it can be concluded that θ is a 2 D vector as well.
Revised: According to (4), z ( x k,t ) is a 2 D vector and asa result it can be concluded that θ is a 2 D vector as well.

Original: Therefore, using RF approximation, thevector θ should be estimated whose dimension does not grow over time.
Revised: Therefore, using RF approximation, thevector θ should be estimated whose dimension does not grow over time.

Original: The performance of a kernel learning algorithm depends on the choice of the kernel.
Revised: The performance of a kernel learning algorithm depends on the choice of the kernel.

Original: Thus, performing the function approximation using a pre-selected kernel requires prior information whichmay not be available.
Revised: Thus, performing the function approximation using a pre-selected kernel requires prior information whichmay not be available.

Original: To cope with this, employing a dictionary of kernels in lieu of a pre-selectedsingle kernel has been proposed in the literature (see e.g. [44, 23, 30]).
Revised: To cope with this, employing a dictionary of kernels in lieu of a pre-selectedsingle kernel has been proposed in the literature (see e.g. [44, 23, 30]).

Original: Specifically, the kernel is learned as a combination of kernels in the dictionary.
Revised: Specifically, the kernel is learned as a combination of kernels in the dictionary.

Original: Let κ 1 ( · ) , . .
Revised: Let κ 1 ( · ) , . .

Original: , κ N ( · ) be a set of
Revised: , κ N ( · ) be a set of

Original: N kernels where κ i ( · ) denotes the i -th kernel.
Revised: N kernels where κ i ( · ) denotes the i -th kernel.

Original: The function ¯ κ ( · ) belongs to the convex hull
Revised: The function ¯ κ ( · ) belongs to the convex hull

Original: K :=
Revised: K :=

Original: { ¯ κ ( x ) = (cid:80)
Revised: { ¯ κ ( x ) = (cid:80)

Original: Ni
Revised: Ni

Original: =1 β
Revised: =1 β

Original: i κ i ( x ) , β i ≥ 0 , ∀ i, (cid:80)
Revised: i κ i ( x ) , β i ≥ 0 , ∀ i, (cid:80)

Original: Ni =
Revised: Ni =

Original: 1 β i = 1 } is a kernel [40].
Revised: 1 β i = 1 } is a kernel [40].

Original: Therefore, in onlinemulti-kernel learning, the goal is to learn the convex combination of kernels in the dictionary tominimize the cumulative regret with respect to the best function approximator in hindsight.
Revised: Therefore, in onlinemulti-kernel learning, the goal is to learn the convex combination of kernels in the dictionary tominimize the cumulative regret with respect to the best function approximator in hindsight.

Original: Thecumulative regret is defined as the cumulative difference between loss of the online multi-kernellearning algorithm and that of the best function approximator in hindsight.
Revised: Thecumulative regret is defined as the cumulative difference between loss of the online multi-kernellearning algorithm and that of the best function approximator in hindsight.

Original: Furthermore, for adataset { ( x t , y t ) } Tt =1 , the best function approximator is f ∗ ( · ) ∈ arg min
Revised: Furthermore, for adataset { ( x t , y t ) } Tt =1 , the best function approximator is f ∗ ( · ) ∈ arg min

Original: f ∗ i ,i ∈
Revised: f ∗ i ,i ∈

Original: [ N ] (cid:80) Tt =1 L ( f ∗ i ( x t ) , y t )
Revised: [ N ] (cid:80) Tt =1 L ( f ∗ i ( x t ) , y t )

Original: where f ∗ i ( ·
Revised: where f ∗ i ( ·

Original: ) ∈ arg min f ∈ H
Revised: ) ∈ arg min f ∈ H

Original: i (cid:80) Tt =
Revised: i (cid:80) Tt =

Original: 1 L ( f ( x t ) , y t ) such that H
Revised: 1 L ( f ( x t ) , y t ) such that H

Original: i is an RKHS induced by κ i ( · ) and[ N ] := { 1 , . . .
Revised: i is an RKHS induced by κ i ( · ) and[ N ] := { 1 , . . .

Original: , N } .
Revised: , N } .

Original: Enabled by random feature approximation, centralized and scalable online multikernel learning algorithms have been proposed in literature (see e.g. [39, 42, 15]).
Revised: Enabled by random feature approximation, centralized and scalable online multikernel learning algorithms have been proposed in literature (see e.g. [39, 42, 15]).

Original: The present paperproposes an algorithmic framework for personalized online federated MKL using RF approximationof kernels in the dictionary.
Revised: The present paperproposes an algorithmic framework for personalized online federated MKL using RF approximationof kernels in the dictionary.

Original:  Personalized Online Federated Multi-Kernel Learning
Revised:  Personalized Online Federated Multi-Kernel Learning

Original: The present section proposes an algorithmic framework for online federated multi-kernel learning which can deal with heterogeneous data among clients.
Revised: The present section proposes an algorithmic framework for online federated multi-kernel learning which can deal with heterogeneous data among clients.

Original: To perform function approximation, RF approximations of kernel functions are employed.
Revised: To perform function approximation, RF approximations of kernel functions are employed.

Original: For the i -th kernel κ i , vectors ρ i, 1 , . .
Revised: For the i -th kernel κ i , vectors ρ i, 1 , . .

Original: , ρ i,D are drawn i.i.d from π κ i ( · ) to construct the random feature
Revised: , ρ i,D are drawn i.i.d from π κ i ( · ) to construct the random feature

Original: vector z i ( x ) =1 √ D [sin( ρ ⊤ i, 1 x ) , . .
Revised: vector z i ( x ) =1 √ D [sin( ρ ⊤ i, 1 x ) , . .

Original: , sin( ρ ⊤ i,D x ) , cos( ρ ⊤ i, 1 x ) , . . .
Revised: , sin( ρ ⊤ i,D x ) , cos( ρ ⊤ i, 1 x ) , . . .

Original: , cos( ρ ⊤ i,D x )] .
Revised: , cos( ρ ⊤ i,D x )] .

Original: Then, at time instant t , the randomfeature approximation associated with κ i ( · ) can be obtained as ˆ f RF ,it ( x ) =
Revised: Then, at time instant t , the randomfeature approximation associated with κ i ( · ) can be obtained as ˆ f RF ,it ( x ) =

Original: θ ⊤ i,t z i ( x )
Revised: θ ⊤ i,t z i ( x )

Original: where θ i,t isthe global function parameter associated the i -th kernel at time t .
Revised: where θ i,t isthe global function parameter associated the i -th kernel at time t .

Original: 3.1 Algorithm
Revised: 3.1 Algorithm

Original: At each time instant t , the server transmits global function parameters θ i,t , ∀ i ∈
Revised: At each time instant t , the server transmits global function parameters θ i,t , ∀ i ∈

Original: [ N ] to all clients.
Revised: [ N ] to all clients.

Original: Thek -th client, assigns the weight w ik,t to the i -th kernel which indicates the confidence of the k -th clientat time t in the function approximation given by the i -th kernel.
Revised: Thek -th client, assigns the weight w ik,t to the i -th kernel which indicates the confidence of the k -th clientat time t in the function approximation given by the i -th kernel.

Original: Upon receiving new data samplex k,t , the k -th client performs the function approximation combining kernels’ RF approximations asˆ f ( x k,t ; ˆ Θ t , w k,t ) =
Revised: Upon receiving new data samplex k,t , the k -th client performs the function approximation combining kernels’ RF approximations asˆ f ( x k,t ; ˆ Θ t , w k,t ) =

Original: N (cid:88) i = w ik,t
Revised: N (cid:88) i = w ik,t

Original: W k,t θ ⊤ i,t z
Revised: W k,t θ ⊤ i,t z

Original: i ( x k,t ) =
Revised: i ( x k,t ) =

Original: N (cid:88) i= w ik,t
Revised: N (cid:88) i

Original: W k,t ˆ f RF ,it ( x k,t ; θ i,t ) (6) where ˆ Θ t = [ θ 1 ,t , . . 
Revised: [Equation] (6) where ˆ Θ t = [ θ 1 ,t , . . .

Original: , θ N,t ] , w k,t
Revised: , θ N,t ] , w k,t

Original: =
Revised: =

Original: [ w 1 k,t , . . .
Revised: [ w 1 k,t , . . .

Original: , w Nk,t ] and W k,t = (cid:80) Ni =1 w ik,t .
Revised: , w Nk,t ] and W k,t = (cid:80) Ni =1 w ik,t .

Original: As it can beinferred from (6), each client constructs its own personalized combination of kernels.
Revised: As it can beinferred from (6), each client constructs its own personalized combination of kernels.

Original: Upon observingthe true label y k,t , the k -th client calculates the losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .
Revised: Upon observingthe true label y k,t , the k -th client calculates the losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .

Original: Then, thek -th client leverages calculated losses to locally update both global and local parameters.
Revised: Then, thek -th client leverages calculated losses to locally update both global and local parameters.

Original: Let θ ik,t +1and w ik,t +1 denote the k -th client’s local updates of θ i,t and w ik,t , respectively.
Revised: Let θ ik,t +1and w ik,t +1 denote the k -th client’s local updates of θ i,t and w ik,t , respectively.

Original: Specifically, the k -thclient utilizes multiplicative update rule to update w ik,t as
Revised: Specifically, the k -thclient utilizes multiplicative update rule to update w ik,t as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where η k is the learning rate of the k -th client.
Revised: where η k is the learning rate of the k -th client.

Original: Note that the k -th client ( ∀ k ∈
Revised: Note that the k -th client ( ∀ k ∈

Original: [ K ] ) does not send itsupdated local parameter w k,t +1 to the server.
Revised: [ K ] ) does not send itsupdated local parameter w k,t +1 to the server.

Original: Clients send their locally updated global parameters tothe server (i.e. θ ik,t +1 ).
Revised: Clients send their locally updated global parameters tothe server (i.e. θ ik,t +1 ).

Original: Aggregating local updates, the server updates global parameters to ˆ Θ t +1 .
Revised: Aggregating local updates, the server updates global parameters to ˆ Θ t +1 .

Original: If all clients send updates associated with all kernels (i.e. θ ik,t +1 , ∀ i ∈ [ N ] ), this requires sending2 ND parameters by each client at each time instant.

Original: When the number of both clients and kernelsare large, the available client-to-server communication bandwidth may not be enough to affordsending 2 NDK parameters per time instant even for small values of D .

Original: Note that reducing N and D

Original: Algorithm
Revised: Algorithm

Original: Input: Weights w ik,t , ∀ i ∈ [ N ] , parameter M and exploration rate 0 < ξ k ≤ 1 .
Revised: Input: Weights w ik,t , ∀ i ∈ [ N ] , parameter M and exploration rate 0 < ξ k ≤ 1 .

Original: Sort the kernels in descending order with respect to weights { w ik,t } Ni =1 .
Revised: Sort the kernels in descending order with respect to weights { w ik,t } Ni =1 .

Original: Obtain the index sequence s 1 , . . .
Revised: Obtain the index sequence s 1 , . . .

Original: , s N such that w s b k,t ≤ w s a k,t
Revised: , s N such that w s b k,t ≤ w s a k,t

Original: if b > a , ∀ a, b ∈ [ N ] .
Revised: if b > a , ∀ a, b ∈ [ N ] .

Original: Open bin B 1 and initialize j = 1 .
Revised: Open bin B 1 and initialize j = 1 .

Original: for all i ∈ [ N ] , the k -th client do if the bin B j includes less than M kernels then
Revised: for all i ∈ [ N ] , the k -th client do if the bin B j includes less than M kernels then

Original: Adds the s i -th kernel to B j . else
Revised: Adds the s i -th kernel to B j . else

Original: Opens new bin B j +1 , adds the s i -th kernel to B j +1 and updates j ← j + 1 .
Revised: Opens new bin B j +1 , adds the s i -th kernel to B j +1 and updates j ← j + 1 .

Original: end if end for
Revised: end if end for

Original: Draw an index I k,t via PMF q k,t in (8).
Revised: Draw an index I k,t via PMF q k,t in (8).

Original: Output: S k,t : indices set of kernels in the selected bin B I k,t degrade the performance of online federated MKL.
Revised: Output: S k,t : indices set of kernels in the selected bin B I k,t If all clients send updates associated with all kernels (i.e. θ ik,t +1 , ∀ i ∈ [ N ] ), this requires sending2 ND parameters by each client at each time instant. When the number of both clients and kernelsare large, the available client-to-server communication bandwidth may not be enough to affordsending 2 NDK parameters per time instant even for small values of D . Note that reducing N and Ddegrade the performance of online federated MKL.

Original: Reducing N (the number of kernels), decreasesthe flexibility of clients to construct their ideal kernel using convex combination of kernels in thedictionary.
Revised: Reducing N (the number of kernels), decreasesthe flexibility of clients to construct their ideal kernel using convex combination of kernels in thedictionary.

Original: Reducing D can degrade the accuracy of RF approximation.
Revised: Reducing D can degrade the accuracy of RF approximation.

Original: The present paper proposes an algorithmic framework to enable clients to perform online functionapproximation with sufficiently large dictionary of kernels while the available clients-to-servercommunication bandwidth can afford sending updates from clients to the server when a desirablevalue for the number of random features D is chosen.
Revised: The present paper proposes an algorithmic framework to enable clients to perform online functionapproximation with sufficiently large dictionary of kernels while the available clients-to-servercommunication bandwidth can afford sending updates from clients to the server when a desirablevalue for the number of random features D is chosen.

Original: To this end, at each time instant, each clientrandomly chooses a subset of M kernels among all N kernels in the dictionary.
Revised: To this end, at each time instant, each clientrandomly chooses a subset of M kernels among all N kernels in the dictionary.

Original: Then, each clientupdates and sends the global parameters of the chosen M kernels to the server instead of updatingand sending global parameters of all kernels.
Revised: Then, each clientupdates and sends the global parameters of the chosen M kernels to the server instead of updatingand sending global parameters of all kernels.

Original: To choose a subset of M kernels, each client splitskernels into some bins and draws randomly one of the bins at each time instant.
Revised: To choose a subset of M kernels, each client splitskernels into some bins and draws randomly one of the bins at each time instant.

Original: Each bin containsat most M kernels and each client updates and sends global parameters associated with kernels inthe chosen bin.
Revised: Each bin containsat most M kernels and each client updates and sends global parameters associated with kernels inthe chosen bin.

Original: In order to distribute kernels among bins, at first the k -th client sorts kernels in descending order according to kernels’ weights { w ik,t } Ni =1 .
Revised: In order to distribute kernels among bins, at first the k -th client sorts kernels indescending order according to kernels’ weights { w ik,t } Ni =1 .

Original: Let B j represents the j -th bin of kernels.
Revised: Let B j represents the j -th bin of kernels.

Original: The k -th client adds kernels from sorted list one by one to B j until either all kernels are assigned to abin or the number of kernels in B j reaches M .
Revised: The k -th client adds kernels from sorted list one by one to B j until either all kernels are assigned to abin or the number of kernels in B j reaches M .

Original: When there are some kernels that are not assigned toany bins while there are M kernels in B j , the k -th client opens the bin B j +1 and adds kernels to thisbin.
Revised: When there are some kernels that are not assigned toany bins while there are M kernels in B j , the k -th client opens the bin B j +1 and adds kernels to thisbin.

Original: This continues until all kernels are assigned to a bin.
Revised: This continues until all kernels are assigned to a bin.

Original: As it can be inferred from the procedureof distributing kernels into bins, the number of bins at every client is m = (cid:6) NM (cid:7) .
Revised: As it can be inferred from the procedureof distributing kernels into bins, the number of bins at every client is m = (cid:6) N M (cid:7) .

Original: Furthermore, itcan be concluded that B 1 includes M kernels with the largest weights while the bin B m includes
Revised: Furthermore, itcan be concluded that B 1 includes M kernels with the largest weights while the bin B m includes

Original: N − ( m − 1) M kernels with lowest weights.
Revised: N − ( m − 1) M kernels with lowest weights.

Original: The k -th client assigns the weight u jk,t at time t to B jdefined as u jk,t = (cid:80)
Revised: The k -th client assigns the weight u jk,t at time t to B jdefined as u jk,t = (cid:80)

Original: κ i ∈B j w ik,t .
Revised: κ i ∈B j w ik,t .

Original: The k -th client draws one of the bins according to the probabilitymass function (PMF) q k,t defined as
Revised: The k -th client draws one of the bins according to the probabilitymass function (PMF) q k,t defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation] where U k,t = (cid:80) mj =1 u jk,t and 0 < ξ k ≤ 1 is an exploration rate determined by the k -th client.
Revised: (8) where U k,t = (cid:80) mj =1 u jk,t and 0 < ξ k ≤ 1 is an exploration rate determined by the k -th client.

Original: Let
Revised: Let

Original: I k,t be the index of the chosen bin by the k -th client at time t .
Revised: I k,t be the index of the chosen bin by the k -th client at time t .

Original: The PMF in (8) constitutes trade-offbetween exploitation and exploration.
Revised: The PMF in (8) constitutes trade-offbetween exploitation and exploration.

Original: According to the first term in the right hand side of (8), itis more probable that the k -th client draws a bin which includes kernels with larger weights w ik,t .
Revised: According to the first term in the right hand side of (8), itis more probable that the k -th client draws a bin which includes kernels with larger weights w ik,t .

Original: Hence, it is more probable that the k -th client collaborates in updating the global parameters of akernel with larger weight w ik,t .
Revised: Hence, it is more probable that the k -th client collaborates in updating the global parameters of akernel with larger weight w ik,t .

Original: Let S k,t denotes the set which includes the indices of kernels in thechosen bin at time t .
Revised: Let S k,t denotes the set which includes the indices of kernels in thechosen bin at time t .

Original: The Algorithm 1 summarizes the procedure that the k -th client determines theset S k,t .
Revised: The Algorithm 1 summarizes the procedure that the k -th client determines theset S k,t .

Original: According to Algorithm 1, kernel subset selection is personalized since each client choosesits own subset of kernels to update their parameters.
Revised: According to Algorithm 1, kernel subset selection is personalized since each client choosesits own subset of kernels to update their parameters.

Original: Input: Kernels κ i , i = 1 , ..., N , learning rate η > 

Revised: Input: Kernels κ i , i = 1 , ..., N , learning rate η >

Original: and the number of random features D .
Revised: and the number of random features D .

Original: Initialize: θ i, 1 = 0 , w ik, 1 = 1 , ∀ i ∈
Revised: Initialize: θ i, 1 = 0 , w ik, 1 = 1 , ∀ i ∈

Original: [ N ] , ∀ k ∈
Revised: [ N ] , ∀ k ∈

Original: [ K ] .
Revised: [ K ] .

Original: for t = 1 , . . .
Revised: for t = 1 , . . .

Original: , T do
Revised: , T do

Original: The server transmits the global parameters ˆ Θ t =
Revised: The server transmits the global parameters ˆ Θ t =

Original: [ θ 1 ,t , . .
Revised: [ θ 1 ,t , . .

Original: , θ N,t ] to all clients.
Revised: , θ N,t ] to all clients.

Original: for all k ∈ [ K ] , the k th client do
Revised: for all k ∈ [ K ] , the k th client do

Original: Receive one datum x k,t .
Revised: Receive one datum x k,t .

Original: Predicts ˆ f ( x k,t ;  Θ t , w k,t ) via (6).
Revised: Predicts ˆ f ( x k,t ; ˆ Θ t , w k,t ) via (6).

Original: Calculates losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .
Revised: Calculates losses L ( ˆ f RF ,it ( x k,t ; θ i,t ) , y k,t ) , ∀ i ∈ [ N ] .

Original: Updates
Revised: Updates

Original: Selects a  of kernel
Revised: Selects a subset of kernel

Original: Algorithm
Revised: Algorithm

Original: end for
Revised: end for

Original: The server updates θ
Revised: The server updates θ

Original: N end for
Revised: N end for

Original: Let p ik,t denotes the probability that i ∈
Revised: Let p ik,t denotes the probability that i ∈

Original: S k,t .
Revised: S k,t .

Original: Then p ik,t = q b i k,t where b i is the index of the binwhich includes the i -th kernel.
Revised: Then p ik,t = q b i k,t where b i is the index of the binwhich includes the i -th kernel.

Original: The k -th client updates global parameters locally as follows
Revised: The k -th client updates global parameters locally as follows

Original: [Equation]
Revised: [Equation]

Original: (9) where 1 i ∈S k,t denotes an indicator function and it is 1 when i ∈ S k,t .
Revised: (9) where 1 i ∈S k,t denotes an indicator function and it is 1 when i ∈ S k,t .

Original: The update rule in (9) impliesthat when i / ∈ S
Revised: The update rule in (9) impliesthat when i / ∈ S

Original: k,t , the k -th client does not update θ i,t (i.e. θ ik,t +

Revised: k,t , the k -th client does not update θ i,t (i.e. θ ik,t +

Original: = θ i,t ).
Revised: = θ i,t ).

Original: Therefore, the k -th clientsends θ ik,t +1 to the server only if i ∈ S k,t .
Revised: Therefore, the k -th clientsends θ ik,t +1 to the server only if i ∈ S k,t .

Original: Therefore, at each time instant, each client needs to sendat most 2 MD parameters to the server.
Revised: Therefore, at each time instant, each client needs to sendat most 2 MD parameters to the server.

Original: Let C i,t be a set of client indices such that k ∈ C i,t if the k -thclient sends θ ik,t +1 to the server.
Revised: Let C i,t be a set of client indices such that k ∈ C i,t if the k -thclient sends θ ik,t +1 to the server.

Original: Upon aggregating updates from clients, the server updates θ i,t as
Revised: Upon aggregating updates from clients, the server updates θ i,t as

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Algorithm 2 summarizes the proposed personalized online federated multi-kernel learning algorithmcalled POF-MKL.
Revised: Algorithm 2 summarizes the proposed personalized online federated multi-kernel learning algorithmcalled POF-MKL.

Original: It is useful to note that using our proposed POF-MKL, the server cannot find thegradients ∇L ( θ ⊤ i,
Revised: It is useful to note that using our proposed POF-MKL, the server cannot find thegradients ∇L ( θ ⊤ i,

Original: t z i ( x k,t ) , y k,t ) from updates received from clients.
Revised: t z i ( x k,t ) , y k,t ) from updates received from clients.

Original: Instead, the server can find∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p
Revised: Instead, the server can find∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p

Original: ik,t where p ik,t is a time-varying value determined locally by the k -th client.
Revised: ik,t where p ik,t is a time-varying value determined locally by the k -th client.

Original: This can promote the privacy of the proposed POF-MKL since exchanging the gradients can behazardous to the privacy of federated learning (see e.g. [52, 14]).
Revised: This can promote the privacy of the proposed POF-MKL since exchanging the gradients can behazardous to the privacy of federated learning (see e.g. [52, 14]).

Original: Complexity.
Revised: Complexity.

Original: Each client needs to store d -dimensional D random feature vectors for each kernel.
Revised: Each client needs to store d -dimensional D random feature vectors for each kernel.

Original: Therefore, the memory requirement of each client to implement function approximation using POFMKL is O ( dND ) .
Revised: Therefore, the memory requirement of each client to implement function approximation using POFMKL is O ( dND ) .

Original: Using POF-MKL, at each time instant, each client needs to perform O ( dND )operations including inner products and summations.
Revised: Using POF-MKL, at each time instant, each client needs to perform O ( dND )operations including inner products and summations.

Original: Furthermore, when ξ k < 1 , in order to choosea subset of kernels, the k -th client needs to sort kernels which imposes worst case computationalcomplexity of O ( N log N ) .
Revised: Furthermore, when ξ k < 1 , in order to choosea subset of kernels, the k -th client needs to sort kernels which imposes worst case computationalcomplexity of O ( N log N ) .

Original: However, when ξ k = 1 , according to PMF in (8), the k -th client choosesone bin uniformly at random and as a result in this case the k -th client does not need to sort kernels.
Revised: However, when ξ k = 1 , according to PMF in (8), the k -th client choosesone bin uniformly at random and as a result in this case the k -th client does not need to sort kernels.

Original: Therefore, setting ξ k < 1 , the computational complexity for the k -th client is O ( dND + N log N )while setting ξ k = 1 , the computational complexity of the k -th client at each time instant is O ( dND ) .
Revised: Therefore, setting ξ k < 1 , the computational complexity for the k -th client is O ( dND + N log N )while setting ξ k = 1 , the computational complexity of the k -th client at each time instant is O ( dND ) .

Original: 3.2 Regret Analysis
Revised: 3.2 Regret Analysis

Original: The present section analyzes the regret of the proposed POF-MKL.
Revised: The present section analyzes the regret of the proposed POF-MKL.

Original: Specifically, two types of regret
Revised: Specifically, two types of regret

Original: R k,T and R s,T are considered for the k -th client and the server, respectively.
Revised: R k,T and R s,T are considered for the k -th client and the server, respectively.

Original: The performance of thek -th client utilizing POF-MKL is analyzed in terms of regret defined as
Revised: The performance of thek -th client utilizing POF-MKL is analyzed in terms of regret defined as

Original: [Equation]
Revised: [Equation]

Original: where R k,T measures the cumulative difference between the loss of the k -th client and the lossof the RF approximation of the kernel with minimum loss among all kernels’ RF approximations.
Revised: where R k,T measures the cumulative difference between the loss of the k -th client and the lossof the RF approximation of the kernel with minimum loss among all kernels’ RF approximations.

Original: Let α ∗ ik,t ,
Revised: Let α ∗ ik,t ,

Original: ∀ t ∈
Revised: ∀ t ∈

Original: [ T ] , ∀ k ∈
Revised: [ T ] , ∀ k ∈

Original: [ K ] represents the optimal coefficients associated with the i -th kernelsuch that f ∗ i ( x )
Revised: [ K ] represents the optimal coefficients associated with the i -th kernelsuch that f ∗ i ( x )

Original: = (cid:80) Tt =1 (cid:80)
Revised: = (cid:80) Tt =1 (cid:80)

Original: Kk
Revised: Kk

Original: =1 α ∗ ik,t κ i ( x , x k,t ) .
Revised: =1 α ∗ ik,t κ i ( x , x k,t ) .

Original: Then the best function approximator is definedas f ∗ ( · ) ∈ arg min f ∗ i ,i ∈
Revised: Then the best function approximator is definedas f ∗ ( · ) ∈ arg min f ∗ i ,i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) .
Revised: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) .

Original: Furthermore, the regret of the server isdefined as the cumulative difference between the loss of POF-MKL and that of the best functionapproximator over all data samples distributed among clients which can be expressed as
Revised: Furthermore, the regret of the server isdefined as the cumulative difference between the loss of POF-MKL and that of the best functionapproximator over all data samples distributed among clients which can be expressed as

Original: [Equation]
Revised: [Equation]

Original: In order to analyze the regret of POF-MKL, suppose that the following assumptions hold true:(as1) L ( θ ⊤ i,
Revised: In order to analyze the regret of POF-MKL, suppose that the following assumptions hold true: (as1) L ( θ ⊤ i,t z

Original: t z i ( x k,t ) , y k,t ) , ∀ k ∈
Revised: i ( x k,t ) , y k,t ) , ∀ k ∈

Original: [ K ] is convex with respect to θ i,t at each time instant t .(as2)
Revised: [ K ] is convex with respect to θ i,t at each time instant t . (as2)

Original: For θ in a bounded set satisfying ∥ θ ∥ ≤ C , the loss function and its gradient are boundedas 0 ≤ L ( θ
Revised: For θ in a bounded set satisfying ∥ θ ∥ ≤ C , the loss function and its gradient are bounded

Original:  ⊤ z i ( x k,t ) , y k,t ) ≤ 1 and ∥∇L ( θ ⊤ z i ( x k,t ) , y k,t ) ∥ ≤ L .
Revised: as 0 ≤ L ( θ ⊤ z i ( x k,t ) , y k,t ) ≤ 1 and ∥∇L ( θ ⊤ z i ( x k,t ) , y k,t ) ∥ ≤ L .

Original: Moreover, each data sample isbounded as ∥ x k,t ∥ ≤ 1 , ∀ k ∈
Revised: Moreover, each data sample is bounded as ∥ x k,t ∥ ≤ 1 , ∀ k ∈

Original: [ K ] , ∀ t ∈ [ T ] .(as3)
Revised: [ K ] , ∀ t ∈ [ T ] .

Original:  Kernels κ i ( · ) , ∀ i ∈
Revised: (as3) Kernels κ i ( · ) , ∀ i ∈

Original: [ N ] are shift-invariant with κ i ( 0 )
Revised: [ N ] are shift-invariant with κ i ( 0 )

Original: = 1 , ∀ i ∈
Revised: = 1 , ∀ i ∈

Original: [ N ] .
Revised: [ N ] .

Original: The following theorem investigates the regret of the k -th client according to the k -th client data.
Revised: The following theorem investigates the regret of the k -th client according to the k -th client data.

Original: Theproof of the following Theorem can be found in Appendix A.
Revised: Theproof of the following Theorem can be found in Appendix A.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under (as1)–(as3), the regret of the k -th client with respect to the best kernel satisfies
Revised: Under (as1)–(as3), the regret of the k -th client with respect to the best kernel satisfies

Original: T(cid:88)
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 shows that by setting η k
Revised: Theorem 1 shows that by setting η k

Original: = O (cid:16) 1 √ T (cid:17) , the k -th client achieves sub-linear regret of O ( √ T ) .
Revised: = O (cid:16) 1 √ T (cid:17) , the k -th client achieves sub-linear regret of O ( √ T ) .

Original: Furthermore, Theorem 1 shows that POF-MKL can deal with heterogeneous data among clients sincethe regret of each client defined in (11) is calculated with respect to the corresponding client data.
Revised: Furthermore, Theorem 1 shows that POF-MKL can deal with heterogeneous data among clients sincethe regret of each client defined in (11) is calculated with respect to the corresponding client data.

Original: The following theorem studies the regret of the server with respect to the best function approximator.
Revised: The following theorem studies the regret of the server with respect to the best function approximator.

Original: The proof can be found in Appendix B.
Revised: The proof can be found in Appendix B.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let i ∗ :=
Revised: Let i ∗ :=

Original: arg min i ∈
Revised: arg min i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) and σ i be the second Fouriermoment of the i -th kernel.
Revised: Kk =1 L ( f ∗ i ( x k,t ) , y k,t ) and σ i be the second Fouriermoment of the i -th kernel.

Original: Under (as1)–(as3), the regret of the server with respect to the best functionapproximator satisfies
Revised: Under (as1)–(as3), the regret of the server with respect to the best functionapproximator satisfies

Original: [Equation] (cid:88) 
Revised: T (cid:88) t

Original: [Equation]
Revised: [Equation]

Original: K (cid:88)
Revised: K (cid:88)

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: L (  f 
Revised: L ( ˆ f ( x k [Equation] t ; ˆ Θ t [Equation]

Revised: w k

Original: [Equation]
Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Revised: [Equation]

Revised: −

Revised: T (cid:88) t

Original: [Equation]
Revised: [Equation]

Revised: K (cid:88) k

Original:  (14) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) where C := max i ∈
Revised: = L ( f ∗ ( x k,t [Equation] (14) with probability at least 1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) where C := max i ∈

Original: [ N ] (cid:80) Tt =1 (cid:80)
Revised: [ N ] (cid:80) Tt =1 (cid:80)

Original: Kk
Revised: Kk

Original: =1 α ∗ ik,t .
Revised: =1 α ∗ ik,t .

Original: As it can be inferred from (14), the regret of the server with respect to the best function approximatordepends on 1 p i ∗ k,t .
Revised: As it can be inferred from (14), the regret of the server with respect to the best function approximatordepends on 1 p i ∗ k,t .

Original: From (8) and the fact that p ik,t = q b i k,t , it can be concluded that p i ∗ k,t > ξ k m .
Revised: From (8) and the fact that p ik,t = q b i k,t , it can be concluded that p i ∗ k,t > ξ k m .

Original: Thus,setting ξ k = O (1) , then p ik,t > O ( MN ) .
Revised: Thus,setting ξ k = O (1) , then p ik,t > O ( MN ) .

Original: The regret bound in (14) shows that setting η = O (cid:16)(cid:113) MNT (cid:17)and ϵ = η k = 1 √ T , ∀ k ∈
Revised: The regret bound in (14) shows that setting η = O (cid:16)(cid:113) MNT (cid:17)and ϵ = η k = 1 √ T , ∀ k ∈

Original: [ K ] , the server obtains regret of O (cid:16)(cid:113) NM T (cid:17) with probability at least1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) .
Revised: [ K ] , the server obtains regret of O (cid:16)(cid:113) NM T (cid:17) with probability at least1 − 2 8 (cid:0) σ i ∗ ϵ (cid:1) 2 exp (cid:16) − Dϵ 2 4( d +2) (cid:17) .

Original: This shows that increasing M tighten the regret bound and increasing
Revised: This shows that increasing M tighten the regret bound and increasing

Original: D increases the probability that the regret bound in (14) holds true.
Revised: D increases the probability that the regret bound in (14) holds true.

Original: However, using POF-MKL, eachclient needs to transmit MD parameters at each time instant.
Revised: However, using POF-MKL, eachclient needs to transmit MD parameters at each time instant.

Original: Since both M and D are determined bythe algorithm POF-MKL, this shows that POF-MKL can provide flexibility to tighten regret boundwhile the available clients-to-server communication bandwidth can afford transmission of clients’updates to the server.
Revised: Since both M and D are determined bythe algorithm POF-MKL, this shows that POF-MKL can provide flexibility to tighten regret boundwhile the available clients-to-server communication bandwidth can afford transmission of clients’ updates to the server.

Revised: It is useful to mention that choosing larger value for ξ k increases the lowerbound of p i ∗ k,t and as a result the optimal choice for ξ k in terms of regret is ξ k = 1 .

Revised: However,choosing smaller values for ξ k makes the value of p ik,t more dependent on weights { w ik,t } Kk =1(c.f.

Revised: Therefore, choosing smaller values for ξ k makes p ik,t less predictable.

Revised: This makesestimating ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) given ∇L ( θ ⊤ i,t z i ( x k,t ) , y k,t ) /p

Revised: ik,t more difficult which leads tobetter protection of privacy.

Original: Comparison with personalized federated learning.
Revised: Comparison with personalized federated learning.

Original: In order to deal with heterogeneous data amongclients, personalized federated learning has been studied extensively in literature (see [43, 8, 9, 12,19, 10, 28, 27, 41, 6, 1, 31, 49, 2, 45, 5]).
Revised: In order to deal with heterogeneous data amongclients, personalized federated learning has been studied extensively in literature (see [43, 8, 9, 12,19, 10, 28, 27, 41, 6, 1, 31, 49, 2, 45, 5]).

Original: Utilizing model-agnostic meta-learning [13], personalizedfederated learning algorithms have been proposed in [12, 1].
Revised: Utilizing model-agnostic meta-learning [13], personalizedfederated learning algorithms have been proposed in [12, 1].

Original: In [41, 5], personalized federatedlearning algorithms have been designed by learning hyper-networks [17].
Revised: In [41, 5], personalized federatedlearning algorithms have been designed by learning hyper-networks [17].

Original: In [31], a personalizedmodel is a linear combination of a set of shared component models such that each client constructsits personalized mixture of models.
Revised: In [31], a personalizedmodel is a linear combination of a set of shared component models such that each client constructsits personalized mixture of models.

Original: However, in aforementioned personalized federated learningworks, clients are assumed to store a dataset to perform local updates with.
Revised: However, in aforementioned personalized federated learningworks, clients are assumed to store a dataset to perform local updates with.

Original: Therefore, when clientsare not able to store data in batch and they have to make a decision upon receiving a new datasample, aforementioned works in personalized federated learning cannot guarantee sub-linear regretfor clients.
Revised: Therefore, when clientsare not able to store data in batch and they have to make a decision upon receiving a new datasample, aforementioned works in personalized federated learning cannot guarantee sub-linear regretfor clients.

Original: However, according to Theorems 1 and 2, POF-MKL provides sub-linear regret for clientswhen clients cannot store data in batch and make decision in an online fashion.
Revised: However, according to Theorems 1 and 2, POF-MKL provides sub-linear regret for clientswhen clients cannot store data in batch and make decision in an online fashion.

Original: Comparison with online federated learning [33].
Revised: Comparison with online federated learning [33].

Original: Fed-OMD algorithm has been proposed in [33]which enables clients to perform their learning task in an online and federated fashion while it is provedthat Fed-OMD enjoys sub-linear regret when the loss function is convex with respect to parametersrequired to be learnt at each time instant.
Revised: Fed-OMD algorithm has been proposed in [33]which enables clients to perform their learning task in an online and federated fashion while it is provedthat Fed-OMD enjoys sub-linear regret when the loss function is convex with respect to parametersrequired to be learnt at each time instant.

Original: The proposed POF-MKL differs from Fed-OMD in the sensethat Fed-OMD cannot guarantee sub-linear regret when it comes to performing the online learningtask with RF approximations of multiple kernels since the loss function L ( (cid:80) Ni
Revised: The proposed POF-MKL differs from Fed-OMD in the sensethat Fed-OMD cannot guarantee sub-linear regret when it comes to performing the online learningtask with RF approximations of multiple kernels since the loss function L ( (cid:80) Ni

Original: =1 w i,t θ ⊤ i,t z i ( x ) , y ) isnot convex with respect to both θ i,t and w i,t .
Revised: =1 w i,t θ ⊤ i,t z i ( x ) , y ) isnot convex with respect to both θ i,t and w i,t .

Original: However, according to Theorems 1 and 2, the proposed
Revised: However, according to Theorems 1 and 2, the proposed

Original: POF-MKL guarantees sub-linear regret.
Revised: POF-MKL guarantees sub-linear regret.

Original: Comparison with [21].
Revised: Comparison with [21].

Original: Online federated MKL algorithms called vM-KOFL and eM-KOFL havebeen presented in [21].
Revised: Online federated MKL algorithms called vM-KOFL and eM-KOFL havebeen presented in [21].

Original: Both POF-MKL and algorithms in [21] exploit random feature approximationto alleviate computational complexity of online kernel learning.
Revised: Both POF-MKL and algorithms in [21] exploit random feature approximationto alleviate computational complexity of online kernel learning.

Original: Furthermore, both POF-MKL andalgorithms in [21] learn a linear combination of kernels.
Revised: Furthermore, both POF-MKL andalgorithms in [21] learn a linear combination of kernels.

Original: The proposed POF-MKL has the followingadvantages and innovations compared to vM-KOFL and eM-KOFL: i)
Revised: The proposed POF-MKL has the followingadvantages and innovations compared to vM-KOFL and eM-KOFL: i)

Original: The proposed POF-MKLallows clients to learn their own personalized combination of kernels (c.f. (6)).
Revised: The proposed POF-MKLallows clients to learn their own personalized combination of kernels (c.f. (6)).

Original: As it is proved in
Revised: As it is proved in

Original: Theorem 1, the proposed POF-MKL can deal with heterogeneous data among clients in the sensethat using POF-MKL each client guarantees sub-linear regret with respect to the best kernel RFapproximation according to the corresponding client data.
Revised: Theorem 1, the proposed POF-MKL can deal with heterogeneous data among clients in the sensethat using POF-MKL each client guarantees sub-linear regret with respect to the best kernel RFapproximation according to the corresponding client data.

Original: However, both vM-KOFL and eM-KOFLare not able to provide such guarantee.
Revised: However, both vM-KOFL and eM-KOFLare not able to provide such guarantee.

Original: ii) Using vM-KOFL, each client needs to send ( N + 1) Dparameters to the server.
Revised: ii) Using vM-KOFL, each client needs to send ( N + 1) Dparameters to the server.

Original: However, using the proposed POF-MKL, each client needs to send MDparameters to the server such that M ≤ N is determined by POF-MKL and can be chosen to bemuch smaller than N . iii)
Revised: However, using the proposed POF-MKL, each client needs to send MDparameters to the server such that M ≤ N is determined by POF-MKL and can be chosen to bemuch smaller than N . iii)

Original: In eM-KOFL, the server chooses a kernel at each time instant and clientssend their local updates associated with the chosen kernel by the server.
Revised: In eM-KOFL, the server chooses a kernel at each time instant and clientssend their local updates associated with the chosen kernel by the server.

Original: The proposed POF-MKLprovides more flexibility compared to eM-KOFL in the sense that using POF-MKL each client cansend local updates of M ≥ 1 kernels to the server.
Revised: The proposed POF-MKLprovides more flexibility compared to eM-KOFL in the sense that using POF-MKL each client cansend local updates of M ≥ 1 kernels to the server.

Original: And each client chooses its own subset of kernelsto send their updates to the server.
Revised: And each client chooses its own subset of kernelsto send their updates to the server.

Original: Therefore, even though POF-MKL sets M to 1 , it is possible thatat a time instant the server receives updates associated with all kernels in the dictionary.
Revised: Therefore, even though POF-MKL sets M to 1 , it is possible thatat a time instant the server receives updates associated with all kernels in the dictionary.

Original: It is useful tomention that using eM-KOFL the cumulative regret of all clients is sub-linear with respect to the bestkernel RF approximation with probability 1 − δ
Revised: It is useful tomention that using eM-KOFL the cumulative regret of all clients is sub-linear with respect to the bestkernel RF approximation with probability 1 − δ

Original: where 0 < δ ≤ 1 .
Revised: where 0 < δ ≤ 1 .

Original: However, utilizing the updaterule in (9), using the proposed POF-MKL, each client obtains sub-linear regret with respect to RFapproximation of its best kernel with probability 1.
Revised: However, utilizing the updaterule in (9), using the proposed POF-MKL, each client obtains sub-linear regret with respect to RFapproximation of its best kernel with probability 1.

Original:  Experiments
Revised:  Experiments

Original: We tested the performance of the proposed POF-MKL for online regression task through a set ofexperiments.
Revised: We tested the performance of the proposed POF-MKL for online regression task through a set ofexperiments.

Original: The performance of POF-MKL is compared with the baselines PerFedAvg [12], OFSKL[33], OFMKL-Avg [33], vM-KOFL [21] and eM-KOFL [21].
Revised: The performance of POF-MKL is compared with the baselines PerFedAvg [12], OFSKL[33], OFMKL-Avg [33], vM-KOFL [21] and eM-KOFL [21].

Original: PerFedAvg refers to the personalizedfederated averaging algorithm in [12].
Revised: PerFedAvg refers to the personalizedfederated averaging algorithm in [12].

Original: In the experiments, PerFedAvg employs a feedforward neuralnetwork model.
Revised: In the experiments, PerFedAvg employs a feedforward neuralnetwork model.

Original: More information about the implementation of PerFedAvg can be found in Appendix
Revised: More information about the implementation of PerFedAvg can be found in Appendix

Original: C. OFSKL and OFMKL-Avg are two variations of Fed-OMD [33].
Revised: C. OFSKL and OFMKL-Avg are two variations of Fed-OMD [33].

Original: OFSKL leverages Fed-OMD[33] when a single radial basis function (RBF) with bandwidth of 10 is employed to perform thelearning task.
Revised: OFSKL leverages Fed-OMD[33] when a single radial basis function (RBF) with bandwidth of 10 is employed to perform thelearning task.

Original: In OFMKL-Avg, kernels are learned independently from each other using Fed-OMD [33] and the prediction is the average of approximations given by kernels.
Revised: In OFMKL-Avg, kernels are learned independently from each other using Fed-OMD[33] and the prediction is the average of approximations given by kernels.

Original: Moreover, vM-KOFL andeM-KOFL are online federated MKL algorithms of [21] such that vM-KOFL requires transmissionof all kernel updates at every time instant while eM-KOFL requires transmission of a kernel update ateach time instant.
Revised: Moreover, vM-KOFL andeM-KOFL are online federated MKL algorithms of [21] such that vM-KOFL requires transmissionof all kernel updates at every time instant while eM-KOFL requires transmission of a kernel update ateach time instant.

Original: In the experiments, each client observes 500 samples until the end of the learningtask meaning that T = 500 .
Revised: In the experiments, each client observes 500 samples until the end of the learningtask meaning that T = 500 .

Original: The performance of the proposed POF-MKL and other baselines aretested on the following real datasets downloaded from UCI machine learning repository [11]: Naval[7], UJI [46], Air [50] and WEC [34].
Revised: The performance of the proposed POF-MKL and other baselines aretested on the following real datasets downloaded from UCI machine learning repository [11]: Naval[7], UJI [46], Air [50] and WEC [34].

Original: More detailed information about datasets can be found in
Revised: More detailed information about datasets can be found in

Original: Appendix C. Data samples of Naval and UJI datasets are distributed i.i.d among clients.
Revised: Appendix C. Data samples of Naval and UJI datasets are distributed i.i.d among clients.

Original: Data samplesin Air and WEC datasets are distributed non-i.i.d among clients.
Revised: Data samplesin Air and WEC datasets are distributed non-i.i.d among clients.

Original: More inforamtion about distributingdata samples among clients can be found in Appendix C.
Revised: More inforamtion about distributingdata samples among clients can be found in Appendix C.

Original: The number of clients for Naval, UJI,
Revised: The number of clients for Naval, UJI,

Original: Air and WEC datasets are 23 , 42 , 240 and 560 , respectively.
Revised: Air and WEC datasets are 23 , 42 , 240 and 560 , respectively.

Original: The dictionary of kernels consists of51 RBFs with different bandwidth such that the bandwidth of the i -th kernel is σ i =
Revised: The dictionary of kernels consists of51 RBFs with different bandwidth such that the bandwidth of the i -th kernel is σ i =

Original: i − 52 25 .
Revised: i − 52 25 .

Original: Weconsider the case where the clients-to-server communication bandwidth is limited such that at eachtime instant, the maximum number of parameters that a client is allowed to transmit to the serveris 1000 .
Revised: Weconsider the case where the clients-to-server communication bandwidth is limited such that at eachtime instant, the maximum number of parameters that a client is allowed to transmit to the serveris 1000 .

Original: Furthermore, the memory and computational capability of clients are limited such that themaximum value can be picked for the number of random features D is 100 .
Revised: Furthermore, the memory and computational capability of clients are limited such that themaximum value can be picked for the number of random features D is 100 .

Original: The experiments werecarried for 20 different sets of random feature vectors.
Revised: The experiments werecarried for 20 different sets of random feature vectors.

Original: The performance of algorithms is measuredusing average of mean squared error (MSE) defined as
Revised: The performance of algorithms is measuredusing average of mean squared error (MSE) defined as

Original: [Equation]
Revised: [Equation]

Original: where ˆ y jk,t denotes the prediction of the k -th client at time instant t corresponding to the j -th set ofrandom feature vectors.
Revised: where ˆ y jk,t denotes the prediction of the k -th client at time instant t corresponding to the j -th set ofrandom feature vectors.

Original: Learning rates are set to η = η k = 1 √ T , ∀ k .
Revised: Learning rates are set to η = η k = 1 √ T , ∀ k .

Original: Also, exploration rates are set toξ k = 1 , ∀ k .
Revised: Also, exploration rates are set toξ k = 1 , ∀ k .

Original: The performance of POF-MKL with different ξ k is studied in Appendix C.
Revised: The performance of POF-MKL with different ξ k is studied in Appendix C.

Original: Table 1 presents the MSE and run time performance of online federated kernel learning algorithmson real datasets.
Revised: Table 1 presents the MSE and run time performance of online federated kernel learning algorithmson real datasets.

Original: Run time refers to average total run time of clients to perform online learning taskon the entire data samples that they observe.
Revised: Run time refers to average total run time of clients to perform online learning taskon the entire data samples that they observe.

Original: In Table 1, M refers to the number of kernels whoseupdates are sent to the server after prediction at each time instant.
Revised: In Table 1, M refers to the number of kernels whoseupdates are sent to the server after prediction at each time instant.

Original: And, D is the number of randomfeatures.
Revised: And, D is the number of randomfeatures.

Original: Comparing MSE of POF-MKL with that of OFMKL-Avg, it can be concluded that learningthe weights to combine kernels provides higher accuracy than averaging kernels’ predictions.
Revised: Comparing MSE of POF-MKL with that of OFMKL-Avg, it can be concluded that learningthe weights to combine kernels provides higher accuracy than averaging kernels’ predictions.

Original: Table 1shows that POF-MKL with M = 1 provides lower MSE than eM-KOFL.
Revised: Table 1shows that POF-MKL with M = 1 provides lower MSE than eM-KOFL.

Original: Using eM-KOFL, at eachtime instant, the server receives updates belong to only one kernel.
Revised: Using eM-KOFL, at eachtime instant, the server receives updates belong to only one kernel.

Original: However, using POF-MKL with
Revised: However, using POF-MKL with

Original: M = 1 , each client sends an update belongs to a kernel which is selected by the client.
Revised: M = 1 , each client sends an update belongs to a kernel which is selected by the client.

Original: Therefore, theserver receives updates associated with different kernels even though M = 1 .
Revised: Therefore, theserver receives updates associated with different kernels even though M = 1 .

Original: Therefore, experimentalresults show the effectiveness of the personalized kernel selection provided by POF-MKL.
Revised: Therefore, experimentalresults show the effectiveness of the personalized kernel selection provided by POF-MKL.

Original: It can beobserved that POF-MKL with M = 25 obtains lower MSE than those of POF-MKL with M = 51and vM-KOFL.
Revised: It can beobserved that POF-MKL with M = 25 obtains lower MSE than those of POF-MKL with M = 51and vM-KOFL.

Original: Since each client is allowed to send at most 1000 parameters per time instant, ifclients send updates of all kernels at every time instant as this is the case in vM-KOFL, D cannotbe chosen to be greater than 9 .
Revised: Since each client is allowed to send at most 1000 parameters per time instant, ifclients send updates of all kernels at every time instant as this is the case in vM-KOFL, D cannotbe chosen to be greater than 9 .

Original: However, setting M = 25 , POF-MKL can set D = 

Revised: However, setting M = 25 , POF-MKL can set D =

Original: whichcan improve the accuracy of online regression task compared to the case where D = 9 .
Revised: whichcan improve the accuracy of online regression task compared to the case where D = 9 .

Original: Note thataccording to Theorem 2, increase in D increases the probability that the server achieves sub-linearregret with respect to the best function approximator.
Revised: Note thataccording to Theorem 2, increase in D increases the probability that the server achieves sub-linearregret with respect to the best function approximator.

Original: Furthermore, POF-MKL with M = 51 achieveslower MSE than vM-KOFL even if data samples are distributed i.i.d among clients. 
Revised: Furthermore, POF-MKL with M = 51 achieveslower MSE than vM-KOFL even if data samples are distributed i.i.d among clients. This shows [Table]

Original: This showsthat the proposed POF-MKL can better cope with heterogeneous data among clients which is inagreement with theoretical results in Theorem 1.
Revised: [Figure] [Table] [Figure] that the proposed POF-MKL can better cope with heterogeneous data among clients which is inagreement with theoretical results in Theorem 1.

Original: In fact, the optimal combination of kernels can bedifferent across clients.
Revised: In fact, the optimal combination of kernels can bedifferent across clients.

Original: Using POF-MKL, each client constructs its own personalized combination ofkernels which results in lower MSE compared to vM-KOFL.
Revised: Using POF-MKL, each client constructs its own personalized combination ofkernels which results in lower MSE compared to vM-KOFL.

Original: The proposed POF-MKL with M = 1and M = 25 runs faster than eM-KOFL.
Revised: The proposed POF-MKL with M = 1and M = 25 runs faster than eM-KOFL.

Original: In fact, using POF-MKL, clients only need to updateparameters associated with M kernels while employing vM-KOFL and eM-KOFL, clients have toupdate parameters of all kernels.
Revised: In fact, using POF-MKL, clients only need to updateparameters associated with M kernels while employing vM-KOFL and eM-KOFL, clients have toupdate parameters of all kernels.

Original: Moreover, POF-MKL obtains lower MSE than PerFedAvg.
Revised: Moreover, POF-MKL obtains lower MSE than PerFedAvg.

Original: Notethat since clients are not able to store data in batch, at each time instant clients update PerFedAvg’smodel using only the newly observed data sample.
Revised: Notethat since clients are not able to store data in batch, at each time instant clients update PerFedAvg’smodel using only the newly observed data sample.

Original: Therefore, convergence of PerFedAvg is notguaranteed.
Revised: Therefore, convergence of PerFedAvg is notguaranteed.

Original: Experimental results show that POF-MKL achieves higher accuracy than PerFedAvg in [Table] [Figure] online regression task when it is not possible for clients to store data in batch.
Revised: Experimental results show that POF-MKL achieves higher accuracy than PerFedAvg inonline regression task when it is not possible for clients to store data in batch.

Original: Since OFSKL employsonly a pre-selected single kernel, OFSKL runs faster than POF-MKL.
Revised: Since OFSKL employsonly a pre-selected single kernel, OFSKL runs faster than POF-MKL.

Original: However, utilizing multiplekernels enables POF-MKL to obtain lower MSE than that of OFSKL.
Revised: However, utilizing multiplekernels enables POF-MKL to obtain lower MSE than that of OFSKL.

Original: In fact, using POF-MKL clientslearn a linear combination of kernels which is proved to enjoy sub-linear regret with respect to the bestkernel in hindsight while employing OFSKL clients have to make predictions using a pre-selectedkernel.
Revised: In fact, using POF-MKL clientslearn a linear combination of kernels which is proved to enjoy sub-linear regret with respect to the bestkernel in hindsight while employing OFSKL clients have to make predictions using a pre-selectedkernel.

Original: Furthermore, Figure 1 illustrates the average regret of clients when clients employ vM-KOFLand proposed POF-MKL with different M parameters.
Revised: Furthermore, Figure 1 illustrates the average regret of clients when clients employ vM-KOFLand proposed POF-MKL with different M parameters.

Original: From Figure 1, it can be observed that theproposed POF-MKL achieves sub-linear regret.
Revised: From Figure 1, it can be observed that theproposed POF-MKL achieves sub-linear regret.

Original:  Conclusions
Revised:  Conclusions

Original: The present paper proposed a personalized online federated MKL algorithm called POF-MKL basedon RF approximation.
Revised: The present paper proposed a personalized online federated MKL algorithm called POF-MKL basedon RF approximation.

Original: Employing the proposed POF-MKL, each client updates the parameters of asubset of kernels which alleviates the computational complexity of the client as well as communicationcost of sending updated parameters of kernels.
Revised: Employing the proposed POF-MKL, each client updates the parameters of asubset of kernels which alleviates the computational complexity of the client as well as communicationcost of sending updated parameters of kernels.

Original: Theoretical analysis proved that using POF-MKL,each client achieves sub-linear regret with respect to the RF approximation of its best kernel inhindsight which indicates that POF-MKL can deal heterogeneous data among clients.
Revised: Theoretical analysis proved that using POF-MKL,each client achieves sub-linear regret with respect to the RF approximation of its best kernel inhindsight which indicates that POF-MKL can deal heterogeneous data among clients.

Original: While eachclient updates a subset of kernels, it was proved that the server achieves sub-linear regret withrespect to the best function approximator.
Revised: While eachclient updates a subset of kernels, it was proved that the server achieves sub-linear regret withrespect to the best function approximator.

Original: Experiments on real datasets showcased the advantages of
Revised: Experiments on real datasets showcased the advantages of

Original: POF-MKL compared with other online federated kernel learning algorithms.Checklist
Revised: POF-MKL compared with other online federated kernel learning algorithms.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] In section 4, we mention that

Original: the proposed multi-kernl leanring algorithm POF-MKL runs slower than single kernel

Original: learning algorithm OFSKL.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We do

Original: not foresee any direct societal impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Please see

Original: section 3.2 where we explicitly explain the assumptions.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Complete proofs of

Original: theoretical results are presented in Appendices A and B.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide code,

Original: data, and instructions to reproduce the main experimental results in the supplementary

Original: material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were

Original: chosen)?

Original: [Yes] In section 4 and Appendix C, detailed information about experimental

Original: setup is provided.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] We did the experiments for 20 sets of random features.

Original: In Appendix C, the standard deviation of MSE is reported.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type

Original: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] In Table 1, we report run time of

Original: algorithms.

Original: Also, in Appendix C, we explain that all experiments were carried out

Original: using Intel(R) Core(TM) i7-10510U CPU @ 1.80 GHz 2.30 GHz processor with a

Original: 64-bit Windows operating system.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] In section 4, we citedall datasets used for experiments.

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [Yes] In section 4, we point out that datasets are downloaded from UCI

Original: Machine Learning Repository.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]