D ECENTRALIZED L EARNING FOR O VERPARAMETERIZED P ROBLEMS : A M ULTI -A GENT K ERNEL A PPROXIMATION A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
This work develops a novel framework for communication-efﬁcient distributed learning where the models to be learnt are overparameterized.
We focus on a class of kernel learning problems (which includes the popular neural tangent kernel (NTK) learning as a special case) and propose a novel multi-agent kernel approximation technique that allows the agents to distributedly estimate the full kernel function, and subsequently perform distributed learning, without directly exchanging any local data or parameters.
The proposed framework is a signiﬁcant departure from the classical consensus-based approaches, because the agents do not exchange problem parameters, and consensus is not required.
We analyze the optimization and the generalization performance of the proposed framework for the (cid:96) 2 loss.
We show that with M agents and N total samples, when certain generalized inner-product (GIP) kernels (resp.
the random features (RF) kernel) are used, each agent needs to communicate O (cid:0) N 2 /M (cid:1) bits (resp.
O (cid:0) N √ N/M (cid:1) real values) to achieve minimax optimal generalization performance.
Further, we show that the proposed algorithms can signiﬁcantly reduce the communication complexity compared with state-of-the-art algorithms, for distributedly training models to ﬁt UCI benchmarking datasets. Moreover, each agent needs to share about 200 N/M bits to closely match the performance of the centralized algorithms, and these numbers are independent of parameter and feature dimension.
I NTRODUCTION
Recently, decentralized optimization has become a mainstay of the optimization research.
In decentralized optimization, multiple local agents hold small to moderately sized private datasets, and collaborate by iteratively solving their local problems while sharing some information with other agents.
Most of the existing decentralized learning algorithms are deeply rooted in classical consensus-based approaches (Tsitsiklis, 1984), where the agents repetitively share the local parameters with each other to reach an optimal consensual solution.
However, the recent trend of using learning models in the overparameterized regime with very high-dimensional parameters (He et al., 2016; Vaswani et al., 2017; Fedus et al., 2021) poses a signiﬁcant challenge to such parameter sharing approaches, mainly because sharing model parameters iteratively becomes excessively expensive as the parameter dimension grows.
If the size of local data is much smaller than that of the parameters, perhaps a more efﬁcient way is to directly share the local data.
However, this approach raises privacy concerns, and it is rarely used in practice.
Therefore, a fundamental question of decentralized learning in the overparameterized regime is:
(Q) For overparameterized learning problems, how to design decentralized algorithms that achieve the best optimization/generalization performance by exchanging minimum amount of information
We partially answer (Q) in the context of distributed kernel learning (Vert et al., 2004).
We depart from the popular consensus-based algorithms and propose an optimization framework that does not require the local agents to share model parameters or raw data.
We focus on kernel learning because: (i) kernel methods provide an elegant way to model non-linear learning problems with complex data dependencies as simple linear problems (Vert et al., 2004; Hofmann et al., 2008), and (ii) kernelbased methods can be used to capture the behavior of a fully-trained deep network with large width (Jacot et al., 2018; Arora et al., 2019; 2020).
[Table]
Distributed implementation of kernel learning problems is challenging.
Current state-of-the-art algorithms for kernel learning either rely on sharing raw data among agents and/or imposing restrictions on the number of agents (Zhang et al., 2015;
Lin et al., 2017; Koppel et al., 2018; Lin et al., 2020; Hu et al., 2020; Pradhan et al., 2021; Predd et al., 2006).
Some recent approaches rely on speciﬁc random feature (RF) kernels to alleviate some of the above problems.
These algorithms reformulate the (approximate) problem in the parameter domain and solve it by iteratively sharing the (potentially high-dimensional) parameters (Bouboulis et al., 2017; Richards et al., 2020; Xu et al., 2020; Liu et al., 2021) 1 .
These algorithms suffer from excessive communication overhead, especially in the overparameterized regime where the number of parameters is larger than the data size N .
For example, implementing the neural tangent kernel (NTK) with RF kernel requires at least O ( N β ) , β ≥ 2 , random features (parameter dimension) using ReLU activation (Arora et al., 2019;
Han et al., 2021) 2 .
For such problems, in this work, we propose a novel algorithmic framework for decentralized kernel learning.
Below, we list the major contributions of our work.
[GIP Kernel for Distributed Approximation]
We deﬁne a new class of kernels suitable for distributed implementation, Generalized inner-product (GIP) kernel, that is fully characterized by the angle between a pair of feature vectors and their respective norms.
Many kernels of practical importance including the NTK can be represented as GIP kernel.
Further, we propose a multi-agent kernel approximation method for estimating the GIP and the popular RF kernels at individual agents.
[One-shot and Iterative Scheme]
Based on the proposed kernel approximation, we develop two optimization algorithms, where the ﬁrst one only needs one-shot information exchange, but requires sharing data labels among the agents; the second one needs iterative information exchange, but does not need to share the data labels.
A key feature of these algorithms is that neither the raw data features nor the (high-dimensional) parameters are exchanged among agents.
[Performance of the Approximation Framework] We analyze the optimization and the generalization performance of the proposed approximation algorithms for (cid:96) 2 loss.
We show that GIP kernel requires communicating O ( N 2 /M )
bits and the RF kernel requires communicating O ( N √ N/M ) real values per agent to achieve minimax optimal generalization performance.
Importantly, the required communication is independent of the function class and the optimization algorithm. We validate the performance of our approximation algorithms on UCI benchmarking datasets. In Table 1, we compare the communication requirements of the proposed approach to popular distributed kernel learning algorithms.
Speciﬁcally, DKRR-CM (Lin et al., 2020) relies on sharing data and is therefore not preferred in practical settings. For the RF kernel, the proposed algorithm outperforms other algorithms in both non-overparameterized and the overparameterized regimes when T > N/M . In the overparameterized regime the GIP kernel is more communication efﬁcient compared to other algorithms. Finally, note that since our analysis is developed using the multiagent-kernel-approximation, it does not impose any upper bounds on the number of machines in the network.
Notations: We use R , R d , and R n × m to denote the sets of real numbers, d -dimensional Euclidean space, and real matrices of size n × m , respectively.
We use N to denote the set of natural numbers.
N (0 , Σ) is multivariate normal distribution with zero mean and covariance Σ .
Uniform distribution with support [ a, b ] is denoted by U [ a, b ] .
(cid:104) a, b (cid:105) (resp.
(cid:104) a, b (cid:105) H ) denotes the inner-product in Euclidean space (resp.
Hilbert space H ).
The inner-product deﬁnes the usual norms in corresponding spaces.
Norm (cid:107) A (cid:107) of matrix
A denotes the operator norm induced by (cid:96) 2 vector norm.
We denote by [ a ] i or [ a ] ( i ) the i th element of a vector a .
[ A · a ] ( i )
j denotes the ( i · j ) th element of vector A · a .
Moreover,
A (: ,i ) is the i th column of A and [ A ] mk is the element corresponding to m th row and k th column.
Notation m ∈ [ M ] denotes m ∈ { 1 , .., M } .
Finally, 1 [ E ] is the indicator function of event E .
P ROBLEM S TATEMENT
Given a probability distribution π ( x, y ) over X × R , we want to minimize the population loss
[Equation]
where x ∈ X ⊂ R d and y ∈ R denote the features and the labels, respectively.
Here, f : X → R is an estimate of the true label y .
We consider a distributed system of M agents, with each agent m ∈
[ M ] having access to a locally available independently and identically distributed (i.i.d) dataset
N m =
x ( i ) m , y ( i ) m } n i =1 with 3 ( x ( i ) m , y ( i ) m ) ∼ π ( x, y ) .
The total number of samples is N = nM .
The goal of kernel learning with kernel function, k ( · , · ) :
X × X → R , is to ﬁnd a function f ∈ H (where H is the reproducing kernel Hilbert space (RKHS) associated with k (Vert et al., 2004)) that minimizes (1).
We aim to solve the following (decentralized) empirical risk minimization problem
[Equation]
where λ > 0 is the regularization parameter and ˆ L m ( f ) = 1 n (cid:80)
i ∈N m (cid:96) ( f ( x ( i ) m ) , y ( i ) m ) is the local loss at each m ∈
[ M ] .
Problem (2) can be reformulated using the Representer theorem (Sch ¨ olkopf et al., 2002) with ˆ L m ( α ) =
1 n (cid:80) i ∈N m (cid:96) (cid:0)(cid:2) K α (cid:3) ( i ) m , y ( i ) m (cid:1) , ∀ m ∈
[ M ] , as
[Equation]
where K ∈ R N × N is the kernel matrix with elements k
( x ( i ) m , x ( j )¯ m ) , ∀ m, ¯ m ∈
[ M ] , ∀ i ∈ N m and ∀ j ∈ N ¯ m .
The supervised (centralized) learning problem (3) is a classical problem in statistical learning (Caponnetto & De Vito, 2007) and has been popularized recently due to connections with overparameterized neural network training (Jacot et al., 2018; Arora et al., 2019).
An alternate way to solve problem (2) (and (3)) is by parameterizing f in (2) by θ ∈ R D as f D ( x ; θ ) = (cid:104) θ, φ D ( x ) (cid:105) where φ D : X → R D is a ﬁnite dimensional feature map. Here, φ D ( · ) is designed to approximate k ( · , · ) with k D ( x, x (cid:48) ) = (cid:104) φ D ( x ) , φ D ( x (cid:48) ) (cid:105) (Rahimi & Recht, 2008).
Using this approximation, problem (2) (and (3)) can be written in the parameter domain with ˆ L m,D ( θ ) = 1 n (cid:80) i ∈N m (cid:96) (cid:0) (cid:104) θ, φ D ( x ( i ) m ) (cid:105) , y ( i ) m (cid:1) , ∀ m ∈ [ M ] , as
[Equation] Note that (4) is a D -dimensional problem, whereas (3) is an N -dimensional problem. Since (4) is in the standard ﬁnite-sum form, it can be solved using the standard parameter sharing decentralized optimization algorithms (e.g., DGD (Richards et al., 2020) or ADMM (Xu et al., 2020) ), which share D -dimensional vectors iteratively.
However, when (4) is overparameterized with very large
D (e.g., D = O ( N β ) with β ≥ 2 for the NTK), such parameter sharing approaches are no longer feasible because of the increased communication complexity.
An intuitive solution to avoid sharing these high-dimensional parameters is to directly solve (3).
However, it is by no means clear if and how one can efﬁciently solve (3) in a decentralized manner .
The key challenge is that, unlike the conventional decentralized learning problems, here each loss term (cid:96) ([ K α ] ( i ) m , y ( i ) m ) is not separable over the agents.
Instead, each agent m ’s local problem is dependent on k ( x ( i ) m , x ( j )¯ m ) with m (cid:54) =
¯ m .
Importantly, without directly transmitting the data itself (as has been done in Predd et al.
Koppel et al.
; Lin et al.
, it is not clear how one can obtain the required ( m · i ) th element of K α .
Therefore, to develop algorithms that avoid sharing high-dimensional parameters by directly (approximately) solving (3), it is important to identify kernels that are suitable for decentralized implementation and propose efﬁcient algorithms for learning with such kernels.
T HE P ROPOSED A LGORITHMS
In this section, we deﬁne a general class of kernels referred to as the generalized inner product (GIP) kernels that are suitable for decentralized overparameterized learning.
By focusing on GIP kernels, we aim to understand the best possible decentralized optimization/generalization performance that can be achieved for solving (3).
Surprisingly, one of our proposed algorithm only shares O ( nN ) =
O ( N 2 /M )
bits of information per node, while achieving the minimax optimal generalization performance.
Such an algorithm only requires one round of communication, where the messages transmitted are independent of the actual parameter dimension (i.e., D in problem (4)); further, there is no requirement for achieving consensus among the agents.
The proposed algorithm represents a signiﬁcant departure from the classical consensus-based decentralized learning algorithms.
We ﬁrst deﬁne a class of kernels that we will focus on in this work.
Deﬁnition 3.1.
[Generalized inner-product (GIP) kernel]
We deﬁne a GIP kernel as:
[Equation]
where ψ ( x, x (cid:48) ) =
arccos( x T x (cid:48) /
( (cid:107) x
cid:107)(cid:107) x (cid:48) (cid:107) ))
∈ [0 , π ] denotes the angle between the feature vectors x and x (cid:48) ; and g ( · , (cid:107) x (cid:107) , (cid:107) x (cid:48) (cid:107) ) is assumed to be Lipschitz continuous (cf.
Assumption 2).
Remark 1 .
Note that the GIP kernel is a generalization of the inner-product kernels (Sch ¨ olkopf et al., 2002), i.e., kernels of the form k
( x, x (cid:48) )
=
k ( (cid:104) x, x (cid:48) (cid:105) ) .
Clearly, k ( (cid:104) x, x (cid:48) (cid:105) ) can be represented as k ( (cid:104) x, x (cid:48) (cid:105) )
=
g ( ψ ( x, x (cid:48) ) , (cid:107) x (cid:107) , (cid:107) x (cid:48) (cid:107) ) for some function g ( · ) .
Moreover, many kernels of practical interest can be represented as GIP kernels, some examples include NTK (Jacot et al., 2018; Chizat et al., 2019; Arora et al., 2019), arccosine (Cho & Saul, 2009), polynimal, Gaussian, Laplacian, sigmoid, and inner-product kernels (Sch ¨ olkopf et al., 2002).
The main reason we focus on the GIP kernels for decentralized implementation is that, this class of kernels can be fully speciﬁed at each agent if the norms of all the feature vectors and the pairwise angles between them are known at each agent.
For example, consider an NTK of a single hiddenlayer ReLU neural network: k
( x, x (cid:48) )
=
x T x (cid:48) ( π − ψ ( x, x (cid:48) )) / 2 π (Chizat et al., 2019). This kernel can be fully learned with just the knowledge of norms and the pairwise angles of the feature vectors.
For many applications of interest (Bietti & Mairal, 2019; Geifman et al., 2020; Pedregosa et al., 2011), normalized feature vectors are used, and for such problems, the GIP kernel at each agent can be computed only by using the knowledge of the pairwise angles between the feature vectors.
We show in Sec. 3.1 that such kernels can be efﬁciently estimated by each agent while sharing only a few bits of information.
Importantly, the communication requirement for such a kernel estimation procedure is independent of the problem’s parameter dimension (i.e., D in (4)), making them suitable for decentralized learning in overparameterized regime. Next, we deﬁne the RF kernel.
Deﬁnition 3.2.
[Random features (RF) kernel] RF kernel is deﬁned as (Rahimi & Recht, 2008; Rudi & Rosasco, 2017; Li et al., 2019):
[Equation]
(6) with (Ω , q ) being the probability space and ¯ ζ : X × Ω → R .
Remark 2 .
The RF kernel can be approximated as: k ( · , · ) ≈ k P ( x, x [Equation]
) = (cid:104) φ P ( x ) , φ P ( x (cid:48) ) (cid:105) , with φ P ( x ) = 1 √ P [ ¯ ζ ( x, ω 1 ) , . . . , ¯ ζ ( x, ω P )] T ∈ R P and { ω i } Pi =1 drawn i.i.d.
from distribution q ( ω ) .
A popular example of the RF kernels is the shift-invariant kernels, i.e., kernels of the form k
( x, x (cid:48) )
=
k ( x − x (cid:48) )
(Rahimi & Recht, 2008).
The RF kernels generalize the random Fourier features construction (Rudin, 2017) for shift-invariant kernels to general kernels.
Besides the shiftinvariant kernels, important examples of the RF kernels include the inner-product, and (Kar & Karnick, 2012), the homogeneous additive kernels (Vedaldi & Zisserman, 2012).
Next, we propose a multi-agent approximation algorithm to effectively learn the GIP and the RF kernels at each agent, as well as the optimization algorithms to efﬁciently solve the learning problem.
Our proposed algorithms will follow an approximation – optimization strategy, where the agents ﬁrst exchange some information so that they can locally approximate the full kernel matrix K ; then they can independently optimize the resulting approximated local problems.
Below we list a number of key design issues arising from implementing such an approximation – optimization strategy: [Kernel approximation] How to accurately approximate the kernel K , locally at each agent?
For example, for the GIP kernels, how to accurately estimate the angles ψ ( x ( i ) m , x ( j )¯ m ) at a given agent m , where j ∈ N ¯ m and ¯ m (cid:54)
= m ?
This is challenging, especially when raw data sharing is not allowed.
[Effective exchange of local information] How shall we design appropriate messages to be exchanged among the agents?
The type of messages that gets exchanged will be dependent on the underlying kernel approximation schemes.
Therefore, it is critical that proposed approximation methods are able to utilize as little information from other agents as possible.
[Iterative or one-shot scheme]
It is not clear if such an approximation – optimization scheme should be one-shot or iterative – that is, whether it is favourable that the agents iteratively share information and perform local optimization (similar to classical consensus-based algorithms), or they should do it just once.
Again, this will be dependent on the underlying information sharing schemes.
Next, we will formally introduce the proposed algorithms.
Our presentation follows the approximation – optimization strategy outlined above.
We ﬁrst discuss the proposed decentralized kernel approximation algorithm, followed by two different ways of performing decentralized optimization.
3. M ULTI - AGENT KERNEL APPROXIMATION
The kernel K is approximated locally at each agent using Algorithm 1.
Note that in Step 3, each agent randomly samples { ω i } Pi =1 from distribution p ( ω ) .
This can be easily established via random seed sharing as in Xu et al.
; Richards et al.
In Step 6, each agent shares a locally constructed matrix A m of size P × n , whose elements ζ ( x ( i ) m , ω i ) will be deﬁned shortly.
The choices of p ( ω ) and ζ ( · , · ) in Step 1 depend on the choice of kernel.
Speciﬁcally, we have: [Approximation for GIP kernel]
For the GIP kernel, we ﬁrst assume that the feature vectors are normalized (Pedregosa et al., 2011).
We then choose p ( ω ) to be any circularly symmetric distribution, for simplicity we choose p ( ω ) as N (0 , I d ) .
Moreover, we use ζ ( x, ω ) = 1 [ ω T x ≥ 0] such that
A m is a binary matrix with entries { 0 , 1 } .
Note that such matrices are easy to communicate.
Next, we approximate the kernel K with K P as
[Equation]
where k ( x ( i ) m , x ( j )¯ m ) and k P ( x ( i ) m , x ( j )¯ m ) ∀ i ∈ N m , ∀ m ∈
[ M ] and ∀ j ∈ N ¯ m and ∀ ¯ m ∈
[ M ] are the individual elements of K and K P , resp., and ψ P ( x ( i ) m , x ( j )¯ m ) is an approximation of the angle ψ ( x ( i ) m , x ( j )¯ m ) evaluated using A m , A ¯ m as
[Equation]
1: Initialize: α 1 m ∈
N ,
{ η tm } T t each agent m
[ M ] 2: for m
M do 3:
Using Algorithm 1 construct K P 4:
Communicate ¯ y m =
[ y (1) m , . . .
, y ( n ) m ] T ∈ R n 5:
Using K P and ¯ y m construct ˆ L P ( α ) (cf.
(10)) locally using ˆ L m,P ( α ) 6:
Option I: Solve exactly at each agent 7:
Option II Solve agent 8: for t 1 to T m 9:
GD Update : α t +1 m = α tm − η tm ∇ ˆ R P ( α tm ) 10: end for 11: end for 12: Return: α T +1 m for all m M ]
This implies that K can be approximated for the GIP kernel by communicating only nP bits of information per agent.
Note that in the general case if the feature vectors are not normalized, then (7) can be evaluated by communicating additional n real values of the norms of the feature vectors by each agent; see Step 7 in Algorithm 1.
[Approximation for RF kernel] For the RF kernel, we choose ζ ( · , · ) = ¯ ζ ( · , · ) and p ( ω ) = q ( ω ) as deﬁned in (6) and approximate K with K P as [Equation]
[Equation]
where k ( x ( i ) m , x ( j )¯ m ) and k P ( x ( i ) m , x ( j )¯ m ) are elements of K and K P , resp., φ P ( x ( i ) m )
= 1 / √ P
[ A (: ,i ) m ] and φ P ( x ( j )¯ m ) = 1 / √ P [ A (: ,j ) ¯ m ] .
Note that K can be approximated for the RF kernel by sharing only nP real values per agent.
Further, the distribution q ( ω ) and the mapping ¯ ζ ( · , · ) depend on the type of RF kernel used.
For example, for shift-invariant kernels with random Fourier features, we can choose ¯ ζ ( x, ω ) = √ 2 cos( ω T x + b ) with ω ∼ q ( ω ) and b ∼ U [0 , 2 π ] (Rahimi & Recht, 2008).
Now that using Algorithm 1 we have approximated the kernel matrix at all the agents, we are ready to solve (3) approximately.
3. T HE DECENTRALIZED OPTIMIZATION STEP
The approximated kernel regression problem (3) with K P obtained using Algorithm 1, and local loss ˆ L m,P ( α ) :
= 1 n (cid:80) i [Equation] K P α (cid:3) (
[Equation]
) m , y ( i ) m (cid:1) is
[Equation]
Remark 3 .
For the approximate problem (10), we would want K P constructed using the multi-agent kernel approximation approach to be positive semi-deﬁnite (PSD), i.e., the kernel function k P ( · , · ) is a positive deﬁnite (PD) kernel.
From the deﬁnition of the approximate RF kernel (9), it is easy to verify that it is PD.
However, it is not clear if the approximated GIP kernel is PD.
Certainly, for the GIP kernel we expect that as P → ∞ we have K P → K , i.e., asymptotically K P is PSD, since K is PSD.
In the Appendix, we introduce a sufﬁcient condition (Assumption 6) that ensures K P to be
PSD for the GIP kernel.
In the following, for simplicity we assume K P is PSD.
Decentralized optimization based on one-shot communication:
In this setting, we share the information among all the agents in one-shot , then each agent learns its corresponding minimizer using the gathered information.
We assume that each agent can communicate with every other agent either in a decentralized manner (or via a central server) before initialization.
This is a common assumption in distributed learning with RF kernels where the agents need to share random seeds before initialization to determine the approximate feature mapping (Richards et al., 2020; Xu et al., 2020).
Here, consensus is not enforced as each agent can learn a local minimizer which has a good global property .
The label information is also exchanged among all the agents.
In Algorithm 2, we list the steps of the algorithm.
In Step 3, the agents learn K P (the local estimate of the kernel matrix) using Algorithm 1.
In Step 4, the agents share the labels ¯ y m so that each agent can (approximately) reconstruct the loss ˆ L ( α ) (cf.
(10)) locally.
Then each agent can either choose Option I or Option
II to solve (10).
A few important properties of Algorithm 2 are: [Communication] Each agent communicates a total of O ( nP ) =
O ( NP/M ) bits (if the norms also need to be transmitted, then with an additional N/M real values) for the GIP kernel, and O ( NP/M ) real values for the RF kernels.
Importantly, for the GIP kernel the communication is independent of the parameter dimension, making it suitable for decentralized overparameterized learning problems; see Table 1 for a comparison with other approaches.
[No consensus needed] Each agent executes Algorithm 2 independently to learn α m , without needing to reach any kind of consensus.
They are free to choose different initializations, step-sizes, and even regularizers (i.e., λ in (10)).
In contrast to the classical learning, where algorithms are designed to guarantee consensus (Koppel et al., 2018; Richards et al., 2020; Xu et al., 2020), our algorithms allow each agent to learn a different function.
The proposed framework relies on sharing matrices
A m ’s that are random functions of the local features.
Note that problem (10) can also be solved by using an iterative distributed gradient tracking algorithm
(Qu & Li, 2018), with the beneﬁt that no label sharing is needed; see Appendix D.
Remark 4 (Optimization performance) .
Note that using Algorithm 2, we can solve the approximate problem (10) to arbitrary accuracy using either Option I or Option II .
However, it is by no means clear if the solution obtained by Algorithm 2 will be close to the solution of (3).
Therefore, after problem (10) is solved, it is important to understand how close the solutions returned by Algorithm 2 are to the original kernel regression problem (3).
M AIN R ESULTS
In this section, we analyze the performance of Algorithm 2.
Speciﬁcally, we are interested in understanding the training loss and the generalization error incurred due to the kernel approximation (cf.
Algorithm 1).
For this purpose, we focus on (cid:96) 2 loss functions for which the kernel regression problem (10) can be solved in closed-form.
Speciﬁcally, we want to minimize the loss:
[Equation]
Then we solve the following kernel ridge regression problem with the choice ˆ L ( α )
=
12 N (cid:107) [Equation] [Equation] α
[Equation]
R N [Equation]
[Equation]
where we denote ¯ y
=
[¯ y T 1 , . .
, ¯ y TM ] T ∈
R N with ¯ y m =
[ y (1) m , y (2) m , . .
y ( n ) m ] T ∈ R n .
The above problem can be solved in closed form with ˆ α ∗ =
[ K + N · λ · I ] − 1 ¯ y .
The approximated problem at each agent with the kernel K P and with the loss function ˆ L P ( α ) = 1 2 N (cid:107) ¯ y − K P α (cid:107) 2 is
[Equation]
13) with the optimal solution returned by Option I in Algorithm 2 as ˆ α ∗ P =
[ K P + N · λ · I ] − 1 ¯ y .
The goal is to analyze the impact of the approximation on the performance of Algorithm 2.
Speciﬁcally, we bound the difference between the optimal losses of the exact and the approximated Kernel ridge regression.
We begin with some assumptions.
Assumption 1.
We assume | k
( x, x (cid:48) )
| ≤ κ 2 and | k P ( x, x (cid:48) )
| ≤ κ 2 for some κ ≥ 1 .
Assumption 2.
The function g ( · ) in (5) used to construct the GIP kernel is G-Lipschitz w.r.t.
ψ , i.e., ∃ G ≥ 0 such that: | g ( ψ, z 2 , z 3 ) − g ( ˆ ψ, z 2 , z 3 ) | ≤ G | ψ − ˆ ψ | , ∀ ψ, ˆ ψ ∈ [0 , π ] and ∀ z 2 , z 3 ∈ R .
Assumption 3.
We assume that the data labels | y | ≤ R almost surely for some R > 0 .
Assumption 4.
There exists f H ∈ H such that L ( f H )
=
inf h ∈H L ( h ) .
A few remarks are in order.
Note that Assumptions 1, 3 and 4 are standard in the statistical learning theory (Cucker & Zhou, 2007; Caponnetto & De Vito, 2007; Ben-Hur & Weston, 2010; Rudi &
Rosasco, 2017).
Moreover, for RF kernel Assumption 1 is automatically satisﬁed if | ζ ( x, ω ) | ≤ κ almost surely (Rudi & Rosasco, 2017) (cf.
(6) and (9)).
Assumption 2 is required for estimating the kernel by approximating the pairwise angles between feature vectors.
It is easy to verify that the popular kernels including, NTK (15), Arccosine, Gaussian and Polynomial kernels satisfy Assumption 2 with feature vectors belonging to a compact domain (this ensures that the Lipschitz constant
G is independent of the feature vector norms).
Now we are ready to present the results.
We analyze how well Algorithm 1 approximates the exact kernel.
We are interested in the approximation error as a function of the number of random samples P .
We have the following lemma.
Lemma 4.1 (Kernel Approximation) .
For K P returned by Algorithm 1, the following holds with probability at least 1 − δ : (i) For the GIP kernel, (cid:107) K − K P (cid:107) ≤ GN (cid:16)(cid:113) 32 π 2 P log 2 Nδ + 8 π 3 P log 2 Nδ (cid:17) .
(ii) Similarly, for the RF kernel, (cid:107) K − K P (cid:107) ≤ κ 2 N (cid:16)(cid:113) 8 P log {+
+}
Nδ + 43 P log {+
+} Nδ (cid:17) .
Note that as P increases K P → K , in particular, to achieve an approximation error of (cid:15) > 0 , we need P
=
O
( (cid:15) − 2 ) .
Importantly, Lemma 4.1 plays a crucial role in analyzing the optimization performance of the kernel approximation approach.
Next, we state the training loss incurred as a consequence of solving the approximate decentralized problem (13) in Algorithm 2 instead of (12).
Theorem 4.2 (Approximation: Optimal Loss) .
Suppose P ≥ 2 9 log [-
 Nδ-]2 N δ , then for both the GIP and the RF kernels, the solution returned by Algorithm 2 ( Option I ) for solving (12) approximately (i.e, (13)), satisﬁes the following with probability at least 1 − δ (cid:12)(cid:12) ˆ L P
[Equation]
Theorem 4.2 states that as P increases, the optimal training loss achieved by solving approximate problem (13) via Algorithm 2 ( Option I ) will approach the performance of the centralized system (12) for both the GIP and the RF kernels. The proof of the above result utilizes Lemma 4.1 and the deﬁnition of the loss functions in (12) and (13).
See Appendix G for a detailed proof.
The results of Lemma 4.1 and Theorem 4.2 characterize the approximation performance of the proposed approximation – optimization framework on ﬁxed number of training samples.
Of course, it is of interest to analyze how the proposed approximation algorithms will perform on unseen test data.
Towards this end, it is essential to analyze the performance of the function ˆ f P learned from solving (13) via Algorithm 2.
We have the following result.
Theorem 4.3 (Generalization performance) .
Let us choose λ = 1 / √ N , δ ∈ (0 , 1) , and N ≥ max (cid:110)
[Equation]
[Equation]
(cid:111) , also choose P ≥ max (cid:110)
[Equation]
[Equation]
K (cid:107) 2 , 288 π 2 G 2 N (cid:111) log 16 δ for the GIP kernel and P ≥ max (cid:110) 8 κ 2 , 32 κ 2 (cid:107) K (cid:107) 2 , 72 κ 2 √ N (cid:111) log 128 κ 2 √ N δ for the RF kernel, where K is deﬁned in Appendix F.
Then with probability at least 1 − δ , we have for ˆ f P returned by Algorithm( Option I ) for approximately solving (12)
[Equation]
i.e.
[Equation]
[Equation]
The proof of Theorem 4.3 utilizes a result similar to Lemma 4.1 but for integral operator deﬁned using kernels k ( · , · ) and k P ( · , · ) .
Theorem 4.3 states that with appropriate choice of λ (the regularization parameter), N (the number of overall samples), and P (the messages communicated per agent), the proposed algorithm achieves the minimax optimal generalization performance (Caponnetto & De Vito, 2007). Also, note that the the requirement of P = O ( √ N ) for the RF kernel compared to P
= O ( N ) for the GIP kernel is due to the particular structure of the RF kernel (cf.
It can be seen from Lemmas H.4 and H.5 in Appendix H, that the approximation obtained with the RF kernel allows the derivation of tighter bounds compared to the GIP kernel.
The next corollary precisely states the total communication required per agent to achieve this optimal performance.
Corollary 1 (Communication requirements for the GIP and RF kernels) .
Suppose Algorithm 2 uses the choice of parameters stated in Theorem 4.3 to approximately optimize (12). Then it requires a total of O ( N 2 /M ) bits (resp.
O ( N √ N/M ) real values) of message exchanges per node when the GIP kernel (resp. the RF kernel) is used, to achieve minimax optimal generalization performance.
Moreover, if unnormalized feature vectors are used, then the GIP kernel requires an additional O ( N/M ) real values of message exchanges per node.
Compared to DKRR-RF-CM (Liu et al., 2021), Decentralized RF (Richards et al., 2020), DKLA, and COKE (Xu et al., 2020), the number of message exchanges required by the proposed algorithm [Table] [Table] [Table]
is independent of the iteration numbers
[Table]
and it is much less compared to other algorithms, especially for the GIP kernel in the overparameterized regime; see Table 1 for detailed comparisons
[Table]
We compare the performance of the proposed algorithm to DKRR-RF-CM (Liu et al., 2021), Decentralized RF (Richards et al., 2020)
[Table]
and DKLA (Xu et al., 2020). We evaluate the performance of all the algorithms on real world datasets from the UCI repository. Speciﬁcally, we present the results on National Advisory Committee for Aeronautics (NACA) airfoil noise dataset (Lau & L ´ opez, 2009), where the goal is to predict aircraft noise based on a few measured attributes. The dataset consists of N = 1503 samples that are split equally among M =nodes.
Each node utilizes 70% of its data for training and 30% for testing purposes.
Each feature vector x
( i ) m ∈ R 5 represents the measured attributes such as, frequency, angle, etc., and each label y ( i ) m represents the noise level.
Additional experiments on different datasets and classiﬁcation problems, as well as the detailed parameter settings, are included in the Appendix A. We evaluate the performance of all the algorithms with the Gaussian kernel. Note that the algorithms DKRR-RF-CM, Decentralized RF, and DKLA can only be implemented using the RF approach while our proposed algorithm utilizes the GIP kernel. Also, in contrast to these benchmark algorithms that use iterative parameter exchange, the proposed Algorithm 2 uses only one-shot communication.
First, in Table 2, we compare the communication required by each algorithm with the
Gaussian kernel for P = 100 , 500 , and 1000 to achieve the same test mean squared error (MSE) for each setting, see last row of Table 2. Note that for P = 100 , the communication required by Algorithm 2 is less than 50% of that required by DKLA and Decentralized RF while it is only slightly less than that of DKRR-RF-CM.
Moreover, as P increases to 500 and 1000 , it can be seen that
Algorithm 2 only requires a fraction of communication compared to other algorithms, and this fact demonstrates the utility of the proposed algorithms for over-parameterized learning problems.
In
Table 3, we compare the averaged MSE achievable by different algorithms, when a ﬁxed total communication budget (in bits) is given for each setting (see the last row of Table 3 for the budget).
Note that Algorithm 2 signiﬁcantly outperforms all the other methods as P increases.
This is expected since Algorithm 2 essentially solves a centralized problem (cf.
Problem (10)) after the multi-agent kernel approximation (cf. Algorithm 1), and a large P provides a better approximation of the kernel (cf.
Lemma 4.1).
In contrast, for[-β > 1 per iteration (resp. 
 · d f , where d f is-] the parameter sharing based algorithms the performance deteriorates even though the kernel approximation improves with large P as learning a high-dimensional parameter naturally requires more communication rounds as well as a higher communication budget per communication round.
Please note that we also compare the performance of Algorithm 2 with the benchmarking algorithms discussed above for the NTK.
We further benchmark the performance of Algorithm 2 against the centralized algorithms for the Gaussian, the Polynomial, and the NTK.
However, due to space limitations, we relegate these numerical results to the Appendix A.
A PPENDIX
In this section, we discuss in detail some important aspects of the proposed kernel approximation framework along with additional experiments and the proofs of Theorems 4.2 and 4.3.
A A DDITIONAL E XPERIMENTS
The goal of this section is to empirically analyze the performance of the proposed approximation – optimization framework on real datasets.
We evaluate the performance of the proposed approach on regression and classiﬁcation tasks on datasets from the UCI repository. First for the regression task, we compare the mean squared error (MSE) and the communication performance of the proposed algorithm to popular decentralized regression algorithms DKRR-RF-CM (Liu et al., 2021), Decentralized RF (Richards et al., 2020), and DKLA (Xu et al., 2020). For the classiﬁcation task, we benchmark the classiﬁcation performance of the proposed algorithm against centralized algorithms.
Speciﬁcally, via this set of experiments we determine the communication required by each agent to achieve performance similar to the centralized algorithms.
To present the results, we use three types of popular GIP kernels, namely, the Polynomial, the Gaussian, and the NTK.
Below, we discuss the dataset and the implementation details for both the tasks. Regression Problem. We consider the regression task on the following UCI datasets: (i) Airfoil self-noise dataset.
We compare the performance of the proposed algorithm with the benchmarking algorithms on National Advisory Committee for Aeronautics (NACA) airfoil noise dataset (Lau & L ´ opez, 2009), where the goal is to predict aircraft noise based on a few measured attributes.
The dataset consists of N = 1503 samples with each feature vector x ( i ) m
∈ R 5 representing the measured attributes such as, frequency, angle, etc., and each label y ( i ) m ∈ R represents the noise level. (ii) Energy dataset. We also evaluate the performance of the algorithms on Appliances energy prediction dataset (Candanedo et al., 2017), where the goal is to predict the total energy consumption of a house based on attributes like, pressure, wind speed, and temperature and humidity in different areas of the house, etc.
This dataset contains T = 19735 samples with the feature vector x ( i ) m ∈
Rdescribing the measured attributes and y ( i ) m ∈ R representing the total energy consumption in the house.
Algorithm and Parameter Settings.
For each setting we assume that there are M = 10 nodes in the network connected via a star topology.
The overall data is split equally among the nodes.
Each node utilizes 70% of its data for training and 30% for testing purposes.
The learning performance of all the algorithms is evaluated by average MSE across all the nodes.
Moreover, for our algorithm the testing is performed using Algorithm 4 in Appendix E.
For all the algorithms, the regularization parameter λ is tuned from the set { 10 − 3 , 10 − 2 , 10 − 1 , 1 , 10 } while the step-sizes for Decentralized
RF (Richards et al., 2020) and DKLA (Xu et al., 2020) are tuned from { 10 − 3 , 10 − 2 , 10 − 1 } .
For the regression task, we compare the performance of the proposed algorithms with the Gaussian kernel and the NTK.
Note that DKRR-RF-CM (Liu et al., 2021), Decentralized RF (Richards et al., 2020) and DKLA (Xu et al., 2020), can only be implemented using parameter sharing based RF approach (please see the respective papers) while our algorithm utilizes the GIP kernel approximation (cf.
Algorithm 1) for the decentralized implementation.
For the Gaussian kernel, we choose the scale parameter to be 1 and the P -dimensional random feature mapping is constructed using Algorithmin Rahimi & Recht (2008).
For the NTK, we generate the P -dimensional feature mapping utilizing the gradient of a single-hidden layer ReLU network with weights of the ﬁnal layer chosen from
U [ {− 1 , 1 } ] (Cho & Saul, 2009; Chizat et al., 2019).
Discussion.
In Tables 2 and 3 of Section 5, we compared the performance of Algorithm 2 against the benchmarking algorithms on the Airfoil self-noise dataset with the Gaussian kernel.
Next, in Table 4 we evaluate the performance of Algorithm 2 against other algorithms on the same dataset with the NTK.
We choose P = 100 and compare the communication performance (measured in bits) for all the algorithms for a ﬁxed MSE in Table 4a.
Note that Algorithm 2 requires only a fraction of communication compared to other algorithms.
Next, in Table 4b we compare the MSE performance of the algorithms for a ﬁxed communication budget.
Again, we observe that the proposed approximation – optimization approach achieves better performance compared to the state-of-the-art algorithms.
[Table]
[Table]
In Tables 5 and
[Table]
we compare the communication and the MSE performance, respectively
[Table]
of the algorithms for the energy dataset.
We compare the performance with the Gaussian kernel for P = 100 , 500 , and 1000 .
In Table
[Table]
we compare the communication required by each algorithm to achieve the same MSE for each setting, see last row of Table 5.
We execute the parameter sharing algorithms DKRR-RF-CM (Liu et al., 2021), Decentralized RF (Richards et al., 2020), and (Xu et al., 2020) for a maximum of 100 iterations and report the total communication at the end ofrounds if the algorithm does not achieve the target MSE.
Note that for all the algorithms and under all the settings, Algorithm 2 requires fewer bits of communication compared to other algorithms.
In Table 6, we compare the averaged MSE achievable by different algorithms, when a ﬁxed total communication budget (in bits) is given for each setting (see the last row of Table 6 for the budget).
Again, note that Algorithm 2 signiﬁcantly outperforms all the other methods.
Classiﬁcation Problem.
We benchmark the performance of the proposed approximation – optimization framework against the centralized algorithms on multi-class classiﬁcation tasks.
For this purpose, we use 90 UCI benchmarking datasets for evaluating the classiﬁcation performance of the algorithms.
For the datasets, we follow the pre-processing suggested in Fern´andez-Delgado et al.
The datasets are chosen such that the sample sizes for each task are smaller than(Arora et al., 2020), with an average of approximately 965 samples per task.
The samples are randomly split into equal sized training and test sets, and typical 4-fold cross-validation is used for performance comparison (Pedregosa et al., 2011).
Note that for testing we utilize Algorithm 4 in Appendix E. For communication comparison, we consider three settings where the data is equally split randomly among agents with number of agents chosen as 5 , 10 and 20 .
Next, we discuss the algorithms and parameter setting.
Algorithm and parameter settings. We evaluate the classiﬁcation performance (test accuracy) of the proposed algorithm on three popular GIP kernels, namely the Polynomial, the Gaussian, and the NTK.
For multi-class classiﬁcation, we utilize kernel SVM with soft margin and regularization hyper-parameter λ tuned from the set { 10 − 2 , 10 − 1 , 10 0 , 10 1 , 10 2 } .
For all the kernels, the hyperparameters are tuned using the validation method in Fern´andez-Delgado et al.
For the NTK, we choose the kernel corresponding to a single hidden-layer ReLU network (please see (15)).
Moreover, for the decentralized implementation all the kernels are estimated locally at individual nodes using the GIP kernel approximation in Algorithm 1.
All the presented results are averaged overindependent runs of the algorithms. [Table] [Figure] [Table]
[Figure]
[Table] Discussion. In Figure 1, we compare the test accuracy of the proposed approximation – optimization framework against the baseline centralized algorithm for the Gaussian, the Polynomial [Table]
and the
NTK as P increases.
Note that for all the kernels, the test accuracy of Algorithm 2 increases rapidly in the beginning and approaches the centralized benchmarks as P increases.
Importantly, note that with only P = 1000 , the test accuracy of Algorithm 2 is within 2% of the centralized benchmarks under all the settings.
Since, this P is independent of the parameter dimension (i
[Table]
e., D in (4)), this implies that the proposed approximation – optimization approaches can be utilized to alleviate the communication burden for distributed learning algorithms, especially for overparameterized learning problems.
To analyze the communication performance of the proposed approach, in Table 7 we compare the overall bits communicated by each agent to achieve an accuracy of 75% .
We note that for learning with the NTK, each agent needs to communicate only 220 real values 4 on average for a network with 10 agents.
This implies that a total of 2200 real values are shared across the network, which is only about 2 .
5 times the total sample size ( N ≈ 965 on average).
In contrast, if the agents share high-dimensional parameters (cf.
Table 1) (resp.
{+partial data), then the total real values shared will be 965 β for β ≥ 2 per iteration (resp. 
+}
· d f , where d f is the dimension of the (shared) features), which will be much higher than our proposed approximation – optimization schemes even for moderate values of d f .
B R ELEVANT L ITERATURE
Current state-of-the-art decentralized/distributed kernel regression problems can be classiﬁed into three main categories.
Below, we discuss each class of algorithms and their properties.
One of the most popular approaches for distributed kernel learning is the divide-and-conquer approach, where the overall dataset is partitioned among multiple agents, and each agent locally learns its corresponding optimal function (Zhang et al., 2015).
These locally learned functions at each agent are then shared via one-shot communication to construct the global model.
Divide-and-conquer approaches can guarantee the same generalization performance as of a centralized system (Zhang et al., 2015; Chang et al., 2016; Lin et al., 2017; Guo et al., 2017; M¨ucke & Blanchard, 2018).
However, both practically and theoretically, divide-and-conquer approaches impose strict restriction on the number of agents, i.e., the performance of the distributed kernel regression degrades signiﬁcantly as the number of agents increase (see Figure 2 in Lin et al.
In Lin et al.
(2020), the au-
[Table]
thors proposed distributed kernel ridge regression with communications (DKRR-CM), where they showed that the dependence on the number of nodes can be improved by allowing some iterative communication between the nodes.
However, a major issue with the state-of-the-art communication based approaches is that they require (neighboring) agents to share their local data with each other (or with the server), hence raising privacy concerns.
A number of works have been proposed for decentralized/distributed kernel learning including Predd et al.
; Gao et al.
Shin et al.
; Chouvardas & Draief (2016); Koppel et al.
; Pradhan et al.
(2021) that require the (neighboring) agents to share raw data among each other that is undesirable for many practical applications.
Random Fourier features approach ﬁrst proposed in Rahimi & Recht (2008) for centralized learning, and later adopted for decentralized kernel learning in (Bouboulis et al., 2017; Richards et al., 2020;
Xu et al., 2020; Liu et al., 2021) alleviates some of the privacy concerns by allowing nodes to share ﬁnite dimensional parameters.
In the random features (RF) approach, all the agents share a random seed to construct a ﬁnite dimensional feature map (Bouboulis et al., 2017; Xu et al., 2020).
The kernel learning problem is then solved in the parameter domain with each agent iteratively updating the local parameters while sharing the learned parameters with the server (or the neighboring agents) in each cycle to achieve consensus.
Speciﬁcally, Bouboulis et al.
(2017) and Richards et al.
(2020) developed distributed GD based algorithms for the kernel regression problems.
In Bouboulis et al.
(2017), the authors adopted a simple combine then adapt (CTA) algorithm, and provided asymptotic consensus guarantees and regret bounds for the optimization problem with general loss functions.
In Richards et al.
(2020) the authors proposed Decentralized RF algorithm based on RF approximation, they focused on (cid:96) 2 loss and provided generalization guarantees for the decentralized kernel learning problem.
In contrast to Bouboulis et al.
(2017) and Richards et al.
(2020), Xu et al.
(2020) proposed DKLA and COKE, where they utilized decentralized ADMM (Shi et al., 2014) to solve the (RF approximated) decentralized kernel learning problem, and provided both optimization and generalization guarantees for the proposed algorithms.
The work Liu et al.
(2021) proposed DKRRRF-CM and improved upon the results of Lin et al.
by adopting the RF approach.
Speciﬁcally,
DKRR-RF-CM avoids data sharing and improves the dependence on the number of agents to guarantee optimal generalization guarantees compared to DKRR-CM (Lin et al., 2020).
Please see Table 1 for a comparison of these approaches to that of the learning framework proposed in this work.
Despite of all the beneﬁts of the RF approaches, they suffer from excessive communication overhead when the parameter dimension is large compared to the data size (i.e., the overparameterized regime)
(Belkin et al., 2019).
For example, the number of random features required to approximate the NTK (Jacot et al., 2018; Arora et al., 2019) of a multi-layer neural network increase exponentially with the number of layers and can be much larger than the number of data samples even for a single layer NTK (Arora et al., 2019; Han et al., 2021).
For such problems, it is prohibitively expensive to share the learned local parameters at each communication round.
To reduce the communication overhead, there are two classical approaches: (i) sharing model parameters intermittently between computations (McMahan et al., 2017; Stich, 2018; Lin et al., 2018), and/or, (ii) compressing model parameters before sharing (Seide et al., 2014; Alistarh et al., 2017; Bernstein et al., 2018).
Although these approaches can lead to some communication savings, they require iteratively communicating high-dimensional parameters or might lead to increased variance (McMahan et al., 2017; Alistarh et al., 2017), hence slowing down convergence.
As discussed in the introduction section, perhaps a more efﬁcient way is to directly share the local data if the size of local data is much smaller than that of the parameters.
However, this approach raises privacy concerns, and is rarely used in practice.
Therefore, it is necessary to devise techniques for decentralize kernel learning where one can avoid sharing very high-dimensional parameters iteratively, and at the same time does not require raw data sharing among agents.
In this work, we take an approach orthogonal to all the above mentioned works and propose a decentralized kernel learning framework that is suitable for learning in the overparameterized regime.
Speciﬁcally, the proposed framework departs from the popular consensus-based approaches by locally solving the learning problem independently at each agent.
This is accomplished by approximating the “global” learning problem locally at each agent via one-shot information sharing among agents before initialization (see Section 3 for more details).
C N EURAL TANGENT KERNEL
In this section, we discuss the NTK as a motivation to deﬁne the GIP kernels.
Neural tangent kernel (NTK): The NTK of an L hidden-layer fully-connected neural network with element wise activation a : R → R is Jacot et al.
; Chizat et al.
Arora et al.
[Equation]
where Σ ( h ) and ˙Σ ( h ) are deﬁned recursively as: [Equation]
[Equation]
)] , where ˙ a ( · ) is the derivative of a .
For ReLU activation the NTK in (14) is a GIP kernel (Han et al., 2021) and can be deﬁned in closed form (Arora et al., 2020; Han et al., 2021).
Note that to approximate the multi-layer NTK one needs at least P
= O ( N 2 ) parameters (Arora et al., 2019).
Therefore, implementing a decentralized method that utilizes parameter sharing will incur heavy communication cost.
Next, we show that the NTK can easily be approximated in a decentralized manner using the GIP kernel characterization.
For ease of exposition, we consider the NTK with ReLU activation for a single hidden-layer neural network (Cho & Saul, 2009; Cho, 2012; Chizat et al., 2019) where the weights of the ﬁnal layer are chosen from U [ {− 1 , 1 } ]
[Equation]
where ψ ( x, x (cid:48) ) =
arccos( x T x (cid:48) / (cid:107) x (cid:107)(cid:107) x (cid:48) (cid:107) ) ∈
[0 , π ] .
Note that the NTK (15) depends on the innerproduct as well as the angle between x and x (cid:48) , and the former can be constructed by the latter if one also knows the norms of the data samples.
The main reason we focus on the GIP kernels for decentralized implementation is that, this class of kernels can be fully speciﬁed at each agent if the norms of all the feature vectors as well as the angles between any pair of them are known at each agent.
Moreover, for many applications of practical interest (Bietti & Mairal, 2019; Geifman et al., 2020; Pedregosa et al., 2011) normalized feature vectors are utilized for learning, for such problems the GIP kernel at each agent can be fully computed only with the knowledge of the pairwise angles between the (normalized) feature vectors. Further, the RF kernels also exhibit a special structure where they can be approximated with the knowledge of feature mapping φ P ( · ) ∈ R P corresponding to all the feature vectors. The RF kernel approximation is expected to work well for generic kernels (Rahimi & Recht, 2008; Rudi & Rosasco, 2017), however, they might not be suited for all the problems, especially for overparameterized problems (like NTK in (15)), where P can be very large , i.e., P = O ( N β ) with β > 1 (Han et al., 2021; Arora et al., 2019) (cf. Table 1 and discussion in Section 1).
For such overparameterized problems, GIP kernel is an appealing alternative to the
RF kernel.
The above discussed properties of the GIP (and the RF) kernels enable us to design an efﬁcient decentralized kernel approximation and message exchange scheme, which allows the agents to estimate randomly approximated versions of these kernels, but without directly sharing the (raw) local data.
1: Initialize: α 1 m = α 1 ∈ R N , ∆ 1 m = ∇ ˆ R P ( α 1 m ) , step-sizes { η t } Tt =1 at each agent m ∈
[ M ] 2: for m ∈
[ M ] do 3:
Using Algorithm 1 construct K P 4:
Construct local functions ˆ L m,P ( α ) (cf. equation 10) 5: for t = 1 to T do 6:
Update : α t +1 m =
(cid:80) k
∈NB m [ W ] mk α tm − η t ∆ tm 7:
Gradient Tracking : ∆ t +1 m =
(cid:80) k
∈NB m [ W ] mk ∆ tk + ∇ ˆ R P ( α t +1 m )
− ∇ ˆ R P ( α tm ) 8: end for 9: end for 10: Return: α T +1 m for all m ∈
[ M ]
D A N ALGORITHM BASED ON DECENTRALIZED GRADIENT TRACKING (DGT)
Note that in this setting we utilize DGT to learn α locally.
Since, this algorithm is a natural extension of DGT in the functional space, there will be consensus in parameters α m learned at each agent m ∈
[ M ] (Qu & Li, 2018).
Note that in contrast to Algorithm 2, DGT does not require sharing labels.
For DGT we assume that the agents communicate via a undirected graph represented by a doubly stochastic matrix W .
We denote by NB m the neighbourhood of agent m ∈
[ M ] , i.e.,
NB m = { k ∈ [ M ] : [ W ] mk (cid:54) = 0 } .
The steps of DGT are listed in Algorithm 3.
In Step 3, each agent locally computes K P .
We utilize the fact that the once K P is known, the loss in (10) is decomposable in m ∈
[ M ] .
This implies that each agent can update the local parameters α m using
Steps 6 and 7.
Below we list important properties of Algorithm 3:
[Communication] The total communication required at each agent is O ( nP + NT ) values (where the term nP are bits for the GIP and real values for the RF kernels).
The DGT algorithm is useful when problem (10) is strongly-convex (or PL), in that case T
=
O (log N ) (Qu & Li, 2018) and total communication becomes ˜ O
( nP + N ) .
Note that even for non-convex losses ˆ L m,P ( · ) , the local objective at each agent can be made strongly-convex with large regularizer, λ .
[Consensus] DGT with appropriately chosen step-sizes { η t } Tt =1 will guarantee consensus of α m ’s, this will ensure functional consensus, i.e., ˆ f m,P ≈ ˆ f ¯ m,P for m (cid:54) =
¯ m and m, ¯ m ∈
[ M ] .
E T ESTING ALGORITHM FOR DECENTRALIZED KERNEL REGRESSION
In this section, we outline how to perform testing once the kernel model has been trained to solve (10).
Suppose that there is a set N test test samples { ( x ( i ) , y ( i ) )
} N test i =1 .
The idea is to utilize the local information at each agent, { A m } Mm =1 , combined with the new data, to construct the testing kernel matrix, based on which the testing process can be carried out.
Speciﬁcally, we use Algorithm 4 to construct the kernel matrix K P,
test ∈ R N test × N as 1.
GIP kernel: For the GIP kernel, we consider the same setting as (7) and construct K P. test as
[Equation]
) where k P ( x ( i ) , x ( j ) m ) ∀ i ∈ [ N test ] and ∀ j ∈ N m , ∀ m ∈ [ M ] are the individual elements of K P, test and where ψ P ( x ( i ) is deﬁned using A test and A m as:
[Equation]
, where A (: ,i ) ∈ R P represents the i th column of matrix A .
RF kernel: Similarly, for the RF kernel, we construct K P. test as
[Equation]
where again k P ( x ( i ) , x ( j ) m ) are the individual elements of K P, test , similar to (9), we have φ P x i )
=
√ P
[ A ,i test ] ) m ]
Once we have K P, test , then the testing is performed as:
[Equation]
) where and (cid:2) K P, test · α ( T +1) m (cid:3) ( i ) represents the i th element of the vector [ K P, test · α ( T +1) m ] ∈ R N test .
Using (18) we can evaluate the test loss
[Equation]
Next, we present the performance of the multi-agent kernel approximation framework proposed in this work.
Note that the algorithms presented above work for arbitrary loss functions.
However, in the sequel we focus on the (cid:96) 2 loss and provide training and generalization loss incurred because of the multi-agent kernel approximation.
Remark 5 (Optimization performance of GD (Nesterov, 2003) and DGT (Qu & Li, 2018)) .
Algorithms 2 and 3 can solve the approximated kernel problem (10) to arbitrary accuracy.
In particular, for strongly-convex and smooth objectives, both GD and DGT converge at a linear rate.
On the other hand, for convex and smooth objectives both GD and DGT converge sub-linearly with rate O (1 /T ) .
F N OTATIONS
First we deﬁne some notations below.
• Recall that the features and the labels are generated as ( x, y ) ∼ π ( x, y ) where π ( x, y ) deﬁnes the joint probability distribution on X × R .
• π x denotes the marginal distribution of the features on X .
• π ( y | x ) denotes the conditional distribution of the label y given the feature x on R .
• We deﬁne by L 2 ( X , π x ) as the space of square integrable functions with Lebesgue measure deﬁned by π x . • The inner-product and the norm on space
L 2 ( X , π x ) is deﬁned by Inner-product: (cid:104) f, g (cid:105)
π x = (cid:90) x ∈X f ( x ) g ( x ) dπ x ( x ) , Norm: (cid:107) f (cid:107) π x = (cid:0) (cid:104) f, f (cid:105)
π x (cid:1) 1 / 2 , ∀ f, g ∈ L 2 ( X, π x ) .
• We deﬁne the RKHS induced by mapping k and k P as H and H P , respectively, as H = span { k ( x, · ) : x ∈ X} , and H P = span { k P ( x, · ) : x ∈ X} .
with inner-products deﬁned as (cid:104) k
( x, · ) , k ( x (cid:48) , · ) (cid:105) H =
k ( x, x (cid:48) )
in H and (cid:104) k P ( x, · ) , k P ( x (cid:48) [Equation]
We also note that H P = R P for the RF kernels.
Next, we deﬁne some operators which shall be utilized in bounding the optimization and generalization losses.
• We deﬁne K :
L 2 ( X , π x )
→ L 2 ( X , π x ) and K P : L 2 ( X , π x ) → L 2 ( X , π x ) as ( Kh )
[Equation] x ) = [Equation] z ∈X k ( x, z ) h ( z ) dπ z
and ( K P h )( x ) =
(cid:90)
z ∈X k P ( x, z ) h ( z ) dπ z .
• Next, we deﬁne Φ :
H → L 2 ( X , π x ) and Φ P : H P
→ L 2 ( X , π x ) as
[Equation]
[Equation]
and (Φ
[Equation]
(cid:104)
[Equation]
, β (cid:105) • Moreover, using the deﬁnitions of Φ and Φ P we deﬁne the covariances C : H → H and C P : H P → H P as C =
Φ ∗ Φ and C P
[Equation] P [Equation] P , Also, note that from the deﬁnition of Φ we have K = ΦΦ ∗ and K P =
Φ P Φ ∗ P .
• We deﬁne the ﬁnite sample version of the operators Φ ∗ and Φ ∗ P as ˆΦ ∗ : R N → H and ˆΦ ∗ P : R N → H P , respectively as
[Equation]
and
[Equation]
where ˆ y ∈ R N .
• From the deﬁnition of ˆΦ and ˆΦ P – We deﬁne ˆ C : H → H and ˆ C P : H P → H P as ˆ C = ˆΦ ∗ ˆΦ and ˆ
[Equation]
= ˆΦ ∗ P ˆΦ P .
– Moreover, we deﬁne ˆ K : R N → R N and ˆ K P : R N → R N as ˆ K = ˆΦˆΦ ∗ and ˆ K P = ˆΦ P ˆΦ ∗ P .
• For an operator A , we deﬁne the operator A λ as A λ
=
A + λI .
We also denote by λ max ( A ) as the largest eigen value of A .
We denote by (cid:107) A (cid:107) the operator norm and (cid:107) A (cid:107) HS the Hilbert-Schmidt norm of a linear operator A .
Remark 6 .
Note from the deﬁnition of operators ˆ K and ˆ K P we have [Equation] and
[Equation]
We will utilize this fact and the operators deﬁned above in the proofs for bounding the optimization and generalization errors for the kernel methods.
G O PTIMIZATION PERFORMANCE OF MULTI - AGENT KERNEL APPROXIMATION FRAMEWORK : P ROOF OF T HEOREM 4.
In this section, we present the complete proof of Theorem 4.2.
Speciﬁcally, the main technical results in this section is to analyze the impact of the multi-agent kernel approximation framework proposed in this work on the training performance for the proposed algorithms.
For this purpose, we bound difference between the optimal losses for the exact and the approximated kernel ridge regression problems (12) and (13).
Towards this end, we ﬁrst provide a few technical lemmas.
First, we present two concentration inequalities to bound the distance between the empirical mean of a sequence of random matrices (and random variables) from their mean.
These results will be later used in Proposition G.4 to bound the operator norms of difference between the kernel matrix K and its approximation K P .
First, we present the Bernstein’s inequality for random matrices.
Proposition G.1 (Bernstein’s inequality for random matrices, Theorem 6.1.1 in Tropp (2012)) .
Consider a sequence X (cid:96) for (cid:96) ∈ N of independent, random, Hermitian matrices of dimension N .
Assume that
E [ X (cid:96) ]
= 0 and (cid:107) X (cid:96) (cid:107) ≤ C almost surely .
Deﬁne a random matrix as Y :=
(cid:80) (cid:96) X (cid:96) , and denote its variance as σ 2 :
= (cid:107) E [ Y 2 ] (cid:107) .
Then for all (cid:15) ≥ 0 , we have
[Equation]
[Equation] Moreover, with probability at least 1 − δ for δ ∈ (0 , 1] we have:
[Equation]
In the next proposition, we present the Bernstein’s inequality to bound the distance between the empirical mean of a sequence of random variables and their mean.
Proposition G.2 (Bernstein’s inequality for random variables) .
Consider a sequence { x (cid:96) } (cid:96) ∈ N of independent, random variables such that the following holds:
[Equation]
almost surely .
Deﬁne the random variable y :
= (cid:80) (cid:96) x (cid:96) , and denote its variance as σ 2 :=
E [ y 2 ] .
Then for all (cid:15) ≥ 0 , we have:
[Equation]
Next, using the above propositions we derive a union bound on the estimate ψ P ( x ( i ) m , x ( j )¯ m ) in (8) from its mean ψ ( x ( i ) m , x ( j )¯ m ) for all training data samples, i.e., for all i ∈ N m and j ∈ N ¯ m with m, ¯ m ∈
[ M ] .
Recall from Algorithm 1 and (8) that we have
[Equation]
where A (: ,i ) m ∈ R P .
Speciﬁcally, the matrix A m for the GIP kernel for the m th agent is
[Equation]
where we choose ζ ( x, ω )
= 1[ ω T x ≥ 0] and with { ω (cid:96) } L(cid:96) =
1 chosen uniformly randomly from a circularly symmetric distribution.
Please see discussion in Section 3.1 for details.
Below, we present the union bound.
Proposition G.3 (Union Bound: GIP kernel) .
Let ψ ( x ( i ) m , x ( j )¯ m )
= arccos (cid:16) (cid:104) x
( i ) m ,x ( j )¯ m (cid:105) (cid:107) x ( i ) m (cid:107)(cid:107) x ( j )¯ m (cid:107) (cid:17) be the angle between two feature vectors x ( i ) m and x ( j )¯ m , and let ψ P ( x ( i ) m , x ( j )¯ m ) be an estimate of ψ ( x ( i ) m , x ( j )¯ m ) as deﬁned in (20).
Then with probability at least 1 − δ , the following holds:
[Equation]
∀ i ∈ N m , ∀ j ∈ N ¯ m and ∀ m, ¯ m ∈
[ M ] .
Proof.
Note from the deﬁnition of ψ P ( x ( i ) m , x ( j ) ¯ m ) in (20) for the GIP kernel
[Equation]
we have
[Equation] [Equation]
with ζ ( x, ω ) = 1[ ω T x ≥ 0] and where ( i ) uses E
[ ζ ( x ( i ) m , ω ) ζ ( x [Equation] ¯ [Equation]
[Equation] [Equation] for ω ∼
N (0 , I ) Chizat et al.
; Cho & Saul (2009); Cho (2012), and ( ii ) follows from the reverse triangle inequality.
Next, note from the deﬁnition of A m , we have
[Equation]
Let us deﬁne the random variable z (cid:96) as
[Equation]
Note from the deﬁnition of z (cid:96) that we have
[Equation]
[Equation]
To proceed, let us deﬁne the sum of z (cid:96) ’s as Z ( x ( i ) m , x ( j )¯ m )
= (cid:80) L(cid:96)
=1 z (cid:96) .
Noticing that we can relate
Z ( x ( i ) m , x ( j )¯ m ) to the quantity (21) we want to bound in as
[Equation]
where the equality ( i ) follows from (22).
Moreover, note that this term is zero mean which follows from the fact that each z (cid:96) is zero mean in the ﬁrst equality.
Further, we bound the variance of
Z ( x ( i ) m , x ( j )¯ m ) as
[Equation]
where ( i ) comes from the facts that (a): each z (cid:96) for (cid:96) ∈ [ P ] is zero mean, and (b): z (cid:96) and z k are independent for all (cid:96) (cid:54)
=
k ; ( ii ) uses the fact that z 2 (cid:96) ≤ 1 /P 2 for all (cid:96) ∈
[ P ] .
Applying Proposition I.1 to the random variable Z
( x ( i ) m , x ( j )¯ m ) , we get P
[Equation]
− [Equation] [Equation]
P
[Equation]
( (cid:15) [Equation] P ) (cid:1) (cid:19) . Taking the union bound over all the training samples, i.e., ∀ i ∈ N m , ∀ j ∈ N ¯ m with ∀ m, ¯ m ∈
[ M ] , we get
[Equation]
i
[Equation]
m
[Equation]
j
[Equation]
m
[Equation]
i
[Equation]
j
[Equation]
m [Equation] Note from above that we have that with probability at least 1 − δ
[Equation]
j ∈ N ¯ m with ∀ m, ¯ m ∈
[ M ] .
Substituting the above in (21) completes the proof..
Next, we provide a key technical result, which bounds the operator norm of the difference between the kernel, K
and it’s approximation K P .
We present the results for both th GIP and the RF kernel.
The following proposition is utilized in the proof of Theorem 4.2.
Proposition G.4.
The following holds with probability at least 1 − δ
For GIP kernel, we have
[Equation]
For RF kernel, we have
[Equation]
Proof.
First, we consider the GIP kernel.
( i ) :
GIP kernel: Note from the deﬁnition of approximated GIP kernel in (7) that the individual elements of K P ∈ R N × N are g ( ψ P ( x ( i ) m , x ( j )¯ m ) , (cid:107) x
( i ) m (cid:107) , (cid:107) x
( j )¯ m (cid:107) ) .
Similarly, from (5) individual elements of K ∈ R N × N are g ( ψ ( x ( i ) m , x ( j )¯ m ) , (cid:107) x
( i ) m (cid:107) , (cid:107) x
( j )¯ m (cid:107) ) , ∀ i ∈ N m , ∀ j ∈ N ¯ m with ∀ m, ¯ m ∈
[ M ] .
Next, we use the notation B = (cid:2) b ij (cid:3) ∈ R N × N to denote the matrix with individual elements b ij , i.e.,
B =   b [Equation]
b [Equation] N . [Equation]
[Equation]
Then we have with probability at least 1 − δ [Table]
[Equation]
[Table]
[Equation]
[Table]
m
[Equation]
x
[Equation]
(cid:12)(cid:12)(cid:12)(cid:21)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ( iv
[Equation]
P log
[Equation]
N δ + 8 π 3 P log 2 N δ (cid:19) · (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) 
  1 1 · · ·.. .
1 1 · · ·   (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) (
[Equation]
P
[Equation]
N δ (cid:19) , where equality ( i ) uses the deﬁnition of the GIP kernel and its approximation in (5) and (7), resp.
; inequality ( ii ) results from the fact that for L p -operator norm with p an even integer we have for any matrix B (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:13)   bb· · · b 1 N .
b Nb N· · · b NN   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)   | b 11 | | b 12 | · · · | b 1 N | .
| b N 1 | | b N 2 | · · · | b NN | 
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) , inequality ( iii ) results from the Lipschitzness of function g ( · , z 1 , z 2 ) in Assumption 2 and the fact that for any matrices A and B with positive entries, we have (cid:107) B (cid:107) ≤ (cid:107)
A + B (cid:107) ; inequality ( iv ) utilizes Proposition G.3 and again the fact that for any matrices A and B with positive entries, we have (cid:107) B (cid:107) ≤ (cid:107)
A + B (cid:107) ; Finally, ( v ) uses the fact that the L 2 -operator norm (maximum eigenvalue) of the all one matrix of dimension N × N is N .
Therefore, the ﬁrst result has been proven.
( ii ) :
RF kernel: First, we note that the matrix K P can be decomposed as
[Equation]
where K (cid:96) ∈ R N × N consists of individual elements of the form ζ ( x ( i ) m , ω (cid:96) )
· ζ ( x ( j )¯ m , ω (cid:96) ) , ∀ i ∈ N m , ∀ j ∈ N ¯ m with ∀ m, ¯ m ∈
[ M ] .
Recall that { ω (cid:96) } P(cid:96)
=1 are drawn i.i.d from distribution p ( ω ) .
Please see discussion in Section 3.1 on RF kernel approximation.
Then using the matrix notation deﬁned in (23) we we write K (cid:96) as
[Equation]
∀
[Equation] [ M ] [Equation] We apply Proposition G.1 on the sequence of random matrices deﬁned by [ K (cid:96) − K ] /P .
Note that we have:
[Equation]
and
[Equation]
where the ﬁrst equality follows from the deﬁnition of the RF kernel in (6) and the deﬁnition of K (cid:96) , and the second inequality follows from Assumption 1.
To apply Proposition G.1, we need to upper bound the variance of the sum (cid:80)
[Equation]
K P = K P − K .
Towards this end, we obtain:
[Equation]
where ( i ) follows from the fact that K (cid:96) − K are zero mean and independent across (cid:96) ∈
[ P ] ; ( ii ) utilizes that K (cid:96) − K are i.i.d.
across (cid:96) ∈
[ P ] ; for inequality ( iii ) , we used the notation that for a matrix B , | B | denotes the matrix with absolute values of B and combined it with the fact that for
L p -operator norm with p as an even integer we have for any matrix B that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) bb· · · b 1 N .
b Nb N· · · b NN 
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) | b 11 | | b 12 | · · · | b 1 N | .
| b N 1 | | b N 2 | · · · | b NN | 
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Inequality, ( iv ) results from the use of Assumption 1, speciﬁcally, using Assumption 1 verify that each element of E [( K (cid:96) − K ) 2 ] is bounded by 4 κ 4 N and combine with the fact that for any matrices
A and B with positive entries, we have (cid:107) B (cid:107) ≤ (cid:107)
A + B (cid:107) ; ﬁnally, ( v ) utilizes that for all one matrix of dimension
N × N the L 2 operator norm (maximum eigen-value) is equal to N .
Therefore, we have the result.
Next, using Proposition G.1 with X (cid:96) =
K (cid:96) − K P , we have (cid:107) K P − K (cid:107) ≤ (cid:114) 8 κ 4 N
P log 2 N δ + 4 Nκ3 P log 2 N δ = κ 2 · N · (cid:18)(cid:114) P log 2 N δ + 3 P log 2 N δ (cid:19) .
with probability at least 1 − δ .
Therefore, the proof is complete.
Finally, we state our main result.
Theorem G.5.
For any ﬁxed λ > 0 , the following holds with probability at least 1 − δ 1.
For the GIP kernel, we have:
[Equation]
Moreover, we have:
[Equation]
For the RF kernel, we have:
[Equation]
Moreover, we have
[Equation]
Proof.
Recall from the notations in Appendix F that we have ˆ K = 1 N K , and
[Equation]
and we deﬁned
[Equation] [Equation]
where I is the identity matrix of some appropriate size.
It is easy to observe that the optimal solutions to problems (12) and (13) are given by the following closed-form solutions
[Equation]
From the deﬁnition of the loss function for the (cid:96) 2 loss
[Equation]
we obtain:
[Equation]
where ( i ) utilizes (25) and (26); ( ii ) uses the following
[Equation]
and ( iii ) results from the application of Cauchy-Schwartz and triangle inequality
[Equation]
Next, we consider the term(A) and term(B) in (27), and provide upper bounds for them.
Let us ﬁrst analyze term(B).
we have:
[Equation]
where ( i ) follows from the deﬁnition of ˆ K λ and ˆ K P,λ ; ( ii ) results from A − 1 − B − 1 =
A − 1 ( B − A ) B − 1 for invertible matrices A and B and ( iii ) again results from the deﬁnition of ˆ K λ and ˆ K P,λ .
This implies that we have:
[Equation]
where inequality ( i ) results from the Cauchy-Schwartz inequality; ( ii ) uses the fact that (cid:107) ˆ K − 1 P,λ (cid:107) ≤ 1 /λ , (cid:107) ˆ K − 1 λ (cid:107) ≤ 1 /λ and (cid:107) ¯ y (cid:107) ≤ √ N · R , and ( iii ) utilizes the deﬁnition of ˆ K and ˆ K P .
Next, we utilize high probability bounds for the term (cid:107) K − K P (cid:107) , which characterize the effect of sampling using P observations.
Towards this end, we invoke Proposition G.4, which provides two high probability bounds for this term, one for the GIP, and one for the RF kernel.
Speciﬁcally, for the former kernel, we have the following
[Equation]
with probability at least 1 − δ .
Similarly, we have for the RF kernel
[Equation]
with probability at least 1 − δ .
This completes the analysis for term(B) in (27).
Next, we consider the term(A) in (27).
We have:
[Equation]
Now substituting the bounds for term(A) and term(B) into (27)
[Equation]
we get the following bounds:
For the GIP kernel:
[Equation]
For the RF kernel:
[Equation]
Next, we bound the loss incurred because of multi-agent kernel approximation for the oveall objective for both the kernels.
Note from the deﬁnition of the ˆ R ( α ) and ˆ R P ( α )
[Equation]
which follows from the use of triangle inequality.
Note that we have already bounded the ﬁrst term.
In the following, we bound the second term of (31) above.
[Equation]
where ( i ) follows from the deﬁnition of ˆ α ∗ and ˆ α ∗ P along with the notation in (24) and (25); ( ii ) results from Cauchy-Schwartz inequality, and ( iii ) uses Assumption 3 and the triangle inequality.
Next, we bound term(C) and term(D) in (32).
First, from the deﬁnition of term(C), we have term(
[Equation]
= [Equation] where ( i ) results from the Cauchy-Schwartz inequality; ( ii ) utilizes A − 1 − B − 1 =
A − 1 ( B − A ) B −for invertible matrices A and B along with Cauchy-Schwartz inequality; ( iii ) uses that fact that we have (cid:107) ˆ K − 1 λ (cid:107) ≤ 1 /λ , (cid:107)
ˆ K − 1 P,λ (cid:107) ≤ 1 /λ and (cid:107) ˆ K · ˆ K − 1 λ (cid:107) ≤ 1 , and ( iv ) results from the notation in (24).
Next, we bound term(D) as term
[Equation]
where inequality ( i ) follows from Cauchy-Schwartz inequality; ( ii ) results from (28) and the application of Cauchy-Schwartz inequality; ( iii ) follows from the fact that we have (cid:107) ˆ K − 1 λ (cid:107) ≤ 1 /λ and (cid:107)
ˆ K − 1 P,λ (cid:107) ≤ 1 /λ , and ( iv ) results from the notation deﬁned in (24).
Substituting the upper bounds of term(C) and term(D) in (32), we get
[Equation]
Finally, using Lemma G.4, we have for the GIP kernel
[Equation]
Substituting, the above expression in (31) and combining with the bound in (29), we get the statement of the theorem.
Similarly, we have for the RF kernel
[Equation]
Finally, substituting the above expression in (31) and combining with the bound in (30) [Equation] we get the statement of the theorem.
Therefore, the proof is complete.
Finally, the proof of Theorem 4.2 is a simple consequence of Theorem G.5.
H G ENERALIZATION PERFORMANCE OF MULTI - AGENT KERNEL APPROXIMATION FRAMEWORK : P ROOF OF T HEOREM 4.
To prove generalization bounds we extensively use the notations deﬁned in Appendix F.
Deﬁnition H.1.
We deﬁne by f π as the MMSE estimator for f as
[Equation]
We make the following assumption.
Assumption 5.
There exists f H ∈ H s.t.
L ( f H )
=
inf h ∈H L ( h ) .
From the deﬁnition of MMSE estimator above we have f π = f H .
Moreover, this assumption implies that there exists h ∈ L 2 ( X , π x ) s.t.
we have
[Equation]
33) for some (cid:107) h (cid:107) π x ≤ R .
Note that Assumption 5 is same as Assumption 4 in the main paper.
The existence of f H ensures that (33) holds for some h ∈ L 2 ( X , π x ) .
Moreover, without loss of generality we assume (cid:107) h (cid:107) π x to be bounded by R (same R as in Assumption 3).
Note that for the (cid:96) 2 loss deﬁned in (11) and under Assumption 5, we have for any f ∈ L 2 ( X , π x ) (Caponnetto & De Vito, 2007)
[Equation]
Now the goal is to bound the generalization error with f replaced by the ˆ f P (learned by solving the approximate kernel problem stated in (13)).
Using the notations deﬁned in Appendix F, we have
[Equation]
where we have deﬁned ˆ y = 1 √ N ¯ y .
Using the above we have
[Equation]
Next, we bound each term separately below for both the GIP and the RF kernels.
We note that for the
RF kernel we follow the proofs developed in Rudi & Rosasco (2017) with some minor corrections.
Bound for term(A)
First, we consider term(A) given in (35).
Lemma H.1.
For any δ ∈ (0 , 1] , with λ ≤ 3 4 (cid:107) K (cid:107) and N ≥ max (cid:26) 72 κ 2 λ , [Equation] λ 9 (cid:27)
[Equation]
κ
[Equation]
· δ .
Moreover, with P ≥ max
[Equation] [Table] π 2 G [Equation]
K (cid:107) 2 (cid:111) log 2 δ for the GIP kernel and with P ≥ max (cid:110) 8 κ
[Table]
[Table]
for the RF kernel, we have with probability at least 1 − 3 δ
[Equation]
Proof.
From the deﬁnition of term(A) in (35), we have
[Equation]
where ( i ) follows from C 1 / 2 P,λ · C − 1 / 2 P, λ
= I .
Next, we individually bound term(A1) and term(A2) below.
First
[Equation]
we bound term(A1) using Proposition 8 in Rudi & Rosasco (2017)
[Equation]
we deﬁne β 1 := λ max
[Equation]
where ( i ) utilizes the Cauchy-Schwartz inequality; ( ii ) follows from the fact that:
[Equation]
ﬁnally, ( iii ) utilizes Proposition 8 in Rudi & Rosasco (2017).
Next, we bound term β 1 .
Using
Lemma H.2, with the choice of λ ≤ (cid:107) C P (cid:107) and with N ≥ max [Equation] κ
[Equation]
λ , 2 λ
[Equation]
log 4 κ 2 λδ , we have with probability at least 1 − δ
[Equation]
Finally, we need to ensure that the random variable (cid:107) C P (cid:107) ≥ λ with high probability for both the
GIP and the RF kernel.
Note from Proposition H.7 that for: ( i ) the GIP kernel with P ≥ max (cid:110) 8 ,
π 2 G 2 (cid:107) K (cid:107) 2 (cid:111) log 2 δ and ( ii ) for the RF kernel with P ≥ max (cid:110) 8 κ 2 , 32 κ 2 (cid:107) K (cid:107) 2 (cid:111) log 2 δ , we have with probability at least 1 − δ (cid:107) K − K P (cid:107) ≤ 1 4 (cid:107) K (cid:107) .
Next, choosing λ s.t.
we have λ ≤ 34 (cid:107) K (cid:107) .
Note that the choice of λ and the choice of P implies that we have with probability at least 1 − δ
[Equation]
[-≥ 
 [Equation] K (cid:107) ≥ λ.-] Now combining the two facts that β 1 ≤ 1 / 3 holds with probability 1 − δ conditioned on (cid:107) C P (cid:107) ≥ λ and (cid:107) C P (cid:107) ≥ λ holds with probability at least 1 − δ , it is easy to verify that we have β 1 ≤ 1 / 3 with probability at least 1 − 2 δ .
This further implies that we have with probability at least 1 − 2 δ (cid:107) term ( A 1) (cid:107) ≤ 3 2 .
Next, we consider term(A2) in (39)term
[Equation]
We modify Lemma 6 in Rudi & Rosasco (2017) to bound this term.
First, note that the random variable φ P ( x i ) · y i − Φ ∗ P · f π is zero mean.
Next, we apply Bernstein’s inequality stated in Proposition
I.2.
Consider the random vector z i deﬁned as
[Equation]
with mean E [ z i ] = µ = C − 1 / 2 P,λ · Φ ∗ L · f π .
Note that
z
i are i.i.d.
with the moments of random variable z = C [Equation] [Equation]
L
[Equation]
· φ P ( x ) · y bounded as [Equation] where ( i ) follows from the Jensen’s inequality and ( ii ) utilizes the convexity of (cid:107) · (cid:107) p H P .
Next, we bound E (cid:107) z (cid:107) p H P as
[Equation]
where ( i ) utilizes Cauchy-Schwartz inequality; ( ii ) follows from Assumption 3 and, ( iii ) results from the following
[Equation]
where we utilized the deﬁnition of k P ( · , · ) and Assumption 1.
Combining with the above we get
[Equation]
This implies that we can apply Bernstein’s inequality stated in Proposition I.2 with σ = B = 2 Rκ/ √ λ , therefore, we get with probability at least 1 − δ
[Equation]
Finally, substituting the bound for term (A1) derived in (40) and the bound for term (A2) derived in (41) in (39) and noting the fact that (40) holds with probability at least 1 − 2 δ and (41) holds with probability at least 1 − δ , we have with probability at least 1 − 3 δ
[Equation]
Therefore, the lemma is proved.
Lemma H.2.
For some δ ∈ (0 , 1] and with λ ≤ (cid:107) C P (cid:107) we have with probability at least 1 − δ
[Equation]
Proof.
The proof follows from Proposition 6 in Rudi & Rosasco (2017).
Next, we bound term(B).
Bound for term(B)
Let us consider term(B) in (36), we have
Lemma H.3.
For δ ∈ (0 , 1] , with P ≥ max (cid:110) 8 , 288 π 2 G 2 λ(cid:111) log 2 δ for the GIP kernel and with
P ≥ max (cid:110) 2 λ 9 , 72 κ 2 λ (cid:111) log 4 κ 2 λ · δ for the RF kernel, we have with probability at least 1 − 4 δ (cid:107) term(B) (cid:107)
[Equation]
Proof.
Consider term(B), note that from Lemma 3 in Rudi & Rosasco (2017), we have
[Equation]
Next, we bound each term individually.
First, we consider term(B1) in (42) above, we deﬁne β 2 :=
λ max (cid:2) K − 1 / 2 λ
[Equation]
K
[Equation]
K −
[Equation]
/ 2 λ (cid:3) , then we have
[Equation]
1 /, where ( i ) follows from the fact that (cid:107) K 1 / 2 (cid:107) ≤ (cid:107) K 1 / 2 λ (cid:107) and ( ii ) utilizes Proposition 8 in
Rudi & Rosasco (2017).
Note that from Lemma H.4 for ( i ) the GIP kernel with P ≥ max (cid:110) 8 , 288 π 2 G 2 λ(cid:111) log 2 δ and for ( ii ) the RF kernel with P ≥ max (cid:110) 2 λ 9 , 72 κ 2 λ (cid:111) log 4 κ 2 λ · δ , we have with probability at least 1 − δ β 2 ≤ 1 3 .
This implies that with probability at least 1 − δ , we have (cid:107) term(B1) (cid:107) ≤ 2 .
Next, note from the deﬁnition of term(B2) in (42) that it is same as term(A1) in (39).
Therefore, using the same bound as derived in (40), we have with probability at least 1 − 2 δ
[Equation]
[Equation]
Next, we bound term(B3) in (42).
Using Equation (28) of Lemma 7 in Rudi & Rosasco (2017), we have with probability at least 1 − δ (cid:107) term(B3) (cid:107) ≤
[Equation]
Finally, substituting the bounds derived in (43), (44) and (45) in (42) and utilizing the fact that (43) holds with probability at least 1 − δ , (44) holds with probability at least 1 − 2 δ and (45) holds with probability at least 1 − δ , we get with probability at least 1 − 4 δ (cid:107) term(B) (cid:107)
[Equation]
N logδ .
Therefore, the lemma is proved.
Lemma H.4.
For some δ ∈ (0 , 1] , then with probability at least 1 − δ for 1.
We have for the GIP kernel
[Equation]
We have for the RF kernel (Proposition 6 in Rudi & Rosasco (2017))
[Equation]
Proof.
For the GIP kernel, we have
[Equation]
where ( i ) follows from the deﬁnition of the operator norm and ( ii ) uses the fact that (cid:107) K − 1 / 2 λ (cid:107) ≤ 1 / √ λ .
Next, utilizing the bound for (cid:107) K − K P (cid:107) from Proposition H.7, we have the proof of the lemma.
Next, we consider term(C).
Bound for term(C)
We consider term(C) in (37).
Lemma H.5.
For δ ∈ (0 , 1] , with probability at least 1 − δ 1.
For the GIP kernel, we have
[Equation]
For the RF kernel, with P ≥ max (cid:110) 2 λ 9 , 72 κ 2 λ (cid:111) log 4 κ 2 λ · δ we have from Lemma 8 in Rudi & Rosasco (2017)
[Equation]
Proof.
Let us consider the ﬁrst term of term(C) for the GIP kernel, we have from the deﬁnition of
C P,λ
[Equation]
where ( i ) follows from the deﬁnition of C P ; ( ii ) uses the fact that for any continuous spectral function and compact operator Z , we have F ( Z ∗ Z ) Z ∗ =
Z ∗ F ( ZZ ∗ )
; ( iii ) uses the deﬁnition of
K P and ﬁnally, ( iv ) results from the notation of K P,λ .
Therefore, we have
[Equation]
where ( i ) uses the identity A · ( A + λ · I ) −
= I − λ
( A + λ · I ) − 1 for bounded positive operators; ( ii ) results from A − 1 − B − 1 =
A − 1 · ( B − A ) · B − 1 for invertible bounded positive operators, and ( iii ) utilizes Assumption 5.
Taking the norm on both sides and utilizing Cauchy-Schwartz inequality, we get [Equation]
· (cid:13)(cid:13) K − 1 / 2 λ
[Equation]
where the last inequality follows from (cid:13)(cid:13) λ · K − 1 P,λ (cid:13)(cid:13) ≤ 1 , (cid:13)(cid:13) K − 1 / 2 λ
[Equation]
≤ [Equation] (cid:13)(cid:13) K [Equation]
2 λ
[Equation]
and (cid:107) h (cid:107) π x ≤ R from Assumption 5.
Finally, using Proposition H.7 we have for the generalized inner-product kernel with probability at least 1 − δ [Equation] δ
[Equation]
Therefore, the lemma is proved.
Let us next consider term(D).
Bound for term(D)
We consider term(D) in equation 38.
Lemma H.6.
We have
[Equation]
Proof.
We have
[Equation]
where ( i ) uses the identity A · ( A + λ · I ) −
= I − λ
( A + λ · I ) − 1 and ( ii ) follows from Assumption 4.
Taking the norm on both sides and applying Cauchy-Schwartz inequality, we get
[Equation]
where the last inequality results from (cid:107) λ 1 / 2 · K − 1 / 2 λ
[Equation]
− [Equation] [Equation] ≤ 1 and (cid:107) h (cid:107) π x ≤ R follows from Assumption 5.
Next, we combine the four terms to bound (cid:107) ˆ f P − f π (cid:107) π x .
Proposition H.7.
The following holds with probability at least 1 − δ 1.
For the GIP kernel, we have
[Equation]
For the RF kernel, we have
[Equation]
Proof.
We ﬁrst prove ( i )
[Equation]
note from the deﬁnition of K − K P and the norm in L 2 ( X , π x ) -space
we have
[Equation]
h ( z ) dπ z (cid:13)(cid:13)(cid:13)
π x ≤ G sup (cid:107) h (cid:107) πx =
(cid:13)(cid:13)(cid:13) (cid:90) z ∈X (cid:12)(cid:12) ψ
( x, z ) − ψ P ( x, z ) (cid:12)(cid:12) h
( z ) dπ z (cid:13)(cid:13)(cid:13)
π x
Recall using ζ ( x, ω ) = 1 [ ω T x ≥ 0] for the GIP kernel, we have from the deﬁnition of ψ ( x, z ) and ψ P ( x, z ) ψ ( x, z ) = (cid:12)(cid:12)(cid:12)(cid:12) π − 2 π E (cid:2) ζ ( x, ω ) ζ ( z, ω ) (cid:3)(cid:12)(cid:12)(cid:12)(cid:12) and ψ P ( x, z ) = (cid:12)(cid:12)(cid:12)(cid:12)
π − 2 π P
P (cid:88) (cid:96) =
ζ ( x, ω (cid:96) ) ζ ( z, ω (cid:96) ) (cid:12)(cid:12)(cid:12)(cid:12) .
Substituting in the above
[Table]
we
[Table] [Equation]
Finally, we bound (cid:13)(cid:13)
Ω − Ω P (cid:13)(cid:13) using Proposition I.2.
Using the notation ζ ω (cid:96) =
ζ ( · , ω (cid:96) ) , we deﬁne the random variable Θ (cid:96) =
Ω − ζ ω (cid:96) ⊗ ζ ω (cid:96) , this implies that from the deﬁnition of Ω − Ω P , we have from Lemma 1 in Rudi & Rosasco (2017)
[Equation]
with
[Equation]
Next, note that Θ (cid:96) are random vectors belonging to the Hilbert space of Hilbert-Schmidt operators on L 2 ( X , π x ) .
Therefore, we can apply Proposition I.2.
Moreover, from the deﬁnition of Ω and Ω P , [Equation] Ω [Equation]
2 and E [Equation] Θ (cid:96) (cid:107) 2 HS ≤ 1 . Therefore, from the application of Proposition I.2, we have with probability at least 1 − δ
[Equation]
Substituting in the expression above we get the result of statement ( i ) .
The proof of ( ii ) follows from Lemma 9 in Rudi & Rosasco (2017).
Combining bounds on term(A), term(B), term(C) and term(D)
Here, we combine the results of Lemmas H.1, H.3, H.5 and H.6 to bound (cid:107) ˆ f P − f π (cid:107) π x .
We get the following theorem.
Theorem H.8.
For the two classes of kernels with λ ≤ 34 (cid:107) K (cid:107) , the following is satisﬁed with probability at least 1 − δ 1.
For the GIP kernel, with
[Equation]
and with P ≥ max
[Equation]
[Equation]
[Equation]
, we have
[Equation]
For the RF kernel, with
[Equation]
and with
P ≥ max
[Equation] [Equation]
we have
[Equation]
Finally, using the above Theorem H.8 and (34), we bound the generalization error.
Theorem H.9.
For the two classes of kernels with λ ≤ 34 (cid:107) K (cid:107) , the following is satisﬁed with probability at least 1 − δ 1.
For the GIP kernel, with
[Equation]
and with
P ≥ max
[Equation]
we have
[Equation]
For the RF kernel, with
[Equation]
and with
P ≥ max
[Equation]
we have
[Equation]
Finally, we choose the parameters to get the result of Theorem 4.3.
parameters:
Here, we optimally choose the parameters λ , P , N such that all the conditions in Theorem H.8 and H.9 are satisﬁed and the proposed approximation approach achieves the (minimax) optimal generalization performance.
• GIP kernel: For the GIP kernel, we choose the parameters
as follows:
– Regularization parameter: λ = 1 / √ N .
– Number of overall samples: N
[Equation]
– Number of bits communicated:
[Equation]
• RF kernel: For the RF kernel we choose the parameters as follows: – Regularization parameter: λ = 1 / √ N .
– Number of overall samples:
[Equation]
– Number of real values communicated:
[Equation]
δ
Now with the above choice of parameters we have Theorem 4.3 stated beolw [Equation]
Theorem H.10.
With the choice of parameters stated above [Equation] we have with probability at least 1 − δ for both the GIP and RF kernels that
[Equation]
I C ONCENTRATION INEQUALITIES
Proposition I.1.
For x 1 , . . .
, x N , a sequence of zero-mean i.i.d.
random variables with x i ∈ R for all i ∈
[ N ] .
If there exists T, S ∈ R s.t.
x i ≤ T a.s. and E x 2 i ≤ S for all i ∈
[ N ] .
Then with probability at least 1 − δ we have
[Equation]
Proposition I.2.
For x 1 , . . .
, x N , a sequence of i.i.d.
random vectors on a separable Hilbert space
H .
Assume µ = E x i and let σ, B ≥ 0 s.t.
[Equation]
for any i ∈ [ N ] .
Then we have with probability at least 1 − δ
[Equation]
Proposition I.3.
Let F be a separable Hilbert space and let X 1 , . . .
X N be a sequence of i.i.d.
self-adjoint positive random operators on F . Assume that E [ X i ] = 0 and λ max ( X i ) ≤ T a.s. for some T > 0 for all i ∈ [ N ] .
Let S be such that E ( X i ) 2 ≤ S .
Then with probability at least 1 − δ we have
[Equation]
(cid:107) S (cid:107) β N , with β = log 2 Tr S (cid:107) S (cid:107)· δ .
J S UFFICIENT CONDITION FOR POSITIVE DEFINITE GIP KERNEL
Assumption 6.
We assume that there exists a mapping from { x i , . . .
, x N } (cid:55)→ { ˜ x i , . . . ,
˜ x N } with x i ∈ R d and ˜ x i ∈ R D for some D ∈ N such that the following is satisﬁed:
(cid:107) ˜ x
i (cid:107) =
(cid:107) x
i (cid:107) for all i ∈ [ N ] .
ψ (˜ x i , ˜ x j ) =
ψ P ( x i , x j ) for all i, j ∈ [ N ] where ψ P ( x i , x j ) is deﬁned in (8).
The above assumption implies that there exists a mapping that transforms the feature vectors such that their norms are preserved and the pairwise angles between the transformed set of vectors are same as the approximated angles for the GIP kernel (cf.
Proposition J.1.
Under Assumption 6, the kernel K P approximated using (7) for the GIP kernel is positive semi-deﬁnite.
Proof.
From the deﬁnition of the approximated GIP kernel (7) and Assumption 6, we have k
[Equation]
Using the above fact, we have for the approximated GIP kernel for any vector v ∈ R N v T K P v =
N (cid:88) i,j = [ v ] i [ v ] j k P ( x i , x j ) = N (cid:88) i,j = [ v ] i [ v ] j k (˜ x i , ˜ x j ) ( i ) = (cid:28) N (cid:88) i = [ v ] i φ ( x i ) , N (cid:88) j
= [ v ] j φ ( x j ) (cid:29) = (cid:13)(cid:13)(cid:13)(cid:13) N (cid:88) i = [ v ] i φ ( x i ) (cid:13)(cid:13)(cid:13)(cid:13) ≥ 0 , where ( i ) follows from Mercer’s theorem, i.e., for positive deﬁnite kernel k , there exists φ ( · ) such that k
( x i , x j ) = (cid:104) φ ( x i ) , φ ( x j ) (cid:105) .
Moreover, we used the fact that the GIP kernel k is well deﬁned irrespective of the dimension of the feature vectors.
Therefore, we have the proof.