E FFECTIVE D ISTRIBUTED L EARNING WITH R ANDOM F EATURES : I MPROVED B OUNDS AND A LGORITHMS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we study the statistical properties of distributed kernel ridge regression together with random features (DKRR-RF), and obtain optimal generalization bounds under the basic setting, which can substantially relax the restriction on the number of local machines in the existing state-of-art bounds.
Speciﬁcally, we ﬁrst show that the simple combination of divide-and-conquer technique and random features can achieve the same statistical accuracy as the exact KRR in expectation requiring only O ( |D| ) memory and O ( |D| 1 . 5 ) time.
Then, beyond the generalization bounds in expectation that demonstrate the average information for multiple trails, we derive generalization bounds in probability to capture the learning performance for a single trail.
Finally, we propose an effective communication strategy to further improve the performance of DKRR-RF, and validate the theoretical bounds via numerical experiments
I NTRODUCTION
Kernel ridge regression (KRR) is one of the most popular nonparametric learning methods (Vapnik, 2000).
Despite the excellent theoretical guarantees, KRR does not scale well in large scale settings because of high time and memory complexities.
Distributed learning (Zhang et al., 2013; Hsieh et al., 2014; Chang et al., 2017b; Li et al., 2019b; Lin et al., 2020), random features (Rahimi & Recht, 2007;
Sutherland & Schneider, 2015; Rudi & Rosasco, 2017; Rudi et al., 2018; Liu et al., 2020; Avron et al., 2017a; Yu et al., 2016; Jacot et al., 2020), and Nystr ¨ om methods (Drineas & Mahoney, 2005;
Ding & Liao, 2012; Yang et al., 2012; Camoriano et al., 2016; Si et al., 2016; Musco & Musco, 2017; Kriukova et al., 2017) are the most widely used large scale techniques to address the scalability issues.
Recent statistical learning works on KRR together with large scale approaches demonstrate that these large scale approaches can not only obtain great computational gains but also can guarantee the optimal theoretical properties, such as KRR with divide-and-conquer (Zhang et al., 2013; 2015;
Chang et al., 2017b;a; Guo et al., 2017; Lin et al., 2017; Li et al., 2019b; Lin et al., 2020), with random features (Rudi & Rosasco, 2017;
Li et al., 2019c; Carratino et al., 2018;
Yang et al., 2012), and with Nystr ¨ om methods (Bach, 2013; Alaoui & Mahoney, 2015; Rudi et al., 2015; 2017).
The combinations of distributed learning and other large scale approaches are very intuitive but effective strategies to further improve the effectiveness, such as distributed learning with gradient descent algorithms (Lin & Zhou, 2018; Richards et al., 2020), with multi-pass SGD (Lin & Rosasco, 2017; Lin & Cevher, 2018; 2020), with random features (Li et al., 2019b), and with Nystr¨om methods (Yin et al., 2020).
The optimal generalization performance of these combining approaches has been studied, however, the main theoretical problem is that there is a strict restriction on the number of local machines.
For sample, in (Lin & Zhou, 2018; Li et al., 2019b; Yin et al., 2020), to guarantee the optimal generalization performance in the basic setting, the upper bounds of the local machines are restricted to be a constant, which is difﬁcult to be satisﬁed in real applications.
In this paper, we aim at enlarging the number of local machines by considering communications among different local machines.
This paper makes the following three main contributions.
Firstly, we improve the existing state-of-art results of the divide-and-conquer technique together with random features.
We prove that the optimal generalization performance can be guaranteed even the partitions reach Ω( (cid:112) |D| ) , which are limited to a constant Ω(1) for the existing bounds in the basic setting, |D| is the size of the data sets.
Secondly, to essentially reﬂect the generalization performance, beyond the minimax optimal rates in expectation, we derive optimal learning rates in probability, which can capture the learning performance for a single trail.
Finally, we develop a communication strategy to further improve the performance of our proposed method, and validate the effectiveness of the proposed communications via both theoretical assessments and numerical experiments.
Related Work The most related work includes the statistical analysis of distributed learning and random features.
Distributed learning.
Optimal learning rates for divide-and-conquer KRR in expectation were established in the seminal work (Zhang et al., 2013; 2015).
An improved bound was derived in (Lin et al., 2017) based on a novel tool of integral operator.
Based on the proof techniques proposed in (Zhang et al., 2013; 2015; Lin et al., 2017), optimal learning rates were established for distributed spectral algorithms (Guo et al., 2017; Blanchard & M¨ucke, 2018; Lin & Cevher, 2020), distributed gradient descent algorithms (Lin & Zhou, 2018; Richards et al., 2020), distributed semi-supervised learning (Chang et al., 2017b), distributed local average regression (Chang et al., 2017a;
Lin & Cevher, 2020), localized SVM (Meister & Steinwart, 2016), etc.
Some other communication strategies for distributed learning have been provided, see e.g. (Fan et al., 2019;
Li et al., 2019a; Lin & Cevher, 2020; Li et al., 2020), and references therein.
The theoretical analysis mentioned above shows that the divide-and-conquer learning can achieve the same statistical accuracy as the exact KRR, however, there is a strict restriction on the number of local machines.
The optimal learning rates with a less strict condition on the number of local machines for distributed stochastic gradient methods and spectral algorithms were established in (Lin & Cevher, 2020).
In (Lin et al., 2020), they considered the communications among different local machines to enlarge the number of local machines.
However, the communication strategy proposed in (Lin et al., 2020) based on an operator representation, which requires communicating the input data between each local machine.
Thus, it is difﬁcult to protect the data privacy of each local machine.
Furthermore, for each iteration, the communication complexity of each local machine is O ( |D| d ) , where d denotes the dimension, which is infeasible in practice for large scale data sets.
Random Features.
The generalization bound of random features was ﬁrst proposed in (Rahimi & Recht, 2008), which shows that O ( |D| ) random features are needed for O (1 / (cid:112) |D| ) learning rate.
Some works further studied its theoretical performance (Cortes et al., 2010; Yang et al., 2012;
Sutherland & Schneider, 2015; Sriperumbudur & Szab ´ o, 2015; Bach, 2017; Avron et al., 2017b).
By applying the standard integral operator framework (Smale & Zhou, 2007; Caponnetto & Vito, 2007), the optimal generalization bounds of KRR with random features were established in (Rudi & Rosasco, 2017), which requires only O ( (cid:112) |D| log( |D| )) random features.
To decrease the size of random features, an improved approach was proposed based on a novel leverage score sampling strategy (Rudi et al., 2018).
Sun et al.
(2018) extended the result of (Rudi & Rosasco, 2017) to SVM.
In (Li et al., 2019c), they further devised a simple framework for the uniﬁed analysis of random Fourier features, which can be applied to KRR, as well as SVM and logistic regression.
To further improve the effectiveness, recently, Li et al.
(2019b) considered the simple combination of divide-and-conquer and random features.
However, to guarantee the optimal generalization performance, the number of local machines should be restricted to a constant, degenerating it into a single random features-based large scale KRR.
B ACKGROUND
In a standard framework of supervised learning, there is a probability space
X
× Y with an unknown distribution ρ , where X is the input and Y is the output space.
The sample set D
= { ( x i , y i ) } ni =1 of size n is drawn i.i.d from X × Y with respect to ρ .
Let K : X × X
→ R be a Mercer kernel, and
H K be its reproducing kernel Hilbert space (RKHS) (Steinwart & Christmann, 2008; Vapnik, 2000), and assume that K ( x , x (cid:48) ) ≤ κ, ∀ x , x (cid:48) ∈ X .
Throughout, we will denote the inner product in H K by (cid:104)· , ·(cid:105) K , and corresponding norm by (cid:107) · (cid:107) K .
Kernel Ridge Regression (KRR)
KRR is one of the most popular nonparametric learning methods (Shawe-Taylor & Cristianini, 2000;
Vapnik, 2000), which can be stated as
[Equation]
where λ > 0 is the regularization parameter, |D| is the size of D .
Using the representation theorem (Shawe-Taylor & Cristianini, 2000; Vapnik, 2000), f D ,λ can be written as f D ,λ ( x ) = (cid:80) |D| i =
α i K
( x i , x ) with α
=
( K D + λ I ) − 1 y D , where K D = 1 |D| [ K D ( x i , x j )] |D| i,j =1 is the |D| × |D| kernel matrix and y D
= 1 |D| ( y 1 , . . .
, y |D| ) T .
Despite the excellent theoretical guarantees (Blanchard & Kr¨amer, 2010; Caponnetto & Vito, 2007),
KRR requires O ( |D| 2 ) memory to store K D , and O ( |D| 3 ) time to solve inverse of K D
+ λ I , which is infeasible for large scale settings.
KRR with Random Features (KRR-RF)
Assuming the spectral measure has a density function (cid:37) ( · ) , the corresponding shift-invariant kernel can be written as K ( x , x (cid:48) )
= (cid:82) Ω ψ ( x , ω ) ψ ( x (cid:48) , ω ) (cid:37) ( ω ) d ω , where ψ : X × Ω → R is a continuous and bounded function with respect to ω and x .
The main idea behind random
Fourier features is to approximate the kernel function K ( x , x (cid:48) ) by its Monte-Carlo estimation (Rahimi & Recht, 2007): K M ( x , x (cid:48) )
= 1 M (cid:80) Mi
=1 ψ ( x , ω i ) ψ
( x (cid:48) , ω i ) =
(cid:104) φ M ( x ) , φ M ( x (cid:48) )
(cid:105) , where φ M ( x ) = 1 √ M ( ψ ( x , ω 1 ) , . . .
, ψ ( x , ω M )) T .
The solution of KRR with random features can be written as f M, D ,
[Equation]
x ) =
w
[Equation]
M, D ,
[Equation]
x )
[Equation]
w
[Equation]
(2) where Φ M, D = 1 √ |D| ( φ M ( x 1 ) , . .
, φ M ( x |D| )) and ¯ y D = 1 √ |D| ( y 1 , . .
, y |D| ) T .
KRR-RF requires O ( M |D| ) to store Φ M, D , O ( M 3 ) and O ( M 2 |D| ) time to solve the inverse of ( Φ M, D Φ T M, D + λ I ) and the matrix multiplication Φ M, D Φ M, D , respectively.
Thus, the total space and time complexity of KRR-RF are O ( M |D| ) and O ( M 2 |D| ) , M (cid:28) |D| , respectively.
Distributed KRR with Random Features (DKRR-RF)
Let {D j } mj
=1 be m disjoint subsets with D = ∪ mj
=1 D j .
The distributed KRR with random features (DKRR-RF) is deﬁned as
[Equation] [Equation]
) where f M, D j ,λ ( x ) = w T M, D j ,λ φ M ( x ) with w M, D j ,λ = ( Φ M, D j Φ T M, D j + λ I ) − 1 Φ M, D j ¯ y D j .
The space complexity, time complexity and communication complexity of DKRR-RF for each local machine are O ( M |D j | ) , O ( M 3 + M 2 |D j | ) and O ( M ) , respectively.
DKRR-RF WITH C OMMUNICATIONS (DRKK-RF-CM)
In this section, we will present an effective communication strategy to enlarge the number of local machines.
We ﬁrst give the motivation of our communication strategy, and then propose a communication-based method, called DRKK-RF-CM.
The proposed communication strategy are adaptations from (Lin et al., 2020) to avoid communicating local data among partition nodes.
Motivation.
Let g M, D ,λ : R M → R M be g M, D ,λ ( w ) :
= (cid:2)
Φ M, D Φ T M, D + λ I (cid:3) w − Φ M, D ¯ y D .
One can see that 2 g M, D ,λ ( w ) is the gradient of the empirical risk of 1 |D| (cid:80) ( x i ,y i ) ∈D (cid:0) w T φ M ( x i ) − y i (cid:1) 2 + λ (cid:107) w (cid:107) 2 on w .
From Eq.2, we know that for any w , the following equation holds:
[Equation]
Deﬁne ¯ w 0 M, D ,λ = (cid:80) mj
=1 |D j | |D| w M, D j ,λ , it is easy to verify that
[Equation]
Local machine compute β t − 1 j = (cid:104)
Φ M, D j Φ T M, D j
+ λ I (cid:105) − 1 g M, D ,λ ( ¯ w t − 1 M, D ,λ ) cate for f pM, D ,λ =
¯ w pM, D ,λ , φ M (
Comparing 4 and 5, and noting that the global gradient g M, D ,λ ( w ) can be achieved via the communications of each local gradient g M, D j ,λ ( w ) , i.e., g M, D ,λ ( w )
=
(cid:80) m j =
|D
j | |D| g M, D j ,λ ( w ) , thus, we consider the following Newton Raphson iteration-based communication strategy:
[Equation]
We propose an iterative procedure to implement the communication strategy Eq.6, which can be broken down into 4 steps.
At ﬁrst, each local machine computes the local gradient and communicates back to the global machine.
Then the global machine computes the global gradient based on the local gradient, and communicates it to each local machine.
In the third step, each local machine computes β t − 1 j , and communicates back to the global machine.
Finally, the global machine obtains the solution ¯ w tM, D ,λ .
More details can be seen in Algorithm 1.
Complexity analysis.
Space complexity: each local machine only needs to store Φ M, D j and the local gradient g M, D j ,λ , thus the space complexity of each local machine is O ( M |D j | + M )
=
O ( M |D j | ) ;
Time complexity: for each local machine, we only need to compute the matrix multiplication Φ M, D j Φ T M, D j and the inverse of Φ M, D j Φ T M, D j + λ I once.
For each iteration, we need to compute the local gradient g M, D j ,λ and β j for each local machine.
Therefore, the total time complexity of each local machine is O ( M 3 + M 2 |D j | + pM |D j | ) , where p is the number of communication;
Communication complexity: for each iteration, we only communicate the local gradient g M, D j ,λ and β j to the global machine, and receive the gradient g M, D ,λ and ¯ w t − 1 M, D ,λ from the global machine, so the total communication complexity is O ( pM ) .
Remark 1.
From the complexity analysis above, we can see that if the number of the communication p satisfying p ≤ M or p ≤ |D j | , then the time and space complexity of DKRR-RF-CM are the same as DKRR-RF.
Only the communication complexity is slightly increased from O ( M ) to O ( pM ) .
T HEORETICAL A NALYSIS
In this section, we analyze the generalization performances of DKRR-RF and DKRR-RF-CM.
The performance of the algorithm is usually measured by the expected risk E ( f ) =
(cid:82) X×Y ( f ( x ) − y ) 2 dρ ( x , y ) .
The optimal hypothesis f H K in H K is denoted by f H K = arg
min f ∈H K E ( f ) , and we assume f H K exists in the paper.
4. O PTIMAL LEARNING RATE FOR DKRR IN E XPECTATION
Theorem 1.
Suppose that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) and | y | ≤ ζ .
If λ = Ω( |D| − 12 ) , |D 1 |
= . . .
|D m | , the number of partitions m and the number of random features M respectively correspond to m (cid:46) (cid:112) |D| and M (cid:38) (cid:112) |D| , then, for every δ ∈ (0 , 1] , with probability at least 1 − δ , we have E [ E ( ¯ f 0 M, D ,λ )] − E
( f H K ) =
O (cid:16) |D| − 1 2 log 2 (1 /δ )
(cid:17) .
From Theorem 1, one can see that if m (cid:46) (cid:112) |D| and M (cid:38) (cid:112) |D| , the learning rate of the generalization bound can reach O (1 / (cid:112) |D| ) , which is optimal in a minmax sense (Rudi & Rosasco, 2017;
Caponnetto & Vito, 2007).
It means that, in this basic setting, as long as the number of partitions and random features are in order Ω
( (cid:112) |D| )
, the corresponding ridge regression estimator has optimal generalization properties.
The assumption of | y | ≤ ζ can be related to the Bernstein condition or moment assumption (Blanchard & Kr¨amer, 2016), but for simplicity, in this paper, we only consider that Y is bounded.
Optimal learning rates for divide-and-conquer KRR in expectation have been established in (Zhang et al., 2013; 2015;
Lin et al., 2017), etc.
However, there is a strict restriction on the number of local machines m .
Speciﬁcally, in (Lin et al., 2017), to reach the optimal rate, m should to restrict to a constant m = Ω(1) .
In (Li et al., 2019b), the authors have studied the generalization performance of the combination of divide-and-conquer technique and random features.
Using the same setting as Theorem 1 (that is r = 1 / 2 and γ = 1 in Theorem 8 of Li et al.
(2019b))
, they prove that, if
M (cid:38) (cid:112) |D| and m (cid:46) Ω(1) , then E [ E ( ¯ f 0 M, D ,λ )] − E ( f H K ) =
O ( |D| − 12 log 2 (1 /δ )) .
It means that to guarantee the optimal generalization properties, the number of partitions should be restricted to a constant, but for our result is Ω( (cid:112) |D| ) .
In (Li et al., 2019b), they also considered using the unlabeled data to enlarge the number of partitions.
They have proved that (see Corollary 12 of Li et al.
(2019b) for detail),
if M (cid:38) (cid:112) |D| and m (cid:46) |D ∗ | / |D| , then E [ E ( ¯ f 0 M, D ,λ )]
−E ( f H K )
=
O ( |D| − 12 log 2 (1 /δ )) , where D ∗ is the dataset includes both labeled and unlabeled data.
Thus, if we want the number of partitions m to reach Ω( (cid:112) |D| ) as the same as our Theorem 1, the size of |D ∗ | should be Ω( (cid:112) |D| ) times of |D| .
In this case, the data size of each local machine is |D| = |D| 3 / 2 / (cid:112) |D| , so the time and space complexity are the same as KRR with a single random feature technique.
4. O PTIMAL LEARNING RATES FOR DKRR IN P ROBABILITY
Note that E [ E ( ¯ f 0 M, D ,λ )] −E ( f H K )
=
E [ (cid:107) ¯ f 0 M, D ,λ − f H K (cid:107) 2 ρ ] (Caponnetto & Vito, 2007), thus Theorem 1 proposes the optimal learning rate in expectation , which demonstrates the average information for multiple trails, but may fail to capture the learning performance for a single trail.
To essentially reﬂect the generalization performance for a single trail, we derive the optimal learning rate in probability :
Theorem 2.
Under the same assumptions as Theorem 1.
If λ = Ω( |D| − 12 ) , |D 1 | = . . .
|D m | , m (cid:46) |D| 14 and M (cid:38) |D| 12 , then, for every δ ∈ (0 , 1] , with probability at least 1 − δ , we have (cid:13)(cid:13)(cid:13) ¯ f 0 M, D , [Equation] − f [Equation]
K (cid:13)(cid:13)(cid:13) 2 ρ = O (cid:16) |D| − 12 log 2 (
[Equation]
) (cid:17) . To guarantee the optimal generalization properties in probability, the number of partitions should be restricted to Ω( |D| 1 / 4 ) , which is stricter than Ω( |D| 1 / 2 ) in Theorem 1.
This is because the generalization error in expectation can be decomposed into approximation error, sample error and distributed error (more details can be seen in Proposition 1 in Appendix), but the error decomposition in probability is not easy to separate a distributed error in probability to control the number of local machines.
The derive the optimal learning rate, we provide a novel decomposition, please see details in Proposition 9 in Appendix.
The following result demonstrates that the proposed communication strategy can enlarge the number of partitions in probability.
Theorem 3.
Under the same assumptions as Theorem 1, If λ = Ω( |D| − 12 ) , |D 1 | = . . .
|D m | , m (cid:46) |D| p +1 2( p +2) and M (cid:38) |D| 12 , then, for every δ ∈ (0 , 1] , with probability at least 1 − δ , we have (cid:13)(cid:13)(cid:13) ¯ f pM, D ,λ − f H K (cid:13)(cid:13)(cid:13) 2 ρ = O (cid:16) |D| − 12 log p +2 (1 /δ ) (cid:17) , where ¯ f pM, D ,λ is returned by Algorithm 1 under p -th iterations.
Compared Theorem 3 with Theorem 2, it is clear that the proposed communication strategy can relax the restriction on m from Ω (cid:0) |D| 1 / 4 (cid:1) to Ω (cid:0) |D| ( p +1) / (2( p +2)) (cid:1) .
Note that m is monotonically increasing with the number of communications p , which can demonstrate the power of the proposed communications.
When p → ∞ , the partitions can reach Ω( (cid:112) |D| ) , which is the same as the generalization bound in expectation. [Table] D| p [Table]
2( p [Table] 2) |D| [Table]
Inprobability |D|
[Table]
[Table]
|D
[Table]
p |D [Table] Remark 2. In the main text of this paper, we only give the optimal rates of DKRR-RF in the basic setting.
The fast learning rates can be achieved under favorable conditions, see in Appendix.
Remark 3 (The Signiﬁcance of Distributed Learning for RF) . At ﬁrst glance, it seems that the bottleneck in learning with random Fourier features is not the size of the dataset but the number of features. However, from (Rudi & Rosasco, 2017), one can see that we only requiring O ( (cid:112) |D| ) random features to guarantee the optimal performance, so the total computational complexity is O ( M 3 + M 2 |D| )
=
O ( |D| 2 ) .
Thus, the computational bottleneck in learning is not only the size of random features, but also the size of dataset .
If we don’t consider reducing the size of D , the computational complexity is |D| 2 in the basic setting, which is not suitable for large scale problems.
Distributed learning is one of the most popular methods to reduce the size of dataset.
The distributed learning bring the distributed error, but can decrease the variance of the model Zhang et al.
Thus, how to choose an appropriate number of partitions to trade off the distributed error and the variance to guarantee the optimal performance is a very interesting and signiﬁcant direction.
4. C OMPARED WITH THE R ELATED W ORK Comparisons of the Time and Space Complexities
Table 1 reports the statistical and computational properties of the related approaches and our theoretical ﬁndings under the basic setting.
We see that our DKRR-RF can guarantee the optimal generalization performance in expectation only requiring |D| memory and |D| 1 . 5 time, which is more effective than other methods.
For DKRR-RF-CM, we can also see that it can guarantee optimal generalization performance in probability requiring less complexity than the communication-based method of DKRR-CM (Lin et al., 2020).
Remark 4.
In (Rudi et al., 2017), the authors considered combining the Nystr¨om method and preconditioned conjugate gradient (PCG) (Cutajar et al., 2016) to scale up KRR.
As far as we know, it is the only existing work that can guarantee optimal statistical accuracy, only requiring |D| memory and |D| 1 . 5 time for KRR.
In this paper, we consider combining distributed learning and random features, a completely different path from (Rudi et al., 2017).
Note that in our proposed method, we need to compute the inverse of Φ M, D j Φ T M, D j + λ I , which requires |D| 1 . 5 time.
Inspired by (Rudi et al., 2017), we can also adopt PCG to avoid the inverse calculation, which can further speed up our proposed DKRR-RF-CM.
The combination of DKRR-RF-CM and PCG may open a path to reach the linear time complexity for optimal learning rate.
Novelty and Proof Techniques
The most related works of our paper are (Li et al., 2019b), (Lin et al., 2020) and (Rudi & Rosasco, 2017).
We discuss the novel techniques adopted to derive the improved results compared with them.
Compared with (Li et al., 2019b).
(a) To derive the learning bounds, Q M, D := (cid:107) ( C M + λI ) − 1 / 2 ( C M − C M, D )( C M + λI ) − 1 / 2 (cid:107) is required to be estimated, where C M and C M, D are self-adjoint and positive operators deﬁned in Deﬁnition 1 (see in Appendix).
In (Li et al., 2019b), they used a classical approach from (Chang et al., 2017b;
Guo et al., 2017) to estimate Q M, D (see Lemmas 21 and 22 in (Li et al., 2019b)), and obtain that Q M, D ≤ 1 √ λ (cid:107) ( C M − C M, D )( C M + λI ) − 1 / 2 (cid:107) =
O (cid:0) 1 /λ |D| + (cid:112) N ( λ ) /λ |D| (cid:1) , where N ( λ ) is the effective dimension deﬁned in Assumption 1 (see in Appendix).
However, in our paper, we directly estimate Q M, D based on the concentration inequality for self-adjoint operators (Lin & Cevher, 2020; 2018; Caponnetto & Yao, 2006), and prove that Q M, D
= O (cid:0) 1 / ( λ |D| ) + (cid:112) 1 / ( λ |D| ) (cid:1) (see in Proposition 6) .
Thus, our estimation of Q M, D is (cid:112) N ( λ ) tighter than that in (Li et al., 2019b).
This is the one of the key reasons why we can substantially relax the restriction on the number of local machines compared with (Li et al., 2019b); (b) We not only present the bounds in expectation, but also in probability.
To derive the tight bound in probability, we provide new decompositions of (cid:107) ¯ f 0 M, D ,λ − f M, D ,λ (cid:107) and (cid:107) ¯ w 0 M, D ,λ − w M, D ,λ (cid:107) , please see Proposition 9 in detail.
As far as we know, these decompositions are novel; (c) We also consider the Newton Raphson iteration-based communication strategy.
To derive the improved high-probability bounds with communication, we introduce a novel decomposition of (cid:107) ¯ f tM, D ,λ − f M, D ,λ (cid:107) ρ
(see in Proposition 10).
Compared with (Lin et al., 2020).
(a) In (Lin et al., 2020), they also considered a communication strategy to enlarge the number of local machines.
At ﬁrst it seems that it only need to communicate the gradient information, but it should be noted that the gradient information is based on an operator representation (see Eq.(7) in (Lin et al., 2020)), which is usually infeasible in practice.
The authors present a realization for the proposed strategy by communicating the data among each local machine, see in Appendix B (page 34 in (Lin et al., 2020), step 1).
Thus, the data privacy of each local machine is difﬁcult to be protected.
Furthermore, since it requires communicating data D j , j = 1 , . . .
, m , among each local machine, for each iteration, the communication complexity of each local machine is O ( |D| d ) , which is too high for large scale data sets.
However, the communication strategy proposed in this paper only requires communicating the gradient g M, D j ,λ ( ¯ w t − 1 M, D ,λ ) and the model parameters β t − 1 j , rather than the data, therefore our proposed strategy do better on privacy protection.
Moreover, the communication complexity is only O ( M ) for each local machine, M (cid:28) |D| , which is suitable for large scale data sets; (b) At ﬁrst it seems that the proof techniques of (Lin et al., 2020) can be easily extended to our paper, but it is not true.
If we use the same proof technical of (Lin et al., 2020), we can only obtain that (cid:107) f (cid:5) M, D ,λ − f M,λ (cid:107) ρ = (cid:107) ( L M, D + λI ) − 1 ( L M, D − L M )( f ρ − f M,λ ) (cid:107) =
O ((1 / ( √ λ |D| ) + (cid:112) N ( λ ) / |D| ) (cid:107) f ρ − f M,λ (cid:107) ρ / √ λ ) , where f (cid:5) M, D ,λ , f M,λ and f ρ are deﬁned in Deﬁnition 12.
Combing with Proposition 2, one can only obtain that (cid:107) f M, D ,λ − f M,λ (cid:107) ρ
= O (cid:0)(cid:0) 1 /λ |D| + (cid:112) N ( λ ) /λ |D| (cid:1) (cid:107) f ρ − f M,λ (cid:107) ρ (cid:1) .
However, in our paper, we introduce new decompositions of (cid:107) f (cid:5) M, D ,λ − f M,λ (cid:107) ρ and (cid:107) f M, D ,λ − f (cid:5) M, D ,λ (cid:107) ρ (see Propositionsand 3 in detail), and further obtain that (cid:107) f M, D ,λ − f M,λ (cid:107) ρ
= O (cid:0)(cid:0) 1 / ( √ λ |D| ) + (cid:112) N ( λ ) / |D| (cid:1) (cid:107)
f ρ − f M,λ (cid:107) ρ (cid:1) (see in Proposition 4) , which is Ω(1 / √ λ ) tighter than the directly use of the techniques of (Lin et al., 2020).
The novel decompositions (cid:107) f
(cid:5) M, D ,λ − f M,λ (cid:107) ρ and (cid:107) f M, D ,λ − f (cid:5) M, D ,λ (cid:107) ρ are the key reasons why we can guarantee the optimal performance even under m = Ω( (cid:112) |D| ) .
We only give an example here, but the novel decompositions have also been embedded in Propositions 5, 9, 10, etc.
Compared with (Rudi & Rosasco, 2017).
(a) We study the statistical properties of the combination of distributed learning and random features, but in (Rudi & Rosasco, 2017), they only consider random features.
As mentioned above, to estimate the tight bound of the distributed error, we introduce a novel decomposition of (cid:107) ¯ f 0 M, D ,λ − f M, D ,λ (cid:107) to derive the tight bound in expectation, and novel decompositions of (cid:107) ¯ f 0 M, D ,λ − f M, D ,λ (cid:107) and (cid:107) ¯ f pM, D ,λ − f M, D ,λ (cid:107) to derive tight bounds in probability; (b) The combination not only brings distributed error, but also brings some other problems.
If you compare the proofs of (Rudi & Rosasco, 2017) with these of our paper in detail, you can ﬁnd that the decompositions of (Rudi & Rosasco, 2017) and ours are very different.
Overall, we improve the existing state-of-art bounds in expectation, and provide novel communicationbased distributed bounds with RF in probability.
Moreover, we introduce some novel techniques and decompositions to substantially relax the restriction on the number of local machines, which are non-trivial extensions of (Li et al., 2019b; Lin et al., 2020; Rudi & Rosasco, 2017).
E XPERIMENTS
In this section, we validate our theoretical ﬁndings by performing experiments on both simulated and real datasets.
[Figure]
Numerical Experiments.
Inspired by numerical experiments in (Rudi & Rosasco, 2017; Li et al., 2019c), we consider a spline kernel of order q : K 2 q ( x , x (cid:48) )
=
1 + (cid:80) ∞
k =1 cos(2 πk ( x − x (cid:48) )) /
( k 2 q ) .
If the marginal distribution of X is uniform on [0,1], then K 2 q ( x , x (cid:48) )
= (cid:82) 1 0 ψ ( x , ω ) ψ ( x , ω ) (cid:37) ( ω ) d ω , where ψ ( x , ω )
=
K q ( x , ω ) and (cid:37) ( ω ) is also uniform on [0,1].
The random features of the spline kernel are φ M ( x ) =
( ψ ( x , ω 1 ) , . . .
, ψ ( x , ω M )) T / √ M.
According to Theorem 1, 2 and 3, we set the size of the random features to be M = (cid:112) |D| , and ﬁne tune λ around |D| − 1 / 2 using 5-fold cross validation 1 , the tuned set is { 2 − 5 , 2 − 3 , . . .
2 5 }|D| − 1 / 2 .
We let the target function f ∗ be a Gaussian random variable with mean µ = K t ( x, 0) and variance σ 2 = 0 .
We generate 10000 samples for training and 10000 samples for testing.
We use the exact KRR as a baseline, which trains all samples in a batch.
We compare our proposed DKRR-RF-CM ( p = 2 , 4 , 8 ) with KRR and DKRR-RF.
We repeat the training 5 times and estimate the averaged error on testing data.
The mean square error on the test set with different partitions is given in Figure 1(a, b), which can be summarized as follows: 1) When m is not too large, the distributed methods (DRKK-RF and DRKK-RF-CM) are always comparable to original KRR.
There exists an upper bound of m , when larger than it, the error increases dramatically and is far from the original KRR.
This veriﬁes the theoretical statement in Theorem 1, 2 and 3; 2)
The upper bound m of DKRR-RF-CM is much larger than DKRR-RF.
This result is aligned with Theorem 3, which demonstrates that the proposed communication strategy can enlarge the upper bound m ; 3)
The upper bound m of DKRR-RF-CM monotonically increases with the number of communications, which veriﬁes Theorem 3.
Real Data.
In this experiment, we consider the performance on real data.
The empirical evaluations with Gaussian kernel, exp( −(cid:107) x − x (cid:48) (cid:107) 2 /σ ) , on MINIST (random selected 10000 samples for training and 10000 for testing) is given in Figure 1(c), where the optimal σ and λ are selected by 5-fold cross-validation, σ ∈ { 2 i , i = − 10 , − 8 , . . .
, 10 } , { 2 − 5 , 2 − 3 , . . .
2 5 }|D| − 1 / 2 , and the number of random features is 300 .
From Figure 1(c), one can ﬁnd that the result is similar as that on the simulated data, which demonstrates that our proposed communication strategy is also effective on the real dataset.
More experiments on different datasets are given in Appendix G.
C ONCLUSION
In this paper, we study the generalization properties of the combination of distributed learning and random features for ridge regression.
We ﬁrst improve the existing results of divide-and-conquer KRR with random features in expectation.
Then, beyond the expectation, we derive generalization error bound in probability.
Finally, we propose a novel effective communication strategy to further improve the learning performance of the combination method, and demonstrate the power of communications via both theoretical assessments and numerical experiments.
Our results may open several venues for both theoretical and empirical work: (a) combine the approach with gradient algorithms such as preconditioned conjugate gradient Avron et al.
(2017a)
and multi-pass SGD (Carratino et al., 2018;
Lin & Cevher, 2018; 2020); (b) replace synchronous distributed methods with asynchronous ones (Suresh et al., 2017); (c) consider the loss functions other than quadratic loss (Li et al., 2019c).
A A PPENDIX : F AST L EARNING R ATES
In the section, we will show that the fast learning rates can be achieved under favorable conditions.
Let
[Equation]
be the square integrable space, (cid:107) · (cid:107)
ρ be its norm.
Denote the integral operator (Smale & Zhou, 2007)
L K by
[Equation]
Assumption 1.
For λ > 0 , N ( λ ) is the effective dimension of the integral operator L K deﬁned as
N ( λ
[Equation]
L K (cid:1) , where Tr is the trace.
Assuming there exists a constant c ≥ 1 , such that
[Equation]
The effective dimension is a common assumption within the framework of learning theory (Caponnetto & Vito, 2007; Smale & Zhou, 2005; Rudi & Rosasco, 2017), which is used to measure the complexity of the hypothesis space.
It is always satisﬁed for γ = 1 and c = κ .
Equation 7 can control the variance of the estimator and is equivalent to the classic entropy and covering number conditions.
In particular, it holds if the eigenvalues of integral operator L K decay as i − 1 /γ , which is satisﬁed by the popular Gaussian and polynomial kernel functions.
More details can be seen in (Steinwart &
Christmann, 2008; Caponnetto & Vito, 2007; Rudi & Rosasco, 2017).
Assumption 2.
Let f ρ ( x ) =
(cid:82) Y ydρ ( y | x ) be the regression function, ρ ( y | x ) be the conditional distribution at x induced by ρ .
For 12 ≤ r ≤ 1 , assume there exists a g ∈ L 2 ρ X such that
[Equation]
where L rK is the r th power of L K .
Regression function f ρ is the best function in L
ρ X , which is the primary objective in regression problem.
Assumption 2 is used to measure the complexity of the regression function f ρ , which is commonly used in approximation theory (Caponnetto & Vito, 2007).
Equation 8 can be used to control the bias of the estimator, it requires the expansion of the regression function f ρ having coefﬁcients that decay faster than the eigenvalues of integral operator L K .
The larger the value of r , the faster the coefﬁcients decay.
The case r = 1 / 2 means that f ρ ∈ H K .
More detail can be seen in (Rudi & Rosasco, 2017; Smale & Zhou, 2007).
Theorem 4.
Suppose ψ is continuous,
such that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) and | y | ≤ ζ .
Under Assumptions 1-2 with r ∈
[1 / 2 , 1] , γ ∈ [0 , 1] , if λ = Ω( |D| − 1 2 r + γ ) , |D 1 |
[Equation]
|D m | ,
[Equation]
then, for every δ ∈ (0 , 1] , with probability at least 1 − δ , we have
[Equation]
The bound above is the same as the original KRR estimator and is optimal in a minimax sense (Caponnetto & Vito, 2007; Lin et al., 2017; Chang et al., 2017b).
In the best case, when r = 1 and γ = 0 , the rate O (1 / |D| ) can be achieved by Ω( (cid:112) |D| ) random features and Ω( (cid:112) |D| ) partitions.
In the worst case, that is r = 1 / 2 and γ = 1 , which has been covered in Theorem 1.
Theorem 5.
Suppose ψ is continuous,
such that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) and | y | ≤ ζ .
Under Assumptions 1-2 with r ∈
[1 / 2 , 1] , γ ∈ [0 , 1] , if λ = Ω( |D| − 1 2 r + γ ) , |D 1 |
= . . .
|D m | , m (cid:46) |D| 2 r + γ − 1 4 r +2 γ and M (cid:38) |D| 1+ γ (2 r − 1) 2 r + γ , (10) then, for every δ ∈ (0 , 1] , with probability at least 1 − δ , we have
[Equation]
One can see that the upper bound of m is |D| 2 r + γ − 1 4 r +2 γ , which is stricter than that of Theorem 4.
In the best case, when r = 1 and γ = 0 , the rate O (1 / |D| ) can be achieved considering Ω( (cid:112) |D| ) random features and Ω( |D| 1 / 4 ) partitions.
For the basic setting, that is r = 1 / 2 and γ = 1 , which has been given in Theorem 2.
Theorem 6.
Suppose ψ is continuous,
such that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) and | y | ≤ ζ .
Under Assumptions 1-2 with r ∈
[1 / 2 , 1] , γ ∈ [0 , 1] , if λ = Ω(
[Equation] [Equation]
) , |D 1 |
= . .
[Equation]
then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,
[Equation]
One can see that the communication can relax the restriction on the number of partitions.
As p → ∞ , the partitions can reach Ω( |D| 2 r + γ − 1 2 r + γ ) , which is the same as Theorem 4.
Remark 5.
In this paper, we focus on enlarging the number of local machines.
In (Rudi & Rosasco, 2017; Rudi et al., 2018), they have proved that when generating random features in a data-dependent manner, fewer random features are required to obtain optimal learning.
Thus, if we adopt the data-dependent manner to generate random features, we can further improve the performance of
DKRR-RF-CM with fewer random features.
B A PPENDIX : N OTATION AND P RELIMINARY
In this paper we denote the operator norm by (cid:107) · (cid:107) , and the square integrable norm by (cid:107) · (cid:107) ρ .
Deﬁnition 1.
[Equation]
Lemma 1.
C M and C M, D are self-adjoint and positive operators, with spectrum is [0 , τ 2 ] .
Moreover we have C M = S ∗ M S M and C M, D =
Φ M, D Φ T M, D
= S ∗ M, D S M .
Proof.
C M and C M, D are self-adjoint and positive operators, with spectrum is [0 , τ 2 ] , and
C M = S ∗ M S M , C M, D = Φ M, D Φ T M, D can be directly obtained from Caponnetto & Vito (2007); Smale & Zhou (2005; 2007); Rosasco et al.
(2010); Rudi & Rosasco (2017); Lin et al.
In the following, we prove that C M, D = S ∗ M, D S M .
From the deﬁnitions of S M , S ∗ M, D and C M, D in
Deﬁnition 1, we have
[Equation]
and thus we can obtain that,
[Equation]
So, the Equation C M, D
= S ∗ M, D S M holds.
Deﬁnition
[Equation]
Remark 6 (From Caponnetto & Vito (2007); Smale & Zhou (2007); Li et al. (2019b); Lin et al.
[Equation]
Deﬁnition 3.
The maximum random feature dimension is denoted as
[Equation]
Remark 7 (From Rudi & Rosasco (2017))
[Equation] N ∞ ( λ ) ≤ τ 2 λ − 1 is always satisﬁed for every λ > 0 .
C A PPENDIX : P ROOF OF T HEOREMC. A PPENDIX : E RROR DECOMPOSITION FOR DKRR-RF IN E XPECTATION
Proposition 1.
Let ¯ f 0 M, D ,λ
[Equation]
and f (cid:5) M, D ,λ , f M,λ , f λ be deﬁned by 3 and 12, respectively.
Then, we have
[Equation]
Proof.
Let ¯ f 0 M, D ,λ , and f (cid:5) M, D ,λ , f M,λ be deﬁned by 3 and 12, respectively.
According to the
Proposition 5 of (Chang et al., 2017b) or Lemma 20 of (Li et al., 2019b), we have
[Equation]
Note that ( a + b + c ) 2 ≤ 3 a 2 + 3 b 2 + 3 c 2 , ∀ a, b, c ≥ 0 .
Thus, we have
[Equation]
Combining the above inequality and Eq.
[Equation]
we can prove this proposition.
Proposition 2.
The follows hold: √
[Equation]
and
[Equation]
M, D ,λ − f (cid:5) M, D ,λ (cid:107) ρ ≤ J 2 M, D ( R M, D + K M, D ) .
where R M, D :=
(cid:13)(cid:13)
( C M + λI ) − 1 / 2 ( Φ M, D ¯ y D − S ∗ M f ρ ) (cid:13)(cid:13) , J M, D :
= (cid:107) ( C M, D + λI ) − 1 / 2 ( C M + λI ) 1 / 2 (cid:107) , K M, D := (cid:107) ( C M + λI ) − 1 / 2 ( S ∗ M f ρ − S ∗ M, D f ρ ) (cid:107) .
Proof.
From 13, we know that w M, D ,λ
=
( C M, D + λI ) − 1 Φ M, D ¯ y D and w (cid:5) M, D ,λ
= ( C M, D + λI ) − 1 S ∗ M, D f ρ , so we have
[Equation]
Note ( C M, D + λI ) − 1 / 2 is a self-adjoint and positive operator, so (cid:107)
[Equation]
≤
[Equation]
[Equation]
, thus, we can obtain that
[Equation]
Note that f M, D ,λ − f (cid:5) M, D ,λ = S M ( w M, D ,λ − w (cid:5) M, D ,λ ) , by 15, we have
[Equation]
Note that
[Equation]
So, by Eq
[Equation]
we have
[Equation]
Proposition 3.
The follows hold: √
[Equation]
and
[Equation]
where J M, D :=
[Equation]
Proof.
By Remark
[Equation]
we have
[Equation]
Note that for any self-adjoint and positive operators A and B ,
[Equation]
so we have
[Equation]
From Lemma
[Equation]
we know that C M = S ∗ M S M and C M, D =
Φ M, D Φ T M, D
= S ∗ M, D S M , thus we can obtain that
[Equation]
Thus, we have
[Equation] [Equation]
Note that
[Equation]
and
[Equation]
since (cid:107) ( C M + λI )
[Equation]
S ∗ M
[Equation]
=
[Equation]
≤ 1 .
Substituting the above two inequalities into Eq.
20, we have
[Equation]
which prove the ﬁrst result of Proposition 3.
In the following, we will prove the second result.
By Eq.
19, we have
[Equation]
Note that (cid:107) S M ( C M + λI
[Equation]
−
[Equation]
λI
[Equation]
2 (cid:107)
[Equation]
≤ 1 , (cid:107) ( C M
[Equation]
D + λI )
[Equation] [Equation]
(cid:107)
[Equation]
λI
[Equation]
M
[Equation]
M
[Equation]
D + λI ) − 1 / 2 (cid:107) 1 /≤ 1 , and (cid:107) ( C M + λI ) − 1 / 2 S ∗ M (cid:107) =
(cid:107) ( C M + λI ) −
[Equation]
M (
[Equation]
M
[Equation]
λI ) − 1 / 2 (cid:107) 1 / 2 ≤ 1 , so we have
[Equation]
Proposition 4.
The follows hold:
[Equation]
, and
[Equation]
where R M, D :=
(cid:13)(cid:13)
[Equation]
C M + λI
[Equation]
−
[Equation] [Equation]
Φ
[Equation]
D ¯ y D
[Equation]
S
[Equation]
f
[Equation]
= (cid:107)
[Equation]
D
[Equation] [Equation]
C
[Equation]
λI ) 1 / 2 (cid:107) , K M, D := (cid:107) ( C M + λI ) − 1 / 2 ( S ∗ M f ρ − S ∗ M
[Equation]
D f ρ ) (cid:107) .
Proof.
Note that
[Equation]
and
[Equation]
Combining Propositions 2 and 3, we can prove this result.
C. P ROOF OF T HEOREM
Lemma 2 (Lemma 23 in (Li et al., 2019b), can be also seen in (Rudi & Rosasco, 2017)) .
For δ ∈ (0 , 1] and λ > 0 , when
[Equation]
Then, with probability at least 1 − δ
[Equation]
we have
[Equation]
where c is a constant.
Lemma 3 (Theorem 4 in (Smale & Zhou, 2005)) .
Under Assumption 2, for r ∈ [1 / 2 , 1] , we have
[Equation]
where c is a constant.
Lemma 4 (Lemma 6 in (Rudi & Rosasco, 2017)) .
For δ ∈ (0 , 1] , with probability at least 1 − δ , we have
[Equation]
where N M ( λ ) :
=
Tr (cid:0) ( L M + λI ) − 1 L M (cid:1) , L M is the integral operator associated with the approximate kernel function K
[Equation]
( x (cid:48) ) .
Lemma 5 (Proposition 10 in (Rudi & Rosasco, 2017)) .
For any δ ∈ (0 , 1] , M ≥ Ω (cid:0) N ∞ ( λ ) log 1 λδ (cid:1) , then with probability at least 1 − δ ,
[Equation]
) where N M ( λ ) :
=
Tr (cid:0) ( L M + λI ) − 1 L M (cid:1) .
Lemma 6 (Lemma E.2, (Blanchard & Kr ¨ amer, 2010)) .
For any self-adjoint and positive semideﬁnite operators A and B , if there exists η > 0 such that the following inequality holds
[Equation]
then
[Equation]
Proposition 5.
For δ ∈ (0 , 1] , with probability at least 1 − δ , we have
[Equation]
, where N M ( λ ) :
=
Tr (cid:0) ( L M + λI ) − 1 L M (cid:1) .
The proof closely follows the proof of Lemma 6 in (Rudi & Rosasco, 2017).
Proof.
Deﬁne µ i =
( C M +
[Equation]
f ρ − ( C M + λI ) 1 / 2 φ M ( x i )
f
ρ ( x i ) .
Note that
[Equation]
µ i .
Since µ 1 , . . . , µ |D| are independent and identically distributed random vector, and
[Equation]
To apply the Bernstein inequality (Arcones, 1995; Rudi & Rosasco, 2017) for random vectors, we need to bound their moments.
Note that
[Equation]
and
[Equation]
Thus, using the Bernstein inequality (Arcones, 1995; Rudi & Rosasco, 2017), for any δ ∈ (0 , 1] , with 1 − δ , we have
[Equation]
Lemma 7 (Lemma 2 in (Smale & Zhou, 2007)) .
Let H be a Hilbert space and ξ be a random variable on Z , ρ with values in H .
Assume (cid:107) ξ (cid:107) ≤ M < ∞ almost surely.
Denote σ ( ξ ) =
E ( ξ 2 ) .
Let { z i } ni
=1 be independent random drawers of ρ .
For any 0 < δ < 1 , with conﬁdence 1 − δ ,
[Equation]
Proposition 6.
For any δ > 0 , with probability at least 1 − δ , we have
[Equation]
and
[Equation]
where N ∞ ( λ ) = sup ω ∈ Ω (cid:107) ( L K + λI ) − 1 / 2 ψ ( · , ω ) (cid:107) 2 ρ , c 1 and c 2 are two constants.
To prove Proposition 6, we ﬁrst prove the following lemma (a similar technique can be found in (Hsu et al., 2012; Rudi et al., 2013; Caponnetto & Yao, 2006; Rudi & Rosasco, 2017)).
The similar result for the matrix case was ﬁrst proved in (Hsu et al., 2012), and later was extended to the operator case in (Rudi et al., 2013; Rudi & Rosasco, 2017).
Lemma 8.
Let ζ 1 , . .
, ζ n with n ≥ 1 , be i.i.d random vectors on a separable Hilbert spaces
H such that H =
E ζ ⊗ ζ is a trace class, and for any λ there exists N ∞ ( λ ) <
∞ such that (cid:104) ζ, ( H + λI ) − 1 ζ (cid:105) ≤ N ∞ ( λ ) .
Denote H n as 1 n (cid:80) ni
=1 ζ i ⊗ ζ i .
Then for any δ ≥ 0 , with probability at least 1 − 2 δ , the following holds
[Equation]
N
[Equation] [Equation]
[Equation]
n .
Proof.
Let H λ
= H + λI , η =
H − 1 / 2 λ
HH
[Equation] [Equation]
2 λ , ξ i = η −
[Equation]
−
[Equation]
λ ζ i ⊗ H − 1 / 2 λ ζ i .
One can see that E ξ i = 0 .
Note that
[Equation]
and
[Equation]
Thus, substituting the above two inequalities to Lemma 7 (Lemma 2 in (Smale & Zhou, 2007)), which ﬁnishes the proof.
Proof of Proposition 6.
Since C M is self-adjoint operator, so we have
[Equation]
According to Lemma 8 with υ i =
φ M ( x i ) , we can obtain this result.
Proposition 7.
If |D| ≥ 32 log(2 /δ )
(1+ N ∞ ( λ )) , then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Proof.
From Proposition 6, we know that if |D| ≥ 32 log(2 /δ )
(1 + N ∞ ( λ )) , then
[Equation]
Combining the above inequality and Lemma 6, we can prove this result.
Proposition 8.
If δ ∈ (0 , 1] , and |D| ≥ Ω( N ∞ ( λ )) , then with 1 − δ , we have
[Equation]
where Υ M, D ,λ :=
[Equation]
√ λ |
[Equation]
|
[Equation]
N
[Equation]
|
[Equation]
| .
Proof.
From Proposition 4, we have
[Equation]
Thus, from Lemmas 4, 5, and Propositions 5, 7, we know that if |D| ≥ Ω( N ∞ ( λ )) , we can prove this result.
Proof of Theorem 4 .
According to Proposition 1, we have
[Equation]
Substituting Lemmas 2,3, Proposition 3, 7, 8 into the above inequality, one can see that if
M ≥ Ω
[Equation]
N 2 r
[Equation]
1 ( λ ) λ
[Equation]
, and
|D j | ≥ Ω( N ∞ ( λ )) , with conﬁdence 1 − δ , we have
[Equation]
where Υ M, D
[Equation]
[Equation]
=
[Equation]
j | + (cid:113) N ( λ ) |D j | .
If setting
[Equation]
D
[Equation] [Equation]
) , we have
[Equation]
Note that if m ≤ Ω (cid:16) |D| 2 r + γ − 1 2 r + γ (cid:17) and |D 1 | = . . .
=
|D m | , and λ = Ω( |D| − 1 2 r + γ ) , we have
[Equation]
Thus, substituting 24 into 23, one can see that if m ≤ Ω (cid:16) |D| 2 r + γ − 1 2 r + γ (cid:17) and
M ≥ Ω (cid:18)(cid:18) N
[Equation]
( λ
[Equation]
λ
[Equation] [Equation]
, with probability at least 1 − δ , we have
[Equation]
D A PPENDIX : P ROOF OF T HEOREMD. A PPENDIX : E RROR D ECOMPOSITION FOR DKRR-RF IN P ROBABILITY
Proposition 9.
The follows hold:
[Equation]
and (cid:107)
[Equation]
where J M, D := (cid:107) ( C M, D + λI ) − 1 / 2 ( C M + λI ) 1 / 2 (cid:107) and Q M, D := (cid:107) ( C M + λI ) − 1 / 2 ( C M −
C M, D )( C M + λ ) − 1 / 2 (cid:107) .
Proof.
Note that w M, D ,λ =
( C M, D + λI ) − 1 Φ M, D ¯ y D , thus we have
[Equation]
Note that m (cid:88)
[Equation]
and m
[Equation]
Substituting the above equations into Eq.
[Equation]
we have
[Equation]
≤
[Equation]
where L M, D := (cid:107) ( C M + λI ) − 1 ( C M − C M, D ) (cid:107) .
From Proposition 6, we know that L M, D = Q M, D , so we have
[Equation]
which prove the ﬁrst result of this proposition.
In the following, we will prove the second result of this proposition.
Note that S M ( ¯ w 0 M, D ,λ − w M, D ,λ ) =
¯ f 0 M, D ,λ − f M, D ,λ .
According to 25, we have
[Equation]
Note that
[Equation]
(27) thus we have (cid:107)
[Equation]
Since (cid:107) S M ( C M + λI ) − 1 / 2 (cid:107) =
(cid:107) ( C M + λI ) − 1 / 2 C M ( C M + λI ) − 1 / 2 (cid:107) 1 / 2 ≤ 1 .
So, we have (cid:107)
[Equation]
the last inequality uses the fact that
[Equation]
Similar as the above process
[Equation]
we can also obtain that
[Equation]
Thus
[Equation]
using
[Equation]
we can prove this result
[Equation]
D. A PPENDIX : P ROOF OF T HEOREM
Proof of Theorem 5 .
Combining Proposition 9 and Proposition
[Equation]
we have
[Equation]
From Propositions 7 and 8, one can see that if |D j | ≥ Ω( N ∞ ( λ )) , and λ ≤ (cid:107) L K (cid:107) , then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
where Υ M, D i ,λ =
[Equation]
λ ) |D i | .
Note that Q M, D ≤ Q M, D j , and by Lemmas 2 and 3, so we have
[Equation]
Note that
[Equation]
Combining E.q 30, Lemmas 2, 3, Proposition 8 and E.q 31, one can see that if M ≥ Ω (cid:16)(cid:16) N
[Equation]
λ )
[Equation]
r ∨ N ∞ ( λ ) [Equation] , with probability 1 − δ , we have
[Equation]
If |D 1 |
= . . .
=
|D m | , λ = Ω( |D|
[Equation]
1 2 r + γ )
[Equation]
we have
[Equation]
Thus
[Equation]
when m ≤ Ω (cid:16) |D| 2 r + γ
[Equation]
1 4 r +
[Equation]
γ (cid:17)
[Equation]
we have
[Equation] [Equation]
(cid:16)
[Equation] [Equation]
Thus, we have (cid:107) ¯ f 0 M, D ,λ − f ρ (cid:107) 2 ρ
=
O (cid:16) |D| − 2 r 2 r + γ log 2 1 δ (cid:17) , which prove this result.
E A PPENDIX : P ROOF OF T HEOREME.
A PPENDIX :
E RROR D ECOMPOSITION F OR DKRR-RF-CM IN P ROBABILITY
Proposition 10.
[Equation]
where J M, D := (cid:107) ( C M, D + λI ) − 1 / 2 ( C M + λI ) 1 / 2 (cid:107) and Q M, D := (cid:107) ( C M + λI ) − 1 / 2 ( C M −
C M, D )( C M + λ ) − 1 / 2 (cid:107) .
Proof.
Note that
[Equation]
Thus
[Equation]
we have
[Equation]
Note that
[Equation]
Note that (cid:107) S M ( C M + λI ) − 1 / 2 (cid:107) =
(cid:107) ( C M + λI ) − 1 / 2 C M ( C M + λI ) − 1 / 2 (cid:107) 1 / 2 ≤ 1 , so, we have (cid:107) S M ℵ j 1 (cid:107) ρ ≤ J
[Equation]
Note that C M = S ∗ M S M , so
[Equation]
Substituting the above inequality into Eq.
[Equation]
we have
[Equation]
the last inequality use the fact that
[Equation]
λI )
[Equation]
(cid:107)
[Equation]
≤ 1 .
Using the same process
[Equation]
we can obtain that
[Equation]
Thus
[Equation]
we have
[Equation]
According to
[Equation]
we know that
[Equation]
Thus, one can obtain that
[Equation]
where L M, D := (cid:107) ( C M + λI ) − 1 ( C M − C M, D ) (cid:107) .
From Proposition
[Equation]
we know that L M, D = Q M, D .
Thus, we have
[Equation]
Combining 37 and 38, we have
[Equation]
which prove the result
[Equation]
E. A PPENDIX : P ROOF OF T HEOREM
Proof of Theorem 6 .
Substituting Propositions 4, 5, 7, 8 and Lemma 4 into Proposition 9, we have
[Equation]
Combining the above inequality and Proposition
[Equation]
and note that Q M
[Equation]
D ≤ Q M
[Equation]
D j
[Equation]
we can obtain that
[Equation]
Note that
[Equation]
Thus
[Equation]
by Lemmas
[Equation] [Equation]
one can see that if
[Equation] [Equation]
we have
[Equation]
From 33, by setting |D 1 | = . . . = |D
[Equation] | , λ [Equation]
Ω [Equation] |D| −
[Equation]
γ ) , we know that
[Equation]
Thus, when m ≤ Ω (cid:16) |D| (2 r + γ − 1)( t +1) (2 r + γ )( t +2) (cid:17) , we have (cid:107) ¯ f tM, D ,λ − f ρ (cid:107) ρ
=
O (cid:16) |D| − r 2 r + γ (cid:17) , which proves the result.
F A PPENDIX : P ROOF OF T HEOREMS 1, 2,
Proof.
From (Smale & Zhou, 2007; Caponnetto & Vito, 2007), if r = 1 / 2 , then f ρ ∈ H K .
Thus, in this case, f H K exists and E ( f H K ) =
E ( f ρ ) .
Note that Assumption 1 is always satisﬁed for γ =and c = τ 2 .
So, using Theorems 4, 5 and 6 with r = 1 / 2 , γ = 1 and c = τ 2 , Theorem 1, 2, andcan be proved.
[Figure]
G A PPENDIX : M ORE E XPERIMENTS
In this section, we add more experiments on different data-sets to further exemplify the performance of the algorithm.
We use 5 publicly available datasets from LIBSVM Data 2 .
The empirical evaluations with Gaussian kernel, exp( −(cid:107) x − x (cid:48) (cid:107) 2 /σ ) , are given in Figure 2, where the optimal σ and λ are selected by 5-fold cross-validation, σ ∈ { 2 i , i = − 10 , − 8 , . . .
, 10 } , { 2 − 5 , 2 − 3 , . . .
2 5 }|D| − 1 / 2 , and the number of random features is 2 (cid:112) |D| .
From Figure 2, one can ﬁnd that: 1) our DKRR-RF-CM are better than the original DKRR-RF on all data-sets; 2) the larger the iterations of the communication, the better the performance.
The above results demonstrate that our communication-based DKRR-RF is effective.