D ISTRIBUTED L EAST S QUARE R ANKING WITH R ANDOM F EATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we study the statistical properties of pairwise ranking using distributed learning and random features (called DRank-RF) and establish its convergence analysis in probability.
Theoretical analysis shows that DRank-RF remarkably reduces the computational requirements while preserving a satisfactory convergence rate.
An extensive experiment veriﬁes the effectiveness of DRankRF.
Furthermore, to improve the learning performance of DRank-RF, we propose an effective communication strategy for it and demonstrate the power of communications via theoretical assessments and numerical experiments.
I NTRODUCTION
Distributed learning has attracted much attention in the literature and has been widely used for kernel learning in large scale scenarios (Zhang et al., 2013; Chang et al., 2017; Lin et al., 2020b).
The distributed kernel learning has mainly three ingredients: Processing the data subset in the local kernel machines and producing a local estimator; Communicating exclusive information such as the data (Bellet et al., 2015), gradients (Zeng & Yin, 2018) and local estimator (Huang & Huo, 2019) between the local processors and the global processor; Synthesizing the local estimators and the communicated information on the global processor to produce a global estimator.
Note that, in the divide-and-conquer learning, the second ingredient communications is not necessary.
In the terms of practical challenges and theoretical analysis, the distributed learning has made signiﬁcant breakthroughs in the multi-penalty regularization (Guo et al., 2019), coefﬁcient-based regularization (Pang & Sun, 2018), spectral algorithms (M ¨ ucke & Blanchard, 2018; Lin et al., 2020a), kernel ridgel regression (Yin et al., 2020; 2021), and semi-supervised regression (Li et al., 2022).
All the above are restricted to pointwise kernel learning.
However, the distributed learning in pairwise kernel learning still has a long way to go.
The existing distributed pairwise learning (Chen et al., 2019; 2021) has high computational requirements, which motivates us to explore theoretic foundations and efﬁcient methods for pairwise ranking kernel methods under distributed learning.
Random features methods (Rahimi & Recht, 2007; Carratino et al., 2018; Liu et al., 2021) have a long and distinguished history, which embed the non-linear feature space (i.e. the Reproducing Kernel Hilbert Space associated with the kernel) into a low dimensional Euclidean space while incurring an arbitrarily small additive distortion in the inner product values.
This enables one to overcome the high computational requirements of kernel learning since one can now work in an explicit low dimensional space with explicit representation whose complexity depends only on the dimensionality of the space.
Random features have gained rapid progress in reducing the complexity of the kernel ridge regression (Liu et al., 2021) and semi-supervised regression (Li et al., 2022).However, it remains unclear for complexity reduction and learning theory analysis to the distributed pairwise ranking kernel learning.
In this paper, to reduce the computational requirements of pairwise ranking kernel learning, we investigate the method of combining distributed learning and random features for pairwise ranking kernel learning, called distributed least square ranking with random features (DRank-RF), to deal with large-scale applications, and study its statistical properties in probability by integral operators framework.
To further improve the performance of DRank-RF, we consider communications among different local processors.
The main contributions of this paper are as follows: 1) We construct a novel method DRank-RF to improve the existing state-of-the-art performance of the distributed pairwise ranking kernel learning.
This work is the ﬁrst time to apply random features to least square ranking and derive the theoretical guarantees, which is a new exploration of random features in least square ranking.
In theoretical analysis, we derive the convergence rate of the proposed method, which is sharper than that of the existing state-of-the-art distributed pairwise ranking kernel learning (See Theorem 1).
In computational complexity, DRank-RF requires essentially O ( m 2 | D j | ) time and O ( m | D j | ) memory, where m is the number of random features, m < | D j | , and | D j | is the number of data in each local processor.
The proposed method can greatly reduce the computational requirements compared with the state-of-the-art works (See Table 1).
Experimental results verify that the proposed method keeps the similar testing error as the exact and state-of-the-art approximate kernel least square ranking and has a great advantage over the exact and state-of-the-art approximate kernel least square ranking in the training time, which is consistent with our theoretical analysis. 2)
We propose a communication strategy to further improve the performance of DRank-RF, called
DRank-RF-C. Statistical analysis shows that DRank-RF-C obtains a faster convergence rate with the help of communication strategy than DRank-RF.
And the numerical results validate the power of the proposed communication strategy.
The paper is organized as follows:
In section 2, we brieﬂy introduce the least square ranking problem and the distributed least square ranking.
In section 3, we introduce the proposed methods.
Section 4 shows the theoretical analysis of the proposed DRank-RF and DRank-RF-C.
In section 5, we compare the related works with the proposed methods.
The following sections are the experiments and conclusions.
B ACKGROUND
There is a compact metric space Z :=
( X , Y ) ⊂
R q +1 , where X ⊂ R q and Y ⊂ [ − b, b ] for some positive constant b .
The sample set D := { ( x i , y i ) } Ni =1 of size N = | D | is drawn independently from an intrinsic Borel probability measure ρ on Z .
ρ ( y | X = x ) denotes the conditional distribution for given input x .
The hypothesis space used is the reproducing kernel Hilbert space (RKHS) ( H K ) associated with a mercer kernel K : X × X
→ R (Aronszajn, 1950).
We will denote the inner product in H K by (cid:104)· , ·(cid:105) , and corresponding norm by (cid:107) · (cid:107) K .
2. L EAST S QUARE R ANKING (LSR ANK )
Least square ranking (LSRank) is one of the most popular learning methods in the machine learning community (Chen, 2012;
Zhao et al., 2017; Chen et al., 2019),which can be stated as f D,λ = arg min f ∈H K (cid:8) E D ( f ) + λ (cid:107)
f
(cid:107) 2 K (cid:9) and E D ( f ) =
[Equation]
where the regularized parameter λ > 0 .
The main purpose of LSRank is to ﬁnd a function f : X → R through empirical observation, so that the ranking risk
[Equation]
can be as small as possible, where x , x (cid:48) ∈ X .
The optimal predictor (Chen, 2012; Chen et al., 2013; Kriukova et al., 2016) under Eq.(1) is the regression function f ρ ( x ) =
(cid:82) Y ydρ ( y | X = x ) , x ∈ X .
Complexity Analysis LSRank requires O ( | D | 3 ) time and O ( | D | 2 ) space, which is prohibitive for large-scale settings.
2. D ISTRIBUTED L EAST S QUARE R ANKING (DR ANK )
Let the dataset D
= ∪ pj =1 D j and each subset D j :=
(cid:110)(cid:16) x ji , y jj (cid:17)(cid:111)
| D j | i =1 be stored in the j -th local processor for 1 ≤ j ≤ p .
The DRank is deﬁned by
[Equation]
where the least squares ranking (LSRank)
f D j ,λ = arg min f ∈H K (cid:8) E D j
( f ) + λ (cid:107) f (cid:107) 2 K (cid:9) and
E D j ( f ) =
[Equation]
[Equation]
Complexity Analysis The time complexity, space complexity, and communication complexity of
DRank for each local processor are O
( | D j | 3 ) , O ( | D j | 2 ) , and O ( | D j | ) , where j = 1 , . .
. , p and p is the number of partitions.
P ROPOSED A LGORITHMS 3. D ISTRIBUTED L EAST S QUARE R ANKING WITH R ANDOM F EATURES (DR ANK -RF)
Here we ﬁrst introduce the main properties of the shift-invariant kernel and the basic idea of random features.
The shift-invariant kernel can be written as K ( x , x (cid:48) )
= (cid:82) Ω ψ ( x , ω ) ψ ( x (cid:48) , ω ) (cid:37) ( ω ) d ω if the spectral measure has a density function (cid:37) ( · ) (Li et al., 2019; Carratino et al., 2018), where ψ :
X × Ω → R is a bounded and continuous function with respect to ω and x .
The basic idea of random features is to approximate the kernel function K ( x , x (cid:48) ) by its Monte-Carlo estimation (Li et al., 2019; Rahimi & Recht, 2007): K m ( x , x (cid:48) )
= 1 m (cid:80) m i =1 ψ ( x , ω i ) ψ
( x (cid:48) , ω i ) =
(cid:104)
φ m ( x ) , φ m ( x (cid:48) )
(cid:105) , where φ m ( x ) = 1 √ m ( ψ ( x , ω 1 ) , . . .
, ψ ( x , ω m )) T .
Back to supervised learning (Chen, 2012), combining random features with the least squares ranking leads to, f m,D,λ ( x ) =
g Tm,D,λ φ m ( x ) with
[Equation]
(3) for Φ m,D = 1 √ | D | ( φ m ( x 1 ) , . .
, φ m ( x | D | )) , W D
= I | D | − 1 | D | 1 | D | 1 T | D | = 1 | D | ( | D | I − 1 | D | 1 T | D | ) , the identity matrix
I | D | , 1 | D | = (1 , . .
, 1) T ∈ R | D | , and ¯y D = 1 √ | D | ( y 1 , . .
, y | D | ) T .
DRank with random features (DRank-RF) is deﬁned as
[Equation]
(4) where f m,D j ,λ = g Tm,D j ,λ φ m ( x ) with g m,D j ,λ = ( Φ m,D j W D
j Φ Tm,D j + λ 2 I ) − 1 Φ m,D j W D
j ¯y D j .
Random features have a long history and have been studied in different learning, for example kernel ridge regression (Liu et al., 2021), kernel classiﬁcation (Liu et al., 2022), kernel k-means (Chitta et al., 2012).
However, random features have not been studied in least square ranking.
Our work is the ﬁrst time to apply random features to least square ranking and derive the theoretical guarantees, which is a new exploration of the application of random features.
In addition, due to the different objective functions and integral operators, the proof of our proposed method is different from the existing methods (See Appendix).
Finally, the proposed methods greatly reduce the computational requirements (See Table 1).
The method of synthesis operation in Eq.(4) is to weighted average the estimated values in each local processor.
Complexity Analysis In time complexity, solving the inverse of Φ m,D j W D j Φ Tm,D j + λ 2 I needs
O ( m 3 ) time and computing the matrices multiplication Φ m,D j W D j Φ Tm,D j requires O ( m 2 | D j | ) cost, where m is the number of random features.
In space complexity, the key is to store Φ m,D j , whose space complexity is O ( m | D j | ) .
Therefore, the time complexity, space complexity, and communication complexity of DRank-RF for each local processor are O ( m 2 | D j | ) , O ( m | D j | ) , and
O ( m ) , where m < | D j | .
Not that, the computational cost of random features model is far less than m 2 | D j | .
It is ignored when expressing the computational complexity.
In the experiments, the training time of our methods includes the time of calculating the random features model.
The way of weighted averaging in Eq.(4) cannot improve the approximation ability of DRank-RF in each local processor (Huang & Huo, 2019; Lin et al., 2020b; Yin et al., 2021).
To further improve the performance, we bring an efﬁcient communication strategy into DRank-RF.
Initialize : ¯ g 0 m,D,λ =
For l = 1 to M do
Local processor : compute the local gradient G m,D j ,λ
(¯ g l − 1 m,D,λ ) and communicate back to the global processor.
Global processor : compute ¯ G m,D,λ (¯ g l − 1 m,D,λ ) =
(cid:80) pj =
| D j | 2 (cid:80) pk =1 | D k | 2 G m,D j ,λ
(¯ g l − 1 m,D,λ ) in Eq.(9) and communicate to each local processor.
Local processor : compute β l − 1 j in Eq.(8) and communicate back to the global processor.
Global processor : compute ¯ g lm,D,λ in Eq.(7), and communicate to each local processor.
End For
Output : ¯ g Mm,D,λ and ¯ f Mm,D,λ = (cid:104)
¯ g Mm,D,λ , φ m ( · ) (cid:105)
3. DR ANK -RF WITH C OMMUNICATIONS (DR ANK -RF-C)
In this section, we introduce the DRank-RF with communications (DRank-RF-C), which can not only improve the approximation ability but also protect the data privacy in each local processor.
For any g , according to Eq.(3), one has the following equation:
[Equation]
(5) where G m,D,λ ( g ) =
( Φ m,D W D Φ Tm,D + λ 2 I ) g − Φ m,D W D ¯y D .
Deﬁne ¯ g 0 m,D,λ = [Equation] |
[Equation]
g m,D j
[Equation]
, we can obtain that
[Equation]
Note that, the gradient of the empirical risk of
[Equation] λ
[-(cid:1) 
-][Equation]
g (cid:107) 2 on g is 4 G m,D j ,λ ( g ) for all ( x i , y i ) , ( x k , y k ) ∈
D j .
Comparing Eq.(5) and Eq.(6), we consider the communication strategy based on the well-known
Newton Raphson iteration (Lin et al., 2020b; Yin et al., 2021; Chen et al., 2021) for DRank-RF, which is formed as:
[Equation]
where
[Equation]
H
[Equation]
and l is the number of iteration.
The method of synthesis operation in DRank-RF-C is to weighted average the model parameters { β j } of each local processor obtained in the last iteration.
Algorithm 1 shows the detail of DRank-RF-C. In step 1, let ¯ g 0 m,D,λ be 0 .
In the following steps, we update the global gradients and model parameters iteratively.
For l = 1 , . . .
, M , we distribute ¯ g l − 1 m,D,λ to each local processor.
In step 2 (on each local processor), compute p local gradient vectors G m,D j ,λ
( ¯ g l − 1 m,D,λ ) and communicate them back to the global processor.
In step 3 (on global processor), according to the received p local gradient vectors, we compute the global gradient ¯
G m,D,λ
(¯ g l − 1 m,D,λ ) and communicate it to each local processor.
In step 4 (on each local processor), each local processor computes β l − 1 j and communicates them back to the global processor.
In step(on global processor), the global processor obtains the solution ¯ g lm,D,λ .
Then we transmit ¯ g lm,D,λ to each local processor and go back to step 2.
Complexity Analysis
In the terms of time complexity, one needs to compute the inverse of Φ m,D j W D
j Φ Tm,D j + λ 2 I and the matrices multiplication Φ m,D j W D
j Φ Tm,D j once for each local processor, and one needs to solve the local gradient G m,D j ,λ and model parameter β j in each iteration for each local processor.
Thus, the total time complexity for each local processor is
O ( m 2 | D j | + mM | D j | ) , where M is the number of communications.
In the terms of space complexity, for each local processor, the key is to store Φ m,D j , thus the space complexity of each local processor is O ( m | D j | ) .
In the terms of communication complexity, the global processor sends the gradient ¯ G m,D,λ and ¯ g lm,D,λ to each local processor and receives the local gradient G m,D j ,λ and model parameter β j from each local processor in each iteration.
Therefore, the total communication complexity is O ( mM ) .
Note that, if the number of communications M ≤ m , the time complexity and space complexity of DRank-RF-C are the same as those of DRank-RF.
T HEORETICAL A NALYSIS
Here, we analyze the convergence rate of DRank-RF and DRank-RF-C in probability.
Deﬁne the optimal hypothesis f λ in H K as f λ = arg min f ∈H K (cid:8) E ( f ) + λ (cid:107) f (cid:107) 2 K (cid:9) .
We assume f λ exists.
4. C ONVERGENCE RATE OF DR ANK -RF
In the following, we state and discuss the convergence rate of DRank-RF in probability.
Theorem 1.
Suppose ψ is continuous,
such that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) .
Assume that L − rK f ρ ∈ H K with 0 < r ≤ 1 , where L rK is the r -th power of L K .
If the regularization parameter λ
= O (cid:18)(cid:16)(cid:80)
[Equation]
(cid:19) and the number of random features m = Ω (cid:18)(cid:16)(cid:80) [Equation] [Equation] r
[Equation]
, for ¯ f 0 m,D,λ deﬁned in Eq.(4) and every δ ∈ (0 , 1] , there holds
[Equation]
with conﬁdence at least 1 − δ .
Remark 1.
From Theorem 1 mentioned above, one can see that if the number of random features m is Ω (cid:18)(cid:16)(cid:80) pj =
[Equation]
[Equation]
− 2 r 1+ r (cid:19) , the convergence rate of the proposed DRank-RF can reach O (cid:18)(cid:16)(cid:80) pj =
[Equation]
r 1+ r (cid:19) 1 , which is sharper than the existing convergence rate
O (cid:18)(cid:16)(cid:80) pj =
| D j | 3 / 2 (cid:80) pk =1 | D k |(cid:17)
r 1+ r
(cid:19) of the state-of-the-art distributed pairwise ranking kernel learning (Chen et al., 2021).
When the number of partitions p = 1 , the convergence rate of the proposed
DRank-RF is O (cid:16) | D | − r 1+ r (cid:17) with m = Ω (cid:16) | D | 2 r 1+ r (cid:17) .
When | D 1 |
= . . .
=
| D p | , the convergence rate of the proposed DRank-RF is O (cid:16) ( | D | p ) − r 1+ r (cid:17) with m = Ω (cid:16) ( | D | p )
r 1+ r (cid:17) .
Theoretical analysis demonstrates that the proposed DRank-RF is sound and effective.
Remark 2.
From a theoretical perspective, this paper is a non-trivial extension of these approximate pairwise ranking methods.
The existing papers mainly use capacity concentration estimation (Rudin, 2009; Rudin & Schapire, 2009; Rejchel, 2012) and algorithmic stability (Cossock & Zhang, 2008;
Chen et al., 2014) for the learning theory analysis of pairwise ranking.
In this paper, we apply the integral operator framework and introduce a novel technique of error decomposition so that the proposed method can achieve a tight bound under the basic condition.
The details can be seen in
[Table]
| D |/
[Table]
| D j || D j || D
j |
[Table]
| D
[Equation]
|
[Equation]
D | | D
[Equation]
|qM | D |
[Table] [Table]
m
[Equation]
D
[Equation]
| m
[Equation]
D
[Equation]
| m
[Table]
Paper
[Table]
m
[Equation]
mM
[Equation]
| m
[Equation]
| mM
Appendix.
This is the ﬁrst time that combined distributed learning and random features in LSRank and achieved such a breakthrough.
4. C ONVERGENCE RATE OF DR ANK -RF-C
Here, we introduce and discuss the convergence analysis of DRank-RF-C in probability.
Theorem 2.
Suppose ψ is continuous,
such that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) .
Assume that L − rK f ρ ∈ H K with 0 < r ≤ 1 , where L rK is the r -th power of L K .
If λ = O ( | D | − 11+ r ) , | D 1 | = . . .
=
| D p | = | D | p , and the number of random features m = Ω (cid:16) | D | 2 r 1+ r (cid:17) , for every δ ∈ (0 , 1] , with conﬁdence at least 1 − δ , we have (cid:13)(cid:13)(cid:13)
¯ f Mm,D,λ − f ρ (cid:13)(cid:13)(cid:13) K
= O (cid:18)(cid:16) p 12 | D | − r 2(1+ r ) (cid:17) M +
(cid:19) .
Proof.
The proof of Theorem 1 and 2 is in Appendix.
The assumption of L − rK f ρ ∈ H K with 0 < r ≤ 1 is commonly used in approximation theory (Smale & Zhou, 2007), which can be seen as regularity assumption.
Remark 3.
Theoretical analysis shows that, when p < | D | rM rM + M +2 , the convergence rate of DRankRF-C is sharper than that of DRank-RF at the same settings.
Note that p is monotonically increasing with the number of communications M , which can demonstrate the power of the proposed communications.
For M → ∞ , it is clear that the convergence rate of DRank-RF-C is always sharper than that of DRank-RF.
The convergence rate in Theorem 2 is also related to δ .
To simplify the representation, we omit it here.
Their detailed relationship is shown in Appendix C.2.
C OMPARED WITH THE R ELATED W ORKS
In this section, we introduce the related distributed pairwise ranking in kernel learning.
In Chen et al.
Chen et al.
construct the divide-and-conquer pairwise ranking in kernel learning, called
DRank.
They study the statistical properties of DRank and establish its convergence analysis in expectation.
The time complexity, space complexity, and communication complexity of DRank are
O ( | D j | 3 ) , O ( | D j | 2 ) , and O ( | D j | ) , respectively.
The convergence rate in expectation only demonstrates the average information for multiple trails but fails to capture the learning performance for a single trail.
Therefore, the probability version of the convergence rate of DRank in a single trial is proposed subsequently in Chen et al.
The statistical properties and the convergence rate of
DRank in probability are carefully analyzed and established in Chen et al.
In addition, the paper Chen et al.
(2021) proposes a communication strategy for DRank, called DRank-C, to improve the learning performance and provides its convergence rate in probability.
The time complexity and space complexity of DRank-C are O
( | D j | 3 + M | D j || D | ) and O ( | D j | 2 ) , respectively.
However, its communication strategy requires communicating the input data between each local processor.
Thus, it is difﬁcult to protect the data privacy of each local processor.
Furthermore, for each iteration, the communication complexity of each local processor is O ( qM | D | ) , where q denotes the dimension, which is infeasible in practice for large-scale datasets.
Table 1 shows the detail complexity of the related methods.
We see that the proposed DRank-RF only requires O ( m 2 | D j | ) time, O ( m | D j | ) memory, and O ( m ) communications, which are smaller
[Figure]
than other methods.
For DRank-RF-C, it requires less complexity than the communication-based method.
In addition, the communication strategy proposed in this paper only requires communicating the gradient and the model parameters, rather than the data, therefore the proposed DRank-RF-C do better on privacy protection.
The convergence rate of the proposed DRank-RF in Theorem 1 is sharper than the convergence rate O (cid:18)(cid:16)(cid:80) pj =
| D
[Equation]
pk =
[Equation]
(cid:17) r 1+ r (cid:19) of the existing state-of-the-art DRank (without communications) (Chen et al., 2021; 2019).
And the convergence rate of the proposed DRank-RF-C in Theorem 2 is also sharper than the convergence rate O (cid:18) max (cid:26)(cid:16) p 12 | D | − r 2(1+ r ) (cid:17) M +1 , | D | − r 2(1+ r ) (cid:27)(cid:19) of the existing communication-based DRank (Chen et al., 2021).
E MPIRICAL E VALUATIONS
We perform experiments to validate our theoretical analysis of DRank-RF and the communication strategy on simulated and real datasets.
The server is 32 cores (2.40GHz) and 32 GB of RAM.
P ARAMETERS AND C RITERION
We use the Gaussian kernel K ( x , x (cid:48) )
=
exp (cid:16) − (cid:107) x − x (cid:48) (cid:107) 22 / (2 d 2 ) (cid:17) .
The optimal bandwidth d ∈ 2 [ − 2:0 . 5:5] and regularization parameter λ ∈ 2 [ − 13:2: − 3] are selected via 5-fold cross-validation.
The criterion of evaluating the methods on testing data is as follows (Chen et al., 2021;
Kriukova et al., 2016):
R
[Equation]
, where I { ϕ } is 1 if ϕ is true and 0 otherwise.
We use the exact LSRank as a baseline, which trains all samples in a batch.
And we compare the proposed DRank-RF and DRank-RF-C ( M = 2 , 4 , 8 ) with DRank, DRank-C, and LSRank by carrying out various settings.
We repeat the training 5 times and estimate the error on testing data.
6. S IMULATED E XPERIMENTS
Inspired by the numerical experiments in Chen et al.
; Kriukova et al.
, we consider the following way to generate the synthetic data.
The inputs { x i } | D (cid:48) | i =1 ∈ R | D (cid:48) |× q are randomly chosen from { 1 , · · · , 100 } , and the corresponding outputs { y i } | D (cid:48) | i =1 are generated from the model
y i
=
[ (cid:107) x
i (cid:107) / 7] + (cid:15) i , 1 ≤ i ≤ | D (cid:48) | , where [ · ] means the integer part of inputs and (cid:15) i is the noise independently sampled from Gaussian distribution N (0 , 0 . 01) .
Dimension q is 7.
We generate 20000 samples.
70% is used for training and 30% for testing.
Figure 1(a) and Figure 1(b) show the testing error and training time (logarithmizing it) in seconds about the number of random features m with p = 2 and indicate that DRank-RF has an obvious advantage over DRank and LSRank, even one order of magnitude less, in time cost.
In the testing error, the gap between DRank-RF and DRank decreases as m increases.
Finally, there is no signiﬁcant dif-
[Figure]
ference between DRank-RF and DRank, both of which are close to the optimal level.
These results are consistent with our theoretical analysis.
With the increase of the number of random features m , the training time of DRank-RF increases, and the testing error becomes smaller, which are in line with the theoretical reasoning.
And the testing error of DRank-RF declines signiﬁcantly when m is a small number.
Therefore, in practice, we only need to take a small m to obtain a satisfactory error, which will result in the savings of computing resources.
Note that, DRank and LSRank have nothing to do with m .
Figure 1(c) and Figure 1(d) show the testing error and training time about the number of partitions p with m = 200 for DRank-RF.
Figure 1(c) shows DRank-RF keeps the same accuracy level as
DRank.
With the increase of the number of partitions p , the testing errors increase in p -related algorithms, which is in line with the theoretical analysis.
In Figure 1(d), with the increase of p , the training time decreases in distributed algorithms (DRank-RF and DRank).
Our algorithm DRankRF has a signiﬁcant advantage over LSRank and DRank in the training time.
In particular, the time cost of DRank with p = 30 is higher than that of DRank-RF with p = 15 , that is to say, the proposed DRank-RF requires less expensive hardware devices, under the same scenario and time cost.
Combining Figure 1(c) and Figure 1(d), we obtain that DRank-RF can use fewer hardware devices (local processors) to achieve a smaller testing error under the same training time, which is consistent with the theoretical analysis.
Figure 2 shows the relation between the testing error, p , and different numbers of communications ( M = 2 , 4 , 8 ) with m = 300 and indicates the following information: 1) With the increase of p , the testing error gaps between p -related algorithms and exact LSRank become larger and larger.
There exists an upper bound of p for DRank-RF and DRank-RF-C respectively, when larger than it, the testing error increases and is far from the exact LSRank.
This is in line with Theorem 1 and Theorem 2. 2)
The upper bound p of DRank-RF-C is much larger than DRank-RF, which is aligned with our theoretical analysis that the bound of p is determined by the communication times.
Under the same p , the performance of DRank-RF-C is better than DRank-RF.
And with the increase of the number of communications M , the testing error of DRank-RF-C is smaller.
These verify the power of the communication strategy for DRank-RF. 4) Under the same conditions, the testing errors of the proposed DRank-RF and DRank-RF-C are similar to those of DRank and DRank-C.
6. R EAL D ATA
The real dataset of MovieLens is from website http://www.grouplens.org/taxonomy/term/14, which is a 62423 × 162541 rating matrix where ( i, j ) entry is the rating score of the j -th reviewer on the i -th movie.
We group the reviewers into 500-1000 movies according to the number of movies they have rated.
And 500 reference reviewers are selected at random from this group.
In addition, we select the test reviewers from those users who had rated more than 5000 movies.
So, we obtain a small matrix with the scale of at least 5000 × 501 , where the last column corresponds to the test reviewer and the other columns correspond to the 500 reference reviewers.
Then the columns without non-zero elements are deleted and the rows without the rating of any reference reviewers or the test reviewer are deleted.
Finally, missing review values of every left movie were replaced
[Table]
DRank-RF
[Table]
DRank-RF-C #
[Table]
DRank-RF-
[Table]
[Table]
DRank-
[Table]
with the median review score of those left reference reviewers on this movie.
Here, we obtain a smaller matrix.
Each row of it is a data pair ( x i , y i ) and the last entry was the label
y i of the input features x i .
The experimental dataset is similar to that in Chen et al.
On the obtained dataset, 70% is used for training and 30% for testing.
The empirical evaluations are given in Table 2 where m = 100 , 150 and p = 2 , 10 , 15 .
In Table 2, we can ﬁnd that the experimental results are similar to those on the simulated data.
The average testing error gaps between our methods and the exact methods are particularly small, which verify the effectiveness of our methods on the real dataset.
Under the conditions of M =16, p =2, and p =10, the testing error of DRank-RF-C is convergent and does not change with the increase of the number of communications.
Under the condition of p =15, the testing error of DRank-RF-C decreases with the increase of the number of communications, which demonstrates the effectiveness of the communication strategy on the real dataset and is consistent with our Theorem 2.
The training time in the distributed algorithms decreases with the increase of p .
The training time in communication-based algorithms increases with the increase of the number of communications.
The proposed DRank-RF and Drank-RF-C have signiﬁcant advantages over LSRank, DRank, and DRank-C in the training time.
These are consistent with the theoretical analysis.
More experiments on different datasets are given in Appendix E.
C ONCLUSIONS
We propose a novel pairwise ranking method (DRank-RF) to scale to large-scale scenarios.
Our work is the ﬁrst time to apply random features to least square ranking, which is a new exploration of the application of random features.
Our theoretical analysis based on the techniques of integral operators shows that its convergence rate is sharper than that of the existing state-of-the-art DRank without communications.
In computational complexity, DRank-RF only requires O ( m 2 | D j | ) time and
O ( m | D j | ) memory, which are the least compared with the existing state-of-the-art DRank.
Experiments verify that our proposed method keeps the similar testing error as the exact and state-of-the-art approximate methods and has a greatly advantage over the exact and state-of-the-art approximate methods in the training time, which are consistent with our theoretical analysis.
To further improve the performance of DRank-RF, we propose a communication strategy to DRank-RF, which is called
DRank-RF-C. Statistical analysis shows that DRank-RF-C obtains a faster convergence rate than
DRank-RF.
Compared with the existing state-of-the-art DRank with communications, DRank-RF-C requires less complexity and keeps a sharper convergence rate.
And the numerical results validate the power of the communication strategy.
A P RELIMINARY DEFINITIONS
There is a compact metric space Z :=
( X , Y ) ⊂
R q +1 , where X ⊂ R q and Y ⊂ [ − b, b ] for some positive constant b .
The sample set D := { ( x i , y i ) } Ni =1 of size N = | D | is drawn independently from an intrinsic Borel probability measure ρ on Z .
ρ ( y | X = x ) denotes the conditional distribution for given input x .
The hypothesis space used is the reproducing kernel Hilbert space (RKHS) ( H K ) associated with a mercer kernel K : X × X
→ R (Aronszajn, 1950; Cucker & Zhou, 2007).
We will denote the inner product in H K by (cid:104)· , ·(cid:105) , and corresponding norm by (cid:107) · (cid:107) K .
K x = K ( x , · ) .
Let ρ X be the margin distribution of ρ with respect to X and L
ρ X be the Hilbert space of ρ X square integrable functions on X .
The Mercer kernel K deﬁnes an integral operator L K on H K (or L 2 ρ X ) (Chen et al., 2021) by
[Equation]
Suppose ψ is continuous, such that | ψ ( x , ω ) | ≤ τ almost surely, τ ∈ [1 , ∞ ) .
Assume that L − rK f ρ ∈
H K with 0 < r ≤ 1 , where L rK is the r -th power of L K .
Before the proof, we give some deﬁnitions: S m : R m →
L 2 ρ X , ( S
[Equation]
g ) (
[Equation]
) = (cid:104) g , φ m ( x ) (cid:105) ,
S ∗ m : L 2 ρ X
[Equation]
ρ X → R m , S ∗ m,D f = 1 | D | (cid:80) x j ∈ D X φ m ( x j ) f
( x j ) .
S ∗ m S m and Φ m,D Φ T m,D
=
S ∗ m,D S m are self-adjoint and positive operators, with spectrum is (cid:2) 0 , τ 2 (cid:3) (Caponnetto & Vito, 2007).
This part is organized as follows:
In section B, we introduce the proof of Theorem 1.
Section B. contains the main lemmas used for the proof of Theorem 1 and Theorem 2.
Section B.2 is the detail proof process of Theorem 1.
In section C, we introduce the proof of Theorem 2.
Section C. contains the main lemmas used for the proof of Theorem 2.
Section C.2 is the detail proof process of Theorem 2.
In section D, we introduce the propositions used for the proof of Theorem 1 and
Theorem 2.
Section E is the experiments on Jester Joke dataset.
B P ROOF OF T HEOREMB. B OUND T ERMS
Lemma 1.
We have √
[Equation]
Proof.
Note that (cid:107) g m,D,λ − g m,λ (cid:107) ≤ (cid:13)(cid:13)(cid:13)
g m,D,λ − g (cid:5) m,D,λ (cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13) g (cid:5) m
[Equation]
D
[Equation]
λ − g m,λ (cid:13)(cid:13)(cid:13) .
Deﬁne f m,D,λ = g T m,D,λ φ m ( · ) , g m,D,λ =
[Equation]
g (cid:5) m,D,λ = arg
min g ∈ R m (cid:110) 1 | D | 2 (cid:80)
z i ,z k ∈ D (cid:0) ( g T φ m ( x i ) − f ρ ( x i ))
− ( g T φ m ( x k ) − f ρ ( x k )) (cid:1) 2 + λ (cid:107) g (cid:107) 2 (cid:111) .
One can have f m,D,λ =
S m g m,D,λ , g m,D,λ = (cid:0) Φ m,
D W D
Φ T m,D + λ 2 I (cid:1) − 1 Φ m,D W D ¯ y D , f
[Equation]
m,D,λ = S m g (cid:5) m,D,λ , and g
[Equation]
m,D
[Equation]
λ =
[Equation]
m,D W
[Equation]
(cid:1)
[Equation]
m
[Equation]
D W D f ρ , so we have g m
[Equation]
D,λ − g (cid:5)
[Equation]
Note that (cid:13)(cid:13)(cid:13)(cid:0) Φ m,D W D Φ Tm,D + λ 2 I (cid:1) − 1 / 2 (cid:13)(cid:13)(cid:13) ≤ (cid:112) 2 /λ .
Thus we can obtain that
[Equation] [Equation]
λ
[Equation]
g (cid:5) m
[Equation]
D
[Equation]
λ (cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) (cid:18)
[Equation]
[Table]
[Equation]
(cid:19)
[Equation]
(cid:19)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:33) .
Deﬁne f m,λ = g T m,λ φ m ( · ) with g m,λ = arg min g ∈ R m (cid:26)(cid:90)
Z (cid:90)
Z (cid:0) ( g T φ m ( x ) − f ρ ( x )) − ( g T φ m ( x (cid:48) )
− f ρ ( x (cid:48) ))
(cid:1) 2 dρ
X ( x , y ) dρ
X ( x (cid:48) , y (cid:48) )
+ λ (cid:107) g (cid:107) 2 (cid:27) .
We know f m,λ =
S m g m,λ and g m,λ
= (cid:0) S ∗ m W D S m
+ λ 2 I (cid:1) − 1 S ∗ m W D
f
ρ .
So one can obtain g (cid:5) m,D,λ − g m,λ = (cid:18) Φ m,
D W D Φ Tm,D + λ 2 I (cid:19) −
S ∗ m,D W D f ρ − (cid:18)
S ∗ m W D S m
+ λ 2 I (cid:19) −
S ∗ m W D
f ρ = (cid:18) Φ m,
[Equation]
For any self-adjoint and positive operators A and B ,
[Equation]
so we have
[Equation]
We know that Φ m,D Φ T m,D = S ∗ m,D S m (Caponnetto & Vito, 2007)
[Equation]
thus we can obtain that
[Equation]
g
[Equation]
<
[Equation]
Thus, we have
[Equation] [Equation]
Note that
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Φ
[Equation]
W
[Equation]
Φ
[Equation]
+ λ 2 I (cid:19) − 1 /
S ∗ m,D (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) Φ m,D W
[Equation]
Φ
[Equation]
I (cid:19) − 1 /
[Equation]
(cid:18)
[Equation]
I (cid:19) − 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 /≤and (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:18)
[Equation]
(cid:19)
[Equation] [Equation]
(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
[Equation]
(cid:19)
[Equation] [Equation]
(cid:18)
[Equation]
I (cid:19) 1 / 2 (cid:18)
[Equation]
I (cid:19) − 1 /
S
[Equation]
m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Φ
[Equation]
Φ
[Equation]
I (cid:19) − 1 / 2 (cid:18)
S
[Equation]
λ
[Equation]
I (cid:19)
[Table]
since
[Equation]
Substituting the above two inequalities into Eq.(13) we have
[Equation]
Combining Eq.(11) and Eq.(14)
[Equation]
we ﬁnish this proof.
Lemma 2.
We have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
According to f m,D,λ − f (cid:5) m,D,λ = S m (cid:16) g m,D,λ − g (cid:5) m,D,λ (cid:17) , by Eq.(10), we have f m,D,
[Equation] [Equation]
Note that
[Equation] [Equation]
(cid:18)
[Equation]
(cid:19) −
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:19)
[Equation]
(cid:18)
[Equation]
(cid:19) −
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 /≤ 1 .
So, by Eq.(15) we have (cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
[Equation]
[Table]
Similarly, according to Eq.(12)
[Equation]
we have
[Equation]
[Table]
[Equation]
so we have (cid:13)
[Equation]
Combining Eq.(16) and Eq.(17), we ﬁnish this proof.
Lemma 3.
For δ ∈ (0 , 1] , with probability at least 1 − δ , we have
[Equation]
δ
[Equation]
where N m ( λ ) =
Tr (cid:16)(cid:0) L m + λ 2 I (cid:1) − 1 L m (cid:17) , L m is the integral operator associated with the approximate kernel function K m , ( L m f ) ( x ) =
(cid:82)
X K m ( x , x (cid:48) )
f ( x (cid:48) )
dρ X ( x (cid:48) ) .
Proof.
We have
[Equation]
According to Lemma 6 in Rudi & Rosasco (2017), we know, with probability at least 1 − δ ,
[Equation]
where N m ( λ ) =
Tr (cid:16)(cid:0) L m + λ 2 I (cid:1) − 1 L m (cid:17) , L m is the integral operator associated with the approximate kernel function K m , ( L m f ) ( x ) =
(cid:82)
X K m ( x , x (cid:48) )
f ( x (cid:48) )
dρ X ( x (cid:48) ) .
Thus, we complete this proof.
Lemma 4.
For δ ∈ (0 , 1] , with probability at least 1 − δ , we have
[Equation]
, where N m ( λ ) =
Tr (cid:16)(cid:0) L m + λ 2 I (cid:1) − 1 L m (cid:17) .
Proof.
We have
[Equation]
According to Proposition 5 in Liu et al.
(2021), with probability at least 1 − δ , we have
[Equation]
where N m ( λ ) =
Tr (cid:16)(cid:0) L m + λ 2 I (cid:1) − 1 L m (cid:17) .
Combining them
[Equation]
we complete this proof.
Lemma 5.
For any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Proof.
Since S ∗ m S m is self-adjoint operator
[Equation]
so we have
[Equation]
According to Proposition 1 with ζ i = φ m ( x i ) , we can obtain (cid:13) (cid:13)(cid:13)
(cid:13)(cid:13)
[Equation]
+
[Equation]
|
[Equation]
log
[Equation] [Equation]
δ )
[Equation]
N
[Equation]
λ ) + 1) | D | , where
N
[Equation]
sup
[Equation]
˜ L K f = (cid:82) X K ( x , · ) f ( x ) dρ X
(Rudi & Rosasco, 2017), c
1 and c 2 are two constants.
Therefore, we have
[Equation]
Lemma 6.
We have
[Equation]
Proof.
Note that g m,D,λ =
[Equation]
Φ m,D W D Φ Tm,D + λ 2 I ) −
[Equation]
Φ m,D W D ¯y D .
Thus we have
[Equation]
By introducing S ∗ m S m term
[Equation]
we can convert the above formula into
[Equation]
+
[Equation]
j
[Equation]
| D
[Equation]
|
[Equation]
D k |
[Equation]
So we have
[Equation]
Note that
[Equation]
=
[Equation]
and
Term-
[Equation]
=
[Equation]
Substituting the above equations into Eq.(21)
[Equation]
we have
[Equation]
(cid:13)
[Equation]
∗
[Equation]
g m
[Equation]
D j
[Equation]
(cid:13)(cid:13) .
Here, we complete this proof
[Equation]
Lemma 7.
We have
[Equation]
Proof.
Note that
[Equation]
According to Eq.(21)
[Equation]
we have
[Equation]
Note that
[Equation]
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Similar as the above process, we can obtain that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
Combining Eq.(23), Eq.(24), and Eq.(25), we obtain this result.
Lemma 8.
For δ ∈ (0 , 1] and λ > 0 , when
[Equation]
=
[Equation]
, with probability at least 1 − δ , we have
[Equation]
where c is a constant.
Proof.
Note that f m,λ =
S m g m,λ and g m,λ
= (cid:0) S ∗ m W D S m
+ λ 2 I (cid:1) − 1 S ∗ m W D
f
ρ .
We have
[Equation]
f
[Equation]
K , where ˜ f λ = arg min f ∈H K { (cid:82) X ( f ( x ) − f ρ ( x ))
2 dρ X ( x ) + λ (cid:107) f (cid:107) 2 K } .
According to Lemma 2 in Liu et al.
(2021) (can be also seen in Li et al.
and Rudi & Rosasco (2017)), one has, when m = Ω (cid:0) λ − 2 r ∨ λ − 1 log 1 λδ (cid:1) , with probability at least 1 − δ ,
[Equation]
Combining the above, we complete this proof.
B. P ROOF OF T HEOREM
Proof.
We have
[Equation] [Equation]
Combining Lemma 1, Lemma
[Equation]
and Lemma
[Equation]
we have
[Table]
S
[Equation]
m S m +
[Equation]
(cid:1) .
From Lemma 5, we know that if | D | ≥ 32 log(2 /δ )
(cid:0) 1 + 2 τ 2 λ −
[Equation] [Equation]
Combining the above inequality and Proposition 2, for any δ > 0 , with probability at least 1 − δ , we can obtain,
[Equation]
From Lemma 2, we have
[Equation] [Equation]
S
[Equation]
+ (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
[Equation]
+ (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) (cid:18)
[Equation]
(cid:19) 1 / 2 (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) 2   (cid:107) f m,λ − f ρ (cid:107) K .
From Proposition 3, Lemma 3, Lemma 4, and Eq.(27), we know that if | D | ≥ Ω (cid:0) τ 2 λ − 1 (cid:1) , we have (cid:107) f m,D,λ − f m,λ (cid:107) K
= O (cid:18) Υ m,D,λ log 1 δ
+ (cid:107) f m,λ − f λ
[Equation]
where
[Equation]
Note that
[Equation]
According to Proposition 4 and Lemma
[Equation]
we have
[Equation]
Combining Eq.(26), Eq.(28), Eq.(30), Proposition
[Equation]
and Lemma 8, one can obtain, if m = Ω (cid:0) λ − 2 r ∨ λ − 1 log 1 λδ (cid:1) , with probability 1 − δ , we have
[Equation]
According to Lemma 5, we have
[Equation]
Set λ
= O (cid:18)(cid:16)(cid:80) pj =
[Equation]
(cid:19) , we have the number of random features m
[Equation]
Ω    
[Equation]
(cid:88)
j =
[Equation]
|
[Equation]
| (cid:80) pk
[Equation]
|
[Equation]
k |
[Equation]
 
[Equation]
r 1+ r   .
Combining Eq.(31), Eq.(29), and Eq.(32), we have (cid:13)(cid:13)
[Equation]
m,D
[Equation]
f
[Equation]
(cid:13)(cid:13) K
[Equation]
We complete this proof
[Equation]
C P ROOF OF T HEOREMC. B OUND T ERMS
Lemma 9.
We have
[Equation]
where
[Equation]
(cid:19) − 1 /
[Equation]
(cid:18)
[Equation]
(cid:19) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
[Equation]
(cid:19)
[Equation]
(cid:16)
[Equation]
(cid:18)
[Equation]
(cid:19)
[Equation] [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Proof.
Note that
[Equation]
(cid:21)
[Equation]
and ¯ g lm,D,λ =¯ g l
[Equation]
1 m,D,
[Equation]
− p (cid:88) j =
[Equation]
|
[Equation]
Thus, we have g m,D,λ − ¯ g lm,D,λ =¯ g l − 1 m,D,λ − (cid:18)
[Equation]
The above can be convert into
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
so, we have
[Equation] [Equation]
Note that
[Equation]
Substituting the above into Eq.(35), we have
[Table]
[Equation]
the last inequality use the fact that
[Equation]
Using the same process
[Equation]
we can obtain that
[Equation]
[Table]
Thus, we have
[Equation]
According to Eq.(34)
[Equation]
we know that
[Equation]
+
[Equation]
j
[Equation]
|
[Equation]
Thus
[Equation]
we obtain that
[Equation]
≤ p
[Equation]
j
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
C. P ROOF OF T HEOREM
Proof.
Note that
[Equation]
Substituting Lemma 1, Lemma 2, Lemma 3, Lemma 4, Eq.(27)
[Equation]
and Eq.(28) into Lemma 6 and
Lemma
[Equation]
we have
[Equation]
where SS
[Equation]
, and Q
[Equation]
Combining the above inequality and Lemma
[Equation]
and note that
[Equation]
we can obtain that
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
Combining Eq.(38), Eq.(39), Proposition 4, and Lemma 8, one can obtain, if m = Ω (cid:0) λ − 2 r ∨ λ − 1 log 1 λδ (cid:1) , with probability 1 − δ , we have (cid:13)(cid:13) ¯
[Equation]
K
= O    
[Equation]
(cid:13)(cid:13)(cid:13) l ∗  
[Equation]
+
[Equation]
m
[Equation]
Set λ
= O ( | D | − 11+ r ) , | D 1 | = . . .
=
| D p | = | D | p , and the number of random features m = Ω (cid:16) | D | 2 r 1+ r (cid:17) , we have
[Equation]
where M = l .
We complete this proof.
D P ROPOSITIONS
Proposition 1 ((Liu et al., 2021)) .
Let ζ 1 , . .
, ζ n with n ≥ 1 , be i.i.d random vectors on a separable
Hilbert spaces H such that H =
E ζ ⊗ ζ is a trace class, and for any λ there exists N ∞ ( λ ) <
∞ such that (cid:10) ζ, ( H + λ 2 I ) − 1 ζ (cid:11) ≤ N ∞ ( λ ) .
Denote H n as 1 n (cid:80) ni
=1 ζ i ⊗ ζ i .
Then for any δ ≥ 0 , with probability at least 1 − 2 δ , the following holds
[Equation]
n .
Proposition 2 ((Blanchard & Kr¨amer, 2010)) .
For any self-adjoint and positive semideﬁnite operators A and B , if there exists η > 0 such that the following inequality holds
[Equation]
then
[Equation]
Proposition 3 (Proposition 10 in Rudi & Rosasco (2017)) .
For any δ ∈ (0 , 1] , m ≥ Ω (cid:0) 2 τ 2 λ − 1 log 1 λδ (cid:1) then with probability at least 1 − δ ,
[Equation]
where N m ( λ ) =
Tr (cid:16)(cid:0) L m + λ 2 I (cid:1) − 1 L m (cid:17) .
Proposition 4 (Eq.(9) in Chen et al.
(2021), Chen (2012)) .
Assume that L − rK f ρ ∈ H K with 0 < r ≤ 1 , where L rK is the r -th power of L K , we have (cid:107) f λ − f ρ (cid:107) K
= O ( λ r ) .
Here we prove the gradient of the empirical risk of
[Equation] λ (cid:107) g (cid:107) 2 on g is 4 G m,D j ,λ ( g ) for all ( x i , y i ) , ( x k , y k ) ∈ D j .
Proof.
We have
[Equation]
So, we have the results.
E S UPPLEMENTARY E XPERIMENTS
We add the experiments on the dataset Jester Joke.
Jester Joke is publicly available from the following URL: http://www.grouplens.org/taxonomy/term/14 and contains over 4 . 1 million continuous anonymous ratings ( − 10 .
to + 10 . 00) of 100 jokes from 73,421 users.
We group the reviewers according to the number of jokes they have reviewed.
The grouping is 40-60 jokes.
For a given test reviewer, 300 reference reviewers are chosen at random from the group and their rating are used to form the input vectors.
70 percent of the test reviewer’s joke ratings are used for training and the rest for testing.
Missing review values in the input features are populated with the median review score of the given reference reviewer.
Here, we add the comparison with MPRank algorithm (Cortes et al., 2007).
It is not a distributed algorithm related to this paper, but it is a representative algorithm in the ﬁeld of least square ranking, so it is compared here. [Table] [Table]
The empirical evaluations are given in Table 3 where the number of random features is m =and 50 and the number of partitions is p = 2 and 4. In Table
[Table] we can ﬁnd that the experimental results are similar to those on the simulated data and MovieLens dataset.
The average testing errors of our methods, the exact method, MPRank [Table] and DRank remain at the same level [Table]
which verify the effectiveness of our methods on the real dataset. The testing error of DRank-RF-C decreases with the increase of the number of communications, which demonstrates the effectiveness of the communication strategy on the real dataset.
The proposed DRank-RF and Drank-RF-C have signiﬁcant advantages over LSRank, MPRank, DRank, and DRank-C in the training time.
These are consistent with the theoretical analysis.
We add the experiments under the same experiments setting as (Chen et al., 2021) on the datasets mentioned in the main paper.
Table 4 shows the experimental results with partitions p = 60 , dimension q = 3 , and random features m = 150 on simulated dataset with the same data generating distribution as (Chen et al., 2021), and p = 60 and m = 150 on MovieLens dataset.
Our algorithm
DRank-RF has a signiﬁcant advantage over DRank and LSRank in the training time.
Under the same conditions, the testing errors of the proposed DRank-RF and DRank-RF-C are similar to those of DRank and DRank-C.