Distributed Learning of Conditional Quantiles in the Reproducing Kernel Hilbert Space
Anonymous Author(s)
AffiliationAddress email
Abstract
We study distributed learning of nonparametric conditional quantiles with Tikhonovregularization in a reproducing kernel Hilbert space (RKHS).
Although distributedparametric quantile regression has been investigated in several existing works,the current nonparametric quantile setting poses different challenges and is stillunexplored.
The difficulty lies in the illusive explicit bias-variance decompositionin the quantile RKHS setting as in the regularized least squares regression.
For thesimple divide-and-conquer approach that partitions the data set into multiple partsand then takes an arithmetic average of the individual outputs, we establish the riskbounds using a novel second-order empirical process for quantile risk.
Introduction
Distributed learning has attracted significant attention recently with the increasing popularity ofdistributed systems and distributed data.
Although the algorithmic aspects have gone far beyondthe simple data parallelism paradigm that uses a simple divide-and-conquer strategy to carry outcomputations on each part of the data separately and then combine them in a finalizing step, thetheoretical verification has lagged behind even though nowadays there are a lot of cases for whichlearning rates have been established, including Zhang et al.
; Rosenblatt and Nadler(2015); Balcan et al.
; Chang et al.
; Lee et al.
; Lin et al.
(2017); Jordan et al.(2018); Volgushev et al.
; Lian and Fan (2018).
It has been shown repeatedly for variousmodels that the divide-and-conquer method can achieve the same learning rate as the standardestimator (assuming the whole data set is available on a single machine and it is computationallyfeasible to compute this central estimator based on the entire sample), as long as the number ofpartitions/machines is not too large.
In this work, we investigate the theoretical underpinning of the divide-and-conquer strategy fordistributed learning of nonparametric quantile regression models in reproducing kernel Hilbert spaces(RKHS).
Under this framework, the distributed learning rate for kernel ridge regression with a leastsquares loss has been established in several works (Lin et al., 2017; Guo et al., 2017; Chang et al.,2017).
The quantile regression proposed by Koenker and Bassett Jr (1978) is an important class ofmodels that can focus on the tail of the response distribution and is more robust the than least squaresregression.
However, the rate for nonparametric distributed learning other than least squares loss inthe RKHS has not been established.
The difficulty is that the nonparametric quantile estimator in the
RKHS does not have an explicit form, which makes the decomposition of the mean squared errorinto bias and variance illusive, while such a decomposition is the key to establishing the rates in thedivide-and-conquer method.
Previously, the quantile regression estimator in the RKHS studied in Liet al.
based on standard empirical process theory does not make it possible to decompose theerror into bias and variance and thus is not adaptable to the distributed setting, although the upperbound derived there can be roughly regarded as a sum of a bias term and a variance term.
On the other hand, for the fixed-dimensional parametric quantile regression, the Bahadur representation forthe estimator is well-established in the literature which can be used in the distributed setting.
Even inhigh-dimensional parametric quantile regression, Zhao et al.
(2020) used a debiasing strategy so thatafter debiasing the bias term is sufficiently small and can be explicitly controlled.
Another relatedwork on distributed high-dimensional parametric quantile regression is Chen et al.
(2020), which ismore closely related to the gradient method.
To address the challenges in trying to control the bias and the variance, we use what we call thesecond-order empirical process involving the quantile loss function, which makes it possible toapproximate more accurately the quantile loss by a weighted least squares loss, and we show thatthis approximation of the loss leads to approximation of the estimator (the weighted least squaresestimator is close to the quantile estimator).
The rest of the paper is organized as follows.
In thenext section, we state and prove the learning rates for both the central estimator and the distributedestimator.
Section 3 contains a simple numerical illustration, which is not used to verify the learningrate (which is difficult if not impossible) but just to illustrate the divide-and-conquer method canwork reasonably.
The paper concludes in Section 4.
Conditional quantile estimation in the RKHS
For the response y , its conditional quantile at quantile level τ is denoted by Q y | x ( τ ) = f 0 ,τ ( x ) .
Forsimplicity of notation, we will suppress τ in our notations since we will focus on only an individualvalue of τ without modelling different quantiles jointly.
In the setting of RKHS, the unknown true function f 0 in the regression problem is assumed to beinside an RKHS H ⊂
L 2 ([0 , 1]) with an associated kernel K ( ., . ) , with the L 2 space defined by aprobability measure ρ X for x on [0 , 1] .
Note that here we assume the covariate x ∈ [0 , 1] withoutmuch loss of generality.
In the following we use ⟨ , ⟩ H and ∥ .
∥ H to denote the RKHS inner product andthe RKHS norm, respectively, while ∥ .
∥ is the L 2 -norm for functions.
The function f 0 is estimated by
[Equation]
where ( y i , x i ) , i = 1 , . .
, N is the entire sample from the true generating model with P ( y i ≤f 0 ( x i )
| x i )
=
τ , ρ τ ( x ) = x ( τ − I { x ≤ 0 } ) is the quantile loss function, and λ > 0 is a tuningparameter.
(cid:98) f is referred to as the central estimator since it utilizes the entire sample.
In the distributed setting, the entire sample is distributed onto m machines and each machine computesa local estimate
[Equation]
where ( y i , x i ) , i = 1 . . .
, n (suppressing the dependence of the index on j ) is the sub-sample on thej -th machine.
We assume n = N/m is an integer and m is allowed to diverge with N .
The finaldistributed estimator is given by ¯ f = (cid:80) mj
=1 (cid:98) f j /m .
Let L : L 2 ([0 , 1])
→ L 2 ([0 , 1]) , L ( f ) = (cid:82) K ( x, . )
f ( x ) dρ
X
( x ) and define L γ : L 2 ([0 , 1])
→
L 2 ([0 , 1]) , L γ ( f ) =
(cid:82) γ (0 | x ) K ( x, . )
f ( x ) dρ
X
( x ) , where γ (0 | x ) is the conditional density of e i :
=
y i − f
( x i ) at 0 .
We will assume that γ (0 | x ) is bounded away from zero and infinity for all x andthus the properties of L and L γ are almost identical for our purpose.
We assume L is compact andhas the spectral decomposition L = (cid:80) ∞
j =1 s j ϕ j ⊗ ϕ j , where s 1 ≥ s 2 ≥ · · · > 0 are the eigenvaluesand ϕ
j are the eigenfunctions.
We assume the following smoothness condition on the true f 0 :f
=
L r g for some g ∈ H , r ∈ [0 , 1 / 2] .
Assumption (3) is often referred to as the “source condition” in the literature.
L r is defined asthe operator (cid:80) j s rj ϕ j ⊗ ϕ j .
The condition can be written more explicitly as follows.
By Lemma1.1.1 of Wahba (1990), any g ∈ H can be written as g = (cid:80) j g j ϕ j with (cid:80) j g 2 j /s j < ∞ , and
L r g = (cid:80) j s rj
g
j ϕ j .
Thus the assumption is the same as (cid:80)
j f 20 j /s 1+2 r j < ∞ , where f 0 j = ⟨ f 0 , ϕ j ⟩is the Fourier coefficient of f 0 in the expansion.
Since s j → 0 , larger r implies faster decay of f 0 j , and the decay rate of the Fourier coefficient is naturally a characterization of its smoothness infunctional analysis.
We establish the error bounds for the central regularized quantile estimator (2) as well as the distributedestimator.
For the central estimator, although quantile regression has been studied in several workssuch as Li et al.
; Zhang et al.
, these works simply assume f ∈ H and does not imposethe additional assumption (3).
As for the least squares regression (Caponnetto and De Vito, 2007),such a regularity condition is expected to lead to faster rates.
However, different from the leastsquares case which relies on the closed-form solution of the regularized estimator, taking into accountthe assumption (3) requires entirely different technical tricks.
Throughout the paper, C denotes a generic positive constant.
The following assumptions are used.
(A1) s j ≤ Cj − α for some α > 1 .
The smoothness assumption (3) holds.
(A2)
We have the entropy condition log N ( ε, H (1) , L ∞ ) ≤ ( C/ε ) 2 /α , where H (1) =
{ f ∈ H :∥ f ∥ H ≤ 1 } .(A3) (Sup-norm assumption)
∥ f ∥ ∞ ≤ C ∥
f ∥ 1 − 1 α ∥ f ∥ 1 α H , ∀ f ∈ H .(A4)
The conditional density of e = y − f 0 ( x ) given x , denoted by γ ( .
| x ) , and its derivative areuniformly bounded.
γ (0 | x ) is bounded away from zero uniformly for x ∈ [0 , 1] .
The assumption (A1) on the polynomial decay of eigenvalues is commonly assumed in the literature.
The entropy condition (A2) is stronger than assuming s j ≤ Cj − α .
Such an assumption is notuncommon and used in Müller and van de Geer (2015).
Theorem 6.26 of Steinwart and Christmann(2008) showed that it is satisfied if the RKHS has a kernel with derivatives of sufficiently high orders.
The sup-norm assumption is also used in Steinwart et al.
(2009); Suzuki and Sugiyama (2013), whichyields a better control for the sup-norm of a function in the RKHS.
The sup-norm assumption issatisfied if the RKHS is a Sobolev space or is continuously embeddable in a Sobolev space.
Assumptions (A1)-(A4) are required for the proof for the distributed estimator.
For the centralestimator, we can remove (A2) and (A3) and add the following assumption.
[Equation]
A2’)
For some sufficiently small constant
[Equation]
Assumption (A2’) is a strong convexity condition of the population loss.
It can be shown that (A2’) isimplied by (A3) and (A4).
It also holds if the conditional density γ ( .
| x ) is uniformly bounded awayfrom zero ((A4) only requires γ (0 | x ) is uniformly bounded away from zero).
The following resultgives the bound for the central estimator.
Theorem 1 Assume (A1), (A2’) and (A4) hold.
Setting λ ≍ N − α (2 r +1)
α +1 , we have ∥ (cid:98) f − f 0 ∥ =
O p ( N − (2 r +1)
α 2(2 r +1) α +2 ) .
We first state a lemma on the relevant empirical process.
The proof relies on the Rademacher complexity E [sup ∥ f ∥≤ u, ∥ f ∥ H ≤ 1 1 N (cid:80) N i =1 σ i f ( x i )] , where σ i ∈ {− 1 , 1 } are i.i.d.
Rademacher variables.
The Rademacher complexity is a characterization of the complexity of the function space.
As shownin Mendelson (2002), we have a bound
[Equation]
N (cid:88)
[Equation]
u
[Equation]
(4) where R ∗ ( u ) =
(cid:113) 1 N (cid:80) ∞
j =1 min { s j , u 2 } .
When s j ≤ Cj − α , it is easy to see R ∗ ( u ) ≤ C R ( u ) with
R ( u ) :
= u 1 − 1 α √ N .
Lemma
For any u with R ( u ) ≤ u 2 , with probability at least 1 − e − CN R 2 ( u ) /u 2 ,(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
N (cid:88)
i
[Equation]
Sketches of Proof of Theorem 1.
We prove by way of contradiction.
Assuming ∥ (cid:98) f − f 0 ∥ is large, we try to establish
[Equation]
which will contradict the fact that (cid:98) f is the minimizer of the penalized loss.
The difference of thequantile loss on the two sides is approximated by its population version with the help of Lemma 1 andbounded below by assumption (A2’), while the difference of the penalties are carefully bounded by anontrivial application of Young’s inequality for operators, taking into account the source condition.
Proof of Theorem 1.
In the proof we choose u that satisfies R ( u ) =
u 2+2 r (that is, u =
N −
[Equation]
, since R ( u ) = u 1 − 1 α / √ N ).
By way of contradiction, suppose ∥ (cid:98) f − f 0 ∥ 2 ≥ Lu 2(1+2 r ) for some sufficiently large constant L .
This means
[Equation]
By the convexity of the functional in f in the above, we have
[Equation]
On the other hand, by assumption (A2’) and Lemma 1, with probability at least
[Equation]
CNu
[Table]
[Equation]
note
[Equation] [Equation]
In the 5th line above
[Equation]
we used that
[Equation]
where the first line used assumption (3), and the second to last line used Young’s inequality forpositive operators λ 1 − 2 r L 2 r ≤ (1 − 2 r )
λ + 2 r L .
(6) implies ∥
f − f 0 ∥
≤ Cu 1+2 r , contradicting∥ f − f 0 ∥ =
Lu 1+2 r for L sufficiently large, and the proof is complete.
□
The rate obtained for the central estimator is the same as the rate obtained for kernel ridge regressionin Caponnetto and De Vito (2007); Lin et al.
and thus cannot be improved in general.
However, the proof of Theorem 1 does not yield clearly any bias and variance terms separately.
As mentionedin the introduction, separate control of bias and variance is usually the key step in obtaining thelearning rate of the distributed estimator.
The empirical process of Lemma 1 cannot be utilized to getthe approximate bias and variance terms, although such type of empirical process was popularly usedin the literature (Belloni and Chernozhukov, 2011; Li et al., 2007;
Lv et al., 2018).
To deal with thisproblem, for the distributed estimator, we use the empirical process techniques not on the differenceof the quantile loss as in Lemma 1, but instead on a higher-order difference taking into account thegradient of the loss also, resulting in what we will call the second-order empirical process.
This willallow us to approximate the quantile loss by a weighted least squares loss, with results on distributedlearning for the latter available from the literature (after minor modifications).
More specifically, we define g ( x, y, f )
=
ρ τ ( y − f ( x )) − ρ τ ( y − f 0 ( x ))
+ ( τ − I { y − f 0 ( x ) ≤0 } )
( f ( x ) − f 0 ( x )) .
If the loss were differentiable, g would be the residual in the first-order Taylor’sexpansion of the loss.
For notational simplicity, we denote ϵ =
τ − I { e ≤ 0 } with e = y − f 0 ( x ) .
The key lemma that controls the difference between the quantile estimator and the weighted leastsquares estimator is the following result involving second-order empirical processes.
Due to the useof an additional term ( τ − I { y − f 0 ( x ) ≤ 0 } )
( f ( x ) − f 0 ( x )) representing the first-order expansionin g , the bound in Lemma 2 is typically much smaller than that in Lemma 1 for risk difference.
Lemma
For any u ∈ (0 , 1) , with probability 1 − exp { u − 2 s/α } where s = 3 2 − 12 α > 1 ,
[Equation]
Based on Lemma 2, we can prove our main result for the distributed estimator.
It is shown that undersuitable conditions, the learning rate for the distributed estimator matches that of the central estimator.
Theorem 2 Under assumptions (A1)-(A4), setting λ ≍
N − α (2 r +1)
α +1 , when m ≲
N min
[Equation]
, we have ∥ ¯
[Equation]
Remark
We note that, unfortunately, the exponent here is quite messy, and the first term inmin (cid:110)
α 2 (
[Equation]
r
[Equation]
[Equation] [Equation]
α (
[Equation]
r +1)+
[Equation]
(cid:111) can be either bigger or smaller than the second term(since the explicit expression on when the first term is bigger is quite messy we do not choose to writeit down).
Also the expression α 2 (2 r + 1) − (8 r + 2) α + 2 r + 1 may not even be positive (if it is notpositive, the bound becomes void).
It is positive if α ≥ 3 .
Sketches of Proof of Theorem 2.
From Lemma 2, we can show that (cid:98) f j which minimizes theregularized empirical loss can be well-approximated by (cid:101)
f j that minimizes the sum of three terms:
P n (( f ( x ) − f 0 ( x )) ϵ )
(the first-order term in the definition of g ), the Tikhorov regularization term, andthe second-order term from Taylor’s expansion of the expected loss.
By definition (cid:101)
f
j is a weightedkernel ridge estimator and the statistical bound for its average over m machines can be more easilyestablished.
However, rigorous establishment of the closeness of (cid:98)
f j and (cid:101)
f j requires more detailedanalysis involving multiple applications of Lemma 2.
Proof of Theorem 2.
By Knight’s identity
that ρ τ ( x − y ) −
ρ τ ( x ) = − y ( τ −
[Equation]
{ x
[Equation]
+ (cid:82) y
( I { x ≤t } − I { x ≤ 0 } ) dt , we have
[Equation]
using E [( f ( x ) − f 0 ( x )) 3 ] ≤ ∥ f − f 0 ∥ ∞
E [( f ( x ) − f 0 ( x )) 2 ] and the sup-norm assumption, where
F ( .
| x ) is the conditional cdf of e = y − f 0 ( x ) .
Let (cid:101)
f
j be the minimizer of 12 E [ γ (0 | x )( f ( x ) − f 0 ( x )) 2 ] + P n (( f ( x ) − f 0 ( x ))
ϵ )
+ λ ∥ f ∥ 2 H .
Thisis basically the least squares problem when the true model is given by y = f
0 ( x ) + ϵ , with twodifferences.
One is that the loss is weighted by γ (0 | x ) and the other is that the first term is thepopulation counterpart of (cid:80) i ( r (0 | x i )
( f ( x i ) − f
( x i )))
2 /n .
It is easy to see that (cid:101)
f j has a closedform (cid:101) f
j =
( L γ + λ ) − 1 L γ f 0 +
( L γ + λ ) − 1 (cid:80) ni
=
ϵ i K
( x i ,. )
n .
If E [ γ (0 | x )( f ( x ) − f 0 ( x )) 2 ] is replacedby (cid:80) i
( r (0 | x i )( f ( x i ) − f
( x i )))
2 /n , the minimizer would be (cid:101)
f ∗ j =
( L nγ + λ ) − 1 L nγ f
+ ( L nγ +λ ) − 1 (cid:80) ni
=
ϵ i K
( x i ,. )
n , where L nγ , defined as L nγ f = 1 n (cid:80) ni
=1 f ( x i ) K
( x i , . ) , is the empirical versionof L γ .
We can see analysis of (cid:101)
f j would be similar to (and actually simpler than) the analysisof (cid:101) f ∗ j , and the latter was basically the problem studied in Lin et al.
(2017), since we assumedγ (0 | x ) is bounded away from zero and infinity and it makes no difference in statistical analysis.
Inparticular, by Corollary 11 of Lin et al.
(2017), we can get ∥ 1 m (cid:80) mj
=1 (cid:101) f
[Equation]
if m ≤ N
[Equation]
Furthermore, by combining Corollary 8 and Lemma 21 of Lin et al.
(2017), wealso get the local estimates satisfy ∥ (cid:101)
f j − f 0 ∥
[Equation]
p
[Equation]
Now we bound ∥ (cid:98) f
j − (cid:101) f j ∥ .
For simplicity of notation, define the function h ( u )
=
u 3 − 1 /α
+
[Equation]
By Lemma 2 and (7), for any f ∈ H with ∥ f − f 0 ∥ ≤ Cu and ∥ f − f
∥
H ≤ 1 ,
[Equation]
( u )
[Equation]
Let u 1 = N 1 2[(2 r +1) α +1] √ n , then n
[Equation]
0 ( x )) 2 } ≤ Ch ( u 1 ) .
Taking the difference of the two displayed above,−
[Equation]
Since (cid:101) f
j minimizes 1 2 P (cid:8) γ (0 | x )( f ( x ) − f 0 ( x )) 2 (cid:9) − 1 n (cid:80) i ϵ i ( f ( x i ) − f 0 ( x i ))
+ λ ∥ f ∥ 2 H , the firstorder optimality condition for (cid:101)
f j yields−n n (cid:88)
i
[Equation]
Thus we have
[Equation]
where the last step used (9).
Using this in (8)
[Equation]
we get
[Equation]
Let u 2 = n −
[Equation]
Note that this u 2 is the value of u that makes u 2 ≍ h ( u ) .
Let u 3 = (cid:112) h ( u 1 )
+ h ( u 2 ) .
Suppose 12 E [ γ (0 | x )( (cid:98) f
j ( x ) − (cid:101)
f j ( x )) 2 ] + λ ∥ (cid:98)
f
j − (cid:101) f
j ∥ 2 H ≥ Lu 23 for some L sufficiently large.
By (10), and using the same arguments as in the proof of Theorem 1 that lead to (5), there exists some f ∈ H with 12 E [ γ (0 | x )( f ( x ) − (cid:101)
f j ( x )) 2 ] + λ ∥
f − (cid:101) f j ∥ 2 H =
Lu 23 , and also satisfying 1 n (cid:80) i ρ τ
( y i − f ( x i ))
+
λ ∥ f ∥ 2 H ≤ 1 n (cid:80) i
ρ τ
( y i − (cid:101)
f j ( x i ))
+ λ ∥ (cid:101) f j ∥ 2 H , such that
[Equation]
Because the left hand side above is Lu 23 , this would lead to contradiction if L is large enough.
Tosee this, we note that
Lu 2 3 > Ch ( √ Lu 3 ) since u 3 > (cid:112) h ( u 2 ) =
u 2 , and also Lu 2 3 >
Ch ( u 1 ) sinceu
= h ( u 1 ) + h ( u 2 ) >
h ( u 1 ) .
Based on the previous arguments, we then have ∥ (cid:98)
f
j − (cid:101) f j ∥ ≤ Cu 3 .
Finally, the distributed estimatorsatisfies
[Equation]
As mentioned before, when m ≤ N
[Equation]
1)+1 , the second term above has the optimal rate
[Equation]
To obtain the same rate for ∥ ¯ f − f 0 ∥ , we would require
[Equation]
Using n = N/m , (11) is equivalent to saying m ≤ N α 2(
r +1) − (8 r +2)
α +2 r
[Equation]
+
[Equation]
In summary, if
[Equation]
N
[Equation]
the optimal rate is achieved.
□
Remark 2 As stated in the theorem, for ¯ f to achieve the same rate as the non-distributed estimator,we need the constraint m ≲ N min
[Equation]
On the other hand, from theproof, we have the convergence rate N −
[Equation]
+
[Equation]
N/m
[Equation]
even if the bound for m is notsatisfied (when the bound is not satisfied, we just have a non-optimal rate).
In practice, of course it ishard to check whether a specific pair ( m, N ) satisfies the bound since it is asymptotic in nature.
Onthe other hand, in the setting that data are physically located on m machines and one is unwilling toshare data across different machines, then we have no other option and just accept this given m .
Inthis situation, we do not have to be concerned with the choice of m .
Some numerical illustrations
Here we illustrate the distributed quantile regression in relatively simple simulations.
For a valueτ ∈ (0 , 1) , the sample is generated from the model
y i =
f 0 ( x i ) +
(1 + x i ) σ ( ϵ i − Φ − 1 ( τ )) , where x iare generated uniformly on [0 , 1] , ϵ i ∼ N (0 , 1) , Φ − 1 is the quantile function of the standard normaldistribution so that f ( x i ) is indeed the conditional τ -quantile of y i .
We set f 0 ( x ) = sin(2 πx ) and σ =0 .
The simulations are carried out for different combinations of n ∈ { 32 , 64 , 128 , 256 , 512 , 1024 }and m ∈ { 1 , 2 , 4 , 8 , 16 , 32 } , with the total sample size of N = nm .
Note that m = 1 actuallycorresponds to the central estimator.
For the RKHS, we use the Sobolev space of the second order.
The tuning parameter λ is chosen to minimize the errors on independently generated test data.
Ineach setting, 100 repetitions are used to compute the average squared estimation error ∥
¯ f − f 0 ∥ 2 .
Choice of λ is an important practical matter.
Theorem 2 shows the choice of λ ∼ N −
[Table]
depends on the total sample size N , which indicates that tuning parameter chosen based on local datais not optimal.
In our simulations, we use independently generated data to select the optimal λ thatensures ¯ f
( ¯ f implicitly depends on λ ) is the one that achieves the smallest prediction error on thetest data.
When independently generated data is not available, either hold-out data or global K -foldcross-validation can be used, which requires coordination of the central machine.
If the hold-out datais maintained by the central machine, the central machine can evaluate and compare different valuesof λ .
If K -fold cross-validation is carried out on the local machines, the central machine needs tosend ¯ f back to the local machines so that the local machines can calculate the cross-validation errors,and then send the errors to the central machine.
This needs to be done K times for K -fold CV.
Xuet al.
(2018, 2019) developed a generalized approximate cross-validation criterion in the least-squarescase and established its asymptotic optimality.
We expect that similar criterion developed for quantileregression (Yuan, 2006) can be adapted in our case.
However, even in the non-distributed setting,
Yuan (2006) did not establish the asymptotic properties of the criterion, mainly investigating itsperformance from a computational viewpoint.
Thus it remains an open question whether such acriterion has a desirable asymptotic property.
The investigation of such problems requires additionalstudy by itself and thus we do not use this in the current work.
The simulation results are shown in Table 1 for τ = 0 .
We can examine the table in several ways.
First, each row of the table indicates what happens when one fixes the number of observations on eachmachine while increasing the number of machines (the total sample size of course also increases).
In this case, we see the errors decrease with the number of machines since the total sample sizeincreases.
Second, each column of the table indicates what happens if the number of machines arefixed while the number of observations increases proportionally with the total sample size.
Here wealso see the error decreases.
Third, the results along the anti-diagonal directions shows the resultswhen the total sample size N is fixed while we change the number of machines m .
For example, thenumbers shaded in yellow all corresponds to N = 512 while the numbers shaded in blue correspondsto N = 1024 .
Here we see that the error in general increases with the number of machines (up tosome random fluctuations).
This is natural since the central estimator should perform best if it isfeasible to compute.
Finally, we observe that the table shows the divide-and-conquer strategy is usefulwhen N is large.
For example, in the table, the smallest error is achieved when n = 1024 , m = 32 ,
[Table]
corresponding to N = 32768 .
It would be hard to deal with such a large sample size in the standardimplementation (the kernel method is always computationally burdensome) without using someadvanced techniques to approximate the solution, if such techniques exist at all.
The simulation results for other values of τ are presented in Appendix A. We also perform an additional simulation with 3-dimensional predictors, as well as analysis of an airline on-time performancedata, which are reported in Appendix B and C, respectively.
Conclusion
In this paper, we established the convergence rate of the nonparametric quantile estimator in thedistributed learning setting under the RKHS framework.
The second-order empirical process allowsus to approximate the estimator by the estimator obtained from a weighted least squares problem,which in turn has an explicit bias and variance decomposition as done in the existing literature.
However, the theoretical drawback here is that this strategy currently does not allow us to obtain theoptimal rate for the entire range α ∈ (1 , ∞ ) (we require α to be sufficiently large).
It may be the casethat some other entirely different proof strategy is required to get a more elegant result.
In practice, it is always hard to estimate extreme conditional quantiles with τ very close to 0 or 1.
For any fixed τ in the open interval (0 , 1) , we do not have theoretical problems since we alwaysrequire m to be much smaller than N and thus typically the local sample size n = N/M is still quitelarge in the big data setting.
Theoretically, the statistical convergence rate inversely depends on thedensity (this is well-known in the parametric setting, although in the nonparametric setting we arenot aware of explicit mentioning of this in the literature).
At the tail of the distribution, the densityis typically small and thus the statistical error is large, which can also be seen from our simulationresults for τ = 0 . 1 and 0.9 in the Supplementary Appendix.
Extreme conditional quantile regressionis a separate discipline of its own and has been studied, for example, in Chernozhukov and Du (2008);
Wang et al.
; Wang and Li (2013).
However, even in the non-distributed setting we are notaware of any theoretical study of this problem in the framework of RKHS and it remains unclearwhether some techniques can be adapted to this setting.
One possible negative impact of the research is that the simple divide-and-conquer method doesnot consider the privacy issue in any mathematical rigorous way, for example using the differentialprivacy concept, which is a concern when using such a simple method.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
conclusion
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] the implementa-
tion is straightforward and is not an important aspect of the work.
We can share the
code upon request if the paper is accepted.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] But only in the last table for real data.
Other tables are
purposely made more succinct and informative without standard errors, which seemssufficient in this work focusing more on theory, and allows us to see the pattern in thetable more easily.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No] Such information does not seem to
add value to this work.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
N/A] e
) Did you discuss whether the data you are using
curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix A: Additional numerical results for one-dimensional case
Here we report the simulation results under the setup of the main paper for τ ∈{ 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Appendix B: Numerical results for three-dimensional case
Here consider a simulation with three-dimensional predictor.
The sample is generated from themodel y i
=
f 0 ( x i )
+
(1 + x 2 i 1 ) σ ( ϵ i − F − 1 ( τ )) , where x i = ( x i 1 , x i 2 , x i 2 ) is uniformly generatedfrom [0 , 1] 3 , f 0 ( x ) = 4 exp {−
x 21 + x 22 } − 4 x 3 , and here F denotes a student’s t-distribution with4 degrees of freedom.
Gaussian kernel is used with bandwidth specified as the median distancebetween two predictors.
Other settings are the same as the one-dimensional case.
The results forτ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } are reported in Tables 10–14.
Qualitatively, the interpretations of theresults are the same as before, distributed learning using simple averaging works satisfactorily, witherrors decreasing with the increase of either m or n .
[Table]
[Table] [Table]
Appendix C: Airline on-time performance data
We now use the air on-time performance data available at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/HG7NV7 (Harvard Dataverse, 2008) whichconsists of flight arrival and departure details for all commercial flights within the USA, from October1987 to April 2008.
The aim is to predict the delay in minutes of the aircraft.
After some initialexploration, we use 3 predictors, including ArrivalTime, DeparturetTime and Distance.
This is a largedataset, and we randomly select N = 10000 observations with another 5000 as validation data for
[Table] [Table]
tuning parameter selection.
The quantile regression model is fitted to the data with m ∈ { 20 , 50 , 100 } .
We see that the errors decrease with increase of m .
Because here the true f 0 is unknown, the errormeasure is the average loss for another randomly selected 5000 observations.
The whole procedure isrepeated 100 times and the results are reported in Table 15.
[Table]
Appendix D: Proof of Lemmas
Proof of Lemma 1.
By the standard symmetrization argument (Pollard, 1984), we have
E
[Equation]
] |
[Equation]
where the second to last inequality follows from the contraction inequality for the Rademachercomplexity (see, e.g., Theorem 2.2 of Koltchinskii (2011)), and the last bound follows from (4).
For the left-hand side of (A.1), since (using the Lipschitz continuity of ρ τ )
[Equation]
and
[Equation]
using the concentration inequality (see, e.g., the Bousquet bound in Chapter 2 of Koltchinskii (2011)),
[Equation]
with probability at least 1 − e − Ct .
We can then set t = N R 2 ( u ) /u 2 to get
[Equation]
≤ C R ( u
[Equation] [Equation]
with probability at least 1 − e − CN R 2 ( u ) /u 2 , which finishes the proof.
□
Proof of Lemma 2.
Define the class of functions G = { g ( x, y, f )
=
ρ τ ( y − f ( x )) − ρ τ ( y − f 0 ( x ))
+ ( τ − I { y − f 0 ( x ) ≤ 0 } )
( f ( x ) − f 0 ( x )) :
∥ f − f 0 ∥ ≤ u, ∥ f − f
∥ H ≤ 1 } .
Obviously | g ( x, y, f 1 ) − g ( x, y, f 2 ) | ≤ C | f 1 ( x ) − f 2 ( x ) | and thus the covering number of G is bounded by
[Equation]
Suppose ∥
f − f 0 ∥ ≤ u and ∥ f − f
∥ H ≤ 1 .
We can also easily see that | g ( x, y, f ) | ≤ C | f ( x ) −f 0 ( x ) | · I {| e | ≤ | f ( x ) − f 0 ( x ) |} .
Thus we have ∥ g ( x, y, f )
∥
≤ Cu ∥ f − f
[Equation]
(using the sup-norm assumption) and ∥ g ( x, y, f ) ∥ ∞ ≤ C ∥ f − f 0 ∥ ∞ ≤ Cu 1 − 1 /α .
Using Theorem 3.12 of Koltchinskii (2011), which provides an upper bound of the Rademachercomplexity in terms of the covering number, we get
[Equation]
Using Talagrand’s concentration inequality, we have with probability 1 − e − t ,
[Equation]
and setting t = u − 2 s/α proves the lemma.
□