Distributed Learning of Conditional Quantiles in the Reproducing Kernel Hilbert Space
Abstract
We study distributed learning of nonparametric conditional quantiles with Tikhonov regularization in a reproducing kernel Hilbert space (RKHS).
Although distributed parametric quantile regression has been investigated in several existing works, the current nonparametric quantile setting poses different challenges and is still unexplored.
The difficulty lies in the illusive explicit bias-variance decomposition in the quantile RKHS setting as in the regularized least squares regression.
For the simple divide-and-conquer approach that partitions the data set into multiple parts and then takes an arithmetic average of the individual outputs, we establish the risk bounds using a novel second-order empirical process for quantile risk.
Introduction
Distributed learning has attracted significant attention recently with the increasing popularity of distributed systems and distributed data.
Although the algorithmic aspects have gone far beyond the simple data parallelism paradigm that uses a simple divide-and-conquer strategy to carry out computations on each part of the data separately and then combine them in a finalizing step, the theoretical verification has lagged behind even though nowadays there are a lot of cases for which learning rates have been established, including Zhang et al.
; Rosenblatt and Nadler (2015); Balcan et al.
; Chang et al.
; Lee et al.
; Lin et al.
(2017); Jordan et al.
; Volgushev et al.
; Lian and Fan (2018).
It has been shown repeatedly for various models that the divide-and-conquer method can achieve the same learning rate as the standard estimator (assuming the whole data set is available on a single machine and it is computationally feasible to compute this central estimator based on the entire sample), as long as the number of partitions/machines is not too large.
In this work, we investigate the theoretical underpinning of the divide-and-conquer strategy for distributed learning of nonparametric quantile regression models in reproducing kernel Hilbert spaces (RKHS).
Under this framework, the distributed learning rate for kernel ridge regression with a least squares loss has been established in several works (Lin et al., 2017; Guo et al., 2017; Chang et al., 2017).
The quantile regression proposed by Koenker and Bassett Jr (1978) is an important class of models that can focus on the tail of the response distribution and is more robust the than least squares regression.
However, the rate for nonparametric distributed learning other than least squares loss in the RKHS has not been established.
The difficulty is that the nonparametric quantile estimator in the RKHS does not have an explicit form, which makes the decomposition of the mean squared error into bias and variance illusive, while such a decomposition is the key to establishing the rates in the divide-and-conquer method.
Previously, the quantile regression estimator in the RKHS studied in Li et al.
; Steinwart and Christmann (2011) based on standard empirical process theory does not make it possible to decompose the error into bias and variance and thus is not adaptable to the distributed setting, although the upper bound derived there can be roughly regarded as a sum of a bias term and a variance term.
On the other hand, for the fixed-dimensional parametric quantile regression, the Bahadur representation for the estimator is well-established in the literature which can be used in the distributed setting.
Even in high-dimensional parametric quantile regression, Zhao et al.
(2020) used a debiasing strategy so that after debiasing the bias term is sufficiently small and can be explicitly controlled.
Another related work on distributed high-dimensional parametric quantile regression is Chen et al.
(2020), which is more closely related to the gradient method.
To address the challenges in trying to control the bias and the variance, we use what we call the second-order empirical process involving the quantile loss function, which makes it possible to approximate more accurately the quantile loss by a weighted least squares loss, and we show that this approximation of the loss leads to approximation of the estimator (the weighted least squares estimator is close to the quantile estimator).
The rest of the paper is organized as follows.
In the next section, we state and prove the learning rates for both the central estimator and the distributed estimator.
Section 3 contains a simple numerical illustration, which is not used to verify the learning rate (which is difficult if not impossible) but just to illustrate the divide-and-conquer method can work reasonably.
The paper concludes in Section 4.
Conditional quantile estimation in the RKHS
For the response y , its conditional quantile at quantile level τ is denoted by Q y | x ( τ ) = f 0 ,τ ( x ) .
For simplicity of notation, we will suppress τ in our notations since we will focus on only an individual value of τ without modelling different quantiles jointly.
In the setting of RKHS, the unknown true function f 0 in the regression problem is assumed to be inside an RKHS H ⊂
L 2 ([0 , 1]) with an associated kernel K ( ., . )
, with the L 2 space defined by a probability measure ρ X for x on [0 , 1] .
Note that here we assume the covariate x ∈ [0 , 1] without much loss of generality.
In the following we use ⟨ , ⟩ H and ∥ .
∥ H to denote the RKHS inner product and the RKHS norm, respectively, while ∥ .
∥ is the L 2 -norm for functions.
The function f 0 is estimated by
[Equation]
where ( y i , x i ) , i = 1 , . .
, N is the entire sample from the true generating model with P ( y i ≤ f 0 ( x i )
| x i )
=
τ , ρ τ ( x ) = x ( τ − I { x ≤ 0 } ) is the quantile loss function, and λ > 0 is a tuning parameter.
(cid:98) f is referred to as the central estimator since it utilizes the entire sample.
In the distributed setting, the entire sample is distributed onto m machines and each machine computes a local estimate
[Equation]
where ( y i , x i ) , i = 1 . . .
, n (suppressing the dependence of the index on j ) is the sub-sample on the j -th machine.
We assume n = N/m is an integer and m is allowed to diverge with N .
The final distributed estimator is given by ¯ f = (cid:80) mj
=1 (cid:98) f j /m .
Let L : L 2 ([0 , 1])
→ L 2 ([0 , 1]) , L ( f ) = (cid:82) K ( x, . )
f ( x ) dρ
X
( x ) and define L γ : L 2 ([0 , 1])
→
L 2 ([0 , 1]) , L γ ( f ) =
(cid:82) γ (0 | x ) K ( x, . )
f ( x ) dρ
X
( x ) , where γ (0 | x ) is the conditional density of e i :
=
y i − f
( x i ) at 0 .
We will assume that γ (0 | x ) is bounded away from zero and infinity for all x and thus the properties of L and L γ are almost identical for our purpose.
We assume L is compact and has the spectral decomposition L = (cid:80) ∞
j =1 s j ϕ j ⊗ ϕ j , where s 1 ≥ s 2 ≥ · · · > 0 are the eigenvalues and ϕ j are the eigenfunctions.
We assume the following smoothness condition on the true f 0 :
[Equation]
for some g
[Equation]
H
[Equation] [Equation]
Assumption (3) is often referred to as the “source condition” in the literature.
L r is defined as the operator (cid:80)
j s rj ϕ j ⊗ ϕ j .
The condition can be written more explicitly as follows.
By Lemma 1.1.1 of Wahba (1990), any g ∈ H can be written as g = (cid:80) j g j ϕ j with (cid:80) j g 2 j /s j < ∞ , and
L r g = (cid:80) j s rj
g
j ϕ j .
Thus the assumption is the same as (cid:80)
j f 20 j /s 1+2 r j < ∞ , where f 0 j = ⟨ f 0 , ϕ j ⟩ is the Fourier coefficient of f 0 in the expansion.
Since s j → 0 , larger r implies faster decay of f 0 j , and the decay rate of the Fourier coefficient is naturally a characterization of its smoothness in functional analysis.
We establish the error bounds for the central regularized quantile estimator (2) as well as the distributed estimator.
For the central estimator, although quantile regression has been studied in several works such as Li et al.
; Zhang et al.
, these works simply assume f ∈ H and does not impose the additional assumption (3).
As for the least squares regression (Caponnetto and De Vito, 2007), such a regularity condition is expected to lead to faster rates.
However, different from the least squares case which relies on the closed-form solution of the regularized estimator, taking into account the assumption (3) requires entirely different technical tricks.
Throughout the paper, C denotes a generic positive constant.
The following assumptions are used.
(A1) s j ≤ Cj − α for some α > 1 .
The smoothness assumption (3) holds.
(A2)
We have the entropy condition log N ( ε, H (1) , L ∞ ) ≤ ( C/ε ) 2 /α , where H (1) =
{ f ∈ H :
∥ f
∥ H ≤ 1 } .
(A3) (Sup-norm assumption)
∥ f ∥ ∞ ≤ C ∥
f ∥ 1 − 1 α ∥ f ∥ 1 α H , ∀ f ∈ H .
(A4)
The conditional density of e = y − f 0 ( x ) given x , denoted by γ ( .
| x ) , and its derivative are uniformly bounded.
γ (0 | x ) is bounded away from zero uniformly for x ∈ [0 , 1] .
The assumption (A1) on the polynomial decay of eigenvalues is commonly assumed in the literature.
The entropy condition (A2) is stronger than assuming s j ≤ Cj − α .
Such an assumption is not uncommon and used in Müller and van de Geer (2015).
Theorem 6.26 of Steinwart and Christmann (2008) showed that it is satisfied if the RKHS has a kernel with derivatives of sufficiently high orders.
The sup-norm assumption is also used in Steinwart et al.
; Suzuki and Sugiyama (2013), which yields a better control for the sup-norm of a function in the RKHS.
The sup-norm assumption is satisfied if the RKHS is a Sobolev space or is continuously embeddable in a Sobolev space.
Assumptions (A1)-(A4) are required for the proof for the distributed estimator.
For the central estimator, we can remove (A2) and (A3) and add the following assumption.
[Equation]
Assumption (A2’) is a strong convexity condition of the population loss.
It can be shown that (A2’) is implied by (A3) and (A4).
It also holds if the conditional density γ ( .
| x ) is uniformly bounded away from zero ((A4) only requires γ (0 | x ) is uniformly bounded away from zero).
The following result gives the bound for the central estimator.
Theorem 1 Assume (A1), (A2’) and (A4) hold.
Setting λ ≍ N − α (2 r +1)
α +1 , we have ∥ (cid:98) f − f 0 ∥ =
O p ( N − (2 r +1)
α 2(2 r +1) α +2 ) .
We first state a lemma on the relevant empirical process.
The proof relies on the Rademacher complexity E [sup ∥ f ∥≤ u, ∥ f ∥ H ≤ 1 1 N (cid:80) Ni =1 σ i f ( x i )] , where σ i ∈ {− 1 , 1 } are i.i.d.
Rademacher variables.
The Rademacher complexity is a characterization of the complexity of the function space.
As shown in Mendelson (2002), we have a bound
[Equation]
where R ∗ ( u ) = (cid:113) 1 N (cid:80) ∞
j =1 min { s j , u 2 } .
When s j ≤ Cj − α , it is easy to see R ∗ ( u ) ≤ C R ( u ) with
R ( u ) :
= u 1 − 1 α √ N .
Lemma
For any u with R ( u ) ≤ u 2 , with probability at least 1 − e − CN R 2 ( u ) /u 2 ,
[Equation]
Sketches of Proof of Theorem 1.
We prove by way of contradiction.
Assuming ∥ (cid:98) f − f 0 ∥ is large, we try to establish
[Equation]
which will contradict the fact that (cid:98) f is the minimizer of the penalized loss.
The difference of the quantile loss on the two sides is approximated by its population version with the help of Lemma 1 and bounded below by assumption (A2’), while the difference of the penalties are carefully bounded by a nontrivial application of Young’s inequality for operators, taking into account the source condition.
Proof of Theorem 1.
In the proof we choose u that satisfies R ( u ) =
u 2+2 r (that is, u =
N − α 2 α
[Equation]
2 r +1)+2 , since R ( u ) =
u 1 − 1 α / √ N ).
By way of contradiction, suppose ∥ (cid:98) f − f 0 ∥ 2 ≥ Lu 2(1+2 r ) for some sufficiently large constant L .
This means
[Equation]
By the convexity of the functional in f in the above
[Equation]
we have
[Equation]
On the other hand, by assumption (A2’) and Lemma 1, with probability at least 1 − e − CNu 2+
[Equation]
r (note
Nu 2+
[Equation]
r = N
[Equation]
→
[Equation] [Equation]
In the 5th line above
[Equation]
we used that
[Equation]
where the first line used assumption (3), and the second to last line used Young’s inequality for positive operators λ 1 − 2 r L 2 r ≤ (1 − 2 r )
λ + 2 r L .
(6) implies ∥
f − f 0 ∥
≤ Cu 1+2 r , contradicting ∥ f − f 0 ∥ =
Lu 1+2 r for L sufficiently large, and the proof is complete.
□
The rate obtained for the central estimator is the same as the rate obtained for kernel ridge regression in Caponnetto and De Vito (2007); Lin et al.
and thus cannot be improved in general.
However, the proof of Theorem 1 does not yield clearly any bias and variance terms separately.
As mentioned in the introduction, separate control of bias and variance is usually the key step in obtaining the learning rate of the distributed estimator.
The empirical process of Lemma 1 cannot be utilized to get the approximate bias and variance terms, although such type of empirical process was popularly used in the literature (Belloni and Chernozhukov, 2011;
Li et al., 2007;
Lv et al., 2018).
To deal with this problem, for the distributed estimator, we use the empirical process techniques not on the difference of the quantile loss as in Lemma 1, but instead on a higher-order difference taking into account the gradient of the loss also, resulting in what we will call the second-order empirical process.
This will allow us to approximate the quantile loss by a weighted least squares loss, with results on distributed learning for the latter available from the literature (after minor modifications).
More specifically, we define g ( x, y, f )
=
ρ τ ( y − f ( x )) − ρ τ ( y − f 0 ( x ))
+ ( τ − I { y − f 0 ( x ) ≤ 0 } )
( f ( x ) − f 0 ( x )) .
If the loss were differentiable, g would be the residual in the first-order Taylor’s expansion of the loss.
For notational simplicity, we denote ϵ =
τ − I { e ≤ 0 } with e = y − f 0 ( x ) .
The key lemma that controls the difference between the quantile estimator and the weighted least squares estimator is the following result involving second-order empirical processes.
Due to the use of an additional term ( τ − I { y − f 0 ( x ) ≤ 0 } )( f ( x ) − f 0 ( x )) representing the first-order expansion in g , the bound in Lemma 2 is typically much smaller than that in Lemma 1 for risk difference.
Lemma
For any u ∈ (0 , 1) , with probability 1 − exp { u − 2 s/α } where s = 32 − 12 α > 1 ,
[Equation]
Based on Lemma 2, we can prove our main result for the distributed estimator.
It is shown that under suitable conditions, the learning rate for the distributed estimator matches that of the central estimator.
Theorem 2 Under assumptions (A1)-(A4), setting λ ≍
N − α (2 r +1)
α +1 , when m ≲
[Equation]
, we have
[Equation]
Remark
We note that, unfortunately, the exponent here is quite messy, and the first term in min (cid:110)
α 2 (2 r +1) − (8 r +2)
α +
r +1
α 2 (2 r +1)+2 α , 2 αr α (2 r +1)+ (cid:111) can be either bigger or smaller than the second term (since the explicit expression on when the first term is bigger is quite messy we do not choose to write it down).
Also the expression α 2 (2 r + 1) − (8 r + 2) α + 2 r + 1 may not even be positive (if it is not positive, the bound becomes void).
It is positive if α ≥ 3 .
Sketches of Proof of Theorem 2.
From Lemma 2, we can show that (cid:98) f j which minimizes the regularized empirical loss can be well-approximated by (cid:101)
f j that minimizes the sum of three terms:
P n (( f ( x ) − f 0 ( x )) ϵ )
(the first-order term in the definition of g ), the Tikhorov regularization term, and the second-order term from Taylor’s expansion of the expected loss.
By definition (cid:101)
f
j is a weighted kernel ridge estimator and the statistical bound for its average over m machines can be more easily established.
However, rigorous establishment of the closeness of (cid:98)
f j and (cid:101)
f j requires more detailed analysis involving multiple applications of Lemma 2.
Proof of Theorem 2.
By Knight’s identity
that ρ τ ( x − y ) −
ρ τ ( x ) = − y ( τ − I { x ≤ 0 } )
+ (cid:82) y
( I { x ≤ t } − I { x ≤ 0 } ) dt , we have
[Equation]
using E [( f ( x ) − f 0 ( x )) 3 ] ≤ ∥ f − f 0 ∥ ∞
E [( f ( x ) − f 0 ( x )) 2 ] and the sup-norm assumption, where
F ( .
| x ) is the conditional cdf of e = y − f 0 ( x ) .
Let (cid:101)
f
j be the minimizer of 12 E [ γ (0 | x )( f ( x ) − f 0 ( x )) 2 ] + P n (( f ( x ) − f 0 ( x ))
ϵ )
+ λ ∥ f ∥ 2 H .
This is basically the least squares problem when the true model is given by y = f 0 ( x ) + ϵ , with two differences.
One is that the loss is weighted by γ (0 | x ) and the other is that the first term is the population counterpart of (cid:80) i ( r (0 | x i )
( f ( x i ) − f
( x i )))
2 /n .
It is easy to see that (cid:101)
f j has a closed form (cid:101)
f
j =
( L γ + λ ) − 1 L γ f 0 +
( L γ + λ ) − 1 (cid:80) ni
=
ϵ i K
( x i ,. )
n .
If E [ γ (0 | x )( f ( x ) − f 0 ( x )) 2 ] is replaced by (cid:80) i ( r (0 | x i )
( f ( x i ) − f
( x i )))
2 /n , the minimizer would be (cid:101)
f ∗ j =
( L nγ + λ ) − 1 L nγ f
+ ( L nγ + λ ) − 1 (cid:80) ni
=
ϵ i K
( x i ,. )
n , where L nγ , defined as L nγ f = 1 n (cid:80) ni
=1 f ( x i ) K
( x i , . ) , is the empirical version of L γ .
We can see analysis of (cid:101)
f j would be similar to (and actually simpler than) the analysis of (cid:101)
f ∗ j , and the latter was basically the problem studied in Lin et al.
(2017), since we assumed γ (0 | x ) is bounded away from zero and infinity and it makes no difference in statistical analysis.
In particular, by Corollary 11 of Lin et al.
(2017), we can get ∥ 1 m (cid:80) mj
=
[Equation]
, if m ≤ N 2 αr α (2 r +1)+1 .
Furthermore, by combining Corollary 8 and Lemma 21 of Lin et al.
(2017), we also get the local estimates satisfy ∥ (cid:101) f
[Equation]
O
[Equation]
( N
[Equation]
Now we bound ∥ (cid:98) f
j − (cid:101) f j ∥ .
For simplicity of notation, define the function h ( u )
=
u 3 − 1 /α + u s √ n (cid:0) 1 u s (cid:1) 1 /α + u 1 − 1 /α n (cid:0) 1 u s (cid:1) 2 /α .
By Lemma 2 and (7), for any f ∈ H with ∥ f − f 0 ∥ ≤ Cu and ∥ f − f
∥
H ≤ 1 ,
[Equation]
−
[Equation] [Equation]
≤ Ch ( u ) .
Let u 1 = N 1 2[(2 r +1) α +1] √ n , then n
[Equation]
≤ Ch ( u 1 ) .
Taking the difference of the two displayed above,
[Equation]
≤ C ( h ( u ) + h ( u 1 )) .
Since (cid:101) f
j minimizes 12 P (cid:8) γ (0 | x )( f ( x ) − f 0 ( x )) 2 (cid:9) − 1 n (cid:80)
i ϵ i ( f ( x i ) − f 0 ( x i ))
+ λ ∥ f ∥ 2 H , the first order optimality condition for (cid:101) f
j yields
[Equation]
Thus we have
[Equation]
where the last step used (9).
Using this in (8)
[Equation]
we get
[Equation]
Let u 2 = n −
[Equation]
Note that this u 2 is the value of u that makes u 2 ≍ h ( u ) .
Let u 3 = (cid:112) h ( u 1 )
+ h ( u 2 ) .
Suppose 12 E [ γ (0 | x )( (cid:98) f
j ( x ) − (cid:101)
f j ( x )) 2 ] + λ ∥ (cid:98)
f
j − (cid:101) f
j ∥ 2 H ≥ Lu 23 for some L sufficiently large.
By (10), and using the same arguments as in the proof of Theorem 1 that lead to (5), there exists some f ∈ H with 12 E [ γ (0 | x )( f ( x ) − (cid:101)
f j ( x )) 2 ] + λ ∥
f − (cid:101) f j ∥ 2 H =
Lu 23 , and also satisfying 1 n (cid:80) i ρ τ
( y i − f ( x i ))
+
λ ∥ f ∥ 2 H ≤ 1 n (cid:80) i
ρ τ
( y i − (cid:101)
f j ( x i ))
+ λ ∥ (cid:101) f j ∥ 2 H , such that
[Equation]
Because the left hand side above is Lu 23 , this would lead to contradiction if L is large enough.
To see this, we note that
Lu 2 3 > Ch ( √ Lu 3 ) since u 3 > (cid:112) h ( u 2 ) =
u 2 , and also Lu 2 3 >
Ch ( u 1 ) since u 23 = h ( u 1 ) + h ( u 2 ) >
h ( u 1 ) .
Based on the previous arguments, we then have ∥ (cid:98)
f
j − (cid:101) f j ∥ ≤ Cu 3 .
Finally, the distributed estimator satisfies
[Equation]
As mentioned before, when m ≤ N
[Equation]
+1 , the second term above has the optimal rate ∥
[Equation]
To obtain the same rate for ∥ ¯ f − f 0 ∥ , we would require
[Equation]
Using n = N/m , (11) is equivalent to saying m ≤ N α 2(
r +1)
−
[Equation]
r +2)
α +
[Equation]
In summary, if
[Equation]
N
[Equation]
the optimal rate is achieved.
□
Remark 2 As stated in the theorem, for ¯ f to achieve the same rate as the non-distributed estimator, we need the constraint m ≲ N min
[Equation]
On the other hand, from the proof, we have the convergence rate N −
[Equation]
+
[Equation]
N/m ) −
[Equation]
even if the bound for m is not satisfied (when the bound is not satisfied, we just have a non-optimal rate).
In practice, of course it is hard to check whether a specific pair ( m, N ) satisfies the bound since it is asymptotic in nature.
On the other hand, in the setting that data are physically located on m machines and one is unwilling to share data across different machines, then we have no other option and just accept this given m .
In this situation, we do not have to be concerned with the choice of m .
Some numerical illustrations
Here we illustrate the distributed quantile regression in relatively simple simulations.
For a value τ ∈ (0 , 1) , the sample is generated from the model
y i =
f 0 ( x i ) +
(1 + x i ) σ ( ϵ i − Φ − 1 ( τ )) , where x i are generated uniformly on [0 , 1] , ϵ i ∼ N (0 , 1) , Φ − 1 is the quantile function of the standard normal distribution so that f ( x i ) is indeed the conditional τ -quantile of y i .
We set f 0 ( x ) = sin(2 πx ) and σ = 0 .
The simulations are carried out for different combinations of n ∈ { 32 , 64 , 128 , 256 , 512 , 1024 } and m ∈ { 1 , 2 , 4 , 8 , 16 , 32 } , with the total sample size of N = nm .
Note that m = 1 actually corresponds to the central estimator.
For the RKHS, we use the Sobolev space of the second order.
The tuning parameter λ is chosen to minimize the errors on independently generated test data.
In each setting, 100 repetitions are used to compute the average squared estimation error ∥
¯ f − f 0 ∥ 2 .
Choice of λ is an important practical matter.
Theorem 2 shows the choice of λ ∼ N − α (2 r +1)
α
+ depends on the total sample size N , which indicates that tuning parameter chosen based on local data is not optimal.
In our simulations, we use independently generated data to select the optimal λ that ensures ¯ f ( ¯ f implicitly depends on λ ) is the one that achieves the smallest prediction error on the test data.
When independently generated data is not available, either hold-out data or global K -fold cross-validation can be used, which requires coordination of the central machine.
If the hold-out data is maintained by the central machine, the central machine can evaluate and compare different values of λ .
If K -fold cross-validation is carried out on the local machines, the central machine needs to send ¯ f back to the local machines so that the local machines can calculate the cross-validation errors, and then send the errors to the central machine.
This needs to be done K times for K -fold CV.
Xu et al.
(2018, 2019) developed a generalized approximate cross-validation criterion in the least-squares case and established its asymptotic optimality.
We expect that similar criterion developed for quantile regression (Yuan, 2006) can be adapted in our case.
However, even in the non-distributed setting,
Yuan (2006) did not establish the asymptotic properties of the criterion, mainly investigating its performance from a computational viewpoint.
Thus it remains an open question whether such a criterion has a desirable asymptotic property.
The investigation of such problems requires additional study by itself and thus we do not use this in the current work.
The simulation results are shown in Table 1 for τ = 0 .
We can examine the table in several ways.
First, each row of the table indicates what happens when one fixes the number of observations on each machine while increasing the number of machines (the total sample size of course also increases).
In this case, we see the errors decrease with the number of machines since the total sample size increases.
Second, each column of the table indicates what happens if the number of machines are fixed while the number of observations increases proportionally with the total sample size.
Here we also see the error decreases.
Third, the results along the anti-diagonal directions shows the results when the total sample size N is fixed while we change the number of machines m .
For example, the numbers shaded in yellow all corresponds to N = 512 while the numbers shaded in blue corresponds to N = 1024 .
Here we see that the error in general increases with the number of machines (up to some random fluctuations).
This is natural since the central estimator should perform best if it is feasible to compute.
Finally, we observe that the table shows the divide-and-conquer strategy is useful when N is large.
For example, in the table, the smallest error is achieved when n = 1024 , m = 32 ,
[Table]
corresponding to N = 32768 .
It would be hard to deal with such a large sample size in the standard implementation (the kernel method is always computationally burdensome) without using some advanced techniques to approximate the solution, if such techniques exist at all.
The simulation results for other values of τ are presented in Appendix A. We also perform an additional simulation with 3-dimensional predictors in Appendix B.
Conclusion
In this paper, we established the convergence rate of the nonparametric quantile estimator in the distributed learning setting under the RKHS framework.
The second-order empirical process allows us to approximate the estimator by the estimator obtained from a weighted least squares problem, which in turn has an explicit bias and variance decomposition as done in the existing literature.
However, the theoretical drawback here is that this strategy currently does not allow us to obtain the optimal rate for the entire range α ∈ (1 , ∞ ) (we require α to be sufficiently large).
It may be the case that some other entirely different proof strategy is required to get a more elegant result.
In practice, it is always hard to estimate extreme conditional quantiles with τ very close to 0 or 1.
For any fixed τ in the open interval (0 , 1) , we do not have theoretical problems since we always require m to be much smaller than N and thus typically the local sample size n = N/M is still quite large in the big data setting.
Theoretically, the statistical convergence rate inversely depends on the density (this is well-known in the parametric setting, although in the nonparametric setting we are not aware of explicit mentioning of this in the literature).
At the tail of the distribution, the density is typically small and thus the statistical error is large, which can also be seen from our simulation results for τ = 0 . 1 and 0.9 in the Supplementary Appendix.
Extreme conditional quantile regression is a separate discipline of its own and has been studied, for example, in Chernozhukov and Du (2008);
Wang et al.
; Wang and Li (2013).
However, even in the non-distributed setting we are not aware of any theoretical study of this problem in the framework of RKHS and it remains unclear whether some techniques can be adapted to this setting.
One possible negative impact of the research is that the simple divide-and-conquer method does not consider the privacy issue in any mathematical rigorous way, for example using the differential privacy concept, which is a concern when using such a simple method.
Acknowledgments and Disclosure of Funding
This project is partially supported by NSFC Project 11871411 at the Shenzhen Research Institute,
City University of Hong Kong; and partially supported by the Hong Kong Research Grants Council (RGC) General Research Fund under Grant 11300519, 11300721, 11311822 and 11301423.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See conclusion (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See conclusion (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] the implementation is straightforward and is not an important aspect of the work.
We can share the code upon request if the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] But only in the last table for real data.
Other tables are purposely made more succinct and informative without standard errors, which seems sufficient in this work focusing more on theory, and allows us to see the pattern in the table more easily.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Such information does not seem to add value to this work.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A: Additional numerical results for one-dimensional case
Here we report the simulation results under the setup of the main paper for τ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Appendix B: Numerical results for three-dimensional case
Here consider a simulation with three-dimensional predictor.
The sample is generated from the model y i
=
f 0 ( x i )
+
(1 + x 2 i 1 ) σ ( ϵ i − F − 1 ( τ )) , where x i = ( x i 1 , x i 2 , x i 2 ) is uniformly generated from [0 , 1] 3 , f 0 ( x ) = 4 exp {− x 21 + x 22 } − 4 x 3 , and here F denotes a student’s t-distribution with 4 degrees of freedom.
Gaussian kernel is used with bandwidth specified as the median distance between two predictors.
Other settings are the same as the one-dimensional case.
The results for τ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } are reported in Tables 10–14.
Qualitatively, the interpretations of the results are the same as before, distributed learning using simple averaging works satisfactorily, with errors decreasing with the increase of either m or n .
[Table]
size is N = nm .
[Table]
[Table]
[Table]
[Table]
Appendix C: Proof of Lemmas
Proof of Lemma 1.
By the standard symmetrization argument (Pollard, 1984), we have
E
[Equation]
≤ C R ( u ) [Equation] (A.1) where the second to last inequality follows from the contraction inequality for the Rademacher complexity (see, e [Equation]
g., Theorem
[Equation]
2 of Koltchinskii (2011))
[Equation] and the last bound follows from (4) [Equation] For the left-hand side of (A.1), since (using the Lipschitz continuity of ρ τ )
[Equation]
f 0 ∥ + ∥ f
[Equation]
f 0 ∥ H (cid:12)(cid:12)(cid:12)(cid:12) ≤ C [Equation] f − f
[Equation]
∞ u
[Equation]
f
[Equation]
+
[Equation]
f
[Equation]
f
[Equation]
H ≤ C, and
V ar (cid:18) ρ τ ( y − f ( x )) − ρ τ ( y − f 0 ( x ))
u − 1 ∥ f
[Equation]
{+f 0 ∥ + ∥ f − f 
+}
∥ H (cid:19) ≤ CV ar (cid:18) f ( x )
[Equation]
f 0 ( x ) u −
[Equation]
∥
[Equation]
− f
[Equation]
f
[Equation]
f
[Equation]
H (cid:19) ≤ Cu 2 , using the concentration inequality (see, e.g., the Bousquet bound in Chapter 2 of Koltchinskii (2011)),
[Equation]
with probability at least 1 − e − Ct .
We can then set t = N R 2 ( u ) /u 2 to get sup
[Equation]
≤ C R ( u ) ,
[Equation]
with probability at least 1 − e − CN R 2 ( u ) /u 2 , which finishes the proof.
□
Proof of Lemma 2.
Define the class of functions G = { g ( x, y, f )
=
ρ τ ( y − f ( x )) − ρ τ ( y − f 0 ( x ))
+ ( τ − I { y − f 0 ( x ) ≤ 0 } )
( f ( x ) − f 0 ( x )) :
∥ f − f 0 ∥ ≤ u, ∥ f − f
∥ H ≤ 1 } .
Obviously | g ( x, y, f 1 ) − g ( x, y, f 2 ) | ≤ C | f 1 ( x ) − f 2 ( x ) | and thus the covering number of G is bounded by
[Equation]
Suppose ∥
f − f 0 ∥ ≤ u and ∥ f − f
∥ H ≤ 1 .
We can also easily see that | g ( x, y, f ) | ≤ C | f ( x ) − f 0 ( x ) | · I {| e | ≤ | f ( x ) − f 0 ( x ) |} .
Thus we have ∥ g ( x,
[Equation]
Cu s (using the sup-norm assumption) and ∥ g ( x, y, f ) ∥ ∞ ≤ C ∥ f − f 0 ∥ ∞ ≤ Cu 1 − 1 /α .
Using Theorem 3.12 of Koltchinskii (2011), which provides an upper bound of the Rademacher complexity in terms of the covering number, we get
[Equation]
Using Talagrand’s concentration inequality, we have with probability 1 − e − t ,
[Equation]
and setting t = u − 2 s/α proves the lemma.
□