B ENEFIT OF I NTERPOLATION IN N EAREST N EIGHBOR A LGORITHMS
Anonymous authors
Paper under double-blind review
A BSTRACT
The over-parameterized models attract much attention in the era of data science and deep learning.
It is empirically observed that although these models, e.g. deep neural networks, over-ﬁt the training data, they can still achieve small testing error, and sometimes even outperform traditional algorithms which are designed to avoid over-ﬁtting.
The major goal of this work is to sharply quantify the beneﬁt of data interpolation in the context of nearest neighbors (NN) algorithm.
Specifically, we consider a class of interpolated weighting schemes and then carefully characterize their asymptotic performances.
Our analysis reveals a U-shaped performance curve with respect to the level of data interpolation, and proves that a mild degree of data interpolation strictly improves the prediction accuracy and statistical stability over those of the (un-interpolated) optimal k NN algorithm.
This theoretically justiﬁes (predicts) the existence of the second U-shaped curve in the recently discovered double descent phenomenon.
Note that our goal in this study is not to promote the use of interpolated-NN method, but to obtain theoretical insights on data interpolation inspired by the aforementioned phenomenon.
I NTRODUCTION
Classical statistical learning theory believes that over-ﬁtting deteriorates prediction performance: when the model complexity is beyond necessity, the testing error must be huge.
Therefore, various techniques have been proposed in literature to avoid over-ﬁtting, such as early stopping, dropout and cross validation.
However, recent experiments reveal that even with over-ﬁtting, many learning algorithms still achieve small generalization error.
For instances, Wyner et al.
(2017) explored the over-ﬁtting in AdaBoost and random forest algorithms; Belkin et al. (2019a) discovered a double descent phenomenon in random forest and neural network: with growing model complexity, testing performance ﬁrstly follows a (conventional) U-shaped curve, and as the level of overﬁtting increases, a second descent or even a second U-shaped testing performance curve occurs.
To theoretically understand the effect of over-ﬁtting or data interpolation, Du & Lee (2018); Du et al.
(2019; 2018); Arora et al.
Xie et al.
(2017) analyzed how to train neural networks under over-parametrization, and why over-ﬁtting does not jeopardize the testing performance; Belkin et al.
(2019c) constructed a Nadaraya-Watson kernel regression estimator which perfectly ﬁts training data but is still minimax rate optimal; Belkin et al.
(2018) and Xing et al.
(2018) studied the rate of convergence of interpolated nearest neighbor algorithm (interpolated-NN); Belkin et al.
(2019b);
Bartlett et al.
(2019) quantiﬁed the prediction MSE of the linear least squared estimator when the data dimension is larger than sample size and the training loss attains zero.
Similar analysis is also conducted by Hastie et al.
for two-layer neural network models with a ﬁxed ﬁrst layer.
In this work, we aim to provide theoretical reasons on whether, when and why the interpolated-NN performs better than the optimal k NN, by some sharp analysis.
The classical k NN algorithm for either regression or classiﬁcation is known to be rate-minimax under mild conditions (Chaudhuri &
Dasgupta (2014)), say k diverges propoerly.
However, can such a simple and versatile algorithm still beneﬁt from intentional over-ﬁtting?
We ﬁrst demonstrate some empirical evidence below.
Belkin et al.
(2018) designed an interpolated weighting scheme as follows:
[Equation]
where x ( i ) is the i -th closest neighbor to x with the corresponding label y ( x ( i ) ) .
The parameter γ ≥ 0 controls the level of interpolation: with a larger γ > 0 , the algorithm will put more weights on the closer neighbors.
In particular when γ = 0 or γ = ∞ , interpolated-NN reduces to k NN or 1NN, respectively.
Belkin et al.
(2018) showed that such an interpolated estimator is rate minimax in the regression setup, but suboptimal in the setting of binary classiﬁcation.
Later, Xing et al.
(2018) obtained the minimax rate of classiﬁcation by adopting a slightly different interpolating kernel.
What is indeed more interesting is the preliminary numerical analysis (see Figure 1) conducted in the aforementioned paper, which demonstrates that interpolated-NN is even better than the rate minimax k NN in terms of MSE (regression) or mis-classiﬁcation rate (classiﬁcation).
This observation asks for deeper theoretical exploration beyond the rate of convergence.
A reasonable doubt is that the interpolated-NN may possess a smaller multiplicative constant for its rate of convergence, which may be used to study the generalization ability within the “over-parametrized regime.”
In this study, we will theoretically compare the minimax optimal k NN and the interpolated-NN (under (1)) in terms of their multiplicative constants.
On the one hand, we show that under proper smooth conditions, the multiplicative constant of interpolated-NN, as a function of interpolation level γ , is U-shaped.
As a consequence, interpolation indeed leads to more accurate and stable performance when the interpolation level γ ∈ (0 , γ d ) for some γ d > 0 only depending on the data dimension d .
The amount of beneﬁt (i.e., the “performance ratio” deﬁned in Section 2) follows exactly the same asymptotic pattern for both regression and classiﬁcation tasks.
In addition, the gain from interpolation diminishes as the dimension d grows to inﬁnity, i.e. high dimensional data beneﬁt less from data interpolation.
We also want to point out that there still exist other “non-interpolating” weighting schemes, such as OWNN, which can achieve an even better performance; see Section 3.4.
More subtle results are summarized in the ﬁgure below.
From Figure 2, we theoretically justify (predict) the existence of the U-shaped curve within the “over-ﬁtting regime” of the recently discovered double descent phenomenon by Belkin et al.
(2019a;b).
As complementary to Belkin et al.
Xing et al.
(2018), we further show in appendix that interpolated-NN reaches optimal rate for both regression and classiﬁcation under more general ( α, β ) -smoothness conditions in Section F.
In the end, we want to emphasize that our goal here is not to promote the practical use of this interpolation method given that k NN is more user-friendly.
Rather, the interpolated-NN algorithm is used to precisely describe the role of interpolation in generalization ability so that more solid theoretical arguments can be made for the very interesting double descent phenomenon, especially in the over-ﬁtting regime.
I NTERPOLATION IN N EAREST N EIGHBORS A LGORITHM
In this section, we review the interpolated-NN algorithm introduced by Belkin et al.
(2018) in more details.
Given x , we deﬁne R k +
( x ) to be the distance between x and its ( k +1)
th nearest neighbor.
W.O.L.G, we let X 1 to X k denote the (unsorted) k nearest neighbors of x , and let { R i ( x ) } ki =1 to be distances between x and X i .
Thus, based on the same argument used in Chaudhuri & Dasgupta (2014) and Belkin et al.
(2018), conditional on R k +
( x ) , X 1 to X k are iid variables whose support is a ball centered at x with radius
R k +1 ( x ) , and as a consequence, R 1 ( x ) to R k ( x ) are conditionally independent given R k +
( x ) as well.
Note that when no confusion is caused, we will write R i ( x ) as
R i .
Thus, the weights of the k neighbors are deﬁned as
[Equation]
for i = 1 , . . .
, k and some γ ≥ 0 .
For regression models, denote η ( x ) as the target function, and Y ( x ) =
η ( x ) + (cid:15) ( x ) where (cid:15) ( x ) is an independent zero-mean noise with E (cid:15) ( x ) 2 :
= σ 2 ( x ) .
The regression estimator at x is thus (cid:98) η k,n,γ ( x ) =
k (cid:88) i =
W i
Y
i .
For binary classiﬁcation, denote η ( x ) =
P ( Y = 1 | X = x ) , with g ( x ) = 1 { η ( x ) > 1 / 2 } as the
Bayes estimator.
The interpolated-NN classiﬁer is deﬁned as
[Equation]
As discussed previously, the parameter γ controls the level of interpolation: a larger value of γ leads to a higher degree of data interpolation.
We adopt the conventional measures to evaluate the theoretical performance of interpolated-NN given a new test data ( X, Y ) :
Regression: MSE ( k, n, γ ) =
E (( (cid:98) η k,n,γ ( X ) − η ( X )) 2 ) .
Classiﬁcation: Regret ( k, n, γ ) = P ( (cid:98) g k,n,γ ( X ) (cid:54) = Y ) − P ( g ( X ) (cid:54) = Y ) .
Q UANTIFICATION OF I NTERPOLATION E FFECT 3. M ODEL A SSUMPTIONS
Recent works by Belkin et al.
(2018) and Xing et al.
(2018) conﬁrm the rate optimality of MSE and regret for interpolated-NN under mild interpolation conditions.
Two deeper questions (hinted by
Figure 1) we would like to address are whether and how interpolation strictly beneﬁts NN algorithm, and whether interpolation affects regression and classiﬁcation in the same manner.
To facilitate our theoretical investigation, we impose the following assumptions:
A.1 X is a d -dimensional random variable on a compact set X with boundary ∂ X .
A.2 For classiﬁcation, S = { x | η ( x ) = 1 / 2 } is non-empty.
A.3 d − 3 γ ≥ C > 0 for some constant C .
A.4 For classiﬁcation, η is continuous in some open set containing X .
The third-order derivative of η is bounded when η ( x ) = 1 / 2 ± c 0 for a small constant c 0 > 0 .
The gradient ˙ η ( x ) (cid:54) =when η ( x ) = 1 / 2 , and with restriction on x ∈ ∂ X , ˙
∂η ( x ) (cid:54) =
if η ( x ) = 1 / 2 and x ∈ ∂ X .
A.
For classiﬁcation, density of X , denoted as f , is twice differentiable and ﬁnite.
A.6 For regression, the third-order derivative of η is bounded for all x .
A.7 For regression, sup x ∈X σ 2 ( x ) is ﬁnite and σ 2 ( x ) has ﬁnite ﬁrst-order derivative in x .
The above assumptions (except A.3) are mostly derived from the framework established by Samworth et al.
Note that the additional smoothness required in η and f is needed to facilitate the asymptotic study of interpolation weighting scheme.
We also want to point out that these assumptions are generally stronger than those used in Chaudhuri & Dasgupta (2014), but necessary to ﬁgure out the multiplicative constant.
Further discussions regarding the conditions can be found in
Remark 3 in appendix.
3. M AIN T HEOREM
The following theorem examines the asymptotic performance ratios of MSE and Regret between interpolated-NN and k NN and discovers that these ratios (under their respective optimal choice of k ) converges to a function of ( d, γ ) only.
In particular, a U-shaped curve is revealed where the ratio is smaller than 1 when γ ∈ (0 , γ d ) for some γ d < d/ 3 .
Deﬁne the minimum MSE and Regret over k ∈ ( n β , n 1 − 4 β/d ) as follows:
MSE
[Equation]
and
[Equation]
Theorem 1 asymptotically compares the interpolated-NN and k NN, i.e., γ = 0 , in terms of the above measures.
Interestingly, it turns out that the performance ratio, deﬁned as
[Equation]
is a function of d and γ only, independent of the underlying data distribution.
Note that PR ( d, γ ) is just the ratio of multiplicative constants before the minimax rate of interpolated-NN and k NN.
Theorem 1 For regression, suppose that assumptions A.1, A.3, A.6, and A.7 hold. For classiﬁcation, under A.1 to A.5, for any γ ∈ [0 , d/ 3) ,
[Equation]
Note that MSE ( n, 0) /Regret
( n, 0) is the optimum MSE/Regret for k NN.
The proof Theorem 1 are postponed to appendix (Section D).
When k can be chosen adaptively based on γ , we can address the second question that interpolation affects regression and classiﬁcation in exactly the same manner through PR ( d, γ ) .
In particular, this ratio exhibits an interesting U-shape of γ for any ﬁxed d .
Speciﬁcally, as γ increases from 0 , PR ( d, γ ) ﬁrst decreases from 1 and then increases above 1 ; see Figures 2 and 3.
Therefore, within the range (0 , γ d ) for some γ d only depending on dimension d , PR ( d, γ ) < 1 , that is, the interpolated-NN is strictly better than the k NN.
Given the imposed condition that γ < d/ 3 , Some further calculations reveal that γ d < d/ 3 when d ≤ 3 ; γ d = d/ 3 when d ≥ 4 .
Remark
It is easy to show
that lim d →∞ [min γ<d/ 3 PR ( d, γ )] = 1 .
This indicates that high dimensional model beneﬁts less from interpolation, or said differently, high dimensional model is less affected by data interpolation.
This phenomenon can be explained by the fact that, as d increases,
R i /R k
+
→ 1 due to high dimensional geometry.
Remark
The optimum k , which leads to the best MSE/regret, depends on the interpolation level γ .
Thus, we denote it as k γ .
As shown in the appendix, k γ (cid:16) k 0 (:=
k γ =0 ) (cid:16) n 4 d +4 , but k γ /k 0 > 1 for γ > 0 , i.e., interpolated-NN needs to employ slightly more neighbors to achieve the best performance.
Empirical support for this ﬁnding can be found in Section A of appendix.
If we insist using the same k (cid:16) n 4 d +4 for interpolated-NN and k NN, we can still verify that
MSE ( k, n, γ ) / MSE ( k, n, 0) < 1 and Regret ( k, n, γ ) / Regret ( k, n, 0) < 1 , when γ < γ x for some γ x depending on the distribution of X and η .
3. S TATISTICAL S TABILITY
In this section, we will explore how the interpolation affects the statistical stability of nearest neighbor classiﬁcation algorithms.
This is beyond the generalization results obtained in Section 3.2.
In short, if we choose the best k in k NN and apply it to the interpolated-NN, then k NN will be more stable; otherwise, the interpolated-NN will be more stable for γ ∈ (0 , γ d ) if the k is allowed to be chosen separately and optimally based on γ .
For a stable classiﬁcation method, it is expected that with high probability, the classiﬁer can yield the same prediction when being trained by different data sets sampled from the same population.
As a result, Sun et al.
(2016) introduced a type of statistical stability, classiﬁcation instability (CIS), which is different from the algorithmic stability in the literature (Bousquet & Elisseeff, 2002).
Denote Dand D 2 as two i.i.d.
training sets with the same sample size n .
The CIS is deﬁned as:
[Equation]
Hence, a larger value of CIS indicates that the classiﬁer is less statistically stable.
In practice, we need to take into account of mis-classiﬁcation rate and classiﬁcation instability at the same time.
Therefore, we are interested in comparing the stability between interpolated-NN and k -NN only when the regrets of both algorithms reach their optimal performance under respective optimal k choices.
Theorem 2 below illustrates how CIS is affected by interpolation through k , n and γ .
Theorem 2 Under the conditions in Theorem 1, the CIS of interpolated-NN is derived as
[Equation]
The proof of Theorem 2 is postponed to Section E in appendix.
Similarly, Corollary 3 asymptotically compares CIS between interpolated-NN and k NN.
Corollary {+
+} Following the conditions in Theorem 2, when the same k value is used for k NN and interpolated-NN, then as n → ∞ ,
[Equation]
On the other hand, if we choose optimum k ’s for k NN and interpolated-NN respectively, i.e. k γ = arg min k Regret ( k, n, γ ) , when n → ∞ , we have
[Equation]
Therefore, when γ ∈ (0 , γ d ) , interpolated-NN with optimal k has higher accuracy and stability than k -NN at the same time.
From Corollary 3, the interpolated-NN is not as stable as k NN if the same number of neighbors is used in both algorithms.
However, this is not the case if an optimal k is chosen separately.
An intuitive explanation is that, under the same k , k NN has a smaller variance (more stable) given equal weights for all k neighbors; on the other hand, by choosing an optimum k , the interpolated-NN can achieve a much smaller bias, which offsets its performance lost in variance through enlarging k .
3. C ONNECTION WITH OWNN AND D OUBLE D ESCENT P HENOMENON
Samworth et al.
(2012) ﬁrstly worked out a general form of regret using a rank-based weighting scheme, and proposed the optimally weighted nearest neighbors algorithm (OWNN).
The OWNN is the best nearest neighbors algorithm in terms of minimizing MSE for regression (and Regret for classiﬁcation), when the weights of neighbors are only rank-based.
Combining Theorem 1 with Samworth et al.
(2012), we can further compare the interpolated-NN against OWNN as follows:
[Equation]
which is always smaller than 1 (just by deﬁnition).
Here R ( n, OWNN ) denotes the MSE/Regret of OWNN given its optimum k , and R ( n, γ ) denotes the one of interpolated-NN given its own optimum k choice.
It is interesting to note from the above ratio that that the advantage of OWNN is only reﬂected at the level of multiplicative constant, and further that the ratio converges to 1 as d diverges (just as the case of PR ( d, γ ) ; see Remark 1).
Thus, under ultra high dimensional setting, the performance differences among k NN, interpolated-NN and OWNN are almost negligible even at the multiplicative constant level.
We ﬁrst describe the framework of the recently discovered double descent phenomenon (e.g., Belkin et al., 2019a;b), and then comment our contributions (summarized in Figure 2) to it in the context of nearest neighbor algorithm.
Speciﬁcally, within the “classical regime” where exact data interpolation is impossible, the testing performance curve is the usual U-shape w.r.t.
model complexity; once the model complexity is beyond a critical point it thus enters the “over-ﬁtting regime,” the testing performance will start to decrease again as severeness of data interpolation increases, which is so-call “double descend”.
In the context of nearest neighbors algorithms, different weighting schemes may be viewed as a surrogate of modeling complexity.
For OWNN, though it allocates more weights on closer neighbors, while none of the weights exceeds (1 + d/ 2) /k .
Thus, OWNN is never an interpolation weighting scheme.
From this aspect, k -NN and OWNN both belong to the “classical regime,” while interpolated-NN is within the “over-ﬁtting regime.”
In particular, the testing performance of OWNN reaches the minimum point of the U-shaped curve inside the “classical regime.”
Deviation from this optimal choice of weight leads to the increase of the MSE/Regret within this “classical regime.”
After the “over-ﬁtting regime” is reached by the interpolated-NN, say from γ = 0 in Figure 2, the
MSE/Regret decreases as the interpolation level γ increases within the range (0, γ d ) and ascends again when γ > γ d (if the dimension allows γ d < d/ 3 ), forming the second U-shaped curve in
Figure 2.
Therefore, we obtain an overall W-shaped performance curve with theoretical guarantee , which coincides the empirical ﬁnding of Belkin et al.
(2019b)
for over-parametrized linear models.
N UMERICAL E XPERIMENTS
In this section, we will present several simulation studies to justify our theoretical discoveries for regression, classiﬁcation and stability performances of the interpolated-NN algorithm, together with some real data analysis.
4. S IMULATIONS
We aim to estimate the performance ratio curve by data simulation and compare it with the theoretical curve PR ( d, γ ) .
The second simulation setting in Samworth et al.
(2012) is adopted here.
Speciﬁcally, the joint distribution of ( X =
( X ( i ) ) di
=1 , Y ) follows P ( Y = 1)
=
P ( Y = 0)
= 1 / 2 , f
( X | Y = 0) =
(cid:81) i [ 12 φ ( X ( i ) ; 0 , 1) + 12 φ ( X ( i ) ; 3 , 2)] and f ( X | Y = 1) =
(cid:81) i [ 12 φ ( X ( i ) ; 1 . 5 , 1) + 1 2 φ ( X ( i ) ; 4 . 5 , 2)] , where φ ( · ; µ, σ 2 ) denotes the density of N ( µ, σ 2 ) .
The sample size n = 2 10 and dimension d = 2 , 5 .
The interpolated-NN regressor and classiﬁer were implemented under different choices of γ/d = 0 , 0 . 05 , 0 . 1 , . . .
35 and k = 1 , . .
. , n .
For regression, the MSE was estimated based on 100 repetitions, and for classiﬁcation, the Regret was based on 500 repetitions.
When n = 1024 , the Regret/MSE ratio for different γ/d is shown in Figure 3.
Here Regret ratio is deﬁned by Regret ( n, γ ) / Regret ( n, 0) , the MSE ratio is deﬁned by MSE ( n, γ ) / MSE ( n, 0) .
The trends for theoretical value and simulation value are mostly close.
The small difference is mostly caused by the small order terms in the asymptotic result and shall vanish if larger n is used.
Note that γ/d = 0 .
35 is outside our theoretical range γ/d < 1 / 3 , but the performance is still reasonable in our numerical experiment.
We further estimate CIS by training two classiﬁers based on two different simulated data sets of 1024 samples.
The CIS was estimated by calculating the proportion of testing samples that have different prediction labels, that is
[Equation]
The CIS result is shown in Figure 4.
One can see that when γ is small, the simulated CIS ratio decreases in a similar manner as the asymptotic value, while simulated value will increase when γ gets larger.
This pattern is the same as the theoretical result predicted in Theorem 2.
An additional experiment is postponed to appendix, which shows how the MSE and optimum k changes in γ and n where n = 2 i for i = 6 , . . .
4. R EAL D ATA A NALYSIS
In real data experiment, we compare the classiﬁcation accuracy of interpolated-NN with k NN.
Five data sets were considered in this experiment.
The data set HTRU2 from Lyon et al.
(2016) uses 17,897 samples with 8 continuous attributes to classify pulsar candidates.
The data set Abalone contains 4,176 samples with 7 attributes.
Following Wang et al.
, we predict whether the number of rings is greater than 10.
The data set Credit (Yeh & Lien, 2009) has 30,000 samples with 23 attributes, and predicts whether the payment will be default in the next month given the current [Table] payment information.
The built-in data set of digits in sklearn (Pedregosa et al., 2011) contains 1,797 samples of 8 × 8 images.
For images in MNIST are 26 × 26 , we will use part of it in our experiment.
Both the data set of digit and MNIST have ten classes.
Here for binary classiﬁcation we group
to 4 as the ﬁrst class and 5 to 9 as the second class.
For each data set, a proportion of data is used for training and the rest is reserved to test the accuracy of the trained classiﬁers.
For Abalone, HTRU2, Credit and Digit, we use 25% data as training data and 75% as testing data.
For MNIST, we use randomly choosen 2000 samples as training data and 1000 as testing data, which is sufﬁcient for our comparison.
The above experiment is repeated for 50 times and the average testing error rate is summarized in Table 1.
For all data sets, the testing error of interpolated-NN (column “best γ/d ”) is always smaller than the k NN(column “ γ = 0 ”), which veriﬁes that nearest neighbor algorithm actually beneﬁts from interpolation.
C ONCLUSION
Our work precisely quantiﬁes how data interpolation affects the performance of nearest neighbor algorithms beyond the rate of convergence.
We ﬁnd that for both regression and classiﬁcation problems, the asymptotic performance ratios between interpolated-NN and k NN converge to the same value, which depends on d and γ only.
More importantly, when the interpolation level γ/d is within a reasonable range, the interpolated-NN is strictly better than k NN as it has a smaller multiplicative constant of the convergence rate, and it has a more stable prediction performance as well.
Classical learning framework opposes data interpolation as it believes that over-ﬁtting means ﬁtting the random noise rather than the model structures.
However, in the interpolated-NN, the weight degenerating occurs only on a nearly-zero-measure set, and thus there is only “local over-ﬁtting”, which may not hurt the overall rate of convergence.
Technically, through balancing the variance and bias, data interpolation can possibly improve the overall performance.
And our work essentially quantify such a bias-variance balance in a very precise way.
It is of great interest to investigate how our theoretical insights can be carried over to the real deep neural networks, leading to a more complete picture of double descent phenomenon
A A DDITIONAL N UMERICAL E XPERIMENT
In this experiment, instead of taking n = 2 10 only, we take n = 2 i for i = 6 , . . .
, 10 to see how the performance ratio and optimum k changes in γ for different n ’s. The phenomenon for classiﬁcation is similar as regression so we only present regression.
Figure 5 summarizes the change of MSE and optimum choice of k with respect to different choices of n and γ/d , when d = 2 .
The plot corresponding to d = 5 is quite similar hence omitted here.
This plot shows that with respect to the increase of n , interpolated-NN converges in the same rate as k NN, and interpolated-NN generally requires larger k than k -NN.
B P RELIMINARY P ROPOSITION
This section provides an useful result when integrating c.d.f:
Proposition 4 From Lemma S.1 in Sun et al.
(2016), we have for any distribution function G , (cid:90)
[Equation]
C P ROOF OF A P RELIMINARY T HEOREM FOR T HEOREM
Deﬁne P 1 and P 2 (density as f 1 , f 2 ) as the conditional distributions of X given Y = 0 ,respectively, and π 1 , π 2 are the marginal probability P ( Y = 0) and P ( Y = 1) , then take
P = π 1 P 2 + π 2 P 2 , ¯ P = π 1 P 1 − π 2 P 2 , f ( x ) =
π 1 f 1 ( x ) + π 2 f 2 ( x ) , and also denote Ψ( x ) =
π 1 f 1 ( x ) − π 2 f 2 ( x ) .
Theorem 5 For regression, suppose that assumptions A.1, A.3, A.6, and A.7 hold.
If k satisﬁes n β ≤ k ≤ n 1 − 4 β/d for some β > 0 , we have
[Equation]
For classiﬁcation, under A.1 to A.5, the excess risk w.r.t. γ becomes Regret [Equation]
where
[Equation]
C. R EGRESSION
Rewrite the interpolated-NN estimate at x given the distance to the k + 1 th neighbor R k +1 , interpolation level γ as
[Equation]
where the weighting scheme is deﬁned as
[Equation]
For regression, we decompose MSE into bias square and variance, where
E [
[Equation]
S k,n
[Equation]
x
[Equation]
R
[Equation]
in which the bias square can be rewritten as
E (cid:34) k (cid:88)
[Equation]
and the variance can be approximated as
[Equation]
Following a procedure similar as Step 1 for classiﬁcation, i.e., use Taylor expansion to approximate the bias square, we obtain that for some function a , the bias becomes
[Equation]
2 1 + o
[Equation]
As a result, the MSE of interpolated-NN estimate given x becomes,
[Equation]
Finally we integrate MSE over the whole support.
C. C LASSIFICATION
The main structure of the proof follows Samworth et al.
As the whole proof is long, we provide a brief summary in Section C.2.1 to describe things we will do in each step, then in Section
C.2.2 we will present the details in each step.
C.2. B RIEF S UMMARY
Step 1 : denote i.i.d random variables
Z i ( x, R k +1 )
for i = 1 , . . .
k where
[Equation]
i
[Equation]
[Equation]
k
[Equation]
[Equation]
then the probability of classifying as 0 becomes
[Equation]
The mean and variance of Z i ( x, R k +1 ) can be obtained through Taylor expansion of η and density function of x :
[Equation]
for some function a .
The smoothness conditions are assumed in A.4 and A.5.
Note that on the denominator of Z i , there is an expectation E ( R i ( x ) /R
k +1 ( x )) γ .
From later calculation in Corollary 1, the value of this expectation in fact has little changes given or without a condition of R k +1 , and it is little affected by x either.
Step 2 : One can rewrite Regret as
[Equation]
From Assumption A.2, A.4, the region where (cid:98) η is likely to make a wrong prediction is near { x | η ( x ) = 1 / 2 } , thus we use tube theory to transform the integral of Regret over the d -dimensional space into a tube, i.e.,
[Equation]
The term (cid:15) will be deﬁned in detail in appendix.
Basically, when (cid:15) is within a suitable range, the integral over t will not depend on (cid:15) asymptotically.
Step 3 : given R k +1 and x , the nearest k neighbors are i.i.d.
random variables distributed in
B ( x, R k +1 ) , thus we use non-uniform Berry-Esseen Theorem to get the Gaussian approximation of the probability of wrong prediction:
[Equation]
[Table] [Table]
[Table]
[Table] [Table]
[Equation]
C.2. D ETAILS
Denote a d is the Euclidean ball volume parameter
[Equation]
Deﬁne p
=
k/n and r 2 p
= sup x E R 2 k ( x ) .
Denote E be the set that there exists R i such that
R i > r 2 p , then for some constant c > 0 ,
[Equation]
Hence from Claim A.5 in Belkin et al.
(2018), there exist c 1 and c 2 satisfying
[Equation]
Step 1: in this step, we ﬁgure out the i.i.d.
random variable in our problem, and calculate its mean and variance given x .
Denote
[Equation]
then the dominant part we want to integrate becomes
[Equation]
Therefore, one can adopt non-uniform Berry-Essen Theorem to approximate the probability using normal distribution.
Unlike Samworth et al.
(2012) in which E Y ( X i ) is calculated, since the i.i.d. item in non-uniform Berry-Essen Theorem is Z rather than Y , we now calculate mean and variance of Z .
Under R k +1 ,
[Equation]
and
[Equation]
Then the mean and variance of Z i can be calculated as
[Equation]
Fixing R 1 and R k
+1 , we have
[Equation]
and tr (cid:18)2 ¨ η
[Equation] [Equation]
= tr (cid:18)
[Equation]
¨
[Equation]
( x
[Equation]
0 ) (cid:90) (
[Equation]
[Equation] [Equation]
= tr (cid:18)
[Equation]
¨ η
[Equation] [Equation]
x
[Equation]
x
[Equation]
o
[Equation]
= tr (cid:18)
[Equation]
) ¨ η
[Equation]
Then taking function a ( x ) for x such that
[Equation]
which can be satisﬁed through taking
[Equation]
The difference caused by the value of R 1 is only a small order term.
Finally,
[Equation]
Step 2: in this step we construct a tube based on the set S
= { x | η ( x ) = 1 / 2 } , then ﬁgure out that the part of Regret outside this tube is a remainder term.
Assume (cid:15) k,n satisﬁes s k,n,γ = o ( (cid:15) k,n ) and (cid:15) k,n = o ( s k,n,γ k 1 / 2 ) , then the residual terms throughout the following steps will be o ( s 2 k,n,γ + t 2 k,n,γ ) .
Hence although the choice of (cid:15) k,n is different among choices of k and n , this does not affect the rate of Regret.
Note that we ignore the arguments x and
R k +1 as from A.2 and A.4, s 2 k,n,γ ( x, R k +1 ) (cid:16) 1 /k and t k,,n,γ ( x, R k +1 ) (cid:16) ( k/n ) 2 /d for all x while
R k +1 (cid:16) ( k, n ) 2 /d in probability.
By Samworth et al.
(2012), recall that Ψ( x ) =
d ( π 1 P 1 ( x ) − π 2 P 2 ( x )) , then (cid:90)
R d (cid:32)
P
[Equation]
k (cid:88)
[Equation] [Equation]
(cid:33)
[Equation]
where
[Equation]
For r 1 ,
[Equation]
d ¯ P ( x
[Equation]
Using non-uniform Berry-Essen Theorem
[Equation]
when E Z 31 ( x, R k +1 ) < ∞ , i.e. γ < d/ 3 , it becomes − (cid:90)
[Equation]
where ¯Φ( x ) = 1 − Φ( x ) .
Since s k,n,γ ( x, R k +1 )
=
o ( (cid:15) k,n ) , r
= o
( s 2 k,n,γ ( x, R k +1 )) .
By the deﬁnition of (cid:15) k,n , we have
[Equation]
As a result, using Berstain inequality, r 1 is a smaller order term compared with s 2 k,n,γ when s 2 k,n,γ ( R k +1 )
=
o (1) , hence r
= o
( s 2 k,n,γ
Step 3 : now we apply non-uniform Berry-Esseen Theorem.
From Step 3 , we have
[Equation]
where based on non-uniform Berry-Esseen Theorem:
[Equation]
E | Z 1 ( x t 0 , R k +1 ) |k 3 / 2 V ar 3 /
( Z 1 ( x t 0 , R k +1 )) 1 + (cid:12)(cid:12)(cid:12)(cid:12) − k E Z 1 ( x t 0 ,R k +1 ) √ kV
ar
Z 1 ( x t 0 ,R k +1 )) (cid:12)(cid:12)(cid:12)(cid:12) 3 , and r≤ (cid:90)
S (cid:90) (cid:15) k,n − (cid:15) k,n
E R k +1 t (cid:107) ˙Ψ( x 0 ) (cid:107)
k E | Z 1 ( x t 0 , R k +1 ) |k 3 / 2 V ar 3 /
( Z 1 ( x t 0 , R k +1 )) 1 + (cid:12)(cid:12)(cid:12)(cid:12) − k E Z 1 ( x t 0 ,R k +1 ) √ kV
ar
Z 1 ( x t 0 ,R k +1 )) (cid:12)(cid:12)(cid:12)(cid:12) 3 dtd Vol d − 1 ( x 0 ) .
For r 2 , r≤ (cid:90)
S (cid:90) (cid:15) k,n − (cid:15) k,n
E R k +1 t (cid:107) ˙Ψ( x 0 ) (cid:107)
k E | Z 1 ( x t 0 , R k +1 ) |k 3 / 2 V ar 3 /
( Z 1 ( x t 0 , R k +1 )) 1 + (cid:12)(cid:12)(cid:12)(cid:12) − k E Z 1 ( x t 0 ,R k +1 ) √ kV
ar
Z 1 ( x t 0 ,R k +1 )) (cid:12)(cid:12)(cid:12)(cid:12) 3 dtd Vol d − 1 ( x 0 )
=
c
[Equation]
k
[Equation]
n
[Equation] [Equation]
k
[Equation]
1 ( x t
[Equation]
k
[Equation]
1 ) √
[Equation] [Equation]
k
[Equation]
1 )) (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
=
[Equation]
= o ( s 2 k,n,γ ) .
Step 4 : the integral becomes (cid:90)
S
E R k
[Equation]
k
[Equation]
n
[Equation]
kV
[Equation]
dtd
[Equation]
= (cid:90)
S
E
[Equation]
(cid:90)
[Equation]
k
[Equation]
n
[Equation]
k,n t (cid:107)
[Equation]
( x 0 ) (cid:107)   Φ   − t (cid:107) ˙
η ( x 0 ) (cid:107) (cid:113) s 2 k,n,γ ( x, R k +1 ) /k
− E ( R 1 /R k +1 ) − γ a ( x t 0 ) R(cid:113) s 2 k,n,γ ( x, R k +1 ) /k 
 − 1 { t< 0 } 
 dtd Vol d − 1 ( x 0 ) + r 3 + o = (cid:90)
S
E R k + (cid:90)
R t (cid:107) ˙Ψ( x 0 ) (cid:107)   Φ   − t (cid:107) ˙
η ( x 0 ) (cid:107) (cid:113) s 2 k,n,γ ( x, R k +1 ) /k
− E ( R 1 /R k +1 ) − γ a ( x t 0 ) R(cid:113) s 2 k,n,γ ( x, R k +1 ) /k 
 − 1 { t< 0 } 
 dtd Vol d − 1 ( x 0 ) + r 3 + r
+ o = (cid:90)
S (cid:90)
R t (cid:107) ˙Ψ( x 0 ) (cid:107)   Φ   − t (cid:107) ˙
η ( x 0 ) (cid:107) (cid:113) s 2 k,n,γ ( x ) /k
− E ( R 1 /R k +1 ) − γ a ( x t 0 ) R(cid:113) s 2 k,n,γ ( x ) /k 
 − 1 { t< 0 } 
 dtd Vol d − 1 ( x 0 ) + r 3 + r
+ r
+ o =
B4 k
E ( R 1 /R k +1 ) − 2 γ
E 2 ( R 1 /R k +1 ) − γ + (cid:90)
S f ( x 0 ) (cid:107)
[Equation]
Note that (cid:107) ˙Ψ( x 0 ) (cid:107) / (cid:107) ˙
η ( x 0 ) (cid:107) =
2 f ( x 0 ) .
The last step follows Proposition 4 and the fact that R k + does not affect the dominant parts.
The term E (( R 1 /R k +1 ) − γ | R k +1 ) is almost the same for all
R k +1 .
For the small order terms, following Samworth et al.
(2012) we obtain
[Equation]
, and
[Equation]
= o ( s 2 k,n,γ ) .
The term r 5 is the difference between the normal probability given R k +1 and the one after taking expectation.
Similar with Cannings et al.
(2017), for each x , when | t | < (cid:15) k,n , we have
[Equation]
Following step 3 in Cannings et al.
(2017), we obtain
[Equation]
For the case when | t | (cid:29) s k,n,γ + t k,n,γ , differentiate normal cdf twice still leads to very small probability, thus for each x , we have
[Equation]
D P ROOF OF T HEOREM
To show the ratio MSE ( γ, n ) / MSE (0 , n ) and Regret ( γ, n ) / Regret (0 , n ) asymptotically converges to a constant, we need to ﬁgure out the corresponding optimum k for each scenario.
For classiﬁcation, given x , we know that if X follows multi-dimensional uniform distribution with density 1 /f ( x ) , for some constant c d that only depends on d ,
[Equation]
Since R i → 0 , this approximation still holds if X follow other distributions.
As a result, through ﬁguring out the optimum k for 0 and γ , we have
[Equation]
For regression, one more step needed compared with classiﬁcation is to evaluate k E
[Equation]
The sum of ratios (cid:80) ki =1 ( R i /R k +1 ) − γ is hard to evaluated directly in the denominator, hence we use upper bound and lower bound on it.
Since d − 3 γ > 0 , using non-uniform Berry-Essen Theorem, given R k +1 , we have
[Equation]
Therefore, taking ξ = δ k k E ( R 1 /R k +1 ) − γ ,
P
[Equation]
i
[Equation]
Note that ( R 1 /R k +1 ) − γ is always larger than 1, hence
E (cid:34) ( R 1 /R k +1 ) − 2 γ ( (cid:80) ki
=
( R i /R k +1 ) − γ )(cid:35)
≤ E R k + (cid:20)
E ( R 1 /R k +1 ) − 2 γ (1 − δ k ) 2 k 2 E 2 ( R 1 /R k +1 )
− γ (cid:21) + E R k
+ (cid:34)
P (cid:32) k (cid:88) i =
( R i /R k +1 )
− γ < (1 − δ k )
k E ( R 1 /R k +1 ) − γ (cid:12)(cid:12)(cid:12)(cid:12)
R k + (cid:33) E ( R 1 /R k +1 ) − 2 γ k(cid:35)
+ o ≤ k 2 (1 − δ k )
E ( R 1 /R k +1 ) − 2 γ
E 2 ( R 1 /R k +1 ) − γ +
E ( R 1 /R k +1 ) − 2 γ k¯Φ (cid:32) δ k √ k E
( R 1 /R k +1 )
− γ (cid:112) V ar (( R 1 /R k +1 )
− γ ) (cid:33) + E ( R 1 /R k +1 )
− 2 γ kc 1 + ( √ kδ k )
+ o, while
E (cid:34) ( R 1 /R k +1 ) − 2 γ ( (cid:80) ki
=
( R i /R k +1 ) − γ )(cid:35) ≥ k 2 (1 + δ k )
E ( R 1 /R k +1 ) − 2 γ
E 2 ( R 1 /R k +1 ) − γ −
E ( R 1 /R k +1 ) − 2 γ kc 1 + ( √ kδ k ) 3 + o.
Hence taking δ k such that δ k → 0 while δ k √ k → ∞ , we have
[Equation]
and similarly
E
[Equation]
After ﬁguring out the optimum k for k NN and interpolated-NN
[Equation]
we ﬁnally obtain
[Equation]
E P ROOF OF T HEOREM
The proof is similar with Theorem 1 in Sun et al.
From the deﬁnition of CIS, we have
[Equation]
Based on the deﬁnition of Z i ( x, R k +1 ) in (2)
[Equation]
the derivation of µ k,n,γ ( x, R k +1 ) and s k,n,γ ( x, R k +1 ) , follow the same procedures as in Theorem
[Equation]
we obtain
[Equation]
and similarly,
[Equation]

[Equation]
dtd
[Equation]
x
[Equation]
Adopting Proposition 4 and the fact that s k,n,γ ( x, R k +1 ) is little changed by x and R k +1 , treating Φ and Φ 2 as two distribution functions, we have
[Equation]
The optimum k for k NN and interpolated-NN satisﬁes
[Equation]
Finally we can obtain the asymptotic ratio of CIS’es.
F I NTERPOLATION DOES NOT A FFECT THE R ATE OF C ONVERGENCE F. M ODEL S ETUP AND T HEOREM
The results in Section 3 are obtained based on smooth η and f , thus the Taylor expansions can be calculated.
In this section, we will provide a weaker result under weaker assumptions as those in
Chaudhuri & Dasgupta (2014) and Belkin et al.
We ﬁrst setup some assumptions:
A.1’ X is a d -dimensional random variable on a compact set and satisﬁes regularity condition: let λ be the Lebesgue measure on R d , then there exists positive ( c 0 , r 0 ) such that for any x in the support X ,
[Equation]
, r )) ≥ c 0 λ ( B ( x, r )) , for any 0 < r ≤ r 0 .
A.4’ Smoothness condition: | η ( x ) − η ( y ) | ≤ A (cid:107) x − y (cid:107) α for some α > 0 .
A.5’
The density of X is ﬁnite, and the model satisﬁes margin condition: P ( | η ( X ) − 1 / 2 | < t ) ≤ Bt β .
The regularity condition in A.1’ is used to ensure the convergence of the distance of the k + 1 th neighbor: if the density is inﬁnite at some points, the neighbors will be the point itself.
The smoothness condition A.4’ describes how smooth the function η is, which affects the minimax performance of k NN given a class of η ’s. Assumption A.5’ describes how far the samples are away from 1 / 2 , and affects the minimax rate for binary classiﬁcation.
When β is small, there is a cluster of samples around 1 / 2 , leading to a large prediction error rate.
In interpolated-NN, if the level of over-ﬁtting is not strong, that is, γ is within some suitable range, then excess risk of classiﬁcation also reaches optimal:
Theorem 6 For regression, under A.1’, A.3, A.4’, A.5’,
[Equation]
For classiﬁcation, under A.1’, A.3, A.4’, A.5’,
[Equation]
In addition, if β ≥ 2 , when d − κ ( β ) γ > 0 , where κ ( β ) denotes the smallest even number that is greater than β
+ 1 , then we have
[Equation]
The main proof of Theorem 6 follows the idea in Chaudhuri & Dasgupta (2014), with Berstein inequality changing to non-uniform Berry Esseen Thoerem.
For the regression part, we refer readers to Belkin et al.
(2018) and Belkin et al. (2019c).
From Theorem 6, one can see that given a reasonable level of over-ﬁtting, the performance of interpolated-NN reaches the minimax rate for both regression and classiﬁcation.
When β < 2 and β ≥ 2 , we technically impose different restrictions on γ .
A quick illustration on this is that we adopt non-uniform Berry-Essen Theorem to bound the probability.
In non-uniform
Berry-Essen Theorem, although the Gaussian probability itself has a exponential tail, the residual part is only 1 / (1 + | t | 3 ) .
As a result, when summing up t β +1 i / (1 + t 3 i ) for t i =
i , β − 2 should be smaller than 0 in order to bound the summation.
Remark
As Chaudhuri & Dasgupta (2014) mentioned, our smoothness condition (A.4’) is in fact a sufﬁcient condition.
In Theorem 6, what we really need is some constant α (cid:48) such that
[Equation]
The result in Samworth et al.
(2012) is a case where α (cid:54) =
α (cid:48) .
We approximate E η ( X ) as η ( x 0 ) + tr [¨ η ( x 0 ) E ( X − x 0 )
( X − x 0 ) (cid:62) ] in Theorem 5, which indicates that we adopt smoothness α (cid:48) = 2 , though α = 1 .
In addition, the margin condition parameter β = 1 under A.1 to A.5 becomes 1.
Therefore, the optimum rate of O ( n − α (cid:48) ( β +1) /
(2 α (cid:48)
+ d ) ) is O ( n − 4 / (4+ d ) ) .
The over-ﬁtting weighting scheme in fact leads to optimal rate for classiﬁcation.
F. P ROOF OF T HEOREM
Let p = 2 k/n .
Denote E
=
{∃ R i > r p , i = 1 , . . .
, k } , and E R k,n ( x ) − R ∗ ( x ) as the excess risk.
Then deﬁne
[Equation]
as well as
[Equation]
with the decision boundary area:
[Equation]
Given ∂ p, ∆ , X + p, ∆ , and X − p, ∆ , similar with Lemma 8 in Chaudhuri & Dasgupta (2014), the event of g ( x ) (cid:54) =
(cid:98) g k,n,γ ( x ) can be covered as:
[Equation]
When ˜ η ( x | R k +1 ) > 1 / 2 and x ∈ X + p, ∆ , assume (cid:98) η k,n,γ ( x ) < 1 / 2 , then ˜ η k,n,γ ( x | R k +1 ) − (cid:98) η k,n,γ ( x ) > ˜ η k,n,γ ( x | R k +1 ) − 1 / 2 ≥ ∆ .
The other two events are easy to ﬁgure out.
In addition, from the deﬁnition of Regret, assume η ( x ) < 1 / 2 ,
P ( (cid:98) g
( x ) (cid:54) =
Y | X = x ) − η ( x ) =
η ( x ) P ( (cid:98) g
( x ) =
| X = x ) + (1 − η ( x ))
P ( (cid:98) g
( x ) = 1 | X = x ) − η ( x ) =
η ( x ) P ( (cid:98) g
( x ) =
g ( x ) | X
= x ) + (1 − η ( x ))
P ( (cid:98) g
( x ) (cid:54) =
g ( x ) | X = x ) − η ( x )
=
η ( x ) − η ( x ) P ( (cid:98) g ( x ) (cid:54) =
g ( x ) | X
= x ) + (1 − η ( x ))
P ( (cid:98) g
( x ) (cid:54) =
g ( x ) | X = x ) − η ( x ) =
(1 − 2 η ( x ))
P ( (cid:98) g
( x ) (cid:54) =
g ( x ) | X = x ) , similarly, when η ( x ) > 1 / 2 , we have
P ( (cid:98) g
( x ) (cid:54) =
Y | X = x ) − 1 + η ( x ) =
(2 η ( x ) − 1) P ( (cid:98) g
( x ) (cid:54) =
g ( x ) | X = x ) .
As a result, the Regret can be represented as
Regret ( k, n, γ ) =
E ( | 1 − 2 η
( X ) | P ( g ( X ) (cid:54) =
(cid:98) g k,n,γ ( X ))) .
For simplicity, denote p = k/n .
We then follow the proof of Lemma 20 of Chaudhuri & Dasgupta (2014).
Without loss of generality assume η ( x ) > 1 / 2 .
Deﬁne
[Equation]
then
[Equation]
hence x ∈ X + p, ∆( x ) − ∆ 0 .
[Table]
W
[Table]
|
[Table] [Table]
[Table] [Table]
η
[Table]
>
[Table]
(˜ η (
[Table]
| R
[Table]
+ ∆
[Table]
) −
[Table] [Table]
[Table]
− k
[Table] [Table]
[Table]
η
[Table] [Table]
=
[Table]
[Table] [Table]
( R
[Table] [Table]
[Table]
( Y ( X
[Table]
) − ˜ η
[Table]
|
[Table]
) − (
[Table] [Table]
[Equation]
>
[Equation]
Since
E k
[Equation]
i
[Equation]
we can use Markov inequality to the power of κ ( β ) to bound the probability.
Denote
Z i
[Equation]
for simplicity.
Note that
[Equation]
For different settings of β and γ , the following steps have the same logic but different details:
Case 1: β ≤ 1 and γ < d/ 3 :
Considering the problem that the upper bound can be much greater than 1 when ∆( x ) is small, we deﬁne ∆ i = 2 i ∆ 0 , taking i
=
min
{ i ≥ 1 | (∆ i − ∆ 0 ) 2 > 1 /k } .
In this situation, since E Z 31 ( x ) < ∞ , we can adopt non-uniform Berry-Essen Theorem for the proof:
[Equation]
The two terms from Berry-Essen Theorem becomes
[Equation]
together with
E
[Equation]
The upper bound is larger than 1 if ∆ i ≤ c 5 / √ k .
When ∆ i > c 5 / √ k ,
[Equation]
Therefore the sum of the excess risk can be bounded.
When β < 2 , β − 2 < 0 , hence
[Equation]
and
[Equation]
Recall that ∆ i 0 > ∆ 0 and ∆ 2 i 0 > 1 /k , hence when ∆ 2 i
= O (1 /k ) , we can obtain the minimum upper bound
[Equation]
Taking k (cid:16) ( n 2 α/ (2 α + d ) ) , the upper bound becomes O ( n − α ( β +1) / (2 α + d ) ) .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the last inequality is obtained since d − κ ( β ) γ > 0 .
Note that κ ( β ) > β + 1 , thus for some 0 < c < 1 ,
[Equation]
Therefore the sum of the excess risk for can be bounded, where
[Equation]
Recall that ∆ i 0 > ∆ 0 and ∆ 2 i 0 > 1 /k , hence when ∆ 2 i
= O (1 /k ) , we can obtain the minimum upper bound
[Equation]
Taking k (cid:16) ( n 2 α/ (2 α + d ) ) , the upper bound becomes O ( n − α ( β +1) / (2 α + d ) ) .