Debiased Machine Learning without Sample-Splitting for Stable Estimators
Abstract
Estimation and inference on causal parameters is typically reduced to a generalized method of moments problem, which involves auxiliary functions that correspond to solutions to a regression or classification problem.
Recent line of work on debiased machine learning shows how one can use generic machine learning estimators for these auxiliary problems, while maintaining asymptotic normality and root- n consistency of the target parameter of interest, while only requiring mean-squared-error guarantees from the auxiliary estimation algorithms.
The literature typically requires that these auxiliary problems are fitted on a separate sample or in a cross-fitting manner.
We show that when these auxiliary estimation algorithms satisfy natural leave-one-out stability properties, then sample splitting is not required.
This allows for sample re-use, which can be beneficial in moderately sized sample regimes.
For instance, we show that the stability properties that we propose are satisfied for ensemble bagged estimators, built via sub-sampling without replacement, a popular technique in machine learning practice.
Introduction
A large variety of problems in causal inference and more generally semi-parametric inference can be framed as finding a solution to a moment condition:
[Equation]
where Z ∈ Z is a vector of random variables that, apart from the target parameter θ 0 ∈ Θ ⊂ R p of interest (we assume p
= O (1) ) also depends on unknown nuisance functions g 0 ∈ G which need to be estimated in a flexible manner from the data.
This framework has a long history in the literature on semi-parametric inference [38, 26, 27, 47, 36, 51, 57, 12, 40, 49, 58, 13, 41, 1, 42, 2, 53, 37, 3], which analyzes the following two-stage estimation process with sample re-use, when having access to n iid samples { Z 1 , . . .
, Z n } and as n grows, treating the target parameter dimension p as a constant:
Obtain an estimate ˆ g ∈ G of the nuisance function g 0 based on all the samples.
Return any estimate ˆ θ ∈ Θ that satisfies:
[Equation]
with
[Equation]
Given that the estimation of function ˆ g is a complex non-parametric problem, it will typically only satisfy slower than parametric error rates.
Moreover, in high dimensional settings, when machine learning techniques are used to estimate ˆ g , then the regularization bias of ˆ g will propagate to the final estimate ˆ θ , leading to non-regular estimates and the inability to construct confidence intervals.
The literature on efficient semi-parametric estimation provides conditions on the moment function so that the influence of the estimation error of the nuisance function ˆ g is of second order importance and does not alter the distributional properties of the second stage [26, 13, 59, 7, 8, 9, 10, 31, 32, 33, 54, 46, 19, 43, 48, 28, 29, 30, 16, 61, 62].
This approach dates back to the classical work on doubly robust estimation and targeted maximum likelihood [50, 49, 56, 55, 39, 52] as well as the more recent work on locally robust or Neyman orthogonal conditions on the moment function [44, 45, 18, 6, 17], typically referred to as double or debiased machine learning.
At a high-level, the earlier literature on semi-parametric inference shows that if the moment function satisfies some form of robustness to nuisance perturbations and, importantly, as long as the space G used in the first stage estimation is a relatively simple function class, in terms of statistical complexity, typically referred to as a Donsker function class, then the second stage estimate is root- n consistent and asymptotically normal.
Hence, one can easily construct confidence intervals for the target parameter of interest.
Crucially the recent literature on debiased machine learning alters the standard two-stage estimation algorithm to introduce the idea of sample-splitting, [11, 60].
In particular, instead of estimating ˆ g on all the samples, the recent work on debiased machine learning [17, 18, 22] estimates ˆ g on a separate sample, or for better sample efficiency invokes “cross-fitting,” where we train a nuisance model on half the data and evaluate it in the second stage on the other half and vice versa.
Sample splitting avoids the Donsker conditions that were prevalent in the classic semi-parametric inference literature and only requires a root-mean-squared-error (RMSE) guarantee of the estimate ˆ g of o p ( n − 1 / 4 ) .
However, sample splitting or cross-fitting still leads to poorer sample usage, as we can lose half of our data when training complex non-parametric or machine learning models, which can be problematic in small and moderate sample regimes.
Our main result is to show that root- n consistency and asymptotic normality of the standard algorithm, without sample splitting, can be achieved without the Donsker property, but solely if one assumes that the first stage estimation algorithm is o ( n − 1 / 2 ) leave-one-out stable, a relatively widely studied property in the statistical machine learning and generalization theory literature [15, 34, 23, 25].
As a leading example we show that our stability conditions are satisfied by bagging estimators formed with sub-sampling without replacement.
Recent prior work of [21] also considered asymptotic normality based on stability conditions, but as we expand in the main text, their requirement on the stability property is much harsher than the one we derive here.
Moreover, [21] analyzes only a special case of the class of moment problems that we consider here.
For instance, in our leading example of bagging estimators, the prior result of [21] would require that the bias of the base estimator decays faster than 1 /n , where n is the sample size, which is typically not the case.
In contrast, our stability condition does not impose any explicit assumption on the bias and solely requires that the sub-sample size m is o ( √ n ) .
To simplify the regularity assumptions required for asymptotic normality, we focus on the case where m ( Z ; θ, g ) is linear in θ , i.e.
[Equation]
where a ( Z ; g ) ∈ R p × p is a p × p matrix and ν
( Z ; g ) ∈
R p is a p -vector, and we denote with:
[Equation]
Many of the leading examples in semi-parametric problems that arise in causal inference correspond to linear moment problems.
We present below a representative set of problems that are widely used in the practice of causal inference.
Example 1 (Partially Linear Treatment Effect [51]) .
If one assumes that the outcome of interest Y is linear in the treatment, i.e. Y = θ ′
T + f 0 ( X ) + ϵ , with E [ ϵ | T, X ] = 0 , then estimating the treatment effect θ 0 boils down to solving the following linear moment:
[Equation]
where the corresponding true values of ( q, p ) are q 0 ( X )
=
E [ Y | X ] , p 0 ( X ) =
E [ T | X ] .
Example 2 (Partially Linear IV [17]) .
If one assumes that the outcome of interest Y is linear in the treatment, i.e. Y = θ ′
T
+ f 0 ( X ) + ϵ ,
but the treatment is endogenous (i.e. there are unobserved confounders) and one has access to a random variable Z , that is referred to as an instrument, which correlates with the treatment but is un-correlated with the residual in the outcome equation, i.e. satisfies that E [ ϵ | Z, X ] = 0 , then estimating θ 0 boils down to solving the following linear moment:
[Equation]
where the true values of ( q, p, r ) are q 0 ( X )
=
E [ Y | X ] , p 0 ( X ) =
E [ T | X ] , r 0 ( X ) =
E [ Z | X ] .
Example 3 (Average linear functionals of regression functions) .
Consider a class of moment functions of the form:
[Equation]
where g = ( q, µ ) , m b is a linear functional of q and the corresponding true value q is a regression function q [-
-]0 ( T, X ) =
E [ Y | T, X ] and µ 0 is Riesz representer of the functional E [ m b ( Z ; q )]
(see [20] for examples).
For instance, in the case of a binary treatment T , where we have that Y =
q
( T, X )+ ϵ and E
[ ϵ | T, X ] = 0 , then the average treatment effect θ 0 = E [ q (1 , X ) − q (0 , X )] is identified by a moment of the latter type, with:
[Equation]
While if we have a target treatment policy π : X → { 0 , 1 } , and we want to estimate its average value θ 0 , we can identify do so with a moment of the aforementioned type, with:
[Equation]
For completeness, we also include in Appendix H an extension of our results to nonlinear moment problems.
Asymptotic Normality without Sample Splitting
We start by providing an asymptotic normality theorem for semi-parametric moment estimators without sample splitting and where the moment satisfies the well-studied property of Neyman orthogonality.
Our theorem requires four main conditions: i) root-mean-squared-error (RMSE) rates for the nuisance function estimates of o p ( n − 1 / 4 ) , ii) Neyman orthogonality of the moment with respect to the nuisances, iii) second-order smoothness of the moment with respect to the nuisance functions and, iv) stochastic equicontinuity of the Jacobian and the offset part of the linear moment function as the nuisance estimate ˆ g converges to g 0 .
For a vector x ∈ R p we denote with ∥ x ∥ 2 the ℓ 2 norm and for a matrix X ∈ R p × p we denote with ∥ X ∥ op the operator norm with respect to the ℓnorm.
Theorem 1.
Suppose that the nuisance estimate ˆ g ∈ G satisfies:
[Equation]
Suppose that the moment satisfies the Neyman orthogonality condition: for all g ∈ G [Equation] Neyman
[Equation]
) and a second-order smoothness condition: for all g ∈ G
[Equation]
(Smoothness) and that the moment m satisfy the stochastic equicontinuity conditions:
[Equation]
(Stochastic Equicontinuity)
Assume that A ( g 0 ) − 1 exists and that for any g, g ′ ∈ G :
[Equation]
Moreover, assume that for any i, j ∈ [ p ] × [ p ] , the random variable a i,j ( Z ; g 0 ) has bounded variance and that ∥ θ 0 ∥ 2 = O (1) .
Then ˆ θ is asymptotically normal:
[Equation]
The first three conditions are standard assumptions in the literature on debiased machine learning.
The final condition (stochastic equicontinuity) is exactly where sample splitting comes very handy in the literature.
To illustrate the reason why sample splitting helps with the stochastic equicontinuity condition, let us consider the first part of the condition (the reasoning is analogous for the second part).
It asks that the difference of two centered empirical processes, namely A n (ˆ g ) − A (ˆ g ) and
A n ( g 0 ) − A ( g 0 ) , goes to zero faster than n − 1 / 2 .
We expect each empirical process to go down to zero at exactly n − 1 / 2 and so this condition asks, since ˆ g converges to g 0 is the empirical process continuous in its argument and for that reason does the difference converge to zero faster than each individual component.
If the estimate ˆ g was fitted on a separate sample, then conditional on ˆ g , we have that each element t =
( i, j ) ∈
[ p ] × [ p ] of A n (ˆ g ) − A n ( g 0 ) is an empirical average of iid random variables with mean A (ˆ g ) − A ( g 0 ) .
Thus a simple Bernstein inequality would show that the difference of the two empirical processes would converge to zero at the order of:
[Equation]
where we also invoked a mean-squared-continuity property of a t ( Z ; g ) and the fact that ∥ ˆ g − g
∥
=
o p (1) .
Thus, with sample splitting, no further constraint is required from ˆ g , other than a convergence rate on ∥ ˆ g − g
∥ 2 .
In fact, as was noted in recent work of [22], in the above step it suffices to assume that E [( a t ( Z ; ˆ g ) − a t ( Z ; g 0 )) 2 ]
=
O ( ∥ ˆ g − g
∥ q 2 ) for any q < ∞ , which is a much weaker mean-squared-continuity assumption, and the property would still hold, since ∥ ˆ g − g
∥
q/ 2 2 n − 1 /
= o ( n − 1 / 2 ) , whenever ∥ ˆ g − g
∥ 2 = o p (1) .
Without sample splitting, note that ˆ g is now correlated with the samples in the empirical averages and hence A n (ˆ g ) − A n ( g 0 ) is no longer an average of i.i.d.
random variables.
Typical approaches would try to prove a uniform stochastic equicontinuity property over the function space G , typically referred to as a Donsker property of the function space G .
In particular, if we could show that w.h.p. : ∀ g ∈ G : O
[Equation]
then the above property would also hold for ˆ g .
Subsequently, since we know that ∥ ˆ g − g
∥ 22 = o p ( n − 1 / 2 ) , by our convergence rate assumptions on ˆ g , then it would suffice that δ 2 n = o ( n − 1 / 2 ) .
Such localized concentration inequalities have been known to hold for Donsker classes, which are typically defined via entropy integrals, and more recently it was also noted that such inequalities are satisfied with δ n being the critical radius of the space G , defined via localized Rademacher complexities.
However, the latter approach is conservative as it requires a uniform control over the function space
G and does not utilize at all the properties of the estimation algorithm itself.
In particular, as we will show in the next section, the main result of our work is that this stochastic equicontinuity condition follows from o ( n − 1 / 2 ) leave-one-out stability conditions on our estimation algorithm, which are typical in the machine learning literature and in the excess risk and generalization bounds literature.
Stochastic Equicontinuity via Stability
We will show that the Condition (Stochastic Equicontinuity) is satisfied, whenever the estimate ˆ g satisfies leave-one-out stability properties and the moment satisfies the weak mean-squared-continuity property of [22].
We start by some preliminary definitions required to state our stability conditions.
Define Z ( − l ) as the data Z 1 , . .
, Z n with the l -th data point Z l replaced with an independent copy ˜ Z l .
Define Z ( − l 1 , − l 2 ) as the data Z 1 , . .
, Z n with both the l 1 -th and the l 2 -th data points Z l 1 , Z lreplaced with independent copies ˜
Z l 1 , ˜ Z l 2 .
Define similarly for Z ( − l 1 , − l 2 , − l 3 ) and so on.
Let ˆ g ( − l ) be the estimator trained on Z ( − l ) instead of Z 1 , . . .
, Z n .
Similarly, let ˆ g ( − l 1 , − l 2 ) be trained on
Z ( − l 1 , − l 2 ) and so on.
Moreover, we will always denote with Z a fresh random variable drawn from the distribution of the samples, but which is not part of any training sample.
For any random variable
X , we denote with ∥ X ∥ 1 :=
E [ | X | ] , with ∥ X ∥ 2 :=
(cid:112) E
[ X 2 ] , and with ∥ X ∥ p :=
( E [ | X | p ]) 1 /p for any p ≥ 1 in general.
Lemma 2 (Main Lemma) .
If the estimation algorithm satisfies the stability conditions: for all i, j ∈ [ p ]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
and the moment satisfies the mean-squared-continuity condition:
[Equation]
E [
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
for some 0 < q < ∞ and some L > 0 , then the Condition (Stochastic Equicontinuity) is satisfied.
Remark 1.
We show in section 3.1 that the stability conditions are tight.
We present a counter example for which (cid:13)(cid:13)(cid:13)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
2 is exactly of order n − 1 / 2 and for which the Stochastic
Equicontinuity condition is not satisfied.
Remark 2.
We note that prior work of [21] that established asymptotic normality without sample splitting via stability, required significantly stronger conditions than what we invoke here.
In particular, if we let β n be the stability of the estimator ˆ g as measured by the quantities in Lemma 2, then the prior work of [21], would require that nβ n ∥ ˆ g − g
∥ 2 → 0 .
If we only know that ∥ ˆ g − g
∥ =
o ( n − 1 / 4 ) , then the above would require β n = o ( n − 3 / 4 ) , which is much slower than o ( n − 1 / 2 ) .
Moreover, for bagged kernel estimators that we analyze in section 4, the prior work would require that if we use bags of size m , then the bias of the base estimator with m samples, denoted as bias ( m ) satisfies that m bias ( m ) → 0 .
This would rarely be satisfied and in prior work, the only concrete case that was given was forest estimators with binary variables under strong sparsity conditions, in which case the bias decays exponentially with the sample size.
For more general estimators, we expect bias ( m ) =
1 /m α , for some α .
For such settings, our work still applies and, as we show in section 4, gives results for bagged 1 -nearest neighbor estimation algorithms, which do not satisfy any entropy or critical radius bound, but are stable.
The key innovation that enables our improved results is a “double centering” approach that derives intuition from techniques invoked in the analysis of cross-validation via stability and the proof of the Efron-Stein inequality[14].
This idea has already been used in the study of the cross validated risk [4, 5].
Proof of Main Lemma.
We will show the first part of the lemma, i.e. that if for all i, j ∈ [ p ]
[Equation]
then
[Equation]
The analogous statement for ν and V , follows in an identical manner.
Since A ( g ) and A n ( g ) are p × p matrices and p =
O (1) , it suffices to show the above property for every element ( i, j ) ∈
[ p ] × [ p ] , i.e. that √
[Equation]
For this it suffices to show that:
[Equation]
In the remainder of the proof we look at a particular ( i, j ) and hence for simplicity we overload notation and we let a :
= a i,j and A :=
A i,j .
By triangle inequality and domination of L p norms we have
[Equation]
[Table]
To ease notations, we denote
[Equation] [Equation]
Now
[Table]
[Equation]
[Table]
[Equation]
We now aim to show that J 3 ,n = o (1) .
Write for simplicity
[Equation]
Now by expanding the square we obtain
[Equation]
To bound the first term
[Equation]
we have by mean-squared continuity:
[Equation]
[Equation]
where we invoked the property that ∥ ˆ g − g
∥
=
o p (1) , and the second to last equality exploited the tower law.
Thus max l ∈ [ n ] E (cid:2) K 2 l (cid:3) =
o (1) .
Double centering.
We now bound the term, ( n − 1) max l 1 ̸ = l 2 ∈ [ n ] E
[ K l 1 K l 2 ] .
Define, for simplicity, for l 1 ̸ = l
[Equation]
Note that K ( l 2 ) ldoes not depend on the l 2 -th data point and is only a function of Z ( − l 2 ) , ˜ Z l 1 .
Moreover, by the definition of A , noting that Z l is independent of Z ( − l ) and Z l 1 is independent of
Z ( − l 1 , − l 2 ) :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and
[Equation]
For simplicity, we show that ( n − 1) E
[ K 1 K 2 ]
= o (1) , i.e. we show that the second term vanishes for l 1 = 1 and l 2 = 2 .
The same exact arguments generalize to arbitrary l 1 , l 2 .
We begin by writing:
[Equation]
since by tower law:
[Equation]
Similarly by conditioning on Z ( − 1) , ˜ Z 2 and using tower law, we can show that
[Equation]
Hence, we have
[Equation]
With identical arguments, the same equality holds for any indices l 1 , l 2 .
By Cauchy-Schwarz:
[Equation]
− K ( l
[Equation]
[Table]
l
[Equation]
(cid:16) K l 2 − K ( l 1 ) l 2 (cid:17)(cid:105) ≤ max
[Equation]
̸
[Equation]
(cid:13)(cid:13)(cid:13) [-
-]K
[Equation] 1 (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13) K [Equation] 2 − K ( l
[Equation]
l 2 (cid:13)(cid:13)(cid:13)
Thus for max l 1 ̸ = l 2 ( n − 1) E [ K l 1 K l 2 ]
{+= o (1) it suffices that: max l 1 ̸ = l 2 (cid:13)(cid:13)(cid:13) K l 1 − K ( l 2 ) l 1 (cid:13)(cid:13)(cid:13) 
+}
=
o ( n − 1 / 2 ) .
Expanding the definitions K l 1 and K ( l 2 ) l 1 , the above simplifies to:
[Equation]
The latter is upper bounded by a triangle inequality and a Jensen’s inequality by:
[Equation]
If we denote with Z a fresh random sample not part of the training sets, then since Z l 1 is not part of
Z ( − l 1 ) , we have:
[Equation]
where we also used the fact that ˜ Z l 1 only appears in the training sets of ˆ g ( − l 1 ) and ˆ g ( − l 1 , − l 2 ) and we can simply rename it to Z l 1 , as they are identically distributed and both independent from all the other data points.
Invoking the second of our stability conditions for a , we have that:
[Equation]
o
[Equation]
n
[Equation]
Hence, J 3 ,n = o (1) , which completes the proof.
3.1 Tightness of Stability Condition
We present here an example that shows that, without further structural constraints (on the moment or the function space G ), the stability condition we impose is required for stochastic equicontinuity condition to hold.
Let ( X i ) i.i.d ∼ unif[0 , 1] be i.i.d uniform random variables.
We set Y i :
= I ( X i ≤ 0 . 5) and set Z i :=
( X i , Y i ) .
For any x ∈ [0 , 1] , we define c ( Z 1: n , x ) :=
arg min i ≤ n | X i − x | the function that returns the index of the nearest example to x in { X 1 , . . .
, X n } and note that the quantity Y c ( Z 1: n ,x ) is its nearest neighbour estimator.
Let
[Equation]
We remark that ν ( · , ˆ g ) does not satisfy our stability conditions as ∥ ν ( Z, ˆ g ) − ν
( Z, ˆ g ( − 1) )
∥ 2 is exactly of order 1 / √ n , neither does it respect the stochastic equicontinuity property.
Lemma 3.
Let ( X i ) i.i.d ∼ unif[0 , 1] be i.i.d uniform random variables.
We set Y i :
= I ( X i ≤ 0 . 5) and set Z i :=
( X i , Y i ) .
There are constants C, c > 0 such that
• Set g 0 ( Z ) :
=
then we have ∥ ˆ g ( Z ) − g
( Z ) ∥ 22 →• C √ n ≥ ∥
ν ( Z, ˆ g ) − ν
( Z, ˆ g ( − 1) )
∥ 2 ≥ c √ n • √ n | V (ˆ g ) − V ( g 0 ) − (cid:0) V n (ˆ g ) − V n ( g 0 ) (cid:1) | ̸ P −→ 0 .
Application: Bagging Estimators
We remark that if the functions a
and ν satisfy certain L p -Lipchitz conditions then the stability conditions of lemma 2 are implied by the algorithmic stability of the estimator ˆ g .
Those conditions are only marginally stronger than the condition of mean-squared-continuity found in lemma[-
-]
Corollary 4.
Fix any constant r > 1 .
Suppose that there is L < ∞ such that the estimation algorithm satisfies the following uniform L 2 r -continuity condition: for all i, j ∈ [ p ] and l ∈ [ n ]
[Equation]
Suppose in addition that
[Equation]
and if in addition mean-squared-continuity condition is satisfied, then the conditions of lemma 2 are satisfied.
The uniform L 2 r -continuity conditions are going to be satisfied by most moment functions m ( · ; · , · ) .
We notably show in the appendix that example 1, example 2 and example 3 with general moment conditions satisfy our conditions.
The condition of (Algorithmic Stability) is a commonly assumed condition in recent literature [4, 5] and is satisfied by various regularized empirical risk minimization estimators and stochastic gradient descent estimators [15, 24].
Notably, it is satisfied by bagged estimators.
We show in theorem 5 that under very general conditions a bagged ensemble of any machine learning estimator is stable.
Let
Z 1 , . .
, Z n ∈ Z be an independent and identically distributed (i.i.d.) sample of size n .
We sample uniformly randomly without replacement from these observations repeatedly and independently, each time taking a sample of size m , for a number of B times.
We denote the resulting samples as
[Equation]
Let ˆ h m : Z m → R P be a base machine learning estimator trained on m observations.
This base estimator can be tree, a CNN, a nearest-neighbour classifier, or any other type of machine learning estimator.
The corresponding bagged estimator is
[Equation]
We will show that subject to B and m being sufficiently big and mild moment conditions on the base estimator ˆ h m , the bagged estimator satisfies condition (Algorithmic Stability).
Theorem 5.
Fix any constants s, k ≥ 2 r such that 1 s + 1 k
= 12 r .
Assume B, m are sufficiently large:
[Equation]
[Equation]
· n
[Equation]
−
[Equation]
and assume the base estimator ˆ h has bounded moments:
[Equation]
for some constant C > 0 .
Then (Algorithmic Stability) is achieved:
[Equation]
Therefore if a and ν satisfy the condition (1) then the condition (2) is satisfied.
In particular, we can take the base machine learning estimator to be the 1-nearest neighbor estimator.
This specific choice leads to a bagged estimator that can be proved to satisfy both our stability conditions and our consistency rate condition in Theorem 1, in regression settings where covariates are of small intrinsic dimension.
Let { Z i
=
( X i , Y i ) } ni =1 be a sample of size n , drawn independently and identically distributed from
Z =
( X, Y ) .
Here X ∈ X ⊂ R D are known covariates and Y ∈ Y
⊂ R P is the response variable.
As in our bagging setting, let Z 11: m , . .
, Z B 1: m be B independent samples of size m drawn without replacement from observations { Z i } ni =1 .
A bagged 1-nearest neighbor (1-NN) estimator takes the following form:
[Equation]
where S b ( x ) is the 1-NN of x in the set Z b 1: m .
The estimator is used to estimate the conditional expectation g 0 ( x ) :
=
E [ Y | X = x ] .
Lemma 6 (Special Case of Theorem 3 of [35]) .
Assume that
• The marginal distribution µ of X 1 satisfies ( C, d ) -homogeneity on the ball B ( x, r ) : µ ( B ( x, r ))
≤
Cα − d µ ( B ( x, αr ))
∀ α ∈ (0 , 1) for some C, r > 0 .
Here d is referred to as the intrinsic dimension of the distribution.
• The conditional expectation E [ Y | X = x ] is a Lipschitz function in the coordinates x . •
The response variable Y is bounded in L ∞ -norm: ∥ Y ∥ ∞ < ∞ .
Then the bagged 1-NN estimator ˆ g with B ≥ nm satisfies the following convergence condition:
[Equation]
In particular, we note that when 0 < d < 2 and m =
O ( n 12 − ϵ ) for some 0 < ϵ ≤ 12 − 14 d, we immediately have that
[Equation]
achieving the convergence rate assumed in Theorem 1.
Moreover, provided that we choose B to be sufficiently large such that
[Equation]
the required stability conditions can also be satisfied.
Experimental Evaluation
We consider a synthetic experimental evaluation of our main theoretical findings.
We focus on the partially linear model with a scalar outcome Y ∈ R , a scalar continuous treatment T ∈ R and many controls X ∈ R n x , where: T = p 0 ( X ) + η , η ∼ N (0 , 1) , Y = θ
0 T + f 0 ( X ) + ϵ , ϵ ∼ N (0 , 1) .
Our goal is the estimation of the treatment effect θ 0 , while estimating the nuisance functions p [-
-]0 ( X ) and q 0 ( X ) = θ 0 p 0 ( X ) + f 0 ( X ) in a flexible manner.
We will consider estimation based on the orthogonal moment presented in Example 1.
We considered sub-sampled 1 -nearest neighbor regression (NN) and sub-sampled fully grown (with only 1 sample minimum leaf size) random forest (RF) regression (see Appendix A) for the estimation of p 0 and q 0 , regressing T on X and Y on X correspondingly.
The true functions p 0 and f 0 are actually linear in our data generating process, with p 0 ( X ) =
β ′
X and f 0 ( X ) =
γ ′
X
and where β 0 and γ 0 have only one non-zero coefficient ( 1 -sparse), and which is the same coefficient for both β 0 and γ 0 .
In other words, only one of the n x potential confounding variables X is actually a confounder.
We evaluate the performance of the estimate for θ 0 for a range of values of the sample size n and the dimension of the controls n x and with or without cross-fitting.
For the cross-fitted estimates we used 2 splits.
For each specification we draw 1000 experimental samples to evaluate the distributional properties of the estimate.
We considered sub-sample sizes for the nuisance regressions based on our theoretical n 0 .
49 specification and for larger specifications too.
We find that the estimate without cross-fitting typically has almost equal bias and smaller (and always comparable) variance (due most probably to the smaller mean squared error of the nuisances, since they are trained on larger sizes), especially in smaller sample sizes, and has better coverage properties, when the sub-sample size is m =
o ( n 1 / 2 ) .
Moreover, the estimate is approximately normally distributed, even without cross-fitting as is verified qualitatively via quantile-quantile (Q-Q) plots (c.f. Appendix A).
We also report the mean of the estimate of the standard error across the 1000 experiments, to evaluate the bias in the estimation of the standard error.
We find that the estimate of the standard error is more accurate without cross-fitting, potentially because the estimate of the standard error does not incorporate the extra variance that stems from the sample-splitting process.
This inaccuracy of the standard error estimate is most probably the reason for the worst coverage properties of the confidence intervals with cross-fitting.
In summary, we verify experimentally that for stable estimators, with the theoretically required level of stability, sample splitting or cross-fitting is not needed to maintain asymptotic normality, small bias and nominal coverage. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
A Further Experimental Results and Details
Run-time specification.
The experiments in the main text and in the appendix were run on a normal PC laptop with Processor Intel(R) Core(TM) i7-8650U CPU @
1.90GHz, 2112 Mhz,[-
-]
Core(s), 8 Logical Processor(s), 16GB RAM.
It took around 1.5 hours to run all the experiments.
A.1 Break-down Point
As expected, we do verify that as the sub-sample size becomes m =
O ( n ) , where the estimate is both not stable and also does not optimize over hypothesis spaces with small critical radius or that satisfy the Donsker property, then the estimate without cross-fitting breaks down, while the estimate with cross-fitting maintains a decent performance, despite the high-variance of the nuisance estimate.
[Figure]
estimation
[Figure]
A.2 Random Forest Experiments
[Figure]
[Table] [Table]
[Figure]
B Proof of Theorem
Proof.
For any g ∈ G , by the linearity of the moment with respect to θ :
[Equation]
Moreover, for any g , with ∥ g − g 0 ∥ 2 = o p (1) : A
[Equation]
g )
[Equation]
Thus for any g , with ∥ g − g 0 ∥ 2 = o p (1) : A
[Equation]
[Equation]
[Equation]
Let G n ( θ, g ) :
= M ( θ, g ) − M n ( θ, g ) , then we have:
[Equation]
Applying the above for g = ˆ g and by the definition of ˆ θ [Equation] we have:
[Equation]
Applying Neyman orthogonality and bounded second derivative of the moment with respect to g :
[Equation]
Thus we have that:
[Equation]
Now we decompose the empirical process part into an asymptotically normal component and asymptotically equicontinuous parts that converge to zero in probability:
[Equation]
By the linearity of the moment, the middle term can be written as:
[Equation]
Note that by a triangle inequality:
[Equation]
Note that the first quantity is a simple centered empirical process and hence assuming that a i,j ( Z ; g 0 ) has bounded variance, by classical results in empirical process theory we have that:
[Equation]
Moreover, by our stochastic equicontinuity condition we have that:
[Equation]
Thus we get that ∥ A (ˆ g ) − A n (ˆ g ) ∥
op = o p (1) , and therefore:
[Equation]
Moreover, since by our stochastic equicontinuity conditions: √
[Equation]
we have by triangle inequality, the definition of the operator norm, and the fact that ∥ θ 0 ∥ 2 = O (1) that:
[Equation]
Thus we can conclude that:
[Equation]
Assuming that the inverse A ( g 0 ) − 1 exists, we can re-arrange to:
[Equation]
Since G n ( θ 0 , g 0 ) is a mean-zero empirical process, we have that ∥ G n ( θ 0 , g 0 )
∥
= O p ( n − 1 / 2 ) .
Thus the above equation implies that ∥ ˆ θ − θ
∥ 2 = O p ( n − 1 / 2 ) .
Thus we get:
[Equation]
or equivalently that:
[Equation]
The first term converges in distribution to the claimed normal limit by invoking the Central Limit
Theorem.
Thus the theorem follows by Slutsky’s theorem.
C Proof of Lemma
Proof.
Before diving into the proof, recall that c ( Z 1: n , x ) =
arg min i ≤ n | X i − x | , and define:
[Equation]
c ∗ 1 ≤ 12 .
That is, we let c ∗ 1 be the index of the nearest example in { X 1 , . . .
, X n } to 12 , and let c ∗ 2 be the index of the nearest example to 12 on the other side of 12 from.
A new observation Z =
( X, Y ) ,
where X ∼ unif[0 , 1] and Y =
I ( X ≤ 0 . 5) , will be misclassified if Y c ( Z 1: n ,X ) is different from Y .
Therefore it is mislabeled if it falls in the following set:
[Equation]
For a given pair of random variables X i 1 , X i 2 , we write:
[Equation]
We note B 1 := { i ≤ n | X i ≤ 1 / 2 } .
Now we remark that if X c ∗ 1 ≤ 0 .
5 then (cid:12)(cid:12)(cid:12) 1 − (cid:104)
X c ∗ 1 + X c ∗ 2 (cid:105)(cid:12)(cid:12)(cid:12) =
X c ∗ 1 + X c ∗ 2 − 1 and
[Equation]
|
[Equation]
∼ min
[Equation]
X
[Equation]
) where U i ( X c ∗ 1 ) | X c ∗ 1 ∼ i.i.d unif[0 , X c ∗ 1 ] .
Here ∼ means "has the same distribution as."
Therefore we have
[Equation]
where | B 1 | denotes the cardinality of set B 1 .
Therefore as X c ∗ 1 → 0 . 5 and | B 1 | → n/ 2 we have
[Equation]
Similarly we remark that if X c ∗ 1 > 0 .
5 then (cid:12)(cid:12)(cid:12) 1 − (cid:104)
X c ∗ 1 + X c ∗ 2 (cid:105)(cid:12)(cid:12)(cid:12)
= 1 − X c ∗ 1 − X c ∗ 2 and
[Equation]
where U bisi ( X c ∗ 1 ) | X c ∗ 1 ∼ i.i.d unif[0 , 1 − X c ∗ 1 ] .
Therefore we have
[Equation]
Therefore as X c ∗ 1 → 0 . 5 and | B 1 | → n/ 2 we have
[Equation]
This directly implies that
[Equation]
where Exp(2) denotes an exponential distribution with rate parameter 2 .
Moreover, we now also show that the expectation E (cid:104) λ
X c ∗ 1 ,c ∗ 2 (cid:105) =
O (1) .
As a first step, we note that we can write
[Equation]
By Azuma’s concentration inequality
[Equation]
we know that
[Equation]
To treat the other term in the sum
[Equation]
we have that
[Equation]
where X 1: n :
= ( X 1 , . .
X n ) and (cid:12)(cid:12)(cid:12) E (cid:16)
X c ∗ 1 , X c ∗ 2 (cid:17)(cid:12)(cid:12)(cid:12) denotes the length of the interval E (cid:16) X c ∗ 1 , X c ∗ 2 (cid:17) .
Now by triangle inequality
[-(cid:20)(cid:12)(cid:12)(cid:12)(cid:12) 
-]
[Equation]
[-[Equation] 1 (cid:12)(cid:12)(cid:12)(cid:12) I (cid:18)(cid:12)(cid:12)(cid:12)(cid:12) | B 1 | n − 1 2 (cid:12)(cid:12)(cid:12)(cid:12) ≤ 1 4 (cid:19)(cid:21) + n 2 E (cid:20)(cid:12)(cid:12)(cid:12)(cid:12) 
-][Table]
[Equation]
where U i := (cid:12) (cid:12) 12 − X i (cid:12) (cid:12) ∼ i .
i .
d . unif[0 , 0 . 5] , and the penultimate line follows from symmetry.
We can express the expectation in terms of integrals of tail probabilities as
[Equation]
| ≥ n 4 (cid:17)
[Equation]
t | B 1 | (cid:12)(cid:12)(cid:12) | B 1 | (cid:19) dt (cid:21) .
Here
[Equation]
where we have used the inequality that 1 − x ≤ e − x for all x .
Hence, we have
[Equation]
Altogether, we have shown that
[Equation]
The key point is to note that
[Equation]
where (a) comes from realizing that
[Equation]
Therefore we proved the first point.
Moreover if we denote
[Equation]
where Z ( − 1) 1: n is Z 1: n with the first observation replaced with an independent copy ˜ Z
= ( ˜ X 1 , ˜ Y 1 ) , then we remark that
[Equation]
where (b) comes from symmetry (each observation has an equal chance of being c ∗ 1 , for example).
Moreover we note that if neither c ∗ ( − 1)̸ =
c ∗ 1 nor c ∗ ( − 1)̸ =
c ∗ 2 then we have E ( X ( − 1) c ∗ ( − 1), X ( − 1) c ∗ ( − 1)) =
E ( X c ∗ 1 , X c ∗ 2 ) .
Now for ease of notation denote
[Equation]
and
[Equation]
Then we have that there is C < ∞ such that
[Equation]
where to get (c) we exploited independence of ( X c ∗ 1 , X c ∗ 2 ) and the events { c ∗ 1 = 1 } , { c ∗ 2 = 1 } , { c ∗
[Equation]
− 1)= 1 } , { c [Equation] 1)= 1 } and where to get (d) we exploited (2).
Moreover we also notice that
[Equation]
] (cid:19) by independence
[Equation]
where c ∗ 3 :=
arg min i ∈ [ n ] \{ c ∗ 1 } (cid:12)(cid:12)
X i − 12 (cid:12)(cid:12) is the index of the second nearest neighbor of 12 among X 1: n .
Note that c ∗ 3 is not necessarily equal to c ∗ 2 , since the definition of c ∗ 2 requires X c ∗ 2 to be on the other side of 12 from X c ∗ 1 while that of c ∗ 3 does not.
By our knowledge of the expectation of the second order statistic among i.i.d.
uniform random variables, we obtain that
[Equation]
Therefore, similarly to before, we also have that there is a constant ˜ c, c > 0 such that
[Equation]
Therefore we proved the second point.
The third point follows because
[Equation]
where V n ( g 0 )
=
V ( g 0 ) = 0 by definition, and V n (ˆ g ) since the nearest neighbor estimator evaluated at a training data point never misclassifies the point.
D Proof of Corollary
Proof.
We note that by monotonicity of L p norms, plugging the bound in (Algorithmic Stability) into the right hand side terms of (1) gives the stability conditions in lemma 2.
Corollary 4 then immediately follows.
E Proof of Theorem
Proof.
Denote Z b 1: m, ( − l ) , b ∈ { 1 , . . .
, B } as the corresponding bagged samples when the l -th data point Z l is replaced with an independent copy ˜ Z l .
We have that for l ∈ [ n ] :
[Equation]
sup x (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 B B (cid:88) b
= (cid:16) ˆ h
( Z b 1: m )( x ) − ˆ h ( Z b 1: m, ( − l ) )
( x ) (cid:17) 1 {∃ t ≤ m s.t. Z bt = Z l } (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r .
The last equality was because
[Equation]
only if
[Equation]
Fix any l ∈ [ n ] .
To simplify notations, let ∇ ˆ h ( Z b )( x ) :
= ˆ h ( Z b 1: m )( x ) − ˆ h ( Z b 1: m, ( − l ) )
( x ) , and let A b be the event {∃ t ≤ m s.t.
Z bt
=
Z l } .
Then (cid:13)(cid:13)(cid:13)(cid:13) sup x ∥ ˆ g ( x ) − ˆ g ( − l ) ( x ) ∥ 2 (cid:13)(cid:13)(cid:13)(cid:13) 2 r = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) sup x (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 B
B (cid:88) b = ∇ ˆ h ( Z b )( x ) 1 A b (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r .
[Table]
(cid:13)(cid:13)(cid:13)
[Equation]
r .
For ease of notations, denote
[Equation]
and denote
R
[Equation]
=
[Equation]
(cid:21) .
For the first term, we have by tower law that
[Equation]
=
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
r
[Equation]
To further simplify
[Equation]
we use the following lemma:
Lemma 7 (Marcinkiewicz-Zygmund inequality) .
Let p ≥ 1 .
If X 1 , . .
, X n are i
[Equation]
i
[Equation]
d. random variables such that E
[ X 1 ] = 0 , then there exists a constant C p such that
[Equation]
Since R b , b ∈
[ B ] are i [Equation] i
[Equation] d. conditional on Z ( l ) [Equation] we then have that
[Equation]
by tower law .
Now also
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
2 r since ∇ ˆ h ( Z 1 )( x ) = 0 for all x on A c
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
2 r since A 1 is independent of Z 1 , . . .
, Z n , ˜ Z l
[Equation]
[Table] x [Table] ∇
[Table]
h
[Table]
Z n ,
[Table]
Z [Table] [Table] ≤ P ( A 1 )
[Table]
sup x
[Table]
∇ ˆ
[Table] Z [Table] [Table]
by Jensen’s inequality and tower law ≤ P ( A [Table] [Table] C by moment condition, since 2 r ≤ s .
By union bound,
P ( A 1 ) ≤ m (cid:88) t =
[Equation]
(cid:1)
=
m n .
Therefore, altogether we obtain (cid:13)(cid:13)(cid:13)(cid:13) sup x ∥ ˆ g ( x ) − ˆ g ( − l ) ( x ) ∥ 2 (cid:13)(cid:13)(cid:13)(cid:13) {+
+} r ≤ C 2 r √
B · ∥ R b ∥ 2 r + 2 C · m n ≤ C 2 r √
B · (cid:13)(cid:13)(cid:13)(cid:13) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z 1 )( x )
A 1 (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13)(cid:13) 2 r + C 2 r √
B · (cid:13)(cid:13)(cid:13)(cid:13) E (cid:20) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z b )( x ) 1 A b (cid:13)(cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:21)(cid:13)(cid:13)(cid:13)(cid:13) 2 r + 2 C · m n ≤ C 2 r √
B · (cid:13)(cid:13)(cid:13)(cid:13) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z 1 )( x )
A 1 (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Hence, we also have max l ≤
[Equation]
Assuming m =
o
[Equation]
√ n )
and
B >> m 2 /k · n 1 − 2 k , this upper bound is of order o ( n −
[Table]
Remark 3.
We could in fact relax the conditions in Theorem 5 by using Rosenthal’s inequality instead of Marcinkiewicz-Zygmund inequality in the proof.
Moreover, we can relax the bounded moments condition to restrict on L 2 r norm instead of on L s norm.
This gives the following theorem.
Theorem 8.
Assume B, m satisfy
[Equation]
and assume the base estimator ˆ h has bounded moments:
[Equation]
for some constant C > 0 .
Then (Algorithmic Stability) is achieved:
[Equation]
Therefore if a and ν satisfy the condition (1) then the condition (2) is satisfied.
Proof of Theorem 8.
We follow the proof of Theorem 5 and obtain
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
by tower law .
We then use the following lemma.
Lemma 9 (Rosenthal’s inequality) .
Let p ≥ 1 .
If X 1 , . .
, X n are i.i.d.
random variables such that E
[ X 1 ] = 0 , then there exists a constant C p such that
[Equation]
Since R b , b ∈
[ B ] are i.i.d.
conditional on Z ( l ) , we then have that by symmetry of distributions
B · E  
E   (cid:32) B (cid:88)
b =
R b (cid:33) 2 r [Equation] Z ( l ) 
 
 1 / 2 r ≤
B · E (cid:34)(cid:16) √ BC 2 r (cid:17) 2 r (cid:18)(cid:16) E (cid:104) R 21 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 + B 12 r − 12 (cid:16) E (cid:104) R 2 r 1 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 r
(cid:19) 2 r (cid:35) 1 / 2 r = C 2 r √
B · (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) E (cid:104) R 21 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 + B 12 r − 12 (cid:16) E (cid:104) R 2 r 1 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 r (cid:13)(cid:13)(cid:13)(cid:13) 2 r .
By triangle inequality, we have
C 2 r √
B · (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) E (cid:104) R 21 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 + B 12 r − 12 (cid:16) E (cid:104) R 2 r 1 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 r (cid:13)(cid:13)(cid:13)(cid:13) 2 r ≤ C 2 r √ B · (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) E (cid:104) R 21 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:105)(cid:17) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13) 2 r + C 2 r √ B · (cid:13)(cid:13)(cid:13)(cid:13)
[Table]
For further ease of notations let
[Equation]
Note that
[Equation]
Then also by triangle inequality and Jensen’s inequality
[Equation]
[Table]
Further
[Table]
we rewrite this term as
[Equation]
[Table]
) (cid:13)(cid:13)(cid:13) 2 2 · 1 A 1 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:21)(cid:19) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r = 2 C 2 r √ B · (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) P (cid:16) A 1 (cid:12)(cid:12)(cid:12) Z ( l ) (cid:17) · E (cid:20) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z 1 )( x ) (cid:13)(cid:13)(cid:13) 2 2 (cid:12)(cid:12)(cid:12) Z ( l ) , A 1 (cid:21)(cid:19) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r since ∇ ˆ h ( Z 1 )( x ) = 0 for all x on A c= 2 C 2 r √ B · (cid:13)(cid:13) (cid:13)(cid:13) (cid:13) (cid:18) P
A 1 ) · E (cid:20) sup x (cid:13) (cid:13)(cid:13) ∇ ˆ
h ( Z 1 )( x ) (cid:13) (cid:13)(cid:13) 2 2 (cid:12) (cid:12)(cid:12) Z
( l ) , A 1 (cid:21)(cid:19) 1 / 2 (cid:13)(cid:13) (cid:13)(cid:13) (cid:13) 2 r since A 1 is independent of Z ( l ) = 2 C 2 r √
B · ( P ( A 1 )) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) E (cid:20) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z 1 )( x ) (cid:13)(cid:13)(cid:13) 2 2 (cid:12)(cid:12)(cid:12) Z ( l ) , A 1 (cid:21)(cid:19) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r .
By Jensen’s inequality and tower law, we have 2 C 2 r √
B · ( P ( A 1 )) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) E (cid:20) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z 1 )( x ) (cid:13)(cid:13)(cid:13) 2 2 (cid:12)(cid:12)(cid:12) Z ( l ) , A 1 (cid:21)(cid:19) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 r ≤ 2 C 2 r √ B · ( P ( A 1 )) 1 / 2 (cid:13)(cid:13)(cid:13)(cid:13) sup x (cid:13)(cid:13)(cid:13) ∇ ˆ h ( Z 1 )( x ) (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13)(cid:13) 2 r ≤ 2 C 2 r √
B · (cid:16) m n (cid:17) 1 / 2 · 2 C.
Similarly, by replacing the powers of 2 and 1 / 2 with 2 r and 1 / 2 r [Equation] we can show that
[Equation]
Altogether [Equation] we have
[Equation]
Assuming
[Equation] and
[Equation]
this upper bound is of order o ( n − 1 / 2 ) :
[Equation]
F Proof of Lemma
Proof.
Lemma 6 follows from Theorem 3 of [35] by taking ψ ( Z ; θ ) :
= Y − θ ( x ) .
G Establishing L 2 r - and Mean-Squared-Continuity for Examples 1, 2, and
Recall that m ( Z ; θ, g ) = a [Equation] Z ; g
[Equation]
+ ν [Equation] Z ; g ) . We will establish L 2 r -continuity and mean-squared-continuity for Examples 1, 2, and 3 in this section.
G.1 Establishing for Example
For this example, we have
[Equation]
Let v ≥ 1 be the constant such that 1 = 1 r + 1 v .
Denote T i , i ∈
[ p ] as the i -th coordinate of T .
Denote Y i , i ∈
[ p ] as the i -th coordinate of Y .
For any function p denote p
i ( X ) , i ∈
[ p ] as the i -th coordinate of p ( X ) .
We will show that subject to
[Equation]
being finite for all i ∈ [ p ] then L 2 r -continuity conditions hold.
Moreover, we will show that if further
[Equation]
are finite for all i ∈
[ p ] then mean-squared-continuity conditions hold with q = 2 .
We will illustrate for function a and for function ν separately.
For function a :
We first verify L 2 r -continuity for function a .
We have that for any i, j ∈ [ p ]
[Equation]
Then by triangle inequality and Hölder’s inequality
[Equation]
we obtain
[Equation]
[Table]
ˆ p ( − l ) i ( x ) − ˆ p
i ( x ) (cid:12)(cid:12)(cid:12)(cid:13)(cid:13)(cid:13)(cid:13) 2 r · (cid:13)(cid:13)(cid:13) ˆ p ( − l ) j
( X )
+ ˆ p j ( X ) (cid:13)(cid:13)(cid:13) 2 v +2 (cid:13)(cid:13)(cid:13)(cid:13) sup
[Equation]
(cid:12)(cid:12)(cid:12) [Equation] l ) j [Equation]
[Equation] ) (cid:12)(cid:12)(cid:12)(cid:13)(cid:13)(cid:13)(cid:13) 2 r · (cid:13)(cid:13)(cid:13) ˆ p ( − l ) i ( X ) + ˆ p
i ( X ) (cid:13)(cid:13)(cid:13) 2 v ( a ) ≤ L 1 · (cid:13)(cid:13)(cid:13)(cid:13)
sup x (cid:13)(cid:13)(cid:13)
[Equation] l
[Equation] [Equation] (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) 2 r where
L 1 :
[Equation]
Here for (a) we have used the triangle inequality, and for (b) we have used the fact that Z l and ˜ Z l have the same distribution.
By replacing Z with Z l , we can similarly show that
[Equation]
where
[Equation]
Hence, the L 2 r -continuity conditions hold for function a provided that all the aforementioned L 2 vnorm quantities are finite.
Now we check the mean-squared-continuity conditions for function a.
For any g, g ′ , any i, j ∈
[ p ] , we have
[Equation]
Then by triangle inequality and Hölder’s inequality [Equation] we obtain
[Equation]
where
[Equation]
Provided all these L ∞ -norm quantities are finite, mean-squared-continuity conditions hold for function a with q = 2 .
For function ν :
We have
[Equation]
Hence, by triangle inequality and Hölder’s inequality
[Equation]
we similarly obtain that
[Equation]
By replacing Z with Z l , we can similarly show that
[Equation]
Therefore, the L 2 r -continuity conditions hold for function ν provided that all these L 2 v -norm quantities are finite.
As for mean-squared-continuity, note that we have that
[Equation]
p
[Equation]
p
[Equation]
X
[Equation]
Hence, by triangle inequality and Hölder’s inequality, we derive that
[Equation]
Provided all these L ∞ -norm quantities are finite, mean-squared-continuity conditions hold for function ν with q = [Equation]
G.2 Establishing for Example
For this example [Equation] we have
[Equation]
Denote Z i , i ∈
[ p ] as the i -th coordinate of Z .
Denote r i ( X )
[Equation]
i ∈ [ p ] as the i -th coordinate of r ( X ) .
Analogously to Example 1, replacing all functions p i and their estimates with r i and their corresponding estimates, replacing all T i with Z i in the analysis of a i,j and ν i , we can show that subject to
[Equation]
being finite for all i ∈ [ p ] then L 2 r -continuity conditions hold.
Moreover, if further ∥
[Equation]
∞ , ∥ q ′ ( X ) ∥ ∞ are finite for all i ∈
[ p ] then mean-squared-continuity conditions hold with q = 2 .
G.3 Establishing for Example
For this example, we have
[Equation]
The L 2 r -continuity and mean-squared-continuity conditions trivially hold for function a .
For function ν , we have
[Equation]
− [Equation] T, X )) . Hence, by triangle inequality and Hölder’s inequality we obtain ∥ ν ( Z ; ˆ g ) − ν
( Z ; ˆ g ( − l ) ) ∥≤ ∥ m b ( Z ; ˆ q ( − l ) ) − m b
( Z ; ˆ q ) ∥+
∥ ˆ µ ( − l ) ( T, X
[Equation]
ˆ
[Equation]
X [Equation] ( − [Equation]
, X
[Equation]
ˆ
[Equation]
( T, X )) (ˆ µ ( T, X ) − ˆ µ ( − l ) ( T, X )) ∥≤ ∥ m b ( Z ; ˆ q ( − l ) ) − m b ( Z ; ˆ q ) ∥+ ∥ ˆ µ ( −
[Equation]
T
[Equation]
X
[Equation]
∥
[Equation]
· ∥ ˆ
[Equation]
( T, X
[Equation] − ˆ [Equation] ( − l ) ( T, X ) ∥
[Equation]
r +
[Equation]
−
[Equation]
∥
[Equation]
( −
[Equation]
T
[Equation]
X
[Equation]
∥
[Equation]
∥
[Equation]
b ( Z ; ˆ q ( − l ) )
− m b ( Z
[Equation]
ˆ q ) ∥+
∥ ˆ µ (
[Equation]
Since m b is a linear functional, there exists L m > 0 such that
[Equation]
Hence, we have
[Equation]
Analogously, by replacing Z with Z l , we can show that
[Equation]
for some constant ˜ L m > 0 .
Therefore, subject to
[Equation]
being finite for all i ∈ [ p ] then L 2 r -continuity conditions also hold for function ν .
Moreover
[Equation]
we have
[Equation]
+ ∥ q
[Equation]
T, X ) ∥ ∞ } where since m b is a linear functional, there exists L b > 0 such that
[Equation]
Provided that
L b , ∥ µ ′ ( T, X ) ∥ ∞ , ∥ Y ∥ ∞ , ∥ q ( T, X ) ∥ ∞ are finite, mean-squared-continuity conditions hold for function ν with q = 2 .
H Extension to Nonlinear Moments
In this section, we extend our results to the case where the moment function m ( Z ; θ, g ) is not necessarily linear in the target parameter θ.
For simplicity, we assume that the nuisance estimator ˆ g is symmetric in each of the training data points Z 1 , . . .
, Z n .
Moreover, we will denote with Z a fresh random draw from the distribution.
We introduce some notation.
We denote with ∥ · ∥ 2 , 2 the norm of a random vector defined as: ∥ Z ∥ 2 , 2 = (cid:112) E [ (cid:80) i
Z
2 i ] , which can also be thought as taking the L 2 norm of each coordinate and and then taking the ℓ 2 vector norm of this vector, or equivalently taking the L 2 norm of the random variable defined as the ℓ 2 norm of the random vector.
For clarity, for any random vector Z we will denote with ∥ Z ∥ v, 2 the random variable that corresponds to the ℓ 2 vector norm of the random vector, i.e. ∥ Z ∥ v, 2 = (cid:112)(cid:80) i
Z 2 i .
For any random variable V , denote with ∥ V ∥ 2 its ℓ 2 norm (cid:112) E
[ V 2 ] .
Note that for any random vector Z , we have ∥ Z ∥ 2 , 2 = ∥∥
Z ∥ v, 2 ∥ 2 .
Firstly, we establish a consistency lemma for ˆ θ .
We note that in the linear moment case, such a separate proof of consistency was not required and a single step proof of asymptotic normality was feasible due to linearity.
In the non-linear case, as is typical for moment based estimators, we first need to show that the estimate will eventually lie in a small ball around θ 0 , and then argue normality.
This is what the consistency lemma achieves.
Lemma 10 (Consistency) .
Assume that
The parameter space Θ ⊂ R p for target parameter θ is compact.
θ 0 is the unique solution of θ to the equation M ( θ, g 0 )
= 0 .
The moment function m ( z ; θ, g ) is uniformly continuous in θ over all Θ and a suficciently small ℓ 2 ball B 2 ( g 0 ) ⊆
G around g 0 .
That is, ∀ ϵ > 0 , ∃ δ > 0 such that for any ˜ θ 1 , ˜ θ 2 ∈ Θ with ∥ ˜ θ 1 − ˜ θ 2 ∥ 2 < δ , ∀ g ∈ B 2 ( g 0 ) , ∀ z , we have
[Equation]
The moment function m ( Z ; θ, g ) is mean-squared-continuous in g , uniformly in θ , i.e. ∃ L > 0 and q > 0 such that:
[Equation]
Estimator ˆ g of the nuisance function is consistent: as n → ∞
[Equation]
The moment function m and estimator ˆ g satisfies the following o (1) leave-one-out stability condition:
[Equation]
as n → ∞ .
Then any estimator ˆ θ that satisfies that M n (ˆ θ, ˆ g ) =
o p (1) , also satisfies that ˆ θ p → θ 0 .
Proof.
Fix any ϵ > 0 .
Since m ( z ; θ, g ) is uniformly continuous in θ ∈ Θ for a sufficiently small ℓ 2 -ball B 2 ( g 0 ) around g 0 , we have that, ∃ δ > 0 such that for any ˜ θ 1 , ˜ θ 2 ∈ Θ with ∥ ˜ θ 1 − ˜ θ 2 ∥ 2 < δ , ∀ g ∈ B 2 ( g 0 ) , ∀ z , we have
[Equation]
Then
[Equation]
(that is, M ( · , g 0 ) is uniformly continuous) and
[Equation]
Since the parameter space Θ is compact, there exist θ j , j = 1 , . . .
, J such that
[Equation]
By Law of Large Numbers, we have ∀ j, as n → ∞
[Equation]
Hence, ∀ η > 0 , for every j there exists n j such that ∀ n > n j
[Equation]
Then ∀ n > max j n j , we have
[Equation]
Moreover, for any j ∈ J , we have that:
[Equation]
q
[Equation]
= o (1) .
Thus we have that M n ( θ j , ˆ g ) − M n ( θ j , g 0 )
=
o p (1) .
Which means that ∀ η > 0 , there exists n j such that for every n > n j :
[Equation]
Then ∀ n > max j ∈ J n j : P
[Equation]
max
[Equation]
(cid:19) <
η 3 .
Now ∀ θ ∈ Θ , since Θ ⊂ ∪ Jj =1 B ( θ j , δ ) , there exists k ∈ { 1 , . .
, J } such that ∥ θ − θ k ∥ 2 < δ .
Then for n sufficiently large, such that ˆ g ∈ B 2 ( g 0 ) : ∥ M n
[Equation]
θ, ˆ
[Equation]
θ, g
[Equation]
v,
[Equation]
,≤ max
[Equation]
Hence, we obtain
P
[Equation]
max
[Equation]
Moreover, note that by the triangle inequality:
[Equation]
Thus:
[Equation]
And we conclude that:
[Equation]
This shows that
[Equation]
as n → ∞ .
In particular, this implies that
[Equation]
as n → ∞ .
Hence, by triangle inequality and the fact that M n (ˆ θ, ˆ g ) =
o p (1) , we obtain
[Equation]
Hence, M (ˆ θ, g 0 ) =
o p (1) .
It remains to show that
[Equation]
as n → ∞ .
To achieve this, again fix any ϵ > 0 .
Then since Θ is compact, B ( θ 0 , ϵ ) c is also compact as a closed subset of Θ .
By continuity of θ (cid:55)→ ∥
M ( θ, g 0 ) ∥ 2 and the fact that θ 0 is the unique solution to
M ( θ 0 , g 0 )
= 0 , we must have that ∥ M ( θ, g 0 )
∥ v, 2 is bounded away from zero on B ( θ 0 , ϵ ) c .
That is, ∃ η > 0 such that for any θ with ∥ θ − θ
∥ v, 2 ≥ ϵ , ∥
M ( θ, g 0 ) ∥ v, 2 > η.
Then since M (ˆ θ, g 0 )
=
o p (1) , there exists N ∈ N such that ∀ n > N
[Equation]
Then ∀ n > N
[Equation]
< ϵ.
This establishes consistency of ˆ θ.
Now we extend Theorem 1 to nonlinear moments.
Theorem 11.
Let A ( θ, g ) :
=
∂ θ M ( θ, g ) denote the Jacobian of the moment vector, with respect to θ and H i ( θ, g ) :
= ∂ 2 θ M i ( θ, g ) denote the Hessian of the i -th moment coordinate.
Suppose that the moment m is twice differentiable with a ( z ; θ, g ) :
= ∂ θ m ( z ; θ, g ) and h i ( z ; θ, g ) :
= ∂ 2 θ m i ( z ; θ, g ) .
Let
A n ( θ, g ) :
= 1 n (cid:80) n i =1 a ( Z i ; θ, g ) and H i,n := 1 n (cid:80) n i =1 h
( Z i ; θ, g ) denote the empirical counterparts of A, H i .
Suppose that the nuisance estimate ˆ g ∈ G satisfies:
[Equation]
Suppose that the moment satisfies the Neyman orthogonality condition: for all g ∈ G
[Equation]
Neyman [Equation] ) and a second-order smoothness condition: for all g ∈ G
[Equation]
Smoothness)
Assume that A ( θ 0 , g 0 ) − 1 exists and that for any g, g ′ ∈ G :
[Equation]
Lipschitz
[Equation]
Suppose that the moment m and estimator ˆ g satisfy the stochastic equicontinuity conditions:
[Equation]
Suppose that the conditions in Lemma 10 are true.
Moreover, assume that for any i, j ∈ [ p ] × [ p ] , the random variable a i,j ( Z ; θ 0 , g 0 ) has bounded variance and that ∥ θ 0 ∥ 2 = O (1) .
Suppose that ∃ open neighborhood W of g 0 such that
[Equation]
Let ˆ θ denote any approximate solution to the plug-in empirical moment equation that satisfies
M n (ˆ θ, ˆ g ) =
o p ( n − 1 / 2 ) .
Then ˆ θ is asymptotically normal:
[Equation]
Proof.
Let A i ( θ, g ) denote the i -th column of the Jacobian matrix.
By Taylor’s expansion, we have ∀ g
and i ∈
[ p ]
[Equation]
for some ˜ θ i between ˆ θ
and θ 0 .
By Lemma 10, we know that
[Equation]
Further, by the bounded Hessian condition and consistency of ˆ θ , we know that uniformly for g ∈ W
[Equation]
Moreover, for any g with ∥ g − g 0 ∥ 2 = o p (1) we have
[Equation]
where the second to last equality uses the Lipschitz condition.
By consistency, we have ∥ ˆ g − g
∥ 2 = o p (1) and that ˆ g ∈ W for n large enough.
Hence, we obtain that for large enough n ,
[Equation]
where the last line follows because by definition M n (ˆ θ, ˆ g )
=
o p ( n − 1 / 2 ) .
By Neyman orthogonality and the boundness condition on the second derivative of M with respect to g , for any g we have
[Equation]
Plugging in g
=
ˆ g , noting that ∥ g − g
∥ 22 = o p ( n − 1 / 2 )
we obtain
[Equation]
p
[Equation]
[Equation]
Let G n ( θ, g ) :
= M ( θ, g ) − M n ( θ, g ) .
Thus we have that
[Equation]
We decompose G n (ˆ θ, ˆ g ) into the following sum:
[Equation]
By Taylor’s expansion [Equation] we have
[Equation]
Now
[Equation]
where the last equality follows from the stochastic equicontinuity condition on the Jacobian.
Since
A ( θ 0 , g 0 ) − A n ( θ 0 , g 0 ) is a mean zero empirical process with ∂ θ m ( Z ; θ 0 , g 0 ) having bounded variance, we have
[Equation]
Hence,
G n
[Equation]
[Equation]
Moreover,
[Equation]
by stochastic equicontinuity.
In summary [Equation] we have
[Equation]
and thus
[Equation]
That is, we have
[Equation]
Since G n ( θ 0 , g 0 ) is a mean-zero empirical process, we have that ∥ G n ( θ 0 , g 0 )
∥
= O p ( n − 1 / 2 ) .
By consistency of ˆ θ , above implies that
[Equation]
Thus we get
[Equation]
By Slutsky’s Theorem [Equation] we conclude that
[Equation]
We can also directly extend Lemma 2 to general nonlinear moments.
Lemma 12 (Non-Linear Main Lemma) .
If the estimation algorithm satisfies the stability conditions: for all i, j ∈ [ p ]
[Equation]
and the moment satisfies the mean-squared-continuity condition:
[Equation]
for some 0 < q < ∞ and some L > 0 , then the Condition (NonLin.
Stoc.
Equi.) is satisfied.
Proof.
The proof follows by replacing all functions a ( z, g ) , ν ( z, g ) in the proof of Lemma 2 correspondingly with the functions a ( z ; θ 0 , g ) and m ( z ; θ 0 , g ) .
Application to bagging estimators.
We finally note that if the moment satisfies Lipschitz conditions of the form:
[Equation]
Then Theorem 5 can be applied to upper bound the right hand side of these inequalities by o p ( n − 1 / 2 ) for bagging estimators.
This would then imply the stability conditions invoked in both the consistency and the normality theorem.
Thus the main application for bagging estimators carries over to non-linear moments.