D EEP
J UMP
Q-E VALUATION
FOR
O FFLINE
P OLICY
E VALUATION IN C ONTINUOUS A CTION S PACE
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider off-policy evaluation (OPE) in continuous action domains, such as dynamic pricing and personalized dose ﬁnding.
In OPE, one aims to learn the value under a new policy using historical data generated by a different behavior policy.
Most existing works on OPE focus on discrete action domains.
To handle continuous action space, we develop a brand-new deep jump Q-evaluation method for OPE.
The key ingredient of our method lies in adaptively discretizing the action space using deep jump Q-learning.
This allows us to apply existing OPE methods in discrete domains to handle continuous actions.
Our method is further justiﬁed by theoretical results, synthetic and real datasets.
I NTRODUCTION
Individualization proposes to leverage omni-channel data to meet individual needs.
Individualized decision making plays a vital role in a wide variety of applications.
Examples include customized pricing strategy in economics (Qiang & Bayati, 2016; Turvey, 2017), individualized treatment regime in medicine (Chakraborty, 2013; Collins & Varmus, 2015), personalized recommendation system in marketing (McInerney et al., 2018; Fong et al., 2018), etc.
Prior to adopting any decision rule in practice, it is crucial to know the impact of implementing such a policy.
In many applications, it is risky to run a policy online to estimate its value (see, e.g., Li et al., 2011).
Off-policy evaluation (OPE) thus attracts a lot of attention by learning the policy value ofﬂine using logged historical data.
Despite the popularity of developing OPE methods with a ﬁnite set of actions (see e.g., Dud´ık et al., 2011; 2014; Swaminathan et al., 2017; Wang et al., 2017), less attention has been paid to continuous action domains, such as dynamic pricing (den Boer & Keskin, 2020) and personalized dose ﬁnding (Chen et al., 2016).
Recently, a few OPE methods have been proposed to handle continuous actions (Kallus & Zhou, 2018;
Sondhi et al., 2020; Colangelo & Lee, 2020).
All these methods rely on the use of a kernel function to extend the inverse probability weighting (IPW) or doubly robust (DR) approaches developed in discrete action domains.
They suffer from three limitations.
First, the validity of these methods requires the conditional mean of the reward given the feature-action pair to be a smooth function over the action space.
This assumption could be violated in applications such as dynamic pricing, where the expected demand for a product has jump discontinuities as a function of the charged price (den Boer & Keskin, 2020).
Second, the value estimator could be sensitive to the choice of the bandwidth parameter in the kernel function.
It remains challenging to select this hyperparameter.
Kallus & Zhou (2018) proposed to tune this parameter by minimizing the mean squared error of the resulting value estimator.
However, their method is extremely computationally intensive in moderate or high-dimensional feature space; see Section 5 for details.
Third, these kernel-based methods typically use a single bandwidth parameter.
This is sub-optimal in cases where the second-order derivative of the conditional mean function has an abrupt change in the action space; see the toy example in Section 3.1 for details.
To address these limitations, we develop a deep jump Q-evaluation (DJQE) method by integrating multi-scale change point detection (see e.g., Fryzlewicz, 2014), deep learning (LeCun et al., 2015) and OPE in discrete action domains.
The key ingredient of our method lies in adaptively discretizing the action space using deep jump Q-learning.
This allows us to apply IPW or DR methods to handle continuous actions.
It is worth mentioning that our method does not require kernel bandwidth selection.
Theoretically, we show it allows the conditional mean to be either a continuous or piecewise function of the action (see Theorems 1 and 2).
Empirically, we show it outperforms existing state-of-the-art OPE methods in both synthetic and real datasets.
P RELIMINARIES
We ﬁrst formulate the OPE problem.
We next discuss the kernel-based OPE methods and multi-scale change point detection, since our proposal is closely related to them.
2. O FF - POLICY EVALUATION
The observed datasets can be summarized into { ( X i , A
i , Y i ) } 1 ≤ i ≤ n where O i =
( X i , A
i , Y i ) denotes the feature-action-reward triplet for the i th subject and n denotes the total sample size.
We assume these data triplets are independent copies of some population variables ( X, A, Y ) .
Let
X and A denote the feature and action space, respectively.
We focus on the setting where A is one-dimensional, as in dynamic pricing and personalized dose ﬁnding.
A deterministic policy π : X → A determines the action to be assigned given the observed feature.
We use b to denote the behavior policy that generates the observed data.
Speciﬁcally, b ( •| x ) denotes the probability density or mass function of A given X = x , depending on whether A is continuous or not.
Deﬁne the expected reward function conditional on the feature-action pair as
[Equation]
We refer to this function as the Q-function, to be consistent with the literature on developing individualized treatment regime (Murphy, 2003).
As standard in the OPE and the causal inference literature (see e.g., Chen et al., 2016), we assume the stable unit treatment value assumption (SUTVA), no unmeasured confounders assumption, and the positivity assumption are satisﬁed.
These assumptions guarantee that a policy’s value is estimable from the observed data.
Speciﬁcally, for a given target policy π , its value can be represented by
[Equation]
The goal of the OPE is to learn the value V ( π ) based on the observed data.
2. K ERNEL -B ASED OPE
For discrete action, Zhang et al.
and Dud ´ ık et al.
(2011) proposed a DR estimator of V ( π ) by
[Equation]
where I denotes the indicator function, (cid:98) Q and (cid:98) b denote some estimators for the Q-function and the behavior policy.
The second term (cid:98) b − 1 ( A i | X i )
I
( A i =
π ( X i ))
{ Y i − (cid:98) Q ( X i , A i ) } inside the bracket corresponds to an augmentation term.
Its expectation equals zero when (cid:98) Q = Q .
The purpose of adding this term is to offer additional protection against potential model misspeciﬁcation of the Qfunction.
Such an estimator is doubly-robust in the sense that its consistency relies on either (cid:98) Q or (cid:98) b to be correctly speciﬁed.
By setting (cid:98) Q = 0 , equation 1 is reduced to the IPW estimator.
In continuous action domains, the indicator function I ( A i =
π ( X i )) equals zero almost surely.
Consequently, naively applying equation 1 yields the plug-in estimator (cid:80) ni
=1 (cid:98) Q
( X i , π ( X i ))
/n .
To address this concern, the kernel-based OPE proposed to replace the indicator function in equationwith a kernel function K { ( A i − π ( X i )) /h } with some bandwidth parameter h , i.e., n
[Equation]
The bandwidth h represents a trade-off.
The variance of the resulting value estimator decays with h .
Yet, its bias increases with h .
More speciﬁcally, it follows from Theorem 1 of Kallus & Zhou (2018) that the leading term of the bias is equal to
[Equation]
To ensure the term in 2 decays to zero as h goes to 0, it requires the expected second derivative of the Q-function exists, and thus Q ( x, a ) needs to be a smooth function of a .
However, as commented in the introduction, this assumption could be violated in applications such as dynamic pricing.
[Table]
[Figure]
2. M ULTI -S CALE C HANGE P OINT D ETECTION
The change point analysis considers an ordered sequence of data, Y 1: n = { Y 1 , · · · , Y n } , with unknown change point locations, τ = { τ 1 , · · · , τ K } for some unknown integer K .
Here, τ i is an integer between 1 and n − 1 inclusive and satisﬁes τ i <
τ j for i < j .
These change points split the data into K + 1 segments.
Within each segments, the expected response is a constant function (see the left panel of Figure 1 for details).
A number of methods have been proposed on estimating change points (see for example, Boysen et al., 2009; Killick et al., 2012; Frick et al., 2014;
Fryzlewicz, 2014, and the references therein), by minimizing a penalized objective function:
[Equation]
where C is a cost function that measures the goodness-of-the-ﬁt of the constant function within each segment and nγK penalizes the number of change points.
We remark that all the above cited works focused on models without features.
Our proposal goes beyond these works in that we consider models with features and use deep neural networks (DNN) to capture the complex relationship between the response and features.
D EEP J UMP Q-E VALUATION
In section 3.1, we use a toy example to demonstrate the limitation of kernel-based methods.
We present the main idea of our algorithm in Section 3.2.
Details are given in Section 3.3.
3. T OY E XAMPLE
As discussed in the introduction, existing kernel-based OPE methods use a single bandwidth to construct the value estimator.
Ideally, the bandwidth h in the kernel K { ( A i − π ( X i ))
/h } shall vary with π ( X i ) to improve the accuracy of the value estimator.
To elaborate this, consider the Q-function
Q ( x, a ) =
10 max { a 2 − 0 .
25 , 0 } log( x + 2) for any x, a ∈ [0 , 1] .
By deﬁnition, the Q-function is smooth over the entire feature-action space.
However, it has different “patterns” when the action belongs to different intervals.
Speciﬁcally, for a ∈ [0 , 0 . 5] , Q ( x, a ) is constant as a function of a .
For a ∈ (0 . 5 , 1] , Q ( x, a ) depends quadratically in a .
See the middle panel of Figure 1 for details.
Consider the target policy π ( x ) = x .
We decompose the value V ( π ) into V (1) ( π ) + V (2) ( π ) where
V (1) ( π ) =
E Q ( X, π ( X ))
I ( π ( X ) ≤ 0 . 5) and V (2) ( π ) =
E Q ( X, π ( X ))
I ( π ( X ) > 0 . 5) .
Similarly, denote the corresponding kernel-based value estimators by
[Equation] i [Equation] Since Q ( x, a ) is a constant function of a ∈ [0 , 0 . 5] , its second-order derivative ∂ 2 Q ( x, a ) /∂aequals zero.
In view of 2, when π ( x ) ≤ 0 . 5 , the bias of (cid:98) V (1) h
( π ) will be small even with a sufﬁciently large h .
As such, a large h is preferred to reduce the variance of (cid:98) V (1) h ( π ) .
When π ( x ) > 0 . 5 , a small h is preferred to reduce the bias of (cid:98) V (2) h ( π ) .
See Table 1 for details where we report the bias and standard deviation of (cid:98) V (1)
h ( π ) and (cid:98) V (2) h ( π ) with two different bandwidths.
Due to the use of a single bandwidth, the kernel-based estimator suffers from either a large bias or a large variance.
To overcome this limitation, we propose to adaptively discretize the action space into a union of disjoint intervals such that within each interval I , the Q-function { Q ( x, a ) : a ∈ I} can be well-approximated by some function Q I ( x ) that is constant in a ∈ I .
Based on the discretization, one can apply IPW or DR to evaluate the value.
The advantage of adaptive discretization is illustrated in the right panel of Figure 1.
When a ≤ 0 .
5 , the Q-function is constant in a .
It is likely that our procedure will not further split the interval [0 , 0 . 5] .
Consequently, the corresponding DR estimator for V (1) ( π ) will not suffer from large variance.
When a > 0 . 5 , our procedure will split (0 . 5 , 1] into a series of sub-intervals, approximating Q by a step function.
This guarantees the resulting DR estimator for V (2) ( π ) will not suffer from large bias.
Consequently, the proposed value estimator achieves a smaller mean squared error than kernel-based estimators.
See Table 1 for details.
3. T HE M AIN I DEA
For simplicity, we set the action space A =
From now on, we focus on a subset of intervals in [0 , 1] .
By interval we always refer to those of the form [ c, d ) for some 0 ≤ c < d < 1 , or [ c, 1] for some 0 ≤ c < 1 , denoted as I .
A discretization D for A is deﬁned as a collection of mutually disjoint intervals that covers A .
Let |D| denote the number of intervals in D and |I| denote the size of the interval
I .
We aim to identify an “optimal” discretization (cid:98) D such that for each interval I ∈ (cid:98) D ,
Q ( x, a ) is approximately a constant function of a ∈ I .
The number of intervals in (cid:98) D represents a trade-off.
If | (cid:98) D| is too large, then (cid:98) D will contain many short intervals, the resulting IPW or DR estimator might suffer from large variance.
Yet, a smaller value of | (cid:98) D| might result in a large bias.
Our proposed method adaptively determines (cid:98) D and its size | (cid:98) D| as illustrated below.
To begin with, we cut the entire action space
A into m initial intervals: [0 , 1 /m ) , [1 /m, 2 /m ) , . . .
, [( m − 1) /m, 1] .
The number m shall be sufﬁciently large such that the Q-function can be wellapproximated by a piecewise function on these intervals.
In practice, we recommend to set the initial number of intervals m to be proportional to the sample size n .
Note the set of these initial intervals is not the ﬁnal partition (cid:98) D that we recommend, but only serve as the initial candidate intervals.
We next adaptively combine some of these initial intervals to form the ﬁnal partition (cid:98) D .
As shown in our numerical studies, the size of the ﬁnal partition | (cid:98) D| is usually much less than m .
More speciﬁcally, denoted by B ( m ) as the set of discretizations D such that the end-points of each interval
I ∈ D lie on the grid { j/m : j = 0 , 1 , · · · , m } .
We associate to each partition D ∈ B ( m ) a collection of functions { Q I } I∈D .
These functions depend only on features, not the action.
They are used to produce a piecewise approximation of the Q-function such that Q ( a, • )
≈ (cid:80)
I I ( a ∈
I ) Q
I ( • ) .
We model these Q I using deep neural networks, to capture the complex dependence between the response and features.
When the Q-function is well-approximated, we expect the least square loss (cid:80) I∈D [-(cid:104)(cid:80) ni =
 I ( A i ∈ I ) (cid:8) Y-](cid:104) (cid:80) n i =
I
( A i ∈ I ) (cid:8) Y i − Q
I ( X i ) (cid:9) 2 (cid:105) , will be small.
Consequently, (cid:98) D can be estimated by solving [Equation] } ) (cid:32)(cid:88)
I∈D (cid:34) n (cid:88) i = I ( A i [Equation] I ) (cid:8)
[Equation]
I (
[Equation]
i
[Equation]
(cid:9)(cid:35)
[Equation] nγ [Equation] D| (cid:33) , (3) for some regularization parameter γ and DNN class Q .
Here, the penalty term nγ |D| in equationcontrols the total number of intervals in (cid:98) D , as in multi-scale change point detection.
The tuning parameter γ can be selected via cross-validation.
We refer to this step as deep jump Q-learning.
Details of this step are given in the next section.
Given (cid:98) D , one can apply IPW or DR (see equation 1) to derive the value estimates.
To further reduce the bias of the value estimator, we employ a data splitting and cross-ﬁtting strategy, which is commonly used in statistics (Romano & DiCiccio, 2019).
That is, we use different subsets of data samples to learn the discretization (cid:98) D and to construct the value estimator.
A pseudocode summarizing our algorithm is given in Algorithm 1 in the appendix.
We present the details below.
3. T HE C OMPLETE A LGORITHM
We present the details for DJQE in this section.
It consists of three steps: data splitting, deep jump
Q-learning, and cross-ﬁtting.
Step 1: Data Splitting: We divide all n samples into L subsets of equal size, where L (cid:96) denotes the indices of samples in the (cid:96) th subset for (cid:96) = 1 , · · · , L .
Let L c(cid:96)
= { 1 , 2 , · · · , n } − L (cid:96) as the complement of L (cid:96) .
Step 2: Deep Jump Q-Learning:
For each (cid:96) = 1 , · · · , L , we propose to apply deep jump Qlearning to compute a discretization (cid:98) D ( (cid:96) ) and { (cid:98) q
( (cid:96) )
I : I ∈ (cid:98) D ( (cid:96) ) } by solving a version of equationusing the data subset in L c(cid:96) only.
We next present the computational details for solving this optimization.
Our approach is motivated by the PELT method (Killick et al., 2012) in multi-scale change point detection.
Speciﬁcally, for any interval I , deﬁne (cid:98) q
( (cid:96) )
I as the minimizer of
[Equation]
where | L c(cid:96) | denotes the number of samples in L c(cid:96) .
Deﬁne the cost function C ( (cid:96) )
( I ) as the minimum value of the objective function 4.
In our implementation, we set Q as the class of multilayer perceptrons (MLPs, see Figure A in the appendix for an illustration) with L hidden layers and H nodes each hidden layer.
The above optimization can be solved via the MLP regressor implementation of
Pedregosa et al.
Computation of (cid:98) D ( (cid:96) ) relies on dynamic programming (Friedrich et al., 2008).
For any integer 1 ≤ v ∗ < m , denote by B ( m, v ∗ ) the set consisting of all possible discretizations D v ∗ of [0 , v ∗ /m ) .
Set
B ( m, m ) =
B ( m ) , we deﬁne the Bellman function as
[Equation]
(cid:33) , and Bell (0)
= − γ. Our algorithm recursively updates the Bellman function based on the following formula,
[Equation]
where R v ∗ is the candidate change points list updated by
[Equation]
during each iteration with R 0 = { 0 } .
The constraint listed in 6 is important as it facilitates the computation by discarding change points not relevant to obtain the ﬁnal discretization, leading to a linear computational cost (Killick et al., 2012).
To solve equation 5, we search the optimal change point location v that minimizes Bell ( v ∗ ) .
This requires to apply the MLP regressor to learn (cid:98) Q ( (cid:96) )
[ v/m,v ∗ /m ) and C ( (cid:96) )
([ v/m, v ∗ /m )) for each v ∈ R v ∗ .
Let v 1 be the corresponding minimizer.
We then deﬁne the change points list τ ( v ∗ )
= { v 1 , τ ( v 1 ) } .
This procedure is iterated to compute Bell ( v ∗ ) and τ ( v ∗ ) for v ∗ = 1 , . . .
, m .
The optimal partition (cid:98) D ( (cid:96) ) is determined by the values stored in τ (see Algorithm 1 in the appendix for details).
Step 3.
Cross-Fitting:
For each interval in the estimated optimal partition (cid:98) D ( (cid:96) ) , we estimate the generalized propensity score function
Pr ( A ∈ I| X = x ) via the MLP regressor using the training dataset L c(cid:96) .
Let (cid:98) b
( (cid:96) )
( I| x ) denote the resulting estimate.
The ﬁnal estimated value for V ( π ) is constructed via cross-ﬁtting, given by,
[Equation]
Note the samples used to construct (cid:98) V inside bracket are independent from those to estimate (
cid:98) q
( (cid:96) )
I , (cid:98) b [Equation]
I and (cid:98) D ( (cid:96) ) .
This helps remove the bias induced by overﬁtting in the estimation of (cid:98) q ( (cid:96) )
I , (cid:98) b ( (cid:96) )
I and (cid:98) D ( (cid:96) ) .
T HEORY
We investigate the theoretical properties of the proposed estimator.
For simplicity, let the support
X = [0 , 1] p .
We will show our estimator is consistent when the Q-function is either a piecewise function or a continuous function of a .
Speciﬁcally, consider the following two model assumptions.
Model 1 (Piece-wise constant function).
Suppose
Q
[Equation]
for some partition D 0 of [0 , 1] and a collection of continuous functions ( q I ) I∈D 0 .
Model 2 (Continuous function).
Suppose Q is a continuous function of a and x .
We ﬁrst consider the case where the value function takes the form of equation 8.
We remark that kernel-based estimators will fail under this model assumption, as it requires the second-order derivative of the Q-function to exist.
Without loss of generality, assume q I 1 (cid:54) =
q I 2 for any two adjacent intervals I 1 , I 2 ∈ D 0 .
This guarantees that the representation in equation 8 is unique.
For any partition
D = { [0 , τ 1 ) , [ τ 1 , τ 2 ) , · · · , [ τ K , 1] } , we use J ( D ) to denote the set of change points { τ 1 , · · · , τ K } .
We impose the following conditions to establish our theories.
Assumption 1.
The number of layers L and the number of nodes each hidden layer H diverge with n , in that HL =
O ( n ρ ) , for some constant 0 < ρ < 1 / 2 .
Assumption 2.
Functions { [Equation]
[Equation]
are uniformly bounded.
Assumption 1 is mild, as both H and L are parameters we specify.
The part that HL =
O ( n ρ ) ensures that the stochastic error resulting from the parameter estimation in the MLP is negligible.
Assumption 2 ensures that the optimizer would not diverge in the (cid:96) ∞ sense.
Similar assumptions are commonly imposed in the literature to derive the convergence rates of DNN estimators (see e.g., Farrell et al., 2018).
These two assumptions guarantee the uniform consistency of the DNN estimator { (cid:98) q [Equation]
See Lemma 1 in the appendix for details.
Theorem 1 Suppose Model 1, Assumptions 1 and 2 hold.
Suppose m diverges to inﬁnity with n .
Then, there exists some constant γ 0 such that as long as 0 < γ ≤ γ 0 , the following events occur with probability approaching 1 (w.p.a.1),
[Equation]
for any policy π such that for any τ 0 ∈ J ( D ) , Pr ( π ( X ) ∈ [ τ 0 − (cid:15), τ 0 + (cid:15) ])
→ 0 as (cid:15) → 0 .
Theorem 1 establishes the properties of our method under settings where the Q ( a, x ) is piecewise function in a .
Results in (i) imply that deep jump Q-learning correctly identiﬁes the number of change points.
Results in (ii) imply that for any change point τ in D 0 , there will be an estimated change point in (cid:98) D ( (cid:96) ) that converges to τ .
Consequently, the change point locations can be consistently estimated.
Combining (i) with (ii) yields the consistency of our estimated partition (cid:98) D ( (cid:96) ) .
To ensure the consistency of the proposed value estimator, we require that the distribution of the random variable π ( X ) induced by the target policy does not have point-masses at the change point locations.
This condition is also mild.
For instance, it automatically holds when π ( X ) has a density function on [0 , 1] . [Figure]
Theorem 2 Suppose Model 2, Assumptions 1 and 2 hold.
Suppose m diverges to inﬁnity and γ decays to zero.
Then we have
[Equation]
Theorem 2 establishes the properties of our method under settings where Q is continuous in a .
Results in (i) imply that (cid:98) q ( (cid:96) )
I ( • ) can be used to uniformly approximate Q ( a, • ) for any a ∈ I .
The consistency of the value in (ii) thus follows.
E XPERIMENTS
In this section, we investigate the ﬁnite sample performance of the proposed DJQE on the synthetic and real datasets, in comparison to two kernel-based methods.
The computing infrastructure used is a virtual machine in the AWS Platform with 72 processor cores and 144GB memory.
The average runtime for our proposed method on each setting is around 5 minutes.
5. S YNTHETIC D ATA
Synthetic data are generated from the following model:
[Equation]
[Equation]
[Equation]
[Equation] A | X [Equation] ∼ Unif [0 , 1] and X (1) , X (2) , . .
, X ( p ) iid ∼ Unif [ − 1 , 1] , where X =
[ X (1) , X (2) , . . .
, X ( p ) ] .
Consider the following different scenarios:
S1 : Q ( x, a ) =
(1 + x (1) )
I ( a < 0 . 35)
+ ( x (1) − x (2) )
I (0 .
35 ≤ a < 0 .
65) + (1 − x (2) )
I ( a ≥ 0 . 65); S2 : Q ( x, a ) =
I ( a < 0 . 25) + sin(2 πx (1) )
I (0 .
25 ≤ a < 0 .
5 − 8( x (1) − 0 .
75) 2 } I (0 .
5 ≤ a < 0 .
+ 0 .
5 I ( a ≥ 0 . 75); S3 (toy) :
Q ( x, a ) = 10 max { a 2 − 0 .
25 , 0 } log( x (1) + 2) ; S4 : Q ( x, a ) = 0 .
2(8 + 4 x (1) − 2 x (2) − 2 x (3) )
− 2(1 + 0 .
5 x (1) + 0 .
5 x (2) − 2 a ) 2 .
The Q-function is a piecewise function of a under Scenarios 1 and 2, and is continuous under Scenarios 3 (toy example considered in Section 3.1) and 4.
We set the target policy to be the optimal policy that achieves the highest possible mean reward.
We list the oracle mean value under the optimal policy for each scenario in the ﬁrst column of Table 3 in the appendix.
We apply the proposed DJQE and two kernel-based methods (Kallus & Zhou, 2018; Colangelo &
Lee, 2020) to Scenario 1-4 with 20-dimensional covariates ( p = 20 ) and n ∈ { 50 , 100 , 200 , 300 } .
For the DJQE, we set m = n/ 10 and select γ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } n 0 . 4 based on ﬁve-fold crossvalidation.
We ﬁnd it extremely computationally intensive to compute the optimal bandwidth h ∗ using Kallus & Zhou (2018)’s method.
For instance, it takes more than 48 hours to implement their method on a data with n = 50 .
Thus, as suggested in Kallus & Zhou (2018), we ﬁrst compute h ∗ using data with sample size n 0 = 50 .
To accommodate data with different sample sizes n , we adjust h ∗ by setting h ∗ { n 0 /n } 0 . 2 .
To implement Colangelo & Lee (2020)’s estimator, we consider a list of bandwidths, given by h
= cσ
A n − 0 .
2 with c ∈ { 0 . 5 , 0 .
75 , 1 . 0 , 1 . 5 } and σ A is the sample standard deviation of the action.
We then manually select the best bandwidth such that the resulting value estimator achieves the smallest mean squared error.
The conditional mean value and generalized propensity score are ﬁtted via the MLP regressor with 10 hidden layer and 10 neurons in each layer.
[Table]
The average estimated value and its standard deviation over 100 replicates are illustrated in Figure 2 for different methods, with detailed values reported in Table 3 in the appendix.
It can be seen from Figure 2 that the proposed DJQE is very efﬁcient and outperforms all competing methods in almost all cases.
We note that the proposed method performs reasonably well even when the sample size is small ( n = 50 ).
In contrast, two kernel-based methods fail to accurately estimate the value even in some cases when n = 300 .
Among the two kernel-based OPE approaches, we observe that the method developed by Kallus & Zhou (2018) performs better in general.
5. R EAL D ATA : P ERSONALIZED D OSE F INDING
Warfarin is commonly used for preventing thrombosis and thromboembolism.
We use the dataset provided by the International Warfarin Pharmacogenetics (Consortium, 2009) for analysis.
We choose p = 81 baseline covariates considered in Kallus & Zhou (2018).
This yields a total of 3964 with complete records of baseline information.
The response is deﬁned as the absolute distance between the international normalized ratio (INR, a measurement of the time it takes for the blood to clot) after the treatment and the ideal value 2 . 5 , i.e, Y =
−| INR − 2 .
5 | .
We use the min-max normalization to convert the range of the dose level A into [0 , 1] .
To compare among different methods, we calibrate the dataset to generate simulated outcomes.
Speciﬁcally, we ﬁrst estimate the Q-function via a MLP regressor with 10 hidden layer andneurons in each layer using the whole dataset.
We next use the ﬁtted Q-function (cid:98) Q ( X, A ) to simulate the data.
Given a randomly sampled feature-action pair ( a j , x j ) from { ( A 1 , X 1 ) , · · · , ( A n , X n ) } , we set the reward r j to N { (cid:98) Q ( x j , a j ) , (cid:98) σ 2 } , where (cid:98) σ is the standard deviation of the ﬁtted residual { Y i − (cid:98) Q ( X i , A i ) } i .
Given the simulated data { ( x j , a j , r j ) : 1 ≤ j ≤ n } , we are interested in evaluating the optimal policy:
[Equation]
The oracle value under the optimal policy is V = − 0 .
We apply the DJQE on the calibrated Warfarin data, against two kernel-based methods.
Due to the extremely intensive computation in Kallus & Zhou (2018), we directly apply the estimated optimal bandwidth h ∗ in their real data analysis, since they used the same dataset.
Biases, standard deviations, and mean squared errors of the estimated values are reported in Table 2 over 20 replicates for sample size n = 500 under different methods.
It can be observed from Table 2 that our proposed DJQE achieves much smaller mean squared error than the two kernel-based methods, when evaluating the optimal policy.
Speciﬁcally, the DJQE yields bias as 0.259 with the standard deviation as 0.416, in contrast to the large bias as 0.662 with the standard deviation as 0.742 under Kallus & Zhou (2018)’s method and the bias as 0.442 with the large standard deviation as 1.164 under Colangelo & Lee (2020)’s method.
Therefore, our proposed DJQE for off-policy evaluation with continuous actions works better than the kernel-based methods in the calibrated real dataset.
C ONCLUSION
We proposed a brand-new OPE algorithm in continuous action domains, via an adaptive discretization technique using deep jump Q-learning.
In the current paper, we focus on settings with a single decision point.
It would be practically interesting to extend our proposal to sequential decision making problems.
A M ORE ON THE IMPLEMENTATION
We summarize our algorithm in Algorithm 1.
I. Split all n samples into L subsets as { L 1 , · · ·
L L } ; (cid:98) V ( π ) ← 0 ; II.
Initialize an even segment on the action space with m pieces: {I} = { [0 , 1 /m ) , [1 /m, 2 /m ) , . . .
, [( m − 1) /m, 1] } ; III.
For (cid:96) = 1 , · · · , L : 1.
Set the training dataset as L c(cid:96)
= { 1 , 2 , · · · , n } − L (cid:96) ; 2.
Bell (0) ← − γ ; (cid:98) D
= [0 , 1] ; τ ← Null ; R (0) ← { 0 } ; 3.
Collect cost function: For r = 1 , . .
, m : For l = 0 , . . .
, ( r − 1) : (i).
Let I =
[ l/m, r/m ) if r <
m else I =
[ l/m, 1] ; (ii).
Fit a MLP regressor: (cid:98) q
I ( · ) ← I ( i ∈ L c(cid:96) )
I ( A i ∈ I )
Y i ∼ I ( A i ∈ I )
MLP ( X i ) ; (iii).
Calculate the cost: C ( I ) ← (cid:80) i ∈
L c(cid:96)
I ( A i ∈ I ) (cid:8)(cid:98)
q
I ( X i ) − Y i (cid:9) 2 ; 4.
Apply the PELT method to get partitions: For v ∗ = 1 , . . .
, m : (i).Bell ( v ∗ ) =
min v ∈R ( v ∗ ) { Bell ( v ) + C
([ v/m, v ∗ /m ))
+ γ } ; (ii).
v 1 ← arg min v ∈R ( v ∗ ) { Bell ( v ) + C ([ v/m, v ∗ /m ))
+ γ } ; (iii).
τ ( v ∗ ) ← { v 1 , τ ( v 1 ) } ; (iv).
R ( v ∗ ) ← { v ∈ R ( v ∗ − 1) ∪ { v ∗ − 1 } :
Bell ( v ) + C ([ v/m, ( v ∗ − 1) /m ))
≤ Bell ( v ∗ − 1) } ; 5.
Construct the DR value estimator: r ← m ; l ← τ [ r ] ; While r > 0 :
(i) Let I =
[ l/m, r/m ) if r <
m else I =
[ l/m, 1] ; (cid:98) D ← (cid:98) D ∪ I ; (ii) Recall ﬁtted MLP: (cid:98) q
I ( · ) ← I ( i ∈ L c(cid:96) )
I ( A i ∈ I )
Y i ∼ I ( A i ∈ I )
MLP ( X i ) ; (iii) Fit propensity score: (cid:98) b
I ( · ) ← I ( i ∈ L c(cid:96) )
I ( A i ∈ I ) ∼ I
( A i ∈ I )
MLP ( X i ) ; (iv) r ← l ; l ← τ ( r ) ; 6.
Evaluation using testing dataset L (cid:96) : (cid:98) V ( π )+ = (cid:80)
return (cid:98) V ( π ) /n .
B A DDITIONAL E XPERIMENT R ESULTS
[Table]
C T ECHNICAL P ROOF
Throughout the proof, we use c, C, c 0 , ¯ c , etc., to denote some universal constants whose values are allowed to change from place to place.
Let O i =
{ X i , Y i } denote the data summarized from the i th observation.
For any interval I , let b ( I| x ) denote the integral (cid:82) a ∈I b ( a | x ) da .
Proofs of Theorems 1 and 2 rely on Lemmas 1 and 2.
Speciﬁcally, Lemma 1 establishes the uniform convergence rate of (cid:98) q I for any I whose length is no shorter than o ( γ ) and belongs to the set of intervals:
I ( m ) =
{ [ i 1 /m, i 2 /m ) : for some integers i
and i 2 that satisfy 0 ≤ i 1 < i 2 < m } ∪ { [ i 3 /m, 1] : for some integers i 3 that satisfy 0 ≤ i 3 < m } .
To state this lemma, we ﬁrst introduce some notations.
For any such interval I , deﬁne the function q I ( x )
=
E ( Y | A ∈ I , X = x ) .
It is immediate to see that the deﬁnition of q I here is consistent with the one deﬁned in equation 8 for any I ⊆ D 0 .
Lemma
The following holds when either the conditions in Theorem 1 or 2 hold: [Equation] , | [Equation]
|
[Equation]
X
[Equation]
for any positive constant c > 0 .
Lemma 2 When either the conditions in Theorem 1 or 2 hold, min I∈ (cid:98) D ( (cid:96) )
|I| ≥ Cγ w.p.a.1 for some constant C > 0 .
We ﬁrst present the proofs for these two lemmas.
Next we present the proofs for Theorems 1 and 2.
C. P ROOF OF L EMMA
For any sufﬁciently small constant (cid:15) > 0 , it sufﬁces to show
[Equation]
w.p.a.1, for some constant C > 1 whose value will be speciﬁed later.
The proof is divided into two parts.
In the ﬁrst part, we show there exist some constants { θ ∗I } I such that
[Equation]
In the second part, we show
[Equation]
The proof is hence completed.
Part 1.
Under the step model assumption, for any I 0 ∈ D 0 , q I 0 ( • ) is continuous.
Since the support
X is compact, q I 0 ( • ) is uniformly continuous.
Similarly, we can show b is uniformly continuous as well.
For any I ∈ I m , we have
[Equation]
It follows from the uniform continuity of q I 0 , b and the positivity assumption that q I ( • ) is continuous for any I ∈ I ( m ) .
Similarly, under the model assumption in Theorem 2, we can show q I ( • ) is continuous for any
I ∈ I ( m ) as well.
Consequently, when either the conditions in Theorem 1 or 2 hold, we obtain that q I ( • ) is continuous for any I ∈ I ( m ) .
By Stone-Weierstrass theorem, there exists a multivariate polynomial function q ∗ I such that the absolute value of the residual q I − q ∗I is uniformly bounded by (cid:15)/ 2 .
By Theorem 1 of Yarotsky (2017), there exists a feedforward neural network with a bounded number of hidden units that uniformly approximates q ∗I , with the approximation error uniformly bounded by (cid:15)/ 2 in absolute value.
By
Lemma 1 of Farrell et al.
(2018), such a feedforward network can be embedded into an MLP with a bounded number of hidden units.
Since we allow H and L to diverge, such an MLP can be further embedded into an MLP with L layers and the widths of all layers being proportional to H .
This yields equation 9.
The proof for Part 1 is thus completed.
Part 2.
We aim to show equation 10 holds.
Under the boundedness assumption on Y , { q I ( • ) } I∈ I ( m ) are uniformly bounded.
We ﬁrst observe that MLP ( • , θ ∗I ) is a bounded function, by equation 9 and the fact that q I ( • ) is bounded.
Next [Equation] it follows from equation 9 that
[Equation]
By deﬁnition
[Equation]
Suppose we can show
[Equation]
where the set Θ ∗ consists of all θ I such that sup x | MLP ( x ; θ I ) | ≤ M for some sufﬁciently large constant M such that both θ ∗I and (cid:98) θ ( (cid:96) )
I satisfy this constraint.
It thus follows from equation 11 and equation 12 that
[Equation]
w.p.a.1.
Similar to equation 11, we have
[Equation]
It follows that
[Equation]
Under the positivity assumption, sup x b ( I| x ) /
Pr ( A ∈ I ) ≥ 2( c − 1) − 1 for some constant c > 1 .
This yields equation [Equation]
To complete the proof [Equation] it remains to show equation 13 holds.
Using similar arguments in Section A.2.2 of Farrell et al.
, for any I , we have with probability at least 1 − 2 exp( − ¯ γ ) that
[Equation]
for some constant ¯ c > 0 , where σ 2 ( I , θ ) corresponds to the variance of
[Equation]
Under the positivity assumption, we have σ 2 ( I , θ ) ≤ O (1) |I| − 1 where O (1) denotes some positive constant.
Consequently, for any I whose length is greater than cγ − 1 , we have with probability at least 1 − 2 exp( − ¯ γ ) that
[Equation]
for some constant ¯ c 0 > 0 .
Note that the number of elements in I ( m ) is upper bounded by ( m + 1) 2 .
Since m is proportional to n , by setting ¯ γ = ¯ c ∗ log n for some constant ¯ c ∗ > 0 , we have 1 − 2( m + 1) 2 exp( − ¯ γ ) → 1 .
Consequently, it follows from Bonferroni’s inequality that the following event occurs w.p [Equation] a.1 for any
I ∈ I ( m )
such that |I| ≥ cγ ,
[Equation]
where O (1) denotes some positive constant.
Under the given conditions, the RHS is o (1) .
The proof is hence completed.
C. P ROOF OF L EMMA
Consider a given interval I ∈ (cid:98) D ( (cid:96) ) .
We can ﬁnd some interval I (cid:48) ∈
I ( m ) ∩ (cid:98) D ( (cid:96) )
that is adjacent to
I .
Consequently
[Equation]
the interval I ∪ I (cid:48) belongs to I ( m ) as well.
It follows that
[Equation]
By deﬁnition, (cid:98) q
( (cid:96) )
I∪I (cid:48) minimizes the loss (cid:80)
[Equation]
i I ( A i ∈
I ∪ I (cid:48) )
{ Y i − (cid:98) q
( (cid:96) )
I∪I (cid:48)
( X i ) } 2 .
It follows that (cid:88) i ∈ L (cid:96) I
[Equation]
Combining this together with equation 15 yields that
[Equation]
As both Y and (cid:98) q ( (cid:96) )
I are uniformly bounded, we have γ ≤ c | L (cid:96) | − 1 (cid:80) i ∈ L (cid:96) I
( A i ∈ I ) for some constant c > 0 .
By Bernstein’s inequality, we have for any t > 0 such that
[Equation]
(17) with probability 1 − exp( − nt 2 / ¯ c
( |I| + t )) for some constant ¯ c > 0 .
Similar to equation 14, by setting t I = ¯ c 0 max( (cid:112) |I| n − 1 log n, n − 1 log n ) for some constant ¯ c 0 > 0 , we obtain w.p.a.1 that
[Equation]
for any I ∈ I ( m ) .
Under the condition that b is continuous and A × X is compact, b is bounded.
Consequently, the probability Pr ( A ∈ I ) is proportional to the length of I .
In view of equation 17, for any I ∈ (cid:98) D ( (cid:96) ) , it shall satisfy
[Equation]
for some constant ¯ c 1 > 0 , w.p.a.1, where the last inequality follows from Cauchy-Schwarz inequality.
As γ (cid:29) n − 1 log n , we obtain |I| ≥ γ/ ¯ c 2 for any I ∈ (cid:98) D ( (cid:96) ) and some constant ¯ c 2 > 0 , w.p.a.1.
The proof is hence completed.
C. P ROOF OF T HEOREM
We begin with an outline of the proof.
The proof is divide into four steps.
In the ﬁrst step, we show
[Equation]
In the second step, we show
[Equation]
19) where δ min = min I∈D 0 |I| / 3 .
By the deﬁnition of δ min , this implies that
[Equation]
Combining equation 21 together with equation 18 proves (i) in Theorem 1.
This proves (ii) in
Theorem 1.
In the third step, we show
[Equation]
[Equation]
In the last step, we show (iii) holds.
The proof is thus completed.
We next detail the proof for each of the step.
Step 1.
Assume |D 0 | > 1 .
Otherwise, equation 18 automatically holds.
Consider the partition
D = { [0 , 1] } which consists of a single interval and a zero q-function q [0 , 1] ( x ) = 0 for any x .
By deﬁnition, we have
[Equation]
Since Y is uniformly bounded and γ = o (1) , the right-hand-side (RHS) is O ( n ) .
Consequently, we obtain
[Equation]
[Equation]
for some constant c 0 > 0 .
Notice that
[Equation]
We next show η 2 , η 3 = o p (1) .
Consider η 2 ﬁrst.
Under Lemma 2, we have w [Equation] p
[Equation]
a.1 that
[Equation]
We decompose the RHS by
[Equation]
The ﬁrst line is o p ( n ) , by Lemma 1.
Using similar arguments in equation 14, we can show the second term is upper bounded by
[Equation]
for some constant ¯ c > 0 , w.p.a.1.
In view of equation
[Equation]
the above expression can be further bounded by
[Equation]
where O (1) denotes some positive constant and the second inequality follows from the CauchySchwarz inequality.
This yields that η 2 = o p ( n ) .
Using similar arguments, we can show that η 3 = o p ( n ) .
It follows that
[Equation]
Notice that
[Equation]
Using similar arguments in bounding η 2 and η [Equation] we can show η 6 =
o p ( n ) and that
[Equation]
The ﬁrst term on the RHS is greater than c 0 n
[Equation]
| Q ( X, a ) − q I ( X )
| 2 da for some constant c 0 > 0 .
It follows that
[Equation]
Note that under the step model assumption, η 4 can be rewritten as (cid:80) I∈D 0 (cid:80)
i ∈ L (cid:96)
I
( A i ∈ I ) { Y i − q I ( X i ) } 2 .
This together with equation 24, yields that
[Equation]
For any integer k such that 1 ≤ k ≤ |D 0 |− 1 , let τ ∗ 0 ,k be the change point location that satisﬁes τ ∗ 0 ,
k
= i/m for some integer i and that | τ 0 ,k − τ ∗ 0 ,
k | < m − 1 .
Denoted by D ∗ the oracle partition formed by the change point locations { τ ∗ 0 ,k } |D 0 |− 1 k = .
Set τ ∗ 0 , 0 = 0 , τ ∗ 0 , |D 0 | = 1 and q ∗ [ τ ∗ 0 ,k − 1 ,τ ∗ 0 ,k [Equation]
= q [ [Equation] [Equation] for 1 ≤ k ≤ |D 0 | − 1 and q ∗ [ τ ∗ 0 ,K − 1 , 1] =
q [ τ 0 ,K − 1 , 1] .
Let ∆ k
= [ τ ∗ 0 ,k − 1 , τ ∗ 0 ,k ) ∩ [ τ 0 ,k − 1 , τ 0 ,k ) c for 1 ≤ k ≤ |D 0 | − 1 and ∆ |D 0 | =
[ τ ∗ 0 , |D 0 |− 1 , 1] ∩ [ τ 0 , |D 0 |− 1 , 1] c .
The length of each interval ∆ k is at most m − 1 .
It follows that
[Equation]
Using similar arguments in bounding the RHS of equation 16 in the proof of Lemma 2, we can show the RHS of equation 25 is o p ( n ) , as m diverges to inﬁnity.
Combining this together with equationyields
[Equation]
E
[Equation]
By deﬁnition, we have (cid:88)
[Equation]
Since |D ∗ | =
|D 0 | , it follows from equation 26 that
[Equation]
[Equation]
Note that the left-hand-side (LHR) is non-negative.
As γ > 0 , we obtain that | (cid:98) D ( (cid:96) ) | ≤ |D 0 | , w.p.a.1.
This completes the proof for the ﬁrst step.
Step 2.
It follows from equation 27 that
[Equation]
and hence (cid:88)
[Equation]
(28) w.p.a.1.
We aim to show equation 19 holds under the event deﬁned in equation 28.
Otherwise, there exists some τ 0 ∈ J ( P 0 )
such that | ˆ τ − τ 0 | ≥ δ min , for all ˆ τ ∈ J ( (cid:98) P ) .
Under the event deﬁned in equation 28, we obtain that
[Equation]
29) w.p.a.1.
On the other hand, since Q ( X, a ) is a constant function on [ τ 0 − δ min , τ 0 ) or [ τ 0 , τ 0 + δ min ) , we have
[Equation]
where κ
[Equation]
≡ min
[Equation]
1 and I 2 are adjacent
[Equation]
This apparently violates equation 29 where γ ≤ δ min κ 0 / (4 |D 0 − 4 | ) .
equation 19 thus holds w.p.a.1.
Step 3:
Combining the results obtained in the ﬁrst two steps yields that |D ( (cid:96) ) | =
|D 0 | , w.p.a.1.
It follows from equation 27 that
[Equation]
Using similar arguments in Step 2, we can show max τ ∈ J ( D 0 )
[Equation]
(cid:98) τ ∈ J [Equation] D
[Equation]
) ) | (cid:98) τ − τ | < (cid:15) w.p.a.1, for any sufﬁciently small (cid:15) > 0 .
The proof is hence completed.
Step 4: We begin with some notations.
For any π , we deﬁne a random policy π (cid:98) D ( (cid:96) ) according to the partition (cid:98) D ( (cid:96) ) as follows:
[Equation]
Note that (cid:82) 1 0 π (cid:98) D
( (cid:96) )
( a | x ) da = (cid:80) I⊆ (cid:98) D ( (cid:96) )
I { π ( x ) ∈ I} = 1 for any x .
Consequently, π (cid:98) D ( (cid:96) ) is a “valid” random policy.
The proposed value estimator (cid:98) V ( (cid:96) ) is doubly-robust to V ( π (cid:98) D ( (cid:96) ) ) .
By Lemma 1, the estimated Qfunction is consistent.
Consequently, (cid:98) V ( (cid:96) ) is consistent to V ( π (cid:98) D ( (cid:96) ) ) .
Since the propose estimator is a weighted average of (cid:98) V ( (cid:96) ) , it sufﬁces to show V ( π (cid:98) D ( (cid:96) ) ) is consistent to V ( π ) .
Note that
[Equation]
Similarly, we can show
[Equation]
For each I 0 ∈ D 0 , there exists an interval
I ∈ (cid:98) D ( (cid:96) )
such that |I ∪ I 0 | / |I| → 1 .
We use (cid:98) I ( (cid:96) )to denote this interval.
Under the given conditions, we have
[Equation]
In view of equation 30, the value difference | V ( π (cid:98) D ( (cid:96) ) )
− V ( π ) | can be upper bounded by
[Equation]
The ﬁrst two terms decay to zero under the given conditions on π ( X ) .
The proof is hence completed.
C. P ROOF OF T HEOREM
We need the following lemma to prove Theorem 2.
Lemma 3 Assume conditions in Theorem 2 hold.
Then for any interval I ∈ I ( m ) with |I| (cid:29) γ and any interval I (cid:48) ∈ (cid:98) D ( (cid:96) ) with I ⊆
I (cid:48) , we have w.p.a.1 that
[Equation]
where the little- o term is uniform in I and I (cid:48) .
The rest of the proof is divided into three steps.
In the ﬁrst step, we show Assertion (i) in Theoremholds.
In the second step, we show Assertion (ii) in Theorem 2 holds.
Finally, we present the proof for Lemma 3.
Step 1.
Consider a sequence { d n } n such that d n → 0 and d n (cid:29) γ .
We aim to show
[Equation]
By Lemma 1, it sufﬁces to show
[Equation]
Suppose |I (cid:48) | ≥ d n .
Then according to Lemma 3, we can ﬁnd some I such that a ∈
I ⊆
I (cid:48) , |I| → 0 , E | q I ( X ) − q I (cid:48)
( X ) |
= o (1) .
Since |I| → 0 and a ∈ I , it follows from the uniform continuity of the Q-function that | q I ( X ) − Q ( X, a ) | → 0 .
The assertion thus follows.
Next, suppose |I (cid:48) | < d n .
Then |I (cid:48)
| → 0 as well.
It follows from the uniform continuity of the
Q-function that inf a ∈I (cid:48)
| q I (cid:48)
( X ) − Q
( X, a ) | → 0 .
The assertion thus follows.
This completes the proof for the ﬁrst step.
Step 2.
Using similar arguments in Step 4 of the proof of Theorem 1, it sufﬁces to show V ( π (cid:98) D ( (cid:96) ) ) =
V ( π ) + o p (1) .
By deﬁnition,
[Equation]
It follows that
[Equation]
In Step 1 of the proof, we have shown that inf a ∈I , I∈ (cid:98) D ( [Equation]
E | Q ( X, a ) − q I ( X ) |
= o (1) .
It follows that inf a,a [Equation] I∈ [Equation]
( (cid:96) ) E | Q ( X, a ) − Q ( a (cid:48) , X ) | 
 = o
(1) and hence inf a [Equation] a
[Equation] D ( (cid:96) [Equation] E | Q ( a (cid:48) , X ) −
Q ( a (cid:48)(cid:48) , X ) |
= o (1) , by Cauchy-Schwarz inequality.
This completes the proof for the second step.
Step 3.
For a given interval I (cid:48) ∈ (cid:98) D ( (cid:96) ) , the set of intervals I considered in Lemma 3 can be classiﬁed into the following three categories.
Category 1: I = I (cid:48) .
It is immediate to see that q
I
=
q
I (cid:48) and the assertion automatically holds.
Category 2: There exists another interval I ∗ ∈
I ( m ) that satisﬁes I (cid:48)
=
I ∗ ∪ I .
Notice that the partition (cid:98) D ( (cid:96) ) ∗
=
(cid:98) D ( (cid:96) ) ∪ {I ∗ } ∪
I − {I (cid:48) } corresponds to another partition.
By deﬁnition, we have
[Equation]
and hence |
[Equation]
|
[Equation]
It follows from the deﬁnition of (cid:98) q I ∗ that
[Equation]
Therefore
[Equation]
we obtain
[Equation]
Category 3: There exist two intervals I ∗ , I ∗∗ ∈ I ( m ) that satisfy I (cid:48)
=
I ∗ ∪
I ∪ I ∗∗ .
Using similar arguments in proving equation 31, we can show that
[Equation]
Hence
[Equation]
regardless of whether I belongs to Category
[Equation]
or it belongs to Category [Equation] we have
[Equation]
Using similar arguments in equation
[Equation]
we can show w.p
[Equation]
a.1 that
[Equation]
where the little- o terms are uniform in I and I (cid:48) .
Combining these together with equation 32 yields
[Equation]
γ | [Equation] |
[Equation]
, for any I and I (cid:48) , w
[Equation]
p.a.1.
Note that q I satisﬁes E
[ I ( A ∈ I ) { Y − q I ( X ) }| X ] = 0 .
We have
[Equation]
Consider the ﬁrst term on the RHS.
Note that
[Equation]
+
[Equation]
[Equation]
By Cauchy-Schwarz inequality, the last term on the RHS can be lower bounded by
[Equation]
It follows that
[Equation]
and hence
[Equation]
By Lemma 1 and the positivity assumption
[Equation]
the RHS is o p ( |I| ) .
Note that the little-
o terms are uniform in I and I (cid:48) .
As |I| (cid:29) γ , we obtain that
[Equation]
uniformly for any I and I (cid:48) , or equivalently,
[Equation]
By the positivity assumption, we have
[Equation]
uniformly for any I and I (cid:48) , or equivalently,
[Equation]
This yields that
[Equation]
uniformly for any I and I (cid:48) .
The proof is hence completed.