D EEP
J UMP
Q-E VALUATION
FOR
O FFLINE
P OLICY
E VALUATION IN C ONTINUOUS A CTION S PACE
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider off-policy evaluation (OPE) in continuous action domains, such as dynamic pricing and personalized dose ﬁnding.
In OPE, one aims to learn the value under a new policy using historical data generated by a different behavior policy.
Most existing works on OPE focus on discrete action domains.
To handle continuous action space, we develop a brand-new deep jump Q-evaluation method for OPE.
The key ingredient of our method lies in adaptively discretizing the action space using deep jump Q-learning.
This allows us to apply existing OPE methods in discrete domains to handle continuous actions.
Our method is further justiﬁed by theoretical results, synthetic and real datasets.
I NTRODUCTION
Individualization proposes to leverage omni-channel data to meet individual needs.
Individualized decision making plays a vital role in a wide variety of applications.
Examples include customized pricing strategy in economics (Qiang & Bayati, 2016; Turvey, 2017), individualized treatment regime in medicine (Chakraborty, 2013; Collins & Varmus, 2015), personalized recommendation system in marketing (McInerney et al., 2018; Fong et al., 2018), etc.
Prior to adopting any decision rule in practice, it is crucial to know the impact of implementing such a policy.
In many applications, it is risky to run a policy online to estimate its value (see, e.g., Li et al., 2011).
Off-policy evaluation (OPE) thus attracts a lot of attention by learning the policy value ofﬂine using logged historical data.
Despite the popularity of developing OPE methods with a ﬁnite set of actions (see e.g., Dud ´ ık et al., 2011; 2014; Swaminathan et al., 2017; Wang et al., 2017), less attention has been paid to continuous action domains, such as dynamic pricing (den Boer & Keskin, 2020) and personalized dose ﬁnding (Chen et al., 2016).
Recently, a few OPE methods have been proposed to handle continuous actions (Kallus & Zhou, 2018;
Sondhi et al., 2020; Colangelo & Lee, 2020).
All these methods rely on the use of a kernel function to extend the inverse probability weighting (IPW) or doubly robust (DR) approaches developed in discrete action domains.
They suffer from three limitations.
First, the validity of these methods requires the conditional mean of the reward given the feature-action pair to be a smooth function over the action space.
This assumption could be violated in applications such as dynamic pricing, where the expected demand for a product has jump discontinuities as a function of the charged price (den Boer & Keskin, 2020).
Second, the value estimator could be sensitive to the choice of the bandwidth parameter in the kernel function.
It remains challenging to select this hyperparameter.
Kallus & Zhou (2018) proposed to tune this parameter by minimizing the mean squared error of the resulting value estimator.
However, their method is extremely computationally intensive in moderate or high-dimensional feature space; see Section 5 for details.
Third, these kernel-based methods typically use a single bandwidth parameter.
This is sub-optimal in cases where the second-order derivative of the conditional mean function has an abrupt change in the action space; see the toy example in Section 3.1 for details.
To address these limitations, we develop a deep jump Q-evaluation (DJQE) method by integrating multi-scale change point detection (see e.g., Fryzlewicz, 2014), deep learning (LeCun et al., 2015) and OPE in discrete action domains.
The key ingredient of our method lies in adaptively discretizing the action space using deep jump Q-learning.
This allows us to apply IPW or DR methods to handle continuous actions.
It is worth mentioning that our method does not require kernel bandwidth selection.
Theoretically, we show it allows the conditional mean to be either a continuous or piecewise function of the action (Theorems 1 and 2) and converges faster than kernel-based OPE (Theorem 3).
Empirically, we show it outperforms state-of-the-art OPE methods in synthetic and real datasets.
P RELIMINARIES
We ﬁrst formulate the OPE problem.
We next discuss the kernel-based OPE methods and multi-scale change point detection, since our proposal is closely related to them.
2. O FF - POLICY EVALUATION
The observed datasets can be summarized into { ( X i , A
i , Y i ) } 1 ≤ i ≤ n where O i =
( X i , A
i , Y i ) denotes the feature-action-reward triplet for the i th subject and n denotes the total sample size.
We assume these data triplets are independent copies of some population variables ( X, A, Y ) .
Let
X and A denote the feature and action space, respectively.
We focus on the setting where A is one-dimensional, as in dynamic pricing and personalized dose ﬁnding.
A deterministic policy π : X → A determines the action to be assigned given the observed feature.
We use b to denote the behavior policy that generates the observed data.
Speciﬁcally, b ( •| x ) denotes the probability density or mass function of A given X = x , depending on whether A is continuous or not.
Deﬁne the expected reward function conditional on the feature-action pair as
[Equation]
We refer to this function as the Q-function, to be consistent with the literature on developing individualized treatment regime (Murphy, 2003).
As standard in the OPE and the causal inference literature (see e.g., Chen et al., 2016), we assume the stable unit treatment value assumption (SUTVA), no unmeasured confounders assumption, and the positivity assumption are satisﬁed.
These assumptions guarantee that a policy’s value is estimable from the observed data.
Speciﬁcally, for a given target policy π , its value can be represented by
[Equation]
The goal of the OPE is to learn the value V ( π ) based on the observed data.
2. K ERNEL -B ASED OPE
For discrete action, Zhang et al.
(2012) and Dud´ık et al.
(2011) proposed a DR estimator of V ( π ) by
[Equation]
where I denotes the indicator function, (cid:98) Q and (cid:98) b denote some estimators for the Q-function and the behavior policy.
The second term (cid:98) b − 1 ( A i | X i )
I
( A i =
π ( X i ))
{ Y i − (cid:98) Q ( X i , π ( X i )) } inside the bracket corresponds to an augmentation term.
Its expectation equals zero when (cid:98) Q = Q .
The purpose of adding this term is to offer additional protection against potential model misspeciﬁcation of the Q-function.
Such an estimator is doubly-robust in the sense that its consistency relies on either (cid:98) Q or (cid:98) b to be correctly speciﬁed.
By setting (cid:98) Q = 0 , equation 1 is reduced to the IPW estimator.
In continuous action domains, the indicator function I ( A i =
π ( X i )) equals zero almost surely.
Consequently, naively applying equation 1 yields the plug-in estimator (cid:80) ni
=1 (cid:98) Q
( X i , π ( X i ))
/n .
To address this concern, the kernel-based OPE proposed to replace the indicator function in equationwith a kernel function K { ( A i − π ( X i )) /h } with some bandwidth parameter h , i.e.,
[Equation]
The bandwidth h represents a trade-off.
The variance of the resulting value estimator decays with h .
Yet, its bias increases with h .
More speciﬁcally, it follows from Theorem 1 of Kallus & Zhou (2018) that the leading term of the bias is equal to
[Equation]
To ensure the term in 2 decays to zero as h goes to 0, it requires the expected second derivative of the Q-function to exist, and thus Q ( x, a ) needs to be a smooth function of a .
However, as commented in the introduction, this assumption could be violated in applications such as dynamic pricing.
[Table]
[Figure]
2. M ULTI -S CALE C HANGE P OINT D ETECTION
The change point analysis considers an ordered sequence of data, Y 1: n = { Y 1 , · · · , Y n } , with unknown change point locations, τ = { τ 1 , · · · , τ K } for some unknown integer K .
Here, τ i is an integer between 1 and n − 1 inclusive, and satisﬁes τ i < τ j for i < j .
These change points split the data into K +1 segment.
Within each segments, the expected response is a constant function (see the left panel of Figure 1 for details).
A number of methods have been proposed on estimating change points (see for example, Boysen et al., 2009; Killick et al., 2012; Frick et al., 2014; Fryzlewicz, 2014, and the references therein), by minimizing a penalized objective function:
[Equation]
where C is a cost function that measures the goodness-of-the-ﬁt of the constant function within each segment and γK penalizes the number of change points with some regularization parameter γ .
We remark that all the above cited works focused on models without features.
Our proposal goes beyond these works in that we consider models with features and use deep neural networks (DNN) to capture the complex relationship between the response and features.
D EEP J UMP Q-E VALUATION
In section 3.1, we use a toy example to demonstrate the limitation of kernel-based methods.
We present the main idea of our algorithm in Section 3.2.
Details are given in Section 3.3.
3. T OY E XAMPLE
As discussed in the introduction, existing kernel-based OPE methods use a single bandwidth to construct the value estimator.
Ideally, the bandwidth h in the kernel K { ( A i − π ( X i ))
/h } shall vary with π ( X i ) to improve the accuracy of the value estimator.
To elaborate this, consider the Q-function
Q ( x, a ) =
10 max { a 2 − 0 .
25 , 0 } log( x + 2) for any x, a ∈ [0 , 1] .
By deﬁnition, the Q-function is smooth over the entire feature-action space.
However, it has different “patterns” when the action belongs to different intervals.
Speciﬁcally, for a ∈ [0 , 0 . 5] , Q ( x, a ) is constant as a function of a .
For a ∈ (0 . 5 , 1] , Q ( x, a ) depends quadratically in a .
See the middle panel of Figure 1 for details.
Consider the target policy π ( x ) = x .
We decompose the value V ( π ) into V (1) ( π ) + V (2) ( π ) where
V (1) ( π ) =
E Q ( X, π ( X ))
I ( π ( X ) ≤ 0 . 5) and V (2) ( π ) =
E Q ( X, π ( X ))
I ( π ( X ) > 0 . 5) .
Similarly, denote the corresponding kernel-based value estimators by
(cid:98) V (1)
h ( π ) = 1 n n (cid:88) i = ψ h ( O i , π, (cid:98) Q, (cid:98) b ) I
( π ( X i ) ≤ 0 . 5) and (cid:98) V (2) h ( π ) = 1 n n (cid:88) i = ψ h ( O i , π, (cid:98) Q, (cid:98) b ) I
( π ( X i ) > 0 . 5) . Since Q ( x, a ) is a constant function of a ∈ [0 , 0 . 5] , its second-order derivative ∂ 2 Q ( x, a ) /∂aequals zero.
In view of 2, when π ( x ) ≤ 0 . 5 , the bias of (cid:98) V (1) h
( π ) will be small even with a sufﬁciently large h .
As such, a large h is preferred to reduce the variance of (cid:98) V (1) h ( π ) .
When π ( x ) > 0 . 5 , a small h is preferred to reduce the bias of (cid:98) V (2) h ( π ) .
See Table 1 for details where we report the bias and standard deviation of (cid:98) V (1)
h ( π ) and (cid:98) V (2) h ( π ) with two different bandwidths.
Due to the use of a single bandwidth, the kernel-based estimator suffers from either a large bias or a large variance.
To overcome this limitation, we propose to adaptively discretize the action space into a union of disjoint intervals such that within each interval I , the Q-function { Q ( x, a ) : a ∈ I} can be well-approximated by some function Q I ( x ) that is constant in a ∈ I .
Based on the discretization, one can apply IPW or DR to evaluate the value.
The advantage of adaptive discretization is illustrated in the right panel of Figure 1.
When a ≤ 0 .
5 , the Q-function is constant in a .
It is likely that our procedure will not further split the interval [0 , 0 . 5] .
Consequently, the corresponding DR estimator for V (1) ( π ) will not suffer from large variance.
When a > 0 . 5 , our procedure will split (0 . 5 , 1] into a series of sub-intervals, approximating Q by a step function.
This guarantees the resulting DR estimator for V (2) ( π ) will not suffer from large bias.
Consequently, the proposed value estimator achieves a smaller mean squared error than kernel-based estimators.
See Table 1 for details.
3. T HE M AIN I DEA
For simplicity, we set the action space A =
From now on, we focus on a subset of intervals in [0 , 1] .
By interval we always refer to those of the form [ c, d ) for some 0 ≤ c < d < 1 , or [ c, 1] for some 0 ≤ c < 1 , denoted as I .
A discretization D for A is deﬁned as a collection of mutually disjoint intervals that covers A .
Let |D| denote the number of intervals in D and |I| denote the length of the interval
I .
We aim to identify an “optimal” discretization (cid:98) D such that for each interval I ∈ (cid:98) D , Q ( x, a ) is approximately a constant function of a ∈ I .
The number of intervals in (cid:98) D represents a trade-off.
If | (cid:98) D| is too large, then (cid:98) D will contain many short intervals, the resulting IPW or DR estimator might suffer from large variance.
Yet, a smaller value of | (cid:98) D| might result in a large bias.
Our proposed method adaptively determines (cid:98) D and its size | (cid:98) D| as illustrated below.
To begin with, we cut the entire action space
A into m initial intervals: [0 , 1 /m ) , [1 /m, 2 /m ) , . . .
, [( m − 1) /m, 1] .
The number m shall be sufﬁciently large such that the Q-function can be wellapproximated by a piecewise function on these intervals.
In practice, we recommend to set the initial number of intervals m to be proportional to the sample size n .
Note the set of these initial intervals is not the ﬁnal partition (cid:98) D that we recommend, but only serve as the initial candidate intervals.
We next adaptively combine some of these initial intervals to form the ﬁnal partition (cid:98) D .
As shown in our numerical studies (see Table 5 in Appendix B for more details), the size of the ﬁnal partition | (cid:98) D| is usually much less than m .
More speciﬁcally, denote by B ( m ) as the set of discretizations D such that the end-points of each interval
I ∈ D lie on the grid { j/m : j = 0 , 1 , · · · , m } .
We associate to each partition D ∈ B ( m ) a collection of functions { Q I } I∈D .
These functions depend only on features, not the action.
They are used to produce a piecewise approximation of the Q-function such that Q ( a, • )
≈ (cid:80)
I I ( a ∈
I ) Q
I ( • ) .
We model these Q I using deep neural networks, to capture the complex dependence between the response and features.
When the Q-function is well-approximated, we expect the least square loss (cid:80) I∈D {+(cid:104)(cid:80) ni =
 I ( A i ∈ I ) (cid:8) Y+} i − Q
I
( X i ) (cid:9) 2 (cid:105) , will be small.
Consequently, (cid:98) D can be estimated by solving ( (cid:98) D , { (cid:98) q
I : I ∈ (cid:98) D} ) =
[Equation]
for some regularization parameter γ and DNN class Q .
Here, the penalty term γ |D| in equationcontrols the total number of intervals in (cid:98) D , as in multi-scale change point detection.
A large γ results in few intervals in (cid:98) D and a potential large bias of the value estimator, whereas a small γ could procedure a large number of intervals in (cid:98) D , leading to a noisy value estimator.
In practice, we use cross-validation to select the regularization parameter γ that minimizes the mean square error of the ﬁtted Q-function.
We refer to this step as deep jump Q-learning.
Details of this step are given in the next section.
Given (cid:98) D , one can apply IPW or DR (see equation 1) to derive the value estimates.
To further reduce the bias of the value estimator, we employ a data splitting and cross-ﬁtting strategy, which is commonly used in statistics (Romano & DiCiccio, 2019).
That is, we use different subsets of data samples to learn the discretization (cid:98) D and to construct the value estimator.
A pseudocode summarizing our algorithm is given in Algorithm 1 in Appendix A.
We present the details below.
3. T HE C OMPLETE A LGORITHM
We present the details for DJQE in this section.
It consists of three steps: data splitting, deep jump
Q-learning, and cross-ﬁtting.
Step 1: Data Splitting: We divide all n samples into L subsets of equal size, where L (cid:96) denotes the indices of samples in the (cid:96) th subset for (cid:96) = 1 , · · · , L .
Let L c(cid:96)
= { 1 , 2 , · · · , n } − L (cid:96) as the complement of L (cid:96) .
Step 2: Deep Jump Q-Learning:
For each (cid:96) = 1 , · · · , L , we propose to apply deep jump Qlearning to compute a discretization (cid:98) D ( (cid:96) ) and { (cid:98) q
( (cid:96) )
I : I ∈ (cid:98) D ( (cid:96) ) } by solving a version of equationusing the data subset in L c(cid:96) only.
We next present the computational details for solving this optimization.
Our approach is motivated by the PELT method (Killick et al., 2012) in multi-scale change point detection.
Speciﬁcally, for any interval I , deﬁne (cid:98) q
( (cid:96) )
I as the minimizer of
[Equation]
where | L c(cid:96) | denotes the number of samples in L c(cid:96) .
Deﬁne the cost function C ( (cid:96) )
( I ) as the minimum value of the objective function 4, i.e, C ( (cid:96) ) ( I ) = [Equation]
c(cid:96) | (cid:80)
i ∈
L c(cid:96)
I ( A i ∈ I ) (cid:8)(cid:98) q ( (cid:96) ) I ( X i ) − Y i (cid:9) 2 . In our implementation, we set Q as the class of multilayer perceptrons (MLPs, see Figure 3 in Appendix
A for an illustration) with L hidden layers and H nodes each hidden layer.
The above optimization can be solved via the MLP regressor implementation of Pedregosa et al.
Computation of (cid:98) D ( (cid:96) ) relies on dynamic programming (Friedrich et al., 2008).
For any integer 1 ≤ v ∗ < m , denote by B ( m, v ∗ ) the set consisting of all possible discretizations D v ∗ of [0 , v ∗ /m ) .
Set
B ( m, m ) =
B ( m ) , we deﬁne the Bellman function as
[Equation]
Our algorithm recursively updates the Bellman function based on the following formula,
[Equation]
where R v ∗ is the candidate change points list updated by
[Equation]
during each iteration with R 0 = { 0 } .
The constraint listed in 6 is important as it facilitates the computation by discarding change points not relevant to obtain the ﬁnal discretization, leading to a linear computational cost (Killick et al., 2012).
To solve equation 5, we search the optimal change point location v that minimizes Bell ( v ∗ ) .
This requires to apply the MLP regressor to learn (cid:98) q ( (cid:96) )
[ v/m,v ∗ /m ) and C ( (cid:96) )
([ v/m, v ∗ /m )) for each v ∈ R v ∗ .
Let v 1 be the corresponding minimizer.
We then deﬁne the change points list τ ( v ∗ )
= { v 1 , τ ( v 1 ) } .
This procedure is iterated to compute Bell ( v ∗ ) and τ ( v ∗ ) for v ∗ = 1 , . . .
, m .
The optimal partition (cid:98) D ( (cid:96) ) is determined by the values stored in τ (see Algorithm 1 in Appendix A for details).
Step 3.
Cross-Fitting:
For each interval in the estimated optimal partition (cid:98) D ( (cid:96) ) , we estimate the generalized propensity score function
Pr ( A ∈ I| X = x ) via the MLP regressor using the training dataset L c (cid:96) .
Let (cid:98) b
( (cid:96) )
( I| x ) denote the resulting estimate.
The ﬁnal estimated value for V ( π ) is constructed via cross-ﬁtting, given by,
[Equation]
Note the samples used to construct (cid:98) V inside bracket are independent from those to estimate (
cid:98) q
( (cid:96) )
I , (cid:98) b ( (cid:96) )
I and (cid:98) D ( (cid:96) ) .
This helps remove the bias induced by overﬁtting in the estimation of (cid:98) q ( (cid:96) )
I , (cid:98) b ( (cid:96) )
I and (cid:98) D ( (cid:96) ) .
T HEORY
We investigate the theoretical properties of the proposed estimator.
For simplicity, let the support
X = [0 , 1] p .
We will show our estimator is consistent when the Q-function is either a piecewise function or a continuous function of a .
Speciﬁcally, consider the following two model assumptions.
Model 1 (Piece-wise constant function).
Suppose
[Equation]
for some partition D 0 of [0 , 1] and a collection of continuous functions ( q I ) I∈D 0 .
Model 2 (Continuous function).
Suppose Q is a continuous function of a and x .
We ﬁrst consider the case where the value function takes the form of equation 8.
We remark that kernel-based estimators will fail under this model assumption, as it requires the second-order derivative of the Q-function to exist.
Without loss of generality, assume q I 1 (cid:54) =
q I 2 for any two adjacent intervals I 1 , I 2 ∈ D 0 .
This guarantees that the representation in equation 8 is unique.
For any partition
D = { [0 , τ 1 ) , [ τ 1 , τ 2 ) , · · · , [ τ K , 1] } , we use J ( D ) to denote the set of change points { τ 1 , · · · , τ K } .
We impose the following conditions to establish our theories.
Assumption 1.
The number of layers L and the number of nodes in each hidden layer H diverge with n , in that HL =
O ( n ρ ) , for some constant 0 < ρ < 1 / 2 .
Assumption 2.
Functions { (cid:98) q
( (cid:96) )
I ( • ) } I∈ (cid:98) D ( (cid:96) ) are uniformly bounded.
Assumption 1 is mild, as both H and L are parameters we specify.
The part that HL =
O ( n ρ ) ensures that the stochastic error resulting from the parameter estimation in the MLP is negligible.
Assumption 2 ensures that the optimizer would not diverge in the (cid:96) ∞ sense.
Similar assumptions are commonly imposed in the literature to derive the convergence rates of DNN estimators (see e.g., Farrell et al., 2018).
These two assumptions guarantee the uniform consistency of the DNN estimator { (cid:98) q I } I∈ (cid:98) D ( (cid:96) ) .
See Lemma 1 in Appendix D for details.
Theorem 1 Suppose Model 1, Assumptions 1 and 2 hold.
Suppose m diverges to inﬁnity with n .
Then, there exists some constant γ 0 such that as long as 0 < γ ≤ γ 0 , the following events occur with probability approaching 1 (w.p.a.1), [Equation] ) =
[Equation]
) + o p [Equation] 1) for any policy π such that for any τ 0 ∈ J ( D 0 ) , Pr ( π ( X ) ∈ [ τ 0 − (cid:15), τ 0 + (cid:15) ]) → 0 as (cid:15) → 0 .
Theorem 1 establishes the properties of our method under settings where the Q ( a, x ) is piecewise function in a .
Results in (i) imply that deep jump Q-learning correctly identiﬁes the number of change points.
Results in (ii) imply that any change point in D 0 can be consistently identiﬁed. In particular, (cid:98) D ( (cid:96) ) corresponds to a subset of { 1 /m, 2 /m, · · · , ( m − 1) /m } . With a sufﬁciently large m , for any true change point τ in D 0 , there will be a change point in (cid:98) D ( (cid:96) ) that approaches τ .
Consequently, the change point locations can be consistently estimated.
To ensure the consistency of the proposed value estimator, we require that the distribution of the random variable π ( X ) induced by the target policy does not have point-masses at the change point locations.
This condition is also mild.
For instance, it automatically holds when π ( X ) has a density function on [0 , 1] .
Theorem 2 Suppose Model 2, Assumptions 1 and 2 hold.
Suppose m diverges to inﬁnity and γ decays to zero.
Then we have
[Equation]
[Figure] Theorem 2 establishes the properties of our method under settings where Q is continuous in a .
Results in (i) imply that (cid:98) q ( (cid:96) )
I ( • ) can be used to uniformly approximate Q ( a, • ) for any a ∈ I .
The consistency of the value in (ii) thus follows.
Finally, we conduct in-depth theoretical analysis to demonstrate the advantage of our estimator.
Due to space constraint, we present an informal statement below.
Details are given in Appendix C. Theorem 3 (Informal Statement) (i) When the Q-function belongs to a class of piecewise constant functions of the action a , the minimax rate of convergence of the proposed value estimator is O p ( n − 1 / 2 ) . In contrast, the minimax convergence rate of kernel-based estimator is O p ( n − 1 / 3 ) .
(ii) When the Q-function belongs to a class of smooth functions of a , the minimax convergence rate of our estimator is faster than kernel-based estimators if the bandwdith undersmoothes or oversmoothes the data.
In this section, we investigate the ﬁnite sample performance of the proposed DJQE on the synthetic and real datasets, in comparison to two kernel-based methods.
The computing infrastructure used is a virtual machine in the AWS Platform with 72 processor cores and 144GB memory.
5. S YNTHETIC D ATA
Synthetic data are generated from the following model:
[Equation]
) , X (2) , . . .
, X ( p ) iid ∼ Unif [ − 1 , 1] , where X =
[ X (1) , X (2) , . . .
, X ( p ) ] .
Consider the following different scenarios:
S1 : Q ( x, a ) =
(1 + x (1) )
I ( a < 0 . 35)
+ ( x (1) − x (2) )
I (0 .
35 ≤ a < 0 .
65) + (1 − x (2) )
I ( a ≥ 0 . 65); S2 : Q ( x, a ) =
I ( a < 0 . 25) + sin(2 πx (1) )
I (0 .
25 ≤ a < 0 .
5 − 8( x (1) − 0 .
75) 2 } I (0 .
5 ≤ a < 0 .
+ 0 .
5 I ( a ≥ 0 . 75); S3 (toy) :
Q ( x, a ) = 10 max { a 2 − 0 .
25 , 0 } log( x (1) + 2) ; S4 : Q ( x, a ) = 0 .
2(8 + 4 x (1) − 2 x (2) − 2 x (3) )
− 2(1 + 0 .
5 x (1) + 0 .
5 x (2) − 2 a ) 2 .
The Q-function is a piecewise function of a under Scenarios 1 and 2, and is continuous under Scenarios 3 (toy example considered in Section 3.1) and 4.
We set the target policy to be the optimal policy that achieves the highest possible mean reward.
We list the oracle mean value under the optimal policy for each scenario in the ﬁrst column of Table 4 in Appendix B.
We apply the proposed DJQE and two kernel-based methods (Kallus & Zhou, 2018; Colangelo &
Lee, 2020) to Scenario 1-4 with 20-dimensional covariates ( p = 20 ) and n ∈ { 50 , 100 , 200 , 300 } .
For the DJQE, we select γ ∈ { 0 . 1 , 0 . 2 , 0 . 5 } n 0 . 4 based on ﬁve-fold cross-validation. Here, we set m = n/ 10 to achieve a good balance between the bias and the computational cost (see Figure 4 in Appendix B for the detailed computational cost of the DJQE and the resulting bias as a function of m in Scenario 1 with n = 100 ).
We ﬁnd it extremely computationally intensive to compute the optimal bandwidth h ∗ using Kallus & Zhou (2018)’s method (see the detailed comparison of computational cost under different methods based on Scenario 1 in Table 3).
Thus, as suggested in
Kallus & Zhou (2018), we ﬁrst compute h ∗ using data with sample size n 0 = 50 .
To accommodate data with different sample sizes n , we adjust h ∗ by setting h ∗ { n 0 /n } 0 . 2 .
To implement Colangelo [Table] Lee (2020)’s estimator , we consider a list of bandwidths, given by h
= cσ
A n −
[Table] 2 with c ∈ { 0 . 5 , 0 .
75 , 1 . 0 , 1 . 5 } and σ A is the sample standard deviation of the action.
We then manually select the best bandwidth such that the resulting value estimator achieves the smallest mean squared error.
The conditional mean value and generalized propensity score are ﬁtted via the MLP regressor with 10 hidden layer and 10 neurons in each layer.
The average estimated value and its standard deviation over 100 replicates are illustrated in Figure 2 for different methods, with detailed values reported in Table 4 in Appendix B. In addition, we provide the size of the ﬁnal estimated partition under the
DJQE in Table 5 in Appendix B, which is much smaller than m in most cases. It can be seen from Figure 2 that the proposed DJQE is very efﬁcient and outperforms all competing methods in almost all cases.
We note that the proposed method performs reasonably well even when the sample size is small ( n = 50 ).
In contrast, two kernel-based methods fail to accurately estimate the value even in some cases when n = 300 .
Among the two kernel-based OPE approaches, we observe that the method developed by Kallus & Zhou (2018) performs better in general.
5. R EAL D ATA : P ERSONALIZED D OSE F INDING
Warfarin is commonly used for preventing thrombosis and thromboembolism.
We use the dataset provided by the International Warfarin Pharmacogenetics (Consortium, 2009) for analysis.
We choose p = 81 baseline covariates considered in Kallus & Zhou (2018).
This yields a total of 3964 with complete records of baseline information.
The response is deﬁned as the absolute distance between the international normalized ratio (INR, a measurement of the time it takes for the blood to clot) after the treatment and the ideal value 2 . 5 , i.e, Y =
−| INR − 2 .
5 | .
We use the min-max normalization to convert the range of the dose level A into [0 , 1] .
To compare among different methods, we calibrate the dataset to generate simulated outcomes.
Speciﬁcally, we ﬁrst estimate the Q-function via a MLP regressor with 10 hidden layers andneurons in each layer using the whole dataset.
The goodness-of-the-ﬁt of the ﬁtted model under the MLP regressor is reported in Table 6 in Appendix B. We next use the ﬁtted Q-function (cid:98) Q ( X, A ) to simulate the data.
Given a randomly sampled feature-action pair ( a j , x j ) from { ( A 1 , X 1 ) , · · · , ( A n , X n ) } , we set the reward r j to N { (cid:98) Q ( x j , a j ) , (cid:98) σ 2 } , where (cid:98) σ is the standard deviation of the ﬁtted residual { Y i − (cid:98) Q ( X i , A i ) } i .
Given the simulated data { ( x j , a j , r j ) : 1 ≤ j ≤ n } , we are interested in evaluating the optimal policy: π (cid:63) ( X ) ≡ arg max a ∈ [0 , 1] (cid:98) Q
( X, a ) .
The oracle value under the optimal policy is V = − 0 .
We apply the DJQE on the calibrated Warfarin data, against two kernel-based methods.
Due to the extremely intensive computation in Kallus & Zhou (2018), we directly apply the estimated optimal bandwidth h ∗ in their real data analysis, since they used the same dataset.
Biases, standard deviations, and mean squared errors of the estimated values are reported in Table 2 over 20 replicates for sample size n = 500 under different methods.
It can be observed from Table 2 that our proposed DJQE achieves much smaller mean squared error than the two kernel-based methods, when evaluating the optimal policy.
Speciﬁcally, the DJQE yields bias as 0.259 with the standard deviation as 0.416, in contrast to the large bias as 0.662 with the standard deviation as 0.742 under Kallus & Zhou (2018)’s method and the bias as 0.442 with the large standard deviation as 1.164 under Colangelo & Lee (2020)’s method.
Therefore, our proposed
DJQE for off-policy evaluation with continuous actions works better than the kernel-based methods.
D ISCUSSION
Currently, we focus on settings with a single decision point.
It would be practically interesting to extend our proposal to sequential decision making.
A potential drawback of our method is that it would be very computationally intensive for a large m as the runtime increases linearly in m .
A M ORE ON THE IMPLEMENTATION
We summarize our algorithm in Algorithm 1.
[Equation]
; 4. Apply the PELT method to get partitions: For v ∗ = 1 , . . .
, m : (i).Bell ( v ∗ ) =
min v ∈R ( v ∗ ) { Bell ( v ) + C
([ v/m, v ∗ /m ))
+ γ } ; (ii).
v 1 ← arg min v ∈R ( v ∗ ) { Bell ( v ) + C ([ v/m, v ∗ /m ))
+ γ } ; (iii).
τ ( v ∗ ) ← { v 1 , τ ( v 1 ) } ; (iv).
R ( v ∗ ) ← { v ∈ R ( v ∗ − 1) ∪ { v ∗ − 1 } :
Bell ( v ) + C ([ v/m, ( v ∗ − 1) /m ))
≤ Bell ( v ∗ − 1) } ; 5.
Construct the DR value estimator: r ← m ; l ← τ [ r ] ; While r > 0 :
(i) Let I =
[ l/m, r/m ) if r <
m else I =
[ l/m, 1] ; (cid:98) D ← (cid:98) D ∪ I ; (ii) Recall [Table] MLP: (cid:98) q
I ( · ) ← I ( i ∈ L c(cid:96) )
I ( A i ∈ I )
Y i ∼ I ( A i ∈ I )
MLP ( X i ) ; (iii) Fit propensity score: (cid:98) b
I ( · ) ← I ( i ∈ L c(cid:96) )
I ( A i ∈ I ) ∼ I
( A i ∈ I )
MLP ( X i ) ; (iv) r ← l ; l ← τ ( r ) ; 6.
Evaluation using testing dataset L : (cid:98) V ( π )+ = (cid:80) I∈ (cid:98) D (cid:16)(cid:80) i ∈ L (cid:96) I
( A i ∈ I ) (cid:104)
I { π ( X i ) ∈I} (cid:98) b I [Equation] [Figure]
[Figure]
[Figure]
B i a s    16 Initial number of intervals (m)
T i m e
C o s t ( M i nu t e s )
B A DDITIONAL E XPERIMENTAL R ESULTS
We include additional experimental results in this section.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
V = 1.
Kallus & Zhou (2018) 1.061(1.124) 1.363(
[Table]
[Table]
Colangelo & Lee (2020) 1.827(1.371) 2.292(
[Table]
[Table]
[Table]
[Table]
[Table]
Scenario
[Table]
[Table]
[Table]
Kallus & Zhou
[Table]
Colangelo & Lee (2020)
[Table]
Scenario
[Table]
[Table]
Kallus & Zhou
[Table]
[Table]
Colangelo & Lee (2020)
[Table] | (cid:98) D| / m n =n =n =n = Scenario [Table]
{+Scenario4 /
+}
{+Scenario4 /
+}
{+Scenario4 /
+}
[Table]
C A DDITIONAL T HEORETICAL R ESULTS
In this section, we conduct in-depth theoretical analysis to compare the minimax convergence rate of the proposed value estimator with kernel-based value estimator.
We ﬁrst brieﬂy summarize our theoretical ﬁndings.
When the Q-function belongs to a class of piecewise constant functions of the action a , the proposed estimator converges at a faster rate than kernel-based estimators.
Speciﬁcally, the kernel-based OPE converges at a rate of n − 1 / 3 whereas our estimator converges at a rate of n − 1 / 2 .
When the Q-function belongs to a class of Lipschitz continuous functions of a , our estimator converges at a rate of n − 1 / 5 whereas kernel-based estimators converge at a slower rate when the bandwdith undersmoothes or oversmoothes the data.
As we have commented, it remains extremely challenging to tune the bandwidth of kernel-based
OPE in practice, since OPE is an unsupervised problem.
The use of a suboptimal bandwidth would lead to a slower rate of convergence for kernel-based estimators.
In contrast, we develop a supervised learning algorithm to adaptively discretize the action space.
The tuning parameters in our procedure can be selected via cross-validation.
All the above ﬁndings are supported by our observations in the toy example (Section 3.1) and numerical experiments (Section 5) as well.
We next present our theoretical results.
We ﬁrst introduce some notations.
Deﬁne the following classes of Q-functions:
[Equation]
for some sufﬁciently small constant (cid:15) > 0 and some sufﬁciently large constants C 1 , C 2 > 0 where Λ( β, C 2 ) denotes the class of β -smooth functions (see e.g., Section 3.1.1, Shi et al., 2020).
By deﬁnition, the ﬁrst set Q 1 consists of all piecewise constant functions of a with at most C 1 −change points.
The second set Q 2 contains the class of Lipschitz continuous functions of a .
Finally, deﬁne the policy class Π = { π : π ( X ) has a density function bounded by C 4 } for some sufﬁciently large constant C 4 > 0 .
To simplify the analysis, we assume the behavior policy is
[Equation]
[Equation]
See known
such that (cid:98) b = (cid:98) b ( (cid:96) )
I =
b .
In addition, assume the conditional variance of Y given A and X is uniformly bounded away from zero.
Theorem 3 Suppose 2 β > p .
Then for any π ∈ Π and any Q ∈ Q 1 , with proper choice of γ and the MLP class, the minimax rate of convergence of the proposed value estimator is O p ( n − 1 / 2 ) .
In contrast, the minimax convergence rate of kernel-based estimator is O p ( n − 1 / 3 ) .
Suppose 4 β > 3 p .
For any Q ∈ Q 2 , with proper choice of γ and the MLP class, the minimax rate of convergence of the proposed value estimator is O p ( n − 1 / 5 ) , up to some logarithmic factors.
In contrast, the minimax convergence rate of kernel-based estimator is slower when h (cid:16) n − κ for some κ < 1 / 5 or κ > 3 / 5 .
We brieﬂy comment on the condition 2 β > p .
This condition allows that the MLP regressor to converge at a rate faster than n − 1 / 4 (see e.g., Imaizumi & Fukumizu, 2019).
Such a condition is commonly assumed in the literature on evaluating average treatment effects (see e.g., Chernozhukov et al., 2017; Farrell et al., 2018).
In the second part of Theorem 3, we further require the MLP regressor to converge at a rate faster than n − 3 / 10 .
Throughout the proof, we use c, C, c 0 , ¯ c , etc., to denote some universal constants whose values are allowed to change from place to place.
Let O i =
{ X i , Y i } denote the data summarized from the i th observation.
For any interval I , let b ( I| x ) denote the integral (cid:82) a ∈I b ( a | x ) da .
Proofs of Theorems 1 and 2 rely on Lemmas 1 and 2.
Speciﬁcally, Lemma 1 establishes the uniform convergence rate
of (cid:98) q I for any I whose length is no shorter than o γ ) and belongs to the set of intervals:
I ( m ) =
{ [ i 1 /m, i 2 /m ) : for some integers i
and i 2 that satisfy 0 ≤ i 1 < i 2 < m } ∪ { [ i 3 /m, 1] : for some integers i 3 that satisfy 0 ≤ i 3 < m } .
To state this lemma, we ﬁrst introduce some notations.
For any such interval I , deﬁne the function q I ( x )
=
E ( Y | A ∈ I , X = x ) .
It is immediate to see that the deﬁnition of q I here is consistent with the one deﬁned in equation 8 for any I ⊆ D 0 .
Lemma
The following holds when either the conditions in Theorem 1 or 2 hold:
[Equation]
for any positive constant c > 0 .
x
Lemma 2 When either the conditions in Theorem 1 or 2 hold, min I∈ (cid:98) D
[Equation]
) |I| ≥ Cγ w.p.a.1 for some constant C > 0 .
We ﬁrst present the proofs for these two lemmas.
Next we present the proofs for Theorems 1 and 2.
D. P ROOF OF L EMMA
For any sufﬁciently small constant (cid:15) > 0 , it sufﬁces to show
[Equation]
w.p.a.1, for some constant C > 1 whose value will be speciﬁed later.
The proof is divided into two parts.
In the ﬁrst part, we show there exist some constants { θ ∗ I } I such that [Equation]
[Equation]
In the second part [Equation] we show
[Equation]
The proof is hence completed.
Part 1.
Under the step model assumption, for any I 0 ∈ D 0 , q I 0 ( • ) is continuous.
Since the support
X is compact, q I 0 ( • ) is uniformly continuous.
Similarly, we can show b is uniformly continuous as well.
For any I ∈ I m , we have
[Equation]
It follows from the uniform continuity of q I 0 , b and the positivity assumption that q I ( • ) is continuous for any I ∈ I ( m ) .
Similarly, under the model assumption in Theorem 2, we can show q I ( • ) is continuous for any
I ∈ I ( m ) as well.
Consequently, when either the conditions in Theorem 1 or 2 hold, we obtain that q I ( • ) is continuous for any I ∈ I ( m ) .
By Stone-Weierstrass theorem, there exists a multivariate polynomial function q ∗I such that the absolute value of the residual q I − q ∗I is uniformly bounded by (cid:15)/ 2 .
By Theorem 1 of Yarotsky (2017), there exists a feedforward neural network with a bounded number of hidden units that uniformly approximates q ∗I , with the approximation error uniformly bounded by (cid:15)/ 2 in absolute value.
By
Lemma 1 of Farrell et al.
(2018), such a feedforward network can be embedded into an MLP with a bounded number of hidden units.
Since we allow H and L to diverge, such an MLP can be further embedded into an MLP with L layers and the widths of all layers being proportional to H .
This yields equation 10.
The proof for Part 1 is thus completed.
Part 2.
We aim to show equation 11 holds.
Under the boundedness assumption on Y , { q I ( • ) } I∈ I ( m ) are uniformly bounded.
We ﬁrst observe that MLP ( • , θ ∗I ) is a bounded function, by equation 10 and the fact that q I ( • ) is bounded.
Next
[Equation]
it follows from equation 10 that
[Equation]
By deﬁnition,
[Equation]
Suppose we can show
[Equation]
where the set Θ ∗ consists of all θ I such that sup x | MLP ( x ; θ I ) | ≤ M for some sufﬁciently large constant M such that both θ ∗I and (cid:98) θ ( (cid:96) )
I satisfy this constraint.
It thus follows from equation 12 and equation 13 that
[Equation]
w.p [Equation] a
[Equation] Similar to equation 12, we have
[Equation]
It follows that
[Equation]
Under the positivity assumption, sup x b ( I| x ) /
Pr ( A ∈ I ) ≥ 2( c − 1) − 1 for some constant c > 1 .
This yields equation
[Equation]
To complete the proof
[Equation]
it remains to show equation 14 holds.
Using similar arguments in Section A.2.2 of Farrell et al. [Equation]
for any I , we have with probability at least 1 − 2 exp( − ¯ γ ) that
[Equation]
for some constant ¯ c > 0 , where σ 2 ( I , θ ) corresponds to the variance of
[Equation]
Under the positivity assumption, we have σ 2 ( I , θ ) ≤ O (1) |I| − 1 where O (1) denotes some positive constant.
Consequently, for any I whose length is greater than cγ − 1 , we have with probability at least 1 − 2 exp( − ¯ γ ) that
[Equation]
for some constant ¯ c 0 > 0 .
Note that the number of elements in I ( m ) is upper bounded by ( m + 1) 2 .
Since m is proportional to n , by setting ¯ γ = ¯ c ∗ log n for some constant ¯ c ∗ > 0 , we have 1 − 2( m + 1) 2 exp( − ¯ γ ) → 1 .
Consequently, it follows from Bonferroni’s inequality that the following event occurs w.p
[Equation]
a.1 for any
I ∈ I ( m )
such that |I| ≥ cγ ,
[Equation]
where O (1) denotes some positive constant.
Under the given conditions, the RHS is o (1) .
The proof is hence completed.
D. P ROOF OF L EMMA
Consider a given interval I ∈ (cid:98) D ( (cid:96) ) .
We can ﬁnd some interval I (cid:48) ∈
I ( m ) ∩ (cid:98) D ( (cid:96) )
that is adjacent to
I .
Consequently
[Equation]
the interval I ∪ I (cid:48) belongs to I ( m ) as well.
It follows that
[Equation]
By deﬁnition, (cid:98) q
( (cid:96) )
I∪I (cid:48) minimizes the loss (cid:80)
[Equation]
i I ( A i ∈
I ∪ I (cid:48) )
{ Y i − (cid:98) q
( (cid:96) )
I∪I (cid:48)
( X i ) } 2 .
It follows that (cid:88) i ∈
L (cid:96)
I
[Equation]
Combining this together with equation 16 yields that
[Equation]
As both Y and (cid:98) q ( (cid:96) )
I are uniformly bounded, we have γ ≤ c | L (cid:96) | − 1 (cid:80) i ∈ L (cid:96) I
( A i ∈ I ) for some constant c > 0 .
By Bernstein’s inequality, we have for any t > 0 such that
[Equation]
(18) with probability 1 − exp( − nt 2 / ¯ c ( |I| + t )) for some constant ¯ c > 0 .
Similar to equation 15, by setting t I = ¯ c 0 max( (cid:112) |I| n − 1 log n, n − 1 log n ) for some constant ¯ c 0 > 0 , we obtain w.p.a.1 that
[Equation]
for any I ∈ I ( m ) .
Under the condition that b is continuous and A × X is compact, b is bounded.
Consequently, the probability Pr ( A ∈ I ) is proportional to the length of I .
In view of equation 18, for any I ∈ (cid:98) D ( (cid:96) ) , it shall satisfy
[Equation]
for some constant ¯ c 1 > 0 , w.p.a.1, where the last inequality follows from Cauchy-Schwarz inequality.
As γ (cid:29) n − 1 log n , we obtain |I| ≥ γ/ ¯ c 2 for any I ∈ (cid:98) D ( (cid:96) ) and some constant ¯ c 2 > 0 , w.p.a.1.
The proof is hence completed.
D. P ROOF OF T HEOREM
We begin with an outline of the proof.
The proof is divide into four steps.
In the ﬁrst step, we show
[Equation]
In the second step, we show
[Equation]
20) where δ min = min I∈D 0 |I| / 3 .
By the deﬁnition of δ min , this implies that
[Equation]
Combining equation 22 together with equation 19 proves (i) in Theorem 1.
This proves (ii) in
Theorem 1.
In the third step, we show
[Equation] [Equation]
In the last step, we show (iii) holds.
The proof is thus completed.
We next detail the proof for each of the step.
Step 1.
Assume |D 0 | > 1 .
Otherwise, equation 19 automatically holds.
Consider the partition
D = { [0 , 1] } which consists of a single interval and a zero q-function q [0 , 1] ( x ) = 0 for any x .
By deﬁnition, we have
[Equation]
Since Y is uniformly bounded and γ = o (1) , the right-hand-side (RHS) is O ( n ) .
Consequently, we obtain
[Equation] [Equation]
for some constant c 0 > 0 .
Notice that
[Equation]
We next show η 2 , η 3 = o p (1) .
Consider η 2 ﬁrst.
Under Lemma 2, we have w
[Equation]
p
[Equation]
a.1 that
[Equation]
We decompose the RHS by
[Equation]
The ﬁrst line is o p ( n ) , by Lemma 1.
Using similar arguments in equation 15, we can show the second term is upper bounded by
[Equation]
for some constant ¯ c > 0 , w.p.a.1.
In view of equation
[Equation]
the above expression can be further bounded by
[Equation]
where O (1) denotes some positive constant and the second inequality follows from the CauchySchwarz inequality.
This yields that η 2 = o p ( n ) .
Using similar arguments, we can show that η 3 = o p ( n ) .
It follows that
[Equation]
Notice that
[Equation]
Using similar arguments in bounding η 2 and η
[Equation]
we can show η 6 =
o p ( n ) and that
[Equation]
The ﬁrst term on the RHS is greater than c 0 n
[Equation]
| Q ( X, a ) − q I ( X )
| 2 da for some constant c 0 > 0 .
It follows that
[Equation]
Note that under the step model assumption, η 4 can be rewritten as (cid:80) I∈D 0 (cid:80)
i ∈ L (cid:96)
I
( A i ∈ I )
{ Y i − q
I ( X i ) } 2 .
This together with equation 25, yields that
[Equation]
For any integer k such that 1 ≤ k ≤ |D 0 |− 1 , let τ ∗ 0 ,k be the change point location that satisﬁes τ ∗ 0 ,
k
= i/m for some integer i and that | τ 0 ,k − τ ∗ 0 ,
k | < m − 1 .
Denoted by D ∗ the oracle partition formed by the change point locations { τ ∗ 0 ,k } |D 0 |− 1 k = .
Set τ ∗ 0 , 0 = 0 , τ ∗ 0 , |D 0 | = 1 and q ∗ [ τ ∗ 0 ,k − 1 ,τ ∗ 0 ,k
[Equation]
= q [
[Equation] [Equation]
for 1 ≤ k ≤ |D 0 | − 1 and q ∗ [ τ ∗ 0 ,K − 1 , 1] =
q [ τ 0 ,K − 1 , 1] .
Let ∆ k
= [ τ ∗ 0 ,k − 1 , τ ∗ 0 ,k ) ∩ [ τ 0 ,k − 1 , τ 0 ,k ) c for 1 ≤ k ≤ |D 0 | − 1 and ∆ |D 0 | =
[ τ ∗ 0 , |D 0 |− 1 , 1] ∩ [ τ 0 , |D 0 |− 1 , 1] c .
The length of each interval ∆ k is at most m − 1 .
It follows that
[Equation]
Using similar arguments in bounding the RHS of equation 17 in the proof of Lemma 2, we can show the RHS of equation 26 is o p ( n ) , as m diverges to inﬁnity.
Combining this together with equationyields
[Equation]
E
[Equation]
By deﬁnition, we have (cid:88)
[Equation]
Since |D ∗ | =
|D 0 | , it follows from equation 27 that
[Equation]
[Equation]
Note that the left-hand-side (LHR) is non-negative.
As γ > 0 , we obtain that | (cid:98) D ( (cid:96) ) | ≤ |D 0 | , w.p.a.1.
This completes the proof for the ﬁrst step.
Step 2.
It follows from equation 28 that
[Equation]
and hence (cid:88)
[Equation]
(29) w.p.a.1.
We aim to show equation 20 holds under the event deﬁned in equation 29.
Otherwise, there exists some τ 0 ∈ J ( P 0 )
such that | ˆ τ − τ 0 | ≥ δ min , for all ˆ τ ∈ J ( (cid:98) P ) .
Under the event deﬁned in equation 29, we obtain that
[Equation]
w.p.a.1.
On the other hand, since Q ( X, a ) is a constant function on [ τ 0 − δ min , τ 0 ) or [ τ 0 , τ 0 + δ min ) , we have
[Equation]
where κ
[Equation]
≡ min
[Equation]
1 and I 2 are adjacent
[Equation]
This apparently violates equation 30 where γ ≤ δ min κ 0 / (4 |D 0 − 4 | ) .
equation 20 thus holds w.p.a.1.
Step 3:
Combining the results obtained in the ﬁrst two steps yields that |D ( (cid:96) ) | =
|D 0 | , w.p.a.1.
It follows from equation 28 that
[Equation]
Using similar arguments in Step 2, we can show max τ ∈ J ( D 0 )
[Equation]
(cid:98)
τ ∈ J
[Equation]
D (
[Equation]
) ) | (cid:98) τ − τ | < (cid:15) w.p.a.1, for any sufﬁciently small (cid:15) > 0 .
The proof is hence completed.
Step 4: We begin with some notations.
For any π , we deﬁne a random policy π (cid:98) D ( (cid:96) ) according to the partition (cid:98) D ( (cid:96) ) as follows:
[Equation]
Note that (cid:82) 1 0 π (cid:98) D
( (cid:96) )
( a | x ) da = (cid:80) I⊆ (cid:98) D ( (cid:96) )
I { π ( x ) ∈ I} = 1 for any x .
Consequently, π (cid:98) D ( (cid:96) ) is a “valid” random policy.
The proposed value estimator (cid:98) V ( (cid:96) ) is doubly-robust to V ( π (cid:98) D ( (cid:96) ) ) .
By Lemma 1, the estimated Qfunction is consistent.
Consequently, (cid:98) V ( (cid:96) ) is consistent to V ( π (cid:98) D ( (cid:96) ) ) .
Since the propose estimator is a weighted average of (cid:98) V ( (cid:96) ) , it sufﬁces to show V ( π (cid:98) D ( (cid:96) ) ) is consistent to V ( π ) .
Note that
[Equation]
Similarly, we can show
[Equation] [Equation]
For each I 0 ∈ D 0 , there exists an interval
I ∈ (cid:98) D ( (cid:96) )
such that |I ∪ I 0 | / |I| → 1 .
We use (cid:98) I ( (cid:96) )to denote this interval.
Under the given conditions, we have
[Equation]
In view of equation 31, the value difference | V ( π (cid:98) D ( (cid:96) ) )
− V ( π ) | can be upper bounded by
[Equation]
The ﬁrst two terms decay to zero under the given conditions on π ( X ) .
The proof is hence completed.
D. P ROOF OF T HEOREM
We need the following lemma to prove Theorem 2.
Lemma 3 Assume conditions in Theorem 2 hold.
Then for any interval I ∈ I ( m ) with |I| (cid:29) γ and any interval I (cid:48) ∈ (cid:98) D ( (cid:96) ) with I ⊆
I (cid:48) , we have w.p.a.1 that
[Equation]
where the little- o term is uniform in I and I (cid:48) .
The rest of the proof is divided into three steps.
In the ﬁrst step, we show Assertion (i) in Theoremholds.
In the second step, we show Assertion (ii) in Theorem 2 holds.
Finally, we present the proof for Lemma 3.
Step 1.
Consider a sequence { d n } n such that d n → 0 and d n (cid:29) γ .
We aim to show
[Equation]
By Lemma 1, it sufﬁces to show
[Equation]
Suppose |I (cid:48) | ≥ d n .
Then according to Lemma 3, we can ﬁnd some I such that a ∈
I ⊆
I (cid:48) , |I| → 0 , E | q I ( X ) − q I (cid:48)
( X ) |
= o (1) .
Since |I| → 0 and a ∈ I , it follows from the uniform continuity of the Q-function that | q I ( X ) − Q ( X, a ) | → 0 .
The assertion thus follows.
Next, suppose |I (cid:48) | < d n .
Then |I (cid:48)
| → 0 as well.
It follows from the uniform continuity of the
Q-function that inf a ∈I (cid:48)
| q I (cid:48)
( X ) − Q
( X, a ) | → 0 .
The assertion thus follows.
This completes the proof for the ﬁrst step.
Step 2.
Using similar arguments in Step 4 of the proof of Theorem 1, it sufﬁces to show V ( π (cid:98) D ( (cid:96) ) ) =
V ( π ) + o p (1) .
By deﬁnition,
[Equation]
It follows that
[Equation]
In Step 1 of the proof, we have shown that inf a ∈I , I∈ (cid:98) D (
[Equation]
E | Q ( X, a ) − q I ( X ) |
= o (1) .
It follows that inf a,a
[Equation]
I∈
[Equation]
( (cid:96) ) E | Q ( X, a ) − Q ( a (cid:48) , X )
|
=
o
(1) and hence inf
a
[Equation]
a
[Equation]
D ( (cid:96)
[Equation]
E | Q ( a (cid:48) , X ) −
Q ( a (cid:48)(cid:48) , X ) |
= o (1) , by Cauchy-Schwarz inequality.
This completes the proof for the second step.
Step 3.
For a given interval I (cid:48) ∈ (cid:98) D ( (cid:96) ) , the set of intervals I considered in Lemma 3 can be classiﬁed into the following three categories.
Category 1: I = I (cid:48) .
It is immediate to see that q
I
=
q
I (cid:48) and the assertion automatically holds.
Category 2: There exists another interval I ∗ ∈
I ( m ) that satisﬁes I (cid:48)
=
I ∗ ∪ I .
Notice that the partition (cid:98) D ( (cid:96) ) ∗
=
(cid:98) D ( (cid:96) ) ∪ {I ∗ } ∪
I − {I (cid:48) } corresponds to another partition.
By deﬁnition, we have
[Equation]
and hence |
[Equation]
|
[Equation]
It follows from the deﬁnition of (cid:98) q I ∗ that
[Equation]
Therefore
[Equation]
we obtain
[Equation]
Category 3: There exist two intervals I ∗ , I ∗∗ ∈ I ( m ) that satisfy I (cid:48)
=
I ∗ ∪
I ∪ I ∗∗ .
Using similar arguments in proving equation 32, we can show that
[Equation]
Hence
[Equation]
regardless of whether I belongs to Category
[Equation]
or it belongs to Category
[Equation]
we have
[Equation] [Equation]
Using similar arguments in equation
[Equation]
we can show w.p
[Equation]
a.1 that
[Equation]
where the little- o terms are uniform in I and I (cid:48) .
Combining these together with equation 33 yields
[Equation]
γ |I| ) , for any
I and I (cid:48) , w
[Equation]
p.a.1.
Note that q I satisﬁes E
[ I ( A ∈ I ) { Y − q I ( X ) }| X ] = 0 .
We have
[Equation]
Consider the ﬁrst term on the RHS.
Note that
[Equation]
+
[Equation]
By Cauchy-Schwarz inequality, the last term on the RHS can be lower bounded by
[Equation]
It follows that
[Equation]
and hence
[Equation]
By Lemma 1 and the positivity assumption, the RHS is o p ( |I| ) .
Note that the little-
o terms are uniform in I and I (cid:48) .
As |I| (cid:29) γ , we obtain that
[Equation]
, uniformly for any I and I (cid:48) , or equivalently,
[Equation]
By the positivity assumption, we have
[Equation]
uniformly for any I and I (cid:48) , or equivalently,
[Equation]
This yields that
[Equation]
uniformly for any I and I (cid:48) .
The proof is hence completed.
D. P ROOF OF T HEOREM
In the ﬁrst two steps, we compare the minimax convergence rate when Q ∈ Q 1 .
In the next two steps, we compare the minimax convergence rate when Q ∈ Q 2 .
Step 1: We provide a lower bound for the minimax convergence of kernel-based OPE when Q ∈ Qin this step.
Consider a piecewise constant Q-function
[Equation]
[Equation]
otherwise .
Apparently, we have Q ∈ Q 1 when C 2 ≥ 1 ≥ (cid:15) .
Deﬁne a policy π such that the density function of π ( X ) equals (cid:40) 4 / 3 , if 1 / 4 ≤ π ( x ) ≤ 1 / 2 , 2 / 3 , else if 1 / 2 ≤ π ( x ) < 4 / 3 , 0 , otherwise .
We aim to show for such Q and π , the best possible convergence rate of kernel-based estimator is n − 1 / 3 .
We ﬁrst consider its variance.
Since the conditional variance of Y | A, X is uniformly bounded away from 0 and 1 , similar to Theorem 1 of Colangelo & Lee (2020), we can show the variance is lower bounded by O (1)( nh ) − 1 where O (1) denotes some positive constant.
We next consider its bias.
Since the behavior policy is know.
The bias is equal to
[Equation]
Using the change of variable a = ht + π ( X )
[Equation]
the bias equals
[Equation]
Consider any 0 < h ≤ (cid:15) for some sufﬁciently small (cid:15) > 0 .
The bias is then equal to
[Equation]
Under the symmetric condition on the kernel function, the above quantity is equal to
[Equation]
Consequently, the bias is lower bounded by O (1) h where O (1) denotes some positive constant.
To summarize, the root mean squared error of kernel based estimator is lower bounded by
O (1) { ( nh ) − 1 / 2 + h } where O (1) denotes some positive constant.
The optimal choice of h that minimizes such lower bound would be of the order O ( n − 1 / 3 ) .
Consequently, the convergence rate is lower bounded by O (1) n − 1 / 3 .
Step 2: We derive an upper bound for the minimax convergence rate of our estimator when Q ∈ Qin this step.
Using similar arguments in the proof of Lemma 1 (see Section D.1) and the proof of
Theorem 1 in Imaizumi & Fukumizu (2019), we can show that with proper choices of the MLP networks, the following holds with probability at least 1 − O ( n − C ) for some sufﬁciently large constant C > 0 ,
[Equation]
where β is deﬁned in Q 1 and Q 2 .
By equation 34 and the condition that γ (cid:29) n − 2 β/ (2 β + p ) log 2 n , using similar arguments in Steps 1 and 2 of the proof of Theorem 1, we can show that the estimated change point locations are consistent and D ( (cid:96) )
=
D 0 with probability 1 − O ( n − C ) .
Similar to equation 27, it follows from equation 34 that (cid:88)
[Equation]
up to some logarithmic factors.
This further implies that the change point locations in (cid:98) D ( (cid:96) ) converge at a rate of O ( n − 2 β/
(2 β + p ) )
up to some logarithmic factors.
We next establish the statistical properties of our value estimates.
We ﬁrst observe that when b is known, our estimator is unbiased to the inverse-propensity score weighted estimator
[Equation]
Consequently, its bias is equal to
[Equation]
Since b is bounded away from zero and inﬁnity
[Equation]
and that sup I Pr ( π ( X ) ∈ I ) / |I| ≤ C 4 , the absolute value of the bias is upper bounded by
[Equation]
where O (1) denotes some positive constant.
{+For each I ∈ (cid:98) D ( (cid:96) ) , we could ﬁnd some I 0 ∈ D 0 such that the lengths of I −I 
+}
and I 0 −I are upper bounded by O ( n − 2 β/
(2 β + p ) )
up to some logarithmic factors.
By deﬁnition, we have
[Equation]
Consequently, sup x | q I ( x ) − q I 0 ( x ) | =
O ( n − 2 β/
(2 β + p ) )
up to some logarithmic factors.
It follows that the absolute value of the bias of bias is upper bounded by O ( |D ( (cid:96) )
| n − 2 β/ (2 β + p ) ) up to some logarithmic factors.
As |D ( (cid:96) ) | =
|D 0 | ≤ C 1 with probability tending to 1 , the absolute value of the bias is upper bounded by
[Equation]
) up to some logarithmic factors.
We next consider the variance.
The variance is of the same order of magnitude of that of equation 35.
Since L is ﬁnite, it sufﬁces to consider the variance of
[Equation]
The variance of equation 37 is upper bounded by
O ( n − 1 ) E
[Equation]
)  
= O ( n − 1 )   (cid:88)
I∈ (cid:98) D ( (cid:96) )
EI ( A
[Equation]
I ) I ( π ( X ) ∈ I )
b 2 ( I| X )  
= O ( n − 1 )   (cid:88)
I∈ (cid:98) D ( (cid:96) )
EI ( π ( X ) ∈ I ) b ( I| X )  
= O ( n − 1 | (cid:98) D ( (cid:96) ) | ) , where the last equality is due to that b is uniformly bounded way from zero and that Pr ( π ( X ) ∈
I ) ≤ O (1) |I| for some positive constant O (1) .
As |D ( (cid:96) ) | =
|D 0 | ≤ C 1 with probability tending to 1 , the variance of equation 37 is upper bounded by O p ( n − 1 ) .
Consequently, the variance of our value estimator is bounded by O p ( n − 1 ) as well.
In addition, we note this bound and the bias bound in equation 36 is uniform in Q ∈ Q 1 ∪ Q 2 and π ∈ Π .
The proposed value converges at a rate of
O p ( n − 2 β/ (2 β + p ) )
up to some logarithmic terms.
When 2 β/ (2 β + p ) > 1 / 2 , both the bias and the variance decay at a rate of n − 1 / 2 up to some logarithmic factors.
This completes the proof for the ﬁrst part.
Step 3.
We provide a lower bound for the minimax convergence of kernel-based OPE when Q ∈ Qin this step.
Similar to Step 1, we can show its variance is lower bounded by O ( n − 1 h − 1 ) .
When h (cid:16) n − κ for some κ > 3 / 5 , the root mean squared error is larger than or equal to O ( n − (1 − κ ) / 2 ) .
Consider the Q-function
[Equation]
for some constant C > 0 .
With proper choice of C , we can show that such a choice of Q -function belongs to Q 2 .
Using similar arguments in Step 1, we can show the bias equals
[Equation]
Using similar arguments in Step 1, we can show the right-hand-side is lower bounded O (1) h .
When h (cid:16) n − κ for some κ < 1 / 5 , the root mean squared error is larger than or equal to O ( n − κ ) .
To summarize,
when h (cid:16) n − κ for some κ < 1 / 5 or κ > 3 / 5 , kernel-based OPE converges at a rate of n − κ ∗ for some κ ∗ < 1 / 5 .
Step 4.
We provide an upper bound for the minimax convergence of our estimator when Q ∈ Q 2 in this step.
Consider its bias ﬁrst.
Using similar arguments in Step 2, the bias is upper bounded by
O ( L − 1 ) (cid:80) L (cid:96) =1 (cid:80)
I
[Equation]
(cid:96) ) (cid:82)
[Equation]
∈I (cid:48) sup x | Q ( a, x ) − q I (cid:48) ( x ) | da .
Similar to Lemma 3, we can show for any interval I ∈ I ( m ) with |I| (cid:29) γ and any interval I (cid:48) ∈ (cid:98) D ( (cid:96) ) with I ⊆ I (cid:48) , we have w.p.a.1 that
[Equation]
[Equation]
provided that γ is proportional to n − 2 β/ (2 β + p ) up to some logarithmic factors.
For any a , there exists an interval I whose length is proportional to γ log n that covers a .
Since Q ∈ Q 2 , we have sup x | Q ( a, x ) − q I ( x ) |
=
O ( |I| ) .
This together with equation 38 yields that
[Equation]
Set I =
γ 1 / 3 , the bias is upper bounded by O (1) γ 1 / 3 .
Using similar arguments in Step 2 of the proof, the standard deviation is upper bounded by (cid:80) L (cid:96) =
1 L − 1 (cid:113) n − 1 | (cid:98) D ( (cid:96) )
| .
By equation 23, It is upper bounded by O ( n − 1 / 2 γ − 1 / 2 ) .
By setting γ to be proportional to n − 3 / 5 (this rate is achievable under the condition that 4 β > 3 p ), we obtain the rate of n − 1 / 5 .