Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider local kernel metric learning for off-policy policy evaluation (OPE)of deterministic policies in contextual bandits with continuous action spaces.
Ourwork is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosageand duration in medicine.
Although importance sampling (IS) provides a basicprinciple for OPE, it is ill-posed for the deterministic target policy with continuous actions.
Our main idea is to relax the target policy and pose the problem askernel-based estimation, while learning the kernel metric in order to minimize theoverall mean square error (MSE).
We present an analytic solution for the optimalmetric, based on the analysis of bias and variance.
Whereas prior work has beenlimited to scalar action spaces or kernel bandwidth optimization, our work takesa step further being capable of vector action spaces and metric optimization.
Weshow that our estimator is consistent, and significantly reduces MSE compared tobaseline OPE methods through experiments on various domains.
Introduction
In order to deploy a contextual bandit policy to an actual environment, such as personalized pricing[1],treatments [2], recommendation [3], advertisements [4], the performance of the policy being deployedshould be evaluated to decide whether the trained policy is suitable for the deployment.
This isbecause the interaction of the policy with the environment could be costly and dangerous.
Forexample, using an erroneous medical treatment policy to prescribe drugs for patients could resultin dire consequences.
Therefore, emerges a necessity for an algorithm that can evaluate a policy’sperformance without having it interact with the environment in an on-line manner.
Such algorithmis called the "off-policy policy evaluation" (OPE) algorithms.
OPE algorithms for contextualbandits evaluate a target policy by estimating the expected reward the it would get given an offlinedata sampled from a data collecting (behavior) policy.
Previous work on OPE for contextual bandits has mainly focused on environments with finite actions[4–10].
The work can be largely divided into three approaches [6, 9, 11].
The first approach is directmethod (DM) which learns the environment model to evaluate a target policy.
DM learns to modelthe environment for estimating the expected reward, which is used for policy evaluation.
Since theenvironment model is learned with a function approximation, the estimator is biased.
Although itis biased, it is known to have a low variance [9].
The second approach is importance sampling (IS)which corrects the data distribution induced by a behavior policy to that of a target policy [12], anduses the corrected distribution to approximate the expected reward of the target policy.
IS estimateis unbiased when behavior and target policies are given.
However, IS estimates can have a largevariance when there is large mismatch between behavior and target policy distributions [9].
Thelast approach is doubly robust (DR), which uses DM to reduce the variance of IS while keeping theunbiasedness of an IS estimate [6].
However, there is also a need for OPE algorithms for contextual bandits with continuous actions, as itcan be used to evaluate the policies for personalized continuous decision making, such as evaluating acontinuous treatment prescription policies.
And due to practical constraints, we also need
OPE algorithms for deterministic contextual bandit policies.
For example, treatment prescriptionpolicies cannot stochastically prescribe drugs to patients.
Unfortunately, most studies for OPE withfinite actions cannot be easily extended to evaluate deterministic contextual bandit policies withcontinuous actions.
This is because IS weights used for both IS and DR estimates are almost surelyzero for a deterministic target policies in continuous action space.
Thus, the work for evaluatingdeterministic contextual bandit policies continuous action space focus on this problem [2, 14, 15].
The work on OPE of deterministic contextual bandits with continuous actions can be largely dividedinto work on IS [2, 15] and doubly robust estimators (DR) [13, 14].
Both methods use IS and eliminatethe problem of almost surely having zero IS estimate when given a deterministic target policy and astochastic behavior policy in continuous action space.
Some work relaxes the deterministic policyto a kernel [2, 13, 15], and the other work discretizes the action space [14].
These methods reducethe mismatch between behavior and target policies by relaxing the Dirac-delta target policies (adeterministic target policy density can be seen as a Dirac-delta function [2]) to a kernel function andthus reduce the variance with some addition of bias.
The kernel-based IS methods for OPE use kernels to measure the similarities between the target andbehavior actions.
Furthermore, the similarities are measured according to the Euclidean distancesbetween the actions [2, 13, 15].
However, the Euclidean distance is inadequate for measuring thesimilarities in the case of kernel-based IS for OPE.
In the OPE case, assigning high similarity(or, assigning high importance weight) to the actions having similar rewards will induce less biasfor the OPE estimate when relaxing the Dirac-delta target policy density to a kernel.
In kernellearning literature, there are studies that reduce the bias of kernel regression [16] and nearest neighborclassification [17] by learning kernel metrics.
Similar to these methods, we focus on learning kernelmetric that reduces estimation bias (definition in Section 3, Eq.
(7) ) of kernel-based IS methods.
We propose local kernel metric learning for IS (KMIS) for OPE estimation of deterministic contextualbandit policies with multiple continuous actions.
Our proposed method learns a Mahalanobis distancemetric [16–19] locally at each state that shortens the distance between behavior and target actionshaving similar rewards.
The metric applied kernels measure similarities between actions according tothe Mahalanobis distances induced by the metric.
In our work, we first analytically show that the biasof a kernel-based IS estimator is a dominant factor given the optimal bandwidth that minimizes theleading MSE without a metric.
Then we derive the kernel metric that minimizes the leading orderbias given the optimal bandwidth.
In the experiments section, we show that when the proposed kernelmetric learning is used with the previous works in kernel-based IS that learn bandwidths, MSEs ofthe estimations are significantly reduced.
We report empirical results in various synthetic domainsand in real-world data.
Preliminaries2.1 Problem Setting
In this work, we focus on off-policy policy evaluation (OPE) of a deterministic target policy inan environment with the action space
A ⊂ R D , state space S ⊂ R D S , the distribution of thereward p ( r | s, a ) , the state distribution p ( s ) .
The target policy which selects an action given a stateπ ( s ) : S
→ A can be regarded to have a Dirac-delta distribution π ( a | s )
=
δ ( π ( s ) − a ) , where theprobability density value goes to infinity at the selected action given a state, and the integration ofthe function is one.
The offline data D = { s i , a i , r i } Ni =1 used for the evaluation of the deterministictarget policy π is sampled with a known stochastic behavior policy π b ( a | s ) : S → ∆( A ) .
We assumethat the support of the behavior policy π b ( a | s ) contains the actions selected by the target policy π ( s ) .
One of the methods to evaluate the target policy value ρ π = E s ∼ ( s ) ,a ∼ π ( a | s ) ,r ∼ p ( r | s,a ) [ r ] with the 85 offline data D sampled from π b is to do importance sampling (IS) estimation as in Eq.
In ourcase, the IS ratios correct the action sampling distribution π b ( a | s ) to π ( a | s ) .
But since π ( a | s ) is a
Dirac-delta function, the probability density function (PDF) value at the action in the offline data Dwhich is sampled with π b ( a | s ) is almost surely 0.
Existing works deal with the problem by relaxingthe target policy π , which is a Dirac-delta function, to an isotropic kernel K and enable IS estimationof ˆ ρ K as in Eq.
[Equation]
where, π ( a | s )
=
[Equation]
∞
[Equation]
π
[Equation]
s
[Equation]
=
a
[Equation]
2.2 Bandwidth Learning for the Isotropic Kernel Relaxed IS Estimator
As the Dirac-delta function can be regarded as a kernel having its bandwidth approaching zero, therelaxation of the Dirac-delta function to a kernel can be seen as increasing its bandwidth h .
By therelaxation, the bias (definition in Section 3, Eq.
(7)) of the kernel-relaxed IS estimate increases whileits variance is reduced.
However, as the bias and the variance compose the MSE of the estimate,the bandwidth that best balances between them should be selected.
And previous works focus onchoosing the bandwidth of the relaxed kernel [2, 15].
In the works of Su et al.
[15], they proposed abandwidth selection method that selects a bandwidth among the list of given bandwidths by using
Lepski’s principle for bandwidth selection in nonparametric statistics [20–22].
Kallus and Zhou[2] derived the leading order MSE (LOMSE) in terms of bandwidth h , sample size D , and actiondimension D which is denoted as LOMSE( h, N, D ) in Eq.
[Equation]
[Equation]
where, C b and C v are constants related to the bias (definition in Eq.
(7)) and the variance of theestimation, respectively, ∇ 2 a denotes the Laplacian operator w.r.t.
action a , the roughness ofthe kernel is R ( K ) :
= (cid:82) K ( u ) 2 d u .
For simplicity, we assumed a Gaussian kernel and used theproperty (cid:82) uu T K ( u ) d u = I .
The kernel used for the derivation satisfies (cid:82) K ( u ) d u = 1 and 106 (cid:82) u K ( u ) d u = 0 .
With the derived LOMSE in Eq.
(3), Kallus and Zhou [2] computed the bandwidththat minimizes the LOMSE as in Eq.
In the process, they assumed that h → 0 and 1 Nh D → 0 as
N
→ ∞ .
[Equation]
2.3 Mahalanobis Distance Metric
Although relaxing the Dirac-delta target policy π to a kernel K makes importance sampling possiblein our problem setting, the kernel is isotropic, and similarity of target and behavior actions in alldirections are treated equally in weighing the reward in Eq.
But the similarities between twoactions in some directions can be ignored relative to the other directions.
Figure 1 illustrates the casewhere a Mahalanobis metric is locally learned at the target action for bias reduction in making akernel-relaxed IS estimate.
The bias of the estimate is reduced by altering the shape of the kernel toproduce a higher similarity measure on a behavior action that has a similar reward to the target action.
The Mahalanobis distance metric can shorten or lengthen distances in certain directions.
We definethe Mahalanobis distance between two vectors a i ∈ R D and a j ∈ R D with the metric A as Eq.
Applying the Mahalanobis distance metric A ( = LL T ) to a kernel function can be seen as linearlytransforming the kernel inputs with the transformation matrix
L ( L T a = z ).
[Equation]
𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕2𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 ; 𝑨𝑨 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕𝑇𝑇 𝐴𝐴𝒂𝒂−𝒂𝒂 𝒕𝒕 2 (a) K ( a , a t ) = 12 π exp (cid:16) − ( a − a t ) 2(cid:17) 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕
𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕2𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 ; 𝑨𝑨 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕𝑇𝑇 𝐴𝐴𝒂𝒂−𝒂𝒂 𝒕𝒕 2 (b) K ( a , a t ; A )
= 12 π exp (cid:16) − ( a − a t ) T A ( a − a t )(cid:17)
Local Metric Learning for Kernel-Relaxed IS
To minimize the LOMSE of the kernel-relaxed IS estimate by learning a metric, we first analyze the
LOMSE when the optimal bandwidth derived by Kallus and Zhou [2] is applied to the kernel.
Theoptimal bandwidth presented in Eq.
(4) balances between the leading bias and the leading variancein first and the second term of Eq.
(3), respectively.
However the balancing is conducted with anisotropic kernel and estimation bias that could have been avoided with proper use of kernel metriccould be added in exchange of reduction in estimation variance as we see in Figure 1 which canworsen in a high dimensional space.
To analytically show this, we adopt Proposition 4 in the workof Noh et al.
[16] and show that the C b contained in the leading order bias becomes a dominant termin the LOMSE of kernel-relaxed IS OPE given an optimal bandwidth in Eq.
(4) and as the actiondimension increases.
Proposition 1.
For action dimension D ≫ 4 and given h ∗ (Eq.
(4) ), C b contained in the leading biasin Eq.
(3) is the dominant term of LOMSE( h ∗ , N, D )
(Eq.
By plugging in Eq.
(4) to Eq.
(3) , weget the following relation for an environment with high action dimension:
[Equation]
(The proof is provided in Appendix A.3.)
The proposition 1 implies that in high action dimensional environment, MSE can be significantlyreduced by the bias reduction.
Therefore, we aim to reduce the leading estimation bias in Eq.
(3)regardless of the bandwidth used in kernel-relaxed IS by using the Mahalanobis distance metric thatshortens the distance between target and behavior actions in the direction where their correspondingrewards are similar.
As mentioned in Section 2.3, applying a metric ( A = LL T ) to a kernel is equivalent to linearlytransforming input vectors of the kernel.
Therefore, the leading order bias with a metric should beanalyzed with linearly transformed actions ( z = L T a ).
By applying Taylor expansion on p ( r | s, a )and using change of variable, we get: (derivation in A.5.).
[Equation]
where, H is the Hessian operator and the expected reward is r ( s, a ) :
=
E [ r | s, a ] .
In the derivation,we assumed using a Gaussian kernel and used the property (cid:82) uu T K ( u ) d u = I .
From Eq.
(9), theleading order bias in Eq.
(9) which compose LOMSE can be minimized with A ( s ) .
In the following
Theorem 1, we introduce the optimal metric matrix A opt ( s ) that minimizes the squared leading orderbias by using the semi-definite programming solution presented in the work of Noh et al.
[17] on theleading order bias where a kernel metric is applied in Eq.
Theorem 1.
Under the assumption that the p ( r | s, a ) is twice differentiable w.r.t.
an action a ,mahalanobis metric can be applied to reduce the absolute value of the leading order bias in Eq.
(3) .when Λ + ( s ) and Λ − ( s ) are diagonal matrices of positive and negative eigenvalues of the Hessian
H a E [ r | s , a ] | a = π ( s ) , U + ( s ) and U − ( s ) are matrices of eigenvectors corresponding to Λ + ( s ) andΛ − ( s ) , respectively.
And d + ( s ) and d − ( s ) are the numbers of positive and negative eigenvalues ofthe Hessian, then the metric A opt ( s ) that minimizes the squared leading order bias is proportional to:
[Equation]
The derivation of the metric involves solving a Lagrangian equation to minimize the absolute value ofthe trace term in Eq.
(9) with constraints on A opt ( s ) (Detailed proof is provided in Appendix A.6.)
The derived metric matrix proportional to A opt ( s ) can be divided by its determinant to be A opt ( s ) .
And in the special case where the Hessian matrix contains both positive and negative eigenvalues, the metric reduces the leading order bias in Eq.
(3) to zero (see Appendix A.6).
A opt ( s ) islocally computed at a state s and target action π ( s ) as it is derived from the Hessian at that point
H a r ( s, a ) | a =
π ( s ) .
The eigenvalues and eigenvectors of the Hessian matrix represent the principaldirection of the expected reward curvatures and principal curvatures in the corresponding directions.
The principal directions of curvatures are the directions where the expected reward has the largestcurvatures.
When the metric learned from the Hessian is applied to the kernel, it can be seen as using
Mahalanobis distance instead of Euclidean distance, as shown in Figure 1 and in Eq.
[Equation]
where, u + ,i and u − ,i are positive and negative eigenvectors, respectively, and their correspondingeigenvalues are denoted as λ
+ ,i and λ − ,i , respectively.
The Mahalanobis distance is formed toincrease the distance between the behavior action a
and the target action π ( s ) on the expectedreward’s principal directions of curvatures weighted by (cid:112) d
+ λ + ,i or (cid:112) − d − λ − ,i .
To sum up, thederived A opt ( s ) is locally learned at state s and corresponding target action π ( s ) in a way thatincreases the Mahalanobis distance between a behavior action a and a target action π ( s ) on thedirections where the expected reward’s rate of change is the steepest.
We showed that by applying our metric to a kernel-relaxed IS estimator, the bias of the IS estimationcan be reduced, and therefore, the MSE is reduced.
Next, we see how the KMIS metric affectsthe convergence rate of a kernel-relaxed IS estimator w.r.t.
datasize N and the action dimension Dsimilar to Theorem 3 in the work of Kallus and Zhou [1].
Theorem 2.
Kernel-relaxed IS estimator with the KMIS metric and the optimal bandwidth h ∗ in
Eq.
(4) is a consistent estimator in which convergence rate is equal or faster than the isotropickernel-relaxed IS estimator with h ∗ .
The KMIS applied estimator converges to the true policy valuefaster than the one without the metric by the rate of O ( D − 12 ) as the action dimension increases.
The proof is made by analyzing the complexity of the sample size N and the action dimension D ofthe LOMSE in Eq.
(3) with and without the proposed metric.
(The proof is provided in Appendix A.7.)
Since the derived optimal bandwidth A opt ( s ) in Theorem 1 assumes that p ( r | s, a ) is twice differentiable, we add in a regularizers β ( s ) and γ ( s ) as in Eq.
(12) so that the metric applied kernel can fallback to an isotropic kernel in the case where the assumption is not met in practice.
The regularizersare also adjusted to satisfy | A | = 1 (See Appendix A.8 for more detail).
For the simplicity of thealgorithm, we did not discard the components along the eigenvectors with zero eigenvalues ( U 0 )but let the metric have a longer relative bandwidth on the directions compared to the others viaregularizers.
[Equation]
For the implementation of the proposed local metric A MIS for kernel-relaxed IS estimation, we fit aneural network based reward regressor with a Gaussian output layer.
The regressor learns the meanand the variance of the reward distribution from data which can be used for estimating the Hessianrequired for our algorithm.
The reward regressor may have some error regarding the estimation ofthe Hessian and could affect the estimation error of our algorithm.
We analytically analyze how theerror in the reward regressor (error in Hessian estimation) can affect the estimation error and theconvergence speed of our algorithm in Appendix A.10).
And we empirically test if our metric is stillaccurate in case where noise in the reward data increases and degrades the estimation performanceof DM in Appendix A.4.
The reward regressor also can be used to estimate the optimal bandwidth.
After obtaining the Hessian from the regressor, we linearly transform the kernel inputs with thetransformation matrix L from the proposed metric A MIS = LL T as in Eq.
We provide pseudocode in Algorithm 1 for a better understanding of the readers.
[Equation]
) T ( π ( s ) − a ) 6:
1 Nh D (cid:80) Ni
=1 K (cid:0) z h (cid:1) r i π b ( a i | s i )
Experiments
The experiments show that KMIS can be used with the existing works on bandwidth learning, andtheir MSEs are reduced by our metric.
Since our algorithm learns the metric for kernel-relaxed IS, wecompare our algorithm with the other kernel based IS methods.
The baselines include a discretized
OPE estimator [2] that discretizes the action space by 10 for each action dimension (100 bins) for an
IS estimation (for details, see Appendix A.9).
The work of Kallus and Zhou [2] and SLOPE [15]are also compared, where both learn bandwidths for kernel-relaxed IS.
For both baselines, we usethe code provided by the authors.
For Kallus and Zhou’s estimator, we use a reward regressor forfor bandwidth learning instead of making kernel density estimation (KDE) as in their work sinceusing KDE to estimate the bandwidth is computationally expensive.
KMIS uses both the regressorand the learned bandwidth by the baselines for OPE.
Kallus and Zhou’s estimator and our KMISshare the same neural network reward regressor composed of two hidden layers of 128 neurons forlearning bandwidth and metric, respectively.
For the activation function of the neural network, weused tanh.
For more details and codes for our algorithm, look at Appendix A.9.
For SLOPE, we selecta bandwidth from geometrically spaced bandwidths (cid:8) 2 − i : i ∈ [1 , 7] , i ∈
Z (cid:9) as in their work [15] forthe synthetic domain.
For the Warfarin data, we selected a bandwidth from (cid:8) 2 − i : i ∈ [ − 2 , 7] , i ∈
Z (cid:9) .
We do not correct the boundary bias as in the works of Kallus and Zhou [2], Su et al.
[15] since weregard the bias induced by the boundaries are negligible compared to the bias reduced by our locallylearned metric.
We test our algorithm and baselines on various synthetic domains and the Warfarindosing dataset provided by International Warfarin Pharmacogenetics.
4.1 Synthetic Data
In the experiment with the synthetic data, we show that our metric can be applied to offline datasampled from environments with various reward functions.
Furthermore, we show that learninglocal metrics can reduce MSEs of existing kernel-relaxed IS estimators.
To show this, three kinds ofsynthetic test domains are prepared.
We first test on the synthetic domain with a quadratic reward function w.r.t.
actions.
As our metricis computed from the Hessian H a r ( s , a ) | a
= π ( s ) , we first test if the metric can be learned fromthe neural network reward regressor.
If the metric is learned successfully, the metric would beformed in an ellipsoidal shape and would be the same for every state and target action as the Hessianof a quadratic function is a constant.
In the experiment, although our estimator uses the Hessianinformation from the neural network reward regressor, our metric learning applied kernel-relaxed ISestimator outperforms the reward regressor, and other baselines.
After checking if the metric can be properly learned from the reward regressor, we see if the metriccan still be learned in an environment where negative absolute error is given as a reward.
Since the
Hessian of the reward function is zero at the reward, it is reasonable to think that the metric learningwould fail in such a case.
However, since we are dealing with finite samples and because we areusing a neural network reward regressor, we conjecture that some form of convex shaped rewardestimate is learned by the regressor and makes metric learning possible.
We also let one of the actiondimensions be unrelated to the reward.
If metric learning is successful, the metric will be learned tohave a relatively larger bandwidth to the direction of the action dimension.
Lastly, we test KMIS on an exponential reward whose shape is similar to the rewards used inmulti-goal environments, which was introduced in the works of Haarnoja et al.
Since the twosynthetic test domains previously introduced have true Hessian H a r ( s , a ) | a
= π ( s ) of constant andzero, respectively, we test our algorithm on an exponential reward where it has Talyor expansionterms larger than the second order.
[Figure]
Means and standard errors of squared errors obtained from 80 trials are drawn in the figures.
The second row also marks the results of the first row when the datasize used in the estimation is 20k.
The last row shows the learned metric at each test domain.
All synthetic test domains use actions and states that are a ∈ R 2 , and s ∈ R 2 .
For the test domainwith the quadratic reward, the states are uniformly sampled in the range of [ − 1 , 1] , behavior policy isthe Gaussian π b ( a | s ) =
N ( s + [ 0 . 2 0 .
5 2 I ) , target policy is π ( s )
= s , and the reward is a quadraticfunction w.r.t.
actions.
For the details, look at Appendix A.9.
The test environment with the negativeabsolute error is prepared by having both states and actions uniformly sampled in the range of [ − 1 , 1] ,the target policy is designed to be π ( s ) = 0 .
5 s , and the reward is r ( s , a ) =
−| 0 .
5 s 1 − a 1 | , where s 1and a 1 are the first dimensions of the state and action vectors.
The test domain with the exponential reward is made with states and actions uniformly sampled from the range of [ − 1 , 1] , and target policyπ ( s )
= s + [ 0 . 50 ] .
For the details of the exponential reward, look at Appendix A.9.
The experimental results in Figure 2 show that metric learning reduces the MSEs in the kernel relaxed
IS estimates and also outperforms the reward regressor, which is used for computing the Hessianrequired for learning the metric.
The results in the first row of Figure 2 show that the MSE of SLOPEand Kallus et al.
’s estimator are reduced by our metric.
To see whether the learning metrics alwaysreduce MSE in the kernel-relaxed IS estimator, we test KMIS on given bandwidths.
Seeing the resultsin the second row of Figure 2, we can see that most of the time, the MSE is reduced with metriclearning.
For the discretized OPE estimator, it performs worst in the quadratic reward domain.
Butfor the other domains, it performs better than the other baselines.
The absolute error domain and themulti-goal domain have uniform action distributions and no noise in the rewards.
In this condition,because we apply self-normalization to all estimators, the discretized OPE would take an averageof the behavior rewards which corresponding behavior actions are in the same discretized bins withthe target actions which can be accurate in this condition.
In the second row of Figure 2, the dottedvertical lines present the average of the bandwidths learned by Kallus et al.
(blue) or SLOPE (red),and the markers show the average of the learned bandwidths and the MSE by each estimator when20k data points are used to obtain the OPE result.
Since the learned bandwidth can vary for eachrun, the markers may not exactly lie on the black lines denoting the fixed bandwidth’s results.
To seehow metric is learned and also to see how it helps reduce the bias of the kernel-relaxed IS estimate,we show the reward landscape at s =
[ 00 ] and the learned metric in black crosses in the last row.
Bylooking at the metric learned for the quadratic reward, we see that our metric is properly learned withthe Hessian H a r ( s , a ) | a = π ( s ) provided by the reward regressor as the learned metrics are similarfor most actions.
For the negative absolute error test domain with the dummy action dimension, wesee that the metric is learned to ignore the dummy action dimension a 2 by having longer relativebandwidth to the direction of a 2 while having shorter relative bandwidth in the direction of a 1 whichis used for computing the reward.
This result empirically verifies our conjecture made earlier, whichsuggests that our metric learning algorithm can be generally applied to the dataset where the expectedreward or p ( r | s, a ) is not twice differentiable.
The metric learned on the multi-goal reward madewith exponents verifies that our metric indeed learns the local metric for each action according to thereward landscape.
[Figure]
In Proposition 1, we discussed that the leading order bias becomes a dominant term in the LOMSEof isotropic kernel smoothed IS estimator.
To validate the Proposition, we show the empirical biasvariance decomposition of MSEs ( MSE[ (cid:98) ρ K ] =
Bias[ (cid:98) ρ K ] 2 + V ar [ (cid:98) ρ K ] ) of the Kallus&Zhou and
Kallus&Zhou+KMIS estimates in Figure .5.
Resultsshow that empirical estimation bias increases as theaction dimension increases and both algorithms suffer from high bias.
But our method reduces biaswith the learned metric and shows best performance.
While Kallus&Zhou+KMIS has reduced empiricalbias compared to Kallus&Zhou, its variance is increased more than that of Kallus&Zhou.
Althoughthe leading order variance is unaffected by the metric,the ignored components of the estimation variancecould be affected by the metric and resulted in increase in the variance.
4.2 Warfarin Dosing Data
To test our algorithm on a more realistic dataset, we test our algorithm on the Warfarin dosing dataset[23].
Warfarin is a treatment that is commonly used for preventing blood clots.
The dataset containsinformation on the patients who got treated, therapeutic doses, and the resulting outcomes.
Theoutcomes of the treatments are reported in the international normalized ratio (INR), which measureshow quickly blood clots.
We use a similar experimental setting used by Kallus and Zhou [2].
Asin their work, we use the 81 features of the patient information selected by Kallus and Zhou [2] asstates s .
For the behavior action vectors, the first dimension is sampled from the normal distribution
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Sample Size (10 3 ) 0.
l o g ( M S E )
Kallus & Zhou Kallus & Zhou + KMIS (ours)
SLOPESLOPE + KMIS (ours)
DM (a)    2 log(h)     l o g ( M S E )
Bandwidth selected by SLOPE
Bandwidth selected by Kallus & Zhou
No metric KMIS (ours)
Kallus & Zhou Kallus & Zhou + KMIS (ours)
SLOPESLOPE + KMIS (ours)
[Figure]
N ( µ ∗ + σ ∗ √ 0 .
5 z BMI , ( σ ∗ √ 0 .
5) 2 ) truncated by the minimum and maximum therapeutic doses a ∗ minand a ∗ max in the data.
z BMI is the z score of patients’ BMIs, µ ∗ and σ ∗ are the mean and the standarddeviation of the therapeutic dosages, respectively.
To test our algorithm we add a dummy actiondimension sampled from a uniform distribution a 2 ∼ unif[ a ∗ min , a ∗ max ] .
For the reward, since Kallusand Zhou [2] reported that the INR in the dataset is inadequate for testing OPE algorithms, we usetheir cost function as a negative reward function r = − max( | a 1 − a ∗ | − 0 .
1 a ∗ , 0) .
The rewardfunction is only dependent on the first action dimension.
The target policy is π ( s ) =
[ s BMI 0 ] .
Experimental results in Figure 4a show that our metric reduces the MSE of the kernel-relaxed ISestimate in Warfarin data that is more realistic than that of the synthetic domain.
In the figure, we seethat even though the MSE of the reward regressor is high, SLOPE + KMIS is able to learn the metricthat reduces the MSE of SLOPE.
Kallus and Zhou + KMIS also reduce the MSE of the estimatorproposed by Kallus and Zhou.
However, since the bandwidth learned by Kallus and Zhou’s model istoo big, as shown in Figure 4 there are limitations in the reduction of its MSE via metric.
Discussion and Conclusion
We present local metric learning for kernel-relaxed IS (KMIS).
KMIS locally learns metrics at agiven state, and the metric reduces the bias of the kernel-relaxed IS estimate.
This is done by learningthe metric to assign high similarity to the behavior action that has similar rewards to that of the targetaction for IS.
Specifically, given a state, our proposed method learns the Mahalanobis distance metricthat shortens the distance between a pair of behavior and target actions when their correspondingrewards are similar.
Our experimental results suggest that applying KMIS to a kernel-relaxed ISestimation reduces MSE in most cases regardless of given bandwidths.
Moreover, we also showedthat KMIS could be used for offline data sampled from environments with various reward functions.
And most importantly, KMIS can be used with other kernel-based IS estimators that learn bandwidths.
The limitation of our work is that KMIS assumes that a optimal bandwidth is provided.
Becauseof this, when suboptimal bandwidth is given, our algorithm may not fully function as intended.
Another limitation is that our algorithm is dependent on the Hessian estimated by the reward regressorwhich can be erroneous.
Although our work has some limitations, we have shown in the empiricalstudies that KMIS reduces MSEs of kernel-relaxed IS estimates for almost every bandwidths.
Ourwork also opens a new research direction for the kernel-based methods.
Our experimental results in
Figure 2, especially the results from quadratic reward test domain, suggests that optimal bandwidthsare different for a kernel with and without a metric.
This result opens an interesting future studydirection of jointly optimizing the bandwidth and the metric.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default to [Yes] , [No] or [N/A].
You are strongly encouraged to include a justification to your answer , either by referencing theappropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 Bias of the Kernel-Relaxed IS Estimate w.r.t Bandwidths
Bias of the kernel-relaxed IS estimate w.r.t bandwidths as derived in the work of Kallus and Zhou [1].
Define E π b [ · ] := E
[Equation]
s ) , a ∼ π b ( a | s ) , r ∼ p
[Equation]
r | s , a ) [ · ] , then the bias of a kernel-relaxed IS estimate is:
[Equation] [Equation]
For the first term of Eq.
[Equation]
where the following relations are used in the derivation,
[Equation]
Taylor expansion of p ( r | s , a ) at a = π ( s ) ,
[Equation]
using the relation h u = a − π ( s ) ,
[Equation]
A.2 Variance of the Kernel-Relaxed IS Estimate w.r.t Bandwidths
Variance of the kernel-relaxed IS estimate w.r.t bandwidths as derived in the work of
Kallus and Zhou [1].
Define E π b [ · ] :=
E s ∼ p
[Equation]
s ) , a ∼ π b ( a | s ) , r ∼ p
[Equation]
r | s , a ) [ · ] , then the variance of a kernel-relaxed IS estimate is:
Var[
[Equation]
Var
[Equation]
The second term of the Eq.
(16) is,
[Equation] [Equation]
[Equation]
The first term of the Eq.
(16) is,
[Equation]
By substituting a = h u + π ( s ) (cid:16) which comes from u = a − π ( s ) h
[Equation] [Equation]
Let g r ( s , a ) :
= p ( r | s , a ) π b ( a | s ) , and apply Taylor expansion at a = π ( s ) ,g r
[Equation]
a )
[Equation]
( s , π ( s ))
+ ( a − π ( s ))
T ∇ a g r ( s , a ) | a = π ( s ) +2( a − π ( s
[Equation]
r
[Equation] [Equation]
| a =
[Equation]
s
[Equation]
a
[Equation] [Equation]
substitute a = h u + π
[Equation]
[Equation]
Plugging in Eq.
(24) to Eq.
[Equation]
(cid:20)(cid:90)
[Equation]
u
[Equation]
s
[Equation] [Equation]
( s
[Equation]
u
[Equation]
(cid:17) d u
[Equation]
(cid:18)
where, R ( K ) :=
(cid:90) K ( u )
[Equation]
d u (cid:19)
[Equation]
K
[Equation]
(cid:20)(cid:90) r
[Equation]
p
[Equation]
s
[Equation]
From Eq.
(20) and Eq.
(25), the variance can be represented as,
[Equation]
A.3 Proof of Proposition 1
We adopt Proposition 4 in the work of Noh et al.
[2] which shows that the bias is a dominant term inthe MSE of kernel regression to our problem setting in kernel-relaxed IS OPE.
We show that the biasis also a dominant term in the MSE of kernel-relaxed IS OPE when the action dimension is muchbigger than 4.
Proposition 1.
For action dimension D ≫ 4 and given h ∗ (Eq.
(4) ), C b contained in the leading biasin Eq.
(3) is the dominant term of LOMSE( h ∗ , N, D )
(Eq.
By plugging in Eq.
(4) to Eq.
(3) , weget the following relation for an environment with high action dimension:
[Equation]
Proof.
The leading order MSE derived w.r.t.
h by Kallus and Zhou [2] is,
[Equation] [Equation]
When using a kernel without a metric, the optimal bandwidth ( h ∗ ) that minimizes the leading order
MSE is,
[Equation] [Equation]
[Equation]
And the leading order MSE with the optimal kernel bandwidth is,
[Equation]
A.4 Experiment with Varying Noise Level in the Rewards
In Figure 5, we also observe increasing estimation bias when the noise in the rewards increases.
DMsuffers the most as the function estimation becomes difficult as the noise increases.
However, ourmetric reduces leading order bias and shows best performance.
[Figure]
A.5 Derivation of Eq. (9)
For the derivation, we use the following relations:
[Equation]
Plugging in Eq.
(35) to Eq.
(8), and changing the ordering inside the trace operator,
[Equation]
[Table]
A.6 Proof of Theorem 1
We use the semi-definite programming solution presented in the work of Noh et al.
[3] forcomputing the metric matrix A opt that minimizes the tr (cid:0) A − 1 ( s ) B ( s ) (cid:1) 2 for nearest neighborclassification.
We use the solution to minimize the bias of kernel-relaxed IS OPE.
In our case, we use
B ( s ) :
= H a E
[ r | s , a ] | a
= π ( s ) which comes from Eq.
Theorem 1.
Under the assumption that the p ( r | s , a ) is twice differentiable w.r.t.
an action a ,when Λ + ( s ) and Λ − ( s ) are diagonal matrices of positive and negative eigenvalues of the Hessian
H a E [ r | s , a ] | a = π ( s ) , U + ( s ) and U − ( s ) are matrices of eigenvectors corresponding to Λ + ( s ) andΛ − ( s ) , respectively.
And d + ( s ) and d − ( s ) are the numbers of positive and negative eigenvalues ofthe Hessian, then the metric A opt ( s ) that minimizes the squared leading order bias is proportional to:
[Equation]
Proof.
Define B ( s ) :
= H a r ( s , a ) | π ( s ) , then we want A ( s ) that,
[Equation]
[Equation]
In the cases where some of the eigenvalues of B ( s ) are 0, the components along those directions canbe discarded.
When B ( s ) has both positive and negative eigenvalues, A opt ( s ) makes the trace 0.
A
[Equation]
where β ( s ) is a scalar value dependent on a state s that scales the metric matrix to be | A ( s ) | = 1 .
When B ( s ) only has negative or positive values, than we can solve the Lagrangian equation F ( s ) .
Incase where all eigenvalues of B ( s ) are positive,
[Equation]
38) (cid:0) since A ( s ) =
A T ( s ) , B ( s ) =
B T ( s ) , | A ( s ) | = 1 (cid:1) (i) Assume c ( s ) = 0 ,
Then either tr (cid:0) A − 1 ( s ) B ( s ) (cid:1) = 0 , or, A − 1 ( s ) B ( s ) =
We first check if tr (cid:0) A − 1 ( s ) B ( s ) (cid:1) = 0 .
[Equation]
B
[Equation] [Equation]
s
[Equation]
1 ( s
[Equation]
B
[Equation] [Equation]
Since B ( s ) =
U + ( s )Λ + ( s ) U T + ( s ) ,
B 12 ( s ) =
B T 2 ( s ) =
U + ( s )Λ 12 + ( s ) U T + ( s ) .
And as
A ( s ) ≻ 0 , its inverse is also positive definite A − 1 ( s ) ≻ 0 .
Using these relations, we canshow that the term inside the trace in Eq.
(39) is a positive definite matrix.
[Equation]
B T 2 ( s ) a (cid:17) T A − 1 ( s ) (cid:16) B T 2 ( s ) a (cid:17) > 0 , ∀ a ∈ R D \{ 0 }
Because B 12 ( s ) A − 1 ( s ) B 12 ( s ) ≻ 0 , tr (cid:16) B 12 ( s ) A − 1 ( s ) B 12 ( s ) (cid:17) > 0 .
Then, from Eq.
(39),tr (cid:0)
A − 1 ( s ) B ( s ) (cid:1) >
A − 1 ( s ) B ( s ) ̸ = 0 , as it’s trace is positive
Therefore, the assumption is wrong, and c ( s ) ̸ = 0 (ii) Assume c ( s ) ̸ = 0 ,
Since the Lagrangian multiplier c ( s ) and tr (cid:0) A − 1 ( s ) B ( s ) (cid:1) in Eq.
(38) are scalar values,
A ( s ) needs to be a scalar multiple of B ( s ) to match the scalar multiple of identity matrix inthe LHS of Eq.
Therefore the A opt ( s ) is a scalar multiple of B ( s ) .
Since A ( s ) need tosatisfy | A ( s ) | = 1 , the optimal metric is,
[Equation]
(40) where the β ( s ) and d + ( s ) are scalars dependent on a state s , and β ( s ) scales the A opt ( s ) tobe | A opt | = 1 .
Similarly, we can derive A opt ( s ) for the case where all eigenvalues of B ( s ) are negative.
A.7 Proof of Theorem 2
We compare the convergence speed of kernel-relaxed IS estimators with and without metrics learnedby KMIS when both are provided with the optimal bandwidth by the work of Kallus and Zhou [1].
The convergence speed is analyzed similarly to Theorem 3 in the work of Kallus and Zhou [1].
Theorem 2.
Kernel-relaxed IS estimator with the KMIS metric and the optimal bandwidth h ∗ in
Eq.
(4) is a consistent estimator in which convergence rate is equal or faster than the isotropickernel-relaxed IS estimator with h ∗ .
The KMIS applied estimator converges to the true policy valuefaster than the one without the metric by the rate of O ( D − 12 ) as the action dimension increases.
Proof.
The MSE of a kernel-relaxed IS estimation without a metric derived by Kallus and Zhou [1] interms of action dimension D and data size of N is (from the derived bias and variance in Appendix A.1and Appendix A.2),
[Equation]
where, the MSE g consists of squared bias and the variance of the estimate ˆ ρ K .
Since the optimalbandwidth presented in Eq.
(4) is h ∗
=
O (cid:16) DN 1 D +4 (cid:17) , the convergence of the MSE g without themetric is:
[Equation]
Since the MSE of the kernel-relaxed IS estimator with h ∗ converges to 0 by O (cid:16)(cid:0) DN (cid:1) 4 D +4 (cid:17) , Theestimation approaches to the true policy value by the rate of O (cid:16)(cid:0) DN (cid:1) 2 D +4 (cid:17) .
With our metric learning, C b can be reduced.
When the Hessian H a r ( s, a ) | a
= π ( s ) contains bothnegative and positive eigenvalues, C b is reduced to 0.
And when the Hessian is a zero matrix, wecannot use metric and no bias can be reduced.
Therefore, if we denote the C b reduced by our metricas C b,A ,
[Equation]
[Equation]
In the best case, where C b,A = 0 , MSE with our metric and the optimal bandwidth
MSE( h ∗ , A ( s ) , N, D ) converges to zero by:
[Equation]
where
[Equation] [Equation]
The MSE of our metric applied kernel-realxed IS estimation converges to 0 faster than the one withoutthe metric by O ( D − 1 )
Therefore, our algorithm converges to the true policy value faster than the onewithout the metric by O ( D − 12 ) .
A.8 Regularizers for the KMIS Metric
To get β ( s ) and γ ( s ) , we first derive the metric that is proportional to A opt ( s ) in Eq.
Then weadd a small positive real coefficient c ( s ) to it to avoid the metric matrix from having determinant of0 .
The coefficient c ( s ) is selected to be the maximum value among eigenvalues in the Hessian ofreward multiplied by 0.01.
We let the coefficient depend on the state since the scale of eigenvalues ofa Hessian (of reward) matrix differs depending on experiment environments.
With the coefficient c ( s ) added to the metric matrix, we compute the determinant of the resultingmatrix M ( s ) .
To scale the metric matrix to have determinant of 1, We scale the matrix by multiplyingβ ( s )
= 1 | M | ( − 1 /D )
(where D is an action dimension).
Then the γ ( s ) =
β ( s ) ∗ c ( s )
A.9 Experiment Details
For all experiments, we used a neural network reward regressor with 2 fully-connected layers of 128hidden units with tanh activations, and a Gaussian output layer.
The neural network was trained withlearning rate of 5e-4.
The learning rate was chosen by conducting grid search over {1e-4, 5e-4, 1e-3}.
The reward regressor was trained with early stopping rule where we stop training when the validationerror does not decrease for 20 training epochs.
And we used the trained weights that showed thelowest validation error during the training.
For the train and validation split, 20% of the availableoffline data was used as the validation data, and the rest of the data was used for the training.
Forthe estimation time, when we test our algorithm on the synthetic domain with absolute error anda dummy action dimension, our algorithm takes 40 seconds on average to make an OPE estimatewith 20k offline data on the i7 CPU with 32GB RAM.
The experimental results were made with
C2-standard-4 instances on the Google Cloud Platform where each instance has 4 virtual CPUs and16GB of RAM.
Self-Normalization All estimators in the experiments are self-normalized as in the work of
Kallus and Zhou [1].
Self-normalization is used because it reduces the estimation variance significantly with the addition of a small bias and yields a smaller MSE in result compared to theestimation without self-normalization [1, 4].
The self-normalized kernel-relaxed IS estimator withouta metric is shown in Eq.
[Equation]
No Boundary Bias Correction Action space can be bounded, and the kernels used in importancesampling can be extended past the bounds.
As the actions of the offline data are only observed insidethe bounds, the kernels extending outside the bounds will induce bias.
Previous works [1, 5] made acorrection for the induced bias by truncating the kernel by the action bounds of each dimension andnormalizing the kernel.
However, we do not correct the boundary bias for all estimators in this worksince we regard the boundary bias as negligible compared to the bias reduced by our locally learnedmetric.
Removing the boundary bias correction makes kernel-relaxed IS estimators simpler to usethan those with corrections, as the implementation of boundary bias correction requires informationon the action bounds, and the integration of the kernel within the action bounds.
Discretized OPE Estimator Discretized ope estimator discretizes the action space bounded by theminimum and maximum value of the behavior actions for each dimension with given number binsassigned for each dimension.
In our experiment we discretized 10 for each dimension resulting in
discretized bins in 2D action space.
The IS ratio of the discretized OPE estimator is composed ofthe indicator function in the numerator and the behavior policy probability density function integratedover the bin in the denominator.
A.9.1 Synthetic Domains Experiments
For the synthetic domain, we used dropout after each hidden layer with dropout rate of 0.5.
Thedropout rate was chosen by conducting grid search over {0.5, 0.8}.
Behavior policy density valueswere clipped below 032 for the quadratic reward domain.
The clipping threshold is chosen byconducting grid search over {3e-1, 1e-1, 1e-2}.
For absolute error domain and multi-goal rewarddomain, as their behavior policy density is uniform density, we did not apply behavior density valueclipping.
The clipping threshold that showed stable convergence and low MSE was chosen.
Clippingthe behavior policy density value adds a small bias to the estimation, but reduces a significant variance,and therefore MSE is smaller compared to the estimations without the clipping [1].
Quadratic Reward Domain The conditional reward distribution given a state s and a action a is:
[Equation] [Equation]
where, r ( s , a ) :
=
E [ r | s , a ] .
The mean reward of a quadratic reward domain given a state s and aaction a is:
[Equation] [Equation]
[Equation]
Dummy Action Dimension Domain
Each dimension of actions and states are sampled uniformrandomly in the range of [ − 1 , 1] .
The absolute error which is only dependent on the first dimensionof action is designed as shown below.
[Equation]
where the reward is not sampled from a distribution but determined by states s and actions a .
Multi-Goal Reward Domain Each dimension of actions and states are sampled uniform randomlyin the range of [ − 1 , 1] .
The multi-goal reward with exponential functions is designed as shown below.
[Equation] [Equation]
where the reward is not sampled from a distribution but determined by states s and actions a .
A.9.2 Warfarin Data Experiments
For the warfarin data, we used L2 regularizer for the 2 hidden layers with the coefficient of 1e-1.
The
L2 regularizer coefficient was chosen by conducting grid search over {1e-3, 1e-2, 1e-1}.
We chose thecoefficient that trains reward regressor in a stable manner and have low validation error for the rewardregressor.
Behavior policy density values below 1e-1.
The clipping threshold is chosen by conductinggrid search over {1e-3, 1e-2, 1e-1}.
The clipping threshold that showed stable convergence and low
MSE was chosen.
The behavior policy density values are clipped for the same reason to the syntheticdomain.
A.10 Effect of
Error in a Reward Regression (Hessian Estimator for KMIS)
on theEstimation Error and Convergence Speed of KMISError Analysis
To reduce the squared leading order bias LOBIAS( h, A ( s )) 2 in Eq.
(54) (theequation can be derived with bias from Eq.
(15), variance from Eq.
(26), and the leading orderbias with metric from Eq.
(9) ) of a kernel-relaxed IS estimator, KMIS learns a metric that reducesthe absolute value of trace that is contained in the LOBIAS.
To reduce the absolute value of thetrace | tr( A ( s ) − 1 H ( s )) | , we derived the optimal metric matrix
A ( s ) given the true Hessian matrixof expected reward w.r.t.
an action selected by the target policy H ( s ) :
= H a E
[ r | s , a ] | a = π ( s ) in
Appendix A.6.
However, since we will be using estimated Hessian matrix (cid:101) H ( s ) acquired from areward regressor, we analyze how the error in the reward regressor (error in the estimated Hessian(cid:101) H ( s ) ) affects KMIS estimation error.
[Equation]
where,
[Equation]
In Appendix A.6, we proved that when H ( s ) contains both positive and negative eigenvalues, ourmetric makes the squared leading order bias LOBIAS( h, A ( s ))
2 = 0 by making the absolute value ofthe trace | tr( A ( s ) − 1 H ( s )) | zero.
In this section we analyze the upper bound of LOBIAS( h, (cid:101)
A ( s )) 2in the same setting.
For analyzing, we use the following notations regarding the true Hessian H ( s ) ,metric computed from the true Hessian A ( s ) , estimated Hessian from the reward regressor (cid:101)
H ( s ) andthe metric computed from it (cid:101)
A ( s ) :
[Equation]
A
[Equation]
(cid:18)
[Equation] [Equation]
(58) Λ( s ) =
(cid:18) Λ + ( s )
Λ − ( s ) (cid:19) Λ + ( s ) =   λ + , 1 ( s ) · · · λ +
[Equation]
2 ( s ) .
 
U + ( s ) =
(cid:32) | | u + , 1 ( s ) u + , 2 ( s ) · · · | | (cid:33) (cid:101) H ( s ) = (cid:104)(cid:101) U
+ ( s ) (cid:101)
U − ( s )
[Equation]
where we used eigendecomposition on H ( s ) .
Λ + ( s ) and Λ − ( s ) are diagonal matrices containingpositive and negative eigenvalues, respectively.
U + ( s ) and U − ( s ) are matrices containing eigenvectors corresponding to Λ + ( s ) and Λ − ( s ) , respectively.
Similarly, (cid:101) Λ + ( s ) , (cid:101) Λ − ( s ) , (cid:101) U + ( s ) , (cid:101)
U − ( s ) aredecomposed from (cid:101) H ( s ) .
We assume that (cid:101) H ( s ) correctly estimates the sign (positive, negative signs)of
the eigenvalues in H ( s ) .
Because of the assumption, there are d + and d − in Eq.
We quantifythe error in (cid:101) H ( s ) with constants ϵ and η ( ϵ ≥ 0 , η > 0 ) by:
[Equation] [Equation]
65) where, a, b ∈ { + , −} i, j ∈ { 1 , 2 , 3 ,
[Equation]
The trace term tr( (cid:101)
A ( s ) − 1 H ( s )) in the LOBIAS( h, (cid:101)
A ( s )) 2 is,tr( (cid:101)
A ( s ) − 1 H ( s ))
= tr (cid:16) [
[Equation]
where in the third equality (Eq.68) we used,
[Equation]
( s )
[Equation]
, where, a, b ∈ { + , −}
[Equation]
Expanding X ( s ) we get,
[Equation]
We get the following inequality for X ( s ) by applying Eq.
[Equation]
Similarly, we get,
[Equation]
The trace term tr( (cid:101)
A ( s ) − 1 H ( s ))
( = X ( s ) − Y ( s ) ) satisfies the following inequality:
[Equation]
[Equation]
Then the absolute value of the trace term satisfies the following inequality:
[Equation]
74) where, | X u − Y l | , | X l − Y u | are,
[Equation] [Equation]
1) , where, d
[Equation]
By defining d max :=
max { d + , d − } , we can rewrite Eq.
[Equation]
Then we get the following inequality for LOBIAS( h, (cid:101)
A ( s )) 2 :
[Equation]
When there is no error in the Hessian estimation (when ϵ = 0 ) acquired from the reward regressor, then the LOBIAS( h, (cid:101)
A ( s )) 2 becomes zero in Eq.
But when there is an error inthe Hessian estimation (when ϵ > 0 ), the KMIS estimator may have LOBIAS( h, (cid:101)
A ( s )) 2 up to
[Equation]
Convergence Speed Analysis We analyzed the convergence speed of KMIS applied kernel-relaxed
IS OPE estimator in Appendix A.7 when both positive and negative eigenvalues exist in the providedtrue Hessian H ( s ) , and given the optimal bandwidth h ∗ .
In this section we analyze the convergencespeed of KMIS estimator with (cid:101) H ( s ) in the same setting.
We derived the upper bound of LOBIAS( h, (cid:101)
A ( s )) 2 in Eq.
By plugging in the derived upperbound instead of LOBIAS( h, A ( s )) 2 in Eq.
(54), and also using the relation h ∗
=
O (cid:16)(cid:0) DN (cid:1) 1 D +4 (cid:17)from Eq.
(4), we get:
[Equation]
[Equation]
where, N ≫ D
When the estimated Hessian (cid:101) H ( s ) has no error (when ϵ = 0 ), the convergence speed derivedin Eq.
(80) matches the convergence speed of KMIS estimator with a true Hessian derived in Appendix A.7.
However, when there is an error in the estimated Hessian (when ϵ > 0 ), KMISestimator with the estimated Hessian has a slower convergence rate to a true policy value than the
KMIS estimator with a true Hessian by the rate of O ( D 12 d max ) (MSE of a KMIS estimator with theestimated Hessian converges to zero slower by the rate of O ( Dd 2 max ) compared to the KMIS with atrue Hessian).