Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider local kernel metric learning for off-policy policy evaluation (OPE)of deterministic policies in contextual bandits with continuous action spaces.
Ourwork is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosageand duration in medicine.
Although importance sampling (IS) provides a basicprinciple for OPE, it is ill-posed for the deterministic target policy with continuous actions.
Our main idea is to relax the target policy and pose the problem askernel-based estimation, while learning the kernel metric in order to minimize theoverall mean square error (MSE).
We present an analytic solution for the optimalmetric, based on the analysis of bias and variance.
Whereas prior work has beenlimited to scalar action spaces or kernel bandwidth optimization, our work takesa step further being capable of vector action spaces and metric optimization.
Weshow that our estimator is consistent, and significantly reduces MSE compared tobaseline OPE methods through experiments on various domains.
Introduction
In order to deploy a contextual bandit policy to an actual environment, such as personalized pricing[1],treatments [2], recommendation [3], advertisements [4], the performance of the policy being deployedshould be evaluated to decide whether the trained policy is suitable for the deployment.
This isbecause the interaction of the policy with the environment could be costly and dangerous.
Forexample, using an erroneous medical treatment policy to prescribe drugs for patients could resultin dire consequences.
Therefore, emerges a necessity for an algorithm that can evaluate a policy’sperformance without having it interact with the environment in an on-line manner.
Such algorithmis called the "off-policy policy evaluation" (OPE) algorithms.
OPE algorithms for contextualbandits evaluate a target policy by estimating the expected reward the it would get given an offlinedata sampled from a data collecting (behavior) policy.
Previous work on OPE for contextual bandits has mainly focused on environments with finite actions[4–10].
The work can be largely divided into three approaches [6, 9, 11].
The first approach is directmethod (DM) which learns the environment model to evaluate a target policy.
DM learns to modelthe environment for estimating the expected reward, which is used for policy evaluation.
Since theenvironment model is learned with a function approximation, the estimator is biased.
Although itis biased, it is known to have a low variance [9].
The second approach is importance sampling (IS)which corrects the data distribution induced by a behavior policy to that of a target policy [12], anduses the corrected distribution to approximate the expected reward of the target policy.
IS estimateis unbiased when behavior and target policies are given.
However, IS estimates can have a largevariance when there is large mismatch between behavior and target policy distributions [9].
Thelast approach is doubly robust (DR), which uses DM to reduce the variance of IS while keeping theunbiasedness of an IS estimate [6].
However, there is also a need for OPE algorithms for contextual bandits with continuous actions, as itcan be used to evaluate the policies for personalized continuous decision making, such as evaluating acontinuous treatment prescription policies.
And due to practical constraints, we also need
OPE algorithms for deterministic contextual bandit policies.
For example, treatment prescriptionpolicies cannot stochastically prescribe drugs to patients.
Unfortunately, most studies for OPE withfinite actions cannot be easily extended to evaluate deterministic contextual bandit policies withcontinuous actions.
This is because IS weights used for both IS and DR estimates are almost surelyzero for a deterministic target policies in continuous action space.
Thus, the work for evaluatingdeterministic contextual bandit policies continuous action space focus on this problem [2, 14, 15].
The work on OPE of deterministic contextual bandits with continuous actions can be largely dividedinto work on IS [2, 15] and doubly robust estimators (DR) [13, 14].
Both methods use IS and eliminatethe problem of almost surely having zero IS estimate when given a deterministic target policy and astochastic behavior policy in continuous action space.
Some work relaxes the deterministic policyto a kernel [2, 13, 15], and the other work discretizes the action space [14].
These methods reducethe mismatch between behavior and target policies by relaxing the Dirac-delta target policies (adeterministic target policy density can be seen as a Dirac-delta function [2]) to a kernel function andthus reduce the variance with some addition of bias.
The kernel-based IS methods for OPE use kernels to measure the similarities between the target andbehavior actions.
Furthermore, the similarities are measured according to the Euclidean distancesbetween the actions [2, 13, 15].
However, the Euclidean distance is inadequate for measuring thesimilarities in the case of kernel-based IS for OPE.
In the OPE case, assigning high similarity(or, assigning high importance weight) to the actions having similar rewards will induce less biasfor the OPE estimate when relaxing the Dirac-delta target policy density to a kernel.
In kernellearning literature, there are studies that reduce the bias of kernel regression [16] and nearest neighborclassification [17] by learning kernel metrics.
Similar to these methods, we focus on learning kernelmetric that reduces estimation bias (definition in Section 3, Eq.
(7) ) of kernel-based IS methods.
We propose local kernel metric learning for IS (KMIS) for OPE estimation of deterministic contextualbandit policies with multiple continuous actions.
Our proposed method learns a Mahalanobis distancemetric [16–19] locally at each state that shortens the distance between behavior and target actionshaving similar rewards.
The metric applied kernels measure similarities between actions according tothe Mahalanobis distances induced by the metric.
In our work, we first analytically show that the biasof a kernel-based IS estimator is a dominant factor given the optimal bandwidth that minimizes theleading MSE without a metric.
Then we derive the kernel metric that minimizes the leading orderbias given the optimal bandwidth.
In the experiments section, we show that when the proposed kernelmetric learning is used with the previous works in kernel-based IS that learn bandwidths, MSEs ofthe estimations are significantly reduced.
We report empirical results in various synthetic domainsand in real-world data.
Preliminaries2.1 Problem Setting
In this work, we focus on off-policy policy evaluation (OPE) of a deterministic target policy inan environment with the action space
A ⊂ R D , state space S ⊂ R D S , the distribution of thereward p ( r | s, a ) , the state distribution p ( s ) .
The target policy which selects an action given a stateπ ( s ) : S
→ A can be regarded to have a Dirac-delta distribution π ( a | s )
=
δ ( π ( s ) − a ) , where theprobability density value goes to infinity at the selected action given a state, and the integration ofthe function is one.
The offline data D = { s i , a i , r i } Ni =1 used for the evaluation of the deterministictarget policy π is sampled with a known stochastic behavior policy π b ( a | s ) : S → ∆( A ) .
We assumethat the support of the behavior policy π b ( a | s ) contains the actions selected by the target policy π ( s ) .
One of the methods to evaluate the target policy value ρ π = E s ∼ ( s ) ,a ∼ π ( a | s ) ,r ∼ p ( r | s,a ) [ r ] with the 85 offline data D sampled from π b is to do importance sampling (IS) estimation as in Eq.
In ourcase, the IS ratios correct the action sampling distribution π b ( a | s ) to π ( a | s ) .
But since π ( a | s ) is a
Dirac-delta function, the probability density function (PDF) value at the action in the offline data Dwhich is sampled with π b ( a | s ) is almost surely 0.
Existing works deal with the problem by relaxingthe target policy π , which is a Dirac-delta function, to an isotropic kernel K and enable IS estimationof ˆ ρ K as in Eq.
ρ π = E s ∼ p ( s ) ,a ∼ π b ( a | s ) ,r ∼ p ( r | s,a ) (cid:20) π ( a | s ) π b ( a | s ) r (cid:21) , (1) (cid:18) where, π ( a | s ) = (cid:26) ∞ , π ( s ) =
a 0 , π ( s ) ̸
= a , (cid:90)
[Equation]
2.2 Bandwidth Learning for the Isotropic Kernel Relaxed IS Estimator
As the Dirac-delta function can be regarded as a kernel having its bandwidth approaching zero, therelaxation of the Dirac-delta function to a kernel can be seen as increasing its bandwidth h .
By therelaxation, the bias (definition in Section 3, Eq.
(7)) of the kernel-relaxed IS estimate increases whileits variance is reduced.
However, as the bias and the variance compose the MSE of the estimate,the bandwidth that best balances between them should be selected.
And previous works focus onchoosing the bandwidth of the relaxed kernel [2, 15].
In the works of Su et al.
[15], they proposed abandwidth selection method that selects a bandwidth among the list of given bandwidths by using
Lepski’s principle for bandwidth selection in nonparametric statistics [20–22].
Kallus and Zhou [2]derived the leading order MSE (LOMSE) in terms of bandwidth h which is denoted as f in Eq.
The LOMSE is composed of constants C b and C v that are related to bias and variance, respectively :
[Equation]
[Equation]
r | s, a ]) | a = π [Equation] s
[Equation]
(cid:19)
[Equation] , C v := R ( K
[Equation] E s [Equation]
) (cid:20) E [ r 2 | s, a = π [Equation]
s )] π b ( a = π [Equation] s ) | s ) (cid:21) , where, ∇ 2 a denotes the Laplacian operator w.r.t.
action a , the roughness of the kernel is R ( K ) :
= 103 (cid:82)
K ( u ) 2 d u .
For simplicity, we assumed a Gaussian kernel and used the property (cid:82) uu T K ( u ) d u = I .
The kernel used for the derivation satisfies (cid:82) K ( u ) d u = 1 and (cid:82) u K ( u ) d u = 0 .
With the derived
LOMSE in Eq.
(3), Kallus and Zhou [2] computed the bandwidth that minimizes the LOMSE as in
Eq.
In the process, they assumed that h → 0 and
[Equation]
[Equation]
2.3 Mahalanobis Distance Metric
Although relaxing the Dirac-delta target policy π to a kernel K makes importance sampling possiblein our problem setting, the kernel is isotropic, and similarity of target and behavior actions in alldirections are treated equally in weighing the reward in Eq.
But the similarities between twoactions in some directions can be ignored relative to the other directions.
Figure 1 illustrates the casewhere a Mahalanobis metric is locally learned at the target action for bias reduction in making akernel-relaxed IS estimate.
The bias of the estimate is reduced by altering the shape of the kernel toproduce a higher similarity measure on a behavior action that has a similar reward to the target action.
The Mahalanobis distance metric can shorten or lengthen distances in certain directions.
We definethe Mahalanobis distance between two vectors a i ∈ R D and a j ∈ R D with the metric A as Eq.
Applying the Mahalanobis distance metric A ( = LL T ) to a kernel function can be seen as linearlytransforming the kernel inputs with the transformation matrix
L ( L T a = z ).
[Equation]
[Equation]
Local Metric Learning for Kernel-Relaxed IS
To minimize the LOMSE of the kernel-relaxed IS estimate by learning a metric, we first analyze the
LOMSE when the optimal bandwidth derived by Kallus and Zhou [2] is applied to the kernel.
The 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕2𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 ; 𝑨𝑨 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕𝑇𝑇 𝐴𝐴𝒂𝒂−𝒂𝒂 𝒕𝒕 2 (a) K ( a , a t ) = 12 π exp (cid:16) − ( a − a t ) 2(cid:17) 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝒂𝒂𝒂𝒂𝒂𝒂 𝒕𝒕 𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕2𝐾𝐾𝒂𝒂 , 𝒂𝒂 𝒕𝒕 ; 𝑨𝑨 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕𝑇𝑇 𝐴𝐴𝒂𝒂−𝒂𝒂 𝒕𝒕 2 (b) K ( a , a t ; A ) = 12 π exp (cid:16) − ( a − a t ) T A ( a − a t )(cid:17) optimal bandwidth presented in Eq.
(4) balances between the leading bias and the leading variancein first and the second term of Eq.
(3), respectively.
However the balancing is conducted with anisotropic kernel and estimation bias that could have been avoided with proper use of kernel metriccould be added in exchange of reduction in estimation variance as we see in Figure 1 which canworsen in a high dimensional space.
To analytically show this, we adopt Proposition 4 in the workof Noh et al.
[16] and show that the C b contained in the leading order bias becomes a dominant termin the LOMSE of kernel-relaxed IS OPE given an optimal bandwidth in Eq.
(4) and as the actiondimension increases.
Proposition 1.
For action dimension D ≫ 4 and given h ∗ (Eq.
(4) ), C b contained in the leading biasin Eq.
(3) is the dominant term of LOMSE( h ∗ , N, D )
(Eq.
By plugging in Eq.
(4) to Eq.
(3) , weget the following relation for an environment with high action dimension:
[Equation]
(The proof is provided in Appendix A.3.)
The proposition 1 implies that in high action dimensional environment, MSE can be significantlyreduced by the bias reduction.
Therefore, we aim to reduce the leading estimation bias in Eq.
(3)regardless of the bandwidth used in kernel-relaxed IS by using the Mahalanobis distance metric thatshortens the distance between target and behavior actions in the direction where their correspondingrewards are similar.
As mentioned in Section 2.3, applying a metric ( A = LL T ) to a kernel is equivalent to linearlytransforming input vectors of the kernel.
Therefore, the leading order bias with a metric should beanalyzed with linearly transformed actions ( z = L T a ).
By applying Taylor expansion on p ( r | s, a )and using change of variable, we get: (derivation in A.5.).
[Equation]
where, H is the Hessian operator and the expected reward is r ( s, a ) :
=
E [ r | s, a ] .
In the derivation,we assumed using a Gaussian kernel and used the property (cid:82) uu T K ( u ) d u = I .
From Eq.
(9), theleading order bias in Eq.
(9) which compose LOMSE can be minimized with A ( s ) .
In the following
Theorem 1, we introduce the optimal metric matrix A opt ( s ) that minimizes the squared leading orderbias by using the semi-definite programming solution presented in the work of Noh et al.
[17] on theleading order bias where a kernel metric is applied in Eq.
Theorem 1.
Under the assumption that the p ( r | s, a ) is twice differentiable w.r.t.
an action a ,mahalanobis metric can be applied to reduce the absolute value of the leading order bias in Eq.
(3) .when Λ + ( s ) and Λ − ( s ) are diagonal matrices of positive and negative eigenvalues of the Hessian
H a E [ r | s , a ] | a = π ( s ) , U + ( s ) and U − ( s ) are matrices of eigenvectors corresponding to Λ + ( s ) andΛ − ( s ) , respectively.
And d + ( s ) and d − ( s ) are the numbers of positive and negative eigenvalues ofthe Hessian, then the metric A opt ( s ) that minimizes the squared leading order bias is proportional to:
[Equation]
The derivation of the metric involves solving a Lagrangian equation to minimize the absolute value ofthe trace term in Eq.
(9) with constraints on A opt ( s ) (Detailed proof is provided in Appendix A.6.)
The derived metric matrix proportional to A opt ( s ) can be divided by its determinant to be A opt ( s ) .
And in the special case where the Hessian matrix contains both positive and negative eigenvalues, the metric reduces the leading order bias in Eq.
(3) to zero (see Appendix A.6).
A opt ( s ) islocally computed at a state s and target action π ( s ) as it is derived from the Hessian at that point
H a r ( s, a ) | a =
π ( s ) .
The eigenvalues and eigenvectors of the Hessian matrix represent the principaldirection of the expected reward curvatures and principal curvatures in the corresponding directions.
The principal directions of curvatures are the directions where the expected reward has the largestcurvatures.
When the metric learned from the Hessian is applied to the kernel, it can be seen as using
Mahalanobis distance instead of Euclidean distance, as shown in Figure 1 and in Eq.
[Equation]
where, u + ,i and u − ,i are positive and negative eigenvectors, respectively, and their correspondingeigenvalues are denoted as λ
+ ,i and λ − ,i , respectively.
The Mahalanobis distance is formed toincrease the distance between the behavior action a
and the target action π ( s ) on the expectedreward’s principal directions of curvatures weighted by (cid:112) d
+ λ + ,i or (cid:112) − d − λ − ,i .
To sum up, thederived A opt ( s ) is locally learned at state s and corresponding target action π ( s ) in a way thatincreases the Mahalanobis distance between a behavior action a and a target action π ( s ) on thedirections where the expected reward’s rate of change is the steepest.
We showed that by applying our metric to the kernel-relaxed IS estimate, the bias of the IS estimatecan be reduced, and therefore, the MSE is reduced.
Next, we see how the metric learning affects theconvergence speed of our estimator w.r.t.
datasize N and the action dimension D .
In the following
Theorem 2, we show that our estimator is consistent and converges faster than the kernel-relaxed ISestimator without metric learning.
Theorem 2.
KMIS with the optimal bandwidth presented in Eq.
(4) is a consistent estimator in whichconvergence rate is equal or faster than the kernel-relaxed IS estimate with the optimal bandwidth.
The convergence speed of our metric applied kernel-relaxed estimate the true policy value is faster asthe action dimension increases up to O
( D − 12 ) compared to the one without the metric.
The proof is made by analyzing the complexity of the sample size N and the action dimension D ofthe LOMSE in Eq.
(3) with and without the proposed metric.
(The proof is provided in Appendix A.7.)
Since the derived optimal bandwidth A opt ( s ) in Theorem 1 assumes that p ( r | s, a ) is twice differentiable, we add in a regularizers β ( s ) and γ ( s ) as in Eq.
(12) so that the metric applied kernel can fallback to an isotropic kernel in the case where the assumption is not met in practice.
The regularizersare also adjusted to satisfy | A | = 1 (See Appendix A.8 for more detail).
For the simplicity of thealgorithm, we did not discard the components along the eigenvectors with zero eigenvalues ( U 0 )but let the metric have a longer relative bandwidth on the directions compared to the others viaregularizers.
[Equation]
For the implementation of the proposed local metric A MIS for kernel-relaxed IS estimation, we fit aneural network based reward regressor with a Gaussian output layer.
The regressor learns the meanand the variance of the reward distribution from data which can be used for estimating the Hessianrequired for our algorithm.
The reward regressor may have some error regarding the estimationof the Hessian and could affect the estimation error of our algorithm.
We empirically test if ourmetric is still accurate in case where noise in the reward data increases and degrades the estimationperformance of DM in Appendix A.4.
The reward regressor also can be used to estimate the optimalbandwidth.
After obtaining the Hessian from the regressor, we linearly transform the kernel inputswith the transformation matrix L from the proposed metric A MIS = LL T as in Eq.
We providepseudo code in Algorithm 1 for a better understanding of the readers.
[Equation]
Algorithm
[Equation]
[Table]
Experiments
The experiments show that KMIS can be used with the existing works on bandwidth learning, andtheir MSEs are reduced by our metric.
Since our algorithm learns the metric for kernel-relaxed IS, we compare our algorithm with the other kernel based IS methods.
The baselines include adiscretized OPE estimator that discretizes the action space by given number of bins for each actiondimension for an IS estimation [2] (for details, see Appendix A.9).
The work of Kallus and Zhou [2]and SLOPE [15] are also compared, where both learn bandwidths for kernel-relaxed IS.
For bothbaselines, we use the code provided by the authors.
For Kallus and Zhou’s estimator, we use a rewardregressor for for bandwidth learning instead of making kernel density estimation (KDE) as in theirwork since using KDE to estimate the bandwidth is computationally expensive.
KMIS uses boththe regressor and the learned bandwidth by the baselines for OPE.
Kallus and Zhou’s estimator andour KMIS share the same neural network reward regressor composed of two hidden layers of 128neurons for learning bandwidth and metric, respectively.
For the activation function of the neuralnetwork, we used tanh.
For more details and codes for our algorithm, look at Appendix A.9.
For
SLOPE, we select a bandwidth from geometrically spaced bandwidths (cid:8) 2 − i : i ∈ [1 , 7] , i ∈ Z (cid:9) asin their work [15] for the synthetic domain.
For the Warfarin data, we selected a bandwidth from 219 (cid:8) 2 − i : i ∈ [ − 2 , 7] , i ∈
Z (cid:9) .
We do not correct the boundary bias as in the works of Kallus and Zhou[2], Su et al.
[15] since we regard the bias induced by the boundaries are negligible compared to thebias reduced by our locally learned metric.
We test our algorithm and baselines on various syntheticdomains and the Warfarin dosing dataset provided by International Warfarin Pharmacogenetics.
4.1 Synthetic Data
In the experiment with the synthetic data, we show that our metric can be applied to offline datasampled from environments with various reward functions.
Furthermore, we show that learninglocal metrics can reduce MSEs of existing kernel-relaxed IS estimators.
To show this, three kinds ofsynthetic test domains are prepared.
We first test on the synthetic domain with a quadratic reward function w.r.t.
actions.
As our metricis computed from the Hessian H a r ( s , a ) | a
= π ( s ) , we first test if the metric can be learned fromthe neural network reward regressor.
If the metric is learned successfully, the metric would beformed in an ellipsoidal shape and would be the same for every state and target action as the Hessianof a quadratic function is a constant.
In the experiment, although our estimator uses the Hessianinformation from the neural network reward regressor, our metric learning applied kernel-relaxed ISestimator outperforms the reward regressor, and other baselines.
After checking if the metric can be properly learned from the reward regressor, we see if the metriccan still be learned in an environment where negative absolute error is given as a reward.
Since the
Hessian of the reward function is zero at the reward, it is reasonable to think that the metric learningwould fail in such a case.
However, since we are dealing with finite samples and because we areusing a neural network reward regressor, we conjecture that some form of convex shaped rewardestimate is learned by the regressor and makes metric learning possible.
We also let one of the actiondimensions be unrelated to the reward.
If metric learning is successful, the metric will be learned tohave a relatively larger bandwidth to the direction of the action dimension.
Lastly, we test KMIS on an exponential reward whose shape is similar to the rewards used inmulti-goal environments, which was introduced in the works of Haarnoja et al.
Since the twosynthetic test domains previously introduced have true Hessian H a r ( s , a ) | a
= π ( s ) of constant andzero, respectively, we test our algorithm on an exponential reward where it has Talyor expansionterms larger than the second order.
[Figure]
Means and standard errors of squared errors obtained from 80 trials are drawn in the figures.
The second row also marks the results of the first row when the datasize used in the estimation is 20k.
The last row shows the learned metric at each test domain.
All synthetic test domains use actions and states that are a ∈ R 2 , and s ∈ R 2 .
For the test domainwith the quadratic reward, the states are uniformly sampled in the range of [ − 1 , 1] , behavior policy isthe Gaussian π b ( a | s ) =
N ( s + [ 0 . 2 0 .
5 2 I ) , target policy is π ( s )
= s , and the reward is a quadraticfunction w.r.t.
actions.
For the details, look at Appendix A.9.
The test environment with the negativeabsolute error is prepared by having both states and actions uniformly sampled in the range of [ − 1 , 1] ,the target policy is designed to be π ( s ) = 0 .
5 s , and the reward is r ( s , a ) =
−| 0 .
5 s 1 − a 1 | , where s 1and a 1 are the first dimensions of the state and action vectors.
The test domain with the exponential reward is made with states and actions uniformly sampled from the range of [ − 1 , 1] , and target policyπ ( s )
= s + [ 0 . 50 ] .
For the details of the exponential reward, look at Appendix A.9.
The experimental results in Figure 2 show that metric learning reduces the MSEs in the kernel relaxed
IS estimates and also outperforms the reward regressor, which is used for computing the Hessianrequired for learning the metric.
The results in the first row of Figure 2 show that the MSE of SLOPEand Kallus et al.
’s estimator are reduced by our metric.
To see whether the learning metrics alwaysreduce MSE in the kernel-relaxed IS estimator, we test KMIS on given bandwidths.
Seeing the resultsin the second row of Figure 2, we can see that most of the time, the MSE is reduced with metriclearning.
For the discretized OPE estimator, it performs worst in the quadratic reward domain.
Butfor the other domains, it performs better than the other baselines.
The absolute error domain and themulti-goal domain have uniform action distributions and no noise in the rewards.
In this condition,because we apply self-normalization to all estimators, the discretized OPE would take an averageof the behavior rewards which corresponding behavior actions are in the same discretized bins withthe target actions which can be accurate in this condition.
In the second row of Figure 2, the dottedvertical lines present the average of the bandwidths learned by Kallus et al.
(blue) or SLOPE (red),and the markers show the average of the learned bandwidths and the MSE by each estimator when20k data points are used to obtain the OPE result.
Since the learned bandwidth can vary for eachrun, the markers may not exactly lie on the black lines denoting the fixed bandwidth’s results.
To seehow metric is learned and also to see how it helps reduce the bias of the kernel-relaxed IS estimate,we show the reward landscape at s =
[ 00 ] and the learned metric in black crosses in the last row.
Bylooking at the metric learned for the quadratic reward, we see that our metric is properly learned withthe Hessian H a r ( s , a ) | a = π ( s ) provided by the reward regressor as the learned metrics are similarfor most actions.
For the negative absolute error test domain with the dummy action dimension, wesee that the metric is learned to ignore the dummy action dimension a 2 by having longer relativebandwidth to the direction of a 2 while having shorter relative bandwidth in the direction of a 1 whichis used for computing the reward.
This result empirically verifies our conjecture made earlier, whichsuggests that our metric learning algorithm can be generally applied to the dataset where the expectedreward or p ( r | s, a ) is not twice differentiable.
The metric learned on the multi-goal reward madewith exponents verifies that our metric indeed learns the local metric for each action according to thereward landscape.
[Figure]
Means and standard errors of squared errors obtained from 100 trials are drawn in the figures.
Empirical bias and variance of the estimates are also drawn with the same estimates.
In Proposition 1, we discussed that the leading order bias becomes a dominant term inthe LOMSE of isotropic kernel smoothed ISestimator.
To validate the Proposition, weplot empirical squared bias and variance ofthe Kallus&Zhou and Kallus&Zhou+KMIS estimates which together compose MSE on the absolute error domain with varying action dimensions in Figure .5.
Results show that empiricalestimation bias increases as the action dimension increases and both algorithms suffer fromhigh bias.
But our method reduces bias withthe learned metric and shows best performance.
While Kallus&Zhou+KMIS has reduced empirical bias compared to Kallus&Zhou, its varianceis increased more than that of Kallus&Zhou.
Although the leading order variance is unaffectedby the metric, the ignored components of the estimation variance could be affected by the metricand resulted in increase in the variance.
4.2 Warfarin Dosing Data
To test our algorithm on a more realistic dataset, we test our algorithm on the Warfarin dosing dataset[23].
Warfarin is a treatment that is commonly used for preventing blood clots.
The dataset containsinformation on the patients who got treated, therapeutic doses, and the resulting outcomes.
Theoutcomes of the treatments are reported in the international normalized ratio (INR), which measureshow quickly blood clots.
We use a similar experimental setting used by Kallus and Zhou [2].
As
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Sample Size (10 3 ) 0.
l o g ( M S E )
Kallus & Zhou Kallus & Zhou + KMIS (ours)
SLOPESLOPE + KMIS (ours)
DM (a)    2 log(h)     l o g ( M S E )
Bandwidth selected by SLOPE
Bandwidth selected by Kallus & Zhou
No metric KMIS (ours)
Kallus & Zhou Kallus & Zhou + KMIS (ours)
SLOPESLOPE + KMIS (ours)
[Figure]
in their work, we use the 81 features of the patient information selected by Kallus and Zhou [2] asstates s .
For the behavior action vectors, the first dimension is sampled from the normal distribution
N ( µ ∗ + σ ∗ √ 0 .
5 z BMI , ( σ ∗ √ 0 .
5) 2 ) truncated by the minimum and maximum therapeutic doses a ∗ minand a ∗ max in the data.
z BMI is the z score of patients’ BMIs, µ ∗ and σ ∗ are the mean and the standarddeviation of the therapeutic dosages, respectively.
To test our algorithm we add a dummy actiondimension sampled from a uniform distribution a 2 ∼ unif[ a ∗ min , a ∗ max ] .
For the reward, since Kallusand Zhou [2] reported that the INR in the dataset is inadequate for testing OPE algorithms, we usetheir cost function as a negative reward function r = − max( | a 1 − a ∗ | − 0 .
1 a ∗ , 0) .
The rewardfunction is only dependent on the first action dimension.
The target policy is π ( s ) =
[ s BMI 0 ] .
Experimental results in Figure 4a show that our metric reduces the MSE of the kernel-relaxed ISestimate in Warfarin data that is more realistic than that of the synthetic domain.
In the figure, we seethat even though the MSE of the reward regressor is high, SLOPE + KMIS is able to learn the metricthat reduces the MSE of SLOPE.
Kallus and Zhou + KMIS also reduce the MSE of the estimatorproposed by Kallus and Zhou.
However, since the bandwidth learned by Kallus and Zhou’s model istoo big, as shown in Figure 4 there are limitations in the reduction of its MSE via metric.
Discussion and Conclusion
We present local metric learning for kernel-relaxed IS (KMIS).
KMIS locally learns metrics at agiven state, and the metric reduces the bias of the kernel-relaxed IS estimate.
This is done by learningthe metric to assign high similarity to the behavior action that has similar rewards to that of the targetaction for IS.
Specifically, given a state, our proposed method learns the Mahalanobis distance metricthat shortens the distance between a pair of behavior and target actions when their correspondingrewards are similar.
Our experimental results suggest that applying KMIS to a kernel-relaxed ISestimation reduces MSE in most cases regardless of given bandwidths.
Moreover, we also showedthat KMIS could be used for offline data sampled from environments with various reward functions.
And most importantly, KMIS can be used with other kernel-based IS estimators that learn bandwidths.
The limitation of our work is that KMIS assumes that a optimal bandwidth is provided.
Becauseof this, when suboptimal bandwidth is given, our algorithm may not fully function as intended.
Another limitation is that our algorithm is dependent on the Hessian estimated by the reward regressorwhich can be erroneous.
Although our work has some limitations, we have shown in the empiricalstudies that KMIS reduces MSEs of kernel-relaxed IS estimates for almost every bandwidths.
Ourwork also opens a new research direction for the kernel-based methods.
Our experimental results in
Figure 2, especially the results from quadratic reward test domain, suggests that optimal bandwidthsare different for a kernel with and without a metric.
This result opens an interesting future studydirection of jointly optimizing the bandwidth and the metric.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default to [Yes] , [No] or [N/A].
You are strongly encouraged to include a justification to your answer , either by referencing theappropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code
and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]