Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions
Abstract
We consider local kernel metric learning for off-policy evaluation (OPE) of deterministic policies in contextual bandits with continuous action spaces.
Our work is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosage and duration in medicine.
Although importance sampling (IS) provides a basic principle for
OPE, it is ill-posed for the deterministic target policy with continuous actions.
Our main idea is to relax the target policy and pose the problem as kernel-based estimation, where we learn the kernel metric in order to minimize the overall mean squared error (MSE).
We present an analytic solution for the optimal metric, based on the analysis of bias and variance.
Whereas prior work has been limited to scalar action spaces or kernel bandwidth optimization, our work takes a step further being capable of vector action spaces and metric optimization.
We show that our estimator is consistent, and significantly reduces the MSE compared to baseline
OPE methods through experiments on various domains.
Introduction
In order to deploy a contextual bandit policy to a real-world environment, such as personalized pricing[1], treatments [2], recommendation [3], advertisements [4] the performance of the policy should be evaluated prior to the deployment to decide whether the trained policy is suitable for deployment.
This is because the interaction of the policy with the environment could be costly and/or dangerous.
For example, using an erroneous medical treatment policy to prescribe drugs for patients could result in dire consequences.
There then emerges a necessity for an algorithm that can evaluate a policy’s performance without having it interact with the environment in an online manner.
Such algorithms are called “off-policy evaluation” (OPE) algorithms.
OPE algorithms for contextual bandits evaluate a target policy by estimating its expected reward from the data sampled by a behavior policy and without the target policy interacting with the environment.
Previous works on OPE for contextual bandits have mainly focused on environments with finite actions [4, 6–10].
The works can be largely divided into three approaches [6, 9, 11].
The first approach is the direct method (DM) which learns an environment model for policy evaluation.
DM is known to have low variance [9].
However, since the environment model is learned with function approximation, the estimator is biased.
The second approach is importance sampling (IS) which corrects the data distribution induced by a behavior policy to that of a target policy [12], and uses the corrected distribution to approximate the expected reward of the target policy.
IS estimates are unbiased when the behavior and target policies are given.
However, IS estimates can have large variance when there is a large mismatch between the behavior and target policy distributions [9].
The last approach is doubly robust (DR), which uses DM to reduce the variance of IS while keeping the unbiasedness of an IS estimate [6].
Although there are existing works on IS and DR that can be applied to continuous action spaces [13–16], most of them cannot be easily extended to evaluate deterministic contextual bandit policies with continuous actions.
This is because IS weights used for both IS and DR estimators are almost surely zero for deterministic target policies in continuous action spaces [2].
However, in practice, such
OPE algorithms are needed.
For example, treatment prescription policies should not stochastically prescribe drugs to patients.
To meet the needs, there are works for evaluating deterministic contextual bandit policies with continuous actions [2, 5, 17].
These works focus on measuring the similarity between behavior and target actions for assigning IS ratios.
However, these works either assume a single-dimensional action space [17] which cannot be straightforwardly extended to multiple action dimensions, or, use
Euclidean distances for the similarity measures [2, 5].
In general, similarity measures should be learned locally at a state to weigh differences between behavior and target actions in each action dimension differently.
For example, in the case of multi-drug prescription, the synergies and side effects of the prescribed drugs are often very complex.
Moreover, the different kinds of drugs are likely to have different degrees of effect from person to person [18, 19].
To this end, we propose local kernel metric learning for IS (KMIS) OPE estimation of deterministic contextual bandit policies with multi-dimensional continuous action spaces.
Our proposed method learns a Mahalanobis distance metric [20–23] locally at each state that lengthens or shortens the distance between behavior and target actions to reduce the MSE.
The metric-applied kernels measure the similarities between actions according to the Mahalanobis distances induced by the metric.
In our work, we first analytically show that the leading-order bias [2] of a kernel-based IS estimator becomes a dominant factor in the leading-order MSE [2] as the action dimension increases given the optimal bandwidth [2] that minimizes the leading-order MSE without a metric.
Then we derive the kernel metric that minimizes the upper bound of the leading-order bias, which is bandwidth-agnostic.
Our analysis shows that the convergence speed of a kernel-based IS OPE estimator can be improved with the application of the KMIS metrics.
In the experiments, we demonstrate that MSEs of kernel-based IS estimators are significantly reduced when combined with the proposed kernel metric learning.
We report empirical results in various synthetic domains as well as a real-world dataset.
Related Work
The works on OPE of deterministic contextual bandits with continuous actions can be largely divided into importance sampling (IS) [2, 5] and doubly robust estimators (DR) [17, 24].
Both methods eliminate the problem of almost surely having zero IS estimates when given a deterministic target policy and a stochastic behavior policy in a continuous action space in two ways.
Most of the works relax the deterministic target policy, which can be seen as a Dirac delta function [2], to a kernel [2, 5, 24], and the other work discretizes the action space [17].
Among these, kernel-based IS methods use a kernel to measure the similarities between the target and behavior actions and focus on learning the bandwidth of the kernel [2, 5, 24].
Su et al.
[5] proposed the bandwidth selection algorithm that uses the Lepski’s principle for bandwidth selection in the study of nonparametric statistics [25].
Kallus and Zhou [2] derived the leading-order MSE of a kernel-based IS OPE estimator and derived the optimal bandwidth that minimizes the leading-order
MSE.
One of the limitations of the existing kernel-based IS methods [2, 5, 24] is that these methods use Euclidean distances for measuring the similarities between behavior and target actions.
The
Euclidean distance is inadequate for measuring the similarities as assigning a high similarity measure to the actions having similar rewards will induce less bias (bias definition in Section 3.2).
The other limitation is that these methods use one global bandwidth for the whole action space [17].
Since the second-order derivative of the expected reward w.r.t.
an action is related to the leading-order bias derived by Kallus and Zhou [2], the optimal bandwidth that balances the bias-variance trade-off may vary depending on actions.
As for discretization methods, Cai et al.
[17] proposed deep jump learning (DJL)
[17] that avoids the limitation of kernel-based methods due to using one global bandwidth by adaptively discretizing one-dimensional continuous action spaces.
The action space is discretized to have similar expected rewards for each discretized action interval given a state.
By using the discretized intervals for
DR estimation, DJL can estimate the policy value more accurately in comparison to kernel-based methods when the action space has second-order derivatives of the expected rewards which change significantly across the action space.
In such cases, kernel-based methods use the same bandwidth for all actions even though the optimal bandwidth varies depending on actions.
On the other hand,
DJL can discretize the action space into smaller intervals for the parts of the action space where the second-order derivative is high, and larger intervals when it is low.
However, their work focuses on domains with a single action dimension and cannot be easily extended to environments with multi-dimensional action spaces.
To tackle the limitation of kernel-based methods caused by using Euclidean distances, kernel metric learning can be applied to shrink or extend distances in the directions that reduce MSE.
Metric learning has been used for nearest neighbor classification [22, 26–29], and kernel regression [22, 23].
Among them, our work was inspired by the work of Noh et al.
that learns a metric for reducing the bias and MSE of kernel regression [23].
In their work, they made the assumption on the generative model of the data where the input and output are jointly Gaussian.
Under this assumption, they derived the Mahalanobis metric that reduces the bias and MSE of Nadaraya-Watson kernel regression.
We learn our metric in a similar fashion in the context of OPE in contextual bandits except that we do not have the generative assumption on the data as the assumption is unnatural in our problem setting.
Preliminaries 3.1 Problem Setting
In this work, we focus on OPE of a deterministic target policy in an environment with multidimensional continuous action space
A ⊂ R D A , state space S
⊂ R D S , reward r ∈ R sampled from the conditional distribution of the reward p ( r | s , a ) , state distribution p ( s ) .
The deterministic target policy π ( s ) : S
→ A can be regarded to have a Dirac delta distribution π ( a | s )
=
δ ( π ( s ) − a ) , where the probability density function (PDF) value is zero everywhere except at the selected target action given a state.
The offline dataset D = { ( s i , a i , r i ) }
Ni =1 used for the evaluation of the deterministic target policy π is sampled with a known stochastic behavior policy π b ( a | s ) : S → ∆( A ) .
We assume that the support of the behavior policy π b ( a | s ) contains the actions selected by the target policy π ( s ) .
The goal of OPE is to evaluate the target policy value
ρ π = E s ∼ p
( s ) , a ∼ π ( a | s ) ,r ∼ p ( r | s , a )
[ r ] using D and without interaction with the environment.
3.2 Bandwidth Learning for the Isotropic Kernel-Based IS Estimator
One of the methods to evaluate the target policy value with the offline data D sampled from π b is to perform IS estimation.
The IS ratios correct the action sampling distribution for the expectation from π b ( a | s ) to π ( a | s ) .
However, since the density of a deterministic target policy π ( a | s ) is a
Dirac delta function, its PDF value at the behavior action sampled from π b ( a | s ) is almost surely 0.
Existing works deal with the problem by relaxing the target policy π to an isotropic kernel K with a bandwidth h and computing the IS estimate of the policy value ˆ ρ K as in Eq.
[Equation]
As the Dirac delta function can be regarded as a kernel having its bandwidth h approaching zero, the relaxation of the Dirac delta function to a kernel can be seen as increasing its h .
By the relaxation, the bias of the kernel-based IS estimation Bias (cid:2) ˆ ρ K (cid:3) :
=
E s ∼
[Equation]
( s ) ,
[Table] [Table]
[Table]
increases while its variance is reduced.
As the bias and the variance compose the MSE of the estimate, the bandwidth that best balances between them and reduce the MSE should be selected.
Kallus and
Zhou [2] derived the leading-order MSE (LOMSE) in terms of bandwidth h , sample size N , and action dimension D A for bandwidth learning (Eq.
(2)) assuming that h → 0 and 1 Nh DA → 0 as N → ∞ (derivation in Appendix A.1).
They also derived the optimal bandwidth h ∗ that minimizes the LOMSE (derivation in Appendix A.2).
𝒂𝒂𝒂 𝒂𝒂𝒂𝒂 𝒂𝒂 𝒕𝒕 𝒂𝒂𝒂 𝒂𝒂𝒂𝒂 𝐾𝐾 𝒂𝒂 , 𝒂𝒂 𝒕𝒕 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕 2𝐾𝐾 𝒂𝒂 , 𝒂𝒂 𝒕𝒕 ; 𝑨𝑨 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕 𝑇𝑇 𝐴𝐴 𝒂𝒂−𝒂𝒂 𝒕𝒕𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝒂𝒂 𝒕𝒕
[Figure]
[Figure]
(cid:17) 𝒂𝒂𝒂 𝒂𝒂𝒂𝒂 𝒂𝒂 𝒕𝒕 𝒂𝒂𝒂 𝒂𝒂𝒂𝒂 𝐾𝐾 𝒂𝒂 , 𝒂𝒂 𝒕𝒕 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕 2𝐾𝐾 𝒂𝒂 , 𝒂𝒂 𝒕𝒕 ; 𝑨𝑨 = 1 2𝜋𝜋 exp − 𝒂𝒂−𝒂𝒂 𝒕𝒕 𝑇𝑇 𝐴𝐴 𝒂𝒂−𝒂𝒂 𝒕𝒕𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎
For camera ready 𝒂𝒂 𝒕𝒕
[Figure]
(cid:17)
Although behavior actions a ′ and a ′′ are away from target action a t ( = π ( s ) ) by an equal Euclidean distance, their corresponding rewards are different.
When an (a) isotropic Gaussian kernel is used, bias can come from a ′ since the similarity measures of a ′ , and a ′′ from a t are the same even though their rewards are different.
However, when the (b) metric is applied, bias is reduced as the similarity measure is higher on a ′′ that has a similar reward to a t compared to that of a ′ .
[Equation]
where the first term in Eq.
(2) is the squared leading-order bias and the second term is the leadingorder variance, C b and C v
are constants related to the leading-order bias and variance, respectively, the expected reward is r ( s , a ) :
=
E [ r | s , a ] , ∇ 2 a denotes the Laplacian operator w.r.t.
action a , the roughness of the kernel is R ( K ) :
= (cid:82) K ( u ) 2 d u .
For simplicity, we assumed a Gaussian kernel and used the property (cid:82) uu ⊤ K ( u ) d u = I .
The kernel used for the derivation satisfies (cid:82) K ( u ) d u = 1 and (cid:82) u K ( u ) d u = 0 .
In Section 4, we use the leading-order bias and variance [2] for the derivation of our proposed metric.
We also use the optimal bandwidth [2] for analyzing the properties of kernel-based IS estimator with and without a metric.
3.3 Mahalanobis Distance Metric
Relaxing the Dirac delta target policy π to a kernel K prevents the kernel-based IS estimator from estimating 0 a.s.
By using an isotropic kernel for the relaxation, the difference between the target and behavior actions in all directions are treated equally for measuring the similarities between the actions with a kernel in Eq.
However, to produce a more accurate OPE estimation, the difference between the two actions in some directions should be ignored relative to the others for measuring the similarity between the actions.
For this, the Mahalanobis distance can be used.
We define the
Mahalanobis distance between two D A -dimensional vectors a i ∈ R D A and a j ∈ R D A with the metric A ∈ R D
A × D A as in Eq.
Applying the Mahalanobis distance metric
A ( = LL ⊤ ) to a kernel function can be seen as linearly transforming the kernel inputs with the transformation matrix
L ( L ⊤ a = z ).
[Equation] [Equation]
Figure 1 illustrates a case where the Mahalanobis metric is locally learned at a given state for reducing the bias of a kernel-based IS estimate.
The bias of the estimate is reduced by altering the shape of the kernel to produce a higher similarity measure on a behavior action that has a similar reward to the target action.
Local Metric Learning for Kernel-Based IS 4.1 Local Metric Learning via LOMSE Reduction
To reduce the LOMSE (Eq.
(2)) of the kernel-based IS estimate by learning a metric, we first analyze the LOMSE when the optimal bandwidth (Eq.
(3)) is applied to the kernel.
In Figure 1 we illustrate how the bias of an IS estimation is induced due to kernel relaxation.
The bias can worsen in high dimensional spaces.
To analytically show this, we adapt Proposition 4 in the work of Noh et al.
[23] and show that the C b contained in the squared leading-order bias (Eq.
(2)) becomes a dominant term in the LOMSE of kernel-based IS OPE given an optimal bandwidth and as the action dimension increases.
Proposition 1.
(Adapted from Noh et al.
For a high dimensional action space D
A ≫ 4 , and given optimal bandwidth h ∗ (Eq.
(3) ), the squared leading-order bias dominates over the leading-order variance in LOMSE.
Furthermore, LOMSE( h ∗ , N, D A ) can be approximated by C b in Eq.
[Equation]
The proof is made by plugging in Eq.
(3) to Eq.
(2) and taking the limit D A → ∞ .
(See Appendix
B.1.)
Proposition 1 implies that in an environment with high dimensional action space, the MSE can be significantly reduced by the reduction of C b contained in the squared leading-order bias (Eq.
Therefore, we aim to reduce C b , or, reduce the leading bias in a bandwidth-agnostic manner with a
Mahalanobis distance metric that shortens the distance between the target and behavior actions in the direction where their corresponding rewards are similar.
As mentioned in Section 3.3, applying a state-dependent metric ( A ( s ) =
L ( s ) L ( s ) ⊤ ) to a kernel is equivalent to linearly transforming input vectors of the kernel.
Therefore, C b with a metric
A : S → R D
A × D A (i.e. C b,A ) should be analyzed with the linearly transformed actions L ( s ) ⊤ a (Derivation in Appendix A.3.), and we aim to find an optimal A that minimizes it:
[Equation]
where H is the Hessian operator.
In the derivation, we assume a Gaussian kernel and use the property (cid:82) uu ⊤ K ( u ) d u = I .
Still, optimizing the function A in Eq.
(6) itself is challenging since it requires considering the overall effect of each metric matrix
A ( s ) on the objective function.
Therefore, we instead consider minimizing the following upper bound, which allows us to compute the closed-form metric matrix for each state in a non-parametric way:
[Equation]
In the following Theorem 1, we introduce the optimal Mahalanobis metric matrix A ∗ ( s ) that minimizes U b,A .
Theorem 1.
(Adapted from Noh et al.
[22]) Assume that the p ( r | s , a ) is twice differentiable w.r.t.
an action a .
Let Λ + ( s ) and Λ − ( s ) be diagonal matrices of positive and negative eigenvalues of the Hessian H a E [ r | s , a ] | a =
π
( s ) , U + ( s ) and U − ( s ) be matrices of eigenvectors corresponding to Λ + ( s ) and Λ − ( s ) respectively, and d + ( s ) and d − ( s ) be the numbers of positive and negative eigenvalues of the Hessian.
Then the metric A ∗ ( s ) that minimizes U b,A ( s ) is:
[Equation]
where α ( s ) := | M ( s ) | − 1 / ( d + ( s )+ d − ( s )) .
The proof involves solving a Lagrangian equation to minimize the square of the trace term in Eq.
(7) with constraints on A ∗ ( s ) .
(See Appendix B.2.)
In the special case where the Hessian matrix contains both positive and negative eigenvalues for all states, the metric reduces the C b,A to zero.
(See
Appendix B.2.)
A ∗ ( s ) is locally computed at a state s and the corresponding target action π ( s ) as the optimal metric is derived from the Hessian at that point H a r ( s , a ) | a = π ( s ) .
When the optimal metric is applied to the kernel, it measures the similarity between a behavior and target action using the Mahalanobis distance instead of the Euclidean distance, as shown in Figure 1 and in Eq.
[Equation]
where λ + ,i ( s ) and λ − ,i ( s ) are positive and negative eigenvalues of the Hessian H a r ( s , a )
| a = π ( s ) , and their corresponding eigenvectors are denoted as u + ,i ( s ) and u − ,i ( s ) respectively.
The Mahalanobis distance is formed to weigh the difference between the behavior action a and the target action π ( s ) w.r.t.
the axes formed by the eigenvectors.
A ∗ ( s ) increases the distance between the actions in the direction of eigenvectors whose corresponding eigenvalues are large, and vice versa.
Now we analyze how the optimal metric affects the convergence rate of a kernel-based IS estimator w.r.t.
data size N and the action dimension D A similar to Theorem 3 in the work of Kallus and Zhou [2].
Theorem 2.
(Adapted from Kallus and Zhou [2]) Kernel-based IS estimator with the optimal metric
A ∗ ( s ) from Eq.
(8) and the optimal bandwidth h ∗ in Eq.
(3) is a consistent estimator in which convergence rate is equal or faster than the isotropic kernel-based IS estimator with h ∗ .
When the
Hessian H a r ( s , a ) |
a = π ( s ) has both positive and negative eigenvalues for all states, A ∗ ( s ) applied estimator converges to the true policy value faster than the one without the metric by the rate of
O ( D − 12 A ) as the action dimension increases.
The proof is made by analyzing the complexity of the sample size N and the action dimension
D A of the LOMSE with and without the proposed metric.
(See Appendix B.3.)
4.2 Practical Algorithm
To avoid the degenerate case of having zero for all eigenvalues of the Hessian H a r ( s , a ) | a = π ( s ) , we add in the regularizer γ ( s ) , and also use β ( s ) to make | ˆ A ( s ) |
= 1 as in Eq.
(10) so that the metric applied kernel can fall back to an isotropic kernel in the degenerate case.
(See Appendix C.1 for more details.)
For the simplicity of the algorithm, we did not discard the components along the eigenvectors with zero eigenvalues ( U 0 ( s ) ) but let the metric have longer relative bandwidths in those directions compared to the others by the regularizers.
The KMIS metric ˆ A ( s ) with the regularizers is, ˆ
[Equation]
To apply the KMIS metric ˆ A ( s ) on a kernel-based IS estimation, we first fit a neural network reward regressor with the dataset D for the estimation of the Hessian matrix H a r ( s , a ) | a
= π ( s ) .
The reward regressor may have some error in the estimation of the Hessian and may have an adverse effect on our algorithm.
(Effect analyzed in Appendix F.1.)
Using the estimated Hessian, we linearly transform the kernel inputs with the transformation matrix ˆ L ( s ) ( ˆ A ( s )
=
ˆ L ( s )ˆ L ( s ) ⊤ ) in Eq.
Then given a kernel, and a bandwidth from the previous works on bandwidth learning for kernel-based IS [2][5], IS estimation can be made with the offline data D as in Algorithm 1.
[Equation]
Input: Offline data D = { s i , a i , r i } Ni =1 , behavior policy π b , deterministic target policy π , reward function parameters ϕ , bandwidth h , kernel K
[Equation]
Experiments
In this section, we show that applying our KMIS metric reduces the MSEs of the kernel-based IS estimators with bandwidths learned from existing works [2, 5] on synthetic domains and Warfarin dataset [30].
For baselines, we use existing works on kernel-based IS [2, 5] and a direct method (DM).
For synthetic domains, we also include a simple discretized OPE estimator [2] as a baseline.
For the baselines of existing works of kernel-based IS, we use the work of Kallus and Zhou [2] and SLOPE [5] which are bandwidths learning algorithms for kernel-based IS.
For the direct method (DM) we use a neural network reward regressor with a Gaussian output layer.
The discretized OPE estimator [2] evenly discretizes the action space by 10 for each action dimension (resulting in 100 bins in 2-dimensional action spaces) for an IS estimation.
The baselines are compared to our proposed KMIS metric applied kernel-based IS estimator with bandwidth learned from Kallus and Zhou’s estimator or SLOPE.
DM’s reward regressor is used for estimating the Hessian H a r ( s , a ) |
a = π ( s ) (Eq.
(8)) required for KMIS metric computation and the optimal bandwidth learning of Kallus and Zhou’s estimator (compute C v and C b in Eq.
(3) with DM).
We use DM for bandwidth learning of Kallus and Zhou’s estimator instead of using kernel density estimation (KDE), which is used in the work of Kallus and Zhou [2], since using KDE to estimate the bandwidth is computationally expensive [17].
For all estimators, we use self-normalization as in the work of Kallus and Zhou [2] as it is known to reduce estimation variance significantly with the addition of a small bias and result in reduced MSE.
We do not correct the boundary bias as opposed to the works of Kallus and Zhou [2], Su et al.
[5] since we regard the bias induced by the boundaries are negligible compared to the bias reduced by our metric.
For more details of the experiments, see Appendix E.
5.1 Synthetic Data
In the experiment with the synthetic data, we show that our KMIS metric can be applied to the offline data sampled from environments with various reward functions.
With such offline data, we first show the MSEs of existing kernel-based IS estimators can be reduced by our KMIS metric.
Furthermore, we empirically validate Proposition 1.
Lastly, we visualize the learned KMIS metrics to see if the metrics are learned as we intended.
We prepare three synthetic domains which are quadratic reward domain, absolute error domain, and multi-modal reward domain.
All synthetic domains use actions and states in R 2 .
In the quadratic reward domain, the rewards of the offline data are sampled from a normal distribution, where the mean is from a quadratic function w.r.t.
states and actions (for details, see Appendix E).
Each dimension of states are independently and uniformly sampled in the range of [ − 1 , 1] , actions are sampled from π b ( a | s ) =
N ( s +0 .
2 I, 0 .
5 2 I ) , target policy is π ( s )
= s .
If the metric is learned successfully from the data, the metric should form an ellipsoidal shape and should be the same for any state and action since the Hessian H a r ( s , a ) of the quadratic reward function is a constant.
In the absolute error domain, the offline data is sampled from the environment with deterministic rewards given states and actions r ( s , a ) =
−| 0 .
5 s 1 − a 1 | , where s 1 and a 1 are the first dimensions of the state and action vectors.
Each dimension of both states and actions are independently and uniformly sampled in the range of [ − 1 , 1] , and the target policy is π ( s ) = 0 .
5 s .
Since the absolute value function has points in its domain where it is not twice differentiable, and the Hessian H a r ( s , a ) is zero at the twice differentiable points, it is reasonable to think that the metric learning would fail in such cases.
However, since we are dealing with finite samples and using a neural network reward regressor, we conjecture that some form of concave-shaped reward estimate can be learned by the reward regressor and make meaningful metric learning possible.
If the conjecture is right and the metric learning is successful, the metric will be learned to have a relatively larger bandwidth in the direction of the dummy action dimension ( a 2 ) that is unrelated to the reward.
Lastly, for the multi-modal reward domain, the multi-modal reward function is made with exponential functions and max operators similar to the multi-modal reward function introduced in the work of Haarnoja et al.
Rewards are deterministic given states and actions.
(For the details see
Appendix E.)
Each dimension of states and actions are independently and uniformly sampled from the range of [ − 1 , 1] , and the target policy is π ( s )
= s + [ 0 . 50 ] .
Due to the max operators, there are points in the reward function domain where it is not twice differentiable.
For the twice differentiable points, the reward function is designed to have varying Hessians H a r ( s , a ) , thus, varying metrics should be learned for varying actions unlike the other synthetic domains.
Also, since the Taylor expansion terms of the reward function w.r.t.
the target action have higher order terms beyond 2nd order, the bias of the kernel-based IS estimation without the metric will contain the terms ignored in the derivation of the squared leading-order bias in Eq.
(2) (the derivation in Appendix A.1) which upper bound is minimized by the KMIS
metric
[Figure]
The experimental results on the synthetic domains are shown in Figure 2.
The first row of Figureshows that for most cases, the KMIS metric reduces the MSEs of the kernel-based IS estimators.
Also, the KMIS metric applied kernel-based IS estimators outperform DM, which is used by the KMIS algorithm to estimate the Hessians of a reward function for the metric learning.
For the multi-modal reward domain, even though the bias contains terms ignored in the leading-order bias, the metrics that reduce the upper bound of squared leading-order bias also reduce MSE.
The discretized OPE estimator performs worst in the quadratic reward domain for the sample size above 10k, but it performs better than some other estimators in the other domains.
As it is unclear how to discretize the multi-dimensional action space [2], even though the same discretization rule is used for all synthetic domains, its performance varies from domain to domain.
The second row of Figure 2 shows that for most of the given bandwidths, the MSEs of the kernelbased IS estimations are reduced by the KMIS metrics.
The gray dotted line denotes the MSEs of kernel-based IS estimators with given bandwidths and without a metric.
The black line denotes the
KMIS
The third row of Figure 2 shows learned KMIS metrics from a trial and the reward landscape when s =
[ 00 ] for each synthetic domain.
The learned metrics are drawn with black crosses.
The metrics are not only drawn for a target action but also for other actions.
For the metrics learned in the quadratic reward domain, the learned metrics are ellipsoidal and similar for most actions.
Therefore, we can see that our metric is learned as intended with the Hessian H a r ( s , a ) provided by the reward regressor (DM).
For the absolute error domain, the metrics are learned to ignore the dummy action dimension a 2 by having longer relative bandwidth in the direction of the action dimension while having shorter relative bandwidth in the direction of a 1 which is used for computing the reward.
This result empirically verifies our conjecture made earlier, which suggests that our metric learning algorithm can be generally applied to the dataset where the true reward function is not twice differentiable at some actions, and the true Hessian is zero at the twice differentiable actions.
The metrics learned on the multi-modal reward made with exponents verifies that our metric indeed learns varying metrics according to actions.
[Figure]
To empirically validate Proposition 1, the empirical squared bias, variance, and MSE ( MSE[ (cid:98) ρ K ]
=
Bias[ (cid:98) ρ K ] 2 + V ar [ (cid:98) ρ K ] ) of the
Kallus and Zhou’s estimator and the KMIS metric applied version of the estimator is observed on the modified absolute error domain with various number of dummy action dimensions.
The modified domain can have additional dummy action dimensions where the value of each dummy dimension is independently and uniformly sampled in the range of [ − 1 , 1] .
The result in Figure 3 shows that the empirical squared bias of
Kallus and Zhou’s estimator is indeed the dominant term in the MSE in high action dimensions.
As the action dimension is increased, both algorithms suffer from high bias.
But the KMIS metric reduces the bias and shows lower MSE than the one without the metric.
However, the
KMIS metric applied estimator shows higher variance than the Kallus and Zhou’s estimator.
This increase in the variance due to the metric may come from the ignored components of the variance, or, the Hessian estimation error.
5.2 Warfarin Data
To test our algorithm on a more realistic dataset, we test our algorithm on the Warfarin dataset [30].
Warfarin is a treatment that is commonly used for preventing blood clots.
The dataset contains information on the patients, therapeutic doses, and the resulting outcomes.
The outcomes of the treatments are reported in the international normalized ratio (INR), which measures how quickly the blood clots.
We use a similar experimental setting used by Kallus and Zhou [2].
The 81 features of the patient information selected by Kallus and Zhou [2] is used as states s .
For the behavior action vectors, the first dimension is sampled from the normal distribution N ( µ ∗ + σ ∗ √ 0 .
5 z BMI , ( σ ∗ √ 0 .
5) 2 ) truncated by the minimum and maximum therapeutic doses a ∗ min and a ∗ max in the data.
z BMI is the z score of patients’ BMIs, µ ∗ and σ ∗ are the mean and the standard deviation of the therapeutic dosages, respectively.
To test our algorithm we add a dummy action dimension sampled from a uniform distribution a 2 ∼ unif[ a ∗ min , a ∗ max ] .
For the reward, since Kallus and Zhou [2] reported that the INR in the dataset is inadequate for testing OPE algorithms, we use their cost function as a negative reward function r = − max( | a 1 − a ∗
| − 0 .
1 a ∗ , 0) .
The reward function is only dependent on the first action dimension.
The target policy is π ( s ) =
[ s BMI 0 ] .
Experimental results in Figure 4a show that our metric reduces the MSE of the kernel-based IS estimate in Warfarin data that is more realistic than that of the synthetic domain.
In the figure, we see that even though the MSE of the reward regressor is high, MSE of SLOPE and Kallus and Zhou’s estimator is reduced with the KMIS metric.
Similar to the results in the synthetic domains, the KMIS metric reduces MSE for almost all given bandwidths in Figure 4b.
[Figure]
Conclusion
We presented KMIS, an algorithm for off-policy evaluation in contextual bandits for a deterministic target policy with multidimensional continuous action space.
KMIS improves the kernel-based IS method by learning a local distance metric used in the kernel function per state, leading to a kernelbased IS OPE estimator that exploits Mahalanobis distance, rather than Euclidean distance.
Based on the observation that the leading-order bias becomes a dominant term in the LOMSE when the action dimensionality is high, we derived an analytic solution to the optimal metric matrix that minimizes the upper bound of the leading-order bias.
The optimal metric matrix is bandwidth-agnostic and computed using the Hessian of the learned reward function.
Experimental results demonstrated that our KMIS significantly improves the performance of the kernel-based IS OPE across different bandwidth selection methods, outperforming baseline algorithms.
As for future work, exploring algorithms that jointly optimize bandwidth and metric or extending to RL beyond contextual bandits case would be interesting directions to pursue.
Acknowledgments and Disclosure of Funding
Haanvid Lee, Yunseon Choi, and Kee-Eung Kim were supported by National Research Foundation (NRF) of Korea (NRF-2019R1A2C1087634), Field-oriented Technology Development Project for
Customs Administration through National Research Foundation (NRF) of Korea funded by the
Ministry of Science & ICT and Korea Customs Service (NRF-2021M3I1A1097938), Institute of
Information & communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its
Applications to Natural Language Processing; No.2022-0-00311, Development of Goal-Oriented
Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects;
No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST); No.2021-0-02068,
Artificial Intelligence Innovation Hub), and Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government (22ZS1100, Core Technology Research for SelfImproving Integrated Artificial Intelligence System), and KAIST-NAVER Hypercreative AI Center.
Byung-Jun Lee was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
(No.2019-0-00079 , Artificial
Intelligence Graduate School Program(Korea University)).
A Derivation Details A.1 Leading-Order MSE Bias of the Kernel-Based IS Estimation [2]
Define E π b [ · ] := E
[Equation]
∼ p( s ) , a ∼ π b ( a | s ) , r ∼ p(r | s , a ) [ · ] , then the bias of a kernel-based IS estimate is:
[Equation]
For the first term of Eq.
[Equation]
where the following relations are used in the derivation,
[Equation]
Taylor expansion of p ( r | s
[Equation]
a ) at a = π ( s ) , p
[Equation]
using the relation h u = a − π ( s ) ,
[Equation]
Variance of the Kernel-Based IS Estimation [2]
Define E π b [ · ] :=
E s ∼ p
( s ) , a ∼ π b ( a | s ) , r ∼ p(r | s , a ) [ · ] , then the variance of a kernel-based IS estimate is:
Var[ˆ ρ K ] =
[Equation]
The second term of the Eq.
(14) is,
[Equation]
The first term of the Eq.
(14) is,
[Equation]
By substituting a = h u + π ( s ) (cid:16) which comes from u = a − π ( s ) h
[Equation] [Equation]
Let g r ( s , a ) :
= p ( r | s , a ) π b ( a | s ) , and apply Taylor expansion at a = π ( s ) , g r ( s , a ) =
g r
( s , π ( s ))
+ ( a − π ( s ))
⊤ ∇ a g r ( s , a ) | a = π ( s )
+
[Equation]
a
[Equation]
g
[Equation]
a
[Equation]
|
[Equation]
substitute a = h u + π
[Equation]
Plugging in Eq.
(17) to Eq.
[Equation]
From Eq.
(15) and Eq.
[Equation]
the variance can be represented as,
[Equation]
From the derived bias (Eq. (13)) and variance (Eq
[Equation]
[Equation]
MSE can be derived:
[Equation]
Assuming that h → 0 and
[Equation]
A.2 Optimal Bandwidth
The optimal bandwidth ( h ∗ ) that minimizes the leading-order MSE [2] is,
[Equation]
A.3 Derivation of Eq. (6)
For the derivation
[Equation]
we use the following relations:
[Equation]
By changing a of C b in Eq.
(2) to z ( = L ( s ) ⊤ a ), we get:
[Equation]
B Proofs B.1 Proof of Proposition
Proposition 1.
(Adapted from Noh et al.
For a high dimensional action space D
A ≫ 4 , and given optimal bandwidth h ∗ (Eq.
(3) ), the squared leading-order bias dominates over the leading-order variance in LOMSE.
Furthermore, LOMSE( h ∗ , N, D A ) can be approximated by C b in Eq.
[Equation]
Proof.
By plugging in h ∗ in Eq.
(3) to the leading-order bias and leading-order variance in Eq.
(2) and by taking D A → ∞ (or, for D
A ≫ 4 ), their ratio is:
[Equation]
order bias
[Equation]
Therefore, the squared leading-order bias dominates over the leading-order variance in the LOMSE in the high dimensional action space ( D A ≫ 4 ).
By plugging in h ∗ Eq.
(3) to the LOMSE in Eq.
(2) we get,
[Equation]
By taking D A → ∞ (or, for D A ≫ 4 ),
[Equation]
lim
[Equation]
∴ LOMSE
[Equation]
C
[Equation]
B.2 Proof of Theorem
We use the semi-definite programming solution presented in the work of Noh et al.
[22] for computing the metric matrix A ∗ ( s ) that minimizes the tr (cid:0) A − 1 ( s ) B ( s ) (cid:1) 2 for nearest neighbor classification.
We use the solution to minimize the squared trace term in Eq.
In our case, we use
B ( s ) :
= H a E
[ r | s , a ] | a
= π ( s ) which comes from Eq.
Theorem 1.
(Adapted from Noh et al.
[22]) Assume that the p ( r | s , a ) is twice differentiable w.r.t.
an action a .
Let Λ + ( s ) and Λ − ( s ) be diagonal matrices of positive and negative eigenvalues of the Hessian H a E [ r | s , a ] | a =
π
( s ) , U + ( s ) and U − ( s ) be matrices of eigenvectors corresponding to Λ + ( s ) and Λ − ( s ) respectively, and d + ( s ) and d − ( s ) be the numbers of positive and negative eigenvalues of the Hessian.
Then the metric A ∗ ( s ) that minimizes U b,A ( s ) is:
[Equation]
where α ( s ) := | M ( s ) | − 1 / ( d + ( s )+ d − ( s )) .
Proof.
Define B ( s ) :
= H a r ( s , a ) | π ( s ) , then we want A ( s ) that,
[Equation]
In the cases where some of the eigenvalues of B ( s ) are 0, the components along the directions of the corresponding eigenvectors can be discarded.
Minimizing the objective, or, minimizing U b,A is the same as minimizing C b,A under the condition that there are not both sets of states with positive and negative trace terms tr (cid:0) A ( s ) − 1 B ( s ) (cid:1) .
The trace term is positive or negative when the eigenvalues of B ( s ) other than 0 are all positive or negative, respectively.
And the trace term becomes 0 when there are both negative and positive eigenvalues in the eigenvalues of B ( s ) .
In cases where all the states have both positive and negative eigenvalues in the eigenvalues of B ( s ) , C b,A = U b,A = 0 .
When there are both positive and negative eigenvalues in the eigenvalues of B ( s ) ,
A ∗ ( s ) =
α ( s ) [ U + ( s ) U − ( s )] (cid:18) d + ( s )Λ + ( s )  − d − ( s )Λ − ( s ) (cid:19)
[ U + ( s ) U − ( s )]
⊤
B ( s ) =
[ U + ( s ) U − ( s )] (cid:18) Λ + ( s )
Λ
− ( s ) (cid:19)
[ U + ( s ) U − ( s )] ⊤ tr (cid:0) A ∗ ( s ) − 1 B ( s ) (cid:1) =
α ( s ) tr  
[ U + ( s ) U − ( s )]   Λ − 1+ ( s ) d + ( s )  − Λ − 1 − ( s ) d − ( s ) 
 (cid:18) Λ + ( s )
[Equation]

[Equation]
When the eigenvalues of B ( s ) other than 0 are all negative or positive, then we can solve the
Lagrangian equation F ( s ) .
We first solve for the case when the eigenvalues of B ( s ) other than 0 are all positive,
[Equation] [Equation]
(cid:0) since A ( s ) =
A ( s ) ⊤ , B ( s ) =
B ( s ) ⊤ , | A ( s )
[Equation]
= 1 (cid:1) (i) Assume c ( s ) = 0 ,
Then either tr (cid:0) A ( s ) − 1 B ( s ) (cid:1) = 0 , or, A ( s ) − 1 B ( s ) =
We first check if tr (cid:0) A ( s ) − 1 B ( s ) (cid:1) = 0 .
[Equation]
B
[Equation] [Equation]
B
[Equation]
Since B ( s ) =
U + ( s )Λ + ( s ) U + ( s ) ⊤ , B ( s )
=
B ( s ) ⊤
= U + ( s )Λ + ( s ) 12 U + ( s ) ⊤ .
And as
A ( s ) ≻ 0 , its inverse is also positive definite A ( s ) − 1 ≻ 0 .
Using these relations, we can show that the term inside the trace in Eq.
(27) is a positive definite matrix.
[Equation]
⊤ A ( s ) − 1 (cid:16) B ( s )
⊤ 2 a (cid:17) > 0 , ∀
a ∈ R D
A \{ 0 }
Because B ( s ) 12 A ( s ) − 1 B ( s ) 12 ≻ 0 , tr (cid:16) B ( s ) 12 A ( s ) − 1 B ( s ) 12 (cid:17) > 0 .
Then, from Eq.
(27), tr (cid:0) A ( s ) − 1 B ( s ) (cid:1) >
A ( s ) − 1 B ( s ) ̸ = 0 , as it’s trace is positive
Therefore, the assumption is wrong, and c ( s ) ̸ =(ii) Assume c ( s ) ̸ = 0 ,
Since the Lagrangian multiplier c ( s ) and tr (cid:0) A ( s ) − 1 B ( s ) (cid:1) in Eq.
(26) are scalar values,
A ( s ) needs to be a scalar multiple of B ( s ) (which is symmetric) to match the scalar multiple of identity matrix in the LHS of Eq.
(26) while satisfying A ( s ) ≻ 0 and | A ( s ) | = 1 .
Therefore the A ∗ ( s ) is,
[Equation]
Similarly, we can also derive A ∗ ( s ) for the case where all eigenvalues of B ( s ) are negative.
B.3 Proof of Theorem
Theorem 2.
(Adapted from Kallus and Zhou [2]) Kernel-based IS estimator with the optimal metric
A ∗ ( s ) from Eq.
(8) and the optimal bandwidth h ∗ in Eq.
(3) is a consistent estimator in which convergence rate is equal or faster than the isotropic kernel-based IS estimator with h ∗ .
When the
Hessian H a r ( s , a ) |
a = π ( s ) has both positive and negative eigenvalues for all states, A ∗ ( s ) applied estimator converges to the true policy value faster than the one without the metric by the rate of
O ( D − 12 A ) as the action dimension increases.
Proof.
The MSE of a kernel-based IS estimation without a metric derived by Kallus and Zhou [2] in terms of action dimension D A and data size of N is (from the derived bias and variance in
Appendix A.1),
[Equation]
where, the MSE consists of squared bias and the variance of the estimate ˆ ρ K .
Since the optimal bandwidth presented in Eq.
(3) is h ∗
=
O (cid:16) D A N
[Equation]
, the convergence of the MSE without a metric is:
[Equation]
Since the MSE of the kernel-based IS estimator with h ∗ converges to 0 by O (cid:16)(cid:0) D A N (cid:1) 4 DA +4 (cid:17) , The estimation approaches to the true policy value by the rate of O (cid:16)(cid:0)
D A N (cid:1) 2 DA +4 (cid:17) .
For the KMIS metric applied kernel-based IS estimator, its convergence rate can be computed similar to that of the isotropic kernel-based IS estimator regarding C b,A as a non-zero constant, then the resulting convergence rate is the same as that of the isotropic kernel-based IS estimator.
However, in the best case where the Hessians H a r ( s , a ) | a = π ( s ) contain both negative and positive eigenvalues for all states, C b,A ∗ = 0 .
Then, the MSE with the optimal metric and the optimal bandwidth MSE( h ∗ , A ∗ , N, D A ) converges to zero by:
[Equation]
In the best case where there are both positive and negative eigenvalues for all states, the MSE of our metric applied kernel-based IS estimation converges to 0 faster than the one without the metric by
O ( D − 1 A ) .
Therefore, our algorithm converges to the true policy value faster than the one without the metric by O ( D − 12 A ) in the best case.
C Algorithm Details
C.1 Regularizers for the KMIS Metric
[Equation]
To compute the regularizers β ( s ) and γ ( s ) in Eq.
(10), first add a positive real coefficient ϵ ( s ) that is relatively smaller than the eigenvalues of A ∗ ( s ) to X ( s ) .
For the experiments, ϵ ( s ) was assigned to be the maximum value among eigenvalues in the Hessian of reward multiplied by 0.01.
[Equation]
With the coefficient ϵ ( s ) I added to X ( s ) , compute the determinant of the resulting matrix Y ( s ) .
To scale the metric matrix to have determinant of 1, We scale the matrix by multiplying β ( s ) =|
Y ( s ) | ( − 1 /D A ) .
Then the γ ( s ) =
β ( s ) ϵ ( s )
D Additional Experiment D.1 Experiment with Varying Noise Level in the Rewards
When the noise in the rewards increases, DM suffers the most as the function estimation becomes difficult as the noise increases.
Other algorithms also show an increase in the MSE as the noise increases.
The KMIS metric applied kernel-based IS estimator performs the best with the learned metric even though the DM which the KMIS uses to estimate the Hessians required for the metric learning performs the worst.
[Figure]
E Experiment Details
For all experiments, we used a neural network reward regressor with 2 fully-connected layers ofhidden units with tanh activations, and a Gaussian output layer.
The neural network was trained with learning rate of 5e-4.
The learning rate was chosen by conducting grid search over {1e-4, 5e-4, 1e-3}.
The reward regressor was trained with early stopping rule where we stop training when the validation error does not decrease for 20 training epochs.
And we used the trained weights that showed the lowest validation error during the training.
For the train and validation split, 20% of the available offline data was used as the validation data, and the rest of the data was used for the training.
For the estimation time, when we test our algorithm on the synthetic domain with absolute error and a dummy action dimension, our algorithm takes 40 seconds on average to make an OPE estimate with 20k offline data on the i7 CPU with 32GB RAM.
The experimental results were made with
C2-standard-4 instances on the Google Cloud Platform where each instance has 4 virtual CPUs and 16GB of RAM.
The code is available at https://github.com/haanvid/kmis.
Self-Normalization All estimators in the experiments are self-normalized as in the work of Kallus and Zhou [2].
Self-normalization is used because it reduces the estimation variance significantly with the addition of a small bias and yields a smaller MSE in result compared to the estimation without self-normalization [2, 13].
The self-normalized kernel-based IS estimator without a metric is shown in Eq.
[Equation]
[Equation]
No Boundary Bias Correction Action space can be bounded, and the kernels used in importance sampling can be extended past the bounds.
As the actions of the offline data are only observed inside the bounds, the kernels extending outside the bounds will induce bias.
Previous works [2, 5] made a correction for the induced bias by truncating the kernel by the action bounds of each dimension and normalizing the kernel.
However, we do not correct the boundary bias for all estimators in this work since we regard the boundary bias as negligible compared to the bias reduced by our locally learned metric.
Removing the boundary bias correction makes kernel-based IS estimators simpler to use than those with corrections, as the implementation of boundary bias correction requires information on the action bounds, and the integration of the kernel within the action bounds.
Discretized OPE Estimator Discretized ope estimator discretizes the action space bounded by the minimum and maximum value of the behavior actions for each dimension with the given number of bins assigned for each dimension.
In our experiment, we discretized each action dimension byintervals resulting in 100 discretized bins in 2D action space.
The IS ratio of the discretized OPE estimator is composed of the indicator function in the numerator and the behavior policy probability density function integrated over the bin in the denominator.
E.1 Synthetic Domains Experiments
For the synthetic domain, we used dropout after each hidden layer with dropout rate of 0.5.
Behavior policy density values were clipped below 0.1 for the quadratic reward domain as in the works of
Kallus and Zhou [2].
For absolute error domain and multi-modal reward domain, as their behavior policy density is uniform density, we did not apply behavior density value clipping.
Clipping the behavior policy density value adds a small bias to the estimation, but reduces a significant variance, and therefore MSE is smaller compared to the estimations without the clipping [2].
For SLOPE, we select a bandwidth from geometrically spaced bandwidths (cid:8) 2 − i : i ∈ [1 , 7] , i ∈ N (cid:9) as in the work of
Su et al.
Quadratic Reward Domain
The conditional reward distribution given a state s and a action a is:
[Equation]
where, r ( s , a ) :
=
E [ r | s , a ] .
The mean reward of a quadratic reward domain given a state s and a action a is:
[Equation]
[Equation]
Dummy Action Dimension Domain
Each dimension of actions and states are sampled uniform randomly in the range of [ − 1 , 1] .
The absolute error which is only dependent on the first dimension of action is designed as shown below.
[Equation]
where the reward is not sampled from a distribution but determined by states s and actions a .
Multi-Modal Reward Domain
Each dimension of actions and states are sampled uniform randomly in the range of [ − 1 , 1] .
The multi-modal reward with exponential functions is designed as shown below.
[Equation]
where the reward is not sampled from a distribution but determined by states s and actions a .
E.2 Warfarin Data Experiments
For the warfarin data, we used L2 regularizer for the 2 hidden layers with the coefficient of 1e-1.
The
L2 regularizer coefficient was chosen by conducting grid search over {1e-5, 1e-4, 1e-3, 1e-2, 1e-1}.
We selected the L2 regularization coefficient that has a lowest validation error (the offline dataset D was split into a train and validation dataset) for the reward regressor.
Behavior policy density values below 1e-1 was clipped as in the work of Kallus and Zhou [2].
For SLOPE, we selected a bandwidth from (cid:8) 2 − i : i ∈ [ − 2 , 7] , i ∈ N (cid:9) similar to the work of Su et al.
F Theoretical Analysis F.1 Effect of the Hessian Estimation Error on the Estimation Error and the Convergence Speed of the KMIS Metric Applied Kernel-Based IS Estimator Error Analysis
To reduce the squared leading-order bias LOBIAS( h, A ( s )) 2 in Eq.
(43) (the equation can be derived with bias from Eq.
(13), variance from Eq.
(19), and the squared leading-order bias with metric from Eq.
(6) ) of a kernel-based IS estimator, KMIS learns the metric that minimizes the upper bound U b,A in Eq.
To minimize U b,A , we derived the optimal metric matrix A ∗ ( s ) given the true Hessian H ( s ) :
= H a E
[ r | s , a ] | a = π ( s ) in Appendix B.2.
However, since we will be using estimated Hessian matrix (cid:101) H ( s ) acquired from a reward regressor, we analyze how the error in (cid:101)
H ( s ) affects KMIS estimation error.
[Equation]
where,
[Equation] [Equation]
[Equation]
In Appendix B.2, we proved that when H ( s ) contains both positive and negative eigenvalues, the optimal metric makes the squared leading-order bias LOBIAS( h, A ∗ ) 2 = 0 by making the the trace term tr( A ( s ) − 1 H ( s )) zero.
In this section we analyze the upper bound of LOBIAS( h, (cid:101) A )in a similar setting where the H ( s ) contains both positive and negative eigenvalues and no zero eigenvalues.
For analyzing, we use the following notations regarding the true Hessian H ( s ) , the optimal metric computed from the true Hessian A ∗ ( s ) , estimated Hessian from the reward regressor (cid:101)
H ( s ) and the metric computed from it (cid:101)
A ( s ) :
[Equation] [Equation]
A
[Equation]
] (cid:18)
[Equation]
− d
[Equation]
(cid:19)
[ U
[Equation]
s
[Equation]
( s )]
⊤ (47) Λ( s ) =
(cid:18) Λ + ( s
[Equation]
Λ − ( s ) (cid:19) Λ + ( s ) =   λ + , 1 ( s ) · · · λ + , 2 ( s ) .
 
U + ( s ) =
(cid:32) | | u + , 1 ( s ) u + , 2 ( s ) · · · | | (cid:33) α ( s ) =
    d
+ ( s ) d + ( s ) d + ( s ) (cid:89) i
= λ
+ ,i ( s ) 
   ( − 1) d − ( s ) d − ( s ) d − (
[Equation]
[Equation]
where we used eigendecomposition on H ( s ) .
Λ + ( s ) and Λ − ( s ) are diagonal matrices containing positive and negative eigenvalues, respectively.
U + ( s ) and U − ( s ) are matrices containing eigenvectors corresponding to Λ + ( s ) and Λ − ( s ) , respectively.
Similarly, (cid:101) Λ + ( s ) , (cid:101) Λ − ( s ) , (cid:101) U + ( s ) , (cid:101)
U − ( s ) are decomposed from (cid:101) H ( s ) .
We assume that (cid:101) H ( s ) correctly estimates the sign (positive, negative signs) of the eigenvalues in H ( s ) .
Because of the assumption, there are d + ( s ) and d − ( s ) in Eq.
(50) and
Eq.
(51) instead of ˜ d
+ ( s ) and ˜ d − ( s ) .
We quantify the error in (cid:101) H ( s ) with constants ϵ and η ( ϵ ≥ 0 , η > 0 ) by:
[Equation]
where, a, b
[Equation]
[Equation] [Equation]
∈
[Equation] [Equation]
The trace term tr( (cid:101)
A ( s ) − 1 H ( s )) in the LOBIAS( h,
(cid:101) A ) 2 is, tr( (cid:101)
A ( s ) − 1 H ( s ))
= (cid:101) α ( s ) tr (cid:16)
[ d + ( s
[Equation]
where in the third equality (Eq.59) we used,
[Equation]
where, a,
[Equation]
{ + , −
[Equation] [Equation]
Expanding X ( s ) we get,
[Equation]
We get the following inequality for X ( s ) by applying Eq.
[Equation]
Similarly, we get,
[Equation]
The trace term tr( (cid:101)
A ( s ) − 1 H ( s ))
( = X ( s ) − Y ( s ) ) satisfies the following inequality:
[Equation]
Then the absolute value of the trace term satisfies the following inequality:
[Equation]
where, | X u ( s ) − Y l ( s ) | , | X l ( s ) − Y u ( s ) | are,
[Equation]
| X l ( s ) − Y u ( s ) | = (cid:101) α ( s ) (cid:8) 6 ϵ + 2 ϵ 3 + ηϵ 2 (2 d − ( s ) − 1) (cid:9) , where, d + ( s ) , d −
[Equation]
By defining d max ( s ) :
= max { d + ( s ) , d − ( s ) } , we can rewrite Eq.
[Equation]
Then we get the following inequality for LOBIAS( h, (cid:101)
A ( s )) 2 :
[Equation]
When there is no error in the Hessian estimation (when ϵ = 0 ) acquired from the reward regressor, then the LOBIAS( h, (cid:101) A ) 2 becomes zero in Eq.
But when there is an error in the Hessian estimation (when ϵ > 0 ), the KMIS estimator may have LOBIAS( h, (cid:101) A ) 2 up to
[Equation]
Convergence Speed Analysis
Now we analyze the convergence speed of KMIS estimator with (cid:101) H ( s ) with additional assumption that | (cid:101) H ( s ) | is bounded.
By replacing in LOBIAS( h, A ) 2 in
Eq.
(43) with the derived upper bound of LOBIAS( h, (cid:101) A ) 2 in Eq.
(69), and also using the relation h ∗ =
O (cid:16)(cid:0)
D
A N (cid:1) 1 DA +4 (cid:17) from Eq.
(3), we get:
[Equation]
where, N ≫ D A
As for the first term of Eq.
[Equation]
∵ |
[Equation]
H
[Equation]
| is bounded
Assuming that d max ( s )
=
d + ( s ) , Q ( s ) monotonically increases w.r.t.
d + ( s ) .
[Equation] [Equation]
The maximum of Q ( s ) is (when d + ( s ) =
D A − 1 ),
[Equation]
The minimum of Q ( s ) is (when d + ( s ) =
D A 2 ),
[Equation]
We can assume d max ( s ) =
d − ( s ) and show that the maximum and minimum of Q ( s ) are the same.
Therefore,
[Equation] [Equation]
With Eq.
(76) and using the relation D
A − 1 ≥ d max ( s ) ≥ D
A 2 on Eq.
O
[Equation]
Plugging in Eq.
(77) to Eq.
[Equation] [Equation]
When the estimated Hessian (cid:101) H ( s ) has no error (when ϵ = 0 ), the convergence speed derived in Eq.
(78) matches the convergence speed of KMIS estimator with a true Hessian derived in
Appendix B.3.
However, when there is an error in the estimated Hessian (when ϵ > 0 ), KMIS estimator with the estimated Hessian has a slower convergence rate to a true policy value than the
KMIS estimator with a true Hessian by the rate of O (( D A ) 52 )
(MSE of a KMIS estimator with the estimated Hessian converges to zero slower by the rate of O (( D A ) 5 ) compared to the KMIS with a true Hessian).