Ofﬂine Policy Selection under Uncertainty
Abstract
The presence of uncertainty in policy evaluation signiﬁcantly complicates the process of policy ranking and selection in real-world settings.
We formally consider ofﬂine policy selection as learning preferences over a set of policy prospects given a ﬁxed experience dataset.
While one can select or rank policies based on point estimates of their expected values or high-conﬁdence intervals, access to the full distribution over one’s belief of the policy value enables more ﬂexible selection algorithms under a wider range of downstream evaluation metrics.
We propose a Bayesian approach for estimating this belief distribution in terms of posteriors of distribution correction ratios derived from stochastic constraints.
Empirically, despite being Bayesian, the credible intervals obtained are competitive with state-ofthe-art frequentist approaches in conﬁdence interval estimation.
More importantly, we show how the belief distribution may be used to rank policies with respect to arbitrary downstream policy selection metrics, and empirically demonstrate that this selection procedure signiﬁcantly outperforms existing approaches, such as ranking policies according to mean or high-conﬁdence lower bound value estimates.
Introduction
Off-policy evaluation (OPE)
[53] in the context of reinforcement learning (RL) is often motivated as a way to mitigate risk in practical applications where deploying a policy might incur signiﬁcant cost or safety concerns [60].
Indeed, by providing a point estimate of the value of a target policy solely from a static ofﬂine dataset of logged experience in the environment, OPE can help practitioners determine whether a target policy is or is not safe and worthwhile to deploy.
Still, in many practical applications the ability to accurately estimate the online value of a speciﬁc policy is less of a concern than the ability to select or rank a given set of policies (one of which may be the currently deployed policy).
For example, in recommendation systems, a practitioner may have a large number of policies trained ofﬂine using various hyperparameters, while cost and safety constraints only allow a few of those policies to be deployed as live experiments.
Which policies should be chosen to form the small subset that will be evaluated online?
This problem, related to but subtly different from OPE, is ofﬂine policy selection [17, 51, 36].
The original motivations for OPE were arguably with ofﬂine policy selection in mind [53, 28], the idea being that one can use estimates of the value of a set of policies to rank and then select from this set.
Accordingly, there is a rich literature of approaches for computing point estimates of the value of the policy [19, 4, 31, 59, 45, 69, 62, 32, 66], as well as estimating high-conﬁdence lower and upper bounds on a target policy’s value [60, 36, 4, 25, 22, 11, 34].
These existing OPE approaches may be readily applied to the recommendation systems example above by using either mean or high-conﬁdence bounds estimates on each candidate policy to rank the set and picking the top few to deploy online.
However, such a naïve approach ignores crucial differences between the OPE problem setting and the downstream evaluation criteria a practitioner prioritizes.
For example, when choosing a few policies out of a large number of policies, a recommendation systems practitioner may have a number of objectives in mind: They may strive to ensure that the policy with the overall highest groundtruth value is within the small subset of selected policies (akin to top- k precision).
Or, in scenarios where the practitioner is sensitive to large differences in achieved value, a more relevant downstream metric may be the difference between the largest groundtruth value within the k selected policies compared to the groundtruth of the best possible policy overall (akin to top- k regret).
With these potential ofﬂine policy selection metrics, it is far from obvious that ranking according to OPE mean or high-conﬁdence bound estimates is ideal [17].
The diversity of downstream metrics for ofﬂine policy selection presents a challenge to any algorithm that produces a point estimate for each policy.
In fact, any one approach to computing point estimates will necessarily be sub-optimal for some adversarially chosen policy selection criteria.
To circumvent this challenge, we propose to compute a belief distribution over groundtruth values for each policy.
Speciﬁcally, with the posteriors of the policy values, one can calculate the distribution of a variety of criteria over the value for each policy.
These posteriors can be used in a straightforward procedure that takes estimation uncertainty into account to rank the policy candidates.
While this belief distribution approach to ofﬂine policy selection is attractive, it also presents its own challenge: how should one estimate such a distribution in the purely ofﬂine setting?
We propose Bayesian Distribution Correction Estimation (BayesDICE) to address this challenge.
BayesDICE works by estimating posteriors over correction ratios for each state-action pair, corresponding to a belief distribution over density ratios between the off-policy data and the stationary distribution of the target policy.
In contrast to the point estimates of state-of-the-art DICE estimators [45, 69, 66], BayesDICE maintains a distribution from which the sampled ratio satisﬁes the stationary distribution condition with high probability .
Given belief distributions over these correction ratios, the belief distribution over a policy value may be estimated by averaging these correction distributions over ofﬂine data, weighted by rewards or other nonlinear utilities in the case of more exotic downstream policy selection criteria.
As a preliminary experiment, we show that the proposed BayesDICE is highly competitive to existing frequentist approaches when applied to conﬁdence interval estimation.
Then, we demonstrate the superiority of BayesDICE applied to ofﬂine policy selection under different utility measures, across a variety of discrete and continuous RL tasks.
Our policy selection experiments suggest that, while conventional wisdom in the OPE literature focuses on using lower bound estimates to select policies (due to safety concerns) [36], policy ranking based on the lower bound estimates may not always lead to lower downstream regret.
Furthermore, when other metrics of policy selection are considered, such as top- k precision, being able to sample from the posterior enables signiﬁcantly better policy selection than only having access to the mean or conﬁdence bounds of the estimated policy values.
We note that the ofﬂine policy selection problem is distinct from ofﬂine policy optimization (OPO) [39, 24, 35, 6], where one seeks a policy from a parameterized class that optimizes a pointwise objective without consideration of its performance relative to an ensemble of reference policies.
(This distinction will become clear in Section 2 below.)
In summary, the contributions of this paper are three-fold:
• We formally deﬁne ofﬂine policy selection and compare and contrast it to traditional OPE (and OPO).
• We propose BayesDICE for characterizing the posterior of the stationary state-action ratio, derived from the perspective of stochastic constraints.
• We design a simulation-based policy ranking algorithm, OfﬂineSelect , that converts the estimated posteriors from BayesDICE to a ranking of policies with respect to a selection criterion.
Ofﬂine Policy Selection
We consider an inﬁnite-horizon Markov decision process (MDP) [54] denoted as M = (cid:104) S, A, R, T, µ 0 , γ (cid:105) , which consists of a state space, an action space, a deterministic reward function, a transition probability function, an initial state distribution, and a discount factor γ ∈ (0 , 1] .
For simplicity, we restrict our analysis to deterministic rewards, and extending our methods to stochastic reward scenarios is straightforward.
In this setting, a policy π ( a t | s t ) interacts with the environment starting at s 0 ∼ µ 0 and receives a scalar reward r t =
R ( s t , a t ) as the environment transitions into a [Figure] new state s t +1 ∼ T ( s t , a t ) at each timestep t .
The value of a policy is deﬁned as
[Equation]
We formalize the ofﬂine policy selection problem as providing a ranking O ∈ Perm([1 , N ]) over a set of candidate policies { π i } Ni =1 given only a ﬁxed dataset D = { x ( j ) : = ( s ( j ) 0 , s ( j ) , a ( j ) , r ( j ) , s (cid:48)
( j ) ) }
nj =1 where s ( j )∼ µ 0 , ( s ( j ) , a ( j ) ) ∼ d D are samples of an unknown distribution d D , r ( j ) = R ( s ( j ) , a ( j ) ) , and s (cid:48) ( j ) ∼ T ( s ( j ) , a ( j ) ) .
The vanilla approach to the ofﬂine policy selection problem is to characterize the value of each policy under some utility function u ( π ) and then sort the policies accordingly; i.e. ,
O ← ArgSortDescending( { u ( π i ) } Ni =1 ) .
The utility u ( π i ) is typically the result of an OPE algorithm applied to D and π i ; i.e. , u ( π i ) is either a mean or lower-conﬁdence bound estimate of the policy’s normalized per-step reward in (1).
2.1 Selection evaluation
A proposed ranking O will eventually be evaluated according to how well its policy ordering aligns with the groundtruth policy values.
In this section, we elaborate on several potential forms of this evaluation score.
The groundtruth policy value for π i is given by ρ ( π i ) , and we use ρ i as shorthand for this expression.
As part of the ofﬂine policy selection problem, we are given a ranking score S , which serves as the downstream selection criterion we want to optimize.
The ranking score is a function that produces a scalar evaluation metric given a proposed ranking O and groundtruth policy values of { ρ i } Ni =1 .
The
S can take on many forms and is application speciﬁc; e.g. ,
• top- k precision : This is an ordinal ranking score.
The score considers the top k policies in terms of groundtruth means ρ i and returns the proportion of these which appear in the top k spots of O .
• top- k accuracy : Another ordinal ranking score, this score considers the top- k policies in sorted order in terms of groundtruth means ρ i and returns the proportion of these which appear in the same ordinal location in O .
• top- k correlation : Another ordinal ranking score, this represents the Pearson correlation coefﬁcient between the ranking of top- k policies in sorted order in terms of groundtruth means ρ ( π i ) and the truly best top- k policies.
• top- k regret : This is a cardinal ranking score.
This score respresents the difference in groundtruth means ρ i between the overall best policy – i.e. , max i ρ i – and the best policy among the top- k ranked policies – i.e. , max i ∈ [1 ,k ] ρ O [ k ] .
• Beyond expected return : One may deﬁne the above ranking scores in terms of statistics of the policy value other than the groundtruth means { ρ i } Ni =1 .
For example, in safety-critical applications, one may be concerned with the variance of the policy return.
Accordingly, one may deﬁne CVaR analogues to top- k precision and regret.
For simplicity, we will restrict the discussion in this paper to ranking scores which only depend on the groundtruth expected returns { ρ i } Ni =1 .
2.2 Bayes ranking simulation from the posterior
It is not clear whether ranking according to vanilla OPE (either mean or conﬁdence based) is ideal for any of the ranking scores above, including, for example, top- 1 regret in the presence of uncertainty.
However, if one has access to an approximate belief distribution over the policy values, one can simply simulate the Bayes risk over all candidate ranks to ﬁnd a near-optimal ranking [18] with respect to an arbitrary speciﬁed downstream ranking score, and we elaborate on this Bayes decision procedure here.
Algorithm
Inputs Posteriors q ( { ρ i } Ni =1 ) , ranking score ˆ S
Initialize O ∗ ; L ∗ (cid:3) Track best score for O in Perm ( [1 , ..., N ] )
do
L =for j = 1 to n do sample { ˆ ρ ( j ) i } Ni =1 ∼ q ( { ρ i } Ni =1 ) (cid:3)
Sum up sample scores
L = L + ˆ S ( { ˆ ρ ( j ) i } Ni =1 , O ) end for if L < L ∗ then (cid:3)
Update best ranking/score
L ∗ =
L ; O ∗ =
O end if end for ; return O ∗ , L ∗
In the ideal case if we have access to the true groundtruth policy values { ρ i } Ni =1 , and the ranking score function S , we can calculate the score value of any ranking O and ﬁnd the ranking O ∗ that optimizes this score.
However, we are limited to a ﬁnite ofﬂine dataset and the full return distributions are unknown.
In this ofﬂine setting, we propose to instead compute a belief distribution q ( { ρ i } Ni =1 ) , and then we can optimize over the expected ranking score, i.e. , ˜ O
[Equation]
as shown in Algorithm 1.
This algorithm computes the Bayes risk by simulating realizations of the groundtruth values { ρ i } Ni =1 with samples from the belief distribution q ( { ρ i } Ni =1 ) , and in this way estimates the expected realized ranking score S over all possible rankings O .
As we will show empirically, matching the Bayes selection process (the S used in Algorithm 1) to the downstream ranking score naturally leads to improved performance.
The question left now becomes how to effectively learn a belief distribution over { ρ i } Ni =1 , and this is answered by the BayesDICE algorithm.
BayesDICE
We propose BayesDICE for estimating the belief distribution over { ρ i } Ni =1 .
We ﬁrst investigate alternative characterizations of policy value to justify a representation in terms of stationary density correction ratios (generally known as DICE or marginalized importance weights).
These correction ratios are characterized by a set of constraints, one for each state-action pair, which presents a challenge for posterior inference.
However, by re-expressing Bayesian inference as an optimization, we bypass this difﬁculty via stochastic constraints , a derivation that is of independent interest.
We then apply the resulting constrained posterior inference to DICE, yielding a novel estimator that is computationally attractive while supporting a broad range of ranking scores for downstream tasks.
3.1 Alternative Representations of Policy Value
To accomplish ofﬂine policy selection one must choose a speciﬁc expression to represent the value of a policy.
There are several principal requirements for such a representation:
• Ofﬂine : Since we focus on ranking policies given only ofﬂine data, the policy value should not depend on on-policy samples or access to a known behavior policy.
Versatility : Since the downstream task may utilize different ranking scores, the policy value representation should be compatible with efﬁcient evaluation of these scores.
With these considerations in mind, we review choices for representing the value of a policy π .
Deﬁne
Q π ( s, a ) =
{+E (cid:2)(cid:80) ∞ t =
+}
γ t R ( s
t , a t ) | s 0 =
s, a 0 = a (cid:3) and d π ( s, a ) =
(1 − γ ) (cid:80) ∞
{+t =
+}
γ t d πt ( s, a ) , with d πt ( s, a ) =
P ( s t = s, a t = a | s 0 ∼ µ 0 , ∀ i < t, a i ∼ π ( ·| s i ) , s i +1 ∼ T ( ·| s i , a i )) , which are the state-action value function and stationary visitations of π .
These quantities satisfy the recursions
[Equation]
where
[Equation]
where
[Equation]
From these identities, the policy value can be expressed in two equivalent ways:
[Equation]
Current OPE methods are generally based on one of the representations (1), (5) or (6).
For example, importance sampling (IS) estimators [53, 44, 19] are based on (1); LSTDQ [37] is a representative algorithm for ﬁtting Q π and thus based on (5); the DICE algorithms [66] estimate the stationary density ratio ζ π ( s, a ) :
= d π ( s,a ) d D so that ρ ( π ) =
E d D [ ζ π · r ] , and are thus based on (6).
To reduce notational clutter, we omit the superscripted π on ζ when it is clear from context.
Among the three representations, the stationary density ratio representation fully supports the stated requirements, and hence is the most promising for the ultimate selection task.
First, IS estimators suffer from an exponential growth in variance [42] and require knowledge of the behavior policy.
By contrast, the functions Q π and d π share common minimax properties [62] and can be estimated without knowledge of the behavior policy enabling behavior-agnostic learning.
However, Q π exhibits a linear dependence on R ( s, a ) , hence, even if Q π is estimated accurately, it is still infeasible to evaluate ranking scores that involve (1 − γ ) E[-[ (cid:80) ∞ t =
-]
[ (cid:80) ∞ t =0 γ t σ ( r t )] with a nonlinear σ (unless one learns a different Q function for each possible ranking score, which may be computationally expensive).
By contrast, the stationary density ratios ζ ( s, a ) are independent of reward, which enables efﬁcient ranking on a variety of downstream ranking scores.
For example, in the case of a nonlinear utility σ , the policy value may be easily computed from the stationary density ratio as E d D [ ζ · σ ( r )] .
Based on these considerations, representing policy value via stationary density ratios best satisﬁes the requirements: it enjoys statistical advantages for ofﬂine setting [67, 29] and is ﬂexible for downstream ranking score calculation.
Therefore, we focus on developing a Bayesian estimator for ζ π .
3.2 Stationary Ratio Posterior Estimation
Typically, a posterior q ( ζ π |D ) is deﬁned in terms of a prior p ( ζ π ) and likelihood function p ( D| ζ π ) via Bayes’ rule i.e. , q ( ζ π |D )
∝ p ( D| ζ π ) p ( ζ π ) .
However, the posterior can also be equivalently expressed as the result of an optimization problem [63, 68] [Equation]
[Equation]
(8) where P is the space of valid densities.
This optimization interpretation of Bayesian inference has been generalized in well known work on posterior regularization and reguarlized Bayes [43, 41, 70], which considers more complex loss functions on ξ and richer constraints on the “posterior”
[Equation]
where P ( D , ξ ) := P ∩ Ω ( D , ξ ) with Ω ( D , ξ ) as a set deﬁned by data-dependent constraints with slack variable ξ and U ( · ) a loss function.
Although (9) can easily express (8), they key advantage is that the more general formulation allows Bayesian inference to be practically applied in scenarios when the likelihood does not have an explicit, tractable form, or when there are additional constraints that cannot be conveniently encoded in the prior or likelihood [43, 41, 70].
This framing allows us to naturally incorporate constraints arising from the stationary density ratio representation (4).
However, previous work only considers ﬁnitely many constraints on posterior expectations , while the constraints for ζ induced by (4) consider each ratio function individually on arbitrary ( s, a ) ∈ S × A , which can potentially be inﬁnitely many.
Therefore, to apply the generalized
Bayesian framework (9) to our scenario, we ﬁrst need to extend the formulation by considering a function space embedding to reduce the number of constraints to ﬁnitely many [12, 38, 11], then reformulate these as chance constraints to ensure ζ satisﬁes the constraints with high probability [47].
Constraints Embedding First, we use a function space embedding to reduce the number of constraints to ﬁnitely many [12, 38, 11].
Let ∆ d ( s, a ) :
= (1 − γ ) µ 0 ( s ) π ( a | s )+ γ ·P π ∗ d ( s, a ) − d ( s, a ) .
Consider a feature mapping φ ( · , · ) : S ×
A → R m and the induced RKHS H φ , and deﬁne (cid:104) φ, ∆ [Equation] (cid:105) : =
E
[Equation]
Then the constraints (4) can be expressed as ∆ d ( s, a ) = 0 .
We can match distributions in terms of their embeddings [57] by measuring (cid:104) φ, ∆ d (cid:105) (cid:62) (cid:104) φ, ∆ d (cid:105) , a generalization of the approximation methods in [12, 38].
In particular, when | S | | A | is ﬁnite and we set φ ( s, a ) = δ s,a , where δ s,a ∈ { 0 , 1 } | S ||
A | is an indicator vector with a single 1 at position ( s, a ) and 0 otherwise, we are matching the distributions pointwise.
The feature map φ ( s, a ) can also be set to general reproducing kernel k
(( s, a ) , · ) ∈ R ∞ .
As long as the kernel k ( · , · ) is characteristic, the embeddings will match if and only if the distributions are identical almost surely [58].
We further re-frame the constraint with Fenchel duality [46] [Equation]
[Equation]
10) resulting in the ﬁnal constraint (cid:96) ( ζ, D ) = 0 .
Chance Constraints Given that the experience is a ﬁnite sample from d D , we have to approximate (cid:96) with a sample estimator ˆ (cid:96) and the constraint for ζ in (10) might not hold exactly using ˆ (cid:96) .
However, under mild conditions, we have d π ( s,a ) d D ∈
[Equation]
= [Equation] [Equation] (cid:111) with high probability (see Appendix A for the precise statement and proof).
Thus, we expect a randomly sampled ratio ζ ∼ q ( ζ ) to be in the relaxed feasible set Ξ with high probability.
Incorporating this into (9) yields
[Equation]
where the chance constraint enforces the probability that ζ is feasible under the posterior.
This formulation can be equivalently rewritten as
[Equation]
Then, by applying Markov’s inequality, i.e. , P
[Equation]
E
[Equation]
, we can obtain an upper bound on (12) as
[Equation]
14) where the last equation follows by interchangeability [56, 10].
Note that (cid:96) ( ζ, D ) (cid:62) 0 since H φ is symmetric, so the outer optimization is lower bounded.
We amortize the optimization for β w.r.t.
each ζ to a distribution q ( β | ζ ) to reduce the computational effort.
The pseudo-code of the BayesDICE algorithm is shown in Algorithm 2.
Finally, with the posterior approximation for ζ i , denoting the estimate for candidate policy i , we can draw posterior samples of ¯ ρ i by drawing a sample ζ i ∼ q ( ζ i ) and computing ˆ ρ i =n (cid:80) ( s,a,r ) ∈D ζ i ( s, a ) r .
This deﬁnes a posterior distribution over ¯ ρ i .
For the joint posterior over { ¯ ρ i } Ni =1 we use a mean ﬁeld approximation to express it as a product of independent marginals, i.e. , q ( { ¯ ρ i } Ni =1 ) =
(cid:81)
i q (¯ ρ i ) .
This deﬁnes the necessary inputs for OfflineSelect to determine a ranking of the candidate policies.
Given the space limits, please see Appendix B and C for a discussion of other important aspects of BayesDICE, including an alternative safe surrogate of the chance constraints, parametrization of the posteriors, variants of BayesDICE for undiscounted MDPs, connections to vanilla Bayesian stochastic processes, and the application of BayesDICE to exploration.
Related work
We categorize the relevant related work into ﬁve categories: ofﬂine policy selection, ofﬂine policy optimization, off-policy evaluation, Bayesian reinforcement learning, and posterior regularization.
Inputs sampled initial states ˆ µ 0 = { s ( j ) 0 } mj =1 , ofﬂine data D = { ( s ( j ) 0 , s ( j ) , a ( j ) , r ( j ) , s (cid:48) ( j ) ) }
nj =1 , target policy π , parametrized distributions q θ 1 ( · , · ) and q θ 2 ( · , · ) , prior p , convex function f (conjugate f ∗ ), constants (cid:15), λ , learning rates η ζ , η β , training iterations T , and batch B .
T do Sample batch { ( s ( j ) , a j , r ( j ) , s (cid:48) j ) ) }
Bj =1 from D { s j ) 0 } Bj =1 from ˆ µ a (cid:48) j ) ∼ π
) and a
Sample β 0 ∼ q θ 1 ( s ( j ) a ( j ) 0 ) β ∼ q θ 1 ( s ( j ) , a ( j ) ) , β (cid:48) ∼ q θ 1 ( s (cid:48) ( j ) , a (cid:48) ( j ) ) , and ζ ∼ q θ 2 ( s ( j ) a ( j ) ) . Compute loss ˆ J = KL ( p (cid:107) q θ 1 ) + KL ( p (cid:107) q θ 2 ) + λ(cid:15)B (cid:80) Bi
=1 ( ζγ ( β − β (cid:48) ) − f ∗ ( β ))
+ (1 − γ ) β 0 .
Update θ 1 ← θ 1 + η β ∇ θ 1 ˆ J and θ 2 ← θ 2 − η ζ ∇ θ 2 ˆ J .
end return Ofﬂine policy selection The decision making problem we formalize as ofﬂine policy selection is a member of a set of problems in RL referred to as model selection . Previously, this term has been used to refer to state abstraction selection [28, 30] as well as learning algorithm and feature selection [23, 50]. More relevant to our proposed notion of policy selection are a number of previous works which use model selection to refer to the problem of choosing a near-optimal Q -function from a set of candidate approximation functions [21, 20, 27, 64].
In this case, the evaluation metric is typically deﬁned as the L ∞ norm of difference of Q versus the state-action value function of the optimal policy Q ∗ .
While one can relate this evaluation metric to the sub-optimality ( i.e. , regret) of the policy induced by the Q -function, we argue that our proposed policy selection problem is both more general – since we allow for the use of policy evaluation metrics other than sub-optimality – and more practically relevant – since in many practical applications, the policy may not be expressible as the argmax of a Q -function.
Lastly, the ofﬂine policy selection problem we describe is arguably a formalization of the problem approached in [51] and referred to as hyperparameter selection .
In contrast to this previous work, we not only formalize the decision problem, but also propose a method to directly optimize the policy selection evaluation metric.
Ofﬂine policy selection has also been studied by [17], who consider desirable properties of a point estimator to yield good rankings in terms of a notion of ranking score referred to as fairness .
Ofﬂine policy optimization While it is possible to integrate desired criteria such as pessimism into ofﬂine policy optimization [35, 6], this requires the desired criteria (e.g., maximum high-conﬁdence lower bound) to be speciﬁed prior to policy learning, which might differ from what a practitioner deploying the policy prefers (e.g., policies that achieve top- k precision or regret).
Furthermore, policies in practical applications may not be amenable to (policy)-gradient-based learning (e.g., policies with business logic and hard-coded rules).
In these cases, it is much easier to rank a set of candidate policies given a set of criteria rather than learning one policy for each criterion.
Off-policy evaluation Off-policy evaluation (OPE) is a highly active area of research.
While the original motivation for OPE was in the pursuit of policy selection [53, 28], the ﬁeld has historically almost exclusively focused on the related but distinct problem of estimating the online value (accumulated rewards) of a single target policy.
In addition to a plethora of techniques for providing point estimates of this groundtruth value [19, 4, 31, 59, 32, 45, 69, 66], there is also a growing body of literature that uses frequentist principles to derive high-conﬁdence lower bounds for the value of a policy [4, 61, 25, 36, 22, 11, 34].
As our results demonstrate, ranking or selecting policies based on either their estimated mean or lower conﬁdence bounds can at times be sub-optimal, depending on the evaluation criteria.
Bayesian reinforcement learning Our proposed method for ofﬂine policy selection relies on
Bayesian principles to estimate a posterior distribution over the groundtruth policy value.
While many Bayesian RL methods have been proposed for policy optimization [14, 52], especially in the context of exploration [26, 13, 33], relatively few have been proposed for policy evaluation.
In one instance, [21] derive PAC-Bayesian bounds on estimates of the Bellman error of a candidate Q -value function.
In contrast to this work, the BayesDICE estimates a distribution over stationary density ratio, and this distribution allows us to directly optimize arbitrary downstream policy selection metrics.
[Table]
I n
[Table]
r v a l l
[Table]
i d
[Table]
I n t
[Table]
r v a l c ov
[Table]
a
[Table]
.95 ° 3 ° 2 ° 110.6 0.7 0.8 0.9 0.
8 1 .0.6 0.7 0.8 0.9 0.95 ° 5 °° 3 ° 2 °
BayesDICE(ours)
BDQN
CoinDICEBootstrapping
BernsteinStudentt
Expectedcoverage
Distinguish distributional RL Although both distributional RL [2, 8, 7] and BayesDICE learn distributions over quantities of interest, these distributions are signiﬁcantly different and with different update rules.
Distributional RL ﬁts a distribution of returns over future trajectories, where the randomness comes from stochasticity of MDP transitions and policy action selections.
In contrast,
BayesDICE learns distributions of stationary density ratios in a Bayesian posterior sense, which captures uncertainty from both model stochasticity and ﬁnite observations, while marginalizing over any stochasticity in MDP transitions and policy action selections.
More importantly, BayesDICE is designed to serve as a component for policy selection derived via Bayes decision theory, with which distributional RL is not compatible.
Bayesian inference with posterior regularization Unlike vanilla Bayesian inference for posterior computation, the proposed BayesDICE does not rely on an explicitly computed log-likelihood, but instead estimates the posterior of the stationary density ratio by enforcing a stochastic constraint.
This formulation of BayesDICE is inspired by the functional optimization view of Bayesian inference [63, 68, 9].
There are several works introducing the data-dependent constraints or regularization to encode the side information of the posterior into the optimization framework, e.g. , generalized expectation criteria [43], learning from measurements [41], and regularized Bayes [70].
The most important difference lies in the formulation of the constraints: the existing works only considers expectation constraints/regularization , while we largely extend the framework to more general chance constraints .
Experiments
We empirically evaluate BayesDICE in estimating conﬁdence intervals (which can be used for policy selection) and ofﬂine policy selection under linear and neural network posterior parametrizations on tabular Bandit, Taxi [16], FrozenLake [5], and continuous-control Reacher [5] tasks.
As shown in Figure 2, BayesDICE outperforms existing methods for conﬁdence interval (CI) estimation based on concentration inequalities, producing accurate coverage while maintaining tight interval width, suggesting that BayesDICE achieves accurate posterior estimation in practice while being robust to approximation errors and potentially misaligned Bayesian priors.
Moreover, in ofﬂine policy selection settings, matching the selection criteria (Algorithm 1) to a variety of ranking scores (enabled by the estimated posterior) shows clear advantage over policy ranking based on point estimates or conﬁdence intervals.
See Appendix D for additional results and implementation details.
5.1 Conﬁdence interval estimation
We ﬁrst evaluate the BayesDICE approximate posterior by computing the accuracy of the credible intervals [40] it produces.
To make comparisons with previous work, we evaluate frequentist conﬁdence interval properties of BayesDICE against a known set of CI estimators based on concentration inequalities, and against CoinDICE [11], which is based-on empirical likelihood.
While the frequentist conﬁdence interval is analagous to the Bayesian credible interval , they have different statistical properties, so we expect that evaluating the credible intervals BayesDICE produces under frequentist measures will give a pessimistic estimate of its true performance.
To compute the concentrationinequality-based baselines, we follow [11] by ﬁrst using weighted ( i.e. , self-normalized) per-step importance sampling [59] to obtain a policy value estimate for each logged trajectory.
These trajectories provide a ﬁnite sample of value estimates.
We use self-normalized importance sampling in MDP [Figure] Corr. [Figure]
[Figure]
α =
[Figure]
Corr.
Top- 3, α =
[Figure]
Corr.
Top- 3, α =0.
environments (which has been found to yield better empirical results on these tasks [42, 45] despite being biased).
We then use empirical Bernstein’s inequality [61], bias-corrected bootstrap [60], and
Student’s t-test to derive lower and upper high-conﬁdence bounds on these estimates.
We further consider Bayesian Deep Q-Networks (BDQN, only applicable to function approximation) [1] with an average empirical reward prior in the function approximation setting.
BDQN applies Bayesian linear regression to the last layer of a deep Q-network to learn a distribution of Q-values.
Both BayesDICE and BDQN output a distribution of parameters, from which we conduct Monte Carlo sampling and use the resulting samples to compute a credible interval at a given conﬁdence level.
We plot the empirical coverage and interval width at different conﬁdence levels in Figure 2.
To compute the empirical interval coverage , we conduct 200 trials with randomly sampled datasets.
The interval coverage is the proportion of the 200 intervals that contains the true value of the target policy.
The interval log-width is the median of the log width of the 200 intervals.
As shown in Figure 2,
BayesDICE’s coverage closely follows the intended coverage (black dotted line), while maintaining narrow interval width across all tasks.
5.2 Policy selection
Next, we demonstrate the beneﬁt of matching the policy selection criteria to the ranking score in ofﬂine policy selection.
Our evaluation is based on a variety of cardinal and ordinal ranking scores deﬁned in Section 2.1.
We begin by considering the use of Algorithm 1 with BayesDICEapproximated posteriors.
By keeping the BayesDICE posterior ﬁxed, we focus our evaluation on the performance of Algorithm 1.
We plot the groundtruth performance of this procedure applied to Bandit and Reacher in Figure 3 (left).
These ﬁgures compare using different ˆ S to rank the policies according to Algorithm 1 across different downstream ranking scores S .
We ﬁnd that aligning the criteria ˆ S used in Algorithm 1 with the downstream ranking score S is empirically the best approach ( ˆ S = S ).
In contrast, using point estimates such as Mean or Mean ± Std can yield much worse downstream performance.
We also see that in the Bandit setting, where we can analytically compute the Bayes-optimal ranking, using aligned ranking scores in conjunction with
BayesDICE-approximated posteriors achieves near-optimal performance.
Having established BayesDICE’s ability to compute accurate posterior distributions as well as the beneﬁt of appropriately aligning the ranking score used in Algorithm 1, we compare BayesDICE to state-of-the-art OPE methods in policy selection.
In these experiments, we use Algorithm 1 with posteriors approximated by BayesDICE and ˆ S = S .
We compare the use of BayesDICE in this way to ranking via point estimates of DualDICE [45] and other conﬁdence-interval estimation methods introduced in Section 5.1.
We present results in Figure 3, in terms of top- k regret and correlation on
Bandit and Reacher tasks across different sample sizes and behavior data.
BayesDICE outperforms other methods on both tasks.
See additional ranking results in Appendix D.
Conclusion
In this paper, we formally deﬁned the ofﬂine policy selection problem, and proposed BayesDICE to ﬁrst estimate posterior distributions of policy values before using a simulation-based procedure to compute an optimal policy ranking.
Empirically, BayesDICE not only provides accurate belief distribution estimation, but also shows excellent performance in policy selection tasks.
Extending
BayesDICE to estimating a posterior distribution over return distributions (instead of the expected return) is an important direction of future research.
Acknowledgments and Disclosure of Funding
Use unnumbered ﬁrst level headings for the acknowledgments.
All acknowledgments go at the end of the paper before the list of references. Moreover, you are required to declare funding (ﬁnancial activities supporting the submitted work) and competing interests (related ﬁnancial activities outside the submitted work). More information about this disclosure can be found at: https://neurips.cc/Conferences/2021/PaperInformation/FundingDisclosure . Do not include this section in the anonymized submission, only in the ﬁnal paper.
You can use the ack environment provided in the style ﬁle to autmoatically hide this section in the anonymized submission.
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Future direction in Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix A (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code submitted in supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix D (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See caption of Figure 2 and Figure(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section(b)
Did you mention the license of the assets?
[Yes] The license is Apache License 2.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
A Proofs for Finite Sample Relaxation
The following lemma will be needed.
Lemma 1.
[[55], Lemma 4] Let X
=
{ x i } ni
=1 be i.i.d.
random variables in a ball H of radius C centered around the origin in a Hilbert space.
Denote their average by ¯ x = 1 n (cid:80) ni
=1 x i .
Then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
Theorem 2.
Denote ζ ∗ ( s, a ) =
[Equation] [Equation] D which is bounded by C ζ , under the assumption that (cid:107) φ (cid:107)
(cid:54)
C φ , (cid:107) β (cid:107) 2 (cid:54) C β , ∀ β ∈ H β and f is L f -Lipschitz continuous, then ζ ∗ ∈
Ξ := { ζ : (cid:96) ( ζ, D ) (cid:54) (cid:15) } with probability 1 − exp (cid:16) − n(cid:15) 2 2 C (cid:17) with C :
= (1 + γ ) (1 + C ζ ) C β C φ + L f C β .
Proof.
Let
[Equation]
[Equation]
, and
[Equation]
a (cid:48) ) − φ ( s, a [Equation]
− f ∗ ( β ) (cid:3) .
We also denote ˆ β = argmax β ∈H φ ι ( ζ, D , β ) .
Following the discussion in footnote 2 in main text, the D ∼ d D i.i.d.
, it is obvious that E [ ι ( ζ, D , β )]
= ι (cid:0) ζ, d D , β (cid:1) .
Under the bounded assumption of ( β, φ ) , we can bound (cid:107) ι (cid:107) ∞ (cid:54) C .
Therefore, by Lemma 1, we have
[Equation]
Since ζ ∗ ( s, a ) =
d π ( s,a ) d D , we have ι (cid:0) ζ ∗ , d D , β (cid:1) = 0 , ∀ β ∈
H φ .
Finally, recall max β ∈H φ ι ( ζ, D , β ) (cid:62) 0 since H φ is symmetric.
We achieve the conclusion.
B More Discussions on BayesDICE
In this section, we provide more details about BayesDICE.
Remark (Alternative safe surrogates of chance constraints):
We apply the Markov’s inequality to (12) for the upper bound (13).
In fact, the optimization with chance constraints has rich literature [3], where plenty of surrogates can be derived with different safe approximations.
For example, if the parametrization of q is simple, one can directly calculate the CDF for the probability P q ( (cid:96) ( ζ, D ) (cid:54) (cid:15) ) ; or one can also exploit different probability inequalities to derive other surrogates, e.g. , condition value-at-risk, i.e. , min
[Equation]
and Bernstein approximation [47].
These surrogates lead to tighter approximation to the chance probability P q ( (cid:96) ( ζ ) (cid:54) (cid:15) ) with the extra cost in optimization.
Remark (parametrization of q ( ζ ) and q ( β | ζ ) ):
We parametrize both q ( ζ ) (and the resulting q ( β | ζ ) ) as Gaussians with the mean and variance approximated by a multi-layer perceptron (MLP), i.e. : ζ = MLP w ( s, a ) + σ w (cid:48) ξ, ξ ∼ N (0 , 1) .
w and w (cid:48) denote the parameters of the MLP.
Remark (connection to Bayesian inference for stochastic processes): Recall the posterior can be viewed as the solution to an optimization [63, 68, 70, 9],
[Equation]
Then (13) mathematically equivalent to deﬁne a log -likelihood log p ( D| ζ ) ∝ (cid:96) ( ζ, D ) , where p ( D| ζ ) is a Gibbs point process [15, 65].
For example, plug f ( β ) =
12 β (cid:62) β back into (13), we have β ∗ =
ˆ E D [ ζ ( s, a ) · ( γφ ( s (cid:48) , a (cid:48) ) − φ ( s, a ))] + (1 − γ ) E µ 0 π
[ φ ] , resulting the optimization
[Equation]
16) with the kernel k (( s 1 , a 1 , s
[Equation]
[Equation]
and h
[Equation]
[Equation]
φ
[Equation]
s [Equation] If the prior p ( ζ ) is a GP , the posterior q ( ζ |D ) will also a GP .
Obviously, with different choices of f ∗ ( · ) , the BayesDICE framework is far beyond GP .
However, we emphasize although the model deﬁne via stochastic processes likelihood in (16) acheives the equivalent optimization, such a likelihood p ( D| ζ ) is improper in the causality sense as we discussed in Section 3.
Remark (auxilary constraints and undiscounted MDP): As [66] suggested, the non-negative and normalization constraints are important for optimization.
We use positive activation functions (ReLU) to ensure the non-negativity of the mean of the q ( ζ ) .
For the normalization, we consider the chance constraints P (cid:18)(cid:16)
ˆ E D ( ζ ) − 1 (cid:17) 2 (cid:54) (cid:15) 1 (cid:19) (cid:62) ξ 1 .
By applying the same technique, it leads to an extra term λ 1 (cid:15) 1 E q (cid:104) max α ∈ R
α · ˆ E D [ ζ − 1] (cid:105) in (13).
With the normalization condition introduced, the proposed BayesDICE is ready for undiscounted
MDP by simply setting γ = 1 in (13) together with the above extra term for normalization.
C BayesDICE for Exploration vs. Exploitation Tradeoff
In main text, we mainly consider exploiting BayesDICE for estimating various ranking scores for both discounted MDP and undiscounted MDP.
In fact, with the posterior of the stationary ratio computed, we can also apply it for better balance between exploration vs. exploitation for policy optimization.
Instead of selecting from a set of policy candidates, the policy optimization is considering all feasible policies and selecting optimistically.
Speciﬁcally, the feasibility of the stationary state-action distribution can be characterized as (cid:88)
[Equation] [Equation]
where P ∗ d ( s ) :
= (cid:80) ¯ s, ¯ a T ( s | ¯ s, ¯ a ) d (¯ s, ¯ a ) .
Apply the feature mapping for distribution matching, we obtain the constraint for ζ · π with ζ ( s, a ) : = d [Equation] s
[Equation]
d D [Equation] s,a ) as max
[Equation]
Then, we have the posteriors for all valid policies should satisﬁes
[Equation]
19) with (cid:96) ( ζ · π, D ) :
= max β ∈H φ β (cid:62) ˆ E D
[ (cid:80) a ( ζ ( s, a ) π ( a | s )) φ ( s ) − γ ( ζ ( s, a ) π ( a | s )) φ ( s (cid:48) )]
+ (1 − γ ) E µ 0 (cid:2) β (cid:62) φ (cid:3) − f ∗ ( β ) .
Meanwhile, we will select one posterior from among these posteriors of all valid policies optimistically, i.e. , max
[Equation] [Equation]
(21) where E q [ U ( τ, r, D )] denotes the optimistic policy score to capture the upper bound of the policy value estimation.
For example, the most widely used one is
[Equation]
where the second term is the empirical variance and usually known as one kind of “exploration bonus”.
Then the whole algorithm is iterating between solving (20) and use the obtain policy collecting data into D in (20).
This Exploration-BayesDICE follows the same philosophy of [49, 48] where the variance of posterior of the policy value is taken into account for exploration.
However, there are several signiﬁcant differences: i), the ﬁrst and most different is the modeling object, [49, 48] is updating with Qfunction, while we are handling the dual representation; ii), BayesDICE is compatible with arbitary nonlinear function approximator, while [49, 48] considers tabular or linear functions; iii), BayesDICE is considering inﬁnite-horizon MDP, while [49, 48] considers ﬁxed ﬁnite-horizon case.
Therefore, the exploration with BayesDICE pave the path for principle and practical exploration-vs-exploitation algorithm.
The regret bound is out of the scope of this paper, and we leave for future work.
D Experiment details and additional discussion and results D.1 Environments and policies.
Bandit.
We create a Bernoulli two-armed bandit with binary rewards where α controls the proportion of optimal arm ( α = 0 and α = 1 means never and always choosing the optimal arm respectively).
Our policy selection experiments are based on 5 target policies with α =
Reacher.
We modify the Reacher task to be inﬁnite horizon, and sample trajectories of lengthin the behavior data.
To obtain different behavior and target policies, We ﬁrst train a deterministic policy from OpenAI Gym [5] until convergence, and deﬁne various policies by converting the optimal policy into a Gaussian policy with optimal mean with standard deviation 0 .
4 − 0 .
3 α .
Our selection experiments are based on 5 target policies with α =
D.2 Parametrization Details
For the convex function f in (14), we used f ( x ) = x 2 .
We parametrize the distribution correction ratio as a Gaussian using a deep neural network for the continuous control task.
Speciﬁcally, we use feed-forward networks with two hidden-layers of 64 neurons each and ReLU as the activation function.
The networks are trained using the Adam optimizer ( β 1 = 0 .
99 , β 2 = 0 .
999 ) with batch size 2048 and learning rate 0 .
0001 on CPUs.
D.3 Additional empirical discussions
BayesDICE v.s. CoinDICE.
Because BayesDICE is a Bayesian method, it produces credible intervals .
While the credible interval is analogous to the frequentist conﬁdence interval, it has different statistical properties, so it is unsurprising
that evaluating the credible intervals BayesDICE produces under frequentist measures favors frequentist methods like CoinDICE.
The beneﬁt of
BayesDICE is its applicability and superior performance for policy selection with arbitrary criteria.
Function approximation in BayesDICE.
Constraint embedding can be generalized to use neural network function approximators with potential approximation error.
Speciﬁcally, as long as the inner product is well-deﬁned, we can characterize the constraints with max f ∈F (cid:104) f, ∆ (cid:105) =
where
F , i.e., testing function space, can be composed of neural networks.
The solution is then known as a “weak solution” in differential equations and ﬁnite-element methods.
The approximation error induced by such embedding depends on the ﬂexibility of the testing function space.
The theoretical analysis considers an idealized scenario which provides guidance.
In practice, however, the limited expressibility of the function approximators used, relaxed constraints, and inexact optimization introduce approximation errors, which are challenging to quantify analytically.
Empirically, Figureshows that BayesDICE parametrized by kernel and neural network exhibit similar performance, demonstrating the practical effectiveness of neural network as function approximators.
Choice of the prior.
The prior of the ratio variables is chosen to be unit Gaussian.
We conducted experiments where the prior mean ranges from [0 . 1 , 10] and prior variance ranges from [0 . 1 , 1] , and observed the resulting conﬁdence intervals to be similar to those in the paper.
Reacher (# trajectories = 25) 0.6 0.7 0.8 0.9 0.95 °°°°
°0.6 0.7 0.8 0.9 0.95 0 .0 .0 .0 .0 .
Interval log-width Interval coverage
BayesDICE w. kernel
BayesDICE w. neural network
Expected coverage
BayesDICE w. kernel
BayesDICE w. neural network
Expected coverage
BayesDICE w. kernel
BayesDICE w. neural network
Expected coverage
Choice of approximate posterior.
We chose a Gaussian variational posterior for simplicity.
A downside of this choice is that sampled correction ratio can be negative.
In practice, we found that is rarely the case, and Gaussian posterior was sufﬁcient to achieve strong performance.
Moreover, BayesDICE can naturally incorporate advanced parameterizations, e.g., ﬂow and stochastic differential equations which can ensure positivity.
Comparison to point estimators.
The posterior mean estimate of BayesDICE differs from the point estimate in DualDICE due to the prior (i.e., regularization).
We summarize the average (across 10 seeds) log RMSE of DualDICE (pt) and of the mean estimate from BayesDICE ( µ ) on Bandit (B),
FrozenLake (F), Taxi (T) and Reacher (R) with varying number of trajectories in the table below.
For our choice of prior and these tasks, the performance of the point and mean estimators are similar.
B
B B
F F
T T
R pt -4.
µ -7.
Scalability of BayesDICE.
Depending on the evaluation metric chosen, its structural properties can be exploited to nullify the need to test all permutations in Algorithm 1.
Such structural properties are present in many natural metrics (such as top- k precision or regret).
Therefore, BayesDICE can easily scale to larger numbers of candidate policies
[Figure]
D.4 Additional experimental results
[Figure]
BayesDICE and CoinDICE still perform much better than methods based on concentration inequality.
[Figure] [Figure]
[Figure] [Figure]