Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
Anonymous Author(s)
AffiliationAddress email
Abstract
Model-based offline reinforcement learning (RL) aims to find highly rewardingpolicy, by leveraging a previously collected static dataset and a learned dynamicsmodel.
While the dynamics model is learned by reusing the static dataset, itsgeneralization ability hopefully promotes policy learning if properly utilized.
Tothat end, several works propose to quantify the uncertainty of predicted dynamics,and explicitly apply it to penalize reward.
However, as the dynamics has differentimplication than the reward, characterizing the impact of dynamics uncertaintythrough reward penalty may incur unexpected tradeoff between model utilizationand risk avoidance.
In this work, we instead maintain a belief distribution overdynamics, and evaluate/optimize policy through biased sampling from the belief.
The sampling procedure, biased towards pessimism, is derived based on an alternating Markov game formulation of offline RL.
We formally show that the biasedsampling naturally induces an updated dynamics belief with policy-dependentreweighting factor, termed Pessimism-Modulated Dynamics Belief .
To improve policy, we devise an iterative regularized policy optimization algorithm for the game,with guarantee of monotonous improvement under certain condition.
To makepractical, we further devise an offline RL algorithm to approximately find the solution.
Empirical results show that the proposed approach achieves state-of-the-artperformance on a wide range of offline RL benchmark tasks.
Introduction
In typical paradigm of RL, the agent actively interacts with environment and receives feedback topromote policy improvement.
The essential trial-and-error procedure can be costly, unsafe or evenprohibitory in practice (e g. robotics [1], autonomous driving [2] and healthcare [3]), thus constitutinga major impediment to actual deployment of RL.
Meanwhile, for a number of applications, historicaldata records are available to reflect the system feedback under a predefined policy.
This raises theopportunity to learn policy in purely offline setting.
In offline setting, as no further interaction with environment is permitted, the dataset provides a limitedcoverage in state-action space.
Then, the policy that induces out-of-distribution (OOD) state-actionpairs can not be well evaluated in offline learning phase, and deploying it online potentially attainsterrible performance.
Recent studies have reported that applying vanilla RL algorithms to offlinedataset exacerbates such a distributional shift [4–6], making them unsuitable for offline setting.
To tackle the distributional shift issue, a number of offline RL approaches have been developed.
Specifically, one category of them propose to directly constrain the policy close to the one collectingdata [4, 5, 7, 8], or penalize Q-value towards conservatism for OOD state-action pairs [9–11].
Whilethey achieve remarkable performance gains, the policy regularizer and the Q-value penalty tightlyrestricts the produced policy within the data manifold.
Instead, more recent works consider to quantify the uncertainty of Q-value with neural network ensembles [12], where the consistent Q-valueestimates indicate high confidence and can be plausibly used during learning process, even for OODstate-action pairs [13, 14].
However, the uncertainty quantification over OOD region highly relieson how neural network generalizes [15].
As the prior knowledge of Q-function is hard to acquireand insert into the neural network, the generalization is unlikely reliable to facilitate meaningfuluncertainty quantification [16].
Notably, all these works are model-free.
Model-based offline RL optimizes policy based on a constructed dynamics model.
Compared to themodel-free approaches, one prominent advantage is that the prior knowledge of dynamics is easierto access.
First, the generic prior like smoothness widely exists in various domains [17].
Second,the sufficiently learned dynamics models for relevant tasks can act as a data-driven prior for theconcerned task [18–20].
With richer prior knowledge, the uncertainty quantification for dynamics ismore trustworthy.
Similar to the model-free approach, the dynamics uncertainty can be incorporatedto find reliable policy beyond data coverage.
However, an additional challenge is how to characterizethe accumulative impact of dynamics uncertainty on the long-term reward, as the system dynamics iswith entirely different meaning compared to the reward or Q-value.
Although existing model-based offline RL literature theoretically bounds the impact of dynamicsuncertainty on final performance, their practical variants characterize the impact through rewardpenalty [6, 21, 22].
Concretely, the reward function is penalized by the dynamics uncertainty for eachstate-action pair [21], or the agent is enforced to a low-reward absorbing state when the dynamicsuncertainty exceeds a certain level [6].
While optimizing policy in these constructed MDPs stimulatesanti-uncertainty behavior, the final policy tends to be over-conservative.
For example, even thetransition dynamics for a state-action pair is ambiguous among several possible candidates, thesecandidates may generate the states from which the system evolves similarly.
Then, such a state-actionpair should not be treated specially.
For a more detailed discussion of related works, see Appendix A.
Motivated by the above intuition, we propose pessimism-modulated dynamics belief for model-basedoffline RL.
In contrast with the previous approaches, the dynamics uncertainty is not explicitlyquantified.
To characterize its impact, we maintain a belief distribution over system dynamics, andthe policy is evaluated and optimized through biased sampling from it.
The sampling procedure,biased towards pessimism, is derived based on an alternating Markov game (AMG) formulation ofoffline RL.
We formally show that the biased sampling naturally induces an updated dynamics beliefwith policy-dependent reweighting factor, termed Pessimism-Modulated Dynamics Belief .
Besides,the degree of pessimism is monotonously determined by the hyperparameters in sampling procedure.
The considered AMG formulation can be regarded as a generalization of robust MDP, which isproposed as an surrogate to optimize the percentile performance in face of dynamics uncertainty[23, 24].
However, robust MDP suffers from two significant shortcomings: 1)
The percentile criterionis over-conservative since it fixates on a single pessimistic dynamics instance [25, 26]; 2)
Robust
MDP is constructed based on an uncertainty set, and the improper choice of uncertainty set wouldfurther aggravate the degree of conservatism [27, 28].
The AMG formulation is kept from theseshortcomings.
To solve the AMG, we devise an iterative regularized policy optimization algorithm,with guarantee of monotonous improvement under certain condition.
To make practical, we furtherderive an offline RL algorithm to approximately find the solution, and empirically evaluate it on theoffline RL benchmark D4RL.
The results show that the proposed approach obviously outperformsprevious state-of-the-art (SoTA) in 9 out of 18 environment-dataset configurations and performscompetitively in the rest, without tuning hyperparameters for each task.
The proof of theorems in thispaper are presented in Appendix B.
Preliminaries
Markov Decision Process (MDP)
A MDP is depicted by the tuple ( S , A , T, r, ρ 0 , γ ) , where S , Aare state and action spaces, T ( s ′ | s, a ) is the transition probability, r ( s, a ) is the reward function,ρ 0 ( s ) is the initial state distribution, and γ is the discount factor.
The goal of RL is to find the policyπ : s → ∆( A ) that maximizes the cumulative discounted reward:
[Equation]
where ∆( · ) denotes the probability simplex.
In typical RL paradigm, this is achieved via activelyinteracting with environment.
Offline RL In offline setting, the environment is unaccessible, and only a static dataset D ={ ( s, a, r, s ′ ) } is provided, containing the previously logged data samples under an unknown behaviorpolicy.
Offline RL aims to optimize the policy by solely leveraging the offline dataset.
To simplify the presentation, we assume the reward function r and initial state distribution ρ 0 areknown.
Then, the system dynamics is unknown only in terms of the transition probability T .
Notethat the considered formulation and the proposed approach can be easily extend to the general casewithout additional technical modification.
Robust MDP With the offline dataset, a straightforward strategy is first learning a dynamics modelτ ( s ′ | s, a ) and then optimizing policy via simulation.
However, due to the limitedness of available data,the learned model is inevitably imprecise.
Robust MDP [23] is an surrogate to optimize policy withconsideration of the ambiguity of dynamics.
Concretely, robust MDP is constructed by introducing anuncertainty set T = { τ } to contain plausible transition probabilities.
If the uncertainty set includesthe true transition with probability of (1 − δ ) , the performance of any policy π in true MDP canbe lower bounded by min τ ∈T J ( π, τ ) with probability of at least (1 − δ ) .
Thus, the percentileperformance for the true MDP can be optimized by finding a solution tomax π min τ ∈T J ( π, τ ) .
Despite its popularity, Robust MDP suffers from two major shortcomings: First, the percentilecriterion overly fixates on a single pessimistic transition instance, especially when there are multipleoptimal policies for this transition but they lead to dramatically different performance for othertransitions [25, 26].
This behavior results in unnecessarily conservative policy.
Second, the level of conservatism can be further aggravated when the uncertainty set is inappropriatelyconstructed [27].
For a given policy π , the ideal situation is that T contains the (1 − δ ) proportionof transitions with which the policy achieves higher performance than with the other δ proportion.
Then, min τ ∈T J ( π, τ ) is exactly the δ -quantile performance.
This requires the uncertainty set tobe policy-dependent, and during policy optimization the uncertainty set should change accordingly.
Otherwise, if T is predetermined and keeps fixed, it is possible to have τ ′ / ∈ T with non-zeroprobability and satisfying J ( π ∗ , τ ′ ) >
min τ ∈T J ( π ∗ , τ ) , where π ∗ is the optimal policy for (2).
Then, adding τ ′ into T does not affect the optimal solution of the problem (2).
This indicates thatwe are essentially optimizing a δ ′ -quantile performance, where δ ′ can be much smaller than δ .
Inliterature, the uncertainty sets are mostly predetermined before policy optimization [23, 29–31].
Pessimism-Modulated Dynamics Belief
In short, robust MDP is over-conservative due to the fixation on a single pessimistic transition instanceand the predetermination of uncertainty set.
In this work, we strive to take the entire spectrum ofplausible transitions into account, and let the algorithm by itself determine which subset deservesmore attention.
To this end, we consider an alternating Markov game formulation of offline RL, basedon which the proposed offline RL approach is derived.
3.1 Formulation
Alternating Markov game (AMG)
The AMG is a specialization of two-player zero-sum game,depicted by ( S , ¯ S , A , ¯ A , G, r, ρ 0 , γ ) .
The game starts from a state sampled from ρ 0 , then two playersalternatively choose actions a ∈ A and ¯ a ∈ ¯
A under states s ∈ S and ¯ s ∈ ¯ S , along with the gametransition defined by G (¯ s | s, a ) and G ( s | ¯ s, ¯ a ) .
At each round, the primary player receives rewardr ( s, a ) and the secondary player receives its negative counterpart − r
( s, a ) .
Offline RL as AMG We formulate the offline RL problem as an AMG, where the primary playeroptimizes a reliable policy for our concerned MDP in face of stochastic disturbance from thesecondary player.
The AMG is constructed by augmenting the original MDP.
As both have thetransition probability, we use game transition and system transition to differentiate them.
For the primary player, its state space S , action space A and reward function r ( s, a ) are same withthose in the original MDP.
After the primary player acts, the game emits a N -size set of systemtransition candidates T sa .
Formally, the set is generated according to
[Equation]
where P saT is a given belief distribution of system transition τ ( ·| s, a ) under state-action pair ( s, a ) , andwe use τ sa ( · ) to re-denote τ ( ·| s, a ) for short.
According to (3), the elements in T sa are independentand identically distributed samples following P saT .
The major difference to uncertainty set in robust
MDP is that the set introduced here is unfixed and stochastic for each step.
To distinguish withuncertainty set we call it candidate set, but to avoid redundancy we reuse the notation of T .
The beliefdistribution P saT can be chosen arbitrarily to incorporate knowledge of system transition.
Particularly,when the prior distribution of system transition is accessible, P saT can be obtained as the posterior byintegrating the prior and the evidence D through Bayes’ rule.
For the secondary player, it receives the candidate set T sa as state.
Thus, we can denote its statespace by ¯ S =
∆ N ( S ) , i.e., an n-fold Cartesian product of probability simplex over S .
Note that thisstate T sa also takes the role of action space, i.e., ¯ A =
T sa , meaning that the action of secondaryplayer is to choose a system transition from the candidate set.
Given the chosen τ sa ∈ T sa , the gameevolves by sampling τ sa , i.e.,
[Equation]
and the primary player receives s ′ to make the game continue.
In the following, we omit thesuperscript sa in τ sa , T sa and P saT when it is clear from the context.
We also use P NT ( T sa ) tocompactly denote the game transition G (¯ s = T sa | s, a ) in (3).
For the above AMG, we consider a specific policy (explained below) for the secondary player, suchthat the cumulative discounted reward of the primary player with policy π can be written as:
[Equation]
where the subscripts of τ and T denote time step, the expectation is over s t ∼ ρ 0 orτ t − 1 ( ·| s t − 1 , a t − 1 ) , a t ∼ π ( ·| s t ) and T t ∼ P NT , and the operator ⌊ min ⌋ kx ∈X f ( x ) denotes findingk th minimum of f ( x ) over x ∈ X .
The policy of secondary player is implicitly defined by theoperator ⌊ min ⌋ kx ∈X f ( x ) .
When changing k ∈ { 1 , 2 , · · · , N } , the secondary player exhibits variousdegree of adversarial or aggressive disturbance to the future reward.
From the view of original MDP,this behavior raises flexible tendency ranging from pessimism to optimism when evaluating policy π .
The distinctions between the above AMG and robust MDP can be highlighted in the two aspects:1) With a belief distribution over transitions, robust MDP will select only part of its supports withnon-zero probability into uncertainty set, and the set elements are treated indiscriminatingly.
Itmeans that both the possibility of transitions out of the uncertainty set and the relative likelihood oftransitions within the uncertainty set are discarded.
However, in the AMG, the candidate set simplycontains samples drawn from the belief distribution, implying no information drop in an averagesense.
Intuitively, by keeping richer knowledge of the system, the performance evaluation is moreexact and away from excessive conservatism; 2) In robust MDP, the level of conservatism is expectedto be controlled by its hyperparameter δ .
However, as illustrated in Section 2, a smaller δ does notnecessarily corresponds to a more conservative performance evaluation, because of the extra impactfrom uncertainty set construction.
In contrast, for the AMG, the degree of conservatism is adjustedby the size of candidate size N and the order of minimum k .
When changing k or N , the impact toperformance evaluation is known for certain, as formalized in Theorem 3.
To evaluate J ( π ) , we define the following Bellman backup operator:
[Equation]
As the operator depends on N , k and we emphasize pessimism in offline RL, we call it ( N, k ) -pessimistic
Bellman backup operator.
Compared to the standard Bellman backup operator in Qlearning, B πN,k additionally includes the expectation over T ∼ P NT and the k -minimum operator overτ ∈ T .
Despite these differences, we prove that B πN,k is still a contraction mapping, based on which
J ( π ) can be easily evaluated.
Theorem 1 (Policy Evaluation) .
The ( N, k ) -pessimistic Bellman backup operator B πN,k is a contraction mapping.
By starting from any function Q : S × A → R and repeatedly applying B πN,k , thesequence converges to Q πN,k , with which we have J ( π ) =
E ρ 0 ,π (cid:2) Q πN,k ( s 0 , a 0 ) (cid:3) .
3.2 Pessimism-Modulated Dynamics Belief
With the converged Q-value, we are ready to establish a more direct connection between the AMGand the original MDP.
The connection appears as the answer to a natural question: the calculation of(6) encompasses biased samples from the dynamics belief distribution, can we treat these samplesas the unbiased ones sampling from another belief distribution?
We give positive answer in thefollowing theorem.
Theorem 2 (Equivalent MDP with Pessimism-Modulated Dynamics Belief) .
The alternating Markovgame in (5) is equivalent to the MDP with tuple ( S , A , (cid:101) T, r, ρ 0 , γ ) , where the transition probability(cid:101) T ( s ′ | s, a ) =
E (cid:101) P saT [ τ sa ( s ′ )] is defined with the reweighted belief distribution (cid:101) P saT : [Equation] [Equation]
[Equation]
and F ( · ) is cumulative density function.
Furthermore, the value of w ( x ; k, N ) first increases andthen decreases with x , and its maximum is obtained at the k − 1 N − 1 quantile, i.e., x ∗ =
F − 1 (cid:16) k − 1 N − 1 (cid:17) .
In right-hand side of (7), τ sa itself is random following the belief distribution, thus 194 E τ sa ,π (cid:2) Q πN,k ( s ′ , a ′ ) (cid:3) , as a functional of τ sa , is also a random variable, and its cumulative densityfunction is determined by the belief distribution P saT .
Intuitively, we can treat E τ sa ,π (cid:2) Q πN,k ( s ′ , a ′ )
(cid:3)as a pessimism indicator for transition τ sa , with larger value indicating less pessimism.
From Theorem 2, the maximum of w is obtained at τ ∗ : F (cid:16) E τ ∗ ,π (cid:2) Q πN,k ( s ′ , a ′ ) (cid:3)(cid:17) =
k − 1 N − 1 , i.e., thetransition with k − 1 N − 1 -quantile pessimism indicator.
Besides, when E τ sa ,π (cid:2) Q πN,k ( s ′ , a ′ ) (cid:3) departs thek − 1 N − 1 quantile, the reweighting coefficient for its τ sa decreases.
Considering the effect of w to(cid:101) P saT and the equivalence between the AMG and the refined MDP, we can say that J ( π ) is a softpercentile performance.
Compared to the standard percentile criteria, J ( π ) is derived by reshapingbelief distribution towards concentrating around a certain percentile, rather than fixating on a singlepercentile point.
Due to this feature, we term (cid:101) P saT Pessimism-Modulated Dynamics Belief (PMDB) .
Lastly, recall that all the above derivations are with hyperparameters k and N , we present themonotonicity of Q πN,k over them in Theorem 3.
Furthermore, by combining Theorem 1 with Theorem3, we conclude that J ( π ) decreases with N and increases with k .
Theorem 3 (Monotonicity) .
The converged Q-function Q πN,k are with the following properties:
• Given any k , the Q-function Q πN,k element-wisely decreases with N ∈ { k, k + 1 , · · · } .
• Given any N , the Q-function Q πN,k element-wisely increases with k ∈ { 1 , 2 , · · · , N } .
• The Q-function Q πN,N element-wisely increases with N .
k 1 2 3 4 5 6
N
RobustMDP .
M BRL
N=k
Remark 1 (Special Cases).
For N = k = 1 , we have(cid:101)
P saT =
P saT .
Then, the performance is evaluated through sampling the initial belief distribution.
This resembles the common methodology in model-based RL (MBRL), with dynamics belief defined by the uniform distribution over dynamicsmodel ensembles.
For k = δ ( N − 1) + 1 and N → ∞ , (cid:101) P saTasymptotically collapses to be a delta function.
Then, J ( π )degrades to fixate on a single transition instance.
It is equivalent to the robust MDP with the uncertainty set constructedas (cid:110)
τ
sa : P saT ( τ sa ) > 0 , E τ sa ,π (cid:2) Q πN,k ( s ′ , a ′ ) (cid:3) ≥ F − 1 ( δ ) (cid:111) .
In this sense, the AMG is a successive interpolation betweenmodel-based RL and robust MDP.
Policy Optimization with Pessimism-Modulated Dynamics Belief
In this section, we optimize policy by maximizing J ( π ) .
The major consideration is that themethodology should adapt well for both discrete and continuous action spaces.
In continuous settingof MDP, the policy can be updated by following stochastic/deterministic policy gradient [32, 33].
However, for the AMG, evaluating J ( π ) itself involves an inner dynamic programming procedureas in Theorem 1.
As each evaluation of J ( π ) can only produce one exact gradient, it is inefficientto maximize J ( π ) via gradient-based method.
Instead, in this section we consider a series of subproblems with Kullback–Leibler (KL) regularization.
Solving each sub-problem makes prominentupdate to the policy, and the sequence of solutions for sub-problems monotonously improve regarding J ( π ) .
Based on this idea, we further derive offline RL algorithm to approximately find the solution.
4.1 Iterative Regularized Policy Optimization
Define the KL-regularized return for the AMG by
[Equation]
(9) where α ≥ 0 is the strength of regularization, and µ is a reference policy to keep close with.
KL-regularized MDP is considered in previous works to enhance exploration, improve robustnessto noise or insert expert knowledge [34–38].
Here, the idea is to constrain the optimized policy inneighbour of a reference policy so that the inner problem is adequately evaluated for such a smallpolicy region.
To optimize ¯ J ( π ; µ ) , we introduce
the soft ( N, k ) -pessimistic
Bellman backup operator:
[Equation]
Theorem 4 (Regularized Policy Optimization) .
The soft ( N, k ) -pessimistic Bellman backup operator¯ B ∗ N,k is a contraction mapping.
By starting from any function Q : S × A → R and repeatedlyapplying ¯ B ∗ N,k , the sequence converges to ¯ Q ∗ N,k , with which the optimal policy for ¯ J ( π ; µ ) is obtainedas ¯ π ∗ ( a | s [Equation] ∝ µ [-( a | s ) exp (cid:16) 
-][Equation]
a [Equation]
Apparently, the solved policy ¯ π ∗ depends on the reference policy µ , and setting µ arbitrarily aforehandcan result in suboptimal policy for J ( π ) .
In fact, we can construct a sequence of sub-problems with theobjective ¯ J ( π ; µ ) , where µ is chosen as an improved policy from last sub-problem.
By successivelysolving them, the impact from the initial reference policy is gradually eliminated.
Theorem 5 (Iterative Regularized Policy Optimization) .
By starting from any stochastic policyπ 0 : s → ∆( A ) and repeatedly finding π i +1 : ¯ J ( π i +1 ; π i ) >
¯ J ( π i ; π i ) , the sequence of { π i }monotonically improves regarding J ( π ) , i.e., J ( π i +1 ) ≥ J ( π i ) . Especially, when π 0 ( a | s ) > 0 , ∀ s, aand { π i } are obtained via regularized policy optimization in Theorem 4, we have π i ( a | s )
π i ( a ′ | s ) → ∞ forany s, a, a ′ such that lim i →∞ Q π i N,k ( s, a ) >
lim i →∞ Q π i N,k ( s, a ′ ) . Ideally, by combining Theorems 4 and 5, the policy for J ( π ) can be continuously improved byinfinitely applying soft pessimistic Bellman backup operator for each of the sequential sub-problems.
Remark 2 (Iterative Regularized Policy Optimization as Expectation–Maximization with Structured
Variational Posterior).
According to Theorem 2, PMDB can be recovered with the converged Qfunction Q π ∗ N,k .
From an end-to-end view, we have an initial dynamics belief P saT , then via thecalculation based on the belief samples and the reward function, we obtain the updated dynamicsbelief (cid:101) P saT .
It is likely that we are doing some form of posterior inference, where the evidence comesfrom the reward function.
In fact, the iterative regularized policy optimization can be formally recastas an Expectation-Maximization algorithm for offline policy optimization, where the Expectation stepcorreponds to a structured variational inference procedure for dynamics.
We elaborate it in Appendix
C.
4.2 Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
While solving each sub-problem makes prominent update to policy compared with the policy gradientmethod, we may need to construct several sub-problems before convergence, then exactly solvingeach of them incurs unnecessary computation.
For practical consideration, next we introduce asmooth-evolving reference policy, with which the explicit boundary between sub-problems is blurred.
Based on this reference policy, and by further adopting function approximator, we devise an offline RL algorithm to approximately maximize J ( π ) .
The idea of smooth-evolving reference policy is inspired by the soft-updated target network indeep RL literature [39, 40].
That is setting the reference policy as a slowly tracked copy of thepolicy being optimized.
Formally, consider a parameterized policy π ϕ with the parameter ϕ .
In theprocess of optimizing π ϕ , the reference policy is set as µ = π ϕ ′ , where ϕ ′ is the moving average ofϕ : ϕ ′ ← ω 1 ϕ
+ (1 − ω 1 ) ϕ ′ .
With small enough ω 1 , the Q-value of the state-action pairs induced byπ ϕ ′ (or its slight variant) can be sufficiently evaluated, before being used to update the policy.
Next,we detail the loss functions to learn Q-value and policy with neural network approximators.
Denote the parameterized Q-function by Q θ with the parameter θ .
It is trained by minimizing the
Bellman residual of both the AMG and the empirical MDP:
[Equation]
11) with(cid:98) Q
[Equation]
, a
[Equation]
=
[Equation]
, a
[Equation]
+
[Equation]
min ⌋ kτ
[Equation] E τ (cid:20) [Equation] where Q θ ′ represent the target Q-value softly updated for stability [40], i.e., θ ′ ← ω 2 θ + (1 − ω 2 ) θ ′ ,and D ′ is the on-policy data buffer for the AMG.
Since the game transition is known, the game can beexecuted with multiple counterparts in parallel, and the buffer only collects the latest sample for eachof them.
To promote direct learning from D , we also include the Bellman residual of the empirical
MDP in (11).
As with policy update, Theorem 4 states that the optimal policy for ¯ J ( π ; µ ) is propotional toµ ( a | s ) exp (cid:16)
α ¯ Q ∗ N,k ( s, a ) (cid:17) .
Then, we update π ϕ by supervisedly learning this policy, with µ and¯ Q ∗ N,k replaced by the smooth-evolving reference policy and the learned Q-value:
[Equation]
where A and B are constant terms.
In general, (14) can be replaced by any tractable function thatmeasures the similarity of distributions.
For example, when π ϕ is Gaussian, we can apply the recentproposed β -NLL [41], in which each data point’s contribution to the negative log-likelihood loss isweighted by the β -exponentiated variance to improve learning heteroscedastic behavior.
To summarize, the algorithm alternates between collecting on-policy data samples and updatingthe function approximators.
In detail, the latter procedure includes updating the policy with (11),updating the Q-value with (14), and updating reference policy as well as target Q-value with themoving-average rule.
The complete algorithm is listed in Appendix D.
Experiments
Through the experiments, we aim to answer the following questions: 1) How does the proposedapproach compared to the previous SoTA offline RL algorithms on standard benchmark? 2)
Howdoes the learning process in the AMG connect with the performance change in the original MDP? 3)
Section 3 presents the monotonicity of J ( π ) over N and k for any specified π , and it is easy to verifythat this statement also holds when considering optimal policy for each setting of ( N, k ) .
However,with neural network approximator, our proposed offline RL algorithm approximately solves the AMG.
Then, how is the monotonicity satisfied in this case?
We consider the Gym domains in the D4RL benchmark [42] to answer these questions.
As PMDBrelies on the initial dynamics belief, inserting additional knowledge into the initial dynamics beliefwill result in unfair comparison.
To avoid that, we consider an uniform distribution over dynamicsmodel ensembles as the initial belief.
The dynamics model ensembles are trained in supervisedmanner with the offline dataset.
This is similar to previous model-based works [6, 21], wherethe dynamics model ensembles are considered for dynamics uncertainty quantification.
Sincehyperparameter tuning for offline RL algorithms requires extra online test for each task, we purposelykeep the same hyperparameters for all tasks, except when answering the last question.
Especially,the hyperparameters in sampling procedure are N = 10 and k = 2 .
The more detailed setup forexperiments and hyperparameters can be found in Appendix E.
5.1 Performance Comparison
We compare the proposed offline RL algorithm with the baselines including: BEAR [5] and BRAC[7], the model-free approaches based on policy constraint, CQL [9], the model-free approach bypenalizing Q-value, EDAC [13], the previous SoTA on the D4RL benchmark, MOReL [6], themodel-based approach which terminates the trajectory if the dynamics uncertainty exceeds a certaindegree, and BC, the behavior cloning method.
These approaches are evaluated on a total of eighteendomains involving three environments (hopper, walker2d, halfcheetah) and six dataset types (random,medium, expert, medium-expert, medium-replay, full-replay) per environment.
The results are summarized in Table 1.
Our approach PMDB obviously improves over the previous
SoTA on 9 tasks and performs competitively in the rest.
Although EDAC achieves better performancein walker2d with several dataset types, its hyperparameters are tuned individually for each task.
As presented later, the experiments on the impact of hyperparameters indicate that larger k orsmaller N could generate better results for walker2d and halfcheetah.
We also find that PMDBsignificantly outperforms MOReL, another model-based approach.
It is encouraging that our modelbased approach achieves competitive or better performance compared with the SoTA model-freeapproach, as model-based approach naturally has better support for multi-task learning and transferlearning, where the offline data from relevant tasks can be further leveraged
[Table] [Table]
5.2 Learning in Alternating Markov Game
Figure 2 presents the learning curves in the AMG, as well as the received return when deployingthe policy being learned in true MDP.
We observe that the performance in the AMG closely tracksthe true performance from the lower side, implying that it can act as a reasonable surrogate toevaluate/optimize performance for the true MDP.
Besides, the performance in the AMG improvesnearly monotonously, verifying the effectiveness of the proposed algorithm to approximately solvethe game.
Recall that PMDB does not explicitly quantify dynamics uncertainty to penalize return, Figure 3checks how the dynamics uncertainty and the Q-value of visited state-action pairs change during the
     7UDLQLQJ6WHSV H     5 H W X U Q
H KRSSHUPHGLXP 0'3 $0*     
7UDLQLQJ6WHSV H      H ZDONHUGPHGLXP     
7UDLQLQJ6WHSV H 
    
H KDOIFKHHWDKPHGLXP learning process.
The uncertainty is measured by the logarithm of standard deviation of the predictedmeans from the N dynamics samples, i.e., log ( std ( E τ [ s ′ ]; τ ∈ T )) .
The policy being learned isperiodically tested in the AMG for ten trials, and we collect the whole ten trajectories of state-actionpairs.
The solid curves in Figure 3 denote the mean uncertainty and Q-value over the collected pairs,and shaded regions denote the standard deviation.
From the results, the dynamics uncertainty firstsharply decreases and then keeps a slowly increasing trend.
Besides, in the long-term view, the
Q-value is correlated with the degree of uncertainty negatively in the first phase and positively in thesecond phase.
This indicates that the policy first moves to the in-distribution region and then tries toget away by resorting to the generalization of dynamics model.     7UDLQLQJ6WHSV H      /RJ V W G  R I S U HG L F W HG  P HDQ V 8QFHUWDLQW\ 4YDOXH      KRSSHUPHGLXP     
7UDLQLQJ6WHSV H 
   
     ZDONHUGPHGLXP     
7UDLQLQJ6WHSV H
          $ Y H U DJH  4  Y D O XH KDOIFKHHWDKPHGLXP
5.3 Practical Impact of Hyperparameters in Sampling Procedure
Table 2 lists the impact of k .
In each setting, we evaluate the learned policy in both the true MDP andthe AMG where the policy is learned.
The performance in the AMGs improve when increasing k .
This is consistent with the theoretical part, even that we approximately solve the game.
Regarding theperformance in true MDPs, we notice that k = 2 corresponds to the best performance for hopper
[Table]
butfor the others k = 3 is better.
This indicates that tuning hyperparameter online can further improvethe performance.
The impact of N is presented in Appendix F, suggesting the opposite monotonicity.
[Table]
Discussion
We proposed model-based offline RL with Pessimism-Modulated Dynamics Belief (PMDB), aframework to reliably learn policy from offline dataset, with the ability of leveraging dynamics priorknowledge.
Empirically, the proposed approach outperforms the previous SoTA in a wide rangeof D4RL tasks.
Compared to the previous model-based approaches, we characterize the impact ofdynamics uncertainty through biased sampling from the dynamics belief, which implicitly induces
PMDB.
As PMDB is with the form of reweighting an initial dynamics belief, it provides a principledway to insert prior knowledge via the belief to boost policy learning.
However, posing a valuabledynamics belief for arbitrary task is challenging, as the expert knowledge is not always available.
Besides, an over-aggressive belief may still incur high-risk behavior in reality.
Encouragingly, recentworks have done active research to learn data-driven prior from relevant tasks.
We believe thatintegrating them as well as developing safe criterion to design/learn dynamics belief would furtherpromote practical deployment of offline RL.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] This is an official example.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6 and Appendix D,we also point out the possible future directions to improve.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code isproprietary for now, but we are processing to make publicly accessible.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix E.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]