Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning
Anonymous Author(s)
Abstract
Provably efficient Model-Based Reinforcement Learning (MBRL) based on optimism or posterior sampling (PSRL) is ensured to attain the global optimalityasymptotically by introducing complexity measure of model.
However, the complexity might grow exponentially for the simplest nonlinear models, where globalconvergence is impossible within finite iterations.
When the model suffers a largegeneralization error, which is quantitatively measured by the model complexity,the uncertainty can be large.
The sampled model that current policy is greedilyoptimized upon will thus be unsettled, resulting in aggressive policy updates andover-exploration.
In this work, we propose Conservative Dual Policy Optimization(CDPO) that involves a Referential Update and a Conservative Update .
The policyis first optimized under a reference model, which imitates the mechanism of PSRLwhile offering more stability.
A conservative range of randomness is guaranteed bymaximizing the expectation of model value.
Without harmful sampling procedures, CDPO can still achieve the same regret as PSRL.
More importantly, CDPO enjoysmonotonic policy improvement and global optimality simultaneously.
Empiricalresults also validate the exploration efficiency of CDPO.
Introduction
Model-Based Reinforcement Learning (MBRL) involves acquiring a model by interacting with theenvironment and learning to make the optimal decision using the model [55, 32].
MBRL is appealingdue to its significantly reduced sample complexity compared to its model-free counterparts.
However,greedy model exploitation that assumes the model is sufficiently accurate lacks guarantees for globaloptimality.
The policies can be suboptimal and get stuck at local maxima even in simple tasks [10].
As such, several provably-efficient MBRL algorithms have been proposed.
Based on the principleof optimism in the face of uncertainty (OFU) [56, 49, 10], OFU-RL achieves the global optimalityby ensuring that the optimistically biased value is close to the real value in the long run.
Basedon Thompson Sampling [62], Posterior Sampling RL (PSRL) [57, 42, 43] explores by greedilyoptimizing the policy in an MDP which is sampled from the posterior distribution over MDPs.
Beyond finite MDPs, to obtain a general bound that permits sample efficiency in various cases, weneed to introduce additional complexity measure.
For example, [49, 43] provide an (cid:101) O ( √ d E T ) regretfor both OFU and PSRL with eluder dimension d E capturing how effectively the model generalizes.
However, it is recently shown [13, 33] that the eluder dimension for even the simplest nonlinearmodels cannot be polynomially bounded.
The effectiveness of the algorithms will thus be crippled.
The underlying reasons for such ineffectiveness are the aggressive policy updates and the overexploration issue.
Specifically, when a nonlinear model is used to fit complex transition functions,its generalizability will be poor compared to simple linear problems.
If a random model is selectedfrom the large hypothesis, e.g., optimistically chosen or sampled from posterior, it is “unsettled”.
In other words, the selected model can change dramatically between successive iterations.
Policyupdates under this model will also be aggressive and thus causing value degradation.
What’s worse,large epistemic uncertainty results in an unrealistically model, which drives agents for uninformativeexploration.
An exploration step can only eliminate an exponentially small portion of the hypothesis.
In this work, we present Conservative Dual Policy Optimization (CDPO), a simple yet provable MBRLalgorithm.
As the sampling process in PSRL harms policy updates due to the unsettled model duringtraining, we propose the Referential Update that greedily optimizes an intermediate policy under areference model .
It mimics the sampling-optimization procedure in PSRL, but offers more stabilitysince we are free to set a steady reference model.
We show that even without a sampling procedure,
CDPO can match the expected regret of PSRL up to constant factors for any proper reference model,e.g., the least squares estimate where the confidence set is centered at.
The Conservative Updatestep then follows to encourage exploration within a reasonable range.
Specifically, the objective ofa reactive policy is to maximize the expectation of model value, instead of a single model’s value.
These two steps are performed in an iterative manner in CDPO.
Theoretically, we show the statistical equivalence between CDPO and PSRL with the same orderof expected regret.
Additionally, we give the iterative policy improvement bound of CDPO, whichguarantees monotonic improvement under mild conditions.
We also establish the sublinear regret of
CDPO, which permits its global optimality equipped with any model function class that has boundedcomplexity measure.
To our knowledge, the proposed framework is the first that simultaneouslyenjoys global optimality and iterative policy improvement.
Experimental results verify the existenceof the over-exploration issue, and demonstrate the practical benefit of CDPO.
Background2.1 Model-Based Reinforcement Learning
We consider the problem of learning to optimize an infinite-horizon γ -discounted Markov Decision
Process (MDP) over repeated episodes of interaction.
Denote the state space and action space as Sand A , respectively.
When taking action a ∈ A at state s ∈ S , the agent receives reward r ( s, a ) andthe environment transits into a new state according to probability s ′ ∼ f ∗ ( ·| s, a ) .
Here, f ∗ is a diracmeasure for deterministic dynamics and is a probability distribution for probabilistic dynamics.
In model-based RL, the true dynamical model f ∗ is unknown and needs to be learned using thecollected data through episodic (or iterative) interaction.
The history data up to iteration t then forms
H t = {{ s h,i , a h,i , s h
+1 ,i } H − 1 h =0 } t − 1 i =1 , where H is the actual timesteps agents run in an episode.
Theposterior distribution of the dynamics model is estimated as ϕ ( ·|H t ) .
Alternatively, the frequentistmodel of the mean and uncertainty can also be estimated.
Specifically, consider the model functionclass F = { f : S ×A → S} with size |F| , which contains the real model f ∗ ∈ F .
The confidence set(or model hypothesis set) F t ⊂ F is introduced to represent the range of dynamics that is statisticallyplausible [49, 43, 10].
To ensure that f ∗ ∈ F t with high probability, one way is to construct theconfidence set as F t := { f ∈ F | ∥ f − (cid:98) f LS t ∥ 2 ,E t ≤ √ β t } .
Here, β t is an appropriately chosenconfidence parameter (via concentration inequality), the cumulative empirical 2-norm is defined by∥ g
[Equation]
,E t :
[Equation]
(cid:80) t
[Equation]
1 i =1 ∥ g ( x i ) ∥ 22 .
The least squares estimate is
[Equation]
Denote the state and state-action value function associated with π on model f by V fπ : S → R and
Q fπ : S ×
A → R , respectively, which are defined as
[Equation]
The objective of RL is to learn a policy π ∗ =
argmax π J ( π ) that maximizes the expected return
J ( π ) .
Denote the initial state distribution as ζ .
Under policy π , the state visitation measure ν π ( s )over S and the state-action visitation measure ρ π ( s, a ) over S ×
A in the true MDP are defined asν π ( s ) =
(1 − γ ) · ∞ (cid:88) h
[Equation] [Equation]
(cid:88)
h
[Equation]
γ
[Equation] [Equation]
[Equation]
where s 0 ∼ ζ , a h ∼ π ( ·| s h ) and s h
+1 ∼ f ∗ ( ·| s h , a h ) .
The objective J ( π ) is then
[Equation]
2.2 Cumulative Regret and Asymptotic Optimality
A common criterion to evaluate RL algorithms is the cumulative regret, defined as the cumulativeperformance discrepancy between policy π t at each iteration t and the optimal policy π ∗ over the runof the algorithm.
The (cumulative) regret up to iteration T is defined as:
[Equation]
In the Bayesian view, the model f ∗ , the learning policy π and the regret are random variables thatmust be learned from the gathered data.
The Bayesian expected regret is defined as:
[Equation]
One way to prove the asymptotic optimality is to show that the (expected) regret is sublinear in T , sothat π t converges to π ∗ within sufficient iterations.
To obtain the regret bound, the width of confidenceset ω t ( s, a ) is introduced to represent the maximum deviation between any two members in F t :
[Equation]
Provable Model-Based Reinforcement Learning
In this section, we analyze the central ideas and limitations of greedy algorithms as well as twopopular theoretically justified frameworks: optimistic algorithms and posterior sampling algorithms.
Greedy Model Exploitation.
Before introducing provable algorithms, we first analyze greedy modelbased algorithms.
In this framework, the agent takes actions assuming that the fitted model sufficientlyaccurately resembles the real MDP.
Algorithms that lie in this category can be roughly divided intotwo groups: model-based planning and model-augmented policy optimization.
For instance, Dynaagents [61, 20, 17] optimize policies using model-free learners with model-generated data.
The modelcan also be exploited in first-order gradient estimators [18, 12, 9] or value expansion [15, 6].
On theother hand, model-based planning, or model-predictive control (MPC) [40, 41], directly generatesoptimal action sequences under the model in a receding horizon fashion.
However, greedily exploiting the model without deep exploration [45] will lead to suboptimal performance.
The resulting policy can suffer from premature convergence, leaving the potentiallyhigh-reward region unexplored.
Since the transition data is generated by the agent taking actions inthe real MDP, the dual effect [4, 27] that current action influences both the next state and the modeluncertainty is not considered by greedy model-based algorithms.
Optimism in the Face of Uncertainty.
A common provable exploration mechanism is to adopt theprinciple of optimism in the face of uncertainty (OFU) [56, 49, 10].
With OFU, the agent assigns toits policy an optimistically biased estimate of virtual value by jointly optimizing over the policies andmodels inside the confidence set F t .
At iteration t , the OFU-RL policy π t is defined as:
[Equation]
Most asymptotic analyses of optimistic RL algorithms can be abstracted as showing two properties:the virtual value V fπ is sufficiently high, and it is close to the real value V f ∗ π in the long run.
However,in complex environments where the generalizability of nonlinear models is limited, large epistemicuncertainty will result in an unrealistically large optimistic return that drives agents for uninformativeexploration.
What’s worse, such suboptimal exploration steps eliminate only a small portion of themodel hypothesis [13], leading to a slow converging process and suboptimal practical performance.
Posterior Sampling Reinforcement Learning.
An alternative exploration mechanism is based on
Thompson Sampling (TS) [62, 52], which involves selecting the maximizing action from a statistically plausibly set of action values.
These values can be associated with the MDP sampled from its posteriordistribution, thus giving its name posterior sampling for reinforcement learning (PSRL) [57, 42, 43].
The algorithm begins with a prior distribution of f ∗ .
At each iteration t , a model f t is sampled fromthe posterior ϕ ( ·|H t ) , and π t is updated to be optimal under f t :
[Equation]
The insight is to keep away from actions that are unlikely to be optimal in the real MDP.
Explorationis guaranteed by the randomness in the sampling procedure.
Unfortunately, executing actions thatare optimally associated with a single sampled model can cause similar over-exploration issues[52, 51].
Specifically, an imperfect model sampled from the large hypothesis can cause aggressivepolicy updates and value degradation between successive iterations.
The suboptimality degree ofthe resulting policies depends on the epistemic model uncertainty.
Besides, executing π t is notintended to offer performance improvement for follow-up policy learning, but only to narrow downthe model uncertainty.
However, this elimination procedure will be slow when the model suffers alarge generalization error, which is quantitatively formulated in the model complexity measure below.
Complexity Measure and Generalization Bounds.
In RL, we seek to have the sample complexityfor finding a near-optimal policy or estimating an accurate value function.
When given access to agenerative model (i.e., an abstract sampling model) in finite MDPs, it is known that the (minimax)number of transitions the agent needs to observe can be sublinear in the model size, i.e. smaller than
O ( |S| 2 |A| ) .
Beyond finite MDPs where the number of states is large (or countably or uncountablyinfinite), we are interested in the learnability or generalization of RL.
Unfortunately, it is impossiblefor agnostic reinforcement learning that finds the best hypothesis in some given policy, value, ormodel hypothesis class: the number of needed samples depends exponentially on the problem horizon[24].
Despite of the structural assumptions, e.g. linear MDPs [67, 22, 66] or low-rank MDPs [21, 38],we focus on the generalization bounds that can cover various cases.
This can be done with additionalcomplexity measure, e.g. eluder dimension [49], witness rank [60], or bilinear rank [14].
By introducing the eluder dimension d E [49], previous work [43, 44] established regret (cid:101) O ( √ d E T )for both OFU-RL and PSRL.
Intuitively, the eluder dimension captures how effectively the modellearned from observed data can extrapolate to future data, and permits sample efficiency in various(linear) cases.
Nevertheless, it is shown in [13, 33] that even the simplest nonlinear models do nothave polynomially-bounded eluder dimension. The following result is from Thm. 5.2 in Dong et al.[13] and similar results are also established in [33]. Theorem 3.1 (Eluder Dimension of Nonlinear Models [13]) .
The eluder dimension dim E ( F , ε ) (c.f.
Definition 5.4) of one-layer ReLU neural networks is at least Ω( ε − ( d − 1) ) , where d is the state-actiondimension, i.e. ( s, a ) ∈ R d . With more layers, the requirement of ReLU activation can be relaxed. As a result, additional complexity is hidden in the eluder dimension, e.g. when we choose ε = T − 1 ,regret (cid:101) O ( √ d E T ) contains d E = Ω( T d − 1 ) and is no longer sublinear in T . In this case, previousprovable exploration mechanisms will lose the desired property of global optimality and sampleefficiency, which is the underlying reason for the over-exploration issue.
Conservative Dual Policy Optimization
When using nonlinear models, e.g. neural networks, the over-exploration issue causes unfavorableperformance in practice, in terms of slow convergence and suboptimal asymptotic values.
To tacklethis challenge, the key is to abandon the sampling process and have guarantees during training.
In this regard, we propose Conservative Dual Policy Optimization (CDPO) that is simple yet provablyefficient.
By optimizing the policy following two successive update procedures iteratively, CDPOsimultaneously enjoys monotonic policy value improvement and global optimality properties.
4.1 CDPO Framework
To begin with, consider the problem of maximizing the expected value, π t = argmax π E
[ V f ∗ π | H t ] ,where E [ V f ∗ | H t ] denotes the expected values over the posterior.
Obviously, we have the expectedvalue improvement guarantee E
[ V f ∗ π t | H t ] ≥ E
[ V f ∗ π t − 1 | H t ] .
We can also perform expected value maximization in a trust-region to guarantee iterative improvement under any f ∗ .
However, suchupdates will lose the desired global convergence guarantee and may get stuck at local maxima evenwith linear models.
For this reason, we propose a dual procedure of policy optimization.
Referential Update.
The first update step returns an intermediate policy, denoted as q t .
This stepis a greedy one in the sense that q t is optimal with respect to the value of a single model (cid:101) f t , whichwe call a reference model .
Selecting a reference model and optimizing a policy w.r.t.
it imitates thesampling-optimization procedure of PSRL.
We will show in Section 5.1 that if we pose the constraint(cid:101)
f t ∈ F t , then CDPO achieves the same expected regret as PSRL, which implies global optimality.
More importantly, policy optimization under (cid:101)
f t is more stable and can avoid the over-explorationissue in PSRL since we are free to set it as a steady reference between successive iterations.
Forexample, we fix the reference model (cid:101) f t as the least squares estimate (cid:98) f
LSt defined in (2.1), instead of arandom model sampled from the large hypothesis that causes aggressive policy update.
This gives us:
Referential Update (with LS Reference):
[Equation]
Constrained Conservative Update.
The conservative update then follows as the second stage of
CDPO, which takes input q t and returns the reactive policy π t +1 : Conservative
[Equation]
[Equation]
where D TV ( · , · ) stands for the total variation distance and η is the hyperparameter that characterizesthe trust-region constraint and controls the degree of exploration.
Compared with OFU-RL and PSRL, the above exploration and policy updates are conservative sincethe policy maximizes the expectation of the model value, instead of a single model’s value (i.e. theoptimistic model in OFU-RL and the sampled model in PSRL).
The conservative update (4.2) avoidsthe pitfalls when the optimistic model or the posterior sampled model suffers large bias, which leadsto aggressive policy update and over-exploration during training.
Notably, the term conservative inour work differs from previous use, e.g. Conservative Policy Iteration [23, 53].
While the latter refersto policy updates with constraints, ours is to emphasize the conservative range of randomness and thereduction of unnecessary over-exploration by shelving the sampling process.
In our analysis, we follow previous work [43, 59, 10, 35] and assume access to a policy optimizationoracle.
In practice, the problem of finding an optimal policy under a given model can be approximatelysolved by model-based solvers listed below.
More fine-grained analysis can be obtained by applyingoff-the-shelf results established for policy gradient or MPC for specific policy or model functionclasses.
This, however, is beyond the scope of this paper.
4.2 Practical Algorithm
Algorithm Input: Prior ϕ , model-based policy optimization solver MBPO ( π, f, J ) .
1: for iteration t = 1 , ..., T do 2: q t ← MBPO ( · , (cid:98) f LSt , (4.1) ) 3:
Sample N models { f t,n } Nn =1 from ϕ 4: π t ← MBPO ( q t , { f t,n } Nn =1 , (4.2) ) 5:
Execute π t in the real MDP 6:
Update H t
+1 =
H t ∪ { s h,t , a h,t , s h
+1 ,t } h 7:
Update (cid:98) f LSt +1 and ϕ 8: end for 9: return policy π T
The pseudocode of CDPO is in Alg.
Themodel-based solver MBPO ( π, f, J ) outputs thepolicy ( q t or π t ) that optimizes the objective J with access to model f .
Several different types of solvers can be leveraged, e.g.,model-augmented model-free policy optimization such as Dyna [61], model-based reparameterization gradient [18, 9], or model-predictivecontrol [64].
Details of different optimizationchoices can be found in Appendix E.
In experiments, we use Dyna and MPC solvers .
With Pinsker’s inequality, the total variationconstraint in (4.2) is replaced by the KL divergence [53, 2] in experiments.
We followprevious work [34] to use neural network ensembles [10, 25] for model estimation and use calibrations [29, 10] for accurate uncertainty measure.
Analysis
In this section, we first show the statistical equivalence between CDPO and PSRL in terms of the same
BayesRegret bound.
Then we give the iterative policy value bound with monotonic improvement.
Finally, we prove the global convergence of CDPO.
The missing proofs can be found in the Appendix.
5.1 Statistical Equivalence between CDPO and PSRL
We begin our analysis by highlighting the connection between CDPO and PSRL with the followingtheorem, from which we also show the role of the dual update procedure and the reference model.
Theorem 5.1 (CDPO Matches PSRL in BayesRegret) .
Let π PSRL be the policy of any posteriorsampling algorithm for reinforcement learning optimized by (3.2).
If the BayesRegret bound of π PSRLsatisfies that for any T > 0 , BayesRegret( T, π PSRL , ϕ ) ≤ D , then for all T > 0 , we have for the
CDPO policy π CDPO that BayesRegret( T, π CDPO , ϕ ) ≤ 3 D .
Sketch proof.
We first sketch the general strategy in the PSRL analysis.
Recall the definition of the
Bayesian expected regret BayesRegret( T, π, ϕ ) :
= E [ (cid:80) Tt =1 R t ] , where R t =
V f ∗ π ∗ − V f ∗ π t .
PSRLbreaks down R t by adding and subtracting value V f t π f t of the imagined optimal policy π f t under asampled model f t , i.e. π f t = argmax π V f t π .
[Equation]
where the second equality follows from the definition of the PSRL policy.
Following the law of totalexpectation and the Posterior Sampling Lemma (e.g. Lemma 1 in [42]), we have E [ V f ∗ π ∗ − V f t π ft ] = 0by noting that f ∗ and f t are identically distributed conditioned upon H t .
Then we obtain
BayesRegret( T, π PSRL , ϕ ) =
[Equation]
where the first inequality follows from the simulation lemma under the L -Lipschitz value assumption[43].
The second inequality follows from the definition of ω t in (2.6) and the construction of confidence set such that P ( f ∗ ∈ (cid:84) F t ) ≥ 1 − 2 δ and P ( f t ∈ (cid:84) F t , f ∗ ∈ (cid:84) F t ) ≥ 1 − 4 δ via a unionbound.
As more data is collected, the model uncertainty is reduced and the sum of confidence setwidth ω t will be sublinear in T (c.f.
Lemma B.4 and B.5), indicating sublinear regret.
When it comes to CDPO, we decompose the regret as
[Equation]
where the CDPO policy π t is defined in (4.2).
Since E [ V f ∗ π ∗ − V f t π ft ] = 0 , we have
BayesRegret( T, π CDPO , ϕ ) =
T (cid:88) t =
[Equation]
≤
T (cid:88) t =
E
[Equation]
t π ft − V
[Equation]
f t π ft + V (cid:101) f t q t −
[Equation]
f t q t
[Equation]
f [Equation] f [Equation]
] ≤ L [Equation] T (cid:88) t = 3 E
[Equation]
[Equation]
) where the first inequality follows from the optimality of q t and π t , i.e., V (cid:101) f t π ft ≤ V (cid:101) f t q t for any π f t and
V f t π t ≥ V f t q t .
The 8 δT term is introduced since (cid:101) f t ∈ F t and P ( f t ∈ (cid:84) F t , (cid:101) f t ∈ (cid:84) F t ) ≥ 1 − 2 δ .
Theorem 5.1 indicates that although CDPO performs conservative updates and abandons the samplingprocess, it matches the statistical efficiency of PSRL up to constant factors.
The importance of the reference model and the dual procedure is also reflected in the proof.
Thereferential update builds the bridge between V f t π ft and V f t π t .
Policy optimization under the referencemodel mimics the sampling-optimization of PSRL.
However, more stability is offered when thereference is steady, e.g., the least squares estimate we use.
We will formalize this idea below.
5.2 CDPO Policy Iterative Improvement
One motivation of the conservative update is that it maximizes (thus improves) the expected valueover the posterior.
In this section, we are interested in the policy value improvement under anyunknown f ∗ .
Namely, we seek to have the iterative improvement bound J ( π t ) − J ( π t − 1 ) , where thetrue objective J is defined in (2.3).
We impose the following regularity condition on the underlying MDP transition.
Assumption 5.2 (Regularity Condition on MDP Transition) .
Assume that the MDP transition functionf ∗ : S ×
A → S is with additive σ -sub-Gaussian noise and bounded norm, i.e., ∥ s ∥ 2 ≤ C .
Theorem 5.3 (Policy Iterative Improvement) .
Suppose we have ∥ (cid:101) f
( · , · ) ∥ ≤ C for (cid:101) f ∈ F wherethe model class F is finite.
Define ι := max s,a | A f ∗ π ( s, a ) | , where A f ∗ π is the advantage functiondefined as A f ∗ π ( s, a ) :=
Q f ∗ π ( s, a ) − V f ∗ π ( s ) .
With probability at least 1 − δ , the policy improvementbetween successive iterations is bounded by
[Equation]
[Equation]
where ∆( t ) :
=
E s ∼ ζ (cid:2) V (cid:101)
f t q t ( s ) − V (cid:101) f
t q t − 1 ( s ) (cid:3) is non-negative due to the greediness of q t .
The above theorem provides the iterative improvement bound following the CDPO algorithm.
When
H is large enough, the policy value improvement is at least ∆( t ) by choosing a properly small η .
In particular, the first term ∆( t ) characterizes the policy improvement brought by the greedy exploitation in (4.1), and ∆( t ) ≥ 0 since q t is optimal under (cid:101) f t .
The second term in (5.5) accounts for thegeneralization error of least square methods.
Specifically, model (cid:101) f t
= (cid:98) f LSt ∈
F t is trained to fit thehistory samples.
However, we seek to have the model error bound over the state-action visitationmeasure, which requires the deviation from the empirical mean to its expectation using Bernstein’sinequality and union bound.
This term will approach 0 with a long length H of the sampling path,which in the ideal case should be set to ∞ since the MDP is infinite-horizon.
Finally, the trust-regionconstraint in (4.2) brings the 4 ηα/ (1 − γ ) term, which reduces to zero if η is small.
It makes intuitivesense as η controls the degree of conservative exploration.
5.3 Global Optimality of CDPO
We now analyze the global optimality of CDPO by studying its expected regret.
As discussed in
Section 3, agnostic reinforcement learning is impossible.
Without structural assumptions, additionalcomplexity measure is required for a generalization bound beyond finite settings.
For this reason, weadopt the notation of eluder dimension [49, 43], defined as follows.
Definition 5.4 (Eluder Dimension) .
The eluder dimension dim E ( F , ε ) is the length of the longestpossible sequence of elements in S×A such that for some ε ′ ≥ ε , every element is ( F , ε ′ )
-independentof its predecessors, where we defer the definition of ( F , ε )-independence to Appendix C.
We make the regularity condition on the Lipschitz continuity of the value function.
Assumption 5.5 (Lipschitz Continuous Value) .
At iteration t , the value function V f t π is Lipschitzcontinuous in the sense that | V f t π ( s 1 ) − V f t π ( s 2 ) | ≤ L t ∥ s 1 − s 2 ∥ 2 .
Notably, Assumption 5.5 holds under certain regularity conditions of the MDP, e.g. when the transitionand rewards are Lipschitz continuous [5, 47].
Under this assumption, many RL settings can be satisfied[13], e.g., nonlinear models with stochastic Lipschitz policies and Lipschitz reward models, and thusis adopted by various model-based RL work [35, 7, 13].
We now study the global optimality of CDPO by the following expected regret theorem.
Theorem 5.6 (Expected Regret of CDPO) .
Let N ( F , α, ∥·∥ 2 ) be the α -covering number of F .
Denote d E :=
dim E ( F , T − 1 ) for the eluder dimension of F at precision 1 /T .
Under Assumption5.2 and 5.5, the cumulative expected regret of CDPO in T iterations is bounded by
[Equation]
where β := 8 σ 2 log (cid:16) 2 N (cid:0) F , 1 / ( T 2 ) , ∥·∥ 2 (cid:1) T (cid:17) + 2 (cid:0) 8 C + (cid:112) 8 σ 2 log(8 T 3 ) (cid:1) /T and L := E [ L t ] .
Here, the covering number is introduced since we are considering F that may contain infinitely manyfunctions, for which we cannot simply apply a union bound.
Besides, β is the confidence parameterthat contains f ∗ with high probability (via concentration inequality).
To clarify the asymptotics of the expected regret bound, we introduce another measure of dimensionality that captures the sensitivity of F to statistical overfitting.
Corollary 5.7 (Asymptotic Bound) .
Define the Kolmogorov dimension w.r.t.
function class F as
[Equation]
Under the assumptions of Theorem 5.6 and by omitting terms logarithmic in T , the regret of CDPO is
[Equation]
The sublinear regret result permits the global optimality and sample efficiency for any model classwith reasonable complexity measure.
Meanwhile, the iterative improvement theorem guaranteesefficient exploration and good performance even when the model class is highly nonlinear.
Empirical Evaluation
We first provide insights and evidence of why CDPO exploration can be more efficient in N -Chain
MDPs, which have optimal right actions and suboptimal left actions at each of the N states.
Settingsand full results are provided in Appendix F.4.
In Fig.
1, we compare the posterior of CDPO and PSRLat the state furthest away from the initial state, i.e. the state that is the hardest for agents to explore.
[Figure]
[Figure]
304 When training starts, both algorithms have a large variance of value estimation.
However, as trainingprogresses, CDPO gives more accurate and certain estimates, but only for the optimal right actions notfor the suboptimal left actions, while PSRL agents explore both directions. This verifies the potentialover-exploration issue in PSRL: as long as the uncertainty contains unrealistically large values, PSRLagents can perform uninformative exploration by acting suboptimally according to an inaccuratesampled model. In contrast, CDPO replaces the sampled model with a stable mean estimate and caresabout the expected value, thus avoiding such pitfalls. We see in Fig.
2 that although CDPO has muchlarger uncertainty for the suboptimal left actions, its regret is lower.
6.2 Exploration Efficiency with Nonlinear Model Class
In finite MDPs, PSRL-style agents can specify and try every action to finally obtain an accuratehigh-confidence prediction. However, our discussion in Sec. 3 indicates that similar over-explorationsteps in more complex environments can only eliminate an exponentially small portion of theuncertainty. We experimentally validate this phenomenon by plotting the average uncertainty scaleon the validation data in Appendix Section F.3.
The slower uncertainty elimination rate of PSRL and HUCRL [10] indicates less informative exploration steps. To see its impact on the training performance, we report the results of provable algorithms withnonlinear models on several MuJoCo tasks in Figure 3. We observe that all algorithms achievethe asymptotic optimality in inverted pendulum.
Since the dimension of the pendulum task is low,learning an accurate (and thus generalizable) model poses no actual challenge.
However, in higherdimensional tasks such as half-cheetah and ant, CDPO achieves higher asymptotic value with fasterconvergence.
Implementation details and hyperparameters are provided in Appendix F.1. [Figure]
6.3 Comparisons with Prior RL Algorithms
[Figure]
For other MBRL algorithms, we examine DPI [59], SLBO [35], ME-TRPO [30], MBPO [20], and VaGraM [63].
Despite the curves in Fig. 4, we report the sample efficiency and running time in
Appendix F.2.
The model-free baselines we compare with include SAC [16], PPO [54], and MPO [2].
[Figure]
6.4 Ablation Study
One can observe from Fig.
5 that the policies updated with only Referential Update or Conservative Update lag behind the dual framework.
We also test the necessity and sensitivity of the constrainthyperparameter η .
We see that constant η and time-decaying η achieve similar asymptotic value withsimilar convergence rate (around 7 . 5 e 5 timesteps), showing the robustness of CDPO.
However, removing the constraint will lose the policy improvement guarantee, thus causing degradation.
Ablationon different choices of MBPO solver (Dyna and POPLIN-P [64]) shows the generalizability of CDPO.
[Figure]
Conclusions & Future Work
In this work, we present Conservative Dual Policy Optimization (CDPO), a simple yet provablemodel-based algorithm.
By iterative execution of the Referential Update and Conservative Update ,
CDPO explores within a reasonable range while avoiding aggressive policy update.
CDPO gets rid ofthe sampling procedure by optimizing under a reference model and maximizing the expected policyvalue.
With the same order of regret as PSRL, the proposed algorithm can achieve global optimalitywhile monotonically improving the policy.
Considering our naive choice of the reference model, othermore sophisticated designs should be a fruitful future direction.
It will also be interesting to exploredifferent choices of MBPO solvers, which we would like to leave as future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Some components in our algo-
rithm are naively designed.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Appendix H.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption5.2 and 5.5.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide
detailed settings and hyperparameters.
Our code will be released once accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
[Table]
ments multiple times)?
[Yes]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., type [Table] of GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[N/A] [Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] [Table] Our code can be found in the supplemental material. [Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]