Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning
Anonymous Author(s)
Abstract
Provably efficient Model-Based Reinforcement Learning (MBRL) based on optimism or posterior sampling (PSRL) is ensured to attain the global optimalityasymptotically by introducing complexity measure of the model class.
However,the complexity might grow exponentially for even the simplest nonlinear models,where global convergence is impossible within finite iterations.
When the modelsuffers a large generalization error, which is quantitatively measured by the modelcomplexity, the uncertainty can be large.
The sampled model that current policyis greedily optimized upon will thus be unsettled, resulting in aggressive policyupdates and over-exploration.
In this work, we propose Conservative Dual Policy Optimization (CDPO) that involves a Referential Update and a Conservative Update .
The policy is first optimized under a reference model, which imitates themechanism of PSRL while offering more stability.
A conservative range of randomness is guaranteed by maximizing the expectation of model value.
Without harmfulsampling procedures, CDPO can still achieve the same regret as PSRL.
Moreimportantly, CDPO enjoys monotonic policy improvement and global optimalitysimultaneously.
Empirical results also demonstrate the superiority of CDPO.
Introduction
Model-Based Reinforcement Learning (MBRL) involves acquiring a model by interacting with theenvironment and learning to make the optimal decision using the model [48, 28].
MBRL is appealingdue to its significantly reduced sample complexity compared to its model-free counterparts.
However,greedy model exploitation that assumes the model is sufficiently accurate lacks guarantees for globaloptimality.
The policies can be suboptimal and get stuck at local maxima even in simple tasks [9].
As such, several provably-efficient MBRL algorithms have been proposed.
Based on the principleof optimism in the face of uncertainty (OFU) [49, 43, 9], OFU-RL achieves the global optimalityby ensuring that the optimistically biased value is close to the real value in the long run.
Basedon Thompson Sampling [54], Posterior Sampling RL (PSRL) [50, 37, 38] explores by greedilyoptimizing the policy in an MDP which is sampled from the posterior distribution over MDPs.
Beyond finite MDPs, to obtain a general bound that permits sample efficiency in various cases, weneed to introduce additional complexity measure.
For example, [43, 38] provide an (cid:101) O ( √ d E T ) regretfor both OFU and PSRL with eluder dimension d E capturing how effectively the model generalizes.
However, it is recently shown [11, 29] that the eluder dimension for even the simplest nonlinearmodels cannot be polynomially bounded.
The effectiveness of the algorithms will thus be crippled.
The underlying reasons for such ineffectiveness are the aggressive policy updates and the overexploration issue.
Specifically, when a nonlinear model is used to fit complex transition functions,its generalizability will be poor compared to simple linear problems.
If a random model is selectedfrom the large hypothesis, e.g., optimistically chosen or sampled from posterior, it is “unsettled”.
In other words, the selected model can change dramatically between successive iterations.
Policyupdates under this model will also be aggressive and thus causing value degradation.
What’s worse,large epistemic uncertainty results in an unrealistically model, which drives agents for uninformativeexploration.
An exploration step can only eliminate an exponentially small portion of the hypothesis.
In this work, we present Conservative Dual Policy Optimization (CDPO), a simple yet provable MBRLalgorithm.
As the sampling process in PSRL harms policy updates due to the unsettled model duringtraining, we propose the Referential Update that greedily optimizes an intermediate policy under areference model .
It mimics the sampling-optimization procedure in PSRL, but offers more stabilitysince we are free to set a steady reference model.
We show that even without a sampling procedure,
CDPO can match the expected regret of PSRL up to constant factors for any proper reference model,e.g., the least squares estimate where the confidence set is centered at.
The Conservative Updatestep then follows to encourage exploration within a reasonable range.
Specifically, the objective ofa reactive policy is to maximize the expectation of model value, instead of a single model’s value.
These two steps are performed in an iterative manner in CDPO.
Theoretically, we show the statistical equivalence between CDPO and PSRL with the same orderof expected regret.
Additionally, we give the iterative policy improvement bound of CDPO, whichguarantees monotonic improvement under mild conditions.
We also establish the sublinear regret of
CDPO, which permits its global optimality equipped with any model function class that has boundedcomplexity measure.
To our knowledge, the proposed framework is the first that simultaneouslyenjoys global optimality and iterative policy improvement.
Experimental results verify the existenceof the over-exploration issue, and demonstrate the practical benefit of CDPO.
Background2.1 Model-Based Reinforcement Learning
We consider the problem of learning to optimize an infinite-horizon γ -discounted Markov Decision
Process (MDP) over repeated episodes of interaction.
Denote the state space and action space as Sand A , respectively.
When taking action a ∈ A at state s ∈ S , the agent receives reward r ( s, a ) andthe environment transits into a new state according to probability s ′ ∼ f ∗ ( ·| s, a ) .
Here, f ∗ is a diracmeasure for deterministic dynamics and is a probability distribution for probabilistic dynamics.
In model-based RL, the true dynamical model f ∗ is unknown and needs to be learned using thecollected data through episodic (or iterative) interaction.
The history data up to iteration t then forms
H t = {{ s h,i , a h,i , s h
+1 ,i } H − 1 h =0 } t − 1 i =1 , where H is the actual timesteps agents run in an episode.
Theposterior distribution of the dynamics model is estimated as ϕ ( ·|H t ) .
Alternatively, the frequentistmodel of the mean and uncertainty can also be estimated.
Specifically, consider the model functionclass F = { f : S ×A → S} with size |F| , which contains the real model f ∗ ∈ F .
The confidence set(or model hypothesis set) F t ⊂ F is introduced to represent the range of dynamics that is statisticallyplausible [43, 38, 9].
To ensure that f ∗ ∈ F t with high probability, one way is to construct theconfidence set as F t := { f ∈ F | ∥ f − (cid:98) f LS t ∥ 2 ,E t ≤ √ β t } .
Here, β t is an appropriately chosenconfidence parameter (via concentration inequality), the cumulative empirical 2-norm is defined by∥ g
[Equation]
,E t :
[Equation]
(cid:80) t
[Equation]
1 i =1 ∥ g ( x i ) ∥ 22 .
The least squares estimate is
[Equation]
Denote the state and state-action value function associated with π on model f by V fπ : S → R and
Q fπ : S ×
A → R , respectively, which are defined as
[Equation]
The objective of RL is to learn a policy π ∗ =
argmax π J ( π ) that maximizes the expected return
J ( π ) .
Denote the initial state distribution as ζ .
Under policy π , the state visitation measure ν π ( s )over S and the state-action visitation measure ρ π ( s, a ) over S ×
A in the true MDP are defined asν π ( s ) =
(1 − γ ) · ∞ (cid:88) h
[Equation] [Equation]
(cid:88)
h
[Equation]
γ
[Equation] [Equation]
[Equation]
where s 0 ∼ ζ , a h ∼ π ( ·| s h ) and s h
+1 ∼ f ∗ ( ·| s h , a h ) .
The objective J ( π ) is then
[Equation]
2.2 Cumulative Regret and Asymptotic Optimality
A common criterion to evaluate RL algorithms is the cumulative regret, defined as the cumulativeperformance discrepancy between policy π t at each iteration t and the optimal policy π ∗ over the runof the algorithm.
The (cumulative) regret up to iteration T is defined as:
[Equation]
where R t denotes the regret over iteration t , defined as the performance discrepancy under ν ( s ) :
[Equation]
In the Bayesian view, the model f ∗ , the learning policy π and the regret are random variables thatmust be learned from the gathered data.
The Bayesian expected regret is defined as:
[Equation]
[Equation]
One way to prove the asymptotic optimality is to show that the (expected) regret is sublinear in T , sothat π t converges to π ∗ within sufficient iterations.
To obtain the regret bound, the width of confidenceset ω t ( s, a ) is introduced to represent the maximum deviation between any two members in F t : [Equation]
[Equation]
Provable Model-Based Reinforcement Learning
In this section, we analyze the central ideas and limitations of greedy algorithms as well as twopopular theoretically justified frameworks: optimistic algorithms and posterior sampling algorithms.
Greedy Model Exploitation.
Before introducing provable algorithms, we first analyze greedy modelbased algorithms.
In this framework, the agent takes actions assuming that the fitted model sufficientlyaccurately resembles the real MDP.
Algorithms that lie in this category can be roughly divided intotwo groups: model-based planning and model-augmented policy optimization.
For instance, Dynaagents [53, 17, 15] optimize policies using model-free learners with model-generated data.
The modelcan also be exploited in first-order gradient estimators [16, 10, 8] or value expansion [13, 5].
On theother hand, model-based planning, or model-predictive control (MPC) [35, 36], directly generatesoptimal action sequences under the model in a receding horizon fashion.
However, greedily exploiting the model without deep exploration [40] will lead to suboptimal performance.
The resulting policy can suffer from premature convergence, leaving the potentiallyhigh-reward region unexplored.
Since the transition data is generated by the agent taking actions inthe real MDP, the dual effect [3, 24] that current action influences both the next state and the modeluncertainty is not considered by greedy model-based algorithms.
Optimism in the Face of Uncertainty.
A common provable exploration mechanism is to adopt theprinciple of optimism in the face of uncertainty (OFU) [49, 43, 9].
With OFU, the agent assigns to itspolicy an optimistically biased estimate of virtual value by jointly optimizing over the policies andmodels inside the confidence set F t .
At iteration t , the OFU-RL policy π t is defined as:
[Equation]
Most asymptotic analyses of optimistic RL algorithms can be abstracted as showing two properties:the virtual value V fπ is sufficiently high, and it is close to the real value V f ∗ π in the long run.
However,in complex environments where the generalizability of nonlinear models is limited, large epistemicuncertainty will result in an unrealistically large optimistic return that drives agents for uninformative exploration.
What’s worse, such suboptimal exploration steps eliminate only a small portion of themodel hypothesis [11], leading to a slow converging process and suboptimal practical performance.
Posterior Sampling Reinforcement Learning.
An alternative exploration mechanism is based on
Thompson Sampling (TS) [54, 45], which involves selecting the maximizing action from a statisticallyplausibly set of action values.
These values can be associated with the MDP sampled from its posteriordistribution, thus giving its name posterior sampling for reinforcement learning (PSRL) [50, 37, 38].
The algorithm begins with a prior distribution of f ∗ .
At each iteration t , a model f t is sampled fromthe posterior ϕ ( ·|H t ) , and π t is updated to be optimal under f t :
[Equation]
The insight is to keep away from actions that are unlikely to be optimal in the real MDP.
Explorationis guaranteed by the randomness in the sampling procedure.
Unfortunately, executing actions thatare optimally associated with a single sampled model can cause similar over-exploration issues[45, 44].
Specifically, an imperfect model sampled from the large hypothesis can cause aggressivepolicy updates and value degradation between successive iterations.
The suboptimality degree ofthe resulting policies depends on the epistemic model uncertainty.
Besides, executing π t is notintended to offer performance improvement for follow-up policy learning, but only to narrow downthe model uncertainty.
However, this elimination procedure will be slow when the model suffers alarge generalization error, which is quantitatively formulated in the model complexity measure below.
Complexity Measure and Generalization Bounds.
In RL, we seek to have the sample complexityfor finding a near-optimal policy or estimating an accurate value function.
When given access to agenerative model (i.e., an abstract sampling model) in finite MDPs, it is known that the (minimax)number of transitions the agent needs to observe can be sublinear in the model size, i.e. smaller than
O ( |S| 2 |A| ) .
Beyond finite MDPs where the number of states is large (or countably or uncountablyinfinite), we are interested in the learnability or generalization of RL.
Unfortunately, it is impossiblefor agnostic reinforcement learning that finds the best hypothesis in some given policy, value, ormodel hypothesis class: the number of needed samples depends exponentially on the problem horizon[21].
Despite of the structural assumptions, e.g. linear MDPs [58, 19, 57] or low-rank MDPs [18, 33],we focus on the generalization bounds that can cover various cases.
This can be done with additionalcomplexity measure, e.g. eluder dimension [43], witness rank [52], or bilinear rank [12].
By introducing the eluder dimension, previous work [38, 39] gives sublinear regret for both OFU-RLand PSRL.
Intuitively, the eluder dimension captures how effectively the model learned from theobserved data can extrapolate to future data (c.f. Definition 5.5), and permits sample efficiencyin various (linear) cases.
Nevertheless, it is shown in [11, 29] that even the simplest nonlinearmodels, e.g., one-layer neural networks, do not have polynomially-bounded eluder dimension (atleast exponential in dimension, c.f. Appendix C).
In other words, additional complexity is hidden inthe eluder dimension, which is the underlying reason for the over-exploration issue.
Conservative Dual Policy Optimization
When using nonlinear models, e.g. neural networks, the over-exploration issue causes unfavorableperformance in practice, in terms of slow convergence and suboptimal asymptotic values.
To tacklethis challenge, the key is to abandon the sampling process and have guarantees during training.
In this regard, we propose Conservative Dual Policy Optimization (CDPO) that is simple yet provablyefficient.
By optimizing the policy following two successive update procedures iteratively, CDPOsimultaneously enjoys monotonic policy value improvement and global optimality properties.
4.1 CDPO Framework
To begin with, consider the problem of maximizing the expected value, π t = argmax π E
[ V f ∗ π | H t ] ,where E [ V f ∗ | H t ] denotes the expected values over the posterior.
Obviously, we have the expectedvalue improvement guarantee E
[ V f ∗ π t | H t ] ≥ E
[ V f ∗ π t − 1 | H t ] .
We can also perform expected valuemaximization in a trust-region to guarantee iterative improvement under any f ∗ .
However, suchupdates will lose the desired global convergence guarantee and may get stuck at local maxima evenwith linear models.
For this reason, we propose a dual procedure of policy optimization.
Referential Update.
The first update step returns an intermediate policy, denoted as q t .
This stepis a greedy one in the sense that q t is optimal with respect to the value of a single model (cid:101) f t , whichwe call a reference model .
Selecting a reference model and optimizing a policy w.r.t.
it imitates thesampling-optimization procedure of PSRL.
We will show in Section 5.1 that if we pose the constraint(cid:101)
f t ∈ F t , then CDPO achieves the same expected regret as PSRL, which implies global optimality.
More importantly, policy optimization under (cid:101)
f t is more stable and can avoid the over-explorationissue in PSRL since we are free to set it as a steady reference between successive iterations.
Forexample, we fix the reference model (cid:101) f t as the least squares estimate (cid:98) f
LSt defined in (2.1), instead of arandom model sampled from the large hypothesis that causes aggressive policy update.
This gives us:
Referential Update (with LS Reference):
[Equation]
Constrained Conservative Update.
The conservative update then follows as the second stage of
CDPO, which takes input q t and returns the reactive policy π t +1 :
[Equation]
where D TV ( · , · ) stands for the total variation distance and η is the hyperparameter that characterizesthe trust-region constraint and controls the degree of exploration.
Compared with OFU-RL and PSRL, the above exploration and policy updates are conservative sincethe policy maximizes the expectation of the model value, instead of a single model’s value (i.e. theoptimistic model in OFU-RL and the sampled model in PSRL).
The conservative update (4.2) avoidsthe pitfalls when the optimistic model or the posterior sampled model suffers large bias, which leadsto aggressive policy update and over-exploration during training.
Notably, the term conservative inour work differs from previous use, e.g. Conservative Policy Iteration [20, 46].
While the latter refersto policy updates with constraints, ours is to emphasize the conservative range of randomness and thereduction of unnecessary over-exploration by shelving the sampling process.
4.2 Practical Algorithm
Input: Prior ϕ , model-based policy optimization solver MBPO ( π, f, J ) .
1: for iteration t = 1 , ..., T do 2: q t ← MBPO ( · , (cid:98) f LSt , (4.1) ) 3:
Sample N models { f t,n } Nn =1 from ϕ 4: π t ← MBPO ( q t , { f t,n } Nn =1 , (4.2) ) 5:
Execute π t in the real MDP 6:
Update H t
+1 =
H t ∪ { s h,t , a h,t , s h
+1 ,t } h 7:
Update (cid:98) f LS t +1 and ϕ 8: end for 9: return policy π T
The pseudocode of CDPO is in Alg.
Themodel-based solver MBPO ( π, f, J ) outputs thepolicy ( q t or π t ) that optimizes the objective J with access to model f .
Several different types of solvers can be leveraged, e.g.,model-augmented model-free policy optimization such as Dyna [53], model-based reparameterization gradient [16, 8], or model-predictivecontrol [56].
Details of different optimizationchoices can be found in Appendix E.
In experiments, we use Dyna and MPC solvers .
With Pinsker’s inequality, the total variationconstraint in (4.2) is replaced by the KL divergence [46, 1] in experiments.
We followprevious work [30] to use neural network ensembles [9, 22] for model estimation and use calibrations[26, 9] for accurate uncertainty measure.
Analysis
In this section, we first show the statistical equivalence between CDPO and PSRL in terms of the same
BayesRegret bound.
Then we give the iterative policy value bound with monotonic improvement.
Finally, we prove the global convergence of CDPO.
The missing proofs can be found in the Appendix.
5.1 Statistical Equivalence between CDPO and PSRL
We begin our analysis by highlighting the connection between CDPO and PSRL with the followingtheorem, from which we also show the role of the dual update procedure and the reference model.
Theorem 5.1 (CDPO Matches PSRL in BayesRegret) .
Let π PSRL be the policy of any posteriorsampling algorithm for reinforcement learning optimized by (3.2).
If the BayesRegret bound of π PSRLsatisfies that for any T > 0 , BayesRegret( T, π PSRL , ϕ ) ≤ D , then for all T > 0 , we have for the
CDPO policy π CDPO that BayesRegret( T, π CDPO , ϕ ) ≤ 3 D .
Sketch proof.
We first sketch the general strategy in the PSRL analysis.
Recall the definition of the
Bayesian expected regret BayesRegret( T, π, ϕ ) :
= E [ (cid:80) Tt =1 R t ] , where R t =
V f ∗ π ∗ − V f ∗ π t .
PSRLbreaks down R t by adding and subtracting value V f t π ft of the imagined optimal policy π f t under asampled model f t , i.e. π f t = max π V f t π . PSRL [Equation]
[Equation]
) where the second equality follows from the definition of the PSRL policy. PSRL then proceeds by
E [ V f ∗ π ∗ − V f t π ft ] = 0 . It can be shown by the law of total expectation and the posterior sampling lemmaby noting that f ∗ and f t are identically distributed conditioned upon H t .
Then we obtain
BayesRegret( T, π PSRL , ϕ ) =
T (cid:88) t =
[Equation]
[Equation]
where the first inequality follows from the simulation lemma under the L -Lipschitz value assumption[38].
The second inequality follows from the definition of ω t in (2.7) and the construction of confidence set such that P ( f ∗ ∈ (cid:84) F t ) ≥ 1 − 2 δ and P ( f t ∈ (cid:84) F t , f ∗ ∈ (cid:84) F t ) ≥ 1 − 4 δ via a unionbound.
As more data is collected, the model uncertainty is reduced and the sum of confidence setwidth ω t will be sublinear in T (c.f.
Lemma B.4 and B.5), indicating sublinear regret.
When it comes to CDPO, we decompose the regret as
[Equation]
where the CDPO policy π t is defined in (4.2).
Since E [ V f ∗ π ∗ − V f t π ft ] = 0 , we have
BayesRegret( T, π CDPO , ϕ ) =
T (cid:88) t =
[Equation]
≤
[Equation]
where the first inequality follows from the optimality of q t and π t , i.e., V (cid:101)
f t π ft ≤ V (cid:101) f t q t for any π f t and
V f t π t ≥ V f t q t .
The 8 δT term is introduced since (cid:101) f t ∈ F t and P ( f t ∈ (cid:84) F t , (cid:101) f t ∈ (cid:84) F t ) ≥ 1 − 2 δ .
Theorem 5.1 indicates that although CDPO performs conservative updates and abandons the samplingprocess, it matches the statistical efficiency of PSRL up to constant factors.
The importance of the reference model and the dual procedure is also reflected in the proof.
Thereferential update builds the bridge between V f t π ft and V f t π t .
Policy optimization under the referencemodel mimics the sampling-optimization of PSRL.
However, more stability is offered when thereference is steady, e.g., the least squares estimate we use.
We will formalize this idea below.
5.2 CDPO Policy Iterative Improvement
One motivation of the conservative update is that it maximizes (so improves) the expected value overthe posterior.
In this section, we are interested in the policy value improvement under any unknownf ∗ .
Namely, we seek to have the iterative improvement bound J ( π t ) − J ( π t − 1 ) , where the trueobjective J is defined in (2.3).
We impose the following regularity condition on the underlying MDP transition.
Assumption 5.2 (Regularity Condition on MDP Transition) .
Assume that the MDP transition functionf ∗ : S ×
A → S is with additive σ -sub-Gaussian noise and bounded norm, i.e., ∥ s ∥ 2 ≤ C .
Theorem 5.3 (Policy Iterative Improvement) .
Suppose we have ∥ (cid:101) f
( · , · ) ∥ ≤ C for (cid:101) f ∈ F wherethe model class F is finite.
Define ι := max s,a | A f ∗ π ( s, a ) | , where A f ∗ π is the advantage functiondefined as A f ∗ π ( s, a ) :=
Q f ∗ π ( s, a ) − V f ∗ π ( s ) .
With probability at least 1 − δ , the policy improvementbetween successive iterations is bounded by
[Equation]
where ∆( t ) :
=
E s ∼ ζ (cid:2) V (cid:101)
f t q t ( s ) − V (cid:101) f
t q t − 1 ( s ) (cid:3) is non-negative due to the greediness of q t .
The above theorem provides the iterative improvement bound following the CDPO algorithm.
When
H is large enough, the policy value improvement is at least ∆( t ) by choosing a properly small η .
Specifically, the first term ∆( t ) characterizes the policy improvement brought by the greedy exploitation in (4.1), and ∆( t ) ≥ 0 since q t is optimal under (cid:101) f t .
The second term in (5.5) accounts for thegeneralization error of least square methods.
Specifically, model (cid:101) f t
= (cid:98) f LSt ∈
F t is trained to fit thehistory samples.
However, we seek to have the model error bound over the state-action visitationmeasure, which requires the deviation from the empirical mean to its expectation using Bernstein’sinequality and union bound.
This term will approach 0 with a long length H of the sampling path,which in the ideal case should be set to ∞ since the MDP is infinite-horizon.
Finally, the trust-regionconstraint in (4.2) brings the 4 ηα/ (1 − γ ) term, which reduces to zero if η is small.
It makes intuitivesense as η controls the degree of conservative exploration.
5.3 Global Optimality of CDPO
We now analyze the global optimality of CDPO by studying its expected regret.
As discussed in
Section 3, agnostic reinforcement learning is impossible.
Without structural assumptions, additionalcomplexity measure is required for a generalization bound beyond finite settings.
For this reason, weadopt the notation of eluder dimension [43, 38], defined as follows.
Definition 5.4 (( F , ε )-Dependence) .
If we say ( s, a ) ∈ S×A is ( F , ε ) -dependent on { ( s i , a i ) } ni =1 ⊆ S × A , then [Equation] Conversely, ( s, a ) ∈ S
×
A is ( F , ε )-independent of { ( s i , a i ) } ni =1 if and only if it does not satisfythe definition for dependence. Definition 5.5 (Eluder Dimension) .
The eluder dimension dim E ( F , ε ) is the length of the longestpossible sequence of elements in S×A such that for some ε ′ ≥ ε , every element is ( F , ε ′ )
-independentof its predecessors.
We make the regularity condition on the Lipschitz continuity of the value function.
Assumption 5.6 (Lipschitz Continuous Value) .
At iteration t , the value function V f t π is Lipschitzcontinuous in the sense that | V f t π ( s [Equation]
Notably, Assumption 5.6 holds under certain regularity conditions of the MDP, e.g. when the transitionand rewards are Lipschitz continuous [4, 41].
Under this assumption, many RL settings can be satisfied[11], e.g., nonlinear models with stochastic Lipschitz policies and Lipschitz reward models, and thusis adopted by various model-based RL work [31, 6, 11].
We now study the global optimality of CDPO by the following expected regret theorem.
Theorem 5.7 (Expected Regret of CDPO) .
Let N ( F , α, ∥·∥ 2 ) be the α -covering number of F .
Denote d E :=
dim E ( F , T − 1 ) for the eluder dimension of F at precision 1 /T .
Under Assumption5.2 and 5.6, the cumulative expected regret of CDPO in T iterations is bounded by
BayesRegret
[Equation]
) where β := 8 σ 2 log (cid:16) 2 N (cid:0) F , 1 / ( T 2 ) , ∥·∥ 2 (cid:1) T (cid:17) + 2 (cid:0) 8 C + (cid:112) 8 σ 2 log(8 T 3 ) (cid:1) /T and L := E [ L t ] .
Here, the covering number is introduced since we are considering F that may contain infinitely manyfunctions, for which we cannot simply apply a union bound.
Besides, β is the confidence parameterthat is set to contain f ∗ with high probability (via concentration inequality).
To clarify the asymptotics of the expected regret bound, we can introduce another measure ofdimensionality that captures the sensitivity of F to statistical overfitting.
Corollary 5.8 (Asymptotic Bound) .
Define the Kolmogorov dimension w.r.t.
function class F as
[Equation]
Under the assumptions of Theorem 5.7 and by omitting terms logarithmic in T , the regret of CDPO is
[Equation]
The sublinear regret result permits the global optimality for any model class with reasonable complexity measure.
Meanwhile, the iterative improvement theorem guarantees efficient exploration andgood performance even when the model class is highly nonlinear.
Empirical Evaluation
In this section, we conduct experiments to support our claims. Firstly, we evaluate the explorationefficiency of previous provable MBRL when equipped with nonlinear models. Then we compare
CDPO with several SOTA model-free and model-based RL baselines.
Finally, we conduct ablationsto better understand the proposed algorithm.
We test on the Mujoco benchmark [55].
6.1 Exploration Efficiency with Nonlinear Model Class
As analyzed in Section 3, over-exploration issues exist in OFU-RL and PSRL when the model classis not compatible with a reasonable complexity measure, e.g., the eluder dimension of nonlinearmodels is exponential.
To see the impact in practice, we test CDPO, OFU-RL, and PSRL equippedwith the same 5 -layer neural network with 200 hidden nodes in each layer.
For OFU-RL, we mainlyevaluate HUCRL [9], a deep algorithm proposed to deal with the joint optimization intractability.
Implementation details and hyperparameters are provided in Appendix F.
[Figure] We conduct experiments in three control tasks: inverted pendulum swing-up, pusher goal-reaching,and half-cheetah locomotion. All the experiments are repeated with 6 random seeds.
We observe in Figure 1 that both HUCRL and PSRL achieve asymptotic optimality in inverted pendulum.
Since thedimension of the pendulum task is low, learning an accurate (and thus generalizable) model posesno actual challenge.
However, in higher dimensional tasks, the incompatibility brought by nonlinearmodels can harm the performance of PSRL and HUCRL.
This phenomenon is most obvious in thehalf-cheetah task, where learning a globally accurate model is hard. When the model uncertaintyis large, the aggressive policy updates and uninformative exploration lead to slow convergence andlower asymptotic value within finite iterations.
6.2 Comparisons with Prior RL Algorithms
We also compare the performance of CDPO with other SOTA MB- and MF-RL algorithms in Fig.
[Figure]
For MBRL algorithms, we examine DPI [51] and SLBO [31], both of which are proved to satisfymonotonic improvement.
In the inverted pendulum task, the convergence is faster than model-freealgorithms by leveraging the acquired model.
However, due to the lack of deep exploration [40]mechanisms, their asymptotic performance in the more complex pusher and half-cheetah tasks islower than provable approaches (PSRL, HUCRL, and CDPO) due to premature convergence.
We alsocompare with model-free baseline SAC [14], as well as trust-region algorithms: PPO [47] and MPO[1].
CDPO achieves higher asymptotic performance while requiring significantly fewer samples, e.g.,5 , 7 , and 10 times fewer samples in the half-cheetah task than SAC, MPO, and PPO, respectively.
6.3 Ablation Study
We conduct ablation studies to better understand the components in CDPO.
One can observe from Fig.
3 that the policies updated with only Referential Update or Conservative Update lag behind thedual procedure in terms of convergence speed and asymptotic value.
We also test the necessity andsensitivity of the constraint hyperparameter η .
We see that simply setting η = 0 . 1 works pretty well. Shrinking the trust-region with a time-decay η also attains a similar result.
However, removing theconstraint will lose the policy improvement guarantee, thus causing degradation.
Ablation on differentchoices of MBPO solver (Dyna and POPLIN-P [56]) shows the generalizability of the framework.
[Figure]
Conclusions & Future Work
In this work, we present Conservative Dual Policy Optimization (CDPO), a simple yet provablemodel-based algorithm.
By iterative execution of the Referential Update and Conservative Update ,
CDPO explores within a reasonable range while avoiding aggressive policy update.
Importantly,
CDPO gets rid of the harmful sampling procedure in previous provable approaches. Instead, wepropose to mimic the update of PSRL by optimizing under a reference model, and to conservativelyexplore by maximizing the expectation of model value.
With the same order of regret as PSRL, theproposed algorithm can achieve global optimality while monotonically improving the policy value.
Considering our naive choice of the reference model, i.e. LS estimate, it will be interesting to expandto other more sophisticated designs, while not degrading the theorems.
Different choices of MBPOsolvers should also be worth exploring, which we would like to leave as future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Some components in our algo-
rithm are naively designed.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Appendix H.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption 5.2 and 5.6.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide
detailed settings and hyperparameters.
Our code will be released once accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]Our code can be found in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]