Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning
Abstract
Provably efficient Model-Based Reinforcement Learning (MBRL) based on optimism or posterior sampling (PSRL) is ensured to attain the global optimality asymptotically by introducing the complexity measure of the model.
However, the complexity might grow exponentially for the simplest nonlinear models, where global convergence is impossible within finite iterations.
When the model suffers a large generalization error, which is quantitatively measured by the model complexity, the uncertainty can be large.
The sampled model that current policy is greedily optimized upon will thus be unsettled, resulting in aggressive policy updates and over-exploration.
In this work, we propose Conservative Dual Policy Optimization (CDPO) that involves a Referential Update and a Conservative Update .
The policy is first optimized under a reference model, which imitates the mechanism of PSRL while offering more stability.
A conservative range of randomness is guaranteed by maximizing the expectation of model value.
Without harmful sampling procedures,
CDPO can still achieve the same regret as PSRL.
More importantly, CDPO enjoys monotonic policy improvement and global optimality simultaneously.
Empirical results also validate the exploration efficiency of CDPO.
Introduction
Model-Based Reinforcement Learning (MBRL) involves acquiring a model by interacting with the environment and learning to make the optimal decision using the model [55, 32].
MBRL is appealing due to its significantly reduced sample complexity compared to its model-free counterparts.
However, greedy model exploitation that assumes the model is sufficiently accurate lacks guarantees for global optimality.
The policies can be suboptimal and get stuck at local maxima even in simple tasks [10].
As such, several provably-efficient MBRL algorithms have been proposed.
Based on the principle of optimism in the face of uncertainty (OFU) [56, 49, 10], OFU-RL achieves the global optimality by ensuring that the optimistically biased value is close to the real value in the long run.
Based on Thompson Sampling [62], Posterior Sampling RL (PSRL) [57, 42, 43] explores by greedily optimizing the policy in an MDP which is sampled from the posterior distribution over MDPs.
Beyond finite MDPs, to obtain a general bound that permits sample efficiency in various cases, we need to introduce additional complexity measure.
For example, [49, 43] provide an (cid:101) O ( √ d E T ) regret for both OFU and PSRL with eluder dimension d E capturing how effectively the model generalizes.
However, it is recently shown [13, 33] that the eluder dimension for even the simplest nonlinear models cannot be polynomially bounded.
The effectiveness of the algorithms will thus be crippled.
The underlying reasons for such ineffectiveness are the aggressive policy updates and the overexploration issue.
Specifically, when a nonlinear model is used to fit complex transition functions, its generalizability will be poor compared to simple linear problems.
If a random model is selected from the large hypothesis, e.g., optimistically chosen or sampled from the posterior, it is “unsettled".
In other words, the selected model can change dramatically between successive iterations.
Policy updates under this model will also be aggressive and thus cause value degradation.
What’s worse, large epistemic uncertainty results in an unrealistic model, which drives agents for uninformative exploration.
An exploration step can only eliminate an exponentially small portion of the hypothesis.
In this work, we present Conservative Dual Policy Optimization (CDPO), a simple yet provable MBRL algorithm.
As the sampling process in PSRL harms policy updates due to the unsettled model during training, we propose the Referential Update that greedily optimizes an intermediate policy under a reference model .
It mimics the sampling-then-optimization procedure in PSRL but offers more stability since we are free to set a steady reference model.
We show that even without a sampling procedure, CDPO can match the expected regret of PSRL up to constant factors for any proper reference model, e.g., the least squares estimate where the confidence set is centered at.
The Conservative Update step then follows to encourage exploration within a reasonable range.
Specifically, the objective of a reactive policy is to maximize the expectation of model value, instead of a single model’s value.
These two steps are performed in an iterative manner in CDPO.
Theoretically, we show the statistical equivalence between CDPO and PSRL with the same order of expected regret.
Additionally, we give the iterative policy improvement bound of CDPO, which guarantees monotonic improvement under mild conditions.
We also establish the sublinear regret of
CDPO, which permits its global optimality equipped with any model function class that has a bounded complexity measure.
To our knowledge, the proposed framework is the first that simultaneously enjoys global optimality and iterative policy improvement.
Experimental results verify the existence of the over-exploration issue and demonstrate the practical benefit of CDPO.
Background 2.1 Model-Based Reinforcement Learning
We consider the problem of learning to optimize an infinite-horizon γ -discounted Markov Decision
Process (MDP) over repeated episodes of interaction.
Denote the state space and action space as S and A , respectively.
When taking action a ∈ A at state s ∈ S , the agent receives reward r ( s, a ) and the environment transits into a new state according to probability s ′ ∼ f ∗ ( ·| s, a ) .
Here, f ∗ is a dirac measure for deterministic dynamics and is a probability distribution for probabilistic dynamics.
In model-based RL, the true dynamical model f ∗ is unknown and needs to be learned using the collected data through episodic (or iterative) interaction.
The history data up to iteration t then forms
H t = {{ s h,i , a h,i , s h
+1 ,i } H − 1 h =0 } t − 1 i =1 , where H is the actual timesteps agents run in an episode.
The posterior distribution of the dynamics model is estimated as ϕ ( ·|H t ) .
Alternatively, the frequentist model of the mean and uncertainty can also be estimated.
Specifically, consider the model function class F = { f : S ×A → S} with size |F| , which contains the real model f ∗ ∈ F .
The confidence set (or model hypothesis set) F t ⊂ F is introduced to represent the range of dynamics that is statistically plausible [49, 43, 10].
To ensure that f ∗ ∈ F t with high probability, one way is to construct the confidence set as F t := { f ∈ F | ∥ f − (cid:98) f LS t ∥ 2 ,E t ≤ √ β t } .
Here, β t is an appropriately chosen confidence parameter (via concentration inequality), the cumulative empirical 2-norm is defined by ∥ g ∥ 22 ,E t :
= (cid:80) t − 1 i =1 ∥ g ( x i ) ∥ 22 .
The least squares estimate is
[Equation]
Denote the state and state-action value function associated with π on model f by V fπ : S → R and
Q fπ : S ×
A → R , respectively, which are defined as
[Equation]
The objective of RL is to learn a policy π ∗ =
argmax π J ( π ) that maximizes the expected return
J ( π ) .
Denote the initial state distribution as ζ .
Under policy π , the state visitation measure ν π ( s ) over S and the state-action visitation measure ρ π ( s, a ) over S ×
A in the true MDP are defined as ν π ( s
[Equation]
=
[Equation]
[Equation]
[Equation]
where s 0 ∼ ζ , a h ∼ π ( ·| s h ) and s h
+1 ∼ f ∗ ( ·| s h , a h ) .
The objective J ( π ) is then
[Equation]
2.2 Cumulative Regret and Asymptotic Optimality
A common criterion to evaluate RL algorithms is the cumulative regret, defined as the cumulative performance discrepancy between policy π t at each iteration t and the optimal policy π ∗ over the run of the algorithm.
The (cumulative) regret up to iteration T is defined as:
[Equation]
In the Bayesian view, the model f ∗ , the learning policy π , and the regret are random variables that must be learned from the gathered data.
The Bayesian expected regret is defined as:
[Equation]
One way to prove the asymptotic optimality is to show that the (expected) regret is sublinear in T , so that π t converges to π ∗ within sufficient iterations.
To obtain the regret bound, the width of confidence set ω t ( s, a ) is introduced to represent the maximum deviation between any two members in F t :
[Equation]
Provable Model-Based Reinforcement Learning
In this section, we analyze the central ideas and limitations of greedy algorithms as well as two popular theoretically justified frameworks: optimistic algorithms and posterior sampling algorithms.
Greedy Model Exploitation.
Before introducing provable algorithms, we first analyze greedy modelbased algorithms.
In this framework, the agent takes actions assuming that the fitted model sufficiently accurately resembles the real MDP.
Algorithms that lie in this category can be roughly divided into two groups: model-based planning and model-augmented policy optimization.
For instance, Dyna agents [61, 20, 17] optimize policies using model-free learners with model-generated data.
The model can also be exploited in first-order gradient estimators [18, 12, 9] or value expansion [15, 6].
On the other hand, model-based planning, or model-predictive control (MPC) [40, 41], directly generates optimal action sequences under the model in a receding horizon fashion.
However, greedily exploiting the model without deep exploration [45] will lead to suboptimal performance.
The resulting policy can suffer from premature convergence, leaving the potentially high-reward region unexplored.
Since the transition data is generated by the agent taking actions in the real MDP, the dual effect [4, 27] that current action influences both the next state and the model uncertainty is not considered by greedy model-based algorithms.
Optimism in the Face of Uncertainty.
A common provable exploration mechanism is to adopt the principle of optimism in the face of uncertainty (OFU) [56, 49, 10].
With OFU, the agent assigns to its policy an optimistically biased estimate of virtual value by jointly optimizing over the policies and models inside the confidence set F t .
At iteration t , the OFU-RL policy π t is defined as:
[Equation]
Most asymptotic analyses of optimistic RL algorithms can be abstracted as showing two properties: the virtual value V fπ is sufficiently high, and it is close to the real value V f ∗ π in the long run.
However, in complex environments where the generalizability of nonlinear models is limited, large epistemic uncertainty will result in an unrealistically large optimistic return that drives agents for uninformative exploration.
What’s worse, such suboptimal exploration steps eliminate only a small portion of the model hypothesis [13], leading to a slow converging process and suboptimal practical performance.
Posterior Sampling Reinforcement Learning.
An alternative exploration mechanism is based on
Thompson Sampling (TS) [62, 52], which involves selecting the maximizing action from a statistically plausibly set of action values.
These values can be associated with the MDP sampled from its posterior distribution, thus giving its name posterior sampling for reinforcement learning (PSRL) [57, 42, 43].
The algorithm begins with a prior distribution of f ∗ .
At each iteration t , a model f t is sampled from the posterior ϕ ( ·|H t ) , and π t is updated to be optimal under f t :
[Equation]
The insight is to keep away from actions that are unlikely to be optimal in the real MDP.
Exploration is guaranteed by the randomness in the sampling procedure.
Unfortunately, executing actions that are optimally associated with a single sampled model can cause similar over-exploration issues [52, 51].
Specifically, an imperfect model sampled from the large hypothesis can cause aggressive policy updates and value degradation between successive iterations.
The suboptimality degree of the resulting policies depends on the epistemic model uncertainty.
Besides, executing π t is not intended to offer performance improvement for follow-up policy learning, but only to narrow down the model uncertainty.
However, this elimination procedure will be slow when the model suffers a large generalization error, which is quantitatively formulated in the model complexity measure below.
Complexity Measure and Generalization Bounds.
In RL, we seek to have the sample complexity for finding a near-optimal policy or estimating an accurate value function.
When given access to a generative model (i.e., an abstract sampling model) in finite MDPs, it is known that the (minimax) number of transitions the agent needs to observe can be sublinear in the model size, i.e. smaller than
O ( |S| 2 |A| ) .
Beyond finite MDPs where the number of states is large (or countably or uncountably infinite), we are interested in the learnability or generalization of RL.
Unfortunately, it is impossible for agnostic reinforcement learning that finds the best hypothesis in some given policy, value, or model hypothesis class: the number of needed samples depends exponentially on the problem horizon [24].
Despite of the structural assumptions, e.g. linear MDPs [66, 22, 65] or low-rank MDPs [21, 38], we focus on the generalization bounds that can cover various cases.
This can be done with additional complexity measure, e.g. eluder dimension [49], witness rank [60], or bilinear rank [14].
By introducing the eluder dimension d E [49], previous work [43, 44] established regret (cid:101) O ( √ d E T ) for both OFU-RL and PSRL.
Intuitively, the eluder dimension captures how effectively the model learned from observed data can extrapolate to future data, and permits sample efficiency in various (linear) cases.
Nevertheless, it is shown in [13, 33] that even the simplest nonlinear models do not have a polynomially-bounded eluder dimension.
The following result is from Thm.
5.2 in Dong et al. [13] and similar results are also established in [33].
Theorem 3.1 (Eluder Dimension of Nonlinear Models [13]) .
The eluder dimension dim E ( F , ε ) (c.f.
Definition 5.6) of one-layer ReLU neural networks is at least Ω( ε − ( d − 1) ) , where d is the state-action dimension, i.e. ( s, a ) ∈ R d .
With more layers, the requirement of ReLU activation can be relaxed.
As a result, additional complexity is hidden in the eluder dimension, e.g. when we choose ε = T − 1 , regret (cid:101) O ( √ d E T ) contains d E = Ω( T d − 1 ) and is no longer sublinear in T .
In this case, previous provable exploration mechanisms will lose the desired property of global optimality and sample efficiency, which is the underlying reason for the over-exploration issue.
Conservative Dual Policy Optimization
When using nonlinear models, e.g. neural networks, the over-exploration issue causes unfavorable performance in practice, in terms of slow convergence and suboptimal asymptotic values.
To tackle this challenge, the key is to abandon the sampling process and have guarantees during training.
In this regard, we propose Conservative Dual Policy Optimization (CDPO) that is simple yet provably efficient.
By optimizing the policy following two successive update procedures iteratively, CDPO simultaneously enjoys monotonic policy value improvement and global optimality properties.
4.1 CDPO Framework
To begin with, consider the problem of maximizing the expected value, π t = argmax π E
[ V f ∗ π | H t ] , where E [ V f ∗ | H t ] denotes the expected values over the posterior.
Obviously, we have the expected value improvement guarantee E
[ V f ∗ π t | H t ] ≥ E
[ V f ∗ π t − 1 | H t ] .
We can also perform expected value maximization in a trust-region to guarantee iterative improvement under any f ∗ .
However, such updates will lose the desired global convergence guarantee and may get stuck at local maxima even with linear models.
For this reason, we propose a dual procedure of policy optimization.
Referential Update.
The first update step returns an intermediate policy, denoted as q t .
This step is a greedy one in the sense that q t is optimal with respect to the value of a single model (cid:101) f t , which we call a reference model .
Selecting a reference model and optimizing a policy w.r.t.
it imitates the sampling-optimization procedure of PSRL.
We will show in Section 5.1 that if we pose the constraint (cid:101) f t ∈ F t , then CDPO achieves the same expected regret as PSRL, which implies global optimality.
More importantly, policy optimization under (cid:101)
f t is more stable and can avoid the over-exploration issue in PSRL since we are free to set it as a steady reference between successive iterations.
For example, we fix the reference model (cid:101) f t as the least squares estimate (cid:98) f
LSt defined in (2.1), instead of a random model sampled from the large hypothesis that causes aggressive policy update.
This gives us:
Referential Update (with LS Reference):
[Equation]
Constrained Conservative Update.
The conservative update then follows as the second stage of
CDPO, which takes input q t and returns the reactive policy π t +1 :
Conservative
[Equation]
[Equation]
where D TV ( · , · ) stands for the total variation distance and η is the hyperparameter that characterizes the trust-region constraint and controls the degree of exploration.
Compared with OFU-RL and PSRL, the above exploration and policy updates are conservative since the policy maximizes the expectation of the model value, instead of a single model’s value (i.e. the optimistic model in OFU-RL and the sampled model in PSRL).
The conservative update (4.2) avoids the pitfalls when the optimistic model or the posterior sampled model suffers large bias, which leads to aggressive policy updates and over-exploration during training.
Notably, the term conservative in our work differs from previous use, e.g. Conservative Policy Iteration [23, 53].
While the latter refers to policy updates with constraints, ours is to emphasize the conservative range of randomness and the reduction of unnecessary over-exploration by shelving the sampling process.
In our analysis, we follow previous work [43, 59, 10, 35] and assume access to a policy optimization oracle.
In practice, the problem of finding an optimal policy under a given model can be approximately solved by model-based solvers listed below.
More fine-grained analysis can be obtained by applying off-the-shelf results established for policy gradient or MPC for specific policy or model function classes.
This, however, is beyond the scope of this paper.
4.2 Practical Algorithm
Input: Prior ϕ , model-based policy optimization solver MBPO ( π, f, J ) .
1: for iteration t = 1 , ..., T do 2: q t ← MBPO ( · , (cid:98) f LSt , (4.1) ) 3:
Sample N models { f t,n } Nn = 4: π t ← MBPO ( q t , { f t,n } Nn =1 , (4.2) ) 5:
Execute π t in the real MDP 6:
Update H t
+1 =
H t ∪ { s h,t , a h,t , s h
+1 ,t } h 7:
Update (cid:98) f LSt +1 and ϕ 8: end for 9: return policy π T
The pseudocode of CDPO is in Alg.
The model-based solver MBPO ( π, f, J ) outputs the policy ( q t or π t ) that optimizes the objective J with access to model f .
Several different types of solvers can be leveraged, e.g., model-augmented model-free policy optimization such as Dyna [61], model-based reparameterization gradient [18, 9], or model-predictive control [63].
Details of different optimization choices can be found in Appendix E.
In experiments, we use Dyna and MPC solvers.
With Pinsker’s inequality, the total variation constraint in (4.2) is replaced by the KL divergence [53, 2] in experiments.
We follow previous work [34] to use neural network ensembles [10, 25] for model estimation and use calibrations [29, 10] for accurate uncertainty measure.
Analysis
In this section, we first show the statistical equivalence between CDPO and PSRL in terms of the same
BayesRegret bound.
Then we give the iterative policy value bound with monotonic improvement.
Finally, we prove the global convergence of CDPO.
The missing proofs can be found in the Appendix.
5.1 Statistical Equivalence between CDPO and PSRL
We begin our analysis by highlighting the connection between CDPO and PSRL with the following theorem, from which we also show the role of the dual update procedure and the reference model.
Theorem 5.1 (CDPO Matches PSRL in BayesRegret) .
Let π PSRL be the policy of any posterior sampling algorithm for reinforcement learning optimized by (3.2).
If the BayesRegret bound of π PSRL satisfies that for any T > 0 , BayesRegret( T, π PSRL , ϕ ) ≤ D , then for all T > 0 , we have for the
CDPO policy π CDPO that BayesRegret( T, π CDPO , ϕ ) ≤ 3 D .
Sketch proof.
We first sketch the general strategy in the PSRL analysis.
Recall the definition of the
Bayesian expected regret BayesRegret( T, π, ϕ ) :
= E [ (cid:80) Tt =1 R t ] , where R t =
V f ∗ π ∗ − V f ∗ π t .
PSRL breaks down R t by adding and subtracting V f t π f t , the value of the imagined optimal policy π f t under a sampled model f t , i.e. π f t = argmax
π V f t π .
[Equation]
[Equation]
(5.1) where the second equality follows from the definition of the PSRL policy.
Following the law of total expectation and the Posterior Sampling Lemma (e.g. Lemma 1 in [42]), we have E [ V f ∗ π ∗ − V f t π ft ] =by noting that f ∗ and f t are identically distributed conditioned upon H t .
Then we obtain
BayesRegret( T, π PSRL , ϕ ) =
T (cid:88)
[Equation]
) where the first inequality follows from the simulation lemma under the L -Lipschitz value assumption [43].
The second inequality follows from the definition of ω t in (2.6) and the construction of confidence set such that P ( f ∗ ∈ (cid:84) F t ) ≥ 1 − 2 δ and P ( f t ∈ (cid:84) F t , f ∗ ∈ (cid:84) F t ) ≥ 1 − 4 δ via a union bound.
As more data is collected, the model uncertainty is reduced and the sum of confidence set width ω t will be sublinear in T (c.f. Lemma B.5 and B.6), indicating sublinear regret.
When it comes to CDPO, we decompose the regret as
[Equation]
[Equation]
where the CDPO policy π t is defined in (4.2).
Since E [ V f ∗ π ∗ − V f t π ft ] = 0 , we have
BayesRegret( T, π [Equation] , ϕ ) =
[Equation] (cid:88)
[Equation]
≤
T (cid:88) t =
[Equation]
where the first inequality follows from the greediness of q t and π t in the dual update steps, i.e.,
V (cid:101) f
t π ft ≤ V (cid:101)
f t q t for any π f t as well as E [ V f t π t ] ≥ E [ V f t q t ] .
The 8 δT term is introduced since (cid:101) f t ∈ F t and P ( f t ∈ (cid:84) F t , (cid:101) f t ∈ (cid:84) F t ) ≥ 1 − 2 δ .
Theorem 5.1 indicates that although CDPO performs conservative updates and abandons the sampling process, it matches the statistical efficiency of PSRL up to constant factors.
The importance of the reference model and the dual procedure is also reflected in the proof.
The referential update builds the bridge between V f t π ft and V f t π t .
Policy optimization under the reference model mimics the sampling-then-optimization procedure of PSRL while offering more stability when the reference is steady, e.g., the least squares estimate we use.
We formalize this idea below.
5.2 CDPO Policy Iterative Improvement
One motivation for the conservative update is that it maximizes (thus improves) the expected value over the posterior.
In this section, we are interested in the policy value improvement under any unknown f ∗ .
Namely, we seek to have the iterative improvement bound J ( π t ) − J ( π t − 1 ) , where the true objective J is defined in (2.3).
We impose the following regularity conditions on the underlying MDP transition and the state-action visitation.
Assumption 5.2 (Regularity Condition on MDP Transition) .
Assume that the MDP transition function f ∗ : S ×
A → S is with additive σ -sub-Gaussian noise and bounded norm, i.e., ∥ s ∥ 2 ≤ C .
Assumption 5.3 (Regularity Condition on State-Action Visitation) .
We assume that there exists κ > 0 such that for any policy π t , t ∈ [1 , T ] ,
[Equation]
where dρ q t +1 /dρ
π t is the Radon-Nikodym derivative of ρ q t +1 with respect to ρ π t .
Theorem 5.4 (Policy Iterative Improvement) .
Suppose we have ∥ (cid:101) f
( · , · ) ∥ ≤ C for (cid:101) f ∈ F where the model class F is finite.
Define ι := max s,a | A f ∗ π ( s, a ) | , where A f ∗ π is the advantage function defined as A f ∗ π ( s, a ) :=
Q f ∗ π ( s, a ) − V f ∗ π ( s ) .
With probability at least 1 − δ , the policy improvement between successive iterations is bounded by
[Equation]
where ∆( t ) :=
E s ∼ ζ (cid:2) V (cid:101)
f t q t ( s ) − V (cid:101) f
t q t − 1 ( s ) (cid:3) ≥ 0 due to the greediness of q t .
The above theorem provides the iterative improvement bound following the CDPO algorithm.
When
H is large enough, the policy value improvement is at least ∆( t ) by choosing a properly small η .
In particular, the first term ∆( t ) characterizes the policy improvement brought by the greedy exploitation in (4.1), and ∆( t ) ≥ 0 since q t is optimal under the reference model (cid:101) f t .
The second term in (5.6) accounts for the generalization error of least square methods.
Specifically, model (cid:101) f t
= (cid:98) f LSt ∈
F t is trained to fit the history samples.
However, we seek to have the model error bound over the stateaction visitation measure, which requires the deviation from the empirical mean to its expectation using Bernstein’s inequality and union bound.
Finally, the trust-region constraint in (4.2) brings the 4 ηα/ (1 − γ ) term, which reduces to zero if η is small.
This makes intuitive sense as η controls the degree of conservative exploration.
5.3 Global Optimality of CDPO
We now analyze the global optimality of CDPO by studying its expected regret.
As discussed in
Section 3, agnostic reinforcement learning is impossible.
Without structural assumptions, additional complexity measure is required for a generalization bound beyond finite settings.
For this reason, we adopt the notation of eluder dimension [49, 43], defined as follows:
Definition 5.5 (( F , ε )-Dependence) .
If we say ( s, a ) ∈ S×A is ( F , ε ) -dependent on { ( s i , a i ) } ni =1 ⊆ S × A , then [Equation] [Equation]
Conversely, ( s, a ) ∈ S × A is ( F , ε )-independent of { ( s i , a i ) } ni =1 if and only if it does not satisfy the definition for dependence.
Definition 5.6 (Eluder Dimension) .
The eluder dimension dim E ( F , ε ) is the length of the longest possible sequence of elements in S×A such that for some ε ′ ≥ ε , every element is ( F , ε ′ )-independent of its predecessors.
We make the following assumption on the Lipschitz continuity of the value function.
Assumption 5.7 (Lipschitz Continuous Value) .
At iteration t , assume the value function V f t π for any policy π is Lipschitz continuous in the sense that | V f t π ( s 1 ) − V f t π ( s 2 ) | ≤ L t ∥ s 1 − s 2 ∥ 2 .
Notably, Assumption 5.7 holds under certain regularity conditions of the MDP, e.g. when the transition and rewards are Lipschitz continuous [5, 47].
Under this assumption, many RL settings can be satisfied [13], e.g., nonlinear models with stochastic Lipschitz policies and Lipschitz reward models, and is thus adopted by various model-based RL work [35, 7, 13].
We now study the global optimality of CDPO by the following expected regret theorem, which can be seen as a direct consequence of Theorem 5.1 that states the statistical equivalence between CDPO and PSRL.
Theorem 5.8 (Expected Regret of CDPO) .
Let N ( F , α, ∥·∥ 2 ) be the α -covering number of F .
Denote d E :=
dim E ( F , T − 1 ) for the eluder dimension of F at precision 1 /T .
Under Assumption 5.2 and 5.7, the cumulative expected regret of CDPO in T iterations is bounded by
[Equation]
where β := 8 σ 2 log (cid:16) 2 N (cid:0) F , 1 / ( T 2 ) , ∥·∥ 2 (cid:1) T (cid:17) + 2 (cid:0) 8 C + (cid:112) 8 σ 2 log(8 T 3 ) (cid:1) /T
and L :=
E [ L t ] .
Here, the covering number is introduced since we are considering F that may contain infinitely many functions, for which we cannot simply apply a union bound.
Besides, β is the confidence parameter that contains f ∗ with high probability (via concentration inequality).
To clarify the asymptotics of the expected regret bound, we introduce another measure of dimensionality that captures the sensitivity of F to statistical overfitting.
Corollary 5.9 (Asymptotic Bound) .
Define the Kolmogorov dimension w.r.t.
function class F as
[Equation]
Under the assumptions of Theorem 5.8 and by omitting terms logarithmic in T , the regret of CDPO is
[Equation]
The sublinear regret result permits the global optimality and sample efficiency for any model class with a reasonable complexity measure.
Meanwhile, the iterative improvement theorem guarantees efficient exploration and good performance even when the model class is highly nonlinear.
Empirical Evaluation 6.1 Understanding Different Exploration Mechanisms
We first provide insights and evidence of why CDPO exploration can be more efficient in the tabular
N -Chain MDPs, which have optimal right actions and suboptimal left actions at each of the N states.
Settings and full results are provided in Appendix F.2.
In Figure 1, we compare the posterior of CDPO and PSRL at the state that is the furthest away from the initial state, i.e. the state that is the hardest for the agents to reach and explore.
[Figure] [Figure]
When training starts, both algorithms have a large variance of value estimation.
However, as training progresses, CDPO gives more accurate and certain estimates, but only for the optimal right actions not for the suboptimal left actions, while PSRL agents explore both directions.
This verifies the potential over-exploration issue in PSRL: as long as the uncertainty contains unrealistically large values, PSRL agents can perform uninformative exploration by acting suboptimally according to an inaccurate sampled model.
In contrast, CDPO replaces the sampled model with a stable mean estimate and cares about the expected value, thus avoiding such pitfalls.
We see in Figure 2 that although CDPO has much larger uncertainty for the suboptimal left actions, its regret is lower.
6.2 Exploration Efficiency with Nonlinear Model Class
In finite MDPs, PSRL-style agents can specify and try every possible action to finally obtain an accurate high-confidence prediction.
However, our discussion in Section 3 indicates that a similar over-exploration issue in more complex environments can lead to less informative exploration steps, which only eliminate an exponentially small portion of the uncertainty.
To see its impact on the training performance, we report the results of provable algorithms with nonlinear models on several MuJoCo tasks in Figure 3. For OFU-RL, we mainly evaluate HUCRL [10], a deep algorithm proposed to deal with the intractability of the joint optimization.
We observe that all algorithms achieve asymptotic optimality in the inverted pendulum.
Since the dimension of the pendulum task is low, learning an accurate (and thus generalizable) model poses no actual challenge.
However, in higher dimensional tasks such as half-cheetah, CDPO achieves a higher asymptotic value with faster convergence.
Implementation details and hyperparameters are provided in Appendix F.1.
[Figure]
6.3 Comparison with Prior RL Algorithms
We also examine a broader range of MBRL algorithms, including MBPO [20], SLBO [35], and
ME-TRPO [30].
The model-free baselines include SAC [16], PPO [54], and MPO [2].
The results are shown in Figure 4.
We observe that CDPO achieves competitive or higher asymptotic performance while requiring fewer samples compared to both the model-based and the model-free baselines.
[Figure]
6.4 Ablation Study
We conduct ablation studies to provide a better understanding of the components in CDPO.
One can observe from Figure 5 that the policies updated with only Referential Update or Conservative
Update lag behind the dual framework.
We also test the necessity and sensitivity of the constraint hyperparameter η .
We see that a constant η and a time-decayed η achieve similar asymptotic values with a similar convergence rate, showing the robustness of CDPO.
However, removing the constraint will lose the policy improvement guarantee, thus causing degradation.
Ablation on different choices of MBPO solver (Dyna and POPLIN-P [63]) shows the generalizability of CDPO.
[Figure]
Conclusions & Future Work
In this work, we present Conservative Dual Policy Optimization (CDPO), a simple yet provable modelbased algorithm.
By iterative execution of the Referential Update and Conservative Update , CDPO explores within a reasonable range while avoiding aggressive policy update.
Moreover, CDPO gets rid of the harmful sampling procedure in previous provable approaches. Instead, an intermediate policy is optimized under a stable reference model, and the agent conservatively explore the environment by maximizing the expected policy value.
With the same order of regret as PSRL, the proposed algorithm can achieve global optimality while monotonically improving the policy.
Considering our naive choice of the reference model, other more sophisticated designs should be a fruitful future direction.
It will also be interesting to explore different choices of the MBPO solvers, which we would like to leave as future work.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Some components in our algorithm are naively designed.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix H. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption 5.2 and 5.7.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Our code can be found in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]