Original: When to Update Your Model: Constrained Model-based Reinforcement Learning
Revised: When to Update Your Model: Constrained Model-based Reinforcement Learning

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Designing and analyzing model-based RL (MBRL) algorithms with guaranteedmonotonic improvement has been challenging, mainly due to the interdependencebetween policy optimization and model learning.
Revised: Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning.

Original: Existing discrepancy boundsgenerally ignore the impacts of model shifts, and their corresponding algorithmsare prone to degrade performance by drastic model updating.
Revised: Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating.

Original: In this work, we firstpropose a novel and general theoretical scheme for a non-decreasing performanceguarantee of MBRL.
Revised: In this work, we first propose a novel and general theoretical scheme for a non-decreasing performance guarantee of MBRL.

Original: Our follow-up derived bounds reveal the relationship betweenmodel shifts and performance improvement.
Revised: Our follow-up derived bounds reveal the relationship between model shifts and performance improvement.

Original: These discoveries encourage us toformulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL.
Revised: These discoveries encourage us to formulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL.

Original: A further example demonstrates that learning models from adynamically-varying number of explorations benefit the eventual returns.
Revised: A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.

Original: Motivatedby these analyses, we design a simple but effective algorithm CMLO (Constrained
Revised: Motivated by these analyses, we design a simple but effective algorithm CMLO (Constrained

Original: Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model.
Revised: Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model.

Original: Experiments show that
Revised: Experiments show that

Original: CMLO surpasses other state-of-the-art methods and produces a boost when variouspolicy optimization methods are employed.
Revised: CMLO surpasses other state-of-the-art methods and produces a boost when various policy optimization methods are employed.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) has driven impressive advances in many complex decision-makingproblems in recent years [33, 43].
Revised: Reinforcement learning (RL) has driven impressive advances in many complex decision-making problems in recent years [36, 48].

Original: Many of these advances are obtained by model-free (MFRL)methods, whose desirable asymptotic performance yet comes at the cost of sample efficiency.
Revised: Many of these advances are obtained by model-free (MFRL) methods, whose desirable asymptotic performance yet comes at the cost of sample efficiency.

Original: Hencetheir applications are mostly limited to simulation scenarios [40, 30, 32].
Revised: Hence their applications are mostly limited to simulation scenarios [45, 33, 35].

Original: In contrast, Model-Based RL(MBRL) methods, which learn a transition model directly from orders-of-magnitude fewer samplesand then derive the optimal policy from the learned model, have become an appealing alternative insmall-data and more practical cases [34, 9, 37, 16].
Revised: In contrast, Model-Based RL (MBRL) methods, which learn a transition model directly from orders-of-magnitude fewer samples and then derive the optimal policy from the learned model, have become an appealing alternative in small-data and more practical cases [37, 9, 41, 18].

Original: In general, MBRL methods alternate between the two stages: model learning and policy optimization( e.g. the general Dyna-style [45, 46]).
Revised: In general, MBRL methods alternate between the two stages: model learning and policy optimization ( e.g. the general Dyna-style [50, 51]).

Original: A more accurate model will lead to a better policy.
Revised: A more accurate model will lead to a better policy.

Original: Variousattempts have been proposed to improve model accuracy by investigating high-capacity models (themodel ensemble technique [25, 8] and better function approximators [14, 35]) or amending the policyoptimization stage based on model bias [22, 36, 26, 18, 7, 51].
Revised: Various attempts have been proposed to improve model accuracy by investigating high-capacity models (the model ensemble technique [27, 8] and better function approximators [15, 38]) or amending the policy optimization stage based on model bias [24, 39, 28, 20, 7, 57].

Original: Even so, their resultant models arejust accurate in a local and relative sense, since the learning is conditional on a fixed number of stateaction tuples explored by the policy at the current step, other than the full transition dynamics of theenvironment.
Revised: Even so, their resultant models are just accurate in a local and relative sense, since the learning is conditional on a fixed number of stateaction tuples explored by the policy at the current step, other than the full transition dynamics of the environment.

Original: Indeed, it is tricky to determine how much we should explore.
Revised: Indeed, it is tricky to determine how much we should explore.

Original: Insufficient explorationwould trap the model and the following policy optimization, whereas excessive newly-encounteredstate-action pairs would confuse the model and later cause policy chattering.
Revised: Insufficient exploration would trap the model and the following policy optimization, whereas excessive newly-encountered state-action pairs would confuse the model and later cause policy chattering.

Original: To derive a “truly”accurate model, we need a smarter scheme to choose different numbers of explorations at differenttimes, instead of the unchanged setting in current methods.
Revised: To derive a “truly” accurate model, we need a smarter scheme to choose different numbers of explorations at different times, instead of the unchanged setting in current methods.

Original: From an optimization point of view, the thinking of how to derive an accurate model for MBRL ina global sense also motivates us to investigate the monotonicity of the optimization target ( i.e. thereturn of the learned model and policy in MBRL), which, unfortunately, is less explored and notwell guaranteed in current works.
Revised: From an optimization point of view, the thinking of how to derive an accurate model for MBRL in a global sense also motivates us to investigate the monotonicity of the optimization target ( i.e. the return of the learned model and policy in MBRL), which, unfortunately, is less explored and not well guaranteed in current works.

Original: However, discussing the monotonicity guarantee for MBRL ischallenging by any means, arising from the coupling of the model learning and policy optimizationprocesses.
Revised: However, discussing the monotonicity guarantee for MBRL is challenging by any means, arising from the coupling of the model learning and policy optimization processes.

Original: Although there has been recent interest in related subjects, most of the theoretical worksseek to characterize the monotonicity in terms of a fixed model of interest [44, 31, 18, 11, 26], whichdoes not naturally fit our case when the model is dynamically shifted.
Revised: Although there has been recent interest in related subjects, most of the theoretical works seek to characterize the monotonicity in terms of a fixed model of interest [49, 34, 20, 12, 28], which does not naturally fit our case when the model is dynamically shifted.

Original: In this paper, we study how to guarantee the optimization monotonicity theoretically, upon which wethen develop an event-triggered strategy that learns the model from a dynamically-varying number ofexplorations.
Revised: In this paper, we study how to guarantee the optimization monotonicity theoretically, upon which we then develop an event-triggered strategy that learns the model from a dynamically-varying number of explorations.

Original: In particular, we interestingly find that the lower bound of the performance improvementbetween two adjacent alternation steps in MBRL is dependent on the one-step model accuracy plusthe constraint of the model shifts under certain mild assumptions.
Revised: In particular, we interestingly find that the lower bound of the performance improvement between two adjacent alternation steps in MBRL is dependent on the one-step model accuracy plus the constraint of the model shifts under certain mild assumptions.

Original: This discovery encourages us toformulate a constrained optimization problem, in order to permit positive performance improvementand thus the optimization monotonicity for MBRL.
Revised: This discovery encourages us to formulate a constrained optimization problem, in order to permit positive performance improvement and thus the optimization monotonicity for MBRL.

Original: We also give a feasible solution example to showthat dynamical alternation between model learning and policy exploration does benefit performancemonotonicity.
Revised: We also give a feasible solution example to show that dynamical alternation between model learning and policy exploration does benefit performance monotonicity.

Original: To resolve the constrained optimization problem, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound) equipped with an event-triggered mechanism.
Revised: To resolve the constrained optimization problem, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound) equipped with an event-triggered mechanism.

Original: This mechanism first estimates whether the model shifts meet the constraint and then decides whento train the model.
Revised: This mechanism first estimates whether the model shifts meet the constraint and then decides when to train the model.

Original: We evaluate CMLO on several continuous control benchmark tasks.
Revised: We evaluate CMLO on several continuous control benchmark tasks.

Original: The results show that CMLOlearns much faster than other state-of-the-art MBRL methods and yields promising asymptoticperformance compared with the model-free counterparts.
Revised: The results show that CMLO learns much faster than other state-of-the-art MBRL methods and yields promising asymptotic performance compared with the model-free counterparts.

Original: Note that our optimization framework isgeneral and can be applied to different backbones of policy optimization algorithms, which is alsoablated in our experiments.
Revised: Note that our optimization framework is general and can be applied to different backbones of policy optimization algorithms, which is also ablated in our experiments.

Original:  Related works
Revised:  Related works

Original: Model-based reinforcement learning methods have shown great potential for sequential decisionmaking both in simulation and in the real world due to their sample efficiency [9, 19].
Revised: Model-based reinforcement learning methods have shown great potential for sequential decisionmaking both in simulation and in the real world due to their sample efficiency [9, 21].

Original: Generally, these
Revised: Generally, these

Original: MBRL algorithms can be grouped into several categories to highlight the range of uses of predictivemodels [50].
Revised: MBRL algorithms can be grouped into several categories to highlight the range of uses of predictive models [55].

Original: And our work falls into the Dyna-style category.
Revised: And our work falls into the Dyna-style category.

Original: In Dyna-style algorithms, trainingalternates between model learning under policy iterations with the real environments, and policyoptimization using the model rollouts [45–47, 12].
Revised: In Dyna-style algorithms, training alternates between model learning under policy iterations with the real environments, and policy optimization using the model rollouts [50–52, 13].

Original: Many attempts have been devoted to improvingthese two stages.
Revised: Many attempts have been devoted to improving these two stages.

Original: For model learning stage, previous main concerns are function approximators and training objectives.
Revised: For model learning stage, previous main concerns are function approximators and training objectives.

Original: The dynamics approximator has advanced from Gaussian processes [24, 9], time-varying lineardynamics [27, 28] to neural network predictive models [14, 35].
Revised: The dynamics approximator has advanced from Gaussian processes [26, 9], time-varying linear dynamics [29, 30] to neural network predictive models [15, 38].

Original: And training objectives vary from
Revised: And training objectives vary from

Original: Mean Square Error (MSE) [35, 31], Negative Log Likelihood (NLL) [8, 18], etc .
Revised: Mean Square Error (MSE) [38, 34], Negative Log Likelihood (NLL) [8, 20], etc .

Original: Moreover, the deepensemble technique is appealing for improving the robustness to model error.
Revised: Moreover, the deep ensemble technique is appealing for improving the robustness to model error.

Original: Our method adopts anensemble of probabilistic networks similarly as in [8, 18].
Revised: Our method adopts an ensemble of probabilistic networks similarly as in [8, 20].

Original: The policy optimization stage allows Dyna-style algorithms to leverage various off-the-shelf modelfree methods, such as SAC [15], TRPO [39], and TD3 [13].
Revised: The policy optimization stage allows Dyna-style algorithms to leverage various off-the-shelf modelfree methods, such as SAC [16], TRPO [44], and TD3 [14].

Original: Much owing to the progress of model-freemethods, our method is to invoke any reasonable optimization oracle for the empirical models, ratherthan entangle a particular policy optimization algorithm.
Revised: Much owing to the progress of model-free methods, our method is to invoke any reasonable optimization oracle for the empirical models, rather than entangle a particular policy optimization algorithm.

Original: A consensus of MBRL is that a smart policy requires an accurate model.
Revised: A consensus of MBRL is that a smart policy requires an accurate model.

Original: However, model bias cannotbe eliminated because the state-action distribution of the samples in the model learning stage andpolicy optimization stage is quite different.
Revised: However, model bias cannot be eliminated because the state-action distribution of the samples in the model learning stage and policy optimization stage is quite different.

Original: Many prior works attend to this distribution mismatchissue and then tailor the data used for policy optimization according to the model bias.
Revised: Many prior works attend to this distribution mismatch issue and then tailor the data used for policy optimization according to the model bias.

Original: Janneret al.
Revised: Janner et al.

Original: [18], Buckman et al.
Revised: [20], Buckman et al.

Original: [7] encourage truncated rollout lengths.
Revised: [7] encourage truncated rollout lengths.

Original: Besides, the ratio of real tomodel-generated data can be dynamically tuned according to the model uncertainty [22, 36, 26].
Revised: Besides, the ratio of real to model-generated data can be dynamically tuned according to the model uncertainty [24, 39, 28].

Original: Yuet al.
Revised: Yu et al.

Original: [51] penalizes rewards by the model uncertainty.
Revised: [57] penalizes rewards by the model uncertainty.

Original: Our method incorporates the truncated modelrollouts mechanism.
Revised: Our method incorporates the truncated model rollouts mechanism.

Original: Moreover, we further explore how real interactions affect overall performancewhich is rarely studied before.
Revised: Moreover, we further explore how real interactions affect overall performance which is rarely studied before.

Original: We construct an event-triggered mechanism to cope with overfitting ina small-data regime and suffering generalization error when facing a drastic distribution shift.
Revised: We construct an event-triggered mechanism to cope with overfitting in a small-data regime and suffering generalization error when facing a drastic distribution shift.

Original: Monotonic improvement guarantee has been a fundamental concern in both model-free and modelbased avenues.
Revised: Monotonic improvement guarantee has been a fundamental concern in both model-free and modelbased avenues.

Original: In MFRL methods, both CPI [20] and TRPO [39] can be understood as approximatingand optimizing the performance gap by forcing the new policy to be not too far away from the currentpolicy.
Revised: In MFRL methods, both CPI [22] and TRPO [44] can be understood as approximating and optimizing the performance gap by forcing the new policy to be not too far away from the current policy.

Original: However, in Model-based settings, their trust-region constraints cannot directly be satisfiedbecause the policies highly depend on the randomness of the models.
Revised: However, in Model-based settings, their trust-region constraints cannot directly be satisfied because the policies highly depend on the randomness of the models.

Original: While constructing such abound for performance gap is straightforward, it has not been explored in previous MBRL theorecticalanalyses, instead they [44, 31, 18, 11, 26] turn to bound the discrepancy between returns under amodel and those in the real environment.
Revised: While constructing such a bound for performance gap is straightforward, it has not been explored in previous MBRL theorectical analyses, instead they [49, 34, 20, 12, 28] turn to bound the discrepancy between returns under a model and those in the real environment.

Original: Although they could guarantee that the lower bound ofpolicy performance improves under a certain model, this guarantee may face several issues regardingmodel updating.
Revised: Although they could guarantee that the lower bound of policy performance improves under a certain model, this guarantee may face several issues regarding model updating.

Original: In contrast, we construct the performance difference scheme for MBRL algorithmsand perform monotonicity analysis under this scheme.
Revised: In contrast, we construct the performance difference scheme for MBRL algorithms and perform monotonicity analysis under this scheme.

Original: Another line of theoretical works focus on regret bounds [10, 21] or sample complexity properties [1,5, 42], focusing on the convergence performance and sample complexity for model-based approaches.
Revised: Another line of theoretical works focus on regret bounds [10, 23] or sample complexity properties [1, 5, 47], focusing on the convergence performance and sample complexity for model-based approaches.

Original:  Preliminaries
Revised:  Preliminaries

Original: Markov Decision Process A discounted Markov Decision Process (MDP) is a quintuple M =( S , A , P M , r M , γ ) , where S is the state space, A represents the action space, P M denotes thetransition function, r : S ×
Revised: Markov Decision Process A discounted Markov Decision Process (MDP) is a quintuple M = ( S , A , P M , r M , γ ) , where S is the state space, A represents the action space, P M denotes the transition function, r : S ×

Original: A → [ − R, R ] stands for the reward function, and γ ∈ (0 , 1) is thediscount factor.
Revised: A → [ − R, R ] stands for the reward function, and γ ∈ (0 , 1) is the discount factor.

Original: For a fixed policy π and model M , we define V πM ( µ ) as the return of the model Mwith the starting state distribution µ , and V π ( µ ) denotes the returns under the real environment,
Revised: For a fixed policy π and model M , we define V πM ( µ ) as the return of the model M with the starting state distribution µ , and V π ( µ ) denotes the returns under the real environment,

Original: [Equation]
Revised: [Equation]

Original: We make a mild assumption that the model M to be identical to the real MDP except the transitionfunction.
Revised: We make a mild assumption that the model M to be identical to the real MDP except the transition function.

Original: Let d π k M i ( s, a ; µ ) denote the visitation probability s, a when starting at s 0 ∼ µ and followingπ k under the dynamics P M i .
Revised: Let d π k M i ( s, a ; µ ) denote the visitation probability s, a when starting at s 0 ∼ µ and following π k under the dynamics P M i .

Original: We will omit it as d π k M i henceforth for brevity.
Revised: We will omit it as d π k M i henceforth for brevity.

Original: Besides, we denote M asa (parameterized) family of models of interest, and let Π be a family of policies.
Revised: Besides, we denote M as a (parameterized) family of models of interest, and let Π be a family of policies.

Original: Generative Model
Revised: Generative Model

Original: Many previous works [29, 23, 1] focus on a stylized generative model.
Revised: Many previous works [31, 25, 1] focus on a stylized generative model.

Original: Byassuming an access to the generative model, we collect N samples for each state-action pair ( s, a ) ∈
Revised: By assuming an access to the generative model, we collect N samples for each state-action pair ( s, a ) ∈

Original: S × A : s is,a i.i.d ∼ P ( ·| s, a ) which allows us to construct an empirical model defined as follows:
Revised: S × A : s is,a i.i.d ∼ P ( ·| s, a ) which allows us to construct an empirical model defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 {·} is the indicator function.
Revised: where 1 {·} is the indicator function.

Original: This leads to an empirical MDP ˆ M = ( S , A , ˆ P, r, γ ) .
Revised: This leads to an empirical MDP ˆ M = ( S , A , ˆ P, r, γ ) .

Original:  Monotonic improvement under model shifts
Revised:  Monotonic improvement under model shifts

Original: This section provides theoretical analyses for monotonic improvement of MBRL, factoring in theinterdependence between policies and models.
Revised: This section provides theoretical analyses for monotonic improvement of MBRL, factoring in the interdependence between policies and models.

Original: We first construct a general scheme for a nondecreasing performance guarantee and follow it up by characterizing the lower bound when shiftingthe model.
Revised: We first construct a general scheme for a nondecreasing performance guarantee and follow it up by characterizing the lower bound when shifting the model.

Original: Towards a non-negative lower bound, we restrict the model shifts and then obtain arefined bound.
Revised: Towards a non-negative lower bound, we restrict the model shifts and then obtain a refined bound.

Original: These discoveries encourage us to translate the bound maximization to a constrainedoptimization problem to permit monotonicity.
Revised: These discoveries encourage us to translate the bound maximization to a constrained optimization problem to permit monotonicity.

Original: By deriving an instance solution under the generativemodel setting, we demonstrate the merits of the dynamic model learning interval.
Revised: By deriving an instance solution under the generative model setting, we demonstrate the merits of the dynamic model learning interval.

Original: 4.1 Monotonic improvement with policy optimization oracle
Revised: 4.1 Monotonic improvement with policy optimization oracle

Original: Our goal is to construct a general recipe for a monotonicity guarantee.
Revised: Our goal is to construct a general recipe for a monotonicity guarantee.

Original: Naturally, we seek to build aperformance difference scheme for model-based algorithms.
Revised: Naturally, we seek to build a performance difference scheme for model-based algorithms.

Original: Definition 4.1 ( Performance Difference Bound Scheme ) .
Revised: Definition 4.1 ( Performance Difference Bound Scheme ) .

Original: V π i | M i ( µ ) denotes the return of thepolicy π i ∈ Π in the real environment, whereas π i is derived from the dynamical model M i ∈ M .
Revised: V π i | M i ( µ ) denotes the return of the policy π i ∈ Π in the real environment, whereas π i is derived from the dynamical model M i ∈ M .

Original: Then, the lower bound on the true return gap of π 1 and π 2 can be stated in the form,[Equation]
Revised: Then, the lower bound on the true return gap of π 1 and π 2 can be stated in the form,

Original: [Equation]
Revised: [Equation]

Original: Such a statement guarantees that the policy allows non-decreasing performance in the real environment when C is non-negative.
Revised: Such a statement guarantees that the policy allows non-decreasing performance in the real environment when C is non-negative.

Original: Although there has been interest in non-decreasing performance guarantee, previous works [31, 18]commonly derive under a "discrepancy bound" scheme disregarding the model shifts ( i.e. M 1 = M 2 ).
Revised: Although there has been interest in non-decreasing performance guarantee, previous works [34, 20] commonly derive under a "discrepancy bound" scheme disregarding the model shifts ( i.e. M 1 = M 2 ).

Original: Their results imply that once a policy update π 1 → π 2 increases the returns under the same model( V π 2 M 1 > V π 1 M 1 ), the lower bound on the policy performance evaluated in the real environment improvesaccordingly, inf { V π 2 | M 1 } > inf { V π 1 | M 1 }.
Revised: Their results imply that once a policy update π 1 → π 2 increases the returns under the same model ( V π 2 M 1 > V π 1 M 1 ), the lower bound on the policy performance evaluated in the real environment improves accordingly, inf { V π 2 | M 1 } > inf { V π 1 | M 1 }.

Original: When the model shift is introduced, establishing the performance difference bound turns out tobe rather difficult, mainly due to the coupling of the policy optimization and model learning: theestimated model is generated from the policy explorations, while the policy derives from the modelrollouts.
Revised: When the model shift is introduced, establishing the performance difference bound turns out to be rather difficult, mainly due to the coupling of the policy optimization and model learning: the estimated model is generated from the policy explorations, while the policy derives from the model rollouts.

Original: Hence, we need to consider the performance gap arising from the integration of the twostages, which, unfortunately, has never been explored before.
Revised: Hence, we need to consider the performance gap arising from the integration of the two stages, which, unfortunately, has never been explored before.

Original: Since the performance of the policywith a fixed model is already well guaranteed, it is natural to make the following assumptions.
Revised: Since the performance of the policy with a fixed model is already well guaranteed, it is natural to make the following assumptions.

Original: Assumption 4.2 ( Policy Optimization Oracle ) .
Revised: Assumption 4.2 ( Policy Optimization Oracle ) .

Original: The policy optimization oracle is defined as theone that takes as input a model M and returns a ϵ opt -optimal policy π satisfying: V ∗ M ( µ ) − ϵ opt ≤
Revised: The policy optimization oracle is defined as the one that takes as input a model M and returns a ϵ opt -optimal policy π satisfying: V ∗ M ( µ ) − ϵ opt ≤

Original: V πM ( µ ) ≤ V ∗ M ( µ ) .
Revised: V πM ( µ ) ≤ V ∗ M ( µ ) .

Original: We assume that our policy optimization stage always meets the policy optimizationoracle given its corresponding estimated model M .
Revised: We assume that our policy optimization stage always meets the policy optimization oracle given its corresponding estimated model M .

Original: Such assumption usually holds in practice when we explore existing off-the-shelf model-free algorithms [38, 1, 15, 39].
Revised: Such assumption usually holds in practice when we explore existing off-the-shelf model-free algorithms [43, 1, 16, 44].

Original: With this assumption, we can focus directly on the eventual performancedifference under the real environment encountering the model updating.
Revised: With this assumption, we can focus directly on the eventual performance difference under the real environment encountering the model updating.

Original: The bound C we seek can be expressed in terms of two kinds of gaps: the inconsistency gap betweenthe model and the environment, and the optimal returns gap between the two models.
Revised: The bound C we seek can be expressed in terms of two kinds of gaps: the inconsistency gap between the model and the environment, and the optimal returns gap between the two models.

Original: When apolicy π k samples in a model M i , it will encounter states not consistent with those generated by thereal environment.
Revised: When a policy π k samples in a model M i , it will encounter states not consistent with those generated by the real environment.

Original: We denote this inconsistency by 
Revised: We denote this inconsistency by ϵ π k M i =

Original: [Equation] [Equation]
Revised: E s,a ∼ d πk [ D TV ( P ( ·| s, a ) ∥

Original:  Besides, for each model M i ∈ M , a remarkable property is that there always exists a policy thatmaximizes the value function V πM i ( µ ) [48].
Revised: P M i ( ·| s, a ))] . Besides, for each model M i ∈ M , a remarkable property is that there always exists a policy that maximizes the value function V πM i ( µ ) [53].

Original: Hence, we define the ceiling performance of model
Revised: Hence, we define the ceiling performance of model

Original: M as V ∗ M i ( µ ) =
Revised: M as V ∗ M i ( µ ) =

Original: sup π ∈ Π V πM i ( µ ) .
Revised: sup π ∈ Π V πM i ( µ ) .

Original: We will omit µ henceforth for simplicity unless confusion exists.
Revised: We will omit µ henceforth for simplicity unless confusion exists.

Original: With these two terms well-defined, we now present our bound.
Revised: With these two terms well-defined, we now present our bound.

Original: Theorem 4.3 ( Performance Difference Bound for Model-based RL ) .
Revised: Theorem 4.3 ( Performance Difference Bound for Model-based RL ) .

Original: Let M i ∈
Revised: Let M i ∈

Original: M be theestimated models and π i be the ϵ opt -optimal policy for M i .
Revised: M be the estimated models and π i be the ϵ opt -optimal policy for M i .

Original: Recalling κ = 2 Rγ (1 − γ ) 2 where R is thebound of the reward function, we have the performance difference of π 1 and π 2 evaluated under thereal environment be bounded as below,
Revised: Recalling κ = 2 Rγ (1 − γ ) 2 where R is the bound of the reward function, we have the performance difference of π 1 and π 2 evaluated under the real environment be bounded as below,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B, Theorem B.1.
Revised: See Appendix B, Theorem B.1.

Original: This bound implies that if the model update M 1 → M 2 can (1) shorten the divergence between theestimated dynamics and the true dynamics and (2) improve the ceiling performance on the model, itmay guarantee overall performance improvement under the true dynamics.
Revised: This bound implies that if the model update M 1 → M 2 can (1) shorten the divergence between the estimated dynamics and the true dynamics and (2) improve the ceiling performance on the model, it may guarantee overall performance improvement under the true dynamics.

Original: 4.2 Lower-bound optimization with model shift constraints
Revised: 4.2 Lower-bound optimization with model shift constraints

Original: Theorem 4.3 provides a general performance difference bound that suggests models with higherceiling performance and lower bias would raise the overall performance.
Revised: Theorem 4.3 provides a general performance difference bound that suggests models with higher ceiling performance and lower bias would raise the overall performance.

Original: However, finding a nonnegative lower bound may face several issues regarding a drastic model shift in practice.
Revised: However, finding a nonnegative lower bound may face several issues regarding a drastic model shift in practice.

Original: On the onehand, performing an abrupt model update could potentially lead to a tumble in ceiling performanceand then fail to make incremental improvements.
Revised: On the one hand, performing an abrupt model update could potentially lead to a tumble in ceiling performance and then fail to make incremental improvements.

Original: On the other hand, huge distribution divergencebetween model rollouts would confound the policy optimization stage and hamper its access tooptimal policies.
Revised: On the other hand, huge distribution divergence between model rollouts would confound the policy optimization stage and hamper its access to optimal policies.

Original: Thus, we shoot for refining the bound upon adding model shifts constraint.
Revised: Thus, we shoot for refining the bound upon adding model shifts constraint.

Original: First, we seek to further unfold the ceiling performance gap, which serves as the main building blocktoward the desired bound.
Revised: First, we seek to further unfold the ceiling performance gap, which serves as the main building block toward the desired bound.

Original: During derivation, we additionally introduce the L -Lipschitz assumption.
Revised: During derivation, we additionally introduce the L -Lipschitz assumption.

Original: Assumption 4.4 ( L -Lipschitzness of Value Function ) .
Revised: Assumption 4.4 ( L -Lipschitzness of Value Function ) .

Original: We call a value function V πM on the estimateddynamical model M is L -Lipschitz w.r.t to some norm ∥ · ∥ in the sense that
Revised: We call a value function V πM on the estimated dynamical model M is L -Lipschitz w.r.t to some norm ∥ · ∥ in the sense that

Original: [Equation]
Revised: [Equation]

Original: [Equation] We assume that our estimated model M ∈ M satisfies the Lipschiz character, inspired from [4, 31].
Revised: We assume that our estimated model M ∈ M satisfies the Lipschiz character, inspired from [4, 34].

Original: Under the L -Lipschitzness assumption, we can derive a bound for the ceiling return gap.
Revised: Under the L -Lipschitzness assumption, we can derive a bound for the ceiling return gap.

Original: Theorem 4.5 ( Ceiling Return Gap under Model Shift ) .
Revised: Theorem 4.5 ( Ceiling Return Gap under Model Shift ) .

Original: For an estimated model M i ∈ M , theceiling return gap is bounded as:
Revised: For an estimated model M i ∈ M , the ceiling return gap is bounded as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B, Theorem B.2.
Revised: See Appendix B, Theorem B.2.

Original: This conclusion reveals the connection between the ceiling return gap and models’ disagreement.
Revised: This conclusion reveals the connection between the ceiling return gap and models’ disagreement.

Original: Ina benign scenario, the term of ceiling performance gap in the RHS of Eq.
Revised: In a benign scenario, the term of ceiling performance gap in the RHS of Eq.

Original: 4.2 should be dominated bythe term κ · ( ϵ π 1 M 1 − ϵ π 2 M 2 ) when the model shift is sufficiently small.
Revised: 4.2 should be dominated by the term κ · ( ϵ π 1 M 1 − ϵ π 2 M 2 ) when the model shift is sufficiently small.

Original: A sharp model shift, however,risks a massive reduction in ceiling performance that is hardly bridged by the other parts in Eq.
Revised: A sharp model shift, however, risks a massive reduction in ceiling performance that is hardly bridged by the other parts in Eq.

Original: 4.2,and therefore corrupts monotonicity.
Revised: 4.2, and therefore corrupts monotonicity.

Original: This inspires us to introduce the constraint of the model shift.
Revised: This inspires us to introduce the constraint of the model shift.

Original: We further refine our performance difference bound to better characterize the relationship betweenperformance gap and model shift.
Revised: We further refine our performance difference bound to better characterize the relationship between performance gap and model shift.

Original: Theorem 4.6 ( Refined Bound with Constraint ) .
Revised: Theorem 4.6 ( Refined Bound with Constraint ) .

Original: Let policy π
Revised: Let policy π

Original: i ∈ Π denotes the ϵ opt optimal policyunder the dynamical model M i ∈ M , and σ M 1 ,M 2 be the constraint threshold for M 1 and M 2 .
Revised: i ∈ Π denotes the ϵ opt optimal policy under the dynamical model M i ∈ M , and σ M 1 ,M 2 be the constraint threshold for M 1 and M 2 .

Original: Note that L ≥ R 1 − γ .
Revised: Note that L ≥ R 1 − γ .

Original: Then we can refine performance difference lower bound under the model shiftsconstraint as,
Revised: Then we can refine performance difference lower bound under the model shifts constraint as,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B, Theorem B.3.
Revised: See AppendixB [Equation] Theorem B.3.

Original: Theorem 4.6 implies that policy π 2 is guaranteed to outperform policy
Revised: Theorem 4.6 implies that policy π 2 is guaranteed to outperform policy

Original: π 1 once it makes the RHS in
Revised: π 1 once it makes the RHS in

Original: Eq.
Revised: Eq.

Original: 4.5 greater than zero under the constraint Eq.
Revised: 4.5 greater than zero under the constraint Eq.

Original: More specifically, to guarantee a non-decreasingperformance, M 2 and π 2 should meet the following two requirements,
Revised: More specifically, to guarantee a non-decreasing performance, M 2 and π 2 should meet the following two requirements,

Original: [Equation]
Revised: [Equation]

Original: R1 encourages us to alleviate model bias as much as possible.
Revised: R1 encourages us to alleviate model bias as much as possible.

Original: Moreover, when the policy π 1 samplestoo many states not encountered by the model M 1 , it may lead to excessive generalization errorsduring model learning, i.e. the estimated model will suffer extrapolation error in these unexperiencedregions, which further causes instability or crashes in the following derived sub-optimal policies.
Revised: Moreover, when the policy π 1 samples too many states not encountered by the model M 1 , it may lead to excessive generalization errors during model learning, i.e. the estimated model will suffer extrapolation error in these unexperienced regions, which further causes instability or crashes in the following derived sub-optimal policies.

Original: Thus, the introduction of the R2 constraint can help solve the above problem.
Revised: Thus, the introduction of the R2 constraint can help solve the above problem.

Original: Finally, we abstractthese two requirements to a constrained optimization problem.
Revised: Finally, we abstract these two requirements to a constrained optimization problem.

Original: Proposition 4.7 ( Constrained Lower-Bound Optimization Problem ) .
Revised: Proposition 4.7 ( Constrained Lower-Bound Optimization Problem ) .

Original: We reduce the issue offinding a non-negative C to the following constrained optimization problem.
Revised: We reduce the issue of finding a non-negative C to the following constrained optimization problem.

Original: Here, π i is still thesub-optimal policy under model M i .
Revised: Here, π i is still the sub-optimal policy under model M i .

Original: The minimal objective in E.q. 4.7 leads to the maximum of C .
Revised: The minimal objective in E.q. 4.7 leads to the maximum of C .

Original: Then the overall optimization problem can be formalized as,
Revised: Then the overall optimization problem can be formalized as,

Original: [Equation]
Revised: [Equation]

Original: 4.3 A feasible example for constrained optimization problem
Revised: 4.3 A feasible example for constrained optimization problem

Original: We first remark that, Proposition 4.7 provides a useful guide for acquiring performance improvementthrough restricting the upcoming model shift into a safe zone.
Revised: We first remark that, Proposition 4.7 provides a useful guide for acquiring performance improvement through restricting the upcoming model shift into a safe zone.

Original: In this section, we give an instance for a feasible solution of the constrained optimization problem under the generative model setting [1, 29].
Revised: In this section, we give an instance for a feasible solution of the constrained optimization problem under the generative model setting [1, 31].

Original: Specifically, we construct an empirical model M 1 from the N samples per state-action pair that stemsfrom the generative model.
Revised: Specifically, we construct an empirical model M 1 from the N samples per state-action pair that stems from the generative model.

Original: Upon encountering another k samples on each state-action pair, wesegue into the model updating stage, which outputs M 2 given these newly collected samples as input.
Revised: Upon encountering another k samples on each state-action pair, we segue into the model updating stage, which outputs M 2 given these newly collected samples as input.

Original: Towards obtaining a non-decreasing performance, we yield the following feasible solution for modeltraining interval through satisfying requirements R1 and R2.
Revised: Towards obtaining a non-decreasing performance, we yield the following feasible solution for model training interval through satisfying requirements R1 and R2.

Original: Corollary 4.8.
Revised: Corollary 4.8.

Original: Here, vol ( S ) denotes the volume of the state coverage simplex.
Revised: Here, vol ( S ) denotes the volume of the state coverage simplex.

Original: For simplicity, wedenote δ M i ( ·| s, a ) =
Revised: For simplicity, we denote δ M i ( ·| s, a ) =

Original: | P ( ·| s, a ) − P M i ( ·| s, a ) | for each model M i ∈ M .
Revised: | P ( ·| s, a ) − P M i ( ·| s, a ) | for each model M i ∈ M .

Original: Under the generative modelsetting, with a probability larger than 1 − ξ , we can provide a non-negative C when given,
Revised: Under the generative model setting, with a probability larger than 1 − ξ , we can provide a non-negative C when given,

Original: [Equation]
Revised: [Equation]

Original: Here, ϵ = δ M 1 ( ·| s, a ) − (1 − γ )
Revised: Here, ϵ = δ M 1 ( ·| s, a ) − (1 − γ )

Original: L R
Revised: L R

Original: [Equation] [Equation] ) − ( [Equation]{+(1 − γ ) 
+}
Revised: · (2 σ M 1 ,M 2 ) − {+(1 − γ ) 
+}

Original: 2 Rγ · ϵ opt and ξ ∈ (0 , 1) is a constant.
Revised: Rγ · ϵ opt and ξ ∈ (0 , 1) is a constant.

Original: Proof.
Revised: Proof.

Original: See Appendix B, Corollary B.4.
Revised: See Appendix B, Corollary B.4.

Original: One can deduce from Corollary 4.8 that dynamically adjusting the model training interval accordingto the model bias and the model shifts constraints threshold does benefit monotonicity.
Revised: One can deduce from Corollary 4.8 that dynamically adjusting the model training interval according to the model bias and the model shifts constraints threshold does benefit monotonicity.

Original: Along withthe model bias δ M 1 ( ·| s, a ) decaying, the model training interval k requires to be scaled up to obtainadequate newly encountered samples for training M 2 .
Revised: Along with the model bias δ M 1 ( ·| s, a ) decaying, the model training interval k requires to be scaled up to obtain adequate newly encountered samples for training M 2 .

Original: Besides, once gathering excessive samples,we risk violating the model shifts constraint, thus impairing performance.
Revised: Besides, once gathering excessive samples, we risk violating the model shifts constraint, thus impairing performance.

Original: This instance supportsthe insight that determining “when to update your model” is vital for performance improvement andmotivates a smarter scheme to choose different numbers of explorations at different times instead ofthe unchanged setting in current methods.
Revised: This instance supports the insight that determining “when to update your model” is vital for performance improvement and motivates a smarter scheme to choose different numbers of explorations at different times instead of the unchanged setting in current methods.

Original:  CLMO framework
Revised:  CLMO framework

Original: It is nontrivial to tackle the proposed constrained optimization problem in Proposition 4.7, sinceone cannot directly assign a value to the optimization variable M 2 .
Revised: It is nontrivial to tackle the proposed constrained optimization problem in Proposition 4.7, since one cannot directly assign a value to the optimization variable M 2 .

Original: We decouple the optimizationobjective and the constraint as “how to train the model” and “when to train the model” through anevent-triggered mechanism towards dynamic alternation.
Revised: We decouple the optimization objective and the constraint as “how to train the model” and “when to train the model” through an event-triggered mechanism towards dynamic alternation.

Original: Objective minimization.
Revised: Objective minimization.

Original: Minimizing the objective function involves improving the model accuracy.
Revised: Minimizing the objective function involves improving the model accuracy.

Original: Specifically, we adopt the model-ensemble technique to reduce model bias.
Revised: Specifically, we adopt the model-ensemble technique to reduce model bias.

Original: For practicalimplementation, the probabilistic models { ˆ f ϕ 1 , ˆ f ϕ 2 , . . .
Revised: For practical implementation, the probabilistic models { ˆ f ϕ 1 , ˆ f ϕ 2 , . . .

Original: , ˆ f ϕ K } are fitted on shared but differentlyshuffled replay buffer D e , and the target is to optimize the Negative Log Likelihood (NLL).
Revised: , ˆ f ϕ K } are fitted on shared but differently shuffled replay buffer D e , and the target is to optimize the Negative Log Likelihood (NLL).

Original: Constraint estimation.
Revised: Constraint estimation.

Original: The unobserved model M 2 makes the constraint function incalculable,and here we seek to design an estimator for it.
Revised: The unobserved model M 2 makes the constraint function incalculable, and here we seek to design an estimator for it.

Original: Recall that D TV ( P M 1 ( ·| s, a ) ∥
Revised: Recall that D TV ( P M 1 ( ·| s, a ) ∥

Original: P M 2 ( ·| s, a ))
Revised: P M 2 ( ·| s, a ))

Original: =12 (cid:80) s
Revised: = 12 (cid:80) s

Original: ′ ∈S | P M 1 ( s ′ | s, a ) − P M 2 ( s ′ | s, a ) | , then we can find that the distance arises from two parts,the state space coverage, and the models’ disagreement.
Revised: ′ ∈S | P M 1 ( s ′ | s, a ) − P M 2 ( s ′ | s, a ) | , then we can find that the distance arises from two parts, the state space coverage, and the models’ disagreement.

Original: We estimate the policy coverage (statespace coverage) by computing the volume vol ( S D ) of the convex closure S D
Revised: We estimate the policy coverage (statespace coverage) by computing the volume vol ( S D ) of the convex closure S D

Original: =
Revised: =

Original: (cid:8) (cid:80) s
Revised: (cid:8) (cid:80) s

Original: i ∈D λ
Revised: i ∈D λ

Original: i S i :λ i ≥ 0 , (cid:80)
Revised: i S i : λ i ≥ 0 , (cid:80)

Original: i λ i = 1 (cid:9) constructed on the replay buffer D .
Revised: i λ i = 1 (cid:9) constructed on the replay buffer D .

Original: We exploit the average prediction erroron these new samples data to estimate the disagreement on newly encountered data ( ∆ D ) and get 
Revised: We exploit the average prediction error on these new samples data to estimate the disagreement on newly encountered data ( ∆ D ) and get L (∆ D ) = E ( s,a,s ′ ) ∈ ∆ D (cid:2) 1 K (cid:80) Ki =1 ∥ s ′ − ˆ f ϕ

Original:  L (∆ D ) =
Revised: i ( s, a ) ∥ (cid:3) . Combining these two components, we can obtain an estimation for the model shift, i.e. , vol ( S D ) · L (∆ D ) .

Revised: This practical overestimation for the model shifts makes the constraint stay satisfied during objective optimization.

Revised: Event-triggered mechanism.

Revised: We design an event-triggered mechanism to determine the occasion to pause collecting and turn to solve the optimization objective.

Revised: Remark that although diverting to train models as long as not to violate the constraint is theoretically reasonable, we refrain from doing so in practice because performing an update on data with a minor shift in coverage and distribution is wasteful and may risk overfitting.

Revised: Therefore, we trigger when the constraint boundary is touched to reduce computational cost and escape overfitting.

Original: E ( s,a,s ′ ) ∈
Revised: The event-triggering mechanism is based on the following condition: [Equation] Here, τ is the event-triggering time, and α is a given constant. Policy optimization oracle.

Revised: Clearly, we can leverage many model-free RL methods (SAC [16],

Revised: TRPO [44], PPO [46] etc. )

Revised: as our policy optimization oracle.

Revised: Besides, we adopt a truncated short model rollouts technique to mitigate compounding error while encouraging model usage.

Revised: Based on the fresh model rollouts, we perform the policy optimization oracle, employing SAC as an example.

Revised: Algorithm Overview.

Revised: We briefly give an overview of our proposed CMLO in algorithm 1.

Revised: Notably, the event-triggered mechanism subtly determines the occasion to perform model updating, promoting performance monotonicity and reducing computation load.

Revised: initialize : policy π θ , ensemble models ˆ f ϕ 1 , ˆ f ϕ 2 , . . .

Revised: , ˆ f ϕ K , environment buffer D e and model

Revised: buffer D m repeat Sample ∆ D e ∼ π θ from real environment; add to D e Estimate model shifts by vol ( S D e ) L (∆ D e ) if Event-triggered condition (E.3) is reached then Train all models ˆ f ϕ 1 , ˆ f ϕ 2 , . .

Revised: , ˆ f ϕ K on D e Perform h -step model rollouts using policy π θ ; add to D m Update π θ on D m through SAC [16]

Revised: until the policy performs well in the environment ;

Revised:  Experiments

Revised: Our experimental evaluation aims to investigate the following questions: (1) How well does our algorithm perform on standard reinforcement learning benchmarks compared to prior state-of-the-art model-based and model-free algorithms?

Revised: (2) Does the performance with or without the constraint consistent with previous theoretical analyses?

Revised: 6.1 Comparative evaluation

Revised: To illustrate the effectiveness of our method, we contrast several popular model-based and model-free baselines.

Revised: Model-free counterparts include: (1) SAC [16], the state-of-the-art in terms of asymptotic performance.

Revised: (2) PPO [46] that explores monotonic improvement as well.

Revised: Model-based baselines include: (3) PETS [8], which employs models directly for planning, different from the Dyna-style.

Revised: (4) SLBO [34], that explores monotonicity under the discrepancy bound scheme.

Revised: (5) MBPO [20], that employs a similar design of model ensemble technique (ensemble of probabilistic dynamics networks) and policy optimization oracle (SAC) as we do.

Revised: (6) AutoMBPO [28], a variant of MBPO, that uses an automatic hyperparameter controller to tune the model-training frequency but suffers from high pre-training cost and lacks theoretical analysis on parameters rationality.

Revised: [Figure]

Revised: evaluate CMLO and these baselines on six continuous control in OpenAI Gym [6] with the MuJoCo [54]

Revised: HalfCheetah

Revised: Hopper

Revised: Walker2d

Revised: Swimmer

Revised: Ant

Revised: Figure 1 shows the learning curves of all compared methods, along with the asymptotic performance.

Revised: These results show that our algorithm is far ahead of the model-free method in terms of sample efficiency, coupled with an asymptotic performance comparable to that of the state-of-the-art modelfree counterparts SAC.

Revised: Compared to model-based baselines, our method gains faster convergence speed and better eventual performance.

Revised: Notably, credit to the event-triggered mechanism, our method enjoys a more stable training curve.

Revised: The better monotonic property of the learning curve agrees with our previous analyses.

Revised: 6.2 Ablation studies

Revised: Next, we make ablations and modifications to our method to validate the effectiveness and generalizability of the mechanism we devised.

Revised: The necessity of event-triggered mechanism.

Revised: To verify the necessity of event-triggered mechanism, we compare to three unconstrained cases (given fixed model training interval) under two environments.

Revised: The training curve and triggered times are shown in Figure 2.

Revised: Clearly, our mechanism improves the performance while reducing the total times of model training.

Revised: Besides, we notice that the performance is comparable to other MBRL baselines (MBPO etc. )

Revised: when fixing our model training interval at 250.

Revised: Still it performs worse than that equipped with a smart mechanism to decide whether to train the model at current exploration step.

Revised: To better understand why the event-triggered mechanism brings up our outperformance, we asses its main bricks.

Revised: Model shift, which reflects the current ability to digest new data, is the basis of triggered condition.

Revised: And we estimate model shift from two parts, policy coverage and prediction error.

Revised: In

Revised: Figure 3 we observe that, gradually, as the training progresses, the policy coverage increases, which reflects our policy has new explorations at every stage without falling into a local optimum prematurely.

Revised: Also, the prediction error gradually decreases, which implies that our estimated dynamics come closer to the true dynamics in the explored region.

Revised: Figure 4 implies that the number of samples required to hit the constraint tends to grows and the model training frequency then goes down.

Revised: This is also consistent with our intuition that, as the model refines and the exploration novelty fades, then the sample size required to reach a certain level of model shifts grows up.

Revised: The result agrees with our

Revised: [Figure]

Revised: These experiments are average over 5 random seeds.

Revised: (a) shows the average return with or without event-triggered mechanism in HalfCheetah and Ant benchmarks.

Revised: (b) shows the number of triggered times per 10k step.

Revised: [Figure]

Revised: [Figure]

Revised: theoretical analyses which describe that a dynamic alternation subject to the model shifts constraint does benefit to monotonicity rather than an assigned one

Revised: [Figure]

Revised: The generalizability of event-triggered mechanism.

Revised: We further investigate the generalizability of our proposed mechanism through ablation on policy optimization oracle, and results are shown in Figure 5.

Revised: 1) under Dyna-style.

Revised: We adopt TRPO [44] as the policy optimization oracle and test the performance with or without event-triggered mechanism in Halfcheetah and Ant benchmarks.

Revised: Observably, our mechanism can effectively guarantee the overall performance improvement and alleviate the local optimization issue.

Revised: 2) jumping off Dyna-style.

Revised: We utilize iLQR [3] and then conduct ablation experiments in the DKitty-Stand [2] and Panda-Reaching [19] scenarios.

Revised: The result shows that adding event-triggered mechanism has a comparable asymptotic performance as the most-frequently triggered case but enjoys a lower computation cost.

Revised:  Conclusion

Revised: We have investigated the role of the decision on "when to update the model" in joint optimization procedures through theoretical and empirical lens.

Revised: We devise a general novel scheme for exploring the monotonicity of MBRL methods, distinguished from the existing discrepancy bound scheme.

Revised: We then derive lower bounds under the scheme, suggesting that models with higher ceiling performance and lower bias guarantee a non-decreasing performance evaluated in the real environment.

Revised: An effective constrained optimization problem comes from the follow-up refined bound to seek a non-negative lower bound.

Revised: Further, the instance under the generative model setting further verifies the effectiveness of learning models from a dynamically varying number of explorations.

Revised: The algorithm CMLO, stemming from these analyses, has asymptotic performance rivaling the best model-free algorithms and boasts better monotonicity.

Revised: Further ablation studies reveal that the proposed mechanism scales to various policy optimization oracles and benefits computation cost reduction.

Revised: Currently, we observed empirically that the event-triggered condition is usually related to specific environments, which will cost a little time for tuning.

Revised: Thus, one direction that merits further investigation is to construct the dual problem of our constrained optimization problem for better exploring optimality and monotonicity.

Revised: Acknowledgments and Disclosure of Funding

Revised: This work was jointly supported by the Sino-German Collaborative Research Project "Crossmodal

Revised: Learning" (NSFC 62061136001/ DFG TRR169), the CAS Project for Young Scientists in Basic

Revised: Research (Grant No.YSBR-040), the National Natural Science Foundation of China (No.62006137) and Beijing Outstanding Young Scientist Program (No.BJJWZYJH012019100020098).

Revised: The authors would also like to thank the anonymous reviewers for their careful reading and their many insightful comments.

Revised: Appendices A Sketch of Theoretical Analyses

Revised: Here, we present our sketch of theoretical analyses.

Revised: We first construct a general scheme (Definition 4.1) for non-decreasing performance guarantee and follow it up by characterizing the lower bound under model shifts (Theorem 4.3).

Revised: Towards seeking a non-negative lower bound, we restrict model shift and refine the bound (Theorem 4.6), then further reduce this issue to a constrained optimization problem (Proposition 4.7).

Revised: With an instance under the generative model setting (Corollary 4.8), we demonstrate the merits of the dynamic model learning interval.

Revised: [Figure]

Revised: B Omitted Proofs

Revised: Theorem B.1 (Performance difference bound for Model-based RL) .

Revised: Let ϵ π j M

Revised: i denote the inconsistency between the learned dynamics P M i and the true dynamics, i.e. ϵ π j M i =

Revised: E s,a ∼ d πj ( s,a ; µ ) [ D TV ( P ( ·| s, a ) ∥

Revised: P M i ( ·| s, a ))] , where d π ( s, a ; µ ) is the probability of visiting s,

Revised: a after starting at state s 0 ∼ µ and following π j ∈ Π thereafter under the true dynamics.

Revised: Let policy π

Revised: i be the ϵ opt -optimal policy under model M i .

Revised: Assume the performance discrepancy of policy π between the estimated model M 1 and the true dynamics be approximated as V π 1 M 1 ( µ ) − V π 1 ( µ )

Revised: = κ · ϵ π 1 M 1 .

Revised: Recall κ = 2 Rγ (1 − γ ) 2 , then the performance gap between π 1 and π 2 evaluated in the true MDP can be bounded by:

Revised: [Equation]

Revised: Proof.

Revised: We overload notation V π i | M i and write V π i for simplicity.

Revised: [Equation]

Revised: We can bound L 1 − L 3 using Lemma C.2, and bound L 2 using the property of ϵ opt -optimal.

Revised: For L 1 − L 3 , with the performance gap approximation of M 1 and π 1 , we apply Lemma C.2, and obtain: L 1 − L 3 ≥ − κ · ( ϵ π 2 M 2 − ϵ π 1 M 1 ) .

Revised: We call a policy π ϵ -optimal under the dynamical model M , if V ∗ M ( s ) ≥ V πM ( s ) ≥ V ∗ M ( s ) − ϵ opt for all s ∈ S .

Revised: Under the assumptions of black-box optimization oracle, we obtain:

Revised: [Equation]

Revised: Adding these two bounds together yields the desired result.

Revised: Remark that, when V π 1 M 1 − V π 1 gets a value far away from κ · ϵ π 1 M 1 , it indicates the performance discrepancy that evaluated between the model M 1 and the environment is near to zero, which further indicates that the inconsistency between P M 1 and the true dynamics is quite small and thus the optimization process has reached a stopping point.

Revised: Theorem B.2 (Ceiling return gap Under model shifts) .

Revised: For a dynamical model M i ∈ M , V ∗ M i ( µ ) denotes the maximal returns on dynamics P M i .

Revised: Then the gap of optimal returns under these two models M 1 , M 2 can be bounded as:

Revised: [Equation]

Revised: Proof.

Revised: Let G πM i ,M j ( s, a ) be the discrepancy between M i and M j on a single state-action pair ( s, a ) , i.e. G πM i ,M j ( s, a ) =

Revised: E

Revised: [Equation]

Revised: V πM

Revised: [Equation]

Revised: ˜

Revised: [Equation]

Revised: ] − E

Revised: [Equation]

Revised: Mi

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: ,a ) [ V πM j ( s ′ )] .

Original: {+We construct Z k be the discounted return when using π to sample in model M i for k steps and then in M j for the rest with a starting point s 
+}
Revised: {+We construct Z k be the discounted return when using π to sample in model M i for k steps and then in M j for the rest with a starting point s 
+}

Revised: =

Revised: s , that is,

Revised: [Equation]

Revised: Base on this definition, we have that V πM i ( s ) =

Revised: Z ∞ and V πM j ( s ) =

Revised: Z 0 .

Revised: Then, we can decompose

Revised: V πM i ( s ) − V πM j ( s ) into a sum of Z k :

Revised: [Equation]

Revised: We can find that, Z k +1 and Z k only differ in their dynamical model used in the k -th step rollout.

Revised: And we can rewrite them to be :

Revised: [Equation]

Revised: Here

Revised: [Equation]

Revised: r k denotes the reward from the first j steps from the real environment.

Revised: Combine the two equations above together and we get:

Revised: [Equation]

Revised: Then

Revised: [Equation]

Revised: we can obtain the following conclusion by adding up all Z k +1 − Z k :

Revised: [Equation]

Revised: Here, d πM i denotes the distribution of state-action pair induced by policy π under the dynamical model M i .

Revised: For a starting state s 0 = s , and two dynamical models M 2 , M 1 , with the definition of G πM 1 ,M 2 we have:

Revised: [Equation]

Original: ∆ D (cid:2) 1 K (cid:80) Ki =1 ∥ s ′  − ˆ f ϕ
Revised: From the definition: G πM 1 ,M 2 ( s, a ) = E ˜ s ′ ∼ P M 2 ( ·| s,a ) [ V πM 2 (˜ s ′ )] − E s ′ ∼ P M 1 ( ·| s,a )

Revised: [ V πM 2 ( s ′ )] .

Revised: In the case of deterministic dynamics: for clarity, we write s ′ =

Revised: M i ( s, a ) instead of s ′ ∼ P M i ( s ′ | s, a ) , then we rewrite G M 1 ,M 2 ( s, a ) as: G πM 1 ,M 2 ( s, a ) =

Revised: V πM 2 ( M 2 ( s, a )) − V πM 2 ( M 1 ( s, a )) , with the

Revised: Lispchitzness, we have that | G πM 1 ,M 2 ( s, a ) | ≤ L · | M 2 ( s, a ) − M 1 ( s, a ) | .

Revised: In the case of stochastic dynamics: when L ≥ R 1 − γ , we have

Revised: [Equation]

Revised: Note that we require L to be larger than R 1 − γ in the infinite horizon settings.

Revised: Previous work [11] has found that L has a dependence on R · H when equipped with a maximum horizon of H .

Revised: Thus, with the Lipschitzness of V πM 1 and V πM 2 , we have that | G πM 1 ,M 2 ( s, a ) | ≤ L · | P M 2 ( ·| s, a ) −

Revised: P M 1 ( ·| s,

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Observe that | sup x f ( x ) − sup x g ( x ) | ≤ sup x | f ( x ) − g ( x ) | , where f And g are real valued functions.

Revised: This implies:

Revised: [Equation]

Revised: Thus

Revised: [Equation]

Revised: we have that for starting state s 0 ∼ µ :

Revised: [Equation]

Revised: Theorem B

Revised: [Equation]

Revised: 3 (Refined bound with constraints) .

Revised: Here, S denotes the state space simplex.

Revised: Here, policy π i ∈ Π is the ϵ opt optimal policy under the dynamical model M i ∈ M .

Revised: For a dynamical model M i ∈ M , V ∗ M i ( µ ) denotes the maximal returns on dynamics P M i .

Revised: We give the model shift constraints under the TV-distance.

Revised: Then the performance difference bound can be refined under the model shift constraints as:

Revised: [Equation]

Revised: Proof.

Revised: Let µ and v be two probability distributions on the configuration space X , according to

Revised: Lemma C.

Revised: [Equation]

Revised: then we have D TV ( µ ∥ v )

Revised: =

Revised: 12 (cid:80) x ∈X | µ ( x ) − v ( x ) | .

Revised: Recall ϵ πM i denote the discrepancy between the learned dynamics P M i and the true dynamics, i.e. ϵ πM

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: , a ))] .

Revised: Under these definitions

Revised: [Equation]

Revised: we can yield the following intermediate outcome by applying the results from B.2 and B.

Revised: [Equation]

Revised: Recall the following constraint on model shift that we subject to:

Revised: [Equation]

Revised: Then, the ceiling performance difference can be further bounded as:

Revised: [Equation]

Revised: And finally, we have the following refined bound:

Revised: [Equation]

Revised: This refined bound is subject to the model shift constraint we set.

Revised: Corollary B.4.

Revised: Under the generative model setting, let model M 1 has already trained on N samples of each ( s, a ) pair and model M 2 on N + k samples per ( s, a ) pair.

Revised: Policy π 1 is the ϵ opt -optimal policy on M 1 and so is π 2 on M 2 .

Revised: Recall ϵ

Revised: =

Revised: δ M 1 ( ·| s, a ) − (1 − γ )

Revised: L

Revised: [Equation]

Revised: · ϵ opt and ξ ∈ (0 , 1) is a constant.

Revised: As the model is trained on true interaction samples, we can work out the amount of samples we need to satisfy the monotonic improvement requirements:

Revised: [Equation]

Revised: Proof.

Revised: For simplicity, we denote δ M i ( s ′ | s, a ) = | P ( s ′ | s, a ) − P M i ( s ′ | s, a ) | .

Revised: Recall the lower bound for performance difference:

Revised: [Equation]

Revised: Towards this lower bound, we give the assumption that similar models derives similar sub-obptimal policies.

Revised: To be specific, when M 1 and M 2 are close to each other in terms of L 1 -norm distance at any transition pair ( s, a ) : | P M 2 ( ·| s, a ) − P M 1 ( ·| s, a ) | 1 ≤ δ , then the sub-optimal policies derived from them are close as well, i.e. , there exists α , subject to | π 2 ( ·| s ) − π 1 ( ·| s ) | 1 ≤ α · δ for all transition pairs.

Revised: Here, we show the feasibility of given the α in the control theory perspective.

Revised: Let a trajectory s 1: N , a 1: N − 1 be generated from the dynamical model M 1 that satisfies:

Revised: [Equation]

Revised: We call it a nominal trajectory.

Revised: Here, f M 1 is a nonlinear function that represents the dynamics of model M 1 .

Revised: Then, we can formalize the inconsistency between M 2 and M 1 by disturbance, w i ∈ W .

Revised: By entering w i into f M 1 in a general nonlinear way we can get the dynamic of model M 2 as:

Revised: [Equation]

Revised: Further, the deviations from the nominal trajectory can be calculated as when giving a disturbance sequence, w 1: N − 1 .

Revised: Let δy denote the deviations of variable y .

Revised: [Equation]

Revised: Assume that the deviations are computed with a linear feedback controller, that is,

Revised: [Equation]

Revised: Actually, we can utilize any reasonable linear controller.

Revised: Here, we take the time-varying linear quadratic regulator as an instance for illustrating the rationality of our assumption on α .

Revised: Based on the dynamic Riccati equation, we have the solution as:

Revised: [Equation]

Revised: where A t =

Revised: ∂f M 1 /∂s | s t ,a t , 0 and B t =

Revised: ∂f M 1 /∂a | s t ,a t , 0 .

Revised: And, Q i ⪰ 0 and R ⪰ 0 are state and input cost matrices.

Revised: That is, we have a feasible solution for α .

Revised: We then seek to explain the state-action distribution is similar thus we can use d π 1 as an approximation of d π 2 .

Revised: First of all, we have the distance between two policies,

Revised: [Equation]

Revised: Denote P hπ as the state distribution resulting from π at time step h with µ as the initial state distribution.

Revised: We consider bounding | P π 2 h − P π 1 h | 1 with h > 1 .

Revised: [Equation]

Revised: Apply absolute value on both sides, we then get:

Revised: [Equation]

Revised: Under the definition of d πµ

Revised: [Equation]

Revised: we have:

Revised: [Equation]

Revised: Upon these analyses

Revised: [Equation]

Revised: we find that similar model derives similar policy, which invokes similar stateaction distribution.

Revised: Thus we can approximate state-action visitation density d π 2 by previous visitation density d π 1 .

Revised: Also, we assume the same state space S .

Revised: Then, we get a approximation of C M 1 ,π ( M 2 , ˆ π ) as following:

Revised: [Equation]

Revised: Thus, when meeting the following requirements for each ( s, a ) pair, we can guarantee the monotonic improvement for that V π 2 | M 2 − V π 1 | M 1 ≥ ˜ C M 1 ,π 1 ( M 2 , π 2 ) ≥ 0 .

Revised: [Equation]

Revised: By applying Lemma C.3, the L 1 deviation of the empirical distribution P M 2 ( ·| s, a ) and true P ( ·| s, a ) over vol ( S ) distinct events from n samples is bounded by:

Revised: [Equation]

Revised: Then for a fixed ( s, a ) , with probability greater than 1 − ξ , we have:

Revised: [Equation]

Revised: Let ϵ

Revised: =

Revised: δ M 1 ( ·| s, a ) − (1 − γ )

Revised: L R · (2 σ M 1 ,M 2 ) − (1 − γ

Revised: [Equation]

Revised: 2 Rγ · ϵ opt .

Revised: Our requirements can be further shown in this form:

Revised: [Equation]

Revised: Finally, with probability greater than 1 − ξ , we can guarantee the monotonic improvement when having:

Revised: [Equation]

Revised: C Toolbox

Revised: Lemma C.1 (Total variation distance) .

Revised: Let µ and v be two probability distributions on the configuration space X .

Revised: Then

Revised: [Equation]

Revised: Proof.

Revised: Let B = { x ∈ X : µ ( x ) ≥ v ( x ) } , and A ⊆

Revised: X be any event.

Revised: Since µ ( x ) − v ( x ) < 0 for any x ∈ A ∩ B c , we have

Revised: [Equation]

Revised: B )

Revised: For all events A , | µ ( A ) − v ( A ) | µ ( B ) − v ( B ) , and the equality is achieved for A = B or A = B c .

Revised: Thus, we get that

Revised: [Equation]

Revised: Lemma C.2 (Relationship between true returns and model returns) .

Revised: Let ϵ πM denote the inconsistency between the learned dynamics P M and the true dynamics, E s,a ∼ d π [ D TV ( P ( ·| s, a ) ∥ P M ( ·| s, a ))] , where d π .

Revised: =

Revised: d π ( s, a ; µ ) is the probability of visiting state-action pair ( s, a ) after starting at state s 0 ∼ µ and following π thereafter under the true dynamics.

Revised: Then the true returns can be represented as below:

Revised: [Equation]

Revised: Proof.

Revised: Given policy π and dynamics P ( ·| s, a ) , we denote the density of state-action visitation after h steps from starting state s 0 ∼ µ as ρ πh ( µ ; P ) =

Revised: E s 0 ∼ µ [ ρ πh ( s h , a h | s 0 ; P )] .

Revised: Then the discounted returns are bounded as:

Revised: [Equation]

Revised: Using the property of Markov chain TV distance bound

Revised: [Equation]

Revised: then,

Revised: [Equation]

Revised: By plugging the results back

Revised: [Equation]

Revised: we then get,

Revised: [Equation]

Revised: Lemma C.3 (Inequalities for the L 1 deviation of the empirical distribution) .

Revised: Let P be a probability distribution on the set

Revised: A = { 1 , . . .

Revised: , a } .

Revised: For a sequence of samples x 1 , . . .

Revised: , x m ∼ P , let ˆ P be the empirical probability distribution on A defined by ˆ P ( j )

Revised: = 1 m (cid:80) mi =1 1 ( x i = j ) .

Revised: The L 1 -deviation of the true distribution P and the empirical distribution ˆ P over A from m independent identically samples is bounded by,

Revised: [Equation]

Revised: Proof.

Revised: For a probability distribution P on A , we define

Revised: [Equation]

Revised: And for p ∈ [0 , 1 / 2) , we define

Revised: [Equation]

Revised: and, by continuity, set φ (1 / 2) = 2 .

Revised: According to Weissman et al.

Revised: [56], the L 1 -deviation of the true distribution P and the empirical distribution ˆ P is bound by,

Revised: [Equation]

Revised: Firstly, for any P , we have

Revised: [Equation]

Revised: and note that π P = 1 / 2 when P ( A ) = 1 / 2 .

Revised: Then, we claim that the function φ ( p ) is strictly decreasing for p ∈ [0 , 1 / 2] .

Revised: Differentiating φ ( p ) with respect to p yields

Revised: [Equation]

Revised: For p ∈ (0 , 1 / 2) , there always exists 1 (1 − 2 p ) 2 > 0 .

Revised: Thus, to show that φ ′ ( p ) < 0 for p ∈ (0 , 1 / 2) , it suffices to show that

Revised: [Equation]

Revised: The derivative of g ( p ) is

Revised: [Equation]

Revised: Note that g (1 / 2) = 0 , thus we have φ ′ ( p ) < 0 for p ∈ (0 , 1 / 2) .

Revised: And continuity arguments complete the claim for p = 1 / 2 and p = 0 .

Revised: It is then no difficult to see that for any probability distribution P ,

Revised: [Equation]

Revised: Therefore

Revised: [Equation]

Revised: D Comparison with Prior Works

Revised: To begin with, an important fact is that the effect of model shifts on trajectories is drastic.

Revised: For example, even when the system dynamics satisfy L - Lipschitz continuity, along with the policy and the initial state are the same, the difference in trajectories sampled in M 1 , M 2 grows at e LH with the length H of the trajectory [32].

Revised: As the model shifts decay, the trajectory discrepancy will also decrease sharply.

Revised: It implies that model shift stays a substantial influence during the MBRL training process.

Revised: There are two main trends of local view analysis:

Revised: API [22] class.

Revised: Their recipe for monotonicity analysis is V π n +1 ( µ ) − V π n ( µ ) ≥ C ( π n , π n +1 , ϵ m ) .

Revised: If policies update π n → π n +1 could provide a non-negative C ( π n , π n +1 , ϵ m ) , then the performance is guaranteed to increase.

Revised: Here, ϵ m = max π ∈ Π ,M ∈M E s,a ∼ d π [ D TV ( P ( ·| s, a ) ∥ P M ( ·| s, a ))] .

Revised: Most previous works [44, 22] were derived under model-free settings ( ϵ m = 0 ).

Revised: they use conservative policy iteration, for example, by forcing D TV ( π n ∥ π n +1 ) ≤ α ), then the state-action distribution are close as well D TV ( d π n ∥ d π n +1 ) ≤ αγ 1 − γ , so that they can optimize over their performance difference lemma, e.g., C ( π n , π n +1 , 0)

Revised: ≈ 11 − γ E s,a ∼ d πn [ A π n ( s, a )] .

Revised: When ϵ m > 0 , this approximation

Revised: C ( π n , π n +1 , 0)

Revised: ≈ 11 − γ E s,a ∼ d πn [ A π n ( s, a )] fails.

Revised: It is non-trivial to apply the results to model-based settings.

Revised: DPI [49] tries to force π n +1 and π n to be close, which will result in a high similarity of the data they sample.

Revised: Thus, a risk arises from it, this approach would limit the growth of the policy exploration in the real environment, thereby leading the inferred models to stay optimized in a restrictive local area.

Revised: For example, in the Humanoid environment, the agent struggles to achieve balance at the beginning of training.

Revised: By then, an updated restricted policy will cause the exploration space to be limited in such an unbalanced distribution for a long time, and the learned model in such highly repetitive data will converge quickly with a validation loss be zero.

Revised: However, the success trajectory has not been explored yet, implying that both the policy and the learned model will fall into a poor local optimum.

Revised: Besides, the definition of model accuracy (Eq.3) is a local view in DPI, i.e., ˆ P is δ -opt under d π n .

Revised: If we replace model accuracy with a more general, global definition (for example, ˆ P is δ -opt under d π n +1 , or ˆ P is δ -opt under all ( s, a, s ′ ) tuples), we find that the δ in (Eq. 3) will be large at the initial steps, making it difficult to obtain a local optimal solution in Theorem 3.1.

Revised: Finally, theoretical analysis in DPI can only guide the policy iteration process, while the update of the model is passive, which is different from our global view theory.

Revised: Discrepancy bound class [34, 20].

Revised: They mostly derive upon V π n ( µ ) ≥ V π n M ( µ ) − C ( ϵ m , ϵ π ) .

Revised: As guaranteed in them, once a policy update π n → π n +1 has improved returns under the same model

Revised: M , i.e., V π n +1 M ( µ ) >

Revised: V π n M ( µ ) + C ( ϵ m , ϵ π ) , it would improve the lower bound on the performance evaluated in the real environment, i.e., inf { V π 2 | M ( µ ) } > inf { V π 1 | M ( µ ) } }.

Revised: Their theory is based on a fixed model M , or an upper bound on the distribution shift of all models ϵ m .

Revised: It does not concern the change in model dynamics during updating, nor the performance varying due to the model shift.

Revised: Moreover, The solution would be very coarse if only the upper bound of the model shift is given.

Revised: Even worse, the given upper bound is likely to be too large, then it will fail to find a feasible solution for V π n +1 M

Revised: [Equation]

Revised: ) in practice, thus making the monotonicity guarantee fails.

Revised: E Experimental Details E.1 Environment Setup

Revised: We evaluate all algorithms on a set of MuJoCo [54] continuous control benchmark tasks.

Revised: We adopt the standard full-length version of all these tasks.

Revised: Among then, we truncate some redundant observations for Hopper, Ant and Humanoid as our model-based baselines (MBPO[20], AutoMBPO[28]) do.

Revised: The details of the experimental environments are provided in Table 1.

Revised: [Table]

Revised: The environment settings for the ablation study on the generalizability of event-triggered mechanism are presented in Table

Revised: [Table]

Revised: E.2 Baselines and implementation

Revised: MFRL algorithms.

Revised: We compare to two state-of-the-art model-free baselines, SAC [16] and

Revised: PPO [46].

Revised: The hyperparameters are kept the same as the authors’.

Revised: Regarding the low sample efficiency of MFRL methods, we ran 5M steps for them, which is an order of magnitude more than in

Revised: MBRL, to fairly evaluate the asymptotic performance of these MFRL algorithms.

Revised: The implementation of SAC is based on the opensource repo (pranz24 [42], MIT License).

Revised: MBRL algorithms.

Revised: As for model-based methods, we compare with several algorithms including

Revised: PETS [8], SLBO [34], MBPO [20] and AutoMBPO [28].

Revised: Our algorithm CMLO is implemented based on the opensource toolbox for MBRL algorithms, MBRL-LIB [40] (MIT License).

Revised: The implementation of SLBO mainly follows Wang et al.

Revised: To ensure a fair comparison, we run

Revised: CMLO and MBPO with the same network architectures and training configurations based on MBRLLIB.

Revised: We report the asymptotic performance on six benchmark tasks in Table 3.

Revised: Results show that our method has comparable asymptotic performance in each benchmarks to both MBRL and MFRL baselines.

Revised: Each result is averaged over seven trials using different random seeds.

Revised: For MBRL baselines, the performances on different tasks are capped at different timesteps when the learning curves come to converge, we choose 125k for Hopper, 350k for Walker2d and Swimmer, 400k for HalfCheetah, 300k for Ant and 250k for Humanoid.

Revised: [Table]

Revised: E.3 Implementation details of CMLO

Revised: Modeling and learning the dynamical models.

Revised: As inferred from the optimization objective, the minimization of the objective function can be achieved when we try to minimize the difference between M 2 and the real environment.

Revised: To reduce model bias

Revised: [Table]

Revised: we chose to use NLL as a loss function in our implementation, which has been shown an effective way to learn model dynamics.

Revised: More specifically

Revised: [Table]

Revised: CMLO adopts a bootstrap ensemble of dynamical models { ˆ f ϕ 1 , ˆ f ϕ 2 , . .

Revised: , ˆ f ϕ K } .

Revised: Specifically, each forward dynamical model f ϕ i approximates the transition function of the real environment, that is ˆ s t +1 ∼ f ϕ

Original: i
Revised: i

Revised: ( s t , a t ) .

Revised: The probabilistic models are fitted on shared but differently shuffled replay buffer D e , and the target is to optimize the Negative Log Likelihood (NLL).

Original: ( s, a ) ∥ (cid:3) .{+[Equation] And the prediction for these ensemble models is, ˆ s t +
 = 1 K (cid:80) Ki =
+}
Revised: {+[Equation] And the prediction for these ensemble models is, ˆ s t +
 = 1 K (cid:80) Ki =
+}

Original: Combining these two components, we canobtain an estimation for  the  model shift, i.e. , vol ( S D ) ·
Revised: f ϕ i ( s t , a t ) . More details on network settings are presented in Table 4. Model shifts estimation. Recall that we partition the incalculable model shifts into two components for estimation, one for state-space coverage and the other for model divergence.

Revised: • state-space coverage .

Revised: State coverage (policy coverage) is the range of state spaces that our algorithm can explore in the real environment under the current policy π i (derived from the learned model M i ).

Revised: In the existing works, [1] defined the return set for two state sub-space as ¯ R ret =

Revised: lim n →∞ R nret ( X, ¯ X ) , where R nret ( X, ¯ X ) means an n-step returnability from X to ¯ X .

Revised: Referring to this definition, the state coverage of π i can be defined as S π i pc : ∀ s ∈ S π i pc , a ∼ π i ( ·| s ) , s ′ ∼ P ( ·| s, a ) ∈ S π i pc .

Revised: Besides, in the description of La Salle’s Invariance Principle [2], we verify the equivalence of Invariant Set and state coverage.

Revised: Intuitively, the Humanoid example in our response to your major concerns also shows that the variation of state coverage in the different training stages.

Revised: We estimate the policy coverage (state-space coverage) by computing the volume vol ( S D ) of the convex closure S D constructed on the replay buffer D .

Revised: Since estimation on the full historical experiences involves a huge computational burden, we instead sample N tuples ( e.g. 1000 tuples) from the replay buffer upon each estimation.

Revised: As for the convex hull, we first perform Principal Component Analysis on the states to reduce the dimension and then leverage the Graham-Scan algorithm to construct a convex hull of these N points, which only takes O ( N log N ) for time complexity.

Revised: • model divergence .

Revised: We estimate the model divergence by computing the average prediction error on newly encountered data.

Revised: Upon it, we get the estimation for the model divergence from the K ensemble models, L (

Original: L (∆ D ) .
Revised: [Equation] D ) = [Equation] ( s

Original: This practical overestimation for themodel shifts makes the constraint stay satisfied during objective optimization.
Revised: [Equation] [Equation] ) ∈ ∆ D (cid:2) 1 K (cid:80) Ki =

Original:  Event-triggered mechanism.
Revised: [Equation] Event-triggered mechanism.

Original:  We design an event-triggered mechanism to determine the occasionto pause collecting and turn to solve the optimization objective.
Revised: Recall our proposed optimization problem: [Equation] We design an event-triggered mechanism to determine the interval instant τ on the condition that the optimization problem is solved at step t + τ .

Original: Remark that although diverting totrain models as long as not to violate the constraint is theoretically reasonable, we refrain from doingso in practice because performing an update on data with a minor shift in coverage and distribution iswasteful andmay risk overfitting. Therefore, we trigger when the constraint boundary is touched toreduce computational cost and escape overfitting.
Revised: The mechanism is developed based on the difference between the current model M 1 (trained at step t ) and the upcoming model M 2 , which is estimated on the newly encountered data. If their model shifts reaches a certain value, it stands to reason that a new dynamic model is required to be trained.

Original: The event-triggering mechanism is based on thefollowing condition:
Revised: Thus, the event-triggered mechanism is based on the condition:

Original: [Equation]
Revised: [Equation]

Original: Here, τ is the event-triggering time, and α is a given constant.
Revised: Here, we adopt the fraction form for the triggered condition. Denominator vol ( S D∪ ∆ D ( τ ) ) ·

Revised: L (∆ D ( τ )) is used to obtain an estimation for the model shift, as detailed in Line 239-248.

Revised: The numerator vol ( S D ) , on the one hand, is to reduce numerical errors; on the other hand, this fraction reflects the relative change of the policy coverage and model shift if we turn to train M 2 under different τ starting from M 1 .

Revised: This fraction reflects the current ability to digest new data.

Revised: It can facilitate the setting of threshold, for we do not need to tune α once the policy coverage updates.

Revised: In CMLO, the condition estimation execute per F steps because excessively frequently estimation doesn’t make huge difference but bring up the computation load.

Revised: In order to reuse the result of intermediate computation, we apply a log value to the result of each estimation so that the log value condition function can be approximated by the sum of each estimation value within the interval.

Revised: We additionally append a constant β = 1 . 0 for the penalty of accumulated interval steps.

Revised: [Equation]

Original: Policy optimization oracle.
Revised: [Figure] Remark 1: We observe from the event-triggered condition that the interevent time can be enlarged by increasing the threshold α , which implies more exploration samples will be collected by current policy and the optimization objective will be solved less frequently. In other words, the triggered threshold is environment-specific. Notably, the tuning load required for our event-triggered mechanism is not heavier than in those fixed settings, due to that only a hyperparameter α is introduced for model training frequency in CMLO, while those algorithms with fixed settings need to tune the fixed model training interval.

Original: Clearly, we can leverage many model-free RL methods (SAC [15],
Revised: We claim that it is crucial to dynamically adapt the numbers of explorations to update the model according to the current training and exploration status. Remark 2: Zeno behavior [17] is common in the event-triggered mechanism, which leads to a most frequently triggering.

Original: TRPO [39], PPO [41] etc. )
Revised: The zeno behavior is naturallt alleviated by the introdution of β , which acts as a penalty for interevnet time.

Original: as our policy optimization oracle. Besides, we adopt a truncated shortmodel rollouts technique to mitigate compounding error while encouraging model usage.
Revised: We additionaly add the lower and upper bounds of the interevent time to further keep the interval in a safe zone to aviod zeno behavior caused by some extreme situations. The minimal and maximal interevent time are given by inf { τ } = T and sup { τ } = T . Policy optimization and model rollouts.

Original: Based onthe fresh model rollouts, we perform the policy optimization oracle, employing SAC as an example. Algorithm Overview. We briefly give an overview of our proposed CMLO in algorithm 1.
Revised: We can adopt a standard off-policy model-free RL method SAC [16] as the policy optimization oracle of CMLO.

Original: Notably,the event-triggered mechanism subtly determines the occasion to perform model updating, promotingperformance monotonicity and reducing computation load. Algorithm 1: initialize : policy π θ , ensemble models ˆ f ϕ 1 , ˆ f ϕ 2 , . . .
Revised: Another key concern is the way of training data generation. We adopt the truncated short model rollouts strategy inspired by some current MBRL works [20, 39, 28], which helps to escape from compounding error and encourage model usage. The main difference from the general rollouts mechanism is that we restrict our rollouts to be generated from fresh models, rather than using outdated models to generate rollout data as MBPO [20] and

Original: , ˆ f ϕ K , environment buffer D e and model buffer D m repeat Sample ∆ D e ∼ π θ from real environment; add to D e Estimate model shifts by vol ( S D e ) L (∆ D e ) if Event-triggered condition (5.1) is reached then
Revised: AutoMBPO [28] do in their implementations.

Original: Train all models ˆ f ϕ 1 , ˆ f ϕ 2 , . . , ˆ f ϕ K on  D e Perform h -step model rollouts using policy  πθ ; add to D m
Revised: Based on the dataset D m of the fresh model rollouts, we perform SAC. In the policy evaluation step, SAC repeatedly apply a Bellman backup operator T π to the soft Q-value, T π Q π ( s, a ) ≜ r ( s, a ) + γ E s ′

Original: Update π θ on D m through SAC [15] until the policy performs well in the environment ;
Revised: [ V π ( s ′ )] , and in the policy improvement step, SAC updates the policy according to π = arg min π ∈

Original: Experiments
Revised: [Equation]

Original: Our experimental evaluation aims to investigate the following questions: (1) How well does ouralgorithm perform on standard reinforcement learning benchmarks compared to prior state-of-the-artmodel-based and model-free algorithms? (2) Does the performance with or without the constraintconsistent with previous theoretical analyses?
Revised: Q

Original: [Figure]
Revised: [Equation]

Original: To illustrate the effectiveness of our method, we contrast several popular model-based and model-freebaselines.
Revised: π

Original: Model-free counterparts include: (1) SAC [15], the state-of-the-art in terms of asymptotic performance.
Revised: [Equation] Remark 3: The data distribution introduced by the outdated model has a drift from the data distribution introduced by the fresh model.

Original: (2) PPO [41] that explores monotonic improvement as well. Model-based baselinesinclude: (3) PETS [8], which employs models directly for planning, different from the Dyna-style.(4) SLBO [31], that explores monotonicity under the discrepancy bound scheme.
Revised: This data shift will somehow mislead policy training and, in addition, the policy trained on the outdated model suffers from the limited sampling coverage during interacting with the real environment, which might in return cause the following models to fall into a local trap. In other words, the less the model differs from the real dynamics, the data it rolls out is more valuable.

Original: (5) MBPO
Revised: E.4 Hyperparameters

Original: [18],that employs a similar design of model ensemble technique (ensemble of probabilistic dynamicsnetworks) and policy optimization oracle (SAC) as we do. (6) AutoMBPO [26], a variant of MBPO,that uses an automatic hyperparameter controller to tune the model-training frequency but suffersfrom high pre-training cost and lacks theoretical analysis on parameters rationality. We evaluate CMLO and these baselines on six continuous control tasks in OpenAI Gym [6] withthe MuJoCo [49] physics simulator, including HalfCheetah, Hopper, Walker2d, Swimmer, Ant, Humanoid.
Revised: Hyperparameters for Main Experiments.

Original: For fair comparison, we adopt the standard full-length version of these tasks and alignthe same environment settings. Figure 1 shows the learning curves of all compared methods, along with the asymptotic performance.
Revised: Table 4 lists the hyperparameters used in training

Original: These results show that our algorithm is far ahead of the model-free method in terms of sampleefficiency, coupled with an asymptotic performance comparable to that of the state-of-the-art modelfree counterparts SAC. Compared to model-based baselines, our method gains faster convergencespeed and better eventual performance.
Revised: CMLO.

Original: Notably, credit to the event-triggered mechanism, our methodenjoys a more stable training curve. The better monotonic property of the learning curve agrees withour previous analyses.
Revised: Here, x → y over epochs a → b denotes a threshold linear function, i.e. , at epoch t , f ( t ) = min(max( x + t − a b − a · ( y − x ) , x ) , y ) .

Original: 6.2 Ablation studies
Revised: Hyperparameters for Ablation Studies.

Original: Next, we make ablations and modifications to our method to validate the effectiveness and generalizability of the mechanism we devised. [Figure]
Revised: Note that other hyperparameters we do not mention below are the same as hyperparameter settings in Table 4. (1) Policy optimization oracle: TRPO. For the TRPO part, the key parameters are listed below:

Original: These experiments are average over 5 random seeds. (a) shows the average return with or without event-triggered mechanism in HalfCheetah and Ant benchmarks. (b) shows the number of triggered times per 10k step.
Revised: • Ant: horizon = 1000 , γ = 0 .

Original: [Figure]
Revised: 99 , gae = 0 . 97 , step _ size = 0 . 01 , iterations = • HalfCheetah: horizon = 1000 , γ = 0 .

Revised: 99 , gae = 0 .

Revised: 95 , step _ size = 0 .

Revised: 01 , iterations =

Revised: About Legend w/o-n, we use a data sampler with batchsize=20, thus we get 20 × n real interactions during the model training interval.

Revised: We compute the total triggered times and scale them to [0,1], which is shown in the bar plots.

Revised: [Table]

Revised: [Table]

Revised: ) Policy optimization oracle: iLQR.

Revised: Dynamical models network: Gaussian MLP with 3 hidden layers of size 200, batch size is 64, and the learning rate is 0.0001.

Revised: For the iLQR part: LQR _ ITER = 10 , R = 0 .

Revised: 001 , Q = 1 , horizon = 5 .

Revised: About Legend w/o-n, we get n real interactions during the model training interval.

Revised: And α = 0 .

Revised: 5 in w/-ours.

Revised: We compute the total triggered times and scale them to [0 , 1] , as shown in the bar plots.

Revised: E.5 Additional Ablation Study

Revised: Estimation on model shifts.

Revised: The constraint function based on model shifts is incalculable due to the unobserved model M 2 .

Revised: We design a practical predictor for the model shifts by computing the state-space coverage and the model prediction error.

Revised: Also, the decoupling of the constraint and the objective is enabled partly owing to the slightly overestimation over the true value.

Original: [Figure]
Revised: Recall our constraint function: [Equation] The updated dynamics P M 2 usually comes closer to the true dynamics than the previous one P M 1 , thus we turn to estimate (cid:80) s ′ ∈S [ | P M 1 ( s ′ | s, a ) − P M ( s ′ | s, a ) | ] .

Original:  we apply log The necessity of event-triggered mechanism. To verify the necessity of event-triggered mechanism, we compare to three unconstrained cases (given fixed model training interval) under twoenvironments.
Revised: Once the model M 2 is trained, we can actually conduct a more realistic calculation for the constraint function. To show the connection between our predicted value in the absence of M 2 and the estimated value after obtaining M 2 , we perform experiments on four environments and results are shown in Figure 8.

Original: The training curve and triggered times are shown in Figure 2. Clearly, our mechanismimproves the performance while reducing the total times of model training.
Revised: The results demonstrate that our prediction is higher than the true estimation and their trends stand consistent, which indicates that our predictor is well designed. This gap helps to decouple the constrained optimization problem, and this overestimation part can be bridged by adjusting the α . [Figure]

Original: Besides, we notice thatthe performance is comparable to other MBRL baselines (MBPO etc. ) when fixing our model traininginterval at 250.
Revised: Model accuracy. Figure 9 shows the one-step model error during the training under four benchmark tasks. We find that CMLO achieves a more accurate model than the state-of-the-art baseline MBPO.

Original: Still it performs worse than that equipped with  a smart mechanism to decide whetherto train the model at  current exploration step.
Revised: This result agrees with our insight that, a smarter scheme to choose different numbers of explorations at different steps instead of the unchanged setting in current methods, will promote a better model [Figure]

Original: To better understand why the event-triggered mechanism brings up our outperformance, we asses itsmain bricks. Model shift, which reflects the current ability to digest new data, is the basis of triggeredcondition.
Revised: 35m ode l e rr o r

Original: And we estimate model shift from two parts, policy coverage and prediction error.
Revised: HalfCheetah CMLOMBPO 50k 100k 150k 200k steps     m ode l e rr o r

Original: In
Revised: Ant

Original: Figure 3 we observe that, gradually, as the training progresses, the policy coverage increases, whichreflects our policy has new explorations at every stage without falling into a local optimum prematurely.

Original: Also, the prediction error gradually decreases, which implies that our estimated dynamics come closerto the true dynamics in the explored region.

Original: Figure 4 implies that the number of samples requiredto hit the constraint tends to grows and the model training frequency then goes down. This is alsoconsistent with our intuition that, as the model refines and the exploration novelty fades, then thesample size required to reach a certain level of model shifts grows up. The result agrees with ourtheoretical analyses which describe that a dynamic alternation subject to the model shifts constraintdoes benefit to monotonicity rather than an assigned one. [Figure]
Revised: CMLOMBPO 50k 100k 150k 200k steps   m ode l e rr o r

Original: The generalizability of event-triggered mechanism.
Revised: Humanoid CMLOMBPO Policy Coverage Comparison. Policy coverage represents the exploration ability of the policy.

Revised: The policy coverage increasing with the stages means that the policy has new explorations at every stage and may not fall into a local optimum.

Revised: Here, we present the numerical comparison to MBPO in

Revised: Table 5.

Original: We further investigate the generalizabilityof our proposed mechanism through ablation onpolicy optimization oracle, and results are shownin Figure 5.
Revised: [Table] Here, each stage i contains (60 × ( i − 1) , 60 × i ] k steps. In HalfCheetah, we find that our policy achieves higher coverage especially in first 4 stages than MBPO. Consistently, we find that our policy enjoys higher performance, with an average return lead of about 1855.29 over MBPO in the first 300k steps.

Original: 1) under Dyna-style.
Revised: Likewise, the growth of policy coverage in Ant is also consistent with the rise in average [Figure] return.

Original: We adopt TRPO [39] asthe policy optimization oracle andtest the performance with or without event-triggered mechanism 
Revised: The increase in policy coverage helps the policy to refrain from falling into a local optimum, thus improving performance. Effectiveness of event-triggered mechanism. We compare applying model shift constraints to the unconstrained cases and the results are shown on Figure 10. To verify the effectiveness of adding suitable constraints, we invalidate the event-triggered mechanism and keep the other part unchanged in our method.

Original: in Halfcheetah and Ant benchmarks.
Revised: As observed, our model is accurate enough when fixing the model training interval at 250 , but it still performs worse than applying the model shift constraints. We attribute our model’s out-performance to our rational model shift design. It improves the performance while minimizing the training cost of the model. Adding such a model shift can protect the model from overfitting on under-explored data, and can also save the model from fitting large data shifts.

Original: Observably, our mechanism can effectively guarantee the over- all performance improvement and alleviate thelocal optimization issue. 2) jumping off Dyna-style.
Revised: To determine whether the event-triggered mechanism has an effect on the training process, we conduct a t-test to compare the average returns of CMLO with or without the mechanism.

Original: We utilize iLQR [3]and then conduct ablation experiments in theDKitty-Stand [2] and Panda-Reaching [17] scenar-

Original: ios.

Original: The result shows that adding event-triggered

Original: mechanism has a comparable asymptotic perfor-

Original: mance as the most-frequently triggered case but

Original: enjoys a lower computation cost  Conclusion We have investigated the role of the decision on "when to update the model" in joint optimizationprocedures through theoretical and empirical lens. We devise a general novel scheme for exploring themonotonicity of MBRL methods, distinguished from the existing discrepancy bound scheme.
Revised: We compare the original CMLO to its variant with a fixed setting (w/o-250) and list the p-values in

Original: We thenderive lower bounds under the scheme, suggesting that models with higher ceiling performance andlower bias guarantee a non-decreasing performance evaluated in the real environment.

Original: An effectiveconstrained optimization problem comes from the follow-up refined bound to seek a non-negativelower bound. Further, the instance under the generative model setting further verifies the effectivenessof learning models from a dynamically varying number of explorations. The algorithm CMLO,stemming from these analyses, has asymptotic performance rivaling the best model-free algorithmsand boasts better monotonicity. Further ablation studies reveal that the proposed mechanism scales tovarious policy optimization oracles and benefits computation cost reduction.
Revised: Table 6.

Original: Currently, we observedempirically that the event-triggered condition is usually related to specific environments, which willcost a little time for tuning. Thus, one direction thatmerits further investigation is to construct the dualproblem of our constrained optimization problem for better exploring optimality and monotonicity. Checklist For all authors...
Revised: Our p-values are much smaller than 0.05, so we say with a high degree of confidence that the smartly choosing dynamically varying number of explorations does make a difference in the overall performance.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes]
Revised: [Table]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 7. (c) Did you discuss any potential negative societal impacts of your work? [No] No potential negative societal impacts have been found yet.
Revised: Besides, we provide visualization of event-triggered mechanism on HalfCheetah and Ant in Figure 11.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them? [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: The y-axis is our estimation of the triggered condition.

Original: [Yes] See Section

Original: and Appendix B.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B and

Original: Appendix C.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: In the

Original: supplemental material. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: This figure shows the constraint estimation and whether it reaches the triggered threshold (when the peak is above the threshold (dashed line), the primary trigger condition is satisfied) within different stages.

Original: [Yes] See Appendix E and our provided code.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] See in corresponding figure captions.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix E. (b) Did you mention the license of the assets? [Yes] See Appendix E.
Revised: Note that in the paper we have shown 4k steps for each stage, and here we present for the whole 60k.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating? [N/A] [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: E.6 Computing Infrastructure

Original: [Table]

Original: information or offensive content? [N/A] Our work does not use existing data. [Table]
Revised: Table 7 lists our computing infrastructure and the corresponding computational time used for training

Original: If you used crowdsourcing or conducted research with human subjects... [Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] No crowdsourcing/human subjects are used.
Revised: CMLO on the six benchmark tasks.

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Figure]

Original: Board (IRB) approvals, if applicable?

Original: [N/A] No crowdsourcing/human subjects are

Original: [Table]

Original: used.

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?

Original: [N/A] No crowdsourcing/human subjects are used.

Original: [Table]