Original: When to Update Your Model: Constrained Model-based Reinforcement Learning
Revised: When to Update Your Model: Constrained Model-based Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Designing and analyzing model-based RL (MBRL) algorithms with guaranteedmonotonic improvement has been challenging, mainly due to the interdependencebetween policy optimization and model learning.
Revised: Designing and analyzing model-based RL (MBRL) algorithms with guaranteedmonotonic improvement has been challenging, mainly due to the interdependencebetween policy optimization and model learning.

Original: Existing discrepancy boundsgenerally ignore the impacts of model shifts, and their corresponding algorithmsare prone to degrade performance by drastic model updating.
Revised: Existing discrepancy boundsgenerally ignore the impacts of model shifts, and their corresponding algorithmsare prone to degrade performance by drastic model updating.

Original: In this work, we firstpropose a novel and general theoretical scheme for a non-decreasing performanceguarantee of MBRL.
Revised: In this work, we firstpropose a novel and general theoretical scheme for a non-decreasing performanceguarantee of MBRL.

Original: Our follow-up derived bounds reveal the relationship betweenmodel shifts and performance improvement.
Revised: Our follow-up derived bounds reveal the relationship betweenmodel shifts and performance improvement.

Original: These discoveries encourage us toformulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL.
Revised: These discoveries encourage us toformulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL.

Original: A further example demonstrates that learning models from adynamically-varying number of explorations benefit the eventual returns.
Revised: A further example demonstrates that learning models from adynamically-varying number of explorations benefit the eventual returns.

Original: Motivatedby these analyses, we design a simple but effective algorithm CMLO (Constrained
Revised: Motivatedby these analyses, we design a simple but effective algorithm CMLO (Constrained

Original: Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model.
Revised: Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model.

Original: Experiments show that
Revised: Experiments show that

Original: CMLO surpasses other state-of-the-art methods and produces a boost when variouspolicy optimization methods are employed.
Revised: CMLO surpasses other state-of-the-art methods and produces a boost when variouspolicy optimization methods are employed.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) has driven impressive advances in many complex decision-makingproblems in recent years [33, 43].
Revised: Reinforcement learning (RL) has driven impressive advances in many complex decision-makingproblems in recent years [33, 43].

Original: Many of these advances are obtained by model-free (MFRL)methods, whose desirable asymptotic performance yet comes at the cost of sample efficiency.
Revised: Many of these advances are obtained by model-free (MFRL)methods, whose desirable asymptotic performance yet comes at the cost of sample efficiency.

Original: Hencetheir applications are mostly limited to simulation scenarios [40, 30, 32].
Revised: Hencetheir applications are mostly limited to simulation scenarios [40, 30, 32].

Original: In contrast, Model-Based RL(MBRL) methods, which learn a transition model directly from orders-of-magnitude fewer samplesand then derive the optimal policy from the learned model, have become an appealing alternative insmall-data and more practical cases [34, 9, 37, 16].
Revised: In contrast, Model-Based RL(MBRL) methods, which learn a transition model directly from orders-of-magnitude fewer samplesand then derive the optimal policy from the learned model, have become an appealing alternative insmall-data and more practical cases [34, 9, 37, 16].

Original: In general, MBRL methods alternate between the two stages: model learning and policy optimization( e.g. the general Dyna-style [45, 46]).
Revised: In general, MBRL methods alternate between the two stages: model learning and policy optimization( e.g. the general Dyna-style [45, 46]).

Original: A more accurate model will lead to a better policy.
Revised: A more accurate model will lead to a better policy.

Original: Variousattempts have been proposed to improve model accuracy by investigating high-capacity models (themodel ensemble technique [25, 8] and better function approximators [14, 35]) or amending the policyoptimization stage based on model bias [22, 36, 26, 18, 7, 51].
Revised: Variousattempts have been proposed to improve model accuracy by investigating high-capacity models (themodel ensemble technique [25, 8] and better function approximators [14, 35]) or amending the policyoptimization stage based on model bias [22, 36, 26, 18, 7, 51].

Original: Even so, their resultant models arejust accurate in a local and relative sense, since the learning is conditional on a fixed number of stateaction tuples explored by the policy at the current step, other than the full transition dynamics of theenvironment.
Revised: Even so, their resultant models arejust accurate in a local and relative sense, since the learning is conditional on a fixed number of stateaction tuples explored by the policy at the current step, other than the full transition dynamics of theenvironment.

Original: Indeed, it is tricky to determine how much we should explore.
Revised: Indeed, it is tricky to determine how much we should explore.

Original: Insufficient explorationwould trap the model and the following policy optimization, whereas excessive newly-encounteredstate-action pairs would confuse the model and later cause policy chattering.
Revised: Insufficient explorationwould trap the model and the following policy optimization, whereas excessive newly-encounteredstate-action pairs would confuse the model and later cause policy chattering.

Original: To derive a “truly”accurate model, we need a smarter scheme to choose different numbers of explorations at differenttimes, instead of the unchanged setting in current methods.
Revised: To derive a “truly”accurate model, we need a smarter scheme to choose different numbers of explorations at differenttimes, instead of the unchanged setting in current methods.

Original: From an optimization point of view, the thinking of how to derive an accurate model for MBRL ina global sense also motivates us to investigate the monotonicity of the optimization target ( i.e. thereturn of the learned model and policy in MBRL), which, unfortunately, is less explored and notwell guaranteed in current works.
Revised: From an optimization point of view, the thinking of how to derive an accurate model for MBRL ina global sense also motivates us to investigate the monotonicity of the optimization target ( i.e. thereturn of the learned model and policy in MBRL), which, unfortunately, is less explored and notwell guaranteed in current works.

Original: However, discussing the monotonicity guarantee for MBRL ischallenging by any means, arising from the coupling of the model learning and policy optimizationprocesses.
Revised: However, discussing the monotonicity guarantee for MBRL ischallenging by any means, arising from the coupling of the model learning and policy optimizationprocesses.

Original: Although there has been recent interest in related subjects, most of the theoretical worksseek to characterize the monotonicity in terms of a fixed model of interest [44, 31, 18, 11, 26], whichdoes not naturally fit our case when the model is dynamically shifted.
Revised: Although there has been recent interest in related subjects, most of the theoretical worksseek to characterize the monotonicity in terms of a fixed model of interest [44, 31, 18, 11, 26], whichdoes not naturally fit our case when the model is dynamically shifted.

Original: In this paper, we study how to guarantee the optimization monotonicity theoretically, upon which wethen develop an event-triggered strategy that learns the model from a dynamically-varying number ofexplorations.
Revised: In this paper, we study how to guarantee the optimization monotonicity theoretically, upon which wethen develop an event-triggered strategy that learns the model from a dynamically-varying number ofexplorations.

Original: In particular, we interestingly find that the lower bound of the performance improvementbetween two adjacent alternation steps in MBRL is dependent on the one-step model accuracy plusthe constraint of the model shifts under certain mild assumptions.
Revised: In particular, we interestingly find that the lower bound of the performance improvementbetween two adjacent alternation steps in MBRL is dependent on the one-step model accuracy plusthe constraint of the model shifts under certain mild assumptions.

Original: This discovery encourages us toformulate a constrained optimization problem, in order to permit positive performance improvementand thus the optimization monotonicity for MBRL.
Revised: This discovery encourages us toformulate a constrained optimization problem, in order to permit positive performance improvementand thus the optimization monotonicity for MBRL.

Original: We also give a feasible solution example to showthat dynamical alternation between model learning and policy exploration does benefit performancemonotonicity.
Revised: We also give a feasible solution example to showthat dynamical alternation between model learning and policy exploration does benefit performancemonotonicity.

Original: To resolve the constrained optimization problem, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound) equipped with an event-triggered mechanism.
Revised: To resolve the constrained optimization problem, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound) equipped with an event-triggered mechanism.

Original: This mechanism first estimates whether the model shifts meet the constraint and then decides whento train the model.
Revised: This mechanism first estimates whether the model shifts meet the constraint and then decides whento train the model.

Original: We evaluate CMLO on several continuous control benchmark tasks.
Revised: We evaluate CMLO on several continuous control benchmark tasks.

Original: The results show that CMLOlearns much faster than other state-of-the-art MBRL methods and yields promising asymptoticperformance compared with the model-free counterparts.
Revised: The results show that CMLOlearns much faster than other state-of-the-art MBRL methods and yields promising asymptoticperformance compared with the model-free counterparts.

Original: Note that our optimization framework isgeneral and can be applied to different backbones of policy optimization algorithms, which is alsoablated in our experiments.
Revised: Note that our optimization framework isgeneral and can be applied to different backbones of policy optimization algorithms, which is alsoablated in our experiments.

Original:  Related works
Revised:  Related works

Original: Model-based reinforcement learning methods have shown great potential for sequential decisionmaking both in simulation and in the real world due to their sample efficiency [9, 19].
Revised: Model-based reinforcement learning methods have shown great potential for sequential decisionmaking both in simulation and in the real world due to their sample efficiency [9, 19].

Original: Generally, these
Revised: Generally, these

Original: MBRL algorithms can be grouped into several categories to highlight the range of uses of predictivemodels [50].
Revised: MBRL algorithms can be grouped into several categories to highlight the range of uses of predictivemodels [50].

Original: And our work falls into the Dyna-style category.
Revised: And our work falls into the Dyna-style category.

Original: In Dyna-style algorithms, trainingalternates between model learning under policy iterations with the real environments, and policyoptimization using the model rollouts [45–47, 12].
Revised: In Dyna-style algorithms, trainingalternates between model learning under policy iterations with the real environments, and policyoptimization using the model rollouts [45–47, 12].

Original: Many attempts have been devoted to improvingthese two stages.
Revised: Many attempts have been devoted to improvingthese two stages.

Original: For model learning stage, previous main concerns are function approximators and training objectives.
Revised: For model learning stage, previous main concerns are function approximators and training objectives.

Original: The dynamics approximator has advanced from Gaussian processes [24, 9], time-varying lineardynamics [27, 28] to neural network predictive models [14, 35].
Revised: The dynamics approximator has advanced from Gaussian processes [24, 9], time-varying lineardynamics [27, 28] to neural network predictive models [14, 35].

Original: And training objectives vary from
Revised: And training objectives vary from

Original: Mean Square Error (MSE) [35, 31], Negative Log Likelihood (NLL) [8, 18], etc .
Revised: Mean Square Error (MSE) [35, 31], Negative Log Likelihood (NLL) [8, 18], etc .

Original: Moreover, the deepensemble technique is appealing for improving the robustness to model error.
Revised: Moreover, the deepensemble technique is appealing for improving the robustness to model error.

Original: Our method adopts anensemble of probabilistic networks similarly as in [8, 18].
Revised: Our method adopts anensemble of probabilistic networks similarly as in [8, 18].

Original: The policy optimization stage allows Dyna-style algorithms to leverage various off-the-shelf modelfree methods, such as SAC [15], TRPO [39], and TD3 [13].
Revised: The policy optimization stage allows Dyna-style algorithms to leverage various off-the-shelf modelfree methods, such as SAC [15], TRPO [39], and TD3 [13].

Original: Much owing to the progress of model-freemethods, our method is to invoke any reasonable optimization oracle for the empirical models, ratherthan entangle a particular policy optimization algorithm.
Revised: Much owing to the progress of model-freemethods, our method is to invoke any reasonable optimization oracle for the empirical models, ratherthan entangle a particular policy optimization algorithm.

Original: A consensus of MBRL is that a smart policy requires an accurate model.
Revised: A consensus of MBRL is that a smart policy requires an accurate model.

Original: However, model bias cannotbe eliminated because the state-action distribution of the samples in the model learning stage andpolicy optimization stage is quite different.
Revised: However, model bias cannotbe eliminated because the state-action distribution of the samples in the model learning stage andpolicy optimization stage is quite different.

Original: Many prior works attend to this distribution mismatchissue and then tailor the data used for policy optimization according to the model bias.
Revised: Many prior works attend to this distribution mismatchissue and then tailor the data used for policy optimization according to the model bias.

Original: Janneret al.
Revised: Janneret al.

Original: [18], Buckman et al.
Revised: [18], Buckman et al.

Original: [7] encourage truncated rollout lengths.
Revised: [7] encourage truncated rollout lengths.

Original: Besides, the ratio of real tomodel-generated data can be dynamically tuned according to the model uncertainty [22, 36, 26].
Revised: Besides, the ratio of real tomodel-generated data can be dynamically tuned according to the model uncertainty [22, 36, 26].

Original: Yuet al.
Revised: Yuet al.

Original: [51] penalizes rewards by the model uncertainty.
Revised: [51] penalizes rewards by the model uncertainty.

Original: Our method incorporates the truncated modelrollouts mechanism.
Revised: Our method incorporates the truncated modelrollouts mechanism.

Original: Moreover, we further explore how real interactions affect overall performancewhich is rarely studied before.
Revised: Moreover, we further explore how real interactions affect overall performancewhich is rarely studied before.

Original: We construct an event-triggered mechanism to cope with overfitting ina small-data regime and suffering generalization error when facing a drastic distribution shift.
Revised: We construct an event-triggered mechanism to cope with overfitting ina small-data regime and suffering generalization error when facing a drastic distribution shift.

Original: Monotonic improvement guarantee has been a fundamental concern in both model-free and modelbased avenues.
Revised: Monotonic improvement guarantee has been a fundamental concern in both model-free and modelbased avenues.

Original: In MFRL methods, both CPI [20] and TRPO [39] can be understood as approximatingand optimizing the performance gap by forcing the new policy to be not too far away from the currentpolicy.
Revised: In MFRL methods, both CPI [20] and TRPO [39] can be understood as approximatingand optimizing the performance gap by forcing the new policy to be not too far away from the currentpolicy.

Original: However, in Model-based settings, their trust-region constraints cannot directly be satisfiedbecause the policies highly depend on the randomness of the models.
Revised: However, in Model-based settings, their trust-region constraints cannot directly be satisfiedbecause the policies highly depend on the randomness of the models.

Original: While constructing such abound for performance gap is straightforward, it has not been explored in previous MBRL theorecticalanalyses, instead they [44, 31, 18, 11, 26] turn to bound the discrepancy between returns under amodel and those in the real environment.
Revised: While constructing such abound for performance gap is straightforward, it has not been explored in previous MBRL theorecticalanalyses, instead they [44, 31, 18, 11, 26] turn to bound the discrepancy between returns under amodel and those in the real environment.

Original: Although they could guarantee that the lower bound ofpolicy performance improves under a certain model, this guarantee may face several issues regardingmodel updating.
Revised: Although they could guarantee that the lower bound ofpolicy performance improves under a certain model, this guarantee may face several issues regardingmodel updating.

Original: In contrast, we construct the performance difference scheme for MBRL algorithmsand perform monotonicity analysis under this scheme.
Revised: In contrast, we construct the performance difference scheme for MBRL algorithmsand perform monotonicity analysis under this scheme.

Original: Another line of theoretical works focus on regret bounds [10, 21] or sample complexity properties [1,5, 42], focusing on the convergence performance and sample complexity for model-based approaches.
Revised: Another line of theoretical works focus on regret bounds [10, 21] or sample complexity properties [1,5, 42], focusing on the convergence performance and sample complexity for model-based approaches.

Original:  Preliminaries
Revised:  Preliminaries

Original: Markov Decision Process A discounted Markov Decision Process (MDP) is a quintuple M =( S , A , P M , r M , γ ) , where S is the state space, A represents the action space, P M denotes thetransition function, r : S ×
Revised: Markov Decision Process A discounted Markov Decision Process (MDP) is a quintuple M =( S , A , P M , r M , γ ) , where S is the state space, A represents the action space, P M denotes thetransition function, r : S ×

Original: A → [ − R, R ] stands for the reward function, and γ ∈ (0 , 1) is thediscount factor.
Revised: A → [ − R, R ] stands for the reward function, and γ ∈ (0 , 1) is thediscount factor.

Original: For a fixed policy π and model M , we define V πM ( µ ) as the return of the model Mwith the starting state distribution µ , and V π ( µ ) denotes the returns under the real environment,
Revised: For a fixed policy π and model M , we define V πM ( µ ) as the return of the model Mwith the starting state distribution µ , and V π ( µ ) denotes the returns under the real environment,

Original: [Equation]
Revised: [Equation]

Original: We make a mild assumption that the model M to be identical to the real MDP except the transitionfunction.
Revised: We make a mild assumption that the model M to be identical to the real MDP except the transitionfunction.

Original: Let d π k M i ( s, a ; µ ) denote the visitation probability s, a when starting at s 0 ∼ µ and followingπ k under the dynamics P M i .
Revised: Let d π k M i ( s, a ; µ ) denote the visitation probability s, a when starting at s 0 ∼ µ and followingπ k under the dynamics P M i .

Original: We will omit it as d π k M i henceforth for brevity.
Revised: We will omit it as d π k M i henceforth for brevity.

Original: Besides, we denote M asa (parameterized) family of models of interest, and let Π be a family of policies.
Revised: Besides, we denote M asa (parameterized) family of models of interest, and let Π be a family of policies.

Original: Generative Model
Revised: Generative Model

Original: Many previous works [29, 23, 1] focus on a stylized generative model.
Revised: Many previous works [29, 23, 1] focus on a stylized generative model.

Original: Byassuming an access to the generative model, we collect N samples for each state-action pair ( s, a ) ∈
Revised: Byassuming an access to the generative model, we collect N samples for each state-action pair ( s, a ) ∈

Original: S × A : s is,a i.i.d ∼ P ( ·| s, a ) which allows us to construct an empirical model defined as follows:
Revised: S × A : s is,a i.i.d ∼ P ( ·| s, a ) which allows us to construct an empirical model defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 {·} is the indicator function.
Revised: where 1 {·} is the indicator function.

Original: This leads to an empirical MDP ˆ M = ( S , A , ˆ P, r, γ ) .
Revised: This leads to an empirical MDP ˆ M = ( S , A , ˆ P, r, γ ) .

Original:  Monotonic improvement under model shifts
Revised:  Monotonic improvement under model shifts

Original: This section provides theoretical analyses for monotonic improvement of MBRL, factoring in theinterdependence between policies and models.
Revised: This section provides theoretical analyses for monotonic improvement of MBRL, factoring in theinterdependence between policies and models.

Original: We first construct a general scheme for a nondecreasing performance guarantee and follow it up by characterizing the lower bound when shiftingthe model.
Revised: We first construct a general scheme for a nondecreasing performance guarantee and follow it up by characterizing the lower bound when shiftingthe model.

Original: Towards a non-negative lower bound, we restrict the model shifts and then obtain arefined bound.
Revised: Towards a non-negative lower bound, we restrict the model shifts and then obtain arefined bound.

Original: These discoveries encourage us to translate the bound maximization to a constrainedoptimization problem to permit monotonicity.
Revised: These discoveries encourage us to translate the bound maximization to a constrainedoptimization problem to permit monotonicity.

Original: By deriving an instance solution under the generativemodel setting, we demonstrate the merits of the dynamic model learning interval.
Revised: By deriving an instance solution under the generativemodel setting, we demonstrate the merits of the dynamic model learning interval.

Original: 4.1 Monotonic improvement with policy optimization oracle
Revised: 4.1 Monotonic improvement with policy optimization oracle

Original: Our goal is to construct a general recipe for a monotonicity guarantee.
Revised: Our goal is to construct a general recipe for a monotonicity guarantee.

Original: Naturally, we seek to build aperformance difference scheme for model-based algorithms.
Revised: Naturally, we seek to build aperformance difference scheme for model-based algorithms.

Original: Definition 4.1 ( Performance Difference Bound Scheme ) .
Revised: Definition 4.1 ( Performance Difference Bound Scheme ) .

Original: V π i | M i ( µ ) denotes the return of thepolicy π i ∈ Π in the real environment, whereas π i is derived from the dynamical model M i ∈ M .
Revised: V π i | M i ( µ ) denotes the return of thepolicy π i ∈ Π in the real environment, whereas π i is derived from the dynamical model M i ∈ M .

Original: Then, the lower bound on the true return gap of π 1 and π 2 can be stated in the form,
Revised: Then, the lower bound on the true return gap of π 1 and π 2 can be stated in the form,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Such a statement guarantees that the policy allows non-decreasing performance in the real environment when C is non-negative.
Revised: Such a statement guarantees that the policy allows non-decreasing performance in the real environment when C is non-negative.

Original: Although there has been interest in non-decreasing performance guarantee, previous works [31, 18]commonly derive under a "discrepancy bound" scheme disregarding the model shifts ( i.e. M 1 = M 2 ).
Revised: Although there has been interest in non-decreasing performance guarantee, previous works [31, 18]commonly derive under a "discrepancy bound" scheme disregarding the model shifts ( i.e. M 1 = M 2 ).

Original: Their results imply that once a policy update π 1 → π 2 increases the returns under the same model( V π 2 M 1 > V π 1 M 1 ), the lower bound on the policy performance evaluated in the real environment improvesaccordingly, inf { V π 2 | M 1 } > inf { V π 1 | M 1 }.
Revised: Their results imply that once a policy update π 1 → π 2 increases the returns under the same model( V π 2 M 1 > V π 1 M 1 ), the lower bound on the policy performance evaluated in the real environment improvesaccordingly, inf { V π 2 | M 1 } > inf { V π 1 | M 1 }.

Original: When the model shift is introduced, establishing the performance difference bound turns out tobe rather difficult, mainly due to the coupling of the policy optimization and model learning: theestimated model is generated from the policy explorations, while the policy derives from the modelrollouts.
Revised: When the model shift is introduced, establishing the performance difference bound turns out tobe rather difficult, mainly due to the coupling of the policy optimization and model learning: theestimated model is generated from the policy explorations, while the policy derives from the modelrollouts.

Original: Hence, we need to consider the performance gap arising from the integration of the twostages, which, unfortunately, has never been explored before.
Revised: Hence, we need to consider the performance gap arising from the integration of the twostages, which, unfortunately, has never been explored before.

Original: Since the performance of the policywith a fixed model is already well guaranteed, it is natural to make the following assumptions.
Revised: Since the performance of the policywith a fixed model is already well guaranteed, it is natural to make the following assumptions.

Original: Assumption 4.2 ( Policy Optimization Oracle ) .
Revised: Assumption 4.2 ( Policy Optimization Oracle ) .

Original: The policy optimization oracle is defined as theone that takes as input a model M and returns a ϵ opt -optimal policy π satisfying: V ∗ M ( µ ) − ϵ opt ≤
Revised: The policy optimization oracle is defined as theone that takes as input a model M and returns a ϵ opt -optimal policy π satisfying: V ∗ M ( µ ) − ϵ opt ≤

Original: V πM ( µ ) ≤ V ∗ M ( µ ) .
Revised: V πM ( µ ) ≤ V ∗ M ( µ ) .

Original: We assume that our policy optimization stage always meets the policy optimizationoracle given its corresponding estimated model M .
Revised: We assume that our policy optimization stage always meets the policy optimizationoracle given its corresponding estimated model M .

Original: Such assumption usually holds in practice when we explore existing off-the-shelf model-free algorithms [38, 1, 15, 39].
Revised: Such assumption usually holds in practice when we explore existing off-the-shelf model-free algorithms [38, 1, 15, 39].

Original: With this assumption, we can focus directly on the eventual performancedifference under the real environment encountering the model updating.
Revised: With this assumption, we can focus directly on the eventual performancedifference under the real environment encountering the model updating.

Original: The bound C we seek can be expressed in terms of two kinds of gaps: the inconsistency gap betweenthe model and the environment, and the optimal returns gap between the two models.
Revised: The bound C we seek can be expressed in terms of two kinds of gaps: the inconsistency gap betweenthe model and the environment, and the optimal returns gap between the two models.

Original: When apolicy π k samples in a model M i , it will encounter states not consistent with those generated by thereal environment.
Revised: When apolicy π k samples in a model M i , it will encounter states not consistent with those generated by thereal environment.

Original: We denote this inconsistency by
Revised: We denote this inconsistency by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Besides, for each model M i ∈ M , a remarkable property is that there always exists a policy thatmaximizes the value function V πM i ( µ ) [48].
Revised: Besides, for each model M i ∈ M , a remarkable property is that there always exists a policy thatmaximizes the value function V πM i ( µ ) [48].

Original: Hence, we define the ceiling performance of model
Revised: Hence, we define the ceiling performance of model

Original: M as V ∗ M i ( µ ) =
Revised: M as V ∗ M i ( µ ) =

Original: sup π ∈ Π V πM i ( µ ) .
Revised: sup π ∈ Π V πM i ( µ ) .

Original: We will omit µ henceforth for simplicity unless confusion exists.
Revised: We will omit µ henceforth for simplicity unless confusion exists.

Original: With these two terms well-defined, we now present our bound.
Revised: With these two terms well-defined, we now present our bound.

Original: Theorem 4.3 ( Performance Difference Bound for Model-based RL ) .
Revised: Theorem 4.3 ( Performance Difference Bound for Model-based RL ) .

Original: Let M i ∈
Revised: Let M i ∈

Original: M be theestimated models and π i be the ϵ opt -optimal policy for M i .
Revised: M be theestimated models and π i be the ϵ opt -optimal policy for M i .

Original: Recalling κ = 2 Rγ (1 − γ ) 2 where R is thebound of the reward function, we have the performance difference of π 1 and π 2 evaluated under thereal environment be bounded as below,
Revised: Recalling κ = 2 Rγ (1 − γ ) 2 where R is thebound of the reward function, we have the performance difference of π 1 and π 2 evaluated under thereal environment be bounded as below,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B, Theorem B.1.
Revised: See Appendix B, Theorem B.1.

Original: This bound implies that if the model update M 1 → M 2 can (1) shorten the divergence between theestimated dynamics and the true dynamics and (2) improve the ceiling performance on the model, itmay guarantee overall performance improvement under the true dynamics.
Revised: This bound implies that if the model update M 1 → M 2 can (1) shorten the divergence between theestimated dynamics and the true dynamics and (2) improve the ceiling performance on the model, itmay guarantee overall performance improvement under the true dynamics.

Original: 4.2 Lower-bound optimization with model shift constraints
Revised: 4.2 Lower-bound optimization with model shift constraints

Original: Theorem 4.3 provides a general performance difference bound that suggests models with higherceiling performance and lower bias would raise the overall performance.
Revised: Theorem 4.3 provides a general performance difference bound that suggests models with higherceiling performance and lower bias would raise the overall performance.

Original: However, finding a nonnegative lower bound may face several issues regarding a drastic model shift in practice.
Revised: However, finding a nonnegative lower bound may face several issues regarding a drastic model shift in practice.

Original: On the onehand, performing an abrupt model update could potentially lead to a tumble in ceiling performanceand then fail to make incremental improvements.
Revised: On the onehand, performing an abrupt model update could potentially lead to a tumble in ceiling performanceand then fail to make incremental improvements.

Original: On the other hand, huge distribution divergencebetween model rollouts would confound the policy optimization stage and hamper its access tooptimal policies.
Revised: On the other hand, huge distribution divergencebetween model rollouts would confound the policy optimization stage and hamper its access tooptimal policies.

Original: Thus, we shoot for refining the bound upon adding model shifts constraint.
Revised: Thus, we shoot for refining the bound upon adding model shifts constraint.

Original: First, we seek to further unfold the ceiling performance gap, which serves as the main building blocktoward the desired bound.
Revised: First, we seek to further unfold the ceiling performance gap, which serves as the main building blocktoward the desired bound.

Original: During derivation, we additionally introduce the L -Lipschitz assumption.
Revised: During derivation, we additionally introduce the L -Lipschitz assumption.

Original: Assumption 4.4 ( L -Lipschitzness of Value Function ) .
Revised: Assumption 4.4 ( L -Lipschitzness of Value Function ) .

Original: We call a value function V πM on the estimateddynamical model M is L -Lipschitz w.r.t to some norm ∥ · ∥ in the sense that
Revised: We call a value function V πM on the estimateddynamical model M is L -Lipschitz w.r.t to some norm ∥ · ∥ in the sense that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We assume that our estimated model M ∈ M satisfies the Lipschiz character, inspired from [4, 31].
Revised: We assume that our estimated model M ∈ M satisfies the Lipschiz character, inspired from [4, 31].

Original: Under the L -Lipschitzness assumption, we can derive a bound for the ceiling return gap.
Revised: Under the L -Lipschitzness assumption, we can derive a bound for the ceiling return gap.

Original: Theorem 4.5 ( Ceiling Return Gap under Model Shift ) .
Revised: Theorem 4.5 ( Ceiling Return Gap under Model Shift ) .

Original: For an estimated model M i ∈ M , theceiling return gap is bounded as:
Revised: For an estimated model M i ∈ M , theceiling return gap is bounded as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B, Theorem B.2.
Revised: See Appendix B, Theorem B.2.

Original: This conclusion reveals the connection between the ceiling return gap and models’ disagreement.
Revised: This conclusion reveals the connection between the ceiling return gap and models’ disagreement.

Original: Ina benign scenario, the term of ceiling performance gap in the RHS of Eq.
Revised: Ina benign scenario, the term of ceiling performance gap in the RHS of Eq.

Original: 4.2 should be dominated bythe term κ · ( ϵ π 1 M 1 − ϵ π 2 M 2 ) when the model shift is sufficiently small.
Revised: 4.2 should be dominated bythe term κ · ( ϵ π 1 M 1 − ϵ π 2 M 2 ) when the model shift is sufficiently small.

Original: A sharp model shift, however,risks a massive reduction in ceiling performance that is hardly bridged by the other parts in Eq.
Revised: A sharp model shift, however,risks a massive reduction in ceiling performance that is hardly bridged by the other parts in Eq.

Original: 4.2,and therefore corrupts monotonicity.
Revised: 4.2,and therefore corrupts monotonicity.

Original: This inspires us to introduce the constraint of the model shift.
Revised: This inspires us to introduce the constraint of the model shift.

Original: We further refine our performance difference bound to better characterize the relationship betweenperformance gap and model shift.
Revised: We further refine our performance difference bound to better characterize the relationship betweenperformance gap and model shift.

Original: Theorem 4.6 ( Refined Bound with Constraint ) .
Revised: Theorem 4.6 ( Refined Bound with Constraint ) .

Original: Let policy π
Revised: Let policy π

Original: i ∈ Π denotes the ϵ opt optimal policyunder the dynamical model M i ∈ M, and σ M 1 ,M 2 be the constraint threshold for M 1 and M 2 . 
Revised: i ∈ Π denotes the ϵ opt optimal policyunder the dynamical model M i ∈ M . We give the model shift constraints under the TV-distance.

Original: Note that L ≥ R 1 − γ .

Original: Then we can refine performance difference lower bound under the model shiftsconstraint as,
Revised: Then we can refine performance difference lower bound under the model shifts constraint as,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix B, Theorem B.3.
Revised: See Appendix B, Theorem B.3.

Original: Theorem 4.6 implies that policy π 2 is guaranteed to outperform policy
Revised: Theorem 4.6 implies that policy π 2 is guaranteed to outperform policy

Original: π 1 once it makes the RHS in
Revised: π 1 once it makes the RHS in

Original: Eq.
Revised: Eq.

Original: 4.5 greater than zero under the constraint Eq.
Revised: 4.5 greater than zero under the constraint Eq.

Original: More specifically, to guarantee a non-decreasingperformance, M 2 and π 2 should meet the following two requirements,
Revised: More specifically, to guarantee a non-decreasingperformance, M 2 and π 2 should meet the following two requirements,

Original: [Equation]
Revised: [Equation]

Original: R1 encourages us to alleviate model bias as much as possible.
Revised: R1 encourages us to alleviate model bias as much as possible.

Original: Moreover, when the policy π 1 samplestoo many states not encountered by the model M 1 , it may lead to excessive generalization errorsduring model learning, i.e. the estimated model will suffer extrapolation error in these unexperiencedregions, which further causes instability or crashes in the following derived sub-optimal policies.
Revised: Moreover, when the policy π 1 samplestoo many states not encountered by the model M 1 , it may lead to excessive generalization errorsduring model learning, i.e. the estimated model will suffer extrapolation error in these unexperiencedregions, which further causes instability or crashes in the following derived sub-optimal policies.

Original: Thus, the introduction of the R2 constraint can help solve the above problem.
Revised: Thus, the introduction of the R2 constraint can help solve the above problem.

Original: Finally, we abstractthese two requirements to a constrained optimization problem.
Revised: Finally, we abstractthese two requirements to a constrained optimization problem.

Original: Proposition 4.7 ( Constrained Lower-Bound Optimization Problem ) .
Revised: Proposition 4.7 ( Constrained Lower-Bound Optimization Problem ) .

Original: We reduce the issue offinding a non-negative C to the following constrained optimization problem.
Revised: We reduce the issue offinding a non-negative C to the following constrained optimization problem.

Original: Here, π i is still thesub-optimal policy under model M i .
Revised: Here, π i is still thesub-optimal policy under model M i .

Original: The minimal objective in E.q. 4.7 leads to the maximum of C .
Revised: The minimal objective in E.q. 4.7 leads to the maximum of C .

Original: Then the overall optimization problem can be formalized as,
Revised: Then the overall optimization problem can be formalized as,

Original: [Equation]
Revised: [Equation]

Original: 4.3 A feasible example for constrained optimization problem
Revised: 4.3 A feasible example for constrained optimization problem

Original: We first remark that, Proposition 4.7 provides a useful guide for acquiring performance improvementthrough restricting the upcoming model shift into a safe zone.
Revised: We first remark that, Proposition 4.7 provides a useful guide for acquiring performance improvementthrough restricting the upcoming model shift into a safe zone.

Original: In this section, we give an instance for a feasible solution of the constrained optimization problem under the generative model setting [1, 29].
Revised: In this section, we give an instance fora feasible solution of the constrained optimization problem under the generative model setting [1, 29].

Original: Specifically, we construct an empirical model M 1 from the N samples per state-action pair that stemsfrom the generative model.
Revised: Specifically, we construct an empirical model M 1 from the N samples per state-action pair that stemsfrom the generative model.

Original: Upon encountering another k samples on each state-action pair, wesegue into the model updating stage, which outputs M 2 given these newly collected samples as input.
Revised: Upon encountering another k samples on each state-action pair, wesegue into the model updating stage, which outputs M 2 given these newly collected samples as input.

Original: Towards obtaining a non-decreasing performance, we yield the following feasible solution for modeltraining interval through satisfying requirements R1 and R2.
Revised: Towards obtaining a non-decreasing performance, we yield the following feasible solution for modeltraining interval through satisfying requirements R1 and R2.

Original: Corollary 4.8.
Revised: Corollary 4.8.

Original: Here, vol ( S ) denotes the volume of the state coverage simplex.
Revised: Here, vol ( S ) denotes the volume of the state coverage simplex.

Original: For simplicity, wedenote δ M i ( ·| s, a ) =
Revised: For simplicity, wedenote δ M i ( ·| s, a ) =

Original: | P ( ·| s, a ) − P M i ( ·| s, a ) | for each model M i ∈ M .
Revised: | P ( ·| s, a ) − P M i ( ·| s, a ) | for each model M i ∈ M .

Original: Under the generative modelsetting, with a probability larger than 1 − ξ , we can provide a non-negative C when given,
Revised: Under the generative modelsetting, with a probability larger than 1 − ξ , we can provide a non-negative C when given,

Original: [Equation]
Revised: [Equation]

Original: Here, ϵ = δ M 1 ( ·| s,a ) − (1 − γ ) L R
Revised: Here, ϵ = δ M 1 ( ·| s,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) − (

Original: [Equation]
Revised: [Equation]

Original: 2 Rγ · ϵ opt and ξ ∈ (0 , 1) is a constant.
Revised: · ϵ opt and ξ ∈ (0 , 1) is a constant.

Original: Proof.
Revised: Proof.

Original: See Appendix B, Corollary B.4.
Revised: See Appendix B, Corollary B.4.

Original: One can deduce from Corollary 4.8 that dynamically adjusting the model training interval accordingto the model bias and the model shifts constraints threshold does benefit monotonicity.
Revised: One can deduce from Corollary 4.8 that dynamically adjusting the model training interval accordingto the model bias and the model shifts constraints threshold does benefit monotonicity.

Original: Along withthe model bias δ M 1 ( ·| s, a ) decaying, the model training interval k requires to be scaled up to obtainadequate newly encountered samples for training M 2 .
Revised: Along withthe model bias δ M 1 ( ·| s, a ) decaying, the model training interval k requires to be scaled up to obtainadequate newly encountered samples for training M 2 .

Original: Besides, once gathering excessive samples,we risk violating the model shifts constraint, thus impairing performance.
Revised: Besides, once gathering excessive samples,we risk violating the model shifts constraint, thus impairing performance.

Original: This instance supportsthe insight that determining “when to update your model” is vital for performance improvement andmotivates a smarter scheme to choose different numbers of explorations at different times instead ofthe unchanged setting in current methods.
Revised: This instance supportsthe insight that determining “when to update your model” is vital for performance improvement andmotivates a smarter scheme to choose different numbers of explorations at different times instead ofthe unchanged setting in current methods.

Original:  CLMO framework
Revised:  CLMO framework

Original: It is nontrivial to tackle the proposed constrained optimization problem in Proposition 4.7, sinceone cannot directly assign a value to the optimization variable M 2 .
Revised: It is nontrivial to tackle the proposed constrained optimization problem in Proposition 4.7, sinceone cannot directly assign a value to the optimization variable M 2 .

Original: We decouple the optimizationobjective and the constraint as “how to train the model” and “when to train the model” through anevent-triggered mechanism towards dynamic alternation.
Revised: We decouple the optimizationobjective and the constraint as “how to train the model” and “when to train the model” through anevent-triggered mechanism towards dynamic alternation.

Original: Objective minimization.
Revised: Objective minimization.

Original: Minimizing the objective function involves improving the model accuracy.
Revised: Minimizing the objective function involves improving the model accuracy.

Original: Specifically, we adopt the model-ensemble technique to reduce model bias.
Revised: Specifically, we adopt the model-ensemble technique to reduce model bias.

Original: For practicalimplementation, the probabilistic models { ˆ f ϕ 1 , ˆ f ϕ 2 , . . .
Revised: The probabilistic models{ ˆ f ϕ 1 , ˆ f ϕ 2 , . . .

Original: , ˆ f ϕ K } are fitted on shared but differentlyshuffled replay buffer D e , and the target is to optimize the Negative Log Likelihood (NLL).
Revised: , ˆ f ϕ K } are fitted on shared but differently shuffled replay buffer D e , and the target is tooptimize the Negative Log Likelihood (NLL).

Original: Constraint estimation.
Revised: Constraint estimation.

Original: The unobserved model M 2 makes the constraint function incalculable,and here we seek to design an estimator for it.
Revised: The unobserved model M 2 makes the constraint function incalculable,and here we seek to design an estimator for it.

Original: Recall that D TV ( P M 1 ( ·| s, a ) ∥
Revised: Recall that D TV ( P M 1 ( ·| s, a ) ∥

Original: P M 2 ( ·| s, a ))
Revised: P M 2 ( ·| s, a ))

Original: =12 (cid:80) s
Revised: =12 (cid:80) s

Original: ′ ∈S | P M 1 ( s ′ | s, a ) − P M 2 ( s ′ | s, a ) | , then we can find that the distance arises from two parts,the state space coverage, and the models’ disagreement.
Revised: ′ ∈S | P M 1 ( s ′ | s, a ) − P M 2 ( s ′ | s, a ) | , then we can find that the distance arises from two parts,the state space coverage, and the models’ disagreement.

Original: We estimate the policy coverage (statespace coverage) by computing the volume vol ( S D ) of the convex closure S D
Revised: We estimate the policy coverage (state-spacecoverage) by computing the volume vol ( S D ) of the convex closure S D

Original: = (cid:8) (cid:80) s
Revised: = (cid:8) (cid:80) s

Original: i ∈D λ
Revised: i ∈D λ

Original: i S i :λ i ≥ 0 , (cid:80) i λ i = 1 (cid:9) constructed on the replay buffer D .
Revised: i S i : λ i ≥0 , (cid:80) i λ i = 1 (cid:9) constructed on the replay buffer D .

Original: We exploit the average prediction erroron these new samples data to estimate the disagreement on newly encountered data ( ∆ D) and get
Revised: We exploit the average prediction error on thesenew samples data to estimate the disagreement on newly encountered data and get ∆ D and get

Original: L (∆ D ) =
Revised: L (∆ D ) =

Original: E ( s,a,s ′ ) ∈
Revised: E ( s,a,s ′ ) ∈

Original: ∆ D (cid:2) 1 K (cid:80) Ki =1 ∥ s ′ − ˆ f ϕ
Revised: ∆ D (cid:2) 1 K (cid:80) Ki =1 ∥ s ′ − ˆ f ϕ

Original: i
Revised: i

Original: ( s, a ) ∥ (cid:3) .
Revised: ( s, a ) ∥ (cid:3) .

Original: Combining these two components, we canobtain an estimation for the model shift, i.e. , vol ( S D ) ·
Revised: Combining these two components, we canobtain an estimation for the model shift, i.e. , vol ( S D ) ·

Original: L (∆ D ) .
Revised: L (∆ D ) .

Original: This practical overestimation for themodel shifts makes the constraint stay satisfied during objective optimization.
Revised: This practical overestimation for themodel shifts makes the constraint stay satisfied during objective optimization.

Original: Event-triggered mechanism.
Revised: Event-triggered mechanism.

Original: We design an event-triggered mechanism to determine the occasionto pause collecting and turn to solve the optimization objective.
Revised: We design an event-triggered mechanism to determine the occasionto pause collecting and turn to solve the optimization objective.

Original: Remark that although diverting totrain models as long as not to violate the constraint is theoretically reasonable, we refrain from doingso in practice because performing an update on data with a minor shift in coverage and distribution iswasteful and may risk overfitting.
Revised: Remark that although diverting totrain models as long as not to violate the constraint is theoretically reasonable, we refrain from doingso in practice because performing an update on data with a minor shift in coverage and distribution iswasteful and may risk overfitting.

Original: Therefore, we trigger when the constraint boundary is touched toreduce computational cost and escape overfitting.
Revised: Therefore, we trigger when the constraint boundary is touched toreduce computational cost and escape overfitting.

Original: The event-triggering mechanism is based on thefollowing condition:
Revised: The event-triggering mechanism is based on thefollowing condition:

Original: [Equation]
Revised: [Equation]

Original: Here, τ is the event-triggering time, and α is a given constant.
Revised: Here, τ is the event-triggering time, and α is a given constant.

Original: Policy optimization oracle.
Revised: Policy optimization oracle.

Original: Clearly, we can leverage many model-free RL methods (SAC [15],
Revised: Clearly, we can leverage many model-free RL methods (SAC [15],

Original: TRPO [39], PPO [41] etc. )
Revised: TRPO [39], PPO [41] etc. )

Original: as our policy optimization oracle.
Revised: as our policy optimization oracle.

Original: Besides, we adopt a truncated shortmodel rollouts technique to mitigate compounding error while encouraging model usage.
Revised: Besides, we adopt a truncated shortmodel rollouts technique to mitigate compounding error while encouraging model usage.

Original: Based onthe fresh model rollouts, we perform the policy optimization oracle, employing SAC as an example.
Revised: Based onthe fresh model rollouts, we perform the policy optimization oracle, employing SAC as an example.

Original: Algorithm Overview.
Revised: Algorithm Overview.

Original: We briefly give an overview of our proposed CMLO in algorithm 1.
Revised: We briefly give an overview of our proposed CMLO in algorithm 1.

Original: Notably,the event-triggered mechanism subtly determines the occasion to perform model updating, promotingperformance monotonicity and reducing computation load.
Revised: Notably,the event-triggered mechanism subtly determines the occasion to perform model updating, promotingperformance monotonicity and reducing computation load.

Original: Algorithm 1: initialize : policy π θ , ensemble models ˆ f ϕ 1 , ˆ f ϕ 2 , . . .
Revised: initialize : policy π θ , ensemble models ˆ f ϕ 1 , ˆ f ϕ 2 , . . .

Original: , ˆ f ϕ K , environment buffer D e and model buffer
Revised: , ˆ f ϕ K , environment buffer D e and model buffer

Original: D m repeat
Revised: D m repeat

Original: Sample ∆ D e ∼ π θ from real environment; add to D e
Revised: Sample ∆ D e ∼ π θ from real environment; add to D e

Original: Estimate model shifts by vol ( S D e ) L (∆ D e ) if Event-triggered condition (5.1) is reached then
Revised: Estimate model shifts by vol ( S D e ) L (∆ D e ) if Event-triggered condition (5.1) is reached then

Original: Train all models ˆ f ϕ 1 , ˆ f ϕ 2 , . .
Revised: Train all models ˆ f ϕ 1 , ˆ f ϕ 2 , . .

Original: , ˆ f ϕ K on D e
Revised: , ˆ f ϕ K on D e

Original: Perform h -step model rollouts using policy π θ ; add to D m
Revised: Perform h -step model rollouts using policy π θ ; add to D m

Original: Update π θ on D m through SAC [15] until the policy performs well in the environment ;
Revised: Update π θ on D m through SAC [15] until the policy performs well in the environment ;

Original:  Experiments
Revised:  Experiments

Original: Our experimental evaluation aims to investigate the following questions: (1) How well does ouralgorithm perform on standard reinforcement learning benchmarks compared to prior state-of-the-artmodel-based and model-free algorithms?
Revised: Our experimental evaluation aims to investigate the following questions: (1) How well does ouralgorithm perform on standard reinforcement learning benchmarks compared to prior state-of-the-artmodel-based and model-free algorithms?

Original: (2) Does the performance with or without the constraintconsistent with previous theoretical analyses?
Revised: (2) Does the performance with or without the constraintconsistent with previous theoretical analyses?

Original: [Figure]
Revised: [Figure]

Original: 6.1 Comparative evaluation
Revised: 6.1 Comparative evaluation

Original: To illustrate the effectiveness of our method, we contrast several popular model-based and model-freebaselines.
Revised: To illustrate the effectiveness of our method, we contrast several popular model-based and model-freebaselines.

Original: Model-free counterparts include: (1) SAC [15], the state-of-the-art in terms of asymptotic performance.
Revised: Model-free counterparts include: (1) SAC [15], the state-of-the-art in terms of asymptoticperformance.

Original: (2) PPO [41] that explores monotonic improvement as well.
Revised: (2) PPO [41] that explores monotonic improvement as well.

Original: Model-based baselinesinclude: (3) PETS [8], which employs models directly for planning, different from the Dyna-style.(4) SLBO [31], that explores monotonicity under the discrepancy bound scheme.
Revised: Model-based baselinesinclude: (3) PETS [8], which employs models directly for planning, different from the Dyna-style. (4) SLBO [31], that explores monotonicity under the discrepancy bound scheme.

Original: (5) MBPO
Revised: (5) MBPO

Original: [18],that employs a similar design of model ensemble technique (ensemble of probabilistic dynamicsnetworks) and policy optimization oracle (SAC) as we do.
Revised: [18],that employs a similar design of model ensemble technique (ensemble of probabilistic dynamicsnetworks) and policy optimization oracle (SAC) as we do.

Original: (6) AutoMBPO [26], a variant of MBPO,that uses an automatic hyperparameter controller to tune the model-training frequency but suffersfrom high pre-training cost and lacks theoretical analysis on parameters rationality.
Revised: (6) AutoMBPO [26], a variant of MBPO,that uses an automatic hyperparameter controller to tune the model-training frequency but suffersfrom high pre-training cost and lacks theoretical analysis on parameters rationality.

Original: We evaluate CMLO and these baselines on six continuous control tasks in OpenAI Gym [6] withthe MuJoCo [49] physics simulator, including HalfCheetah, Hopper, Walker2d, Swimmer, Ant,
Revised: We evaluate CMLO and these baselines on six continuous control tasks in OpenAI Gym [6] withthe MuJoCo [49] physics simulator, including HalfCheetah, Hopper, Walker2d, Swimmer, Ant,

Original: Humanoid.
Revised: Humanoid.

Original: For fair comparison, we adopt the standard full-length version of these tasks and alignthe same environment settings.
Revised: For fair comparison, we adopt the standard full-length version of these tasks and alignthe same environment settings.

Original: Figure 1 shows the learning curves of all compared methods, along with the asymptotic performance.
Revised: Figure 1 shows the learning curves of all compared methods, along with the asymptotic performance.

Original: These results show that our algorithm is far ahead of the model-free method in terms of sampleefficiency, coupled with an asymptotic performance comparable to that of the state-of-the-art modelfree counterparts SAC.
Revised: These results show that our algorithm is far ahead of the model-free method in terms of sampleefficiency, coupled with an asymptotic performance comparable to that of the state-of-the-art modelfree counterparts SAC.

Original: Compared to model-based baselines, our method gains faster convergencespeed and better eventual performance.
Revised: Compared to model-based baselines, our method gains faster convergencespeed and better eventual performance.

Original: Notably, credit to the event-triggered mechanism, our methodenjoys a more stable training curve.
Revised: Notably, credit to the event-triggered mechanism, our methodenjoys a more stable training curve.

Original: The better monotonic property of the learning curve agrees withour previous analyses.
Revised: The better monotonic property of the learning curve agrees withour previous analyses.

Original: 6.2 Ablation studies
Revised: 6.2 Ablation studies

Original: Next, we make ablations and modifications to our method to validate the effectiveness and generalizability of the mechanism we devised. 
Revised: Next, we make ablations and modifications to our method to validate the effectiveness and generalizability of the mechanism we devised. [Figure] d t i [Figure]

Revised: e s

Original: [Figure]
Revised: [Figure]

Original: These experiments are average over 5 random seeds.
Revised: These experiments are average over 5 random seeds.

Original: (a) shows the average return with or withoutevent-triggered mechanism in HalfCheetah and Ant benchmarks. (b) shows the number of triggered times per 10k step.
Revised: (a shows the average return with or without

Original: [Figure]
Revised: -triggered

Original:  [Figure]we apply log
Revised: HalfCheetah the number of triggered [Figure] [Figure]

Original: The necessity of event-triggered mechanism.
Revised: The necessity of event-triggered mechanism.

Original: To verify the necessity of event-triggered mechanism, we compare to three unconstrained cases (given fixed model training interval) under twoenvironments.
Revised: To verify the necessity of event-triggered mechanism, we compare to three unconstrained cases (given fixed model training interval) under twoenvironments.

Original: The training curve and triggered times are shown in Figure 2.
Revised: The training curve and triggered times are shown in Figure 2.

Original: Clearly, our mechanismimproves the performance while reducing the total times of model training.
Revised: Clearly, our mechanism improves the performance while reducing the total times of model training.

Original: Besides, we notice thatthe performance is comparable to other MBRL baselines (MBPO etc. )
Revised: Besides, we notice thatthe performance is comparable to other MBRL baselines (MBPO etc. )

Original: when fixing our model traininginterval at 250.
Revised: when fixing our model traininginterval at 250.

Original: Still it performs worse than that equipped with a smart mechanism to decide whetherto train the model at current exploration step.
Revised: Still it performs worse than that equipped with a smart mechanism to decide whetherto train the model at current exploration step.

Original: To better understand why the event-triggered mechanism brings up our outperformance, we asses itsmain bricks.
Revised: To better understand why the event-triggered mechanism brings up our outperformance, we asses itsmain bricks.

Original: Model shift, which reflects the current ability to digest new data, is the basis of triggeredcondition.
Revised: Model shift, which reflects the current ability to digest new data, is the basis of triggeredcondition.

Original: And we estimate model shift from two parts, policy coverage and prediction error.
Revised: And we estimate model shift from two parts, policy coverage and prediction error.

Original: In
Revised: In

Original: Figure 3 we observe that, gradually, as the training progresses, the policy coverage increases, whichreflects our policy has new explorations at every stage without falling into a local optimum prematurely.
Revised: Figure 3 we observe that, gradually, as the training progresses, the policy coverage increases, whichreflects our policy has new explorations at every stage without falling into a local optimum prematurely.

Original: Also, the prediction error gradually decreases, which implies that our estimated dynamics come closerto the true dynamics in the explored region.
Revised: Also, the prediction error gradually decreases, which implies that our estimated dynamics come closerto the true dynamics in the explored region.

Original: Figure 4 implies that the number of samples requiredto hit the constraint tends to grows and the model training frequency then goes down.
Revised: Figure 4 implies that the number of samples requiredto hit the constraint tends to grows and the model training frequency then goes down.

Original: This is alsoconsistent with our intuition that, as the model refines and the exploration novelty fades, then thesample size required to reach a certain level of model shifts grows up.
Revised: This is alsoconsistent with our intuition that, as the model refines and the exploration novelty fades, then thesample size required to reach a certain level of model shifts grows up.

Original: The result agrees with ourtheoretical analyses which describe that a dynamic alternation subject to the model shifts constraintdoes benefit to monotonicity rather than an assigned one.
Revised: The result agrees with ourtheoretical analyses which describe that a dynamic alternation subject to the model shifts constraintdoes benefit to monotonicity rather than an assigned one.

Original: [Figure]
Revised: [Figure]

Original: The generalizability of event-triggered mechanism.
Revised: The generalizability of event-triggered mecha- nism.

Original: We further investigate the generalizabilityof our proposed mechanism through ablation onpolicy optimization oracle, and results are shownin Figure 5.
Revised: We further investigate the generalizabilityof our proposed mechanism through ablation on policy optimization oracle, and results are shownin Figure 5.

Original: 1) under Dyna-style.
Revised: 1) under Dyna-style.

Original: We adopt TRPO [39] asthe policy optimization oracle and test the performance with or without event-triggered mechanism
Revised: We adopt TRPO [39] as the policy optimization oracle and test the perfor- mance with or without event-triggered mechanism

Original: in Halfcheetah and Ant benchmarks.
Revised: in Halfcheetah and Ant benchmarks.

Original: Observably,
Revised: Observably,

Original: our mechanism can effectively guarantee the over- all performance improvement and alleviate thelocal optimization issue.
Revised: our mechanism can effectively guarantee the overall performance improvement and alleviate thelocal optimization issue.

Original: 2) jumping off Dyna-style.
Revised: 2) jumping off Dyna-style.

Original: We utilize iLQR [3]and then conduct ablation experiments in theDKitty-Stand [2] and Panda-Reaching [17] scenar- ios.
Revised: We utilize iLQR [3]and then conduct ablation experiments in theDKitty-Stand [2] and Panda-Reaching [17] scenarios.

Original: The result shows that adding event-triggered mechanism has a comparable asymptotic perfor-
Revised: The result shows that adding event-triggeredmechanism has a comparable asymptotic perfor-

Original: mance as the most-frequently triggered case but enjoys a lower computation cost
Revised: mance as the most-frequently triggered case butenjoys a lower computation cost.

Original:  Conclusion
Revised:  Conclusion

Original: We have investigated the role of the decision on "when to update the model" in joint optimizationprocedures through theoretical and empirical lens.
Revised: We have investigated the role of the decision on "when to update the model" in joint optimizationprocedures through theoretical and empirical lens.

Original: We devise a general novel scheme for exploring themonotonicity of MBRL methods, distinguished from the existing discrepancy bound scheme.
Revised: We devise a general novel scheme for exploring themonotonicity of MBRL methods, distinguished from the existing discrepancy bound scheme.

Original: We thenderive lower bounds under the scheme, suggesting that models with higher ceiling performance andlower bias guarantee a non-decreasing performance evaluated in the real environment.
Revised: We thenderive lower bounds under the scheme, suggesting that models with higher ceiling performance andlower bias guarantee a non-decreasing performance evaluated in the real environment.

Original: An effectiveconstrained optimization problem comes from the follow-up refined bound to seek a non-negativelower bound.
Revised: An effectiveconstrained optimization problem comes from the follow-up refined bound to seek a non-negativelower bound.

Original: Further, the instance under the generative model setting further verifies the effectivenessof learning models from a dynamically varying number of explorations.
Revised: Further, the instance under the generative model setting further verifies the effectivenessof learning models from a dynamically varying number of explorations.

Original: The algorithm CMLO,stemming from these analyses, has asymptotic performance rivaling the best model-free algorithmsand boasts better monotonicity.
Revised: The algorithm CMLO,stemming from these analyses, has asymptotic performance rivaling the best model-free algorithmsand boasts better monotonicity.

Original: Further ablation studies reveal that the proposed mechanism scales tovarious policy optimization oracles and benefits computation cost reduction.
Revised: Further ablation studies reveal that the proposed mechanism scales tovarious policy optimization oracles and benefits computation cost reduction.

Original: Currently, we observedempirically that the event-triggered condition is usually related to specific environments, which willcost a little time for tuning.
Revised: Currently, we observedempirically that the event-triggered condition is usually related to specific environments, which willcost a little time for tuning.

Original: Thus, one direction that merits further investigation is to construct the dualproblem of our constrained optimization problem for better exploring optimality and monotonicity.
Revised: Thus, one direction that merits further investigation is to construct the dualproblem of our constrained optimization problem for better exploring optimality and monotonicity.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 7.
Revised: [Yes] See Section 7.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] No potential
Revised: [No] No potential

Original: negative societal impacts have been found yet.
Revised: negative societal impacts have been found yet.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section
Revised: [Yes] See Section

Original: and Appendix B.
Revised: and Appendix B.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B and Appendix C.
Revised: [Yes] See Appendix B andAppendix C.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: In the
Revised: In the

Original: supplemental material.
Revised: supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Appendix E and our provided code.
Revised: [Yes] See Appendix E and our provided code.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See in corresponding figure captions.
Revised: [Yes] See in corresponding figure captions.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E.
Revised: [Yes] See Appendix E.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Appendix E.
Revised: [Yes] See Appendix E.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Appendix E.
Revised: [Yes] See Appendix E.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing data.
Revised: Our work does not use existing data.

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?

Original: [N/A] No crowdsourcing/human subjects are used.
Revised: [N/A] No crowdsourcing/human subjects are used.

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A] No crowdsourcing/human subjects are
Revised: [N/A] No crowdsourcing/human subjects are

Original: [Table]
Revised: [Table]

Original: used.
Revised: used.

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A] No crowdsourcing/human subjects are used.
Revised: [N/A] No crowdsourcing/human subjects are used.

Original: [Table]
Revised: [Table]