Original: Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learning
Revised: Plan To Predict: Learning an Uncertainty-Foreseeing Model For Model-Based Reinforcement Learning

Original:  Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress 
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In Model-based Reinforcement Learning (MBRL), model learning is critical sincean inaccurate model can bias policy learning via generating misleading samples.
Revised: In Model-based Reinforcement Learning (MBRL), model learning is critical sincean inaccurate model can bias policy learning via generating misleading samples.

Original: However, learning an accurate model can be difficult since the policy is continuallyupdated and the induced distribution over visited states used for model learningshifts accordingly.
Revised: However, learning an accurate model can be difficult since the policy is continuallyupdated and the induced distribution over visited states used for model learningshifts accordingly.

Original: Prior methods alleviate this issue by quantifying the uncertaintyof model-generated samples.
Revised: Prior methods alleviate this issue by quantifying the uncertaintyof model-generated samples.

Original: However, these methods only quantify the uncertaintypassively after the samples were generated, rather than foreseeing the uncertaintybefore model trajectories fall into those highly uncertain regions.
Revised: However, these methods only quantify the uncertaintypassively after the samples were generated, rather than foreseeing the uncertaintybefore model trajectories fall into those highly uncertain regions.

Original: The resulting lowquality samples can induce unstable learning targets and hinder the optimizationof the policy.
Revised: The resulting lowquality samples can induce unstable learning targets and hinder the optimizationof the policy.

Original: Moreover, while being learned to minimize one-step predictionerrors, the model is generally used to predict for multiple steps, leading to amismatch between the objectives of model learning and model usage.
Revised: Moreover, while being learned to minimize one-step predictionerrors, the model is generally used to predict for multiple steps, leading to amismatch between the objectives of model learning and model usage.

Original: To this end,we propose Plan To Predict (P2P), an MBRL framework that treats the modelrollout process as a sequential decision making problem by reversely consideringthe model as a decision maker and the current policy as the dynamics.
Revised: To this end,we propose Plan To Predict (P2P), an MBRL framework that treats the modelrollout process as a sequential decision making problem by reversely consideringthe model as a decision maker and the current policy as the dynamics.

Original: In thisway, the model can quickly adapt to the current policy and foresee the multi-stepfuture uncertainty when generating trajectories.
Revised: In thisway, the model can quickly adapt to the current policy and foresee the multi-stepfuture uncertainty when generating trajectories.

Original: Theoretically, we show that theperformance of P2P can be guaranteed by approximately optimizing a lower boundof the true environment return.
Revised: Theoretically, we show that theperformance of P2P can be guaranteed by approximately optimizing a lower boundof the true environment return.

Original: Empirical results demonstrate that P2P achievesstate-of-the-art performance on several challenging benchmark tasks.
Revised: Empirical results demonstrate that P2P achievesstate-of-the-art performance on several challenging benchmark tasks.

Original:  Introduction
Revised:  Introduction

Original: Through learning an approximate dynamics of the environment and then using it to assist policylearning, Model-based Reinforcement Learning (MBRL) methods [Moerland et al., 2020] havebeen shown to be much more sample-efficient than model-free methods both theoretically andempirically [Nagabandi et al., 2018, Chua et al., 2018, Luo et al., 2019, Janner et al., 2019].
Revised: Through learning an approximate dynamics of the environment and then using it to assist policylearning, Model-based Reinforcement Learning (MBRL) methods [Moerland et al., 2020] havebeen shown to be much more sample-efficient than model-free methods both theoretically andempirically [Nagabandi et al., 2018, Chua et al., 2018, Luo et al., 2018, Janner et al., 2019].

Original: Thelearning of the model is critical for MBRL, since an inaccurate model can bias policy learning throughgenerating misleading pseudo samples, which is also referred to as the model bias issue [Deisenrothand Rasmussen, 2011].
Revised: Thelearning of the model is critical for MBRL, since an inaccurate model can bias policy learning throughgenerating misleading pseudo samples, which is also referred to as the model bias issue [Deisenrothand Rasmussen, 2011].

Original: However, learning an accurate model can be difficult, since the update of thepolicy continually shifts the distribution over visited states, and due to lack of enough training datafor the modified distribution, the model can suffer from overfitting and become inaccurate in regionswhere the updated policy is likely to visit.
Revised: However, learning an accurate model can be difficult, since the update of thepolicy continually shifts the distribution over visited states, and due to lack of enough training datafor the modified distribution, the model can suffer from overfitting and become inaccurate in regionswhere the updated policy is likely to visit.

Original: This issue is further aggravated when using expressivemodels like deep neural networks, which are inherently prone to overfitting.
Revised: This issue is further aggravated when using expressivemodels like deep neural networks, which are inherently prone to overfitting.

Original: [Figure]
Revised: [Figure]

Original: Prior methods mitigate this issue by applying uncertainty quantification techniques, typically byusing a bootstrapped model ensemble [Kurutach et al., 2018, Chua et al., 2018, Janner et al., 2019].
Revised: Prior methods mitigate this issue by applying uncertainty quantification techniques, typically byusing a bootstrapped model ensemble [Kurutach et al., 2018, Chua et al., 2018, Janner et al., 2019].

Original: However, due to the lack of explicit considerations of the aforementioned policy shift issue, thesemethods can only quantify the uncertainty passively after the samples were generated, and thus fail toprevent the trajectories induced by the current policy from running into regions with high uncertainty.
Revised: However, due to the lack of explicit considerations of the aforementioned policy shift issue, thesemethods can only quantify the uncertainty passively after the samples were generated, and thusfail to prevent the trajectories induced by the current policy from running into regions with highuncertainty.

Original: This incapability to avoid uncertain regions can result in low quality of the generated samples, whichprovides unstable learning targets and thus may hinder the learning of the policy.
Revised: This incapability to avoid uncertain regions can result in low quality of the generatedsamples, which provides unstable learning targets and thus may hinder the learning of the policy.

Original: Moreover, themodel in the existing methods is usually used to predict for multiple steps, while the model learningobjective is commonly to minimize the one-step prediction errors.
Revised: Moreover, the model in the existing methods is usually used to predict for multiple steps, while themodel learning objective is commonly to minimize the one-step prediction errors.

Original: In other words, there exists anobjective mismatch[Lambert et al., 2020] between model learning and model usage, which meansthat the policy optimization generally expects accurate multi-step trajectories (i.e., model usage), yetthe model is only estimated to consider the accuracy of immediate predictions (i.e., model learning).
Revised: In other words,there exists an objective mismatch between model learning and model usage, which means that thepolicy optimization generally expects accurate multi-step trajectories (i.e., model usage), yet themodel is only estimated to consider the accuracy of immediate predictions (i.e., model learning).

Original: To gain a conceptual understanding of the above two motivations, consider the scenario in Figure 1(a):  The approximate model ˆ P of the true environment dynamics P predicts ˆ P ( s 1 | x 0 ) > 0 , ˆ P ( s ′ 1 | x 0 ) >0 and ˆ P ( s | x 0 )
Revised: To gain a conceptual understanding of the above two motivations, consider the scenario in Figure 1(a): 1) The approximate model ˆ P of the true environment dynamics P predicts ˆ P ( s 1 | x 0 ) >0 , ˆ P ( s ′ 1 | x 0 ) > 0 and ˆ P ( s | x 0 )

Original: = 

Revised: =

Original: , ∀ s / ∈ { s 1 , s ′ 1 } at the state-action pair x 0 ; and 2)
Revised: , ∀ s / ∈ { s 1 , s ′ 1 } at the state-action pair x 0 ; and 2)

Original: The policy is updated from π oldto π new to exploit the regions with high valuespredicted by the value function in the model, andthe resulting distribution shift exposes the model’s inaccuracy at s 1 to the current policy, leading tolarge expected prediction error in the next step, i.e., (cid:80) a ′ π new ( a ′ | s 1 ) D (cid:0) ˆ P ( ·| s 1 , a ′ ) , P ( · | s 1 , a ′ ) (cid:1) >>(cid:80) a ′ π new ( a ′ | s ′ 1 ) D (cid:0) ˆ P ( · | s ′ 1 , a ′ ) , P ( ·| s ′ 1 , a ′ ) (cid:1) , where D ( · , · ) is some distance metrics and the symbol>> means significantly greater than.
Revised: The policy is updated from π old to π new to exploit the regions with high values in the model, and theresulting distribution shift exposes the model’s inaccuracy at s 1 to the current policy, leading tolarge expected prediction error in the next step, i.e., (cid:80) a ′ π new ( a ′ | s 1 ) D (cid:0) ˆ P ( ·| s 1 , a ′ ) , P ( ·| s 1 , a ′ ) (cid:1) >>(cid:80) a ′ π new ( a ′ | s ′ 1 ) D (cid:0) ˆ P ( ·| s ′ 1 , a ′ ) , P ( ·| s ′ 1 , a ′ ) (cid:1) , where D ( · , · ) is some distance metrics and the symbol>> means significantly greater than.

Original: Now starting from x 0 , under the updated policy π new , predictings 1 is likely to direct the trajectory into highly uncertain regions with spurious high values (e.g., thegrey region in Figure 1(b)), and cause considerably larger error in the subsequent trajectory generationthan that of predicting s ′ 1 .
Revised: Now starting from x 0 , under the updated policy π new , predictings 1 is likely to direct the trajectory into highly uncertain regions with spurious high values (e.g., thegrey region in Figure 1(b)), and cause considerably larger error in the subsequent trajectory generationthan that of predicting s ′ 1 .

Original: In contrast, if the model can foresee this uncertainty through explicitconsiderations of the current policy, then it can learn to plan for future predictions and thus jumpto s ′ 1 more often to prevent the trajectory from falling into uncertain regions (e.g., the ant falls intothe right hand side of the wall in Figure 1(b)).
Revised: In contrast, if the model can foresee this uncertainty through explicitconsiderations of the current policy, then it can learn to plan for future predictions and thus jumpto s ′ 1 more often to prevent the trajectory from falling into uncertain regions (e.g., the ant falls intothe right hand side of the wall in Figure 1(b)).

Original: Therefore, to generate trajectories that have smallaccumulation errors, the model should be aware of the update of the policy and change its predicting“strategy” by taking into account the long-term effect of immediate predictions.
Revised: Therefore, to generate trajectories that have smallaccumulation errors, the model should be aware of the update of the policy and change its predicting“strategy” by taking into account the long-term effect of immediate predictions.

Original: Inspired by the above intuition, we propose a new framework for MBRL, named Plan To Predict(P2P).
Revised: Inspired by the above intuition, we propose a new framework for MBRL, named Plan To Predict(P2P).

Original: Our core idea is to treat the model rollout process as a sequential decision making problem andlearn the model by the corresponding algorithms, such as reinforcement learning and model predictivecontrol.
Revised: Our core idea is to treat the model rollout process as a sequential decision making problem andlearn the model by the corresponding algorithms, such as reinforcement learning and model predictivecontrol.

Original: During the learning of the model, the current policy is fixed and serves as the environmentdynamics, while at the same time the model becomes a decision maker and is penalized via theprediction errors at each step.
Revised: During the learning of the model, the current policy is fixed and serves as the environmentdynamics, while at the same time the model becomes a decision maker and is penalized via theprediction errors at each step.

Original: Through actively interacting with the current policy and minimizingthe accumulative prediction errors on the resulting state distribution, the model can quickly adapt to the state distribution shift by considering the long-term effect of the immediate predictions, thusforeseeing the uncertainty of regions that model trajectories are likely to fall into.
Revised: Through actively interacting with the current policy and minimizingthe accumulative prediction errors on the resulting state distribution, the model can quickly adaptto the state distribution shift by considering the long-term effect of the immediate predictions, thus foreseeing the uncertainty of regions that model trajectories are likely to fall into.

Original: Such foresight ofuncertainty enables the model to plan to predict for multiple steps, which improves the quality ofthe pseudo samples generated by the model and eventually benefits policy learning.
Revised: Such foresight ofuncertainty enables the model to plan to predict for multiple steps, which improves the quality ofthe pseudo samples generated by the model and eventually benefits policy learning.

Original: Note that thetypical treatment of minimizing one-step prediction errors regardless of the current policy is in facta greedy optimization process, and thus can be viewed as a special case of P2P. Some prior works,such as [Nagabandi et al., 2018, Luo et al., 2019], adopt similar multi-step prediction loss to learn themodel, yet these works simply compute the loss on trajectories collected by previous policies and canstill suffer from severe distribution shift due to the lack of considerations of the policy update.
Revised: Note that thetypical treatment of minimizing one-step prediction errors regardless of the current policy is in facta greedy optimization process, and thus can be viewed as a special case of P2P. Some prior works,such as [Nagabandi et al., 2018, Luo et al., 2018], adopt similar multi-step prediction loss to learn themodel, yet these works simply compute the loss on trajectories collected by previous policies and canstill suffer from severe distribution shift due to the lack of considerations of the policy update.

Original: Theoretically, we prove that P2P approximately optimizes a lower bound of the true environmentreturn.
Revised: Theoretically, we prove that P2P approximately optimizes a lower bound of the true environmentreturn.

Original: Different from prior works [Luo et al., 2019, Janner et al., 2019], our theoretical analysis doesnot scale up the step-wise prediction errors to their maximum over timesteps when considering theimpact of model error on the return approximation, thus resulting in a tighter bound of the true returnand also justifying the objective mismatch issue.
Revised: Different from prior works [Luo et al., 2018, Janner et al., 2019], our theoretical analysis doesnot scale up the step-wise prediction errors to their maximum over timesteps when considering theimpact of model error on the return approximation, thus resulting in a tighter bound of the true returnand also justifying the objective mismatch issue.

Original: Optimizing this bound not only provides a strongerguarantee for policy improvement, but also enables the model to quickly adapt to the current policy.
Revised: Optimizing this bound not only provides a strongerguarantee for policy improvement, but also enables the model to quickly adapt to the current policy.

Original: Empirical results demonstrate that P2P outperforms the existing state-of-the-art methods on severalchallenging benchmark tasks.
Revised: Empirical results demonstrate that P2P outperforms the existing state-of-the-art methods on severalchallenging benchmark tasks.

Original: It also deserves to note that, due to the capability to avoid modelerror, P2P may be particularly suited for offline learning tasks, where incremental data collectionis not allowed and thus the model error is harder to be corrected than in online settings, inducingan irreversible bias on policy learning.
Revised: It also deserves to note that, due to the capability to avoid modelerror, P2P may be particularly suited for offline learning tasks, where incremental data collectionis not allowed and thus the model error is harder to be corrected than in online settings, inducinga irreversible bias on policy learning.

Original: Therefore, we also provide an evaluation of P2P in offlinelearning settings in the experiments, and present an interpretative visualization on how the learningobjective of P2P affects the model predictions.
Revised: Therefore, we also provide an evaluation of P2P in offlinelearning settings in the experiments, and present an interpretative visualization on how the learningobjective of P2P affects the model predictions.

Original:  Preliminaries and Notations
Revised:  Preliminaries and Notations

Original: A discrete time Markov desision process (MDP) M is defined by tuple ( S , A , R, P, ρ 0 , γ ) , where S isthe state space, A the action space, R ( s, a ) a scalar reward function of s ∈ S and a ∈ A , P ( s ′ | s, a ) ∈[0 , 1] the transition dynamics which is assumed to be unknown, γ ∈ [0 , 1) the discount factor andρ 

Revised: A discrete time Markov desision process (MDP) M is defined by tuple ( S , A , R, P, ρ 0 , γ ) , where S isthe state space, A the action space, R ( s, a ) a scalar reward function of s ∈ S and a ∈ A , P ( s ′ | s, a ) ∈[0 , 1] the transition dynamics which is assumed to be unknown, γ ∈ [0 , 1) the discount factor andρ

Original: the initial state distribution.
Revised: the initial state distribution.

Original: A stochastic policy π ( a | s ) maps states to a probability distributionover the action space.
Revised: A stochastic policy π ( a | s ) maps states to a probability distributionover the action space.

Original: The expected return of π defined by J ( π ) :
Revised: The expected return of π defined by J ( π ) :

Original: = E π,P [ (cid:80) ∞ t =

Revised: = E π,P [ (cid:80) ∞ t =

Original: γ t R ( s t , a t )] isthe expectation of the sum of discounted rewards induced by π .
Revised: γ t R ( s t , a t )] isthe expectation of the sum of discounted rewards induced by π .

Original: Following the general setting ofreinforcement learning, the goal is to search for an optimal policy that maximizes the expected returnfrom the environment, i.e., arg max π J ( π ) .
Revised: Following the general setting ofreinforcement learning, the goal is to search for an optimal policy that maximizes the expected returnfrom the environment, i.e., arg max π J ( π ) .

Original: MBRL methods learn a model ˆ P to approximate the unknown dynamics P , and then use this model toaid policy learning.
Revised: MBRL methods learn a model ˆ P to approximate the unknown dynamics P , and then use this model toaid policy learning.

Original: While the model can be utilized in various ways, this paper focuses on the usage ofgenerating pseudo samples to enrich the dataset, which is one of the most common usages of the model.
Revised: While the model can be utilized in various ways, this paper focuses on the usage ofgenerating pseudo samples to enrich the dataset, which is one of the most common usages of the model.

Original: The expected return of π predicted by the model ˆ P is denoted as J ˆ P ( π ) :
Revised: The expected return of π predicted by the model ˆ P is denoted as J ˆ P ( π ) :

Original: = E π, ˆ P [ (cid:80) ∞ t [-=
-]
Revised: = E π, ˆ P (cid:104)(cid:80) ∞ t [-=
-]=0 γ t

Original: γ t R ( s t , a t )] . In our analysis, thereward function is assumed to be available, 1 and the policy obtained in the lastiteration is denoted as the data-collecting policy π D ,
Revised: ˆ R ( s t , a t ) (cid:105) ,where ˆ R approximates the true reward function. In our analysis, the policy obtained in the last iterationis denoted as the data-collecting policy π D .

Original:  The Plan To Predict (P2P) Framework
Revised:  The Plan To Predict (P2P) Framework

Original: In this section, we present a detailed description of our proposed algorithmic framework for MBRL,i.e., Plan To Predict (P2P).
Revised: In this section, we present a detailed description of our proposed algorithmic framework for MBRL,i.e., Plan To Predict (P2P).

Original: In Section 3.1, the model rollout process is reformulated as a sequentialdecision making problem using the notion of MDP and then a meta-algorithm of the P2P frameworkis provided as a generic solution to this reformulated problem.
Revised: In Section 3.1, the model rollout process is reformulated as a sequentialdecision making problem using the notion of MDP and then a meta-algorithm of the P2P frameworkis provided as a generic solution to this reformulated problem.

Original: In Section 3.2, theoretical results areprovided to guarantee the performance of P2P. In Section 3.3, two practical algorithms are developedunder the P2P framework by handling some practical issues.
Revised: In Section 3.2, theoretical results areprovided to guarantee the performance of P2P. In Section 3.3, two practical algorithms are developedunder the P2P framework by handling some practical issues.

Original: 3.1 The Overall Framework
Revised: 3.1 The Overall Framework

Original: We now define the model MDP to reformulate the model rollout problem and outline our overallalgorithm design.
Revised: We now define the model MDP to reformulate the model rollout problem and outline our overallalgorithm design.

Original: Definition 1.
Revised: Definition 1.

Original: The model MDP M π is defined by tuple ( S m , A m , R m , P m , γ, ρ m 0 ) , where attimestep t , s mt :=
Revised: The model MDP M π is defined by tuple ( S m , A m , R m , P m , γ, ρ m 0 ) , where attimestep t , s mt :=

Original: ( s t , a t ) ∈ S m , a mt :=
Revised: ( s t , a t ) ∈ S m , a mt :=

Original:  s t +1  ∈
Revised: ( r t +1 , s t +1 ) ∈

Original: A m , R m ( s mt , a mt ) :
Revised: A m , R m ( s mt , a mt ) :

Original: = − (cid:0) ˆ P ( s t +1 | s t , a t ) −
Revised: = − (cid:0) ˆ P ( s t +1 | s t , a t ) −

Original: P ( s t +1 | s t , a t ) (cid:1) 2 , P m ( s mt +

Revised: P ( s t +1 | s t , a t ) (cid:1) 2 , P m ( s mt +

Original: | s mt , a m t ) :
Revised: | s mt , a mt ) :

Original: = π ( a t +1 | s t +1 )
Revised: = π ( a t +1 | s t +1 )

Original: ˆ P ( s t +1 | s t , a t ) , γ is the discount factor ,ρ m 0 :
Revised: ˆ P ( s t +1 | s t , a t ) , γ is the discount factor ,ρ m 0 :

Original: =
Revised: =

Original: π ( a 0 | s 0 )
Revised: π ( a 0 | s 0 )

Original: ρ 0 , and ρ 0 is the initial state distribution of the true environment.
Revised: ρ 0 , and ρ 0 is the initial state distribution of the true environment.

Original: In the model learning process of P2P, the roles played by the policy π and the model ˆ P are reversed:π is fixed to serve as the environment dynamics and ˆ P , now as the decision maker, interacts withπ and is updated to minimize the accumulative prediction error along the generated trajectories.
Revised: In the model learning process of P2P, the roles played by the policy π and the model ˆ P are reversed:π is fixed to serve as the environment dynamics and ˆ P , now as the decision maker, interacts withπ and is updated to minimize the accumulative prediction error along the generated trajectories.

Original: Similar to the optimization of the policy, the objective of this learning process can be written asargmax
Revised: Similar to the optimization of the policy, the objective of this learning process can be written asarg

Original: ˆ P J π (cid:0) ˆ P (cid:1) where J π (cid:0) ˆ P (cid:1) :
Revised: [Equation] J π [Equation] ˆ P (cid:1) where J π (cid:0) ˆ P (cid:1) :

Original: = E [ (cid:80) t γ t R m ( s mt , a mt )] .
Revised: = E [ (cid:80) t γ t R m ( s mt , a mt )] .

Original: The overall algorithmic framework of
Revised: The overall algorithmic framework of

Original: P2P is depicted in Algorithm 1, where the main difference compared to most existing MBRL methodslies in Line 5, i.e., the learning of the model.
Revised: P2P is depicted in Algorithm 1.

Original: Specifically, this process can be achieved in a waysimilar to policy learning, i.e., generating samples by actively interacting with the policy (nowviewed as the background environment), and optimizing the expected return J π (cid:0) ˆ P (cid:1) accordingly.
Revised: Remarks: 1)

Original: This optimization  can in principle be done by any approaches solving sequential decision makingproblems.
Revised: The optimization of ˆ P (Line 5 in Algorithm 1) can in principle be done by anyapproaches solving sequential decision making problems.

Original: As two representatives of these approaches, Model Predictive Control (MPC)
Revised: As two representatives of these approaches, Model Predictive Control (MPC)

Original: [Camachoand Alba, 2013] and Reinforcement Learning (RL)
Revised: [Camacho and Alba, 2013] and Reinforcement Learning (RL)

Original: [Sutton and Barto, 2018] are both applied inour practical version of P2P, which are detailed in Section 3.3. In addition, one of thebiggest gapsbetween this theoretical construction of the model MDP and the practical implementation may liein the derivation of R m .
Revised: [Sutton and Barto, 2018] are both applied in our practical version of P2P, which are detailed in Section 3.3.2) During the construction of the model MDP , the true environment dynamics is required to computethe reward function, i.e., R m .

Original: According to Definition 1, the true environment dynamics is required tocompute this reward function, while in practice the true dynamics is not accessible in the modellearning phase. Therefore, depending on which underlying approach (i.e., MPC or RL) is chosen formodel optimization, we propose twodifferent alternative plans to approximate R m in Section 3.3.
Revised: While in practice the true dynamics is not accessible in the modellearning phase, we propose two alternative plans to approximate R m in Section 3.3.

Original: 3.2 Theoretical Results
Revised: 3.2 Theoretical Results

Original: In general, MBRL methods learn an approximate model of the real world dynamics and learn thepolicy by maximizing the expected return predicted by the model.
Revised: In general, MBRL methods learn an approximate model of the real world dynamics and learn thepolicy by maximizing the expected return predicted by the model.

Original: The better this pseudo returnapproximates the true environment return, the stronger guarantee the model can provide for policyimprovement.
Revised: The better this pseudo returnapproximates the true environment return, the stronger guarantee the model can provide for policyimprovement.

Original: Thus, a critical problem in theory is how to bound the gap between the expected returnof the model and the true environment, i.e., (cid:12)(cid:12)
Revised: Thus, a critical problem in theory is how to bound the gap between the expected returnof the model and the true environment, i.e., (cid:12)(cid:12)

Original: J ˆ P ( π ) − J ( π ) (cid:12)(cid:12) .
Revised: J ˆ P ( π ) − J ( π ) (cid:12)(cid:12) .

Original: Suppose that this gap can be upperbounded by C , and the policy is updated from π D to π with the expected model return improved by2 C , then the policy improvement in terms of the true environment can be guaranteed:
Revised: Suppose that this gap can be upperbounded by C , and the policy is updated from π D to π with the expected model return improved by2 C , then the policy improvement in terms of the true environment can be guaranteed:

Original: [Equation]
Revised: [Equation]

Original: Our main theoretical result is the following theorem:
Revised: Our main theoretical result is the following theorem:

Original: Theorem 1.
Revised: Theorem 1.

Original: The gap between the expected return of the model and the environment is bounded as:
Revised: The gap between the expected return of the model and the environment is bounded as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: (cid:32)
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: γ

Original: [Equation]
Revised: [Equation]

Original:  ∞ (cid:88) t
Revised: ) ∞ (cid:88) t

Original: = γ t ϵ mt + ∞ (cid:88) t = γ t ϵ mt −(cid:33) , (1) where ϵ π := max s D TV ( π D ( ·| s ) 
Revised: = γ t [Equation] + ∞ (cid:88) t = γ t ϵ mt −(cid:33) , (1) where ϵ π := max s D TV ( π D ( ·| s ) ∥

Original: ∥ π ( ·| s )) denotes the policy distribution shift, ϵ mt :
Revised: π ( ·| s )) denotes the policy distribution shift, ϵ mt :

Original: = 147 Es ∼ ˆ P t − 1 ( s ; π ) (cid:2)max a D TV ( ˆ P ( ·| s, a ) ∥ P ( ·| s, a )) (cid:3) denotes the upper bound of one-step model prediction error at timestep t of the model rollout trajectory, D TV ( p ∥ q ) refers to the total variationbetween distribution p and q , R max := max s,a R ( s, a ) , and ˆ P t − 1 ( s ; π ) denotes the state distributionat t under ˆ P and π .
Revised: = 142 E (cid:2)

Original: Proof.
Revised: Proof.

Original: Please refer to Appendix A.
Revised: Please refer to Appendix A.

Original: Transforming the inequality in Eq.
Revised: Transforming the inequality in Eq.

Original: (1) results in J ( π ) ≥ J ˆ P ( π )
Revised: (1) results in J ( π ) ≥ J ˆ P ( π )

Original: − 2 C ( ˆ P, π ) , where C ( ˆ P, π ) denotesthe right hand side in Eq.
Revised: − 2 C ( ˆ P, π ) , where C ( ˆ P, π ) denotesthe right hand side in Eq.

Original: Thus in principle, monotonic policy improvement can be achieved byiteratively applying the following update rule:
Revised: Thus in principle, monotonic policy improvement can be achieved byiteratively applying the following update rule:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The update rule in Eq.
Revised: The update rule in Eq.

Original: (2) is ideal and usually impractical since it involves an exhaustive search in thestate and action space to compute C , and requires full-horizon rollouts in the model for estimatingthe accumulative model errors.
Revised: (2) is ideal and usually impractical since it involves an exhaustive search in thestate and action space to compute C , and requires full-horizon rollouts in the model for estimatingthe accumulative model errors.

Original: Thus, similar to how algorithms like TRPO [Schulman et al.
Revised: Thus, similar to how algorithms like TRPO [Schulman et al.

Original: , 2015]approximate their theoretically monotonic version, P2P approximates this update rule by maximizingthe expected model return while keeping the accumulative model error small.
Revised: , 2015]approximate their theoretically monotonic version, P2P approximates this update rule by maximizingthe expected model return while keeping the accumulative model error small.

Original: As for the policy shiftterm ϵ π , though the bound suggests that this term may also be constrained, we found empirically thatit is sufficient to only control the model error.
Revised: As for the policy shiftterm ϵ π , though the bound suggests that this term may also be constrained, we found empirically thatit is sufficient to only control the model error.

Original: This may be explained by the relatively small scale ofpolicy shift with respect to model error, as observed in [Janner et al., 2019].
Revised: This may be explained by the relatively small scale ofpolicy shift with respect to model error, as observed in [Janner et al., 2019].

Original: It is worth noting that the accumulative error here is not simply summing up the multi-step errors ontrajectories collected by any other policies as in [Nagabandi et al., 2018, Luo et al., 2019], becausethe definition of the step-wise model error (i.e., ϵ mt ) in Theorem 1 involves a policy-dependentexpectation over the state space, indicating that this error is also affected by the update of the policy.
Revised: It is worth noting that the accumulative error here is not simply summing up the multi-step errors ontrajectories collected by any other policies as in [Nagabandi et al., 2018, Luo et al., 2018], becausethe definition of the step-wise model error (i.e., ϵ mt ) in Theorem 1 involves a policy-dependentexpectation over the state space, indicating that this error is also affected by the update of the policy.

Original: To enable quick adaptation to this policy update, P2P minimizes the accumulative error on the statedistribution induced by an active interaction between the model and the current policy, and thus betterapproximates the accumulative error defined in Theorem 1.
Revised: To enable quick adaptation to this policy update, P2P minimizes the accumulative error on the statedistribution induced by an active interaction between the model and the current policy, and thus betterapproximates the accumulative error defined in Theorem 1.

Original: Different from most of prior work [Luo et al., 2019, Janner et al., 2019], in Theorem 1 the stepwise model prediction errors are not scaled up to their maximum over timesteps (i.e., max t ϵ m t ),leading to a tighter error bound.
Revised: Different from most of prior work [Luo et al., 2018, Janner et al., 2019], in Theorem 1 the stepwise model prediction errors are not scaled up to their maximum over timesteps (i.e., max t ϵ mt ),leading to a tighter error bound.

Original: As a result, this bound not only provides stronger guarantee forpolicy improvement, but also justifies the objective mismatch issue between the model learningand model usage from a theoretical perspective: Scaling up these error terms by such a potentiallylarge magnitude yields a loose bound which leads to algorithms aiming at minimizing the upperbound of individual one-step prediction losses.
Revised: As a result, this bound not only provides stronger guarantee forpolicy improvement, but also justifies the objective mismatch issue between the model learningand model usage from a theoretical perspective: Scaling up these error terms by such a potentiallylarge magnitude yields a loose bound which leads to algorithms aiming at minimizing the upperbound of individual one-step prediction losses.

Original: However, to obtain better policy improvement, thetighter bound in Eq.
Revised: However, to obtain better policy improvement, thetighter bound in Eq.

Original: (1) implies that a more fine-grained model learning process is desired, whichconsiders consecutive rollout steps and minimizes the accumulative prediction error, i.e., (cid:80) t γ t ϵ mt .
Revised: (1) implies that a more fine-grained model learning process is desired, whichconsiders consecutive rollout steps and minimizes the accumulative prediction error, i.e., (cid:80) t γ t ϵ mt .

Original: Using the notion of model MDP defined in Section 3.1, the model induced by minimizing one-stepprediction errors can be intuitively interpreted as a greedy decision maker, which is often consideredshortsighted and may easily lead to severe suboptimality.
Revised: Using the notion of model MDP defined in Section 3.1, the model induced by minimizing one-stepprediction errors can be intuitively interpreted as a greedy decision maker, which is often consideredshortsighted and may easily lead to severe suboptimality.

Original: 3.3 Practical Implementation
Revised: 3.3 Practical Implementation

Original: In this subsection, we instantiate Algorithm 1 by specifying explicit approaches to learn the modeland provides solutions to some practical issues.
Revised: In this subsection, we instantiate Algorithm 1 by specifying explicit approaches to learn the modeland provides solutions to some practical issues.

Original: [Figure] Hopper [Figure] HalfCheetah
Revised: For brevity, when mentioning the model learningprocess in this subsection, we rename the policy π in the original MDP as the policy-dynamics andthe approximate model ˆ P in the original MDP as the model-policy .

Original: [Figure] P2P-MPC: Sincein our problem formulation the “dynamics” of the model rollout process, i.e.,the current policy π, is accessible, one of the simplest yet effective ways to learn the model canbe the model predictive control (MPC)
Revised: P2P-MPC: Since the policy-dynamics π is accessible, one of the simplest yet effective ways to learnthe model-policy can be the model predictive control (MPC)

Original: [Camacho and Alba, 2013], which utilizes the dynamics toplan and optimize for a sequence of actions.
Revised: [Camacho and Alba, 2013], which utilizesthe dynamics to plan and optimize for a sequence of actions.

Original: Given the state s mt at step t , the MPC controller firstoptimizes the sequence of actions a mt : t
Revised: Given the state s mt at step t , the MPCcontroller first optimizes the sequence of actions a mt : t

Original: + H over a finite horizon H , and then employs the first action ofthe optimal action sequence, i.e., a mH,t := arg max a mt : t + H E ˆ P (cid:80) t
Revised: + H over a finite horizon H , and then employs thefirst action of the optimal action sequence, i.e., a mH,t := arg max a mt : t + H E ˆ P (cid:80) t

Original: + H − 1 t ′ =
Revised: + H − 1 t ′ =

Original: t
Revised: t

Original: R m ( s mt ′ , a mt ′ ) .
Revised: R m ( s mt ′ , a mt ′ ) .

Original: Computing theexact arg max is computationally challenging due to the dimension of the state and action spaces,so we adopt the random-sampling shooting method [Rao, 2009] which generates random actionsequences, executes them respectively, and chooses the one with highest return predicted by thedynamics.
Revised: Computing the exact arg max is computationally challenging due to the dimension of the state andaction spaces, so we adopt the random-sampling shooting method [Rao, 2009] which generatesrandom action sequences, executes them respectively, and chooses the one with highest returnpredicted by the dynamics.

Original: Besides, the reward R m , which cannot be directly computed by definition, is approximatedhere using a neural network and trained on the environment dataset. 
Revised: Besides, the reward R m , which cannot be directly computed by definition,is approximated here using a neural network and trained on the environment dataset. [Figure] Hopper [Figure]

Revised: HalfCheetah

Original:  P2P-RL: It is well-known that though effective, RL methods often suffer from high samplecomplexity.
Revised: [Figure] P2P-RL: It is well-known that though effective, RL methods often suffer from high samplecomplexity.

Original: Hence, the model is trained on the environment dataset instead of the samples generatedby the interaction of the model and policy.
Revised: Hence, the model-policy is trained on the environment dataset instead of the samplesgenerated by the interaction of the model-policy and policy-dynamics.

Original: However, there are two issues to be addressed: 1) As the“environment dynamics”, the policy is highly non-stationary due to its continuing update; and 2) Thedecision maker which generates the dataset is the true environment rather than our model.
Revised: However, there are two issuesto be addressed: 1) The policy-dynamics is highly non-stationary due to its continuing update; and2) The decision maker which generates the dataset is the true environment rather than our modelpolicy.

Original: The firstissue can be addressed by updating the next state in each sampled transitions by applying the currentpolicy, i.e., s mt +1 ← ( s t +1 , π ( s t +1 )) .
Revised: The first issue can be addressed by updating the next state in each sampled transitions byapplying the current policy-dynamics, i.e., s mt +1 ← ( s t +1 , π ( s t +1 )) .

Original: As for the second issue, since the model-generated samplesare not used in model learning, it can be seen approximately as an offline learning problem and thuscan be addressed by well-developed offline RL methods [Fujimoto et al., 2019, Levine et al., 2020].
Revised: As for the second issue, sincethe model-generated samples are not used in model learning, it can be seen approximately as anoffline learning problem and thus can be addressed by well-developed offline RL methods [Fujimotoet al., 2019, Levine et al., 2020].

Original: Due to the effectiveness and the implementational simplicity, we adopt SAC [Haarnoja et al. , 2018]with behavior cloning as the underlying learning algorithm, which bears similarities to the TD3+BCapproach proposed by [Fujimoto and Gu, 2021].
Revised: Due to the effectiveness and the implementational simplicity, weadopt SAC [Haarnoja et al., 2018] with behavior cloning as the underlying learning algorithm, whichbears similarities to the TD3+BC approach proposed by [Fujimoto and Gu, 2021].

Original: Besides, an approach called DualDICE [Nachumet al., 2019] is also applied to correct the estimation of the state distribution.
Revised: Besides, anapproach called DualDICE [Nachum et al., 2019] is also applied to correct the estimation of thestate distribution.

Original: Finally, since the samplescome from the true environment, the reward R m can be simply approximated by re-predicting thedynamics in the sampled transitions and computing the prediction errors for the current model.
Revised: Finally, since the samples come from the true environment, the reward R m can besimply approximated by re-predicting the dynamics in the sampled transitions and computing theprediction errors for the current model-policy.

Original: For the detailed description of how to approximate R m , please refer to Appendix B.3.

Original: Besides, thepseudo code of P2P-MPC and P2P-RL can be found in Appendix E.

Original:  Experiments
Revised:  Experiments

Original: In this section, we first evaluate the empirical performance of P2P on a set of MuJoCo continuouscontrol tasks [Todorov et al., 2012] in Section 4.1, and then present an quantitative analysis for 8 Rollout [Figure] A cc u m u l a t
Revised: In this section, we first evaluate the empirical performance of P2P on a set of MuJoCo continuouscontrol tasks [Todorov et al., 2012] in Section 4.1, and then present an quantitative analysis forthe model error in Section 4.2.

Original: i v e E

Original: rr

Original: o r

Original: [-Hopper       8 Rollout Steps
-]
Revised: [-Hopper       8 Rollout Steps
-]

Original: A cc u m u l a t

Original: i v e E

Original: rr

Original: o r

Original: Ant

Original: P2P-MPC

Original: P2P-RL

Original: [-MBPO 1 2 3 4 5 6 7 8 9 Rollout Steps
-]
Revised: [-MBPO 1 2 3 4 5 6 7 8 9 Rollout Steps
-]

Original: A cc u m u l a t

Original: i v e E

Original: rr o r Humanoid the model error in Section 4.2. The ablation study provided in Section 4.3 verifies the necessity ofan active policy-model interaction when computing the multi-step prediction loss.
Revised: The ablation study provided in Section 4.3 verifies the necessity ofan active policy-model interaction when computing the multi-step prediction loss.

Original: Besides, a keyparameter of P2P-MPC is also studied in Section 4.3.
Revised: Besides, a keyparameter of P2P-MPC is also studied in Section 4.3.

Original: Finally, to gain a deeper understanding of how
Revised: Finally, to gain a deeper understanding of how

Original: P2P work in offline settings, a visualization of the model’s rollout trajectories is shown in Section 4.4.
Revised: P2P work in offline settings, a visualization of the model’s rollout trajectories is shown in Section 4.4.

Original: 4.1 Comparisons with Baselines
Revised: 4.1 Comparisons with Baselines

Original: We compare P2P against some state-of-the-art model-free and model-based methods.
Revised: We compare P2P against some state-of-the-art model-free and model-based methods.

Original: The model-freebaseline is SAC [Haarnoja et al., 2018], which is the model-free counterpart of P2P.
Revised: The model-freebaseline is SAC [Haarnoja et al., 2018], which is the model-free counterpart of P2P.

Original: The model-basedbaselines include MBPO [Janner et al., 2019], which uses short-horizon rollouts branched from thestate distribution of previous policies,SLBO [Luo et al., 2019], which enjoys theoretical performanceguarantee and uses model rollouts from the initial state distribution, and STEVE [Buckman et al.,2018], which also generates short rollouts but uses model data for estimating target values insteadof policy learning.
Revised: The model-basedbaselines include MBPO [Janner et al., 2019], which uses short-horizon rollouts branched from thestate distribution of previous policies, and STEVE [Buckman et al., 2018], which also generates shortrollouts but uses model data for estimating target values instead of policy learning.

Original: Note that our framework only focuses on model learning and can be employedas a flexible plug-in component by most MBRL algorithms that generate pseudo samples to assistpolicy learning.
Revised: Note that ourframework only focuses on model learning and can be employed as a flexible plug-in component bymost MBRL algorithms that generate pseudo samples to assist policy learning.

Original: In our experiments,  we plug the model learning process of P2P into MBPO since itis a widely accepted strong baseline in MBRL.
Revised: In our experiments, [Figure] [Figure] we plug the model learning process of P2P into MBPO since it is a widely accepted strong baselinein MBRL.

Original: The implementational details and the hyperparametersettings of P2P, as well as the environment settings, are all described in Appendix B.
Revised: The implementational details and the hyperparameter settings of P2P, as well as theenvironment settings, are all described in Appendix B

Original:  As shown in Figure 2, P2P-MPC significantly outperforms the baselines on several tasks, especiallyon Hopper, Humanoid and Ant, which are generally considered to be the most challenging MuJoCotasks.
Revised: [Figure] As shown in Figure [Figure] P2P-MPC significantly outperforms the baselines on several tasks, especiallyon Hopper, Humanoid and Ant, which are generally considered to be the most challenging MuJoCotasks.

Original: In Walker2d, the best rollout length is one step in our implementation and MBPO, thus resultingin similar performance between these two methods.
Revised: In Walker2d, the best rollout length is one step in our implementation and MBPO, thus resultingin similar performance between these two methods.

Original: Compared to P2P-MPC, the performance of
Revised: Compared to P2P-MPC, the performance of

Original: P2P-RL is less stable.
Revised: P2P-RL is less stable.

Original: Through an investigation on the model learning process (see Appendix C),we find that P2P-RL sometimes struggles to balance the loss of behavior cloning and RL, leading tothe difficulty in hyperparameter tuning and the instability of learning network parameters, while theoptimization of P2P-MPC is non-parametric and thus does not suffer from this issue.
Revised: Through an investigation on the model learning process (see Appendix C),we find that P2P-RL sometimes struggles to balance the loss of behavior cloning and RL, leading tothe difficulty in hyperparameter tuning and the instability of learning network parameters, while theoptimization of P2P-MPC is non-parametric and thus does not suffer from this issue.

Original: 4.2 Model Error Analysis
Revised: 4.2 Model Error Analysis

Original: The theoretical result presented in Section 3.2 implies that to guarantee better policy improvement, themodel should learn to minimize the accumulative errors along the trajectories induced by the currentpolicy and the model.
Revised: The theoretical result presented in Section 3.2 implies that to guarantee better policy improvement, themodel should learn to minimize the accumulative errors along the trajectories induced by the currentpolicy and the model.

Original: Since P2P is designed based on this principle, the empirical results in Figure 2verify the theoretical result in terms of the final performance.
Revised: Since P2P is designed based on this principle, the empirical results in Figure 2verify the theoretical result in terms of the final performance.

Original: To further verify the effectiveness ofthis algorithmic design, we study the accumulative prediction error on three tasks that require rolloutswith relative long horizons, i.e., Hopper, Ant and Humanoid.
Revised: To further verify the effectiveness ofthis algorithmic design, we study the accumulative prediction error on three tasks that require rolloutswith relative long horizons, i.e., Hopper, Ant and Humanoid.

Original: The results shown in Figure 3 togetherwith Figure 2 strongly support our theoretical claims by demonstrating that the algorithm inducingless accumulative error achieves better performance.
Revised: The results shown in Figure 3 togetherwith Figure 2 strongly support our theoretical claims by demonstrating that the algorithm inducingless accumulative error achieves better performance.

Original: Besides, we can also observe that in Hopperthe model errors of P2P-MPC in the first few steps are slightly larger than the other two methods.
Revised: Besides, we can also observe that in Hopperthe model errors of P2P-MPC in the first few steps are slightly larger than the other two methods.

Original: Nevertheless, as the rollout trajectory goes longer, the accumulative error induced by P2P-MPCgrows significantly slower than the other methods.
Revised: Nevertheless, as the rollout trajectory goes longer, the accumulative error induced by P2P-MPCgrows significantly slower than the other methods.

Original: This further agrees with the intuition mentionedin Figure 1 by showing that P2P is able to trade the one-step model error for accumulative modelerror by considering the long-term effect of the immediate prediction.
Revised: This further agrees with the intuition mentionedin Figure 1 by showing that P2P is able to trade the one-step model error for accumulative modelerror by considering the long-term effect of the immediate prediction.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Revised: Multi-Step Prediction Loss

Revised: In Section 3.2, we emphasize that the loss of P2P is essentiallydifferent from the multi-step prediction loss used in prior work, since P2P computes the multi-steploss on the state distribution induced from an active interaction between the model and the currentpolicy, and on the contrary, prior work computes this loss on the trajectories collected by previous

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e
Revised: A v e r a g e

Original: R e t u r n
Revised: R e t u r n

Original: Ant
Revised: Ant

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: Multi-Step Prediction Loss
Revised: MPC MOPO [Figure] [Figure]

Original: In Section 3.2, we emphasize that the loss of P2P is essentiallydifferent from the multi-step prediction loss used in prior work, since P2P computes the multi-steploss on the state distribution induced from an active interaction between the model and the currentpolicy, and on the contrary, prior work computes this loss on the trajectories collected by previouspolicies.
Revised: v 230 . 2 ± [Figure] 174 . 2 ±

Revised: [Figure]

Revised: [Figure]

Revised: large

Revised: [Figure]

Revised: 1 ±

Revised: [Figure]

Revised: ±

Original:  The empirical results in Figure 4 compare the performances induced by these two kind oflosses and highlight the importance of the interactive model learning process in P2P.
Revised: [Figure] policies. The empirical results in Figure 4 compare the performances induced by these two kind oflosses and highlight the importance of the interactive model learning process in P2P.

Original: Planning Horizon Different from most RL methods, MPC methods can assign an explicit horizonfor planning, hence P2P-MPC is able to set the planning horizon of the model to be exactly the samewith the rollout horizon.
Revised: Planning Horizon Different from most RL methods, MPC methods can assign an explicit horizonfor planning, hence P2P-MPC is able to set the planning horizon of the model to be exactly the samewith the rollout horizon.

Original: This difference may partially explain the superior performance of P2P-MPCagainst P2P-RL.
Revised: This difference may partially explain the superior performance of P2P-MPCagainst P2P-RL.

Original: Nevertheless, longer planning horizon incurs higher computational cost, especiallyin tasks with high-dimensional state spaces.
Revised: Nevertheless, longer planning horizon incurs higher computational cost, especiallyin tasks with high-dimensional state spaces.

Original: To trade-off the performance and the computationalcost, our solution is to simply limit the planning horizon, and the empirical results shown in Figure 5suggests that a horizon of 6 is satisfactory for the tasks experimented.
Revised: To trade-off the performance and the computationalcost, our solution is to simply limit the planning horizon, and the empirical results shown in Figure 5suggests that a horizon of 6 is satisfactory for the tasks experimented.

Original: 4.4 An Interpretative Experiment
Revised: 4.4 An Interpretative Experiment

Original: To gain an intuitive understanding of how the learning objective of P2P affects the model predictions,here we present a visualization of the rollout trajectories in maze2d-medium-v1, an offline learning[Figure] task in the D4RL benchmark [Fu et al., 2020].
Revised: To gain an intuitive understanding of how the learning objective of P2P affects the model predictions,here we present a visualization of the rollout trajectories in maze2d-medium-v1, an offline learning task in the D4RL benchmark [Fu et al., 2020].

Original: As shown in Figure 6, the environment is slightly modified by lengthening the wall at the middle of the mazeand fixing the goal to the top right corner.
Revised: As shown in Figure 6, the environment is slightly modified by lengthening the wall at the middle of the mazeand fixing the goal to the top right corner.

Original: Besides, the number of offline samplesthat involve the grey region are partially discarded to increase this region’s uncertainty.
Revised: Besides, the number of offline samplesthat involve the grey region are partially discarded to increase this region’s uncertainty.

Original: Figure 6 shows the top four trajectories that are most frequently generatedfrom a fixed starting point using our methods and MOPO
Revised: Figure 6 shows the top four trajectories that are most frequently generatedfrom a fixed starting point using our methods and MOPO

Original: [Yu et al., 2020], awell-known offline model-based RL method that uses one-step error for modellearning.
Revised: [Yu et al., 2020], awell-known offline model-based RL method that uses one-step error for modellearning.

Original: The visualization clearly demonstrates the different rollout preferencesof the two kinds of models: the model of P2P-MPC tends to generate trajectoriesheading to the regions with low uncertainty, while the model of MOPO fails toprevent the trajectories from being absorbed into the highly uncertain region.The [Figure] [Figure]
Revised: The visualization clearly demonstrates the different rollout preferencesof the two kinds of models: the model of P2P-MPC tends to generate trajectoriesheading to the regions with low uncertainty, while the model of MOPO fails toprevent the trajectories from being absorbed into the highly uncertain region.

Original: [Figure]

Original: [Figure]

Original: u

Original: [Figure]

Original: l a t i

Original: [Figure]

Original: [Figure]

Original: P2P-MPC

Original: [Table]

Original: resulting accumulative model errors are quantified in Figure 7, and as shown in
Revised: Theresulting accumulative model errors are quantified in Figure 7, and as shown in

Original: Table 1, the final performance of the induced policy of P2P-MPC is significantlybetter than MOPO.
Revised: Table 1, the final performance of the induced policy of P2P-MPC is significantlybetter than MOPO.

Original: Note that in online settings, the model error is possible to becorrected by the newly collected data from the environment, while in offline settings this incrementaldata-collecting process is not allowed.
Revised: Note that in online settings, the model error is possible to becorrected by the newly collected data from the environment, while in offline settings this incrementaldata-collecting process is not allowed.

Original: Thus, the model error can have a more detrimental effect onpolicy learning, which may severely degrade the asymptotic performance as evidenced in Table 1.
Revised: Thus, the model error can have a more detrimental effect onpolicy learning, which may severely degrade the asymptotic performance as evidenced in Table 1.

Original: Additional results on a much harder task (i.e., maze2d-large-v1) are also provided in Table 1 to furtherverifies the effectiveness of P2P-MPC.  
Revised: Additional results on a much harder task (i.e., maze2d-large-v1) are also provided in Table 1 to furtherverifies the effectiveness of P2P-MPC.  Related Work

Revised: MBRL aims at improving the sample efficiency of model-free methods while maintaining highasymptotic performance.

Revised: The research of MBRL can be roughly divided into two lines: the modelusage and the model learning.

Original: Conclusion and Future Work
Revised: The most common usage of the model is generating pseudo samples toenrich the data buffer, so as to reduce the interaction with the environment and thus accelerate policylearning [Sutton, 1990, 1991, Deisenroth et al., 2013, Kalweit and Boedecker, 2017, Luo et al., 2018, Janner et al., 2019, Pan et al., 2020]. The main challenge of this kind of usage is the compoundingmodel error of long-term predictions, which is also known as the model bias issue [Deisenroth and Rasmussen, 2011].

Revised: A number of approaches are proposed to address this issue, varying from using

Revised: Dyna-like short-horizon rollouts [Sutton, 1991, Janner et al., 2019], interpolating between real andpseudo samples [Kalweit and Boedecker, 2017], learning an uncertainty-aware probabilistic modelensemble [Chua et al., 2018], to masking the rollouts according to the uncertainty estimation [Panet al., 2020].

Revised: Falling into this category of model usage, the focus of our work is to improve the qualityof model-generated trajectories by reducing the accumulative model error along these trajectories.

Revised: Through an active interaction between the model and the current policy, P2P reinforces a quickmodel adaptation to the update of policy, so as to foresee and thus avoid the uncertainty in the futuretrajectory during rollout.

Revised: Hence, one of the major differences between P2P and prior work can beinterpreted as: not just to passively estimate the uncertainty after encountering it, but to actively avoidit before this encounter.

Revised: In the line of model learning, the most commonly used objective is to simply minimize each one-stepprediction error for transitions available in the environment dataset [Kurutach et al., 2018, Chua et al.

Revised: ,2018, Janner et al., 2019].

Revised: However, since the data distribution of the environment dataset varieswith the update of policy, it is unlikely to learn an accurate model for all the transitions in complexenvironments.

Revised: Thus, when the model is used to predict for multiple steps, there is a mismatch ofobjectives between model usage and model learning.

Revised: To address this issue, we propose to minimizethe accumulative prediction error under the state distribution of the current policy.

Revised: The most relevantwork to ours includes [Nagabandi et al., 2018, Luo et al., 2018], where a similar multi-step predictionloss is employed for model learning.

Revised: The essential difference between their multi-step loss andours is clarified in Section 3.1 and 3.2.

Revised: Generative models [Kingma and Welling, 2014] have alsobeen utilized to perform multi-step prediction [Mishra et al., 2017, Ha and Schmidhuber, 2018].

Revised: However, the need for a highly exploratory dataset limits their application to relatively simple andlow-dimensional tasks.

Revised: Other learning objectives improve traditional model learning from anglesthat are less similar to ours, including goal-aware model learning that predicts how far the currentstate is from the goal state [Nair et al., 2020] and decision-aware model learning that aims to betteraids policy learning by re-weighting the model error on different samples [Farahmand et al., 2017,

Revised: Farahmand, 2018, D’Oro et al., 2020].

Original: In this paper, we propose an MBRL framework named Plan to Predict (P2P), which novelly treats themodel rollout process as a sequential decision making problem.
Revised: In this paper, we propose an MBRL framework named Plan to Predict (P2P), which novelly treats themodel rollout process as a sequential decision making problem.

Original: In this problem reformulation, theroles played by the model and the policy are reversed, where the model becomes a decision makerand the policy instead serves as the outer environment.
Revised: In this problem reformulation, theroles played by the model and the policy are reversed, where the model becomes a decision makerand the policy instead serves as the outer environment.

Original: Through an active interaction between themodel and the current policy, the model can minimize the multi-step accumulative errors on theinduced trajectories and thus quickly adapt to the state distribution of the current policy.
Revised: Through an active interaction between themodel and the current policy, the model can minimize the multi-step accumulative errors on theinduced trajectories and thus quickly adapt to the state distribution of the current policy.

Original: We providetheoretical guarantee for P2P, which also clarifies the difference of our multi-step loss from priormethods and justifies the mismatch of objectives between model learning and model usage.
Revised: We providetheoretical guarantee for P2P, which also clarifies the difference of our multi-step loss from priormethods and justifies the mismatch of objectives between model learning and model usage.

Original: Empiricalresults on several challenging benchmark tasks verify the effectiveness of P2P. Moreover, observingthat the capability to avoid model error makes P2P promising for offline settings, we also providepreliminary results on some offline learning tasks.
Revised: Empiricalresults on several challenging benchmark tasks verify the effectiveness of P2P. Moreover, observingthat the capability to avoid model error makes P2P promising for offline settings, we also providepreliminary results on some offline learning tasks.

Original: While this is not the major focus of this paper,studying how P2P can be applied in offline tasks more properly is left as an important future work.
Revised: While this is not the major focus of this paper,studying how P2P can be applied in offline tasks more properly is left as an important future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A.3.
Revised: [Yes] See Appendix A.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the supplementary material and Appendix B.
Revised: [Yes] See thesupplementary material and Appendix B.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See the supplementary material and Appendix B.
Revised: [Yes] See the supplementary material and Appendix B.(c)

Original: (c) Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using curating?

Original: [N/A]
Revised: [N /A] e

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: ) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]