Original: O N -P OLICY M ODEL E RRORS IN R EINFORCEMENT L 
Revised: O N -P OLICY M ODEL E RRORS IN R EINFORCEMENT L EARNING

Original: EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Model-free reinforcement learning algorithms can compute policy gradients given sampled environment transitions, but require large amounts of data.
Revised: Model-free reinforcement learning algorithms can compute policy gradients given sampled environment transitions, but require large amounts of data.

Original: In contrast, model-based methods can use the learned model to generate new data, but model errors and bias can render learning unstable or sub-optimal.
Revised: In contrast, model-based methods can use the learned model to generate new data, but model errors and bias can render learning unstable or suboptimal.

Original: In this paper, we present a novel method that combines real world data and a learned model in order to get the best of both worlds.
Revised: In this paper, we present a novel method that combines real-world data and a learned model in order to get the best of both worlds.

Original: The core idea is to exploit the real world data for onpolicy predictions and use the learned model only to generalize to different actions.
Revised: The core idea is to exploit the real-world data for onpolicy predictions and use the learned model only to generalize to different actions.

Original: Speciﬁcally, we use the data as time-dependent on-policy correction terms on top of a learned model, to retain the ability to generate data without accumulating errors over long prediction horizons.
Revised: Speciﬁcally, we use the data as time-dependent on-policy correction terms on top of a learned model, to retain the ability to generate data without accumulating errors over long prediction horizons.

Original: We motivate this method theoretically and show that it counteracts an error term for model-based policy improvement.
Revised: We motivate this method theoretically and show that it counteracts an error term for model-based policy improvement.

Original: Experiments on
Revised: Experiments on

Original: MuJoCo- and PyBullet-benchmarks show that our method can drastically improve existing model-based approaches without introducing additional tuning parameters.
Revised: MuJoCo- and PyBullet-benchmarks show that our method can drastically improve existing model-based approaches without introducing additional tuning parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Model-free reinforcement learning (RL) has made great advancements in diverse domains such as single- and multi-agent game playing (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), robotics (Kalashnikov et al., 2018) and neural architecture search (Zoph & Le, 2016).
Revised: Model-free reinforcement learning (RL) has made great advancements in diverse domains such as single- and multi-agent game playing (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), robotics (Kalashnikov et al., 2018), and neural architecture search (Zoph & Le, 2017).

Original: All of these model-free approaches rely on large numbers of interactions with the environment to ensure successful learning.
Revised: All of these model-free approaches rely on large numbers of interactions with the environment to ensure successful learning.

Original: While this issue is less severe for environments that can easily be simulated, it limits the applicability of model-free RL to (real-world) domains where data is scarce.
Revised: While this issue is less severe for environments that can easily be simulated, it limits the applicability of model-free RL to (real-world) domains where data is scarce.

Original: Model-based RL (MBRL) aims to reduce the amount of data required for policy optimization by learning a model that approximates the true environment, which we can use to generatecheap, simulated state transitions (Sutton, 1990; Racanière et al., 2017; Moerland et al., 2020).
Revised: Model-based RL (MBRL) reduces the amount of data required for policy optimization by approximating the environment with a learned model, which we can use to generate simulated state transitions (Sutton, 1990; Racanière et al., 2017; Moerland et al., 2020).

Original: While early approaches on low-dimensional tasks by Schneider (1997); Deisenroth & Rasmussen (2011) used probabilistic models with closed-form posteriors, recent methods rely on neural networks to scale tomore complex tasks on discrete (Kaiser et al., 2019) and continuous (Chua et al., 2018; Kurutach et al., 2018) action spaces.
Revised: While early approaches on lowdimensional tasks by Schneider (1997); Deisenroth & Rasmussen (2011) used probabilistic models with closed-form posteriors, recent methods rely on neural networks to scale to complex tasks on discrete (Kaiser et al., 2020) and continuous (Chua et al., 2018; Kurutach et al., 2018) action spaces.

Original: However, the learned representation of the true environment always remains imperfect and as such introduces an error source to the RL problem (Atkeson & Santamaria, 1997; Abbeel et al., 2006).
Revised: However, the learned representation of the true environment always remains imperfect, which introduces approximation errors to the RL problem (Atkeson & Santamaria, 1997; Abbeel et al., 2006).

Original: Hence, a key challenge in MBRL is model-bias ; small errors in the learned models thatcan compound over multi-step predictions and lead to lower asymptotic performance in contrast to model-free approaches.
Revised: Hence, a key challenge in MBRL is model-bias ; small errors in the learned models that compound over multi-step predictions and lead to lower asymptotic performance than model-free methods.

Original: To address these challenges with both model-free and model-based RL, Levine & Koltun (2013);
Revised: To address these challenges with both model-free and model-based RL, Levine & Koltun (2013);

Original: Chebotar et al.
Revised: Chebotar et al.

Original: (2017) propose to combine the merits of both.
Revised: (2017) propose to combine the merits of both.

Original: While there are multiple possibilities to combine the two methodologies, in this work we focus on improving the model’s predictive state distribution such that it more closely resembles the data distribution of the true environment.
Revised: While there are multiple possibilities to combine the two methodologies, in this work we focus on improving the model’s predictive state distribution such that it more closely resembles the data distribution of the true environment

Original:  Contributions
Revised: . Contributions

Original: The main contribution of this paper is on-policy corrections ( OPC ), a novel hyperparameter-free methodology that uses on-policy transition data on top of a separately learned model to enable accurate long-term predictions for MBRL.
Revised: The main contribution of this paper is on-policy corrections ( OPC ), a novel hyperparameter-free methodology that uses on-policy transition data on top of a separately learned model to enable accurate long-term predictions for MBRL.

Original: A key strength of our approach is that it does not introduce any new parameters that need to be hand-tuned for speciﬁc tasks.
Revised: A key strength of our approach is that it does not introduce any new parameters that need to be hand-tuned for speciﬁc tasks.

Original: We theoretically motivate our approach by means of a policy improvement bound and show that we can recover the true state distribution when generating trajectories on-policy with the model.
Revised: We theoretically motivate our approach by means of a policy improvement bound and show that we can recover the true state distribution when generating trajectories on-policy with the model.

Original: We illustrate how OPC improves the quality of policy gradient estimates in a simple toy example and evaluate it on various continuous control tasks from the MuJoCo control suite and their PyBullet variants.
Revised: We illustrate how OPC improves the quality of policy gradient estimates in a simple toy example and evaluate it on various continuous control tasks from the MuJoCo control suite and their PyBullet variants.

Original: There, we demonstrate that OPC improves current state-of-the-art MBRL algorithms in terms of data efﬁciency, especially for the more difﬁcult PyBullet environments.
Revised: There, we demonstrate that OPC improves current state-of-the-art MBRL algorithms in terms of data-efﬁciency, especially for the more difﬁcult PyBullet environments.

Original: Related Work
Revised: Related Work

Original: To counteract model-bias, several approaches combine ideas from model-free and model-based RL.
Revised: To counteract model-bias, several approaches combine ideas from model-free and model-based RL.

Original: For example, Levine & Koltun (2013) guide a model-free algorithm via modelbased planning towards promising regions in the state space, Kalweit & Boedecker (2017) augment the training data by an adaptive ratio of simulated transitions, Talvitie (2017) use ‘hallucinated’ transition tuples from simulated to observed states to self-correct the model, and Feinberg et al.
Revised: For example, Levine & Koltun (2013) guide a model-free algorithm via modelbased planning towards promising regions in the state space, Kalweit & Boedecker (2017) augment the training data by an adaptive ratio of simulated transitions, Talvitie (2017) use ‘hallucinated’ transition tuples from simulated to observed states to self-correct the model, and Feinberg et al.

Original: ; Buckman et al.
Revised: ; Buckman et al.

Original: (2018) use a learned model to improve the value function estimate.
Revised: (2018) use a learned model to improve the value function estimate.

Original: Janner et al.
Revised: Janner et al.

Original: (2019) mitigate the issue of compounding errors for long-term predictions by simulating short trajectores start from real states.
Revised: (2019) mitigate the issue of compounding errors for long-term predictions by simulating short trajectories that start from real states.

Original: Cheng et al.
Revised: Cheng et al.

Original: (2019) extend ﬁrst-order model-free algorithms via adversarial online learning to leverage prediction models in a regret-optimal manner.
Revised: (2019) extend ﬁrst-order model-free algorithms via adversarial online learning to leverage prediction models in a regret-optimal manner.

Original: Clavera et al.
Revised: Clavera et al.

Original: (2020) employ a model to augment an actor-critic objective, which they optimize via back-propagation through time, adapting the planning horizon interpolates between a purely model-based and a modelfree approach.
Revised: (2020) employ a model to augment an actor-critic objective and adapt the planning horizon to interpolate between a purely model-based and a model-free approach.

Original: Morgan et al.
Revised: Morgan et al.

Original: (2021) combine actor-critic methods with model-predictive rollouts to guarantee near-optimal simulated data and retain exploration on the real environment.
Revised: (2021) combine actor-critic methods with model-predictive rollouts to guarantee near-optimal simulated data and retain exploration on the real environment.

Original: A downside of most existing approaches is that they introduce additional hyperparameters that are critical to the learning performance (Zhang et al., 2021).
Revised: A downside of most existing approaches is that they introduce additional hyperparameters that are critical to the learning performance (Zhang et al., 2021).

Original: In addition to empirical performance, recent work builds on the theoretical guarantees for model-free approaches by Kakade & Langford (2002); Schulman et al.
Revised: In addition to empirical performance, recent work builds on the theoretical guarantees for model-free approaches by Kakade & Langford (2002); Schulman et al.

Original: (2015) to provide guarantees for MBRL.
Revised: (2015) to provide guarantees for MBRL.

Original: Luo et al.
Revised: Luo et al.

Original: (2018) provide a general framework to show monotonic improvement towards a local optimum of the value function, while Janner et al.
Revised: (2019) provide a general framework to show monotonic improvement towards a local optimum of the value function, while Janner et al.

Original: (2019) present a lower-bound on performance for different rollout schemes and horizon lengths.
Revised: (2019) present a lower-bound on performance for different rollout schemes and horizon lengths.

Original: Yu et al.
Revised: Yu et al.

Original: (2020) show guaranteed improvement in the ofﬂine MBRL setting by augmenting the reward with an uncertainty penalty, while Clavera et al.
Revised: (2020) show guaranteed improvement in the ofﬂine MBRL setting by augmenting the reward with an uncertainty penalty, while Clavera et al.

Original: (2020) present improvement guarantees in terms of the model’s and value function’s gradient errors.
Revised: (2020) present improvement guarantees in terms of the model’s and value function’s gradient errors.

Revised: Moreover, Harutyunyan et al.

Revised: (2016) propose a similar correction term as the one introduced in this paper in the context of off-policy policy evaluation and correct the state-action value function instead of the transition dynamics.

Revised: Similarly, Fonteneau et al.

Revised: (2013) consider the problem of off-policy policy evaluation but in the batch RL setting and propose to generate ‘artiﬁcial’ trajectories from observed transitions instead of using an explicit model for the dynamics.

Original: A related ﬁeld to MBRL that also combines models with data is iterative learning control (ILC) (Bristow et al., 2006).
Revised: A related ﬁeld to MBRL that also combines models with data is iterative learning control (ILC) (Bristow et al., 2006).

Original: The differences between the two methods are the following: RL typically focuses on ﬁnding parametric feedback policies for general reward functions,whereas ILC  seeks an open-loop sequence of actions with ﬁxed length to improve state tracking performance.
Revised: While RL typically focuses on ﬁnding parametric feedback policies for general reward functions, ILC instead seeks an open-loop sequence of actions with ﬁxed length to improve state tracking performance.

Original: Further, the model in ILC is often derived from ﬁrst principles and then kept ﬁxed, whereas in MBRL the model is continuously improved upon observing new data.
Revised: Moreover, the model in ILC is often derived from ﬁrst principles and then kept ﬁxed, whereas in MBRL the model is continuously improved upon observing new data.

Original: The method most closely related to RL and our approach is optimization-based ILC (Owens & Hätönen, 2005; Schöllig & D’Andrea, 2009), in which a linear dynamics model is used to guide the search for optimal actions.
Revised: The method most closely related to RL and our approach is optimization-based ILC (Owens & Hätönen, 2005; Schöllig & D’Andrea, 2009), in which a linear dynamics model is used to guide the search for optimal actions.

Original: Recently, Baumgärtner & Diehl (2020) extended the ILC setting to nonlinear dynamics and more general reward signals.
Revised: Recently, Baumgärtner & Diehl (2020) extended the ILC setting to nonlinear dynamics and more general reward signals.

Original: Little work is available that draws connections between RL and ILC (Zhang et al., 2019) with one notable exception: Abbeel et al.
Revised: Little work is available that draws connections between RL and ILC (Zhang et al., 2019) with one notable exception: Abbeel et al.

Original: (2006) use the observed data from the last rollout to account for a mismatch in the dynamics model.
Revised: (2006) use the observed data from the last rollout to account for a mismatch in the dynamics model.

Original: The limitations of this approach are that deterministic dynamics are assumed, the policy optimization itself requires a line search procedure with rollouts on the true environment and that it was not combined with model learning.
Revised: The limitations of this approach are that deterministic dynamics are assumed, the policy optimization itself requires a line search procedure with rollouts on the true environment and that it was not combined with model learning.

Original: We build on this idea and extend it to the stochastic setting of MBRL by making use of recent advances in RL and model learning.
Revised: We build on this idea and extend it to the stochastic setting of MBRL by making use of recent advances in RL and model learning.

Original: Lastly, Harutyunyan et al.

Original: (2016) propose a similar correction term as the one introduced in this paper in the context of off-policy policy evaluation and correct the state-action value function instead of the transition dynamics.

Original: Similarly, Fonteneau et al.

Original: (2013) consider the problem of off-policy policy evaluation but in the batch RL setting and propose to generate ‘artiﬁcial’ trajectories from observed transitions instead of using an explicit model for the dynamics.

Original: P ROBLEM S TATEMENT AND B ACKGROUND

Original: We consider the Markov decision process (MDP) ( S , A , p, r, γ, ρ ) where S ⊆ R d S and A ⊆

Original: R d A are the continuous state and action spaces, respectively.

Original: The unknown environment dynamics are described by the transition probability p ( s t +1 | s t , a t ) , an initial state distribution ρ ( s 0 ) and the reward signal r ( s , a ) .

Original: The goal in RL is to ﬁnd a policy π θ ( a t

Original: | s t ) parameterized by θ that maximizes the expected return, discounted by γ ∈ [0 , 1] over episodes of length T ,

Original: [Equation]

Original: 1: for n = 1 , . . .
Revised: 1: for n = 1 , . . .

Original: do 2: for b = 1 , . . .
Revised: do 2: for b = 1 , . . .

Original: , B do 3:
Revised: , B do 3:

Original: Rollout policy πθ n on environment and store transitions D bn = { (ˆ s n,bt , ˆ a n,bt , ˆ s n,bt +1 )
Revised: Rollout policy π n on environment and store transitions D bn = { (ˆ s n,bt , ˆ a n,bt , ˆ s n,bt +1 )

Original: } T − 1 t = 4: ˜
Revised: } T − 1 t = 4: ˜

Original: p n ( s t +1 | s t , a t ) : Learn a global dynamics model given all data D 1: n = (cid:83) ni =1 (cid:83) Bb =1 D bi 5: θ n +

Revised: p n ( s t +1 | s t , a t ) : Learn a global dynamics model given all data D 1: n = (cid:83) ni =1 (cid:83) Bb =1 D bi 5: θ n +

Original: = θ n + α ∇ ˜
Revised: = θ n + α ∇ ˜

Original: η n : Optimize the policy based on the model ˜ p with any RL algorithm
Revised: η n : Optimize the policy based on the model ˜ p with any RL algorithm

Revised:  P ROBLEM S TATEMENT AND B ACKGROUND

Revised: We consider the Markov decision process (MDP) ( S , A , p, r, γ, ρ ) , where S ⊆ R d S and A ⊆

Revised: R d A are the continuous state and action spaces, respectively.

Revised: The unknown environment dynamics are described by the transition probability p ( s t +1 | s t , a t ) , an initial state distribution ρ ( s 0 ) and the reward signal r ( s , a ) .

Revised: The goal in RL is to ﬁnd a policy π θ ( a t

Original:  The expectation is taken with respect to the trajectory under the stochastic policy π θ starting from some stochastic initial state.
Revised: | s t ) parameterized by θ that maximizes the expected return discounted by γ ∈ [0 , 1] over episodes of length T , η = [Equation] The expectation is taken with respect to the trajectory under the stochastic policy π θ starting from a stochastic initial state s 0 .

Original: Direct maximization of Eq.
Revised: Direct maximization of Eq.

Original: (1) is challenging, since we do not know the environment’s transition probabilities p .
Revised: (1) is challenging, since we do not know the environment’s transition model p .

Original: In MBRL, we learn a model for the transitions and reward function from data, ˜ p
Revised: In MBRL, we learn a model for the transitions and reward function from data, ˜ p

Original: ( s t +1 | s t , a t )
Revised: ( s t +1 | s t , a t )

Original: ≈ p ( s t +1 | s t , a t ) and ˜ r respectively, and subsequently maximize ˜ η =
Revised: ≈ p ( s t +1 | s t , a t ) and ˜ r ( s t , a t ) ≈ r ( s t , a t ) , respectively.

Original: [Equation] ) as a surrogate problem forEq. The choice of initial state distribution ˜ ρ ( s 0 ) depends on the optimization method: For model predictive control, Chua et al.
Revised: Subsequently, we maximize the model-based expected return ˜ η as a surrogate problem for the true RL setting, where ˜ η is deﬁned as in Eq.

Original: (2018) use the current state to optimize the next action based on a rollout, while Janner et al. (2019) use the empirical state distribution observed from the true environment.
Revised: (1) but with ˜ p and ˜ r instead.

Original: In the following, we focus on the latter and for ease of exposition assume a known reward function ˜ r
Revised: For ease of exposition, we assume a known reward function ˜ r

Original: = r , even though we learn it jointly with ˜ p in our experiments.
Revised: = r , even though we learn it jointly with ˜ p in our experiments.

Original: We let η n denote the policy’s performance under the policy π n = π θ n at iteration n and use ˆ s and ˆ a for states and actions that are observed on the true environment.
Revised: We let η n denote the return under the policy π n = π θ n at iteration n and use ˆ s and ˆ a for states and actions that are observed on the true environment.

Original: Algorithm 1 summarizes the overall procedure for MBRL:
Revised: Algorithm 1 summarizes the overall procedure for MBRL:

Original: At each iteration n we store B on-policy trajectories D bn = { (ˆ s n,bt , ˆ a n,bt , ˆ s n,bt +1 ) }
Revised: At each iteration n we store B on-policy trajectories D bn = { (ˆ s n,bt , ˆ a n,bt , ˆ s n,bt +1 ) }

Original: T − 1 t =0 obtained by rolling out the current policy π n on the real environment in Line 3.
Revised: T − 1 t = obtained by rolling out the current policy π n on the real environment in Line 3.

Original: Lastly, we approximate the environment with a learned model ˜ p n ( s t +1 | s t , a t ) based on the data  in Line 4, and optimize the policy based on the proxy objective in Eq. (2) in Line 5.
Revised: Afterwards, we approximate the environment with a learned model ˜ p n ( s t +1 | s t , a t ) based on the data D 1: n in Line 4, and optimize the policy based on the proxy objective ˜ η in Line 5.

Original: Note that the policy optimization algorithm can be off-policy and employ its own, separate replay buffer.
Revised: Note that the policy optimization algorithm can be off-policy and employ its own, separate replay buffer.

Original: Model Choices
Revised: Model Choices

Original: The choice of model ˜ p plays a key role, since itdeﬁnes the surrogate problem Eq. Inside the optimization loop, the model istypically used to predict sequences  of  transitions τ with length H .
Revised: The choice of model ˜ p plays a key role, since it is used to predict sequences τ of states transitions and thus deﬁnes the surrogate problem in MBRL. We assume that the model comes from a distribution family P , which for each state-action pair ( s t , a t ) models a distribution over the next state s t +1 .

Original: One option is  to use a purely data-based model that replays past transitions from a recent on-policy trajectory b starting at time step t , ˜
Revised: The model is then trained to summarize all past data D 1: n = ∪ ni =1 ∪ Bb =1 D bi by maximizing the marginal log-likelihood

Original: [Equation]
Revised: [Equation]

Revised: For a sampled trajectory index b ∼ U ( { 1 , . . .

Original: data{+, B } ) , sequences τ start from the initial state ˆ s n,b 
 and are distributed according to ˜ p model+} n
Revised: {+, B } ) , sequences τ start from the initial state ˆ s n,b 
 and are distributed according to ˜ p model+} n

Original: [Equation]
Revised: [Equation]

Original: with
Revised: s [Equation] [Equation] θ

Original: [Equation]
Revised: [Equation]

Original:  where δ ( · ) denotes a Dirac-delta distribution and 1 ≤ b ≤ B .
Revised: a t | s t ) , where δ ( · ) denotes the Dirac-delta distribution.

Original: This model choice effectively reduces model-based algorithms back to the model-free setting. An alternative is  to summarize all the past data D 1: n = ∪ ni =1 ∪ Bb =1 D bi in a parametric model coming from a distribution family P by maximizing the marginal log-likelihood
Revised: Using model-data for policy optimization is in contrast to model-free methods, which only use observed environment data by replaying past transitions from a recent on-policy trajectory b ∈ { 1 , . . .

Original: [Equation]
Revised: , B } .

Original: which for each state-action pair ( s t , a t ) models a distribution over the next state s t +1 , such that ˜ p modeln
Revised: In our model-based framework, this replay buffer is equivalent to the non-parametric model

Original: [Equation]
Revised: [Equation]

Original: ) with a t
Revised: ˜ p

Original: [Equation]
Revised: [Equation]

Original: ∼ π θ . For both models, we can sample H -step predictions by ﬁrst sampling a trajectory index b ∼ Categorical( { 0 , . . . , B } ) and then rolling out following the respective state sequence distributions ˜ p ( τ t : t + H | t, b ) .
Revised: where we only replay observed transitions instead of sampling new actions from π θ .

Original:  O N - POLICY C ORRECTIONS
Revised:  O N - POLICY C ORRECTIONS

Original: In this section, we analyze how the choice of model impacts policy improvement, develop OPC as a model that can eliminate one term in the improvement bound, and analyze its properties.
Revised: In this section, we analyze how the choice of model impacts policy improvement, develop OPC as a model that can eliminate one term in the improvement bound, and analyze its properties.

Original: In the following, we drop the n sub- and superscript when the iteration is clear from context.
Revised: In the following, we drop the n sub- and superscript when the iteration is clear from context.

Original: 3. P OLICY I MPROVEMENT
Revised: 3. P OLICY I MPROVEMENT

Original: Independent of whether we usea the data directly Eq. (3) or summarize it in a world model Eq. (4), our goal is to ﬁnd an optimal policy that maximizes Eq.
Revised: Independent of whether we use the data directly in ˜ p data or summarize it in a world model ˜ p model , our goal is to ﬁnd an optimal policy that maximizes Eq.

Original: (1) via the corresponding model-based proxy objective Eq.
Revised: (1) via the corresponding model-based proxy objective.

Original: To this end, we would like to know how a policy improvement ˜ η n +1 − ˜ η n ≥based on the model ˜ p , which is what we optimize in Eq. (2), relates to the true gain in performance η n +1 − η n on the environment with unknown transitions p .
Revised: To this end, we would like to know how a policy improvement ˜ η n +1 − ˜ η n ≥ 0 based on the model ˜ p , which is what we optimize in MBRL, relates to the true gain in performance η n +1 − η n on the environment with unknown transitions p .

Original: While the two are equal without model errors,˜ p = p , in general the larger the model error, the worse wecan expect our proxy objective to be.
Revised: While the two are equal without model errors, in general the larger the model error, the worse we expect the proxy objective to be (Lambert et al., 2020).

Original: More concretely, we show in Appendix A.1 that the policy improvement can be decomposed as
Revised: Speciﬁcally, we show in Appendix B.1 that the policy improvement can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: where a performance improvement on our model-based objective in Eq. (2) only translates to a gain in Eq.
Revised: where a performance improvement on our model-based objective ˜ η only translates to a gain in Eq.

Original: (1) if two error terms are sufﬁciently small.
Revised: (1) if two error terms are sufﬁciently small.

Original: These terms depend on how well the performance estimate based on our model, ˜ η , matches the true performance, η .
Revised: These terms depend on how well the performance estimate based on our model, ˜ η , matches the true performance, η .

Original: If the reward function is known, this term only depends on the model quality of ˜ p relative to p .
Revised: If the reward function is known, this term only depends on the model quality of ˜ p relative to p .

Original: Note that in contrast to the result by Janner et al.
Revised: Note that in contrast to the result by Janner et al.

Original: (2019), Eq.
Revised: (2019), Eq.

Original: (5) is a bound on the policy improvement instead of a lower bound on η n +1 .
Revised: (4) is a bound on the policy improvement instead of a lower bound on η n +1 .

Original: The ﬁrst error term compares η n +1 and ˜ η n +1 , the performance estimation gap under the optimized policy π n +1 that we obtain in Line 5 of Algorithm 1.
Revised: The ﬁrst error term compares η n +1 and ˜ η n +1 , the performance estimation gap under the optimized policy π n +1 that we obtain in Line 5 of Algorithm 1.

Original: Since at this point we have only collected data with π n in Line 3, this term depends on the generalization properties of our model to new data; what we call the off-policy model error .
Revised: Since at this point we have only collected data with π n in Line 3, this term depends on the generalization properties of our model to new data; what we call the off-policy model error .

Original: For our data-based model ˜ p data that just replays data under π n independently of the action, this term can be bounded for stochastic policies.
Revised: For our data-based model ˜ p data that just replays data under π n independently of the action, this term can be bounded for stochastic policies.

Original: For example, Schulman et al.
Revised: For example, Schulman et al.

Original: (2015) bound it by the average KL-divergence between π n and π n +1 .
Revised: (2015) bound it by the average KL-divergence between π n and π n +1 .

Original: For learned models ˜ p model , it depends on the generalization properties of the model (Luo et al., 2018; Yu et al., 2020).
Revised: For learned models ˜ p model , it depends on the generalization properties of the model (Luo et al., 2019; Yu et al., 2020).

Original: While understanding model generalization better is an interesting research direction, we will assume that our learned model is able to generalize to new actions in the following sections.
Revised: While understanding model generalization better is an interesting research direction, we will assume that our learned model is able to generalize to new actions in the following sections.

Original: While the ﬁrst term hinges on model-generalization, the second term is the on-policy model error , i.e., the deviation between η n and ˜ η n under the current policy π n .
Revised: While the ﬁrst term hinges on model-generalization, the second term is the on-policy model error , i.e., the deviation between η n and ˜ η n under the current policy π n .

Original: This error term goes to zero for ˜ p data as we use more on-policy
Revised: This error term goes to zero for ˜ p data as we use more on-policy

Original: data B → ∞ , since the transition data are sampled from the true environment and the mean converges, c.f., Appendix A.2.
Revised: data B → ∞ , since the transition data are sampled from the true environment, c.f., Appendix B.2.

Original: While the learned model is also trained with on-policy data, small errors in our model compound as we iteratively predict ahead in time.
Revised: While the learned model is also trained with on-policy data, small errors in our model compound as we iteratively predict ahead in time.

Original: Consequently,for prediction horizons shorter than the effective horizon encoded in the discount factor, H < γ/ (1 − γ ) , the on-policy error term grows as O (min(  H/ (1 − γ ) , H 2 )) , c.f., (Janner et al., 2019) and Appendix A.3.
Revised: Consequently, the on-policy error term grows as O (min( γ/ (1 − γ ) 2 , H/ (1 − γ ) , H 2 )) , c.f., (Janner et al., 2019) and Appendix B.3.

Original: The root cause is the same as in behavior cloning (Ross & Bagnell, 2010, Theorem 2.1), where small errors aggregate over time.

Original: 3. C OMBINING L EARNED M ODELS AND R EPLAY B UFFER
Revised: 3. C OMBINING L EARNED M ODELS AND R EPLAY B UFFER

Original: The key insight of this paper is that the replay buffer in Eq.
Revised: The key insight of this paper is that the learned model in Eq.

Original: (3) and the parametric model in Eq.
Revised: (2) and the replay buffer in Eq.

Original: (4) have opposing strengths: The replay buffer has low error on-policy, but high error off-policy since it replays transitions from past data, i.e., they are independent of the actions chosen under the new policy.
Revised: (3) have opposing strengths: The replay buffer has low error on-policy, but high error off-policy since it replays transitions from past data, i.e., they are independent of the actions chosen under the new policy.

Original: In contrast, the learned model can generalize to new actions by extrapolating from the data and thus has lower error off-policy, but errors compound over the prediction horizon H .
Revised: In contrast, the learned model can generalize to new actions by extrapolating from the data and thus has lower error off-policy, but errors compound over multi-step predictions.

Original: An ideal model would combine the model-free and model-based approaches in a way such that it retains the unbiasedness of on-policy generated data, but also generalizes to new policies via the model.
Revised: An ideal model would combine the model-free and model-based approaches in a way such that it retains the unbiasedness of on-policy generated data, but also generalizes to new policies via the model.

Original: To this end, we propose to adjust the  observed transitions by an offset that accounts forthe model predictions given a new state-action pair.
Revised: To this end, we propose to use the model to predict how observed transitions would change for a new state-action pair.

Original: In particular, we use the model
Revised: In particular, we use the model

Original: ’s mean prediction ˜ f  ( st , at ) =
Revised: ’s mean prediction ˜ f n ( s , a ) =

Original: E[-s t +
-] [˜ p model n ( s t +1 | st , at )] to construct the joint model 
Revised: E[-s t +
-] [˜ p model n ( · | s , a )] to construct the joint model [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where ∗ denotes the convolution of the two distributions and b refers to a speciﬁc rollout stored in the replay buffer that was observed in the true environment.
Revised: where ∗ denotes the convolution of the two distributions and b refers to a speciﬁc rollout stored in the replay buffer that was observed in the true environment.

Original: Given a batch-index b , ˜ p opc  in Eq.
Revised: Given a trajectory-index b , ˜ p opc n in Eq.

Original: (6) transitions deterministically according to s t
Revised: (5) transitions deterministically according to s t

Original: +1 = ˆ s n,bt +1 + ˜ f  ( s t , a t ) − ˜ f
Revised: +1 = ˆ s n,bt +1 + ˜ f n ( s t , a t ) − ˜ f

Original:  (ˆ s n,bt , ˆ a n,bt ) , resembling the equations in ILC (c.f., Baumgärtner & Diehl (2020) and Appendix D).
Revised: n (ˆ s n,bt , ˆ a n,bt ) , resembling the equations in ILC (c.f., Baumgärtner & Diehl (2020) and Appendix E).

Original: If we rollout ˜ p opc  along a trajectory, starting from a state ˆ s n,bt and apply the recorded actions from the replay buffer, ˆ a n,bt , the correction term on the right of Eq.
Revised: If we roll out ˜ p opc n along a trajectory, starting from a state ˆ s n,bt and apply the recorded actions from the replay buffer, ˆ a n,bt , the correction term on the right of Eq.

Original: (6) cancels out and we have ˜ p opc  ( s t +1 | ˆ s n,bt , ˆ a n,bt, b) =
Revised: (5) cancels out and we have ˜ p opc n ( s t +1 | ˆ s n,bt , ˆ a n,bt b

Original:  ˜
Revised: = ˜

Original: p data  ( s t +1 | t + 1 , b ) =
Revised: p data n ( s t +1 | t, b ) =

Original: δ ( s t +1 − ˆ s n,bt +1 ).
Revised: δ ( s t +1 − ˆ s n,b t +1 )

Original: Thus OPC retrieves the true on-policy data distribution independent of the prediction quality of the model, which is why we refer to this method as on-policy

Original: ˆ s b 0 =
Revised: ˆ s b 0 =

Original: sˆ s b 1 = ˜
Revised: sˆ s b 1 = ˜

Original: p opc ( s 

Revised: p opc ( s

Original: | ˆ s b 0 , ˆ a b 0 ,b ) ˜ p model ( s 1 | ˆ s b 0 , ˆ a b 0 ) ˜
Revised: | ˆ s b 0 , ˆ a b 0 ,b ) ˜ p model ( s 1 | ˆ s b 0 , ˆ a b 0 ) ˜

Original: p model ( s 1 | s 0 , a 0 ) ˜
Revised: p model ( s 1 | s 0 , a 0 ) ˜

Original: p opc ( s 1 | s 0 , a 0 ,b ) ˆ s b 2 = ˜
Revised: p opc ( s 1 | s 0 , a 0 ,b ) ˆ s b 2 = ˜

Original: p opc ( s 2 |
Revised: p opc ( s 2 |

Original: ˆ s b 1 , ˆ a b 1 ,b ) ˜
Revised: ˆ s b 1 , ˆ a b 1 ,b ) ˜

Original: p model ( s 2 | ˆ s b 1 , ˆ a b 1 ) 
Revised: p model ( s 2 | ˆ s b 1 , ˆ a b 1 ) ˜

Original: [Figure]

Original:  ( s 2 | s 1 , a 1 ) [Figure]
Revised: p model ( s 2 | s 1 , a 1 ) ˜

Original: p opc ( s 2 | s 1 , a 1 ,b )
Revised: p opc ( s 2 | s 1 , a 1 ,b )

Original: [Figure]
Revised: [Figure]

Original:  corrections ( OPC ).
Revised: distribution independent of the prediction quality of the model, which is why we refer to this method as on-policy corrections ( OPC ).

Original: This behaviour is illustrated in Fig.
Revised: This behavior is illustrated in Fig.

Original: 1a, whereeven though the model (blue) is biased on-policy,  OPC corrects the model’s prediction to match the true data.
Revised: 1a, where the model (blue) is biased on-policy, but OPC corrects the model’s prediction to match the true data.

Original: In Fig.
Revised: In Fig.

Original: 1b, we show how this affects predicted rollouts on a simple stochastic double-integrator environment.
Revised: 1b, we show how this affects predicted rollouts on a simple stochastic double-integrator environment.

Original: Even though small on-policy errors in ˜ p model (blue) compound over time, the corresponding ˜ p opc matches the ground-truth environment data closely.
Revised: Although small on-policy errors in ˜ p model (blue) compound over time, the corresponding ˜ p opc matches the ground-truth environment data closely.

Original: Note that even though the model in Eq.
Revised: Note that even though the model in Eq.

Original: (6) is deterministic, we retain the environment’s stochasticity from the data in the transitions to ˆ s t +1 , so that we recover the on-policy aleatoric uncertainty (noise) from sampling different reference trajectories via indexes b .
Revised: (5) is deterministic, we retain the environment’s stochasticity from the data in the transitions to ˆ s t +1 , so that we recover the on-policy aleatoric uncertainty (noise) from sampling different reference trajectories via indexes b .

Original: When our actions a t are different from ˆ a n,bt , ˜ p opc still uses the data from the environment’s transitions, but the correction term in Eq.
Revised: When our actions a t are different from ˆ a bt , ˜ p opc still uses the data from the environment’s transitions, but the correction term in Eq.

Original: (6) uses the learned model to predict how the next state changes in expectation relative to the prediction under ˆ a n,bt .
Revised: (5) uses the learned model to predict how the next state changes in expectation relative to the prediction under ˆ a bt .

Original: That is, in Fig.
Revised: That is, in Fig.

Original: 1a for a new a t the model predicts the state distribution shown in red.
Revised: 1a for a new a t the model predicts the state distribution shown in red.

Original: Correspondingly, we shift the static prediction ˆ s n,bt +1 by the difference in means (gray arrow) between the two predictions; i.e., the change in trajectory by changing from ˆ a n,bt to a t .
Revised: Correspondingly, we shift the static prediction ˆ s bt +1 by the difference in means (gray arrow) between the two predictions; i.e., the change in trajectory by changing from ˆ a bt to a t .

Original: Since we shift the model predictions by a time-dependent but constant offset, this does not recover the true state distribution unless the model has zero error.
Revised: Since we shift the model predictions by a time-dependent but constant offset, this does not recover the true state distribution unless the model has zero error.

Original: However, regardless of this it can  help with long-term predictions in Fig.
Revised: However, empirically it can still help with long-term predictions in Fig.

Original: 1b by shifting the model off-policy predictions (red) to the
Revised: 1b by shifting the model off-policy predictions (red) to the

Original: OPC predictions (green), which are closer to the environment’s state distribution under the new policy.
Revised: OPC predictions (green), which are closer to the environment’s state distribution under the new policy.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: As mentioned in the previous sections,  OPCis motivated to reduce the on-policy model error from Eq.
Revised: In the previous sections, we have introduced OPC to decrease the on-policy model error in Eq.

Original: (5) to tighten the improvement bound.
Revised: (4) and tighten the improvement bound.

Original: In this section, we analyze the on-policy performance gap from a theoretical perspective and show that with OPC this error can be reduced independently from the learned model’s error.
Revised: In this section, we analyze the on-policy performance gap from a theoretical perspective and show that with OPC this error can be reduced independently of the learned model’s error.

Original: To this end, we derive a more general form of the predictive model for OPC where we consider ˆ s n,bt +1 as a random variable following the true environment’s transition dynamics instead of an observed quantity.
Revised: To this end, we assume inﬁnitely many on-policy reference trajectories, B → ∞ , which is equivalent to a variant of ˜ p opc that considers ˆ s bt +1 as a random variable that follows the true environment’s transition dynamics.

Original: While impossible to implement in practice, this formulation helps in the theoretical analysis.
Revised: While impossible to implement in practice, this formulation is useful to understand our method.

Original: The generalized OPC -modelis deﬁned as
Revised: We deﬁne the generalized OPC -model as

Original: [Equation]
Revised: [Equation]

Original: which highlights that it transitions according to the true on-policy dynamics conditioned on data from the replay buffer, combined with a correction term.
Revised: which highlights that it transitions according to the true on-policy dynamics conditioned on data from the replay buffer, combined with a correction term.

Original: We provide a detailed derivation for the generalized model in Appendix A (Lemma 4.
Revised: We provide a detailed derivation for the generalized model in Appendix B, Lemma 4.

Original: With Eq.
Revised: With Eq.

Original: (7), we have the following result:
Revised: (6), we have the following result:

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ˜ η opc (cid:63) and η be the expected return under the generalized OPC -model Eq.
Revised: Let ˜ η opc (cid:63) and η be the expected return under the generalized OPC -model Eq.

Original: (7) and the true environment, respectively.
Revised: (6) and the true environment, respectively.

Original: Assume that the learned model’s mean transition function ˜ f
Revised: Assume that the learned model’s mean transition function ˜ f

Original: ( s t , a t ) =
Revised: ( s t , a t ) =

Original: E [˜ p model ( s t +1 | s t , a t )] is L f -Lipschitz and the reward r ( s t , a t ) is L r -Lipschitz.
Revised: E [˜ p model ( s t +1 | s t , a t )] is L f -Lipschitz and the reward r ( s t , a t ) is L r -Lipschitz.

Original: Further, if the policy π ( a t | s t ) is L π -Lipschitz with respect to s t under the Wasserstein distance and its (co-)variance Var[ π ( a t
Revised: Further, if the policy π ( a t | s t ) is L π -Lipschitz with respect to s t under the Wasserstein distance and its (co-)variance Var[ π ( a t

Original: | s t )]
Revised: | s t )]

Original: = Σ π ( s t )
Revised: = Σ π ( s t )

Original: ∈ S
Revised: ∈ S

Original: d A
Revised: d A

Original: + is ﬁnite over the complete state space, i.e., max s t ∈S trace { Σ π ( s t ) } ≤ ¯ σ 2 π , then with C 1 = (cid:112) 2(

Revised: + is ﬁnite over the complete state space, i.e., max s t ∈S trace { Σ π ( s t ) } ≤ ¯ σ 2 π , then with C 1 = (cid:112) 2(

Original: + L 2 π ) L f L r and C 2 = (cid:113)
Revised: + L 2 π ) L f L r and C 2 = (cid:113)

Original: L 2 f + L 2 π
Revised: L 2 f + L 2 π

Original: [Equation]
Revised: [Equation]

Original: We provide a proof in Appendix A.4.
Revised: We provide a proof in Appendix B.4.

Original: From Theorem 1, we can observe the key property of OPC : for deterministic policies, the on-policy model error from Eq.
Revised: From Theorem 1, we can observe the key property of OPC : for deterministic policies, the on-policy model error from Eq.

Original: (5) is zero and independent of the learned models’ predictive distribution ˜ p model , so that η = ˜ η opc (cid:63) .
Revised: (4) is zero and independent of the learned models’ predictive distribution ˜ p model , so that η = ˜ η opc (cid:63) .

Original: For policies with non-zero variance, the bound scales exponentially with T , highlighting the problem of compounding errors.
Revised: For policies with non-zero variance, the bound scales exponentially with T , highlighting the problem of compounding errors.

Original: In this case, as in the off-policy case, the model quality determines how well we can generalize to different actions.
Revised: In this case, as in the off-policy case, the model quality determines how well we can generalize to different actions.

Revised: We show in Appendix B.5 that, for one-step predictions, OPC ’s prediction error scales as the minimum of policy variance and model error.

Original: To  alleviate this issue, one could extend Theorem 1  similarly to the results in Janner et al.
Revised: To further alleviate the issue of compounding errors, one could extend Theorem 1 with a branched rollout scheme similarly to the results by Janner et al.

Original: (2019), such that the rollouts are only of length H (cid:28) T .
Revised: (2019), such that the rollouts are only of length H (cid:28) T .

Original: In practice, ˜ p opc (cid:63) can never be realized as it requires the true (unknown) state distribution.
Revised: In practice, ˜ p opc (cid:63) cannot be realized as it requires the true (unknown) state transition model p .

Original: However,  we can approximate it by ˜ p opc  Eq. (6) using an ever-increasing size of reference trajectories such that the on-policy error converges to zero  in probability.
Revised: However, as we use more on-policy reference trajectories for ˜ p opc in Eq. (5), it also converges to zero on-policy error in probability for deterministic policies.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let M be a MDP with dynamics p
Revised: Let M be a MDP with dynamics p

Original: ( s t +1 | s t , a t ) and reward r < r max .
Revised: ( s t +1 | s t , a t ) and reward r < r max .

Original: Let ˜ M be another
Revised: Let ˜ M be another

Original: MDP with dynamics ˜
Revised: MDP with dynamics ˜

Original: p model (cid:54) = p .
Revised: p model (cid:54) = p .

Original: Assume a deterministic policy π : S (cid:55)→ A and a set of trajectories
Revised: Assume a deterministic policy π : S (cid:55)→ A and a set of trajectories

Original: D = (cid:83)
Revised: D = (cid:83)

Original: Bb =1 { (

Original: [Equation]

Original: s bt , ˆ a bt ,

Original: [Equation] s [Equation] +1 ) } T − 1 t =0 collected from M under π .
Revised: Bb =1 { (ˆ s bt , ˆ a bt , ˆ s bt +1 ) } T − 1 t =0 collected from M under π .

Original: If we use OPC Eq.
Revised: If we use OPC Eq.

Original: (6) with data D , then lim 
Revised: (5) with data D , then lim B

Original: [Equation]
Revised: [Equation]

Original: with convergence rate O
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: B )

Original: [Equation]
Revised: [Equation]

Revised: 0 with convergence rate O (1 /B ) .

Original: We provide a proof in Appendix A.4.
Revised: We provide a proof in Appendix B.4.

Original: Lemma 1 states that given sufﬁcient reference on-policy data, the performance gap due to model errors becomes arbitrarily small for any model ˜ p model when using OPC .
Revised: Lemma 1 states that given sufﬁcient reference on-policy data, the performance gap due to model errors becomes arbitrarily small for any model ˜ p model when using OPC .

Original: While the assumption for inﬁnite on-policy data in Lemma 1 is unrealistic for practical applications, we found empirically that OPC drastically reduces the on-policy model error even when the assumptions are violated.
Revised: While the assumption of inﬁnite on-policy data in Lemma 1 is unrealistic for practical applications, we found empirically that OPC drastically reduces the on-policy model error even when the assumptions are violated.

Original: In our implementation, we use stochastic policies as well as trajectories from previous policies, i.e., off-policy data, for the corrections in OPC (see also Section 4.2).
Revised: In our implementation, we use stochastic policies as well as trajectories from previous policies, i.e., off-policy data, for the corrections in OPC (see also Section 4.2).

Original: 3. D ISCUSSION Epistemic uncertainty
Revised: 3. D ISCUSSION Epistemic uncertainty

Original: So far we have only considered aleatoric uncertainty (noise) in our transition models.
Revised: So far, we have only considered aleatoric uncertainty (noise) in our transition models.

Original: In practice, modern methods additionally distinguish epistemic uncertainty that arises from having seen limited data (Deisenroth & Rasmussen, 2011; Chua et al., 2018).
Revised: In practice, modern methods additionally distinguish epistemic uncertainty that arises from having seen limited data (Deisenroth & Rasmussen, 2011; Chua et al., 2018).

Original: This leads to a distribution (or an ensemble) of models, where each sample could explain the data.
Revised: This leads to a distribution (or an ensemble) of models, where each sample could explain the data.

Original: In this setting, we apply OPC by correcting each sample individually .
Revised: In this setting, we apply OPC by correcting each sample individually .

Original: This allows us to retain epistemic uncertainty estimates after applying OPC , while the epistemic uncertainty is zero on-policy.
Revised: This allows us to retain epistemic uncertainty estimates after applying OPC , while the epistemic uncertainty is zero on-policy.

Original: Limitations
Revised: Limitations

Original: Since OPC uses on-policy corrections, it is inherently limited to local policy optimization where the policy changes slowly over time.
Revised: Since OPC uses on-policy data, it is inherently limited to local policy optimization where the policy changes slowly over time.

Original: As a consequence, it is not suitable for global exploration schemes like the one proposed by Curi et al.
Revised: As a consequence, it is not suitable for global exploration schemes like the one proposed by Curi et al.

Original: Similarly, since OPC uses the observed data and corrects it only with the expected learned model, ˜ p opc always uses the on-policy transition noise (aleatoric uncertainty) from the data, even if the model has learned to represent it.
Revised: Similarly, since OPC uses the observed data and corrects it only with the expected learned model, ˜ p opc always uses the on-policy transition noise (aleatoric uncertainty) from the data, even if the model has learned to represent it.

Original: While not having to learn a representation for aleatoric uncertainty can be a strength, it limits our approach to environments where the aleatoric uncertainty does not vary signiﬁcantly with state/actions.
Revised: While not having to learn a representation for aleatoric uncertainty can be a strength, it limits our approach to environments where the aleatoric uncertainty does not vary signiﬁcantly with states/actions.

Original: It is possible to extend the method to the heteroscedastic noise setting under additional assumptions that enable distinguishing model error from transition noise (Schöllig & D’Andrea, 2009).
Revised: It is possible to extend the method to the heteroscedastic noise setting under additional assumptions that enable distinguishing model error from transition noise (Schöllig & D’Andrea, 2009).

Original: Lastly, our method applies directly only to MDPs, since we rely on state-observations.
Revised: Lastly, our method applies directly only to MDPs, since we rely on state-observations.

Original: Extending the ideas to partially observed environments is an interesting direction for future research.
Revised: Extending the ideas to partially observed environments is an interesting direction for future research.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We begin the experimental section with a motivating example on a toy problem to highlight the impact of OPC on the policy gradient estimates in the presence of model errors.
Revised: We begin the experimental section with a motivating example on a toy problem to highlight the impact of OPC on the policy gradient estimates in the presence of model errors.

Original: The remainder of the section focuses on comparative evaluations and ablation studies on complex continuous control tasks.
Revised: The remainder of the section focuses on comparative evaluations and ablation studies on complex continuous control tasks.

Original:  I LLUSTRATIVE E XAMPLE
Revised: 4. I LLUSTRATIVE E XAMPLE

Original: In Section 3.3, we investigate the inﬂuence of the model error directly on the expected return from a theoretical perspective.
Revised: In Section 3.3, we investigate the inﬂuence of the model error directly on the expected return from a theoretical perspective.

Original: From a practical standpoint, another relevant question is how the policy optimization and the respective policy gradients are inﬂuenced by model errors.
Revised: From a practical standpoint, another relevant question is how the policy optimization and the respective policy gradients are inﬂuenced by model errors.

Original: For general environments and reward signals, this question is difﬁcult to answer, due to the typically highdimensional state/action spaces and large number of parameters governing the dynamics model as well as the policy.
Revised: For general environments and reward signals, this question is difﬁcult to answer, due to the typically highdimensional state/action spaces and large number of parameters governing the dynamics model as well as the policy.

Original: To shed light on this open question, we resort to a simple low-dimensional example and investigate how OPC improves the gradient estimates under a misspeciﬁed dynamics model.
Revised: To shed light on this open question, we resort to a simple low-dimensional example and investigate how OPC improves the gradient estimates under a misspeciﬁed dynamics model.

Original: In particular, we assume a one-dimensional deterministic environment with linear transitions and a linear policy.
Revised: In particular, we assume a one-dimensional deterministic environment with linear transitions and a linear policy.

Original: The beneﬁts of this example are that we can 1) compute the true policy gradient based on a single rollout (determinism), 2) determine the environment’s closed-loop stability under the policy (linearity), and 3) visualize the gradient error as a function of all relevant parameters (low dimensionality).
Revised: The beneﬁts of this example are that we can 1) compute the true policy gradient based on a single rollout (determinism), 2) determine the environment’s closed-loop stability under the policy (linearity), and 3) visualize the gradient error as a function of all relevant parameters (low dimensionality).

Original: The dynamics and initial state distribution are speciﬁed by p ( s t
Revised: The dynamics and initial state distribution are speciﬁed by p ( s t

Original: +

Revised: +

Original: | s t , a t ) =
Revised: | s t , a t ) =

Original: δ ( As t + Ba t | s t , a t ) with ρ ( s 0 ) =
Revised: δ ( As t + Ba t | s t , a t ) with ρ ( s 0 ) =

Original: δ ( s 0 ) where A , B ∈ R and δ ( · ) denotes the Dirac-delta distribution.
Revised: δ ( s 0 ) where A , B ∈ R and δ ( · ) denotes the Dirac-delta distribution.

Original: We deﬁne a deterministic linear policy π θ ( a t | s t ) =
Revised: We deﬁne a deterministic linear policy π θ ( a t | s t ) =

Original: δ ( θ s t | s t ) that is parameterized by the scalar θ ∈ R .
Revised: δ ( θ s t | s t ) that is parameterized by the scalar θ ∈ R .

Original: The objective is to drive the state to zero, which we encode with an exponential reward r ( s t , a t ) =
Revised: The objective is to drive the state to zero, which we encode with an exponential reward r ( s t , a t ) =

Original: exp (cid:8) − ( s t /σ r ) 2 (cid:9) .
Revised: exp (cid:8) − ( s t /σ r ) 2 (cid:9) .

Original: Further, we assume an approximate dynamics model
Revised: Further, we assume an approximate dynamics model

Original: [Equation]
Revised: [Equation]

Original: where ∆ A , ∆ B quantify the mismatch between the approximate model and the true environment.
Revised: where ∆ A , ∆ B quantify the mismatch between the approximate model and the true environment.

Original: In practice, the mismatch can arise due to noise-corrupted observations of the true state or, in the case of stochastic environments, due to a ﬁnite amount of training data.
Revised: In practice, the mismatch can arise due to noise-corrupted observations of the true state or, in the case of stochastic environments, due to a ﬁnite amount of training data.

Original: With the setting outlined above, we investigate how model errors inﬂuence the estimation of policy gradients.
Revised: With the setting outlined above, we investigate how model errors inﬂuence the estimation of policy gradients.

Original: To this end, we roll out different policies under models with varying degrees of error ∆ B .
Revised: To this end, we roll out different policies under models with varying degrees of error ∆ B .

Original: For each policy/model combination, we compute the model-based policy gradient and compare it to the true gradient.
Revised: For each policy/model combination, we compute the model-based policy gradient and compare it to the true gradient.

Original: The results are summarized in Fig.
Revised: The results are summarized in Fig.

Original: 2, where the background’s opacity depicts the gradient error’s magnitude and its color indicates whether the respective signs of the gradients are the same (blue, ≥ 0 ) or differ (red, < 0 ).
Revised: 2, where the background’s opacity depicts the gradient error’s magnitude and its color indicates whether the respective signs of the gradients are the same (blue, ≥ 0 ) or differ (red, < 0 ).

Original: For policy optimization, the sign of the gradient estimate is paramount.
Revised: For policy optimization, the sign of the gradient estimate is paramount.

Original: However, we see in the left-hand image that even small model errors can lead to the wrong sign of the gradient.
Revised: However, we see in the left-hand image that even small model errors can lead to the wrong sign of the gradient.

Original: OPC signiﬁcantly reduces the magnitude of the gradient error and increases the robustness towards model errors.
Revised: OPC signiﬁcantly reduces the magnitude of the gradient error and increases the robustness towards model errors.

Original: See also Appendix B for a more in-depth analysis.
Revised: See also Appendix C for a more in-depth analysis.

Original: 4. E VALUATION ON C ONTINUOUS C ONTROL T ASKS
Revised: 4. E VALUATION ON C ONTINUOUS C ONTROL T ASKS

Original: In the following section, we investigate the impact of OPC on a range of continuous control tasks and compare it to the current state-of-the-art model-basedand model-free RL algorithms MBPO (Janner et al., 2019) and SAC (Haarnoja et al., 2018).
Revised: In the following section, we investigate the impact of OPC on a range of continuous control tasks. To this end, we build upon the current state-of-the-art model-based RL algorithm MBPO (Janner et al., 2019).

Original: Further, we aim to answer the important question about how data diversity affects MBRL, and OPC in particular.
Revised: Further, we aim to answer the important question about how data diversity affects MBRL, and OPC in particular.

Original: While having a model that can generate (theoretically) an inﬁnite amount of data is intriguing, thethe beneﬁt of having more data is limited 
Revised: While having a model that can generate (theoretically) an inﬁnite amount of data is intriguing, the beneﬁt of having more data is limited by the model quality in terms

Original: [Figure]
Revised: [Figure]

Original:  J o C o
Revised: M u J o C o

Original: E v a l u a ti on r [Figure] t u r n ×
Revised: E v a l u a ti on r e t u r n ×

Original: [Figure]
Revised: [Figure]

Original: HalfCheetah
Revised: HalfCheetah

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Hopper
Revised: Hopper

Original: [Figure]
Revised: [Figure]

Original: Walker2d
Revised: Walker2d

Original: [Figure]
Revised: [Figure]

Original: AntTruncatedObs
Revised: AntTruncatedObs

Original: [Figure]
Revised: [Figure]

Original:  y [Figure] u ll e t
Revised: P y B u ll e t

Original: E v a l u a ti
Revised: E v a l u a ti

Original: [Figure]
Revised: [Figure]

Original: r e t 
Revised: r e t u r n ×

Original: [Figure]
Revised: [Figure]

Original: by the model quality in terms of being representative of the true environment.
Revised: of being representative of the true environment.

Original: For OPC , the following questions arise from this consideration: Do longer rollouts help to generate better data?
Revised: For OPC , the following questions arise from this consideration: Do longer rollouts help to generate better data?

Original: And is there a limit to the value of simulated transition data, i.e., is more always better?
Revised: And is there a limit to the value of simulated transition data, i.e., is more always better?

Original: For the dynamics model ˜ p model , we follow Chua et al.
Revised: For the dynamics model ˜ p model , we follow Chua et al.

Original: ; Janner et al.
Revised: ; Janner et al.

Original: and use a probabilistic ensemble of neural networks, where each head predicts a Gaussian distribution over the next state and reward.
Revised: and use a probabilistic ensemble of neural networks, where each head predicts a Gaussian distribution over the next state and reward.

Original: For policy optimization, we employ the soft actor critic ( SAC ) algorithm by Haarnoja et al.
Revised: For policy optimization, we employ the soft actor critic ( SAC ) algorithm by Haarnoja et al.

Original: All learning curves are presented in terms of the median (lines) and interquartile range (shaded region) across ten independent experiments, where we smooth the evaluation return with a moving average ﬁlter to accentuate the results of particularly noisy environments.
Revised: All learning curves are presented in terms of the median (lines) and interquartile range (shaded region) across ten independent experiments, where we smooth the evaluation return with a moving average ﬁlter to accentuate the results of particularly noisy environments.

Original: Apart from small variations in the hyperparameters, the only difference between OPC and MBPO is that our method uses ˜ p opc , while MBPO uses ˜ p model .
Revised: Apart from small variations in the hyperparameters, the only difference between OPC and MBPO is that our method uses ˜ p opc , while MBPO uses ˜ p model .

Revised: We provide pseudo-code for the model rollouts of MBPO and

Revised: OPC in Algorithm 2 in Appendix A.1.

Original: Generally, we found that OPC was more robust to the choice of hyperparameters.
Revised: Generally, we found that OPC was more robust to the choice of hyperparameters.

Original: The rollout horizon to generate training data is set to H = 10 for all experiments.
Revised: The rollout horizon to generate training data is set to H = 10 for all experiments.

Original: Note that when using ˜ p data to generate data, we retain the standard (model-free) SAC algorithm.
Revised: Note that when using ˜ p data to generate data, we retain the standard (model-free) SAC algorithm.

Original: Our implementation is based upon the code from Janner et al.
Revised: Our implementation is based upon the code from Janner et al.

Original: We made the following changes to the original implementation: 1)
Revised: We made the following changes to the original implementation: 1)

Original: The policy is only updated at the end of an epoch, not during rollouts on the true environment.
Revised: The policy is only updated at the end of an epoch, not during rollouts on the true environment.

Original: The replay buffer retains data for a ﬁxed number of episodes, to clearly distinguish on- and off-policy data.
Revised: The replay buffer retains data for a ﬁxed number of episodes, to clearly distinguish on- and off-policy data.

Original: For policy optimization, MBPO uses a small number of environment transitions in addition to those from the model.
Revised: For policy optimization, MBPO uses a small number of environment transitions in addition to those from the model.

Original: We found that this design choice did not consistently improve performance and added an additional level of complexity.
Revised: We found that this design choice did not consistently improve performance and added another level of complexity.

Original: Therefore, we refrain from mixing environment and model transitions and only use simulated data for policy optimization.
Revised: Therefore, we refrain from mixing environment and model transitions and only use simulated data for policy optimization.

Original: While we stay true to the key ideas of MBPO under these changes, we denote our variant as MBPO ( (cid:63) ) to avoid ambiguity.
Revised: While we stay true to the key ideas of MBPO under these changes, we denote our variant as MBPO ( (cid:63) ) to avoid ambiguity.

Original: See Appendices C.6 and C.7 for a comparison to the original MBPO algorithm.
Revised: See Appendices D.6 and D.7 for a comparison to the original MBPO algorithm.

Original: Comparative Evaluation
Revised: Comparative Evaluation

Original: We begin our analysis with a comparison of our method to MBPO ( (cid:63) ) and SAC on four continuous control benchmark tasks from the MuJoCo control suite (Todorov et al., 2012) and their respective PyBullet variants (Ellenberger, 2018–2019).
Revised: We begin our analysis with a comparison of our method to MBPO ( (cid:63) ) and SAC on four continuous control benchmark tasks from the MuJoCo control suite (Todorov et al., 2012) and their respective PyBullet variants (Ellenberger, 2018–2019).

Original: The results are presented in Fig.
Revised: The results are presented in Fig.

Original: We see that the difference in performance between both methods is only marginal when evaluated on the MuJoCo environments (Fig. 3, top row).
Revised: We see that the difference in performance between both methods is only marginal when evaluated on the MuJoCo environments (Fig. 3, top row).

Original: Notably, the situation changes drastically for the PyBullet environments (Fig. 3, bottom row).
Revised: Notably, the situation changes drastically for the PyBullet environments (Fig. 3, bottom row).

Original: Here, MBPO ( (cid:63) ) exhibits little to no learning progress, whereas OPC succeeds at learning a good policy with few interactions in the environment.
Revised: Here, MBPO ( (cid:63) ) exhibits little to no learning progress, whereas OPC succeeds at learning a good policy with few interactions in the environment.

Original: One of the main differences between the environments (apart from the physics engine itself) is that the PyBullet variants have initial state distributions with signiﬁcantly larger variance.
Revised: One of the main differences between the environments (apart from the physics engine itself) is that the PyBullet variants have initial state distributions with signiﬁcantly larger variance.

Original: Inﬂuence of State Representation
Revised: Inﬂuence of State Representation

Original: In general, the success of RL algorithms should be agnostic to the way an environment represents its state.
Revised: In general, the success of RL algorithms should be agnostic to the way an environment represents its state.

Original: In robotics, joint angles ϑ are often re-parameterized by a sine/cosine transformation, ϑ (cid:55)→ [sin( ϑ ) , cos( ϑ )] .
Revised: In robotics, joint angles ϑ are often re -parameterized by a sine/cosine transformation, ϑ (cid:55)→ [sin( ϑ ) , cos( ϑ )] .

Original: We show that even for the simple CartPole environment, the parameterization of the pole’s angle has a large inﬂuence on the performance of MBRL.
Revised: We show that even for the simple CartPole envi-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: ronment, the parameterization of the pole’s angle has a large inﬂuence on the performance of MBRL.

Original: In particular, we compare OPC and MBPO ( (cid:63) ) on the RoboSchool and PyBullet variants of the CartPole environment, which represent the pole’s angle with and without the sine/cosine transformation.
Revised: In particular, we compare OPC and MBPO ( (cid:63) ) on the RoboSchool and PyBullet variants of the CartPole environment, which represent the pole’s angle with and without the sine/cosine transformation.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: To rule out other effects than the angle’s representation, we repeat the experiment for each implementation but transform the state to the other representation, respectively.
Revised: To rule out other effects than the angle’s representation, we repeat the experiment for each implementation but transform the state to the other representation, respectively.

Original: Notably, OPC successfully learns a policy irrespective of the state’s representation, whereas MBPO ( (cid:63) ) fails if the angle of the pole is represented by the sine/cosine transformation.
Revised: Notably, OPC successfully learns a policy irrespective of the state’s representation, whereas MBPO ( (cid:63) ) fails if the angle of the pole is represented by the sine/cosine transformation.

Original: Inﬂuence of Data Diversity
Revised: Inﬂuence of Data Diversity

Original: Here, we investigate whether multi-step predictions are in fact beneﬁcial compared to single-step predictions during the data generation process.
Revised: Here, we investigate whether multi-step predictions are in fact beneﬁcial compared to single-step predictions during the data generation process.

Original: To this end, we keep the total number of simulated transitions N for training constant, but choose different horizon lengths H = { 1 , 5 , 10 } .
Revised: To this end, we keep the total number of simulated transitions N for training constant, but choose different horizon lengths H = { 1 , 5 , 10 } .

Original: The corresponding numbers of simulated rollouts are then given by n rollout =
Revised: The corresponding numbers of simulated rollouts are then given by n rollout =

Original: N/H .
Revised: N/H .

Original: The results for N = { 20 , 40 , 100 , 200 } × 10 3 on the HalfCheetah environment are shown in Fig.
Revised: The results for N = { 20 , 40 , 100 , 200 } × 10 3 on the HalfCheetah environment are shown in Fig.

Original: First, note that more data leads to a higher asymptotic return, but after a certain point more data only leads to diminishing returns.
Revised: First, note that more data leads to a higher asymptotic return, but after a certain point more data only leads to diminishing returns.

Original: Further, the results indicate that one-step predictions are not enough to generate sufﬁciently diverse data.
Revised: Further, the results indicate that one-step predictions are not enough to generate sufﬁciently diverse data.

Original: Note that this result contradicts the ﬁndings by
Revised: Note that this result contradicts the ﬁndings by

Original: Janner et al.
Revised: Janner et al.

Original: (2019) that one-step predictions are often sufﬁcient for MBPO .
Revised: (2019) that one-step predictions are often sufﬁcient for MBPO .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have introduced on-policy corrections ( OPC ), a novel method that combines observed transition data with model-based predictions to mitigate model-bias in MBRL.
Revised: In this paper, we have introduced on-policy corrections ( OPC ), a novel method that combines observed transition data with model-based predictions to mitigate model-bias in MBRL.

Original: In particular, we extend a replay buffer with a learned model to account for state-action pairs that have not been observed on the real environment.
Revised: In particular, we extend a replay buffer with a learned model to account for state-action pairs that have not been observed on the real environment.

Original: This approach enables the generation of more realistic transition data to more closely match the true state distribution, which was further motivated theoretically by a tightened improvement bound on the expected return.
Revised: This approach enables the generation of more realistic transition data to more closely match the true state distribution, which was further motivated theoretically by a tightened improvement bound on the expected return.

Original: Empirically, we demonstrated superior performance on high-dimensional continuous control tasks as well as robustness towards state representations.
Revised: Empirically, we demonstrated superior performance on high-dimensional continuous control tasks as well as robustness towards state representations.

Original: S UPPLEMENTARY M ATERIAL
Revised: S UPPLEMENTARY M ATERIAL

Original: In the appendix we provide additional details on our method, ablation studies, and the detailed hyperparameter conﬁgurations used in the paper.
Revised: In the appendix we provide additional details on our method, ablation studies, and the detailed hyperparameter conﬁgurations used in the paper.

Original: An overview is shown below.
Revised: An overview is shown below.

Revised: Table of Contents A

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: C Motivating Example – In-depth Analysis

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: Additional Information

Original: C Additional Experimental Results
Revised: D Additional Experimental Results

Original: [Table]
Revised: [Table]

Original: Comparison with Other Baseline Algorithms

Original: [Table]
Revised: [Table]

Original: C.2 Ablation - Retain Epochs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3 Inﬂuence of State Representation:
Revised: [Table]

Original: In
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Depth Analysis . . . . . . . . . . . . . . .

Original: C.
Revised: D.

Original: [Table]
Revised: [Table]

Original: Improvement Bound: Empirical Investigation . . . . . . . . . . . . . . . . . . .C.5 Off-Policy Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.6 Implementation Changes to Original MBPO . . . . . . . . . . . . . . . . . . . .
Revised: E.1 Norm-optimal ILC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.7 Results for
Revised: E. [Table]

Original: MBPO
Revised: -based

Original: D.
Revised: RL to Norm-optimal ILC

Original: D. D.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: A I MPLEMENTATION D ETAILS AND C OMPUTATIONAL R ESOURCES A. D ETAILED A LGORITHM FOR R OLLOUTS WITH OPC

Revised: In Section 3.2 and Fig.

Revised: 1a, we have introduced the OPC transition model and how to roll out trajectories with the model.

Revised: Here, we will give more details on the algorithmic implementation for the generation of simulated data.

Revised: Algorithm 2 follows the branched rollout scheme from MBPO (Janner et al., 2019).

Original: E Hyperparameter Settings
Revised: Differences to MBPO are highlighted in blue. Generally, OPC only requires a deterministic transition function ˜ f to compute the corrective term in Line 8 in Algorithm 2. For models that include aleatoric uncertainty, we choose ˜ f ( s t , a t ) =

Revised: E s t (cid:48)

Revised: [˜ p ( s t (cid:48) | s t , a t )] .

Revised: If, in addition, the model includes epistemic uncertainty, we refer to the comment in Section 3.4 in the main part of the paper.

Revised: In practice, rollouts on the true environment are terminated early if, for instance, a particular state exceeds a user-deﬁned boundary.

Revised: Consequently, not all trajectories in the replay buffer are necessarily of length T .

Revised: Since the prediction in Line 8 requires valid transition tuples for the correction term, we additionally check in Line 10 whether the next reference state is a terminal state.

Revised: Thus, in contrast to

Revised: MBPO , for OPC we terminate the inner loop in Algorithm 2 early if either a simulated or reference state is a terminal state.

Revised: : • Set of trajectories D bn = { (ˆ s n,bt , ˆ a n,bt , ˆ s n,bt +1 ) } T − 1 t =0 for b ∈ { 1 , . . .

Revised: , B } • Environment model ˜ p ( s t (cid:48) | s t , a t ) .

Original: F Implementation and Computational Resources
Revised: Deﬁne ˜ f ( s , a ) = E s t (cid:48) [˜ p ( s t (cid:48) | s , a )] .

Revised: • Policy π θ •

Original: A T HEORETICAL A NALYSIS OF O N -P OLICY C ORRECTIONS
Revised: Prediction horizon H simulated D sim A. H YPERPARAMETER S ETTINGS

Revised: Below, we list the most important hyperparameter settings that were used to generate the results in the main paper.

Revised: [Table]

Revised: A. I MPLEMENTATION AND C OMPUTATIONAL R ESOURCES

Revised: Our implementation is based on the code from MBPO (Janner et al., 2019)

Revised: [Table]

Revised: which is open-sourced under the MIT license.

Revised: All experiments were run on an HPC cluster

Revised: [Table]

Revised: where each individual experiment used one Nvidia V100 GPU and four Intel Xeon CPUs.

Revised: All experiments (including early debugging and evaluations) amounted to a total of 84‘713 hours

Revised: [Table]

Revised: which corresponds to roughly 9.7 years if the jobs ran sequentially.

Revised: Most of this compute was required to ensure reproducibility (ten random seeds per job and ablation studies over the effects of parameters).

Revised: The Bosch Group is carbon-neutral.

Revised: Administration, manufacturing and research activities do no longer leave a carbon footprint.

Revised: This also includes GPU clusters on which the experiments have been performed.

Revised: B T HEORETICAL A NALYSIS OF O N -P OLICY C ORRECTIONS

Original: In this section, we analyze OPC from a theoretical perspective and how it affects policy improvement.
Revised: In this section, we analyze OPC from a theoretical perspective and how it affects policy improvement.

Original: Notation
Revised: Notation

Original: In the following we drop the n superscript for states and actions for ease of exposition.
Revised: In the following, we drop the n superscript for states and actions for ease of exposition.

Original: That is, ˆ s n,b
Revised: That is, ˆ s n,b

Original: = ˆ s b and ˆ a n,b
Revised: = ˆ s b and ˆ a n,b

Original: = ˆ a b .
Revised: = ˆ a b .

Original: A. G ENERAL P OLICY I MPROVEMENT B OUND
Revised: B. G ENERAL P OLICY I MPROVEMENT B OUND

Original: We begin by deriving inequality Eq.
Revised: We begin by deriving inequality Eq.

Original: (5), which serves as motivation for OPC and is the foundation for the theoretical analysis.
Revised: (4), which serves as motivation for OPC and is the foundation for the theoretical analysis.

Original: Our goal is to bound the difference in expected return for the policies before and after the policy optimization step, i.e., η n +1 − η n .
Revised: Our goal is to bound the difference in expected return for the policies before and after the policy optimization step, i.e., η n +1 − η n .

Original: Since we are considering the MBRL setting, it is natural to express the improvement bound in terms of the expected return under the model Eq.
Revised: Since we are considering the MBRL setting, it is natural to express the improvement bound in terms of the expected return under the model ˜ η

Original: (2) and thus obtain the following
Revised: and thus obtain the following

Original: [Equation]
Revised: [Equation]

Original: According to this bound, the improvement of the policy under the true environment is governed by the three terms on the LHS:
Revised: According to this bound, the improvement of the policy under the true environment is governed by the three terms on the LHS:

Original: • Model policy improvement : This term is what we are directly optimizing in MBRLEq. (2) offset by the return of the previous iteration ˜ η n , which is constant given the current policy π n .
Revised: • Model policy improvement : This term is what we are directly optimizing in MBRL offset by the return of the previous iteration ˜ η n , which is constant given the current policy π n .

Original: Assuming that we are not at an optimum, standard policy optimization algorithms guarantee that this term is non-negative.
Revised: Assuming that we are not at an optimum, standard policy optimization algorithms guarantee that this term is non-negative.

Original: • Off-policy model error :
Revised: • Off-policy model error :

Original: The last term is the difference in return for the true environment and model under the improved policy π n +1 .
Revised: The last term is the difference in return for the true environment and model under the improved policy π n +1 .

Original: This depends largely on the generalization properties of our model, since it is not trained on data under π n +1 .
Revised: This depends largely on the generalization properties of our model, since it is not trained on data under π n +1 .

Original: • On-policy model error :
Revised: • On-policy model error :

Original: This term compares the on-policy return under π n between the true environment and the model and it is zero for any model ˜ p = p .
Revised: This term compares the on-policy return under π n between the true environment and the model and it is zero for any model ˜ p = p .

Original: Since we have access to transitions from the true environment under the π n , the replay buffer Eq.
Revised: Since we have access to transitions from the true environment under the π n , the replay buffer Eq.

Original: (3) fulﬁlls this condition under certain circumstances and the on-policy model error vanishes, see Lemma 
Revised: (3) fulﬁlls this condition under certain circumstances and the on-policy model error vanishes, see Lemma 2.

Original: Note that the learned model Eq.
Revised: Note that the learned model Eq.

Original: (4) is able to generalize to unseen state-action pairs better than the replay buffer Eq.
Revised: (2) is able to generalize to unseen state-action pairs better than the replay buffer Eq.

Original: (3) and accordingly will achieve a lower off-policy model error.
Revised: (3) and accordingly will achieve a lower off-policy model error.

Original: The motivation behind OPC is therefore to combine the best of the learned model and the replay buffer to reduce both on- and off-policy model errors.
Revised: The motivation behind OPC is therefore to combine the best of the learned model and the replay buffer to reduce both on- and off-policy model errors.

Original: A. P ROPERTIES OF THE R EPLAY B UFFER
Revised: B. P ROPERTIES OF THE R EPLAY B UFFER

Original: The beneﬁt of the replay buffer Eq.
Revised: The beneﬁt of the replay buffer Eq.

Original: (3) is that it can never introduce any model-bias such that any trajectory sampled from this model is guaranteed to come from the true state distribution.
Revised: (3) is that it can never introduce any model-bias such that any trajectory sampled from this model is guaranteed to come from the true state distribution.

Original: Accordingly, if we have collected sufﬁcient data under the same policy, the on-policy model error vanishes.
Revised: Accordingly, if we have collected sufﬁcient data under the same policy, the on-policy model error vanishes.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let M be the true MDP with (stochastic) dynamics p , bounded reward r < r max and let ˜ p data be the transition model for the replay buffer Eq.
Revised: Let M be the true MDP with (stochastic) dynamics p , bounded reward r < r max and let ˜ p data be the transition model for the replay buffer Eq.

Original: Further, consider a set of trajectories
Revised: Further, consider a set of trajectories

Original: D = (cid:83)
Revised: D = (cid:83)

Original: Bb =1 { (ˆ s bt , ˆ a bt , ˆ s bt +1 ) } T − 1 t =0 collected from M under policy π .
Revised: Bb =1 { (ˆ s bt , ˆ a bt , ˆ s bt +1 ) } T − 1 t =0 collected from M under policy π .

Original: If we collect more and more on-policy training data under the same policy, then
Revised: If we collect more and more on-policy training data under the same policy, then

Original: [Equation]
Revised: [Equation]

Original: where ˜ η replay is the expected return under the replay buffer using the collected trajectories D .
Revised: where ˜ η replay is the expected return under the replay buffer using the collected trajectories D .

Original: Proof.
Revised: Proof.

Original: First, note that the corresponding expected return for the replay-buffer model is given by
Revised: First, note that the corresponding expected return for the replay-buffer model is given by

Original: [Equation]
Revised: [Equation]

Original: which is a sample-based approximation of the true reward η .
Revised: which is a sample-based approximation of the true reward η .

Original: By the weak law of large numbers (see e.g., Blitzstein & Hwang (2019, Theorem 10.2.2)), the Lemma then holds.
Revised: By the weak law of large numbers (see e.g., Blitzstein & Hwang (2019, Theorem 10.2.2)), the Lemma then holds.

Original: A. P ROPERTIES OF THE L EARNED M ODEL
Revised: B. P ROPERTIES OF THE L EARNED M ODEL

Original: Following Janner et al.
Revised: Following Janner et al.

Original: (2019, Lemma B.3), a general bound on the performance gap between two
Revised: (2019, Lemma B.3), a general bound on the performance gap between two

Original: MDPs with different dynamics can be given by
Revised: MDPs with different dynamics can be given by

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) m ≥ max t
Revised: where (cid:15) m ≥ max t

Original: E s ∼ p t 1 ( s ) KL( p 1 ( s (cid:48) , a ) ||
Revised: E s ∼ p t 1 ( s ) KL( p 1 ( s (cid:48) , a ) ||

Original: p 2 ( s (cid:48) , a )) bounds the mismatch between the respective transition models.
Revised: p 2 ( s (cid:48) , a )) bounds the mismatch between the respective transition models.

Original: Now, the ﬁnal form of Eq.
Revised: Now, the ﬁnal form of Eq.

Original: (11) depends on the horizon length H .
Revised: (10) depends on the horizon length H .

Original: For H → ∞ , we obtain the original result form Janner et al.
Revised: For H → ∞ , we obtain the original result form Janner et al.

Original: (2019) with (cid:80) t ≥ 1 tγ t =
Revised: (2019) with (cid:80) t ≥ 1 tγ t =

Original: γ/ (1 − γ ) 2 .
Revised: γ/ (1 − γ ) 2 .

Original: For the ﬁnite horizon case and H being smaller than the effective horizon encoded in the discount factor H < γ/ (1 − γ ) ,  the following bound holds:
Revised: For the ﬁnite horizon case one can obtain tigther bounds when H is smaller than the effective horizon, H < γ/ (1 − γ ) , encoded in the discount factor:

Original: [Equation]
Revised: [Equation]

Original: which can beeasily veriﬁed by upper bounding t ≤ H to obtain H/ (1 − γ ) or by bounding γ ≤ 1 to obtain
Revised: which can be veriﬁed by upper bounding t ≤ H to obtain H/ (1 − γ ) or by bounding γ ≤ 1 to obtain

Original: O ( H 2 ) .
Revised: O ( H 2 ) .

Original: Note that this bound is vacuous for deterministic policies, since the KL divergence between two distributions with non-overlapping support is inﬁnite.
Revised: Note that this bound is vacuous for deterministic policies, since the KL divergence between two distributions with non-overlapping support is inﬁnite.

Original: In the following we focus on the Wasserstein metric under the euclidean distance.
Revised: In the following we focus on the Wasserstein metric under the Euclidean distance

Original: A. P ROPERTIES OF OPC
Revised: . B. P ROPERTIES OF OPC

Original: In this section, we analyze the properties of OPC relative to the true, unknown environment’s transition distribution p and a learned representation ˜ p .
Revised: In this section, we analyze the properties of OPC relative to the true, unknown environment’s transition distribution p and a learned representation ˜ p .

Original: In general, the OPC -model mixes observed transitions from the environment with the learned model.
Revised: In general, the OPC -model mixes observed transitions from the environment with the learned model.

Original: The resulting transitions are then a combination of the mean transitions from the learned model, the aleatoric noise from the data (the environment), and the mean-error between our learned model and the environment.
Revised: The resulting transitions are then a combination of the mean transitions from the learned model, the aleatoric noise from the data (the environment), and the mean-error between our learned model and the environment.

Original: A.4. P ROOF OF L EMMA
Revised: B.4. P ROOF OF L EMMA

Original: In this section we proof Lemma 1 by showing that the OPC -model coincides with the replay-buffer
Revised: In this section, we prove Lemma 1 by showing that the OPC -model coincides with the replay-buffer

Original: Eq.
Revised: Eq.

Original: (3) in the case of a deterministic policy and thus lead to the same expected return ˜ η .
Revised: (3) in the case of a deterministic policy and thus lead to the same expected return ˜ η .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let M be the true MDP with (stochastic) dynamics p and let ˜ M be a MDP with the same reward function r and initial state distribution ρ 0 , but different dynamics ˜ p model , respectively.
Revised: Let M be the true MDP with (stochastic) dynamics p and let ˜ M be a MDP with the same reward function r and initial state distribution ρ 0 , but different dynamics ˜ p model , respectively.

Original: Further, assume a deterministic policy π : S (cid:55)→ A and a set of trajectories D
Revised: Further, assume a deterministic policy π : S (cid:55)→ A and a set of trajectories D

Original: = (cid:83) Bb =1 { (ˆ s bt , ˆ a bt , ˆ s bt +1 )
Revised: = (cid:83) Bb =1 { (ˆ s bt , ˆ a bt , ˆ s bt +1 )

Original: } T − 1 t = collected from M under π .
Revised: } T − 1 t = collected from M under π .

Original: If we extend the approximate dynamics ˜ p model by OPC with data D , then
Revised: If we extend the approximate dynamics ˜ p model by OPC with data D , then

Original: [Equation]
Revised: [Equation]

Original: where ˜ η replay and ˜ η opc are the model-based returns following models Eqs.
Revised: where ˜ η replay and ˜ η opc are the model-based returns following models Eqs.

Original: (3) and (6) , respectively.
Revised: (3) and (5) , respectively.

Original: Proof.
Revised: Proof.

Original: For the proof, it sufﬁces to show that the resulting state distributions of the two transition models ˜ p data and ˜ p opc under the deterministic policy π are the same for all b with 1 ≤ b ≤ B .
Revised: For the proof, it sufﬁces to show that the resulting state distributions of the two transition models ˜ p data and ˜ p opc under the deterministic policy π are the same for all b with 1 ≤ b ≤ B .

Original: We show this by induction:
Revised: We show this by induction:

Original: Theorem
Revised: Theorem

Original: Lemma Lemma
Revised: Lemma Lemma

Original: Lemma Lemma
Revised: Lemma Lemma

Original: Lemma Lemma
Revised: Lemma Lemma

Original: Lemma Lemma
Revised: Lemma Lemma

Original: Corollary
Revised: Corollary

Original: Theorem
Revised: Theorem

Original: Lemma Lemma
Revised: Lemma Lemma

Original: For t = 0 the states are sampled from the initial state distribution, thus we have s 

Revised: For t = 0 the states are sampled from the initial state distribution, thus we have s

Original: =
Revised: =

Original: ˆ s b 0 by deﬁnition.
Revised: ˆ s b 0 by deﬁnition.

Original: Assume that s t
Revised: Assume that s t

Original: = ˆ s bt as an induction hypothesis.
Revised: = ˆ s bt as an induction hypothesis.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where the second step follows by the induction hypothesis and due to the deterministic policy.
Revised: where the second step follows by the induction hypothesis and due to the deterministic policy.

Original: Thus, for any index b we have τ opc b = τ b and the result follows.
Revised: Thus, for any index b we have τ opc b = τ b and the result follows.

Original: Now, combining Lemmas 2 and 3 proofs the result in Lemma 1.
Revised: Now, combining Lemmas 2 and 3 proofs the result in Lemma 1.

Original: A.4. P ROOF OF T HEOREM
Revised: B.4. P ROOF OF T HEOREM

Revised: In this section we prove our main result.

Revised: An overview of the lemma dependencies is shown in Fig.

Original: Remark on Notation
Revised: Notation

Original: In the main paper, we unify the notation for state sequence probabilities of the different models Eqs.
Revised: In the main paper, we unify the notation for state sequence probabilities of the different models Eqs.

Original: (3), (4) and (6) as ˜ p
Revised: (2), (3) and (5) as ˜ p

Original: x ( τ t : t + H | t, b ) with x ∈ { replay , model , opc } .
Revised: x ( τ t : t + H | t, b ) with x ∈ { replay , model , opc } .

Original: This allows for a consistent description of the respective rollouts independent of the model being a learned representation, a replay buffer or the OPC -model.
Revised: This allows for a consistent description of the respective rollouts independent of the model being a learned representation, a replay buffer or the OPC -model.

Original: For that notation, the index b denotes the sampled trajectory from the collected data on the real environment.
Revised: For that notation, the index b denotes the sampled trajectory from the collected data on the real environment.

Original: Implicitly, we therefore condition the state sequence probability on the observed transition tuples [(ˆ s bt +1 , ˆ s bt , ˆ a bt ) , . . .
Revised: Implicitly, we therefore condition the state sequence probability on the observed transition tuples [(ˆ s bt +1 , ˆ s bt , ˆ a bt ) , . . .

Original: , (ˆ s bt + H , ˆ s bt + H − 1 , ˆ a bt + H − 1 )] , i.e.,
Revised: , (ˆ s bt + H , ˆ s bt + H − 1 , ˆ a bt + H − 1 )] , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where we omit the explicit conditioning for the sake of brevity in the main paper.
Revised: where we omit the explicit conditioning for the sake of brevity in the main paper.

Original: Similarly, we can write the one-step model Eq.
Revised: Similarly, we can write the one-step model Eq.

Original: (6) for OPC in an explicit form as
Revised: (5) for OPC in an explicit form as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that with the explicit notation, the relation between the OPC -model Eq.
Revised: Note that with the explicit notation, the relation between the OPC -model Eq.

Original: (6) and generalized OPCmodel Eq.
Revised: (5) and generalized OPCmodel Eq.

Original: (7) becomes clear:
Revised: (6) becomes clear:

Original: [Equation]
Revised: [Equation]

Original: For the following proofs, wewill stay with the explicit notation for sake of clarity and instead omit the conditioning on b .
Revised: For the following proofs, we stay with the explicit notation for sake of clarity and instead omit the conditioning on b .

Original: Generalized OPC -Model
Revised: Generalized OPC -Model

Original: In this section, we have a closer look at the generalized OPC -model
Revised: In this section, we have a closer look at the generalized OPC -model

Original: Eq.
Revised: Eq.

Original: The main difference between Eqs.
Revised: The main difference between Eqs.

Original: (6) and (7) is that the former is in fact transitioning deterministically (the stochasticity arises from the environment’s aleatoric uncertainty which manifests itself in the observed transitions).
Revised: (5) and (6) is that the former is in fact transitioning deterministically (the stochasticity arises from the environment’s aleatoric uncertainty which manifests itself in the observed transitions).

Original: The two models can be related via marginalization of ˆ s bt +1 , see
Revised: The two models can be related via marginalization of ˆ s bt +1 , see

Original: Eq.
Revised: Eq.

Original: The resulting generalized OPC -model can then be related to the true transition distribution according to the following Lemma.
Revised: The resulting generalized OPC -model can then be related to the true transition distribution according to the following Lemma.

Original: Lemma 4.
Revised: Lemma 4.

Original: For all ˆ s bt , ˆ a bt ∈ S × A with ˆ s bt +1 ∼ p (ˆ s t +1 | ˆ s bt , ˆ a bt ) it holds that
Revised: For all ˆ s bt , ˆ a bt ∈ S × A with ˆ s bt +1 ∼ p (ˆ s t +1 | ˆ s bt , ˆ a bt ) it holds that

Original: [Equation]
Revised: [Equation]

Original: where ˜ f
Revised: where ˜ f

Original: ( s t , a t ) =
Revised: ( s t , a t ) =

Original: E [˜ p model ( s t +1 | s t , a t )] is the mean transition of the learned model Eq.
Revised: E [˜ p model ( s t +1 | s t , a t )] is the mean transition of the learned model Eq.

Original: (4) and ˜ p opc ( s t +1 | s t , a t , ˆ s bt , ˆ a bt ) denotes the OPC -model if we marginalize over the distribution for ˆ s bt + instead of using its observed value.
Revised: (2) and ˜ p opc ( s t +1 | s t , a t , ˆ s bt , ˆ a bt ) denotes the OPC -model if we marginalize over the distribution for ˆ s bt + instead of using its observed value.

Original: Proof.
Revised: Proof.

Original: Using the explicit notation Eq.
Revised: Using the explicit notation Eq.

Original: (14), the OPC -model from Eq.
Revised: (13), the OPC -model from Eq.

Original: (6) is deﬁned as
Revised: (5) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: With ˆ s bt +1 ∼ p (ˆ s t +1 | ˆ s bt , ˆ a bt ) , marginalizing ˆ s t +1 yields (note that we use ˆ s t +1 instead of s t +1 to denote the random variable for the next state in order to distinguish it from the random variable for ˜ p opc ( s t +1 | s t , a t , b ) under the integral)˜ p opc [Equation] +1 | s t
Revised: With ˆ s bt +1 ∼ p (ˆ s t +1 | ˆ s bt , ˆ a bt ) , marginalizing ˆ s t +1 yields (note that we use ˆ s t +1 instead of s t +1 to denote the random variable for the next state in order to distinguish it from the random variable for ˜ p opc ( s t +1 | s t , a t , b ) under the integral)

Original: [Equation]
Revised: [Equation]

Original: a t

Original: [Equation]

Original: Remark 1 .
Revised: Remark 1 .

Original: An alternative way of writing the general OPC -model is the following,
Revised: An alternative way of writing the general OPC -model is the following,

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: which highlights that the ˜ p opc transitions according to the true on-policy dynamics conditioned on data from the replay buffer, combined with a correction term.
Revised: which highlights that the ˜ p opc transitions according to the true on-policy dynamics conditioned on data from the replay buffer, combined with a correction term.

Original: We can further explicitly see why an implementation of this model wouldn’t be possible due to its dependency on the true transition probabilities.
Revised: We can further explicitly see why an implementation of this model wouldn’t be possible due to its dependency on the true transition probabilities.

Original: Thus, in practice, we’re limited to the sample-based approximation shown in the paper.
Revised: Thus, in practice, we’re limited to the sample-based approximation shown in the paper.

Original: The fundamental idea for the proof of Theorem 1 lies in the following Lemma 5, which is the foundation for bounding the on-policy error.
Revised: The fundamental idea for the proof of Theorem 1 lies in the following Lemma 5, which is the foundation for bounding the on-policy error.

Original: The Wasserstein distance naturally arises in bounding this type of error model as it depends on the expected return under two different distributions.
Revised: The Wasserstein distance naturally arises in bounding this type of error model as it depends on the expected return under two different distributions.

Original: The ﬁnal result is then summarized in Theorem 1.
Revised: The ﬁnal result is then summarized in Theorem 1.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let ˜ p opc be the generalized OPC -model
Revised: Let ˜ p opc be the generalized OPC -model

Original: (cf.
Revised: (cf.

Original: Lemma 4) and ˜
Revised: Lemma 4) and ˜

Original: η opc be its corresponding expected return.
Revised: η opc be its corresponding expected return.

Original: Assume that the return r ( s t , a t ) is L r -Lipschitz and the policy π ( a t | s t ) is
Revised: Assume that the return r ( s t , a t ) is L r -Lipschitz and the policy π ( a t | s t ) is

Original: L π -Lipschitz with respect to s t under the Wasserstein distance, then
Revised: L π -Lipschitz with respect to s t under the Wasserstein distance, then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma 8:
Revised: Applying Lemma 8:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Writing the joint distributions for state/action in terms of their conditional (i
Revised: Writing the joint distributions for state/action in terms of their conditional (i

Original: [Equation]
Revised: [Equation]

Original: e., policy) and marginal distributions p ( s t , a t ) =
Revised: e., policy) and marginal distributions p ( s t , a t ) =

Original: π ( a t | s t ) p ( s t ) :
Revised: π ( a t | s t ) p ( s t ) :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that the policies π are L π -Lipschitz under the Wasserstein distance, application of Lemma 10 concludes the proof:
Revised: Under the assumption that the policies π are L π -Lipschitz under the Wasserstein distance, application of Lemma 10 concludes the proof:

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 (Wasserstein Distance between Marginal State Distributions) .
Revised: Lemma 6 (Wasserstein Distance between Marginal State Distributions) .

Original: Let ˜ p opc ( s t ) and p (ˆ s t ) be the marginal state distributions at time t when rolling out from the same initial state ˆ s 0 under the same policy with the OPC -model and the true environment, respectively.
Revised: Let ˜ p opc ( s t ) and p (ˆ s t ) be the marginal state distributions at time t when rolling out from the same initial state ˆ s 0 under the same policy with the OPC -model and the true environment, respectively.

Original: Assume that the underlying learned dynamics model is L f -Lipschitz continuous with respect to both its arguments and the policy π ( a | s ) is L π -Lipschitz with respect to s under the Wasserstein distance.
Revised: Assume that the underlying learned dynamics model is L f -Lipschitz continuous with respect to both its arguments and the policy π ( a | s ) is L π -Lipschitz with respect to s under the Wasserstein distance.

Original: If it further holds that the policy’s (co-)variance Var[ π ( a | s )]
Revised: If it further holds that the policy’s (co-)variance Var[ π ( a | s )]

Original: =
Revised: =

Original: Σ π ( s ) ∈ S
Revised: Σ π ( s ) ∈ S

Original: d A
Revised: d A

Original: + is ﬁnite over the complete state space, i.e., max s ∈S trace { Σ π ( s ) } ≤ ¯ σ 2 π , then the discrepancy between the marginal state distributions of the two models is bounded
Revised: + is ﬁnite over the complete state space, i.e., max s ∈S trace { Σ π ( s ) } ≤ ¯ σ 2 π , then the discrepancy between the marginal state distributions of the two models is bounded

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We proof the Lemma by induction.
Revised: We proof the Lemma by induction.

Original: Base Case: t =
Revised: Base Case: t =

Original: For the base case, we need to show that starting from the same initial state ˆ sthe following condition holds:
Revised: For the base case, we need to show that starting from the same initial state ˆ sthe following condition holds:

Original: [Equation]
Revised: [Equation]

Original: For ease of readability, we deﬁne z =
Revised: For ease of readability, we deﬁne z =

Original: ( s , a ) and use the notation d p ( x , y ) =
Revised: ( s , a ) and use the notation d p ( x , y ) =

Original: p ( x , y )
Revised: p ( x , y )

Original: d x d y whenever no explicit assumptions are made about the distributions.
Revised: d x d y whenever no explicit assumptions are made about the distributions.

Original: [Equation]
Revised: [Equation]

Original: Recall that we can write both ˜ p opc and p as convolution between p (cid:15) and a Dirac delta (see Lemma 4).
Revised: Recall that we can write both ˜ p opc and p as convolution between p (cid:15) and a Dirac delta (see Lemma 4).

Original: Together with the identity Eq.
Revised: Together with the identity Eq.

Original: (55), the Wasserstein distance for sums of random variables Eq.
Revised: (54), the Wasserstein distance for sums of random variables Eq.

Original: (54) and noting W p ( p (cid:15) ( (cid:15) ) , p (cid:15) ( (cid:15)
Revised: (53) and noting W p ( p (cid:15) ( (cid:15) ) , p (cid:15) ( (cid:15)

Original: [Equation]
Revised: [Equation]

Original: ) = 0 : ≤ W(cid:18)(cid:90)(cid:90)
Revised: = 0 : ≤ W(cid:18)(cid:90)(cid:90)

Original: [Equation]
Revised: [Equation]

Original: Squaring and using Lemma 9:
Revised: Squaring and using Lemma 9:

Original: [Equation]
Revised: [Equation]

Original: We are assuming that the initial states of the trajectory rollouts coincide.
Revised: We are assuming that the initial states of the trajectory rollouts coincide.

Original: The joint state/action distribution can then be written as p (ˆ s 0 , ˆ a[Equation] s
Revised: The joint state/action distribution can then be written as p (ˆ s 0 , ˆ a

Original: [Equation]
Revised: [Equation]

Original: Integrating with respect to s 0 leads to:
Revised: with respect to s 0 leads to:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This term describes the mean squared distance between two random actions.
Revised: This term describes the mean squared distance between two random actions.

Original: Since we condition π on the same state ˆ s 0 , the policy distributions coincide.
Revised: Since we condition π on the same state ˆ s 0 , the policy distributions coincide.

Original: Deﬁne ∆ a = 
Revised: Deﬁne ∆ a = a

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: This term is in fact less than Eq.
Revised: This term is in fact less than Eq.

Original: (28) for t = 0 , thus prooﬁng the base case.
Revised: (27) for t = 0 , thus prooﬁng the base case.

Original: [Equation]
Revised: [Equation]

Original: Inductive Step
Revised: Inductive Step

Original: We will show that if the hypothesis holds for t then it holds for t
Revised: We will show that if the hypothesis holds for t then it holds for t

Original: + 1 as well.
Revised: + 1 as well.

Original: We explicitly write the following intermediate bound such that its application in the proof is more apparent,
Revised: We explicitly write the following intermediate bound such that its application in the proof is more apparent,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality immediately follows from the same reasoning as in the base case Eq.
Revised: where the ﬁrst inequality immediately follows from the same reasoning as in the base case Eq.

Original: Eq.
Revised: Eq.

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma 11 to the second integral
Revised: Applying Lemma 11 to the second integral

Original: [Equation]
Revised: [Equation]

Original: We predict along a consistent trajectory,
Revised: We predict along a consistent trajectory,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: , s t
Revised: , s t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assume that the hypothesis Eq.
Revised: Assume that the hypothesis Eq.

Original: (40) holds for t
Revised: (39) holds for t

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ˜ η opc (cid:63) and η be the expected return under the generalized OPC -model Eq.
Revised: Let ˜ η opc (cid:63) and η be the expected return under the generalized OPC -model Eq.

Original: (7) and the true environment, respectively.
Revised: (6) and the true environment, respectively.

Original: Assume that the learned model’s mean transition function ˜ f
Revised: Assume that the learned model’s mean transition function ˜ f

Original: ( s t , a t ) =
Revised: ( s t , a t ) =

Original: E [˜ p model ( s t +1 | s t , a t )] is L f -Lipschitz and the reward r ( s t , a t ) is L r -Lipschitz.
Revised: E [˜ p model ( s t +1 | s t , a t )] is L f -Lipschitz and the reward r ( s t , a t ) is L r -Lipschitz.

Original: Further, if the policy π ( a t | s t ) is L π -Lipschitz with respect to s t under the Wasserstein distance and its (co-)variance Var[ π ( a t
Revised: Further, if the policy π ( a t | s t ) is L π -Lipschitz with respect to s t under the Wasserstein distance and its (co-)variance Var[ π ( a t

Original: | s t )]
Revised: | s t )]

Original: = Σ π ( s t )
Revised: = Σ π ( s t )

Original: ∈ S
Revised: ∈ S

Original: d A
Revised: d A

Original: + is ﬁnite over the complete state space, i.e., max s t ∈S trace { Σ π ( s t ) } ≤ ¯ σ 2 π , then with C 1 = (cid:112)
Revised: + is ﬁnite over the complete state space, i.e., max s t ∈S trace { Σ π ( s t ) } ≤ ¯ σ 2 π , then with C 1 = (cid:112)

Original: [Equation]
Revised: [Equation]

Original: 1 + L 2 π[-) L f L r and C 
-]
Revised: 1 + L 2 π[-) L f L r and C 
-]

Original: =
Revised: [Equation]

Original:  (cid:113) 
Revised: L f L r and C 2 = (cid:113) L 2 f

Original: L 2 f + L 2 π 
Revised: + L 2 π [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From combining Lemmas 5 and 6 it follows that
Revised: From combining Lemmas 5 and 6 it follows that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  with the shorthand notations C 1 = (cid:112) 2(

Revised: 49) with the shorthand notations C 1 = (cid:112) 2(

Original: + L 2 π ) L f L r and C 

Revised: + L 2 π ) L f L r and C

Original: =
Revised: =

Original: L 2 f + L 2 π
Revised: L 2 f + L 2 π

Original: [Equation]
Revised: [Equation]

Original: Since t ≤ T , we have that (cid:80) tt (cid:48)
Revised: Since t ≤ T , we have that (cid:80) tt (cid:48)

Original: ={+=
+}
Revised: {+=
+}

Original: C t(cid:48)
Revised: C t

Original: [Equation]
Revised: [Equation]

Original: Since T ≤ ∞ , we have with the geometric series (cid:80) T t =

Revised: Since T ≤ ∞ , we have with the geometric series (cid:80) T t =

Original: γ t ≤ 1 / (1 − γ )
Revised: γ t ≤ 1 / (1 − γ )

Original: [Equation]
Revised: [Equation]

Original: A.4. D EFINITIONS , H ELPFUL I DENTITIES AND S UPPORTING L EMMAS
Revised: B.4. D EFINITIONS , H ELPFUL I DENTITIES AND S UPPORTING L EMMAS

Original: Here we brieﬂy summarize some basic deﬁnitions and properties that will be used throughout the following.
Revised: Here we brieﬂy summarize some basic deﬁnitions and properties that will be used throughout the following.

Original: • The Wasserstein distance fulﬁlls the properties of a metric: W p ( p 1 , p 3 ) ≤ W p ( p 1 , p 2 ) + W p ( p 2 , p 3 ) .
Revised: • The Wasserstein distance fulﬁlls the properties of a metric: W p ( p 1 , p 3 ) ≤ W p ( p 1 , p 2 ) + W p ( p 2 , p 3 ) .

Original: • Wasserstein distance of sums of random variables (see, e.g., Mariucci et al. (2018, Corollary 1) for a proof):
Revised: • Wasserstein distance of sums of random variables (see, e.g., Mariucci & Reiß (2018, Corollary 1) for a proof):

Original: [Equation]
Revised: [Equation]

Original: • For any function g ( z , ˆ z ) we have
Revised: • For any function g ( z , ˆ z ) we have

Original: [Equation]
Revised: [Equation]

Original: • For any multivariate random variable z 1 and z 2 with probability distributions p ( z 1 ) =
Revised: • For any multivariate random variable z 1 and z 2 with probability distributions p ( z 1 ) =

Original: p 1 ( x ) q ( y ) and p ( z 2 ) =
Revised: p 1 ( x ) q ( y ) and p ( z 2 ) =

Original: p 2 ( x ) q ( y ) , respectively, we have that (Panaretos
Revised: p 2 ( x ) q ( y ) , respectively, we have that (Panaretos

Original: & Zemel
Revised: Zemel

Original: [Equation]
Revised: [Equation]

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: the following Lemmas are helpful for the proof of Theorem 1.
Revised: the following Lemmas are helpful for the proof of Theorem 1.

Original: Lemma 7 (Kantorovich-Rubinstein (cf.
Revised: Lemma 7 (Kantorovich-Rubinstein (cf.

Original: Mariucci et al. Proposition 1.3)) .
Revised: Mariucci & Reiß (2018) Proposition 1.3)) .

Original: Let X and Y be integrable real random variables.
Revised: Let X and Y be integrable real random variables.

Original: Denote by µ and µ their laws [. ..].
Revised: Denote by µ and µ their laws [... ].

Original: Then the following characterization of the Wasserstein distance of order 1 holds:
Revised: Then the following characterization of the Wasserstein distance of order 1 holds:

Original: [Equation]
Revised: [Equation]

Original: where the supremum is being taken over all φ satisfying the Lipschitz condition | φ ( x ) − φ ( y ) | ≤ | x − y | , for all x, y ∈ R .
Revised: where the supremum is being taken over all φ satisfying the Lipschitz condition | φ ( x ) − φ ( y ) | ≤ | x − y | , for all x, y ∈ R .

Original: Lemma 8.
Revised: Lemma 8.

Original: Let f be L f -Lipschitz with respect to a metric d .
Revised: Let f be L f -Lipschitz with respect to a metric d .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst inequality is a direct consequence of Lemma 7 and the second inequality comes from the well-known fact that if 1 ≤ p ≤ q , then W p ( µ, ν ) ≤ W q ( µ, ν ) (cf.
Revised: The ﬁrst inequality is a direct consequence of Lemma 7 and the second inequality comes from the well-known fact that if 1 ≤ p ≤ q , then W p ( µ, ν ) ≤ W q ( µ, ν ) (cf.

Original: Mariucci et al.
Revised: Mariucci & Reiß (2018,

Original: Lemma 1.2)).
Revised: Lemma 1.2)).

Original: Lemma 9.
Revised: Lemma 9.

Original: For any two functions f ( s ) and g ( s ) and probability density p ( s ) that govern the distributions deﬁned by
Revised: For any two functions f ( s ) and g ( s ) and probability density p ( s ) that govern the distributions deﬁned by

Original: [Equation]
Revised: [Equation]

Original: it holds for any q ≥ 1 that
Revised: it holds for any q ≥ 1 that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: Enforcing the following structure on γ ( ξ 1 , ξ 2 ) reduces the space of possible distributions: γ ( ξ 1 , ξ 2 )
Revised: Enforcing the following structure on γ ( ξ 1 , ξ 2 ) reduces the space of possible distributions: γ ( ξ 1 , ξ 2 )

Original: = (cid:82) δ
Revised: = (cid:82) δ

Original: [Equation]
Revised: [Equation]

Original: − f [Equation] s 
Revised: − f ( s ))

Original: [Equation]

Original: δ ( [Equation] [Equation] g ( s )
Revised: δ ( ξ 2 − g ( s ))

Original: [Equation]

Original: p [Equation] s ) d s ,
Revised: p ( s ) d s ,

Original: so that γ ( ξ 1 , ξ 2 ) ∈ Γ( p 1 , p 2 ) and thus
Revised: so that γ ( ξ 1 , ξ 2 ) ∈ Γ( p 1 , p 2 ) and thus

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Integrating over ξ 1 and ξ 2 yields
Revised: Integrating over ξ 1 and ξ 2 yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 10.
Revised: Lemma 10.

Original: If the policy π ( a t | s t ) is L π -Lipschitz with respect to s t under the Wasserstein distance, then with p ( ˆ s , ˆ a )
Revised: If the policy π ( a t | s t ) is L π -Lipschitz with respect to s t under the Wasserstein distance, then with p ( ˆ s , ˆ a )

Original: =
Revised: =

Original: π ( ˆ a | ˆ s ) p ( ˆ s ) and ˜ p ( s , a ) =
Revised: π ( ˆ a | ˆ s ) p ( ˆ s ) and ˜ p ( s , a ) =

Original: π ( a | s )˜ p ( s ) ,
Revised: π ( a | s )˜ p ( s ) ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Enforcing the following structure on γ reduces the space of possible distributions: γ (ˆ s , s , ˆ a , a ) =γ
Revised: Enforcing the following structure on γ reduces the space of possible distributions: γ (ˆ s , s , ˆ a , a ) =

Original: [Equation]

Original: ˆ s , s ) γ (ˆ a , a | ˆ s , s ) with γ (ˆ s , s ) ∈ Γ( p (ˆ s ) , ˜ p ( s )) and γ (ˆ a , a | ˆ s , s ) ∈ Γ( π (ˆ a |
Revised: γ (ˆ s , s ) γ (ˆ a , a | ˆ s , s ) with γ (ˆ s , s ) ∈ Γ( p (ˆ s ) , ˜ p ( s )) and γ (ˆ a , a | ˆ s , s ) ∈ Γ( π (ˆ a |

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: , π
Revised: , π

Original: [Equation]
Revised: [Equation]

Original: a 
Revised: a |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Interchange inﬁmum and the integral:
Revised: Interchange inﬁmum and the integral:

Original: Rockafellar (1976, Theorem 3A)
Revised: Rockafellar (1976, Theorem 3A)

Original: [Equation]
Revised: [Equation]

Original: Using the assumption that the action distribution is L π -Lipschitz continuous under the Wasserstein metric with respect to the state s :
Revised: Using the assumption that the action distribution is L π -Lipschitz continuous under the Wasserstein metric with respect to the state s :

Original: [Equation]
Revised: [Equation]

Original: Lemma 11 (Average Squared Euclidean Distance Between Actions) .
Revised: Lemma 11 (Average Squared Euclidean Distance Between Actions) .

Original: If the policy π ( a | s ) is L πLipschitz with respect to s under the Wasserstein distance and the policy’s (co-)variance Var[ π ( a | s )]
Revised: If the policy π ( a | s ) is L πLipschitz with respect to s under the Wasserstein distance and the policy’s (co-)variance Var[ π ( a | s )]

Original: =
Revised: =

Original: Σ π ( s ) ∈ S
Revised: Σ π ( s ) ∈ S

Original: d A
Revised: d A

Original: + is ﬁnite over the complete state space, i.e., max s ∈S trace { Σ π ( s ) } ≤ ¯ σ 2 π , then
Revised: + is ﬁnite over the complete state space, i.e., max s ∈S trace { Σ π ( s ) } ≤ ¯ σ 2 π , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Straightforward application of Corollary 1 and Lemma
Revised: Straightforward application of Corollary 1 and Lemma

Original: Lemma 12 (Average Squared Euclidean Distance) .
Revised: Lemma 12 (Average Squared Euclidean Distance) .

Original: Consider two random variables x , y with distributions p x , p y , mean vectors µ x , µ y ∈ R m and covariance matrices Σ x , Σ y ∈ S m + , respectively.
Revised: Consider two random variables x , y with distributions p x , p y , mean vectors µ x , µ y ∈ R m and covariance matrices Σ x , Σ y ∈ S m + , respectively.

Original: Then the average squared Euclidean distance between the two is
Revised: Then the average squared Euclidean distance between the two is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne z
Revised: Deﬁne z

Original: = x − y with mean µ z = µ x − µ y and variance Σ z = Σ x + Σ y .
Revised: = x − y with mean µ z = µ x − µ y and variance Σ z = Σ x + Σ y .

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 (Gelbrich Bound (from Kuhn et al.
Revised: Theorem 2 (Gelbrich Bound (from Kuhn et al.

Original: If (cid:107) · (cid:107) is the Euclidean norm, and the distributions p x and p y have mean vectors µ x , µ y ∈ R m and covariance matrices Σ x , Σ y ∈ S m
Revised: If (cid:107) · (cid:107) is the Euclidean norm, and the distributions p x and p y have mean vectors µ x , µ y ∈ R m and covariance matrices Σ x , Σ y ∈ S m

Original: + , respectively, then
Revised: + , respectively, then

Original: [Equation]
Revised: [Equation]

Original: The bound is exact if p x and p y are elliptical distributions with the same density generator.
Revised: The bound is exact if p x and p y are elliptical distributions with the same density generator.

Original: Corollary 1.
Revised: Corollary 1.

Original: Consider the same setting as in Lemma 12, then the average squared Euclidean distance is bounded by
Revised: Consider the same setting as in Lemma 12, then the average squared Euclidean distance is bounded by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Straightforward application of the results from Lemma 12 and Theorem 2.
Revised: Straightforward application of the results from Lemma 12 and Theorem 2.

Original: Lemma 13.
Revised: Lemma 13.

Original: If the policy’s (co-)variance Var[ π ( a | s )]
Revised: If the policy’s (co-)variance Var[ π ( a | s )]

Original: =
Revised: =

Original: Σ π ( s ) ∈ S
Revised: Σ π ( s ) ∈ S

Original: d A
Revised: d A

Original: + is ﬁnite over the complete state space, i.e., max s ∈S trace { Σ π ( s ) } ≤ ¯ σ 2 π , then
Revised: + is ﬁnite over the complete state space, i.e., max s ∈S trace { Σ π ( s ) } ≤ ¯ σ 2 π , then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The trace of a matrix is the same as the sum of its eigenvalues, and the square root of a matrix has eigenvalues that are square root of its eigenvalues.
Revised: The trace of a matrix is the same as the sum of its eigenvalues, and the square root of a matrix has eigenvalues that are square root of its eigenvalues.

Original: From Jensen’s inequality we know that (cid:80) d
Revised: From Jensen’s inequality we know that (cid:80) d

Original: A i =1 √ λ i ≤ (cid:113)
Revised: A i =1 √ λ i ≤ (cid:113)

Original: d A (cid:80) d
Revised: d A (cid:80) d

Original: A i =

Revised: A i =

Original: λ i and consequently it holds for a matrix M ∈ R d
Revised: λ i and consequently it holds for a matrix M ∈ R d

Original: A ×
Revised: A ×

Original: d A that trace { M 1 / 2 } ≤ (cid:112)
Revised: d A that trace { M 1 / 2 } ≤ (cid:112)

Original: d A trace { M } , so that
Revised: d A trace { M } , so that

Original: [Equation]
Revised: [Equation]

Original: The trace is invariant under cyclic permutation
Revised: The trace is invariant under cyclic permutation

Original: [Equation]
Revised: [Equation]

Original: Since both matrices are positive semi-deﬁnite, it follows from the Cauchy-Schwartz inequality that
Revised: Since both matrices are positive semi-deﬁnite, it follows from the Cauchy-Schwartz inequality that

Original: [Equation]
Revised: [Equation]

Original: By assumption, the covariance matrices’ traces are bounded
Revised: By assumption, the covariance matrices’ traces are bounded

Original: [Equation]
Revised: [Equation]

Original: A. M ODEL E RRORS IN OPC
Revised: M ODEL E RRORS IN OPC

Original: While Theorem 1 highlights that OPC counteracts the on-policy error in predicted performance, for stochastic policies we use the Lipschitz continuity of the model to upper-bound errors.
Revised: While Theorem 1 highlights that OPC counteracts the on-policy error in predicted performance, for stochastic policies we use the Lipschitz continuity of the model to upper-bound errors.

Original: In this section, we look at the impact of model errors in combination with OPC .
Revised: In this section, we look at the impact of model errors in combination with OPC .

Original: Speciﬁcally we focus on the one-step prediction case from a known initial state ˆ s 0 .
Revised: Speciﬁcally we focus on the one-step prediction case from a known initial state ˆ s 0 .

Original: There, while for the model ˜ p model without OPC the prediction error only depends on the quality of the model, with OPC it is instead the minimum of the model error and the policy variance.
Revised: There, while for the model ˜ p model without OPC the prediction error only depends on the quality of the model, with OPC it is instead the minimum of the model error and the policy variance.

Original: This is advantageous, since typical environments tend to have more states that actions, so that the trace of the policy variance can be signiﬁcantly smaller than the full-state model error.
Revised: This is advantageous, since typical environments tend to have more states than actions, so that the trace of the policy variance can be signiﬁcantly smaller than the full-state model error.

Original: Lemma 14.
Revised: Lemma 14.

Original: Under the assumptions of Theorem 1, starting from an initial state ˆ s 0 the following condition holds:
Revised: Under the assumptions of Theorem 1, starting from an initial state ˆ s 0 the following condition holds:

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst term in the minimum follows directly from the base-case of Lemma 6.
Revised: The ﬁrst term in the minimum follows directly from the base-case of Lemma 6.

Original: For the second term, follow the same derivation
Revised: For the second term, follow the same derivation

Original: [Equation]
Revised: [Equation]

Original: but note that under the distribution p (ˆ s 0 , ˆ a 0 , s 0 , a 0 ) =
Revised: but note that under the distribution p (ˆ s 0 , ˆ a 0 , s 0 , a 0 ) =

Original: p (ˆ s 0 ) π (ˆ a 0 | ˆ s 0 )
Revised: p (ˆ s 0 ) π (ˆ a 0 | ˆ s 0 )

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: s 0 − ˆ s 0 )
Revised: s 0 − ˆ s 0 )

Original: π ( a 0 | s 0 ) we have (cid:82) p (ˆ s 1 | ˆ z 0 )
Revised: π ( a 0 | s 0 ) we have (cid:82) p (ˆ s 1 | ˆ z 0 )

Original: d p (ˆ z 0 ) =
Revised: d p (ˆ z 0 ) =

Original: (cid:82) p ( s 1 | z 0 )
Revised: (cid:82) p ( s 1 | z 0 )

Original: d p ( z 0 ) .
Revised: d p ( z 0 ) .

Original: Inserting this into the r
Revised: Inserting this into the r

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: s. of Eq.
Revised: s. of Eq.

Original: (29) and following the same steps we obtain
Revised: (28) and following the same steps we obtain

Original: [Equation]
Revised: [Equation]

Original: Note the additional factor of two in front of the upper bound on the model error, which comes from using the model ‘twice’: once with ˆ z and once with z .
Revised: Note the additional factor of two in front of the upper bound on the model error, which comes from using the model ‘twice’: once with ˆ z and once with z .

Original: In practice we do not see any adverse effects of this error, presumably because either the variance of the policy is sufﬁciently small, or due to the upper bound being lose in practice.
Revised: In practice we do not see any adverse effects of this error, presumably because either the variance of the policy is sufﬁciently small, or due to the upper bound being lose in practice.

Original: B M OTIVATING E XAMPLE – I N - DEPTH A NALYSIS
Revised: C M OTIVATING E XAMPLE – I N - DEPTH A NALYSIS

Original: In this section, we re-visit the motivating example presented in Section 4.1 of the main paper.
Revised: In this section, we re-visit the motivating example presented in Section 4.1 of the main paper.

Original: For completeness, we re-state all assumptions that lead to the simpliﬁed system at hand.
Revised: For completeness, we re-state all assumptions that lead to the simpliﬁed system at hand.

Original: We continue with an analysis of the reward landscape and how OPC inﬂuences its shape.
Revised: We continue with an analysis of the reward landscape and how OPC inﬂuences its shape.

Original: Next, we investigate how an increasing mismatch of the dynamics model impacts the gradient error.
Revised: Next, we investigate how an increasing mismatch of the dynamics model impacts the gradient error.

Original: In addition to the result presented in the main paper, we here show the inﬂuence of different model errors, i.e., ∆ A as well as ∆ B .
Revised: In addition to the result presented in the main paper, we here show the inﬂuence of different model errors, i.e., ∆ A as well as ∆ B .

Original: While OPC is motivated for the use case of on-policy RL algorithms, we further show that the resulting gradients are robust with respect to differences in data-generating and evaluation policy, i.e., the off-policy setting.
Revised: While OPC is motivated for the use case of on-policy RL algorithms, we further show that the resulting gradients are robust with respect to differences in data-generating and evaluation policy, i.e., the off-policy setting.

Original: Lastly, we state the the signed gradient distance that we use for evaluation of the gradient errors, state the relevant theorem for determining the closed-loop stability of linear systems, as well as all numerical values used for the motivating example.
Revised: Lastly, we state the the signed gradient distance that we use for evaluation of the gradient errors, state the relevant theorem for determining the closed-loop stability of linear systems, as well as all numerical values used for the motivating example.

Original: B. S ETUP
Revised: C. S ETUP

Original: Here, we assume a linear system with deterministic dynamics
Revised: Here, we assume a linear system with deterministic dynamics

Original: [Equation]
Revised: [Equation]

Original: with A , B ∈ R and δ ( · ) denoting the Dirac-delta distribution.
Revised: with A , B ∈ R and δ ( · ) denoting the Dirac-delta distribution.

Original: The linear policy and bell-shaped reward are given by the following equations
Revised: The linear policy and bell-shaped reward are given by the following equations

Original: [Equation]
Revised: [Equation]

Original: θ

Original: [Equation]

Original: s

Original: [Equation]

Original: Further, we assume to have access to an approximate dynamics model ˜ p
Revised: Further, we assume to have access to an approximate dynamics model ˜ p

Original: [Equation]
Revised: [Equation]

Original: where ∆ A , ∆ B quantify the mismatch between the approximate model and the true system.
Revised: where ∆ A , ∆ B quantify the mismatch between the approximate model and the true system.

Original: For completeness, the (deterministic) policy gradient is deﬁned as
Revised: For completeness, the (deterministic) policy gradient is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where the state/action pairs are obtained by simulating any of the two above models for T time-steps and following policy π nθ resulting in the trajectory ˜ τ n = { ( s nt , a nt ) } T − 1 t =0 .
Revised: where the state/action pairs are obtained by simulating any of the two above models for T time-steps and following policy π nθ resulting in the trajectory ˜ τ n = { ( s nt , a nt ) } T − 1 t =0 .

Original: Because both the model and policy are deterministic, we can compute the analytical policy gradient from only one rollout.
Revised: Because both the model and policy are deterministic, we can compute the analytical policy gradient from only one rollout.

Original: B. R EWARD L ANDSCAPES
Revised: C. R EWARD L ANDSCAPES

Original: In a ﬁrst step, we will look at the cumulative rewards as a function of the policy parameter for the different systems at hand: 1) the true system, 2) the approximate model without OPC and 3) the approximate model with OPC .
Revised: In a ﬁrst step, we will look at the cumulative rewards as a function of the policy parameter for the different systems at hand: 1) the true system, 2) the approximate model without OPC and 3) the approximate model with OPC .

Original: Further, let’s assume that the model mismatch is ﬁxed to some arbitrary value.
Revised: Further, let’s assume that the model mismatch is ﬁxed to some arbitrary value.

Original: The resulting reward landscapes are shown in Fig.
Revised: The resulting reward landscapes are shown in Fig.

Original: We would like to emphasize several key aspects in the plots: First, as one would expect the model mismatch leads to different optimal policies as well as misleading policy gradients for large parts of the policy parameter space.
Revised: We would like to emphasize several key aspects in the plots: First, as one would expect the model mismatch leads to different optimal policies as well as misleading policy gradients for large parts of the policy parameter space.

Original: Second, the reward landscape for the model with OPC depends on the respective reference policy π n that was used to generate the data for the corrections.
Revised: Second, the reward landscape for the model with OPC depends on the respective reference policy π n that was used to generate the data for the corrections.

Original: Consequently, the correct reward is recovered at θ = θ n .
Revised: Consequently, the correct reward is recovered at θ = θ n .

Original: More importantly, the OPC reshape the reward landscape such that the policy gradients point towards the correct optimum (left plot).
Revised: More importantly, the OPC reshape the reward landscape such that the policy gradients point towards the correct optimum (left plot).

Original: Lastly, even when using OPC the policy gradient’s sign is not guaranteed to have the correct sign (right plot).
Revised: Lastly, even when using OPC the policy gradient’s sign is not guaranteed to have the correct sign (right plot).

Original: The extent of this effect strongly depends on the model mismatch, which we will investigate in the next section.
Revised: The extent of this effect strongly depends on the model mismatch, which we will investigate in the next section.

Original: B. I NFLUENCE OF M ODEL E RROR
Revised: C. I NFLUENCE OF M ODEL E RROR

Original: As shown in the previous section, the estimated policy gradient depends on the current policy as well as the mismatch between the true system and the approximate model.
Revised: As shown in the previous section, the estimated policy gradient depends on the current policy as well as the mismatch between the true system and the approximate model.

Original: Fig.
Revised: Fig.

Original: 2 depicts the (signed) differences between the true policy gradient as well as the approximated gradient as a function of model mismatch and the reference policy.
Revised: 2 depicts the (signed) differences between the true policy gradient as well as the approximated gradient as a function of model mismatch and the reference policy.

Original: Here, the opacity of the background denotes the magnitude of the error and the color denotes if the true and estimated gradient have the same (blue) or oppposite (red) sign.
Revised: Here, the opacity of the background denotes the magnitude of the error and the color denotes if the true and estimated gradient have the same (blue) or oppposite (red) sign.

Original: In the context of policy learning, the sign of the gradient is more relevant than the actual magnitude due to internal re-scaling of the gradients in modern implementations of stochastic optimizers such as Adam (Kingma & Ba, 2015).
Revised: In the context of policy learning, the sign of the gradient is more relevant than the actual magnitude due to internal re-scaling of the gradients in modern implementations of stochastic optimizers such as Adam (Kingma & Ba, 2015).

Original: In our example, even for negligible model errors (either in ∆ A or ∆ B ), the model-based approach can lead to gradient estimates with the opposite sign, indicated by the large red areas for the left ﬁgures in Fig.
Revised: In our example, even for negligible model errors (either in ∆ A or ∆ B ), the model-based approach can lead to gradient estimates with the opposite sign, indicated by the large red areas for the left ﬁgures in Fig.

Original: On the other hand, applying OPC to the model, we gradient estimates are signiﬁcantly more robust with respect to errors in the dynamics.x
Revised: On the other hand, applying OPC to the model, we gradient estimates are signiﬁcantly more robust with respect to errors in the dynamics.x

Original: [Figure]
Revised: [Figure]

Original: B. I NFLUENCE OF O FF -P OLICY E RROR
Revised: C. I NFLUENCE OF O FF -P OLICY E RROR

Original: Until now we have considered the case in which the reference trajectory used for OPC is generated with the same policy as the one used for gradient estimation, i.e., the on-policy setting.
Revised: Until now we have considered the case in which the reference trajectory used for OPC is generated with the same policy as the one used for gradient estimation, i.e., the on-policy setting.

Original: In this case, we have observed that the true return could be recovered (see Fig. 7) when using OPC and that the gradient estimates are less sensitive to model errors (see Fig.
Revised: In this case, we have observed that the true return could be recovered (see Fig. 7) when using OPC and that the gradient estimates are less sensitive to model errors (see Fig.

Original: The off-policy case corresponds to the policy gains in Fig.
Revised: The off-policy case corresponds to the policy gains in Fig.

Original: 7 that are different from the reference policy π n indicated by the dashed line.
Revised: 7 that are different from the reference policy π n indicated by the dashed line.

Original: Fig.
Revised: Fig.

Original: 9 summarizes the results for the off-policy setting.
Revised: 9 summarizes the results for the off-policy setting.

Original: Here, we varied the policy error and the reference policy itself for varying model errors.
Revised: Here, we varied the policy error and the reference policy itself for varying model errors.

Original: Note that for the correct model, we always recover the true gradient.
Revised: Note that for the correct model, we always recover the true gradient.

Original: But also for inaccurate models, the gradient estimates retain a good quality in most cases, with the exception for some model/policy combinations that are close to unstable.
Revised: But also for inaccurate models, the gradient estimates retain a good quality in most cases, with the exception for some model/policy combinations that are close to unstable.

Original: [Figure]
Revised: [Figure]

Original: B. A DDITIONAL I NFORMATION
Revised: C. A DDITIONAL I NFORMATION

Original: Next, we provide some additional information about how we compute gradient distances, properties of linear systems, and exact numerical values used.
Revised: Next, we provide some additional information about how we compute gradient distances, properties of linear systems, and exact numerical values used.

Original: B.5. C OMPUTING THE S IGNED G RADIENT D ISTANCE
Revised: C.5. C OMPUTING THE S IGNED G RADIENT D ISTANCE

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: In order to compare two (1-dimensional) gradients in terms of sign and magnitude, we use the following formula
Revised: In order to compare two (1-dimensional) gradients in terms of sign and magnitude, we use the following formula

Original: [Equation]
Revised: [Equation]

Original: The magnitude of this quantity depends on the normalized difference between the tangent’s angles ∆ g and is positive for gradients with the same sign and vice versa it is negative for gradients with opposing signs.
Revised: The magnitude of this quantity depends on the normalized difference between the tangent’s angles ∆ g and is positive for gradients with the same sign and vice versa it is negative for gradients with opposing signs.

Original: See also Fig.
Revised: See also Fig.

Original: 10 for a sketch.
Revised: 10 for a sketch.

Original: B.5. D ETERMINING THE C LOSED - LOOP S TABILITY FOR L INEAR S YSTEMS
Revised: C.5. D ETERMINING THE C LOSED - LOOP S TABILITY FOR L INEAR S YSTEMS

Original: For linear and deterministic systems, we can easily check if the system is (asymptotically) stable for a particular linear policy using the following standard result from linear system theory:
Revised: For linear and deterministic systems, we can easily check if the system is (asymptotically) stable for a particular linear policy using the following standard result from linear system theory:

Original: Theorem 3 (Exponential stability for linear time-invariant systems (Callier & Desoer, 1991)) .
Revised: Theorem 3 (Exponential stability for linear time-invariant systems (Callier & Desoer, 1991)) .

Original: The solution of x t +1 =
Revised: The solution of x t +1 =

Original: Fx t is exponentially stable if and only if σ ( F ) ⊂ D (0 , 1) , i.e., every eigenvalue of F has magnitude strictly less than one.
Revised: Fx t is exponentially stable if and only if σ ( F ) ⊂ D (0 , 1) , i.e., every eigenvalue of F has magnitude strictly less than one.

Original: In our setting, this means that the closed-loop systems fulﬁlling the following are unstable ,
Revised: In our setting, this means that the closed-loop systems fulﬁlling the following are unstable ,

Original: [Equation]
Revised: [Equation]

Original: (92) i.e., the state and input grow exponentially.
Revised: (91) i.e., the state and input grow exponentially.

Original: We therefore refrain from including unstable system in the results to avoid numerical issues for the gradients’ computation.
Revised: We therefore refrain from including unstable system in the results to avoid numerical issues for the gradients’ computation.

Original: The respective areas in the plots are not colored, see e.g., bottom left corner in Fig.
Revised: The respective areas in the plots are not colored, see e.g., bottom left corner in Fig.

Original: 8b.
Revised: 8b.

Original: B.5. N UMERICAL V ALUES
Revised: C.5. N UMERICAL V ALUES

Original: The numerical values for all parameters used in the motivating example are given as follows:
Revised: The numerical values for all parameters used in the motivating example are given as follows:

Original: • True system dynamics:
Revised: • True system dynamics:

Original: A = 1 . 0 , B = 1 .•
Revised: A = 1 . 0 , B = 1 .•

Original: Initial condition: s 0 = 1 .•
Revised: Initial condition: s 0 = 1 .•

Original: Reward width parameter: σ r = 0 .•
Revised: Reward width parameter: σ r = 0 .•

Original: Optimal policy gain: θ ∗ = − 1 .•
Revised: Optimal policy gain: θ ∗ = − 1 .•

Original: Rollout horizon: T =
Revised: Rollout horizon: T =

Original: C A DDITIONAL E XPERIMENTAL R ESULTS
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we provide additional experimental results that did not ﬁt into the main body of the paper.
Revised: we provide additional experimental results that did not ﬁt into the main body of the paper.

Original: C. C OMPARISON WITH O THER B ASELINE A LGORITHMS
Revised: D. C OMPARISON WITH O THER B ASELINE A LGORITHMS

Original: Fig.
Revised: Fig.

Original: 11 shows our method compared to a range of baseline algorithms.
Revised: 11 shows our method compared to a range of baseline algorithms.

Original: The results for all baselines were obtained from Janner et al.
Revised: The results for all baselines were obtained from Janner et al.

Original: via personal communication.
Revised: via personal communication.

Original: Note that all results are presented in terms of mean and standard deviation.
Revised: Note that all results are presented in terms of mean and standard deviation.

Original: The comparison includes the following methods:
Revised: The comparison includes the following methods:

Original: • MBPO (Janner et al., 2019), • PETS (Chua et al., 2018), • SAC (Haarnoja et al., 2018), • PPO (Schulman et al., 2017), • STEVE (Buckman et al., 2018), • SLBO (Luo et al., 2018).
Revised: • MBPO (Janner et al., 2019), • PETS (Chua et al., 2018), • SAC (Haarnoja et al., 2018), • PPO (Schulman et al., 2017), • STEVE (Buckman et al., 2018), • SLBO (Luo et al., 2019).

Original: [Figure]
Revised: [Figure]

Original: We present the mean and standard deviation across 5 independent experiments (10 for OPC and
Revised: We present the mean and standard deviation across 5 independent experiments (10 for OPC and

Original: MBPO ( (cid:63) )).
Revised: MBPO ( (cid:63) )).

Original: The original data for MBPO and the other baselines
Revised: The original data for MBPO and the other baselines

Original: C. A BLATION - R ETAIN E POCHS
Revised: D. A BLATION - R ETAIN E POCHS

Original: One of the hyperparameters that we found is critical to both OPC and MBPO ( (cid:63) ), is retain epochs ,
Revised: One of the hyperparameters that we found is critical to both OPC and MBPO ( (cid:63) ), is retain epochs ,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e., the number of epochs that are kept in the data buffer for the simulated data generated with the ˜ p x with x = { OPC , model } .
Revised: e., the number of epochs that are kept in the data buffer for the simulated data generated with the ˜ p x with x = { OPC , model } .

Original: The results for a comparison are shown in Fig.
Revised: The results for a comparison are shown in Fig.

Original: For MBPO ( (cid:63) ), we found that for some environments (HalfCheetah, AntTruncatedObs) smaller values for retain epochs are helpful,
Revised: For MBPO ( (cid:63) ), we found that for some environments (HalfCheetah, AntTruncatedObs) smaller values for retain epochs are helpful,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e., simulated data is almost only on-policy
Revised: e., simulated data is almost only on-policy

Original: [Figure]
Revised: [Figure]

Original: and for other environments larger values are beneﬁcial (Hopper, Walker2d).
Revised: and for other environments larger values are beneﬁcial (Hopper, Walker2d).

Original: For OPC on the other hand, we found that retain epochs =almost always leads to better results. 
Revised: For OPC on the other hand, we found that retain epochs =almost always leads to better results. [Figure] [Figure]

Revised: Walker2d

Revised: AntTruncatedObs

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e., the number of epochs that are kept in the data buffer for the simulated data.
Revised: ., the number of epochs that are kept in the data buffer for the simulated data.

Original: C. I NFLUENCE OF S TATE R EPRESENTATION : I N -D EPTH A NALYSIS
Revised: D. I NFLUENCE OF S TATE R EPRESENTATION : I N -D EPTH A NALYSIS

Original: In the following, we will have a closer look at the surprising result from Fig.
Revised: In the following, we will have a closer look at the surprising result from Fig.

Original: 4 (left).
Revised: 4 (left).

Original: In there, the results indicate that MBPO ( (cid:63) ) is not able to learn a stabilizing policy within the ﬁrst 7’500 steps on the RoboSchool variant of the CartPole environment.
Revised: In there, the results indicate that MBPO ( (cid:63) ) is not able to learn a stabilizing policy within the ﬁrst 7’500 steps on the RoboSchool variant of the CartPole environment.

Original: We hypothesize that the failure of MBPO ( (cid:63) ) is due to a mismatch of the simulated data and the true state distribution.
Revised: We hypothesize that the failure of MBPO ( (cid:63) ) is due to a mismatch of the simulated data and the true state distribution.

Original: As a result, the policy that is optimized with the simulated data cannot stabilize the pole in the true environment.
Revised: As a result, the policy that is optimized with the simulated data cannot stabilize the pole in the true environment.

Original: To validate our hypothesis, we perform the following experiment: First, we train a policy π ∗ that we know performs well on the true environment, leading to the maximum evaluation return of 1000.
Revised: To validate our hypothesis, we perform the following experiment: First, we train a policy π ∗ that we know performs well on the true environment, leading to the maximum evaluation return of 1000.

Original: With this policy, we rollout a reference trajectory on the true environment τ ref = { (ˆ s t , ˆ a t ) } Tt =0 .
Revised: With this policy, we roll out a reference trajectory on the true environment τ ref = { (ˆ s t , ˆ a t ) } Tt =0 .

Original: To perform branched rollouts with the respective methods, OPC and MBPO ( (cid:63) ), we use the learned transition models after 20 epochs (5’000 time steps) that were logged during a full learning process for each method.
Revised: To perform branched rollouts with the respective methods, OPC and MBPO ( (cid:63) ), we use the learned transition models after 20 epochs (5’000 time steps) that were logged during a full learning process for each method.

Original: We then perform 100 branched rollouts of length H = 20 starting from randomly sampled initial states of the reference trajectories.
Revised: We then perform 100 branched rollouts of length H = 20 starting from randomly sampled initial states of the reference trajectories.

Original: Fig.
Revised: Fig.

Original: 13 shows the difference between the true and predicted state trajectories (median and 95 th percentiles) for each state in [ x, cos( ϑ ) , sin( ϑ ) , ˙ x, ˙ ϑ ] .
Revised: 13 shows the difference between the true and predicted state trajectories (median and 95 th percentiles) for each state in [ x, cos( ϑ ) , sin( ϑ ) , ˙ x, ˙ ϑ ] .

Original: Since we start each branched rollout from a state on the real environment, the initial error is always zero.
Revised: Since we start each branched rollout from a state on the real environment, the initial error is always zero.

Original: Across all states, the errors are drastically reduced when using OPC .
Revised: Across all states, the errors are drastically reduced when using OPC .

Original: Fig.
Revised: Fig.

Original: 14 shows the predicted trajectories for the cosine of the pole’s angle, cos( ϑ ) .
Revised: 14 shows the predicted trajectories for the cosine of the pole’s angle, cos( ϑ ) .

Original: For MBPO ( (cid:63) ), we observe that the trajectories often diverge and attain values that are clearly out of distribution, i.e., cos( ϑ ) > 1 . 
Revised: For MBPO ( (cid:63) ), we observe that the trajectories often diverge and attain values that are clearly out of distribution, i.e., cos( ϑ ) > 1 . time steps

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: S t a t e d i ff e r e n ce : s t r u e t − s p r e d t
Revised: S t a t e d i ff e r e n ce : s t r u e t − s p r e d t

Original: Lateral Position    time steps cos(Angle)    time steps sin(Angle)    
Revised: Lateral Position    time steps cos(Angle)    time steps sin(Angle)    time steps − 0 .

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: S t a t e d i ff e r e n ce 
Revised: S t a t e d i ff e r e n ce : s t r u e t − s p r e d t

Original: [Figure]
Revised: [Figure]

Original: s t r u e t
Revised: Velocity

Original: [Figure]
Revised: [Figure]

Original: r

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: When using the standard predictive model from MBPO ( (cid:63) ) ( ), the state trajectories often diverge and the rollouts are terminated prematurely.
Revised: using the standard predictive model from MBPO ( (cid:63) ) ( ), the state trajectories often diverge and the rollouts are terminated prematurely.

Original: [Figure]
Revised: [Figure]

Revised: O n - P o li c y

Original: [Figure]
Revised: M od e l

Original:  |
Revised: E rr o r | η n − ˜ η n |

Original: Iteration n of Policy Optimization
Revised: Iteration n of Policy Optimization

Original: [Figure]
Revised: O ff-

Original:  o li c y
Revised: P o li c y

Original: M od [Figure] lE
Revised: M od e l

Original: [Figure] | [Figure] |
Revised: E rr o r | η n +− ˜ η n + 1 |

Original: C. I MPROVEMENT B OUND : E MPIRICAL I NVESTIGATION
Revised: D. I MPROVEMENT B OUND : E MPIRICAL I NVESTIGATION

Original: In this section we empirically investigate to what extent OPC is able to tighten the policy improvement bound Eq.
Revised: In this section we empirically investigate to what extent OPC is able to tighten the policy improvement bound Eq.

Original: (5) compared to pure model-based approaches.
Revised: (4) compared to pure model-based approaches.

Original: As mentioned in the main paper, the motivation behind OPC is to reduce the on-policy error and we assume that the off-policy error is not affected too badly by the corrected transitions.
Revised: As mentioned in the main paper, the motivation behind OPC is to reduce the on-policy error and we assume that the off-policy error is not affected too badly by the corrected transitions.

Original: Generally speaking, it is difﬁcult to quantify a-priori how OPC compares to a purely model-based approach as this depends on the generalization capabilities of the learned dynamics model, the environment itself and the reward signal.
Revised: Generally speaking, it is difﬁcult to quantify a-priori how OPC compares to a purely model-based approach as this depends on the generalization capabilities of the learned dynamics model, the environment itself and the reward signal.

Original: Here, we analyze the CartPole environment (PyBullet implementation) and estimate the respective error terms that appear in the policy improvement bound Eq.
Revised: Here, we analyze the CartPole environment (PyBullet implementation) and estimate the respective error terms that appear in the policy improvement bound Eq.

Original: To this end, we roll out a sequence of policies π n on the true environment (to estimate η n and η n +1 ) and on the learned model with and without OPC (to estimate ˜ η n and ˜ η n +1 ).
Revised: To this end, we roll out a sequence of policies π n on the true environment (to estimate η n and η n +1 ) and on the learned model with and without OPC (to estimate ˜ η n and ˜ η n +1 ).

Original: The sequence of policies was obtained during a full policy optimization run (the policy was logged after each update) and we roll out each policy 100 times on the respective environment/models.
Revised: The sequence of policies was obtained during a full policy optimization run (the policy was logged after each update) and we roll out each policy 100 times on the respective environment/models.

Original: The corresponding learned transition models were similarly logged during a full policy optimization.
Revised: The corresponding learned transition models were similarly logged during a full policy optimization.

Original: For OPC , we estimate the off-policy return ˜ η n +1 using the learned model from iteration n and reference trajectories from the true environment that were collected under π n , but then rollout the model with π n +1 .
Revised: For OPC , we estimate the off-policy return ˜ η n +1 using the learned model from iteration n and reference trajectories from the true environment that were collected under π n , but then roll out the model with π n +1 .

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: [Figure]
Revised: [Figure]

Revised: OPC 1 .1 .1 .1 .1 .2 .2 .2 .2 .2 .3 .

Original: Policy Stochasticity Multiplier β
Revised: Policy Stochasticity Multiplier β

Original: [Figure]
Revised: [Figure]

Revised: c y

Revised: [Figure]

Original: C. O FF -P OLICY A NALYSIS
Revised: D. O FF -P OLICY A NALYSIS

Original: In this section, we investigate the robustness of OPC towards ‘off-policy-ness’ of the reference trajectories that are simulated under the data-generating policy π n .
Revised: In this section, we investigate the robustness of OPC towards ‘off-policy-ness’ of the reference trajectories that are simulated under the data-generating policy π n .

Original: To this end, we manually increase the stochasticity of the behaviour policy π rollout by a factor β such that
Revised: To this end, we manually increase the stochasticity of the behavior policy π rollout by a factor β such that

Original: [Equation]
Revised: [Equation]

Original: (93) and we keep the mean the same for both policies.
Revised: (92) and we keep the mean the same for both policies.

Original: Fig.
Revised: Fig.

Original: 16 shows the distributions of the returns from 100 rollouts with varying degree of ‘off-policy-ness’ on the CartPole environment (PyBullet).
Revised: 16 shows the distributions of the returns from 100 rollouts with varying degree of ‘off-policy-ness’ on the CartPole environment (PyBullet).

Original: Note that the data-generating policy consistently leads to the maximum return of 1000.
Revised: Note that the data-generating policy consistently leads to the maximum return of 1000.

Original: For β close to one, both OPC and MBPO ( (cid:63) ) lead to almost ideal behaviour and correctly predict the return in more than 95% of the cases.
Revised: For β close to one, both OPC and MBPO ( (cid:63) ) lead to almost ideal behavior and correctly predict the return in more than 95% of the cases.

Original: As we increase the policy’s stochasticity (from left to right), the respective performance of the policy decreases until all rollouts terminate prematurely ( β ≥ 2 . 3 ).
Revised: As we increase the policy’s stochasticity (from left to right), the respective performance of the policy decreases until all rollouts terminate prematurely ( β ≥ 2 . 3 ).

Original: Notably, the extent of this effect is almost identical between OPC and MBPO ( (cid:63) ).
Revised: Notably, the extent of this effect is almost identical between OPC and MBPO ( (cid:63) ).

Original: We conclude that OPC is robust towards ‘off-policy-ness’ of the reference trajectories.
Revised: We conclude that OPC is, at least empirically, robust towards ‘off-policy-ness’ of the reference trajectories.

Original: Otherwise, we should observe a more pronounced degradation of the policy’s performance with increased stochasticity of the behaviour policy since this leads to observing more off-policy state/actions.
Revised: Otherwise, we should observe a more pronounced degradation of the policy’s performance with increased stochasticity of the behavior policy, since this leads to observing more off-policy state/actions.

Original: 1: Initialize policy π φ , predictive model p θ , environment dataset D env , model dataset D model 2: for N epochs do 3:
Revised: 1: Initialize policy π φ , predictive model p θ , environment dataset D env , model dataset D model 2: for N epochs do 3:

Original: Train model p θ on D env via maximum likelihood 4: for E steps do 5:
Revised: Train model p θ on D env via maximum likelihood 4: for E steps do 5:

Original: Take action in environment according to π φ ; add to D env 6: for M model rollouts do 7:
Revised: Take action in environment according to π φ ; add to D env 6: for M model rollouts do 7:

Original: Sample s t uniformly from D env 8:
Revised: Sample s t uniformly from D env 8:

Original: Perform k -step model rollout starting from s t using policy π φ ; add to D model 9: for G gradient updates do 10:
Revised: Perform k -step model rollout starting from s t using policy π φ ; add to D model 9: for G gradient updates do 10:

Original: Update policy parameters on model data: φ ← φ − λ π ˆ ∇ φ J π ( φ, D model )
Revised: Update policy parameters on model data: φ ← φ − λ π ˆ ∇ φ J π ( φ, D model )

Original: 1: Initialize policy π φ ,predictive model p θ , environment dataset D env , model dataset D model 2: for N epochs do 3:
Revised: 1: Initialize policy π φ , model p θ , environment dataset D env , model dataset D model 2: for N epochs do 3:

Original: Train model p θ on D env via maximum likelihood 4: for E steps do 5:
Revised: Train model p θ on D env via maximum likelihood 4: for E steps do 5:

Original: Take action in environment according to π φ ; add to D env 6: for M model rollouts do 7:
Revised: Take action in environment according to π φ ; add to D env 6: for M model rollouts do 7:

Original: Sample s t uniformly from D env 8:
Revised: Sample s t uniformly from D env 8:

Original: Perform k -step model rollout starting from s t using policy π φ ; add to D model 9: for G gradient updates do 10:
Revised: Perform k -step model rollout starting from s t using policy π φ ; add to D model 9: for G gradient updates do 10:

Original: Update policy parameters on model data: φ ← φ − λ π ˆ ∇ φ J π ( φ, D model )
Revised: Update policy parameters on model data: φ ← φ − λ π ˆ ∇ φ J π ( φ, D model )

Original: C. I MPLEMENTATION C HANGES TO O RIGINAL MBPO
Revised: D. I MPLEMENTATION C HANGES TO O RIGINAL MBPO

Original: Here, we provide details to the changes we made to the original implementation of MBPO .
Revised: Here, we provide details to the changes we made to the original implementation of MBPO .

Original: We denote our variant as MBPO ( (cid:63) ).
Revised: We denote our variant as MBPO ( (cid:63) ).

Original: Episodic Setting
Revised: Episodic Setting

Original: MBPO updates the policy during rollouts on the real environment.
Revised: MBPO updates the policy during rollouts on the real environment.

Original: Algorithmsand 3 show the original and our version of MBPO , respectively.
Revised: Algorithmsand 4 show the original and our version of MBPO , respectively.

Original: While the original version might be more sample efﬁcient because it allows for more policy gradient updates based on more recent data, it is not realistic to update the policy during a rollout on the real environment.
Revised: While the original version might be more sample efﬁcient because it allows for more policy gradient updates based on more recent data, it is not realistic to update the policy during a rollout on the real environment.

Original: Mix-in of Real Data
Revised: Mix-in of Real Data

Original: As mentioned in the main paper, one of the key problems with model-based RL (MBRL) is that the generated data might exhibit so-called model-bias.
Revised: As mentioned in the main paper, one of the key problems with MBRL is that the generated data might exhibit so-called model-bias.

Original: Biased data can be problematic for policy optimization as the transition tuples possibly do not come from the true state distribution of the real environment, thus misleading the RL algorithm.
Revised: Biased data can be problematic for policy optimization as the transition tuples possibly do not come from the true state distribution of the real environment, thus misleading the RL algorithm.

Original: In the original implementation of MBPO , the authors use a mix of simulated data from the model as well as observed data from the true environment for policy optimization ( https://github.com/JannerM/mbpo/blob/ master/mbpo/algorithms/mbpo.py#L430 ) to alleviate the issue of model bias.
Revised: In the original implementation of MBPO , the authors use a mix of simulated data from the model as well as observed data from the true environment for policy optimization ( https://github.com/jannerm/mbpo/blob/ 22cab517c1be7412ec33fbe5c510e018d5813ebf/mbpo/algorithms/mbpo.py# L430 ) to alleviate the issue of model bias.

Original: While this design choice might help in practice, it adds an additional hyperparameter and the exact inﬂuence of this parameter is difﬁcult to interpret.
Revised: While this design choice might help in practice, it adds another hyperparameter and the exact inﬂuence of this parameter is difﬁcult to interpret.

Original: We therefore refrain from mixing in data from the true environment but instead only use simulated transition tuples – staying true to the spirit of MBRL.
Revised: We therefore refrain from mixing in data from the true environment, but instead only use simulated transition tuples – staying true to the spirit of MBRL.

Original: Fix Replay Buffer
Revised: Fix Replay Buffer

Original: The replay buffer that stores the simulated data D model for policy optimization is allocated to a ﬁxed size of rollout length × rollout batch size × retain epochs , meaning that per epoch rollout length × rollout batch size transition tuples are simulated and only the data from the last retain epochs are kept in the buffer ( https://github. com/JannerM/mbpo/blob/master/mbpo/algorithms/mbpo.py#L351 ).
Revised: The replay buffer that stores the simulated data D model for policy optimization is allocated to a ﬁxed size of rollout length × rollout batch size × retain epochs , meaning that per epoch rollout length × rollout batch size transition tuples are simulated and only the data from the last retain epochs are kept in the buffer ( https://github. com/jannerm/mbpo/blob/22cab517c1be7412ec33fbe5c510e018d5813ebf/ mbpo/algorithms/mbpo.py#L351 ).

Original: As the buffer is implemented as a FIFO queue and rollouts might terminate early such that the actual number of simulated transitions is less than rollout length , data from older episodes as speciﬁed by retain epochs are retained in the buffer.
Revised: As the buffer is implemented as a FIFO queue and rollouts might terminate early such that the actual number of simulated transitions is less than rollout length , data from older episodes as speciﬁed by retain epochs are retained in the buffer.

Original: We assume that this is not the intended behaviour and correct for that in our implementation.[Figure] v a l u [Figure]
Revised: We assume that this is not the intended behavior and correct for that in our implementation.

Original: ti

Original: [Figure]

Original: r e t u r n

Original: [Figure]

Original: l u

Original: [Figure]

Original: ti

Original: [Figure]
Revised: [Figure]

Original: r e t u r n

Original: [Figure] HalfCheetah [Figure] Moreover, on the highly stochastic Hopper environment, OPC outperforms both MBPO variants and does not require additional real off-policy data.
Revised: Moreover, on the highly stochastic Hopper environment, OPC outperforms both MBPO variants and does not require additional real off-policy data.

Original: C. R ESULTS FOR O RIGINAL MBPO I MPLEMENTATION
Revised: D. R ESULTS FOR O RIGINAL MBPO I MPLEMENTATION

Original: The following results were obtained with the original MBPO implementation without the changes described in Appendix C.6.
Revised: The following results were obtained with the original MBPO implementation without the changes described in Appendix D.6.

Original: Consequently, the results for OPC in this section also do not include these changes.
Revised: Consequently, the results for OPC in this section also do not include these changes

Original:  Comparative Evaluation
Revised: . Comparative Evaluation

Original: We evaluate the original implementation on three continuous control benchmark tasks from the
Revised: We evaluate the original implementation on three continuous control benchmark tasks from the

Original: MuJoCo control suite (Todorov et al., 2012).
Revised: MuJoCo control suite (Todorov et al., 2012).

Original: The results for OPC , two variants of MBPO and SAC are presented in Fig.
Revised: The results for OPC , two variants of MBPO and SAC are presented in Fig.

Original: Both OPC and MBPO use a rollout horizon of H = 10 to generate the training data.
Revised: Both OPC and MBPO use a rollout horizon of H = 10 to generate the training data.

Original: The difference between the two MBPO variants lies in the mix-in ratio β of real off-policy transitions to the simulated training data.
Revised: The difference between the two MBPO variants lies in the mix-in ratio β of real off-policy transitions to the simulated training data.

Original: Especially for highly stochastic environments such as the Hopper, this mix-in ratio is a critical hyperparameter that requires careful tuning (see also Fig. 18).
Revised: Especially for highly stochastic environments such as the Hopper, this mix-in ratio is a critical hyperparameter that requires careful tuning (see also Fig. 18).

Original: Fig.indicates that OPC is on par with MBPO on both the InvertedPendulum and HalfCheetah environments that exhibit little stochasticity.
Revised: Fig.indicates that OPC is on par with MBPO on both the InvertedPendulum and HalfCheetah environments that exhibit little stochasticity.

Original: On the Hopper environment, OPC outperforms both MBPO variants.
Revised: On the Hopper environment, OPC outperforms both MBPO variants.

Original: Note that the mix-in ratio for MBPO is critical for successful learning (the original implementation uses β = 5% ).
Revised: Note that the mix-in ratio for MBPO is critical for successful learning (the original implementation uses β = 5% ).

Original: OPC on the other hand, does not require any mixed-in real data.
Revised: OPC on the other hand, does not require any mixed-in real data.

Original: SAC learns slower on the more complex Hopper and HalfCheetah environments, re-iterating that model-based approaches are signiﬁcantly more data-efﬁcient than model-free methods.
Revised: SAC learns slower on the more complex Hopper and HalfCheetah environments, re-iterating that model-based approaches are signiﬁcantly more data-efﬁcient than model-free methods.

Original: Large Ablation Study – Hopper
Revised: Large Ablation Study – Hopper

Original: The full study investigates the inﬂuence of the following hyperparameters and design choices:
Revised: The full study investigates the inﬂuence of the following hyperparameters and design choices:

Original: • Rollout length H and total number of simulated transitions N .
Revised: • Rollout length H and total number of simulated transitions N .

Original: • Mix-in ratio of real transitions into the training data β .
Revised: • Mix-in ratio of real transitions into the training data β .

Original: • Deterministic or stochastic rollouts (for MBPO ):
Revised: • Deterministic or stochastic rollouts (for MBPO ):

Original: Current state-of-the-art methods in MBRL rely on probabilistic dynamic models to capture both aleatoric and epistemic uncertainty.
Revised: Current state-of-the-art methods in MBRL rely on probabilistic dynamic models to capture both aleatoric and epistemic uncertainty.

Original: Accordingly, when rolling out the model, these two sources of uncertainty are accounted for.
Revised: Accordingly, when rolling out the model, these two sources of uncertainty are accounted for.

Original: However, we show that in terms of evaluation return, the stochastic rollouts do not always lead to the best outcome.
Revised: However, we show that in terms of evaluation return, the stochastic rollouts do not always lead to the best outcome.

Original: • Re-setting the buffer of simulated data after a policy optimization step: We found that re-setting the replay buffer for simulated data after each iteration of Algorithm 2 can have a large inﬂuence.
Revised: • Re-setting the buffer of simulated data after a policy optimization step: We found that re-setting the replay buffer for simulated data after each iteration of Algorithm 3 can have a large inﬂuence.

Original: In particular, the replay buffer is implemented as a FIFO queue with a ﬁxed size.
Revised: In particular, the replay buffer is implemented as a FIFO queue with a ﬁxed size.

Original: Hence, if the buffer is not emptied after each iteration, it still contains (simulated) off-policy transitions.
Revised: Hence, if the buffer is not emptied after each iteration, it still contains (simulated) off-policy transitions.

Original: Fig.
Revised: Fig.

Original: 18 presents the results of the ablation study.
Revised: 18 presents the results of the ablation study.

Original: We want to highlight a few core insights:
Revised: We want to highlight a few core insights:

Original: When choosing the best setting for each method, OPC improves MBPO by a large margin (bottom row right).
Revised: When choosing the best setting for each method, OPC improves MBPO by a large margin (bottom row, right).

Original: Generally, for long rollouts H = 20 (bottom row), OPC improves MBPO .
Revised: Generally, for long rollouts H = 20 (bottom row), OPC improves MBPO .

Original: Across all settings, OPC performs well and is more robust with respect to the choices of hyperparameters (e.g., bottom row left and center, third row left).
Revised: Across all settings, OPC performs well and is more robust with respect to the choices of hyperparameters (e.g., bottom row left and center, third row left).

Original: Only for few exceptions, re-setting the buffer can have detrimental effects (e.g., top right, third row right).
Revised: Only for few exceptions, re-setting the buffer can have detrimental effects (e.g., top right, third row right).

Original: Mixing in real transition data can be highly beneﬁcial for MBPO (second row left and center) but it can also have the opposite effect (bottom row).
Revised: Mixing in real transition data can be highly beneﬁcial for MBPO (second row left and center) but it can also have the opposite effect (bottom row).

Original: Using deterministic rollouts can be beneﬁcial (third row right and left), detrimental (third row center) or have no inﬂuence (bottom row left) for MBPO .
Revised: Using deterministic rollouts can be beneﬁcial (third row right and left), detrimental (third row center) or have no inﬂuence (bottom row left) for MBPO .

Original: It is not clear, if re-setting the buffer after each iteration should overall be recommended or not.
Revised: It is not clear, if re-setting the buffer after each iteration should overall be recommended or not.

Original: This remains an open question left for future research.
Revised: This remains an open question left for future research.

Original: [Figure]
Revised: [Figure]

Original: v a
Revised: u a

Original: [Figure]
Revised: [Figure]

Revised: v a l u a ti

Revised: [Figure]

Revised: t u r n

Revised: [Figure]

Original:  ua
Revised: l u

Original: [Figure]
Revised: [Figure]

Original: a
Revised: n

Original: [Figure]
Revised: [Figure]

Original:  u a ti 
Revised: v a l u a ti on r e t u r n

Original: [Figure]
Revised: [Figure]

Original: D C ONNECTION BETWEEN MBRL AND ILC
Revised: E C ONNECTION BETWEEN MBRL AND ILC

Original: In this section we compare optimization-based or so-called norm-optimal ILC (NO-ILC) with MBRL.
Revised: In this section we compare optimization-based or so-called norm-optimal ILC (NO-ILC) with MBRL.

Original: In particular, we show that under certain assumptions we can reduce the MBRL setting to NO-ILC.
Revised: In particular, we show that under certain assumptions we can reduce the MBRL setting to NO-ILC.

Original: This comparison is structured as follows: First, we review the basic assumptions and notations for
Revised: This comparison is structured as follows: First, we review the basic assumptions and notations for

Original: NO-ILC.
Revised: NO-ILC.

Original: While there are many variations on NO-ILC, we will only consider the very basic setting, i.e., linear dynamics, fully observed state and deterministic state evolution.
Revised: While there are many variations on NO-ILC, we will only consider the very basic setting, i.e., linear dynamics, fully observed state and deterministic state evolution.

Original: Then, based on the lifted state representation of the problem, we derive the solution to the optimization problem that leads to the input sequence of the next iteration / rollout.
Revised: Then, based on the lifted state representation of the problem, we derive the solution to the optimization problem that leads to the input sequence of the next iteration / rollout.

Original: Next, we will state the general MBRL problem and pose the simpliﬁcations that we need to make in order to be equivalent to the NO-ILC problem.
Revised: Next, we will state the general MBRL problem and pose the simpliﬁcations that we need to make in order to be equivalent to the NO-ILC problem.

Original: Last, we show that the solution to the reduced MBRL problem is equivalent to the one of NO-ILC.
Revised: Last, we show that the solution to the reduced MBRL problem is equivalent to the one of NO-ILC.

Original: D. N ORM - OPTIMAL ILC
Revised: E. N ORM - OPTIMAL ILC

Original: The goal of NO-ILC is to ﬁnd a sequence of inputs a =
Revised: The goal of NO-ILC is to ﬁnd a sequence of inputs a =

Original: [ a (cid:62) 0 , . . .
Revised: [ a (cid:62) 0 , . . .

Original: , a (cid:62) T − 1 ] (cid:62) with length T + 1 such that the outputs y =
Revised: , a (cid:62) T − 1 ] (cid:62) with length T + 1 such that the outputs y =

Original: [ y (cid:62) 0 , . . .
Revised: [ y (cid:62) 0 , . . .

Original: , y (cid:62) T ] (cid:62) follow a desired output trajectory ˆ y =
Revised: , y (cid:62) T ] (cid:62) follow a desired output trajectory ˆ y =

Original: [ˆ y (cid:62) 0 , . . .
Revised: [ˆ y (cid:62) 0 , . . .

Original: , ˆ y (cid:62) T ] (cid:62) .
Revised: , ˆ y (cid:62) T ] (cid:62) .

Original: In the simplest setting we assume that the system evolves according to the following linear dynamics
Revised: In the simplest setting we assume that the system evolves according to the following linear dynamics

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (95) with state s t ∈ R d S , action a t ∈ R d A , output y t ∈ R p and disturbance d t ∈ R d S .
Revised: (94) with state s t ∈ R d S , action a t ∈ R d A , output y t ∈ R p and disturbance d t ∈ R d S .

Original: One of the major assumption in ILC is that the disturbances d t are repetitive , meaning that the sequence d = [ d (cid:62) 0 , . . .
Revised: One of the major assumption in ILC is that the disturbances d t are repetitive , meaning that the sequence d = [ d (cid:62) 0 , . . .

Original: , d (cid:62) T ] (cid:62) does not change (or only varies slightly) across multiple rollouts.
Revised: , d (cid:62) T ] (cid:62) does not change (or only varies slightly) across multiple rollouts.

Original: While these disturbances can be considered to come from some exogenous error source, one can also interpret these as unmodeled effects of the dynamics, e.g., nonlinearities stemming from friction, aerodynamic effects, etc.
Revised: While these disturbances can be considered to come from some exogenous error source, one can also interpret these as unmodeled effects of the dynamics, e.g., nonlinearities stemming from friction, aerodynamic effects, etc.

Original: For the derivation, let’s assume C
Revised: For the derivation, let’s assume C

Original: = I so that we’re operating on an MDP.
Revised: = I so that we’re operating on an MDP.

Original: Consequently, the goal of ILC is to track a sequence of desired states ˆ s instead of outputs ˆ y .
Revised: Consequently, the goal of ILC is to track a sequence of desired states ˆ s instead of outputs ˆ y .

Original: In order to ﬁnd an optimal input sequence, we minimize the squared 2-norm of a state’s deviation from the reference for each time-step t of the sequence, i.e., e t
Revised: In order to ﬁnd an optimal input sequence, we minimize the squared 2-norm of a state’s deviation from the reference for each time-step t of the sequence, i.e., e t

Original: = ˆ s t − s t .
Revised: = ˆ s t − s t .

Original: As is common in the ILC literature, we make use of the so-called lifted system formulation such that we can conveniently write the state evolution for one rollout using a single matrix/vector multiplication.
Revised: As is common in the ILC literature, we make use of the so-called lifted system formulation such that we can conveniently write the state evolution for one rollout using a single matrix/vector multiplication.

Original: Assuming zero initial state (which can always be done in the linear setting by just shifting the state by a constant offset), we obtain the following formulation
Revised: Assuming zero initial state (which can always be done in the linear setting by just shifting the state by a constant offset), we obtain the following formulation

Original: [Equation]
Revised: [Equation]

Original: Thus, we can write the state-error at the j -th iteration of ILC in the lifted representation as (recall that the disturbance d does not change across iterations)
Revised: Thus, we can write the state-error at the j -th iteration of ILC in the lifted representation as (recall that the disturbance d does not change across iterations)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, the resulting optimization problem then becomes
Revised: Now, the resulting optimization problem then becomes

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Original:  (cid:16) a ( 
Revised: J (cid:16) a ( i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In order to be less sensitive to noise and the inherent stochasticity of real-world problems, one typically also adds a regularizing term that penalizes the changes in the input sequence.
Revised: In order to be less sensitive to noise and the inherent stochasticity of real-world problems, one typically also adds a regularizing term that penalizes the changes in the input sequence.

Original: While this additional penalization term can slow down the learning process it makes it more robust by avoiding overcompensation to the disturbances.
Revised: While this additional penalization term can slow down the learning process it makes it more robust by avoiding overcompensation to the disturbances.

Original: The full objective then becomes
Revised: The full objective then becomes

Original: [Equation]
Revised: [Equation]

Original: where we additionally added positive semi-deﬁnite cost matrices M , W for the respective norms in order to facilitate tuning of the corresponding terms in the objective.
Revised: where we additionally added positive semi-deﬁnite cost matrices M , W for the respective norms in order to facilitate tuning of the corresponding terms in the objective.

Original: Given the regularized cost function we obtain the optimal sequence of inputs at the next iteration as
Revised: Given the regularized cost function we obtain the optimal sequence of inputs at the next iteration as

Original: [Equation]
Revised: [Equation]

Original: D. M ODEL - BASED RL TO N ORM - OPTIMAL ILC Assumptions
Revised: E. M ODEL - BASED RL TO N ORM - OPTIMAL ILC Assumptions

Original: In order to show the equivalence of MBRL and NO-ILC we need to make some assumptions to the above stated optimization problem.
Revised: In order to show the equivalence of MBRL and NO-ILC we need to make some assumptions to the above stated optimization problem.

Original: • No aleatoric uncertainty in the model,
Revised: • No aleatoric uncertainty in the model,

Original: i
Revised: i

Original: e., no transition noise ω t = 0 ∀ t .
Revised: e., no transition noise ω t = 0 ∀ t .

Original: • Typically in RL we assume the policy to be stationary, however, in this setting we will allow for non-stationary policies that are indexed by time t (the result will not necessarily depend on the state such that we essentially just obtain a feedforward control sequence.
Revised: • Typically in RL we assume the policy to be stationary, however, in this setting we will allow for non-stationary policies that are indexed by time t (the result will not necessarily depend on the state such that we essentially just obtain a feedforward control sequence.

Original: To include feedback one can always just combine the feedforward signal with a local controller that tracks the desired state/action trajectory).
Revised: To include feedback one can always just combine the feedforward signal with a local controller that tracks the desired state/action trajectory).

Original: • The reward is given as the negative quadratic error of the state w.r.t.
Revised: • The reward is given as the negative quadratic error of the state w.r.t.

Original: a desired state trajectory,r
Revised: a desired state trajectory,

Original: [Equation]
Revised: [Equation]

Original: t [Equation] • Typically in RL we have constraints on the policy such that it does not change too much after every iteration, see e.g. TRPO, REPS, etc.
Revised: • Typically in RL we have constraints on the policy such that it does not change too much after every iteration, see e.g. TRPO, REPS, etc.

Original: While in the mentioned approaches, the policy are often constrained in terms of their parameterization vectors, we constrain it as(cid:107)
Revised: While in the mentioned approaches, the policy are often constrained in terms of their parameterization vectors, we constrain it as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: • Assume that the model is given by ˜ f t ( s , a ) =
Revised: • Assume that the model is given by ˜ f t ( s , a ) =

Original: As + Ba + d t , where A and B are ﬁxed system matrices and d t is a time-dependent offset that we learn.
Revised: As + Ba + d t , where A and B are ﬁxed system matrices and d t is a time-dependent offset that we learn.

Original: The learned dynamicsmodel
Revised: The learned dynamics

Revised: model

Original: Given state/input pairs of the i -th trajectory τ ( i ) = 
Revised: Given state/input pairs of the i -th trajectory τ ( i ) = { ( s

Original: [Equation]
Revised: [Equation]

Original: from the true system, we can now improve our dynamics model.
Revised: from the true system, we can now improve our dynamics model.

Original: In particular, if we minimize the prediction error over τ ( i ) , we obtain
Revised: In particular, if we minimize the prediction error over τ ( i ) , we obtain

Original: [Equation]
Revised: [Equation]

Original: The resulting dynamics for the optimal control problem in Eq.
Revised: The resulting dynamics for the optimal control problem in Eq.

Original: (1) become
Revised: (1) become

Original: [Equation]
Revised: [Equation]

Original: which are the error dynamics around the trajectory.
Revised: which are the error dynamics around the trajectory.

Original: Now in the noisy case, taking the last trajectory is not necessarily the best thing one can do.
Revised: Now in the noisy case, taking the last trajectory is not necessarily the best thing one can do.

Original: E.g., (Schöllig & D’Andrea, 2009) instead integrate the information of all past trajectories via Kalman-ﬁltering.
Revised: E.g., (Schöllig & D’Andrea, 2009) instead integrate the information of all past trajectories via Kalman-ﬁltering.

Original: In the fully observed case, one way to think of this is as low-pass ﬁltering d t in order to account for the transition noise ω .
Revised: In the fully observed case, one way to think of this is as low-pass ﬁltering d t in order to account for the transition noise ω .

Original: The resulting MBRL problem
Revised: The resulting MBRL problem

Original: Now, let’s plug in all assumptions into the MBRL problem and have a look at how to solve it.
Revised: Now, let’s plug in all assumptions into the MBRL problem and have a look at how to solve it.

Original: [Equation]
Revised: [Equation]

Original: where we have ﬂipped the reward’s sign to transform it into a minimization problem.
Revised: where we have ﬂipped the reward’s sign to transform it into a minimization problem.

Original: In the presented form, this optimization problem has a well-deﬁned unique solution, however, it is a-priori not clear if the trust-region constraint is active for some time-steps.
Revised: In the presented form, this optimization problem has a well-deﬁned unique solution, however, it is a-priori not clear if the trust-region constraint is active for some time-steps.

Original: To facilitate an analytical solution to equation 103, we incorporate the constrain on the policy’s stepsize by a soft-constraint such that
Revised: To facilitate an analytical solution to equation 102, we incorporate the constrain on the policy’s stepsize by a soft-constraint such that

Original: [Equation]
Revised: [Equation]

Original: with C πt being constants that weight the respective trust-region terms.
Revised: with C πt being constants that weight the respective trust-region terms.

Original: Generally, the MBRL problemEq. (2) cannot be solved analytically due to the (possibly highly nonlinear, non-differentiable, etc.)
Revised: Generally, the MBRL problem cannot be solved analytically due to the (possibly highly non-linear, non-differentiable, etc.)

Original: reward signal and the need for propagating the (uncertain) dynamics model forwards in time.
Revised: reward signal and the need for propagating the (uncertain) dynamics model forwards in time.

Original: However, using the simplifying assumptions above, the reduced MBRL problem equation 104 can in fact be solved analytically.
Revised: However, using the simplifying assumptions above, the reduced MBRL problem equation 103 can in fact be solved analytically.

Original: We can circumvent the equality constraints by predicting the state trajectory using the error corrected dynamics such that we obtain a lifted dynamics formulation similar to the analysis for ILC,
Revised: We can circumvent the equality constraints by predicting the state trajectory using the error corrected dynamics such that we obtain a lifted dynamics formulation similar to the analysis for ILC,

Original: [Equation]
Revised: [Equation]

Original: with s , a denoting the stacked states and actions of the recorded trajectory.
Revised: with s , a denoting the stacked states and actions of the recorded trajectory.

Original: Using this notation, the optimization problem reduces to
Revised: Using this notation, the optimization problem reduces to

Original: [Equation]
Revised: [Equation]

Original: with C π = diag
Revised: with C π = diag

Original: [ C π 0 , . . .
Revised: [ C π 0 , . . .

Original: , C πH ] .
Revised: , C πH ] .

Original: By inserting equation 105 into equation 106 we obtain the closedform solution as
Revised: By inserting equation 104 into equation 105 we obtain the closedform solution as

Original: [Equation]
Revised: [Equation]

Original: which, clearly, is equivalent to equation 100 for M
Revised: which, clearly, is equivalent to equation 99 for M

Original: = I and W = C π .
Revised: = I and W = C π .

Original: D. E XTENSIONS
Revised: E. E XTENSIONS

Original: In the previous section, we analyzed the most basic setting for NO-ILC, i.e., linear dynamics, no transition noise in the dynamics and no state/input constraints.
Revised: In the previous section, we analyzed the most basic setting for NO-ILC, i.e., linear dynamics, no transition noise in the dynamics and no state/input constraints.

Original: Some of these assumptions can easily be liftd to genearlized the presented framework
Revised: Some of these assumptions can easily be liftd to genearlized the presented framework.

Original: . Nonlinear Dynamics
Revised: Nonlinear Dynamics

Original: Instead of the system being deﬁned by ﬁxed matrices
Revised: Instead of the system being deﬁned by ﬁxed matrices

Original: A , B , we can just linearize a non-linear dynamics model f
Revised: A , B , we can just linearize a non-linear dynamics model f

Original: ( s , a ) around the last state/input trajectory such that
Revised: ( s , a ) around the last state/input trajectory such that

Original: [Equation]
Revised: [Equation]

Original: and the corresponding dynamics matrix for the lifted representation becomes (dropping the superscript notation indicating the iteration for clarity)
Revised: and the corresponding dynamics matrix for the lifted representation becomes (dropping the superscript notation indicating the iteration for clarity)

Original: [Equation]
Revised: [Equation]

Original: State/Input Constraints
Revised: State/Input Constraints

Original: Recall that we could solve the quadratic problems equation 99 and equation 104 analytically because we assumed no explicit inequality constraints on the state and inputs.
Revised: Recall that we could solve the quadratic problems equation 98 and equation 103 analytically because we assumed no explicit inequality constraints on the state and inputs.

Original: In practice, however, both states and actions are limited by physical or safety constraints typically given by polytopes.
Revised: In practice, however, both states and actions are limited by physical or safety constraints typically given by polytopes.

Original: While a closed-form solution is not readily available for this case, one can easily employ any numerical solver to deal with such constraints.
Revised: While a closed-form solution is not readily available for this case, one can easily employ any numerical solver to deal with such constraints.

Original: See, e.g., https://en.wikipedia.org/w/index.php?title=Quadratic_programming for a comprehensive list of available solvers.
Revised: See, e.g., https://en.wikipedia.org/w/index.php?title=Quadratic_programming for a comprehensive list of available solvers.

Original: Stochastic Dynamics
Revised: Stochastic Dynamics

Original: Schöllig & D’Andrea (2009) generalize the ILC setting by considering two separate sources of noise: 1) transition noise in the dynamics, i.e., s t +

Revised: Schöllig & D’Andrea (2009) generalize the ILC setting by considering two separate sources of noise: 1) transition noise in the dynamics, i.e., s t +

Original: = f
Revised: = f

Original: ( s t , a t ) + ω ft as well as 2) varying disturbances, i.e., d ( i +1)
Revised: ( s t , a t ) + ω f t as well as 2) varying disturbances, i.e., d ( i +1)

Original: =
Revised: =

Original: d ( i ) + ω d .
Revised: d ( i ) + ω d .

Original: Based on this model, a Kalman-Filter in the iteration-domain is developed that estimates the respective random variables and the ILC scheme is adapted to account for the estimated quantities.
Revised: Based on this model, a Kalman-Filter in the iteration-domain is developed that estimates the respective random variables and the ILC scheme is adapted to account for the estimated quantities.

Original: E H YPERPARAMETER S ETTINGS

Original: Below we list the most important hyperparameter settings.

Original: [Table]

Original: F I MPLEMENTATION AND C OMPUTATIONAL R ESOURCES

Original: Our implementation is based on the code from MBPO (Janner et al., 2019)

Original: [Table]

Original: which is open-sourced under the MIT license.

Original: All experiments were run on a HPC cluster

Original: [Table]

Original: where each individual experiment used one nVidia V100 GPU and four Intel Xeon CPUs.

Original: All experiments (including early debugging and evaluations) amounted to a total of 84‘713 hours

Original: [Table]

Original: which corresponds to roughly 9.7 years if the jobs ran sequentially.

Original: Most of this compute was required to ensure reproducability (ten random seeds per job and ablation studies over the effects of parameters)

Original: [Table]

Original: Since the cluster is part of an overall carbon-neutral infrastructure

Original: [Table]

Original: these experiments did not contribute to climate change.