Original: L EARNING POWERFUL POLICIES AND BETTER DYNAMICS MODELS BY ENCOURAGING CONSISTENCY
Revised: L EARNING POWERFUL POLICIES AND BETTER DYNAMICS MODELS BY ENCOURAGING CONSISTENCY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Model-based reinforcement learning approaches have the promise of being sample efﬁcient.
Revised: Model-based reinforcement learning approaches have the promise of being sample efﬁcient.

Original: Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning.
Revised: Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning.

Original: There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment.
Revised: There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment.

Original: Interaction with the environment allows humans to carry out experiments : taking actions that help uncover true casual relationships which can be used for building better dynamics models.
Revised: Interaction with the environment allows humans to carry out experiments : taking actions that help uncover true casual relationships which can be used for building better dynamics models.

Original: Analogously, we would expect such interaction to be helpful for a learning agent while learning to model the environment dynamics.
Revised: Analogously, we would expect such interaction to be helpful for a learning agent while learning to model the environment dynamics.

Original: In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by acting in the real world) and what it imagines (by acting in the “learned” world).
Revised: In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by acting in the real world) and what it imagines (by acting in the “learned” world).

Original: Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.
Revised: Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement Learning consists of two fundamental problems: learning and planning .
Revised: Reinforcement Learning consists of two fundamental problems: learning and planning .

Original: Learning comprises of improving the agent’s current policy by interacting with the environment while planning involves improving the policy without interacting with the environment.
Revised: Learning comprises of improving the agent’s current policy by interacting with the environment while planning involves improving the policy without interacting with the environment.

Original: These problems evolve into the dichotomy of model-free methods (which primarily rely on learning ) and modelbased methods (which primarily rely on planning ).
Revised: These problems evolve into the dichotomy of model-free methods (which primarily rely on learning ) and modelbased methods (which primarily rely on planning ).

Original: Recently, model-free methods have shown many successes, such as learning to play Atari games with pixel observations (Mnih et al., 2015b;
Revised: Recently, model-free methods have shown many successes, such as learning to play Atari games with pixel observations (Mnih et al., 2015b;

Original: Mnih et al., 2016) and learning complex motion skills from high dimensional inputs (Schulman et al., 2015a;b).
Revised: Mnih et al., 2016) and learning complex motion skills from high dimensional inputs (Schulman et al., 2015a;b).

Original: But their high sample complexity is a still a major criticism of the model-free approaches.
Revised: But their high sample complexity is a still a major criticism of the model-free approaches.

Original: In contrast, model-based reinforcement learning methods have been introduced in the literature where the goal is to improve the sample efﬁciency by learning a dynamics model of the environment.
Revised: In contrast, model-based reinforcement learning methods have been introduced in the literature where the goal is to improve the sample efﬁciency by learning a dynamics model of the environment.

Original: But model-based RL has several caveats.
Revised: But model-based RL has several caveats.

Original: If the policy takes the learner to an unexplored state in the environment, the learner’s model could make errors in estimating the environment dynamics, leading to sub-optimal behaviour.
Revised: If the policy takes the learner to an unexplored state in the environment, the learner’s model could make errors in estimating the environment dynamics, leading to sub-optimal behaviour.

Original: This problem is referred to as the model-bias problem (Deisenroth & Rasmussen, 2011).
Revised: This problem is referred to as the model-bias problem (Deisenroth & Rasmussen, 2011).

Original: In order to make prediction about the future, dynamics models are unrolled step by step which leads to the issue of “compounding errors” (Talvitie, 2014; Bengio et al., 2015; Lamb et al., 2016): an error in modeling the environment at time t affects the predicted observations at all subsequent time-steps.
Revised: In order to make prediction about the future, dynamics models are unrolled step by step which leads to the issue of “compounding errors” (Talvitie, 2014; Bengio et al., 2015; Lamb et al., 2016): an error in modeling the environment at time t affects the predicted observations at all subsequent time-steps.

Original: This problem is much more challenging for the environments where the agent observes high-dimensional image inputs and not compact state representations.
Revised: This problem is much more challenging for the environments where the agent observes high-dimensional image inputs and not compact state representations.

Original: On the other hand, modelfree algorithms are not limited by the accuracy of the model, and therefore can achieve better ﬁnal performance by trial and error, though at the expense of much higher sample complexity.
Revised: On the other hand, modelfree algorithms are not limited by the accuracy of the model, and therefore can achieve better ﬁnal performance by trial and error, though at the expense of much higher sample complexity.

Original: In the model-based approaches, the dynamics model is usually trained with supervised learning techniques mostly just by observing the data .
Revised: In the model-based approaches, the dynamics model is usually trained with supervised learning techniques mostly just by observing the data .

Original: On the other hand, there’s enough evidence that humans learn the environment dynamics not just by observing the environment but also by interacting with the environment (Cook et al., 2011; Daniels & Nemenman, 2015).
Revised: On the other hand, there’s enough evidence that humans learn the environment dynamics not just by observing the environment but also by interacting with the environment (Cook et al., 2011; Daniels & Nemenman, 2015).

Original: Interaction is useful as it allows the agent to carry out experiments in the real world to determine causality, which is clearly a desirable characteristic when building dynamics models.
Revised: Interaction is useful as it allows the agent to carry out experiments in the real world to determine causality, which is clearly a desirable characteristic when building dynamics models.

Original: This leads to an interesting possibility.
Revised: This leads to an interesting possibility.

Original: The agent could consider two possible pathways: (i) Taking actions in the real world to generate new observations and (ii) Imagining to take actions and predicting the new observations.
Revised: The agent could consider two possible pathways: (i) Taking actions in the real world to generate new observations and (ii) Imagining to take actions and predicting the new observations.

Original: Consider the humanoid robot from the MuJoCo environment (Mordatch et al., 2015).
Revised: Consider the humanoid robot from the MuJoCo environment (Mordatch et al., 2015).

Original: In the ﬁrst case, the humanoid agent takes an action in the real environment, observes the change in its position (and location), takes another step and so on.
Revised: In the ﬁrst case, the humanoid agent takes an action in the real environment, observes the change in its position (and location), takes another step and so on.

Original: In the second case, the agent imagines taking a step, predicts what the observation would look like, imagines taking another step and so on.
Revised: In the second case, the agent imagines taking a step, predicts what the observation would look like, imagines taking another step and so on.

Original: The ﬁrst case is a close-loop setup, where the humanoid observes the state of the world, takes an action, gets the true observation from the environment, which is used to choose the next action, and so on.
Revised: The ﬁrst case is a close-loop setup, where the humanoid observes the state of the world, takes an action, gets the true observation from the environment, which is used to choose the next action, and so on.

Original: The second case is a open-loop setup, where the agent predicts subsequent states for multiple time steps into the future without interacting with the environment.
Revised: The second case is a open-loop setup, where the agent predicts subsequent states for multiple time steps into the future without interacting with the environment.

Original: The two cases have been summarized in ﬁgure 1.
Revised: The two cases have been summarized in ﬁgure 1.

Original: As such, the two pathways may not to be “consistent” given the challenges in learning a multi-step dynamics model.
Revised: As such, the two pathways may not to be “consistent” given the challenges in learning a multi-step dynamics model.

Original: By “consistent”, we mean the behaviour of state transitions along the two paths should be indistinguishable.
Revised: By “consistent”, we mean the behaviour of state transitions along the two paths should be indistinguishable.

Original: Had they been consistent, the learner’s model would be grounded in reality ie the predictions from the open loop would be similar to the predictions from the closed loop over a long time horizon.
Revised: Had they been consistent, the learner’s model would be grounded in reality ie the predictions from the open loop would be similar to the predictions from the closed loop over a long time horizon.

Original: In this work, we propose to ensure consistency by using an auxiliary loss which explicitly seeks to match the generative behaviour (from open loop) and the observed behaviour (from closed loop) as closely as possible.
Revised: In this work, we propose to ensure consistency by using an auxiliary loss which explicitly seeks to match the generative behaviour (from open loop) and the observed behaviour (from closed loop) as closely as possible.

Original: More generally, we show that the proposed approach helps to simultaneously train more powerful policies as well as better dynamics models, by using a training objective that is not solely focused on predicting the next observation.
Revised: More generally, we show that the proposed approach helps to simultaneously train more powerful policies as well as better dynamics models, by using a training objective that is not solely focused on predicting the next observation.

Original: Our evaluation protocol consists of learning both observation-space models and state-space models.
Revised: Our evaluation protocol consists of learning both observation-space models and state-space models.

Original: We consider various continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016), and
Revised: We consider various continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016), and

Original: RLLab (Duan et al., 2016) and show that using the proposed auxiliary loss consistently helps in achieving better performance across tasks.
Revised: RLLab (Duan et al., 2016) and show that using the proposed auxiliary loss consistently helps in achieving better performance across tasks.

Original: We evaluate the proposed approach on the pixel-based cheetah domain from the OpenAI Gym suite (Brockman et al., 2016).
Revised: We evaluate the proposed approach on the pixel-based cheetah domain from the OpenAI Gym suite (Brockman et al., 2016).

Original: This domain is bit difﬁcult for the “baseline” state space models as one can only infer positions and not velocities from the images and hence this makes the task partially observable.
Revised: This domain is bit difﬁcult for the “baseline” state space models as one can only infer positions and not velocities from the images and hence this makes the task partially observable.

Original: We compare the proposed model to the state of the art state space models (Buesing et al., 2018), and show that the proposed method consistently achieves better results.
Revised: We compare the proposed model to the state of the art state space models (Buesing et al., 2018), and show that the proposed method consistently achieves better results.

Original:  P RELIMARIES
Revised:  P RELIMARIES

Original: A ﬁnite time Markov decision process M is generally deﬁned by the tuple ( S , A , f, R, γ ) .
Revised: A ﬁnite time Markov decision process M is generally deﬁned by the tuple ( S , A , f, R, γ ) .

Original: Here, A the action space, S is the set of states, f ( s t +1 | s t , a t ) the transition distribution, r : S ×
Revised: Here, A the action space, S is the set of states, f ( s t +1 | s t , a t ) the transition distribution, r : S ×

Original: A → R is the reward function and γ the discount factor.
Revised: A → R is the reward function and γ the discount factor.

Original: We deﬁne the return as the  sum of rewards r ( s t , a t ) along a trajectory τ :=
Revised: We deﬁne the return as the discounted sum of rewards r ( s t , a t ) along a trajectory τ :=

Original: ( s 0 , a 0 , ..., s T − 1 , a T − 1 , s T ) , here T refers to the effective horizon of the process.
Revised: ( s 0 , a 0 , ..., s T − 1 , a T − 1 , s T ) , here T refers to the effective horizon of the process.

Original: The goal of reinforcement learning is to ﬁnd a policy  that maximizes the expected return.
Revised: The goal of reinforcement learning is to ﬁnd a policy π φ that maximizes the expected return.

Revised: Here φ denotes the parameters of the policy π .

Original: Model-based RL methods learn the dynamics model from the observed transitions.
Revised: Model-based RL methods learn the dynamics model from the observed transitions.

Original: This is usually done with a function approximator usually parameterized as a neural network ˆ f φ ( s t +1 | s t , a t ) .
Revised: This is usually done with a function approximator usually parameterized as a neural network ˆ f θ ( s t +1 | s t , a t ) .

Original: In such case, the parameters φ of the dynamics model are optimized to maximize the log-likelihood of the state transition distribution.
Revised: In such case, the parameters θ of the dynamics model are optimized to maximize the log-likelihood of the state transition distribution.

Original:  E NVIRONMENT M ODEL
Revised:  E NVIRONMENT M ODEL

Original: Consider a learning agent which is training to optimize a reward signal r in a given environment.
Revised: Consider a learning agent which is training to optimize a reward signal r in a given environment.

Original: At a given timestep t , the agent is in some state s t ∈ S .
Revised: At a given timestep t , the agent is in some state s t ∈ S .

Original: It takes an action a t ∈ A according to its policy a t ∼ π t ( a t | s t ) , receives a reward r t (from the environment) and transitions to a new state s t +1 .
Revised: It takes an action a t ∈ A according to its policy a t ∼ π t ( a t | s t ) , receives a reward r t (from the environment) and transitions to a new state s t +1 .

Original: The agent is trying to maximize its expected reward and has two pathways for improving its behaviour:
Revised: The agent is trying to maximize its expected reward and has two pathways for improving its behaviour:

Original: Close-loop path:
Revised: Close-loop path:

Original: Here, the learner interacts with the environment at every step.
Revised: Here, the learner interacts with the environment at every step.

Original: The agent starts in state s0 and is in state s t at time t .
Revised: The agent starts in state s and is in state s t at time t .

Original: It chooses an action a t to perform (using its policy π t ), performs the chosen action, and receives a reward r t .
Revised: It chooses an action a t to perform (using its policy π t ), performs the chosen action, and receives a reward r t .

Original: It then observes the environment to obtain the new state s t +1 , uses this state to decide which action a t +2 to perform next and so on.
Revised: It then observes the environment to obtain the new state s t +1 , uses this state to decide which action a t +2 to perform next and so on.

Original: Open-loop path:
Revised: Open-loop path:

Original: Here, the learner predicts the future observations (or future belief state in case of state space models).
Revised: Here, the learner predicts the future observations (or future belief state in case of state space models).

Original: The agent starts in state s 0 and in state s t at time t .
Revised: The agent starts in state s 0 and in state s t at time t .

Original: Note that the agent ”imagines” itself to be in state s It and can not access the true state of the environment.
Revised: Note that the agent ”imagines” itself to be in state s It and can not access the true state of the environment.

Original: It chooses an action a t to perform (using its policy π t ), performs the action in the “learners” world and imagines to transition to the new state s It +1 .
Revised: It chooses an action a t to perform (using its policy π t ), performs the action in the “learners” world and imagines to transition to the new state s It +1 .

Original: Thus the current “imagined” state is used to predict the next “imagined” state.
Revised: Thus the current “imagined” state is used to predict the next “imagined” state.

Original: During these “imagined” roll-outs, the agent does not interact with the environment but interacts with its “imagined” version of the environment which we call its dynamics model or the learner’s “world”.
Revised: During these “imagined” roll-outs, the agent does not interact with the environment but interacts with its “imagined” version of the environment which we call its dynamics model or the learner’s “world”.

Original: As an alternative, the agent could use both the pathways simultaneously.
Revised: As an alternative, the agent could use both the pathways simultaneously.

Original: The agent could, in parallel, (i) Build a model of the environment and (ii) Engage in interaction with the environment as shown in Figure 1.
Revised: The agent could, in parallel, (i) Build a model of the environment and (ii) Engage in interaction with the environment as shown in Figure 1.

Original: We propose to make the two pathways consistent with each other so as to ensure that the predictions from the learner’s dynamics model are grounded in the observations from the environment.
Revised: We propose to make the two pathways consistent with each other so as to ensure that the predictions from the learner’s dynamics model are grounded in the observations from the environment.

Original: We show that such a “consistency constraint” helps the agent to learn a powerful policy and a better dynamics model of the environment.
Revised: We show that such a “consistency constraint” helps the agent to learn a powerful policy and a better dynamics model of the environment.

Original: 3. C ONSISTENCY C ONSTRAINT
Revised: 3. C ONSISTENCY C ONSTRAINT

Original: We want the “imagined” behaviour (from the open loop) to be consistent with the observed behaviour (from the close loop).
Revised: We want the “imagined” behaviour (from the open loop) to be consistent with the observed behaviour (from the close loop).

Original: Thus making sure that the predictions from the learner’s dynamics model (or the “world”) are similar to the actual observations from the environment.
Revised: Thus making sure that the predictions from the learner’s dynamics model (or the “world”) are similar to the actual observations from the environment.

Original: The learner’s dynamics model could either be in observation space (pixel space) or it could be in state space.
Revised: The learner’s dynamics model could either be in observation space (pixel space) or it could be in state space.

Original: State space models are generally more efﬁcient as they model dynamics at some higher level of abstraction.
Revised: State space models are generally more efﬁcient as they model dynamics at some higher level of abstraction.

Original: In the case of state space models, the learner predicts transitions in the state space by ﬁrst encoding the actual observation from the environment into the state space of the learner and then imposing the consistency constraint in the (learned) state space.
Revised: In the case of state space models, the learner predicts transitions in the state space by ﬁrst encoding the actual observation from the environment into the state space of the learner and then imposing the consistency constraint in the (learned) state space.

Original: In order to apply the consistency constraint in the open loop setup, we need to obtain the corresponding state transitions in the environment.
Revised: In order to apply the consistency constraint in the open loop setup, we need to obtain the corresponding state transitions in the environment.

Original: At a given timestep t , the learner is in some environment state s t while it imagines to be in state s It .
Revised: At a given timestep t , the learner is in some environment state s t while it imagines to be in state s It .

Original: It takes an action a t according to its policy a t ∼ π t ( a t
Revised: It takes an action a t according to its policy a t ∼ π t ( a t

Original: | s t ) .
Revised: | s t ) .

Original: Now the learner can make transition in two ways.
Revised: Now the learner can make transition in two ways.

Original: It could execute the action in the environment and transition to state s t +1 (as governed by the dynamics of the environment).
Revised: It could execute the action in the environment and transition to state s t +1 (as governed by the dynamics of the environment).

Original: Alternatively, it could execute the action in the “learned” dynamics environment ˆ f θ and imagine to transition to the state s It +

Revised: Alternatively, it could execute the action in the “learned” dynamics environment ˆ f θ and imagine to transition to the state s It +

Original: =
Revised: =

Original: ˆ f θ ( s It , a t ) .
Revised: ˆ f θ ( s It , a t ) .

Original: Note that the state s t is not used by the learner’s dynamics model when making state transitions during the open-loop setup.
Revised: Note that the state s t is not used by the learner’s dynamics model when making state transitions during the open-loop setup.

Original: Many possibilities exist for imposing the “consistency constraint”.
Revised: Many possibilities exist for imposing the “consistency constraint”.

Original: In a simple setup, we could enforce the per step output from the open and the closed loops to be similar.
Revised: In a simple setup, we could enforce the per step output from the open and the closed loops to be similar.

Original: The downside of this approach is that it encourages the dynamics model to mimic each and every detail of the environment irrespective of its utility in predicting the state transitions (Lamb et al., 2016).
Revised: The downside of this approach is that it encourages the dynamics model to mimic each and every detail of the environment irrespective of its utility in predicting the state transitions (Lamb et al., 2016).

Original: Hence, we encode the state transitions (during both open-loop and closed-loop) into ﬁxed-size real vectors using recurrent networks and enforce the output of the recurrent networks to be similar in the two cases.
Revised: Hence, we encode the state transitions (during both open-loop and closed-loop) into ﬁxed-size real vectors using recurrent networks and enforce the output of the recurrent networks to be similar in the two cases.

Original: Encoding the sequence can be seen as abstracting out the per-step state transitions into how the dynamics of the environment evolve over time.
Revised: Encoding the sequence can be seen as abstracting out the per-step state transitions into how the dynamics of the environment evolve over time.

Original: This way, we do not focus on mimicking each state but the high level dynamics of the state transitions.
Revised: This way, we do not focus on mimicking each state but the high level dynamics of the state transitions.

Original: This consistency loss encourages the dynamics model to only focus on information that makes the multi-step predictions (from the open-loop) indistinguishable from the actual future observations from the environment.
Revised: This consistency loss encourages the dynamics model to only focus on information that makes the multi-step predictions (from the open-loop) indistinguishable from the actual future observations from the environment.

Original: This is shown in ﬁgure 1.
Revised: This is shown in ﬁgure 1.

Original: Once we have the predicted state transitions and the real state transitions, we could impose the “consistency constraint” in several ways.
Revised: Once we have the predicted state transitions and the real state transitions, we could impose the “consistency constraint” in several ways.

Original: In order to keep our setup simple, we minimize the prediction error ( L 2 error) between the encoding of predicted future observations as coming from the learner’s dynamics model (during open-loop) and the encoding of the future observations as coming from the environment (during closed loop).
Revised: In order to keep our setup simple, we minimize the prediction error ( L 2 error) between the encoding of predicted future observations as coming from the learner’s dynamics model (during open-loop) and the encoding of the future observations as coming from the environment (during closed loop).

Original: Let us assume that the agent started in state s 0 and that a 0: T − 1 denote the sequence of actions that the agent takes in the environment from time t = 0 to T − 1 .
Revised: Let us assume that the agent started in state s 0 and that a 0: T − 1 denote the sequence of actions that the agent takes in the environment from time t = 0 to T − 1 .

Original: Similarly, s 1: T denotes the sequence of states that the agent transitions through.
Revised: Similarly, s 1: T denotes the sequence of states that the agent transitions through.

Original: Alternatively, the agent could have “imagined” a trajectory of state transitions by performing the actions a 0: T − 1 in the learner’s dynamics model.
Revised: Alternatively, the agent could have “imagined” a trajectory of state transitions by performing the actions a 0: T − 1 in the learner’s dynamics model.

Original: This would result in the sequence of states s I 1: T .
Revised: This would result in the sequence of states s I 1: T .

Original: The consistency loss is computed as follows:
Revised: The consistency loss is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)(cid:107) denotes the L2 norm and RNN denotes the GRU model used to encode the sequence of state transitions into a ﬁxed length vector.
Revised: where (cid:107)(cid:107) denotes the L2 norm and RNN denotes the GRU model used to encode the sequence of state transitions into a ﬁxed length vector.

Original: The agent which is trained with the consistency constraint is referred to as the consistent dynamics agent.
Revised: The agent which is trained with the consistency constraint is referred to as the consistent dynamics agent.

Original: The overall loss for such a learning agent can be written as follows:
Revised: The overall loss for such a learning agent can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where θ refers to the parameters of the agent’s transition model ˆ f and φ refers to the parameters of the agent’s policy π .
Revised: where θ refers to the parameters of the agent’s transition model ˆ f and φ refers to the parameters of the agent’s policy π .

Original: The ﬁrst component of the loss function, l rl ( θ, φ ) , corresponds to the RL objective i.e maximizing expected reward and is referred to as the RL loss.
Revised: The ﬁrst component of the loss function, l rl ( θ, φ ) , corresponds to the RL objective i.e maximizing expected return and is referred to as the RL loss.

Original: The second component of the loss, l cc ( θ, φ ) , corresponds to the loss associated with the consistency constraint and is referred to as consistency loss.
Revised: The second component of the loss, l cc ( θ, φ ) , corresponds to the loss associated with the consistency constraint and is referred to as consistency loss.

Original: α is a hyper-parameter to scale the consistency loss component with respect to the RL loss.
Revised: α is a hyper-parameter to scale the consistency loss component with respect to the RL loss.

Original: 3. O BSERVATION S PACE M ODEL
Revised: 3. O BSERVATION S PACE M ODEL

Original: For the observation space models, we represent the environment as a Markov Decision Process M with an unknown state transition function f : S ×
Revised: For the observation space models, we represent the environment as a Markov Decision Process M with an unknown state transition function f : S ×

Original: A → S .
Revised: A → S .

Original: Given a starting state s t ∈ S , the agent learns a policy function π t to choose an action a t ∈ A and a dynamics model ˆ f to predict the next state s t +1 given a state-action pair ( s t , a t ).
Revised: Given a starting state s t ∈ S , the agent learns a policy function π t to choose an action a t ∈ A and a dynamics model ˆ f to predict the next state s t +1 given a state-action pair ( s t , a t ).

Original: We use the hybrid model-based and model-free (Mb-Mf) algorithm
Revised: We use the hybrid model-based and model-free (Mb-Mf) algorithm

Original: (Nagabandi et al., 2017) as the baseline to design and learn the transition function and the policy.
Revised: (Nagabandi et al., 2017) as the baseline to design and learn the transition function and the policy.

Original: They propose to use a trained, deep neural network based dynamics model to initialize a model free learning agent to combine the sample efﬁciency of model-based approaches with the high task-speciﬁc performance of model-free methods.
Revised: They propose to use a trained, deep neural network based dynamics model to initialize a model free learning agent to combine the sample efﬁciency of model-based approaches with the high task-speciﬁc performance of model-free methods.

Original: Both the transition function and the policy are parameterized using neural networks (Gaussian Policies) as ˆ f θ ( s t , a t ) and π φ ( s t ) where θ and φ denote the parameters of the dynamics model and the policy respectively.
Revised: Both the transition function and the policy are parameterized using neural networks (Gaussian Policies) as ˆ f θ ( s t , a t ) and π φ ( s t ) where θ and φ denote the parameters of the dynamics model and the policy respectively.

Original: The details about model and policy implementation are provided in the appendix 7.1.1.
Revised: The details about model and policy implementation are provided in the appendix 7.1.1.

Original: In the close loop setup, the agent starts in a state s 0 .
Revised: In the close loop setup, the agent starts in a state s 0 .

Original: At any time t , it is in state s t , it chooses an action a t ∼ π t ( a t | s t ) , receives a reward r t and observes the next state s t +1 which it uses to choose the next action a t +1 .
Revised: At any time t , it is in state s t , it chooses an action a t ∼ π t ( a t | s t ) , receives a reward r t and observes the next state s t +1 which it uses to choose the next action a t +1 .

Original: In the open loop setup, the agent starts in a state s 0 .
Revised: In the open loop setup, the agent starts in a state s 0 .

Original: At any time t , it is in state s t , while it imagines to be in state s It .
Revised: At any time t , it is in state s t , while it imagines to be in state s It .

Original: It chooses an a t ∼ π t ( s t ) , imagines the next state s [-It +
-]
Revised: It chooses an a t ∼ π t ( s t ) , imagines the next state s [-It +
-]I t

Original:  =
Revised: +1 =

Original: ˆ f ( s It , a t ) .
Revised: ˆ f ( s It , a t ) .

Original: Simultaneously, the action a t is simulated in the environment to obtain the next environment state s t +1 .
Revised: Simultaneously, the action a t is simulated in the environment to obtain the next environment state s t +1 .

Original: These environment states are needed to ensure consistency between the learner’s imagination and actual state transitions.
Revised: These environment states are needed to ensure consistency between the learner’s imagination and actual state transitions.

Original: As described in equation 1, we encode the two state transition sequences into ﬁxed length vectors using recurrent models and then minimize the L norm between them.
Revised: As described in equation 1, we encode the two state transition sequences into ﬁxed length vectors using recurrent models and then minimize the L norm between them.

Original: 3. S TATE S PACE M ODEL
Revised: 3. S TATE S PACE M ODEL

Original: In case the observation space is high dimensional, as in case of pixel-wise observations(from high dimensional image data), state space models may be used to model the dynamics of the environment.
Revised: In case the observation space is high dimensional, as in case of pixel-wise observations(from high dimensional image data), state space models may be used to model the dynamics of the environment.

Original: These models can be computationally more efﬁcient than the observation space models (as in pixel space) as they make predictions at a higher level of abstraction and learn a compact representation of the observation.
Revised: These models can be computationally more efﬁcient than the observation space models (as in pixel space) as they make predictions at a higher level of abstraction and learn a compact representation of the observation.

Original: Further, it may be easier to model the environment dynamics in the latent space as compared to the high dimensional pixel space.
Revised: Further, it may be easier to model the environment dynamics in the latent space as compared to the high dimensional pixel space.

Original: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.
Revised: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.

Original: Consider a learning agent operating in an environment that produces an observation o t at every time-step t .
Revised: Consider a learning agent operating in an environment that produces an observation o t at every time-step t .

Original: These observations can be high-dimensional and highly redundant (for modelling the dynamics of the environment).
Revised: These observations can be high-dimensional and highly redundant (for modelling the dynamics of the environment).

Original: The agent learns to encode these observations ( o t ) into compact state-space representations ( s t ) using an encoder e .
Revised: The agent learns to encode these observations ( o t ) into compact state-space representations ( s t ) using an encoder e .

Original: The agent learns a policy function π to choose actions a t ∼ π ( a t
Revised: The agent learns a policy function π to choose actions a t ∼ π ( a t

Original: | s t ) .
Revised: | s t ) .

Original: The environment dynamics is given by an unknown observation transition function f : O × A → O and the agent aims to learn the model dynamics in state-space representation using a state transition function ˆ f .
Revised: The environment dynamics is given by an unknown observation transition function f : O × A → O and the agent aims to learn the model dynamics in state-space representation using a state transition function ˆ f .

Original: Both the policy and state transition functions are parameterized using neural networks as π φ and ˆ f θ where φ and θ represents the parameters of the policy and the transition function respectively.
Revised: Both the policy and state transition functions are parameterized using neural networks as π φ and ˆ f θ where φ and θ represents the parameters of the policy and the transition function respectively.

Original: A latent variable z t is introduced per timestep to introduce stochasticity in state transition function.
Revised: A latent variable z t is introduced per timestep to introduce stochasticity in state transition function.

Original: The observation space decoding o t +1 can be obtained from the pixel space encoding as o t +1 ∼ p ( o t +1 | s t , z t ) .
Revised: The observation space decoding o t +1 can be obtained from the pixel space encoding as o t +1 ∼ p ( o t +1 | s t , z t ) .

Original: We now describe the steps in the closed loop and open loop setup.
Revised: We now describe the steps in the closed loop and open loop setup.

Original: Close Loop: The agent starts in some state s 0 and receives an observation o 1 from the environment.
Revised: Close Loop: The agent starts in some state s 0 and receives an observation o 1 from the environment.

Original: At time t , the agent is in a state s t − 1 and receives an observation o t from the environment.
Revised: At time t , the agent is in a state s t − 1 and receives an observation o t from the environment.

Original: z t ∼ q ( z t | e ( o t ) , s t − 1 , a t − 1 ) 3.
Revised: z t ∼ q ( z t | e ( o t ) , s t − 1 , a t − 1 ) 3.

Original: Transition to a new state, s t
Revised: Transition to a new state, s t

Original: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.
Revised: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.

Original: Choose an action a t =
Revised: Choose an action a t =

Original: π ( a t | s t ) 5.
Revised: π ( a t | s t ) 5.

Original: Decode the state s t into observation o t +1 ∼ p ( o t +1 | s t , z t ) Open Loop: The agent starts in some state s 0 .
Revised: Decode the state s t into observation o t +1 ∼ p ( o t +1 | s t , z t ) Open Loop: The agent starts in some state s 0 .

Original: At time t , the agent is in a state s t − 1 .
Revised: At time t , the agent is in a state s t − 1 .

Original: z t ∼ p ( z t | s t − 1 , a t − 1 ) 3.
Revised: z t ∼ p ( z t | s t − 1 , a t − 1 ) 3.

Original: Transition to a new state, s t
Revised: Transition to a new state, s t

Original: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.
Revised: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.

Original: Choose an action a t =
Revised: Choose an action a t =

Original: π ( a t | s t )
Revised: π ( a t | s t )

Original: The open loop setup for the state space models is quite similar to the case of observation space models.
Revised: The open loop setup for the state space models is quite similar to the case of observation space models.

Original: At time t , the agent is in state s t , chooses an a t
Revised: At time t , the agent is in state s t , chooses an a t

Original: = π ( a t | s t ) , transitions to the next state s It + (using the learner’s dynamics function).
Revised: = π ( a t | s t ) , transitions to the next state s It + (using the learner’s dynamics function).

Original: Simultaneously, the action a t is simulated in the external environment to obtain the next environment observation o t +1 .
Revised: Simultaneously, the action a t is simulated in the external environment to obtain the next environment observation o t +1 .

Original: These environment observations are then encoded into the latent state and are needed to ensure consistency between the learner’s imagined state transition and the actual state transitions in the real environment.
Revised: These environment observations are then encoded into the latent state and are needed to ensure consistency between the learner’s imagined state transition and the actual state transitions in the real environment.

Original: s I 1: T denotes the sequence of states that the agent imagines and o 1: T denotes the sequence of observations that the agent obtains from the environment.
Revised: s I 1: T denotes the sequence of states that the agent imagines and o 1: T denotes the sequence of observations that the agent obtains from the environment.

Original: These observations are encoded into the state space to yield a sequence of encoded environment observations s 1: T .
Revised: These observations are encoded into the state space to yield a sequence of encoded environment observations s 1: T .

Original: We want to make the behaviour of sequence s 1: T indistinguishable from s I 1: T .
Revised: We want to make the behaviour of sequence s 1: T indistinguishable from s I 1: T .

Original: We follow the same approach as observation space models where we encode the two state transition sequences into ﬁxed length vectors using recurrent models and then minimize the L2 norm between them (as described in equation 1).
Revised: We follow the same approach as observation space models where we encode the two state transition sequences into ﬁxed length vectors using recurrent models and then minimize the L2 norm between them (as described in equation 1).

Original: The agent is trained by imitation learning using trajectories sampled using an expert policy.
Revised: The agent is trained by imitation learning using trajectories sampled using an expert policy.

Original: The details about the model and policy implementation are provided in the appendix 7.1.2.
Revised: The details about the model and policy implementation are provided in the appendix 7.1.2.

Original: Since most of the latent models with stochastic dynamics are trained with one step ahead predictions, they tend to suffer from inconsistent predictions when predicting multiple time steps into the future.
Revised: Since most of the latent models with stochastic dynamics are trained with one step ahead predictions, they tend to suffer from inconsistent predictions when predicting multiple time steps into the future.

Original: On the other hand, stochasticity is also important to capture long term dependencies.
Revised: On the other hand, stochasticity is also important to capture long term dependencies.

Original: By using the proposed consistency loss in the latent space, we can enforce that the multi-step predictions be grounded in the observations from the actual environment.
Revised: By using the proposed consistency loss in the latent space, we can enforce that the multi-step predictions be grounded in the observations from the actual environment.

Original: Hence, using the proposed consistency loss to improve the long term predictions (as shown empirically) can also be seen as a regularizer.
Revised: Hence, using the proposed consistency loss to improve the long term predictions (as shown empirically) can also be seen as a regularizer.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Model based RL A large majority of the literature in policy search relies on model-free methods, where no prior knowledge of the environment is required to ﬁnd an optimal policy, through either policy improvement (value-based methods, Rummery & Niranjan (1994); Mnih et al.
Revised: Model based RL A large majority of the literature in policy search relies on model-free methods, where no prior knowledge of the environment is required to ﬁnd an optimal policy, through either policy improvement (value-based methods, Rummery & Niranjan (1994); Mnih et al.

Original: (2015a)), or direct policy optimization (policy gradient methods, Mnih et al.
Revised: (2015a)), or direct policy optimization (policy gradient methods, Mnih et al.

Original: (2016); Schulman et al.
Revised: (2016); Schulman et al.

Original: (2015a)).
Revised: (2015a)).

Original: Although this is conceptually simple, these algorithms have a high sample complexity.
Revised: Although this is conceptually simple, these algorithms have a high sample complexity.

Original: To improve their sample-efﬁciency, one can learn a model of the environment alongside the policy, to sample experience from.
Revised: To improve their sample-efﬁciency, one can learn a model of the environment alongside the policy, to sample experience from.

Original: PILCO (Deisenroth & Rasmussen, 2011) is a model-based method that learns a probabilistic model of the dynamics of the environment, and incorporates the uncertainty provided by the model for planning on long-term horizons.
Revised: PILCO (Deisenroth & Rasmussen, 2011) is a model-based method that learns a probabilistic model of the dynamics of the environment, and incorporates the uncertainty provided by the model for planning on long-term horizons.

Original: This model of the dynamics induces a bias on the policy search though.
Revised: This model of the dynamics induces a bias on the policy search though.

Original: Previous work has tried to address the model-bias issue of model-based methods, by having a way to characterize the uncertainty of the models, and by learning a more robust policy (Deisenroth & Rasmussen, 2011;
Revised: Previous work has tried to address the model-bias issue of model-based methods, by having a way to characterize the uncertainty of the models, and by learning a more robust policy (Deisenroth & Rasmussen, 2011;

Original: Rajeswaran et al., 2016;
Revised: Rajeswaran et al., 2016;

Original: Lim et al., 2013).
Revised: Lim et al., 2013).

Original: Model Predictive Control (MPC, Lenz et al., 2015) has also been proposed in the literature to account for imperfect models by re-planning at each step, but it suffers from a high computational cost.
Revised: Model Predictive Control (MPC, Lenz et al., 2015) has also been proposed in the literature to account for imperfect models by re-planning at each step, but it suffers from a high computational cost.

Original: There is no sharp separation between model-free and model-based reinforcement learning, and often model-based methods are used in conjunction with model-free algorithms.
Revised: There is no sharp separation between model-free and model-based reinforcement learning, and often model-based methods are used in conjunction with model-free algorithms.

Original: One of the earliest example of this interaction is the classic Dyna algorithm (Sutton, 1991), which takes advantage of the model of the environment to generate simulated experiences, which get included in the training data of a model-free algorithm (like Q-learning, with Dyna-Q).
Revised: One of the earliest example of this interaction is the classic Dyna algorithm (Sutton, 1991), which takes advantage of the model of the environment to generate simulated experiences, which get included in the training data of a model-free algorithm (like Q-learning, with Dyna-Q).

Original: Extensions of Dyna have been proposed (Silver et al., 2008; Sutton et al., 2012), including deep neural-networks as function approximations.
Revised: Extensions of Dyna have been proposed (Silver et al., 2008; Sutton et al., 2012), including deep neural-networks as function approximations.

Original: Recently, the Model-assisted Bootstrapped DDPG (MA-DDPG, Kalweit & Boedecker, 2017) was proposed to incorporate model-based rollouts into a Deep Deterministic Policy Gradient method.
Revised: Recently, the Model-assisted Bootstrapped DDPG (MA-DDPG, Kalweit & Boedecker, 2017) was proposed to incorporate model-based rollouts into a Deep Deterministic Policy Gradient method.

Original: Recently, (Weber et al., 2017) used a predictive model in Imagination-Augmented Agents to provide additional context to a policy network.
Revised: Recently, (Weber et al., 2017) used a predictive model in Imagination-Augmented Agents to provide additional context to a policy network.

Original: Off-policy learning There are different approaches which combine on-policy learning algorithms with off-policy samples.
Revised: Off-policy learning There are different approaches which combine on-policy learning algorithms with off-policy samples.

Original: Recent examples of this approach include the interpolated policy gradient (Gu et al.)
Revised: Recent examples of this approach include the interpolated policy gradient (Gu et al.)

Original: , PGQ (O’Donoghue et al., 2016) and ACER (Wang et al., 2016), which combine policy gradient learning with ideas from off-policy learning, and a methodology inspired by Q-learning.
Revised: , PGQ (O’Donoghue et al., 2016) and ACER (Wang et al., 2016), which combine policy gradient learning with ideas from off-policy learning, and a methodology inspired by Q-learning.

Original: While we can incorporate experience from a behaviour policy to learn both the model of the environment as well as the policy (see Section 5.2.2), our method remains orthogonal to these works.
Revised: While we can incorporate experience from a behaviour policy to learn both the model of the environment as well as the policy (see Section 5.2.2), our method remains orthogonal to these works.

Original: We propose to ensure consistency between the open-loop and the closed-loop pathways as a means to learn a stronger policy, and better dynamics model.
Revised: We propose to ensure consistency between the open-loop and the closed-loop pathways as a means to learn a stronger policy, and better dynamics model.

Original: As such, our approach can be applied to a wide range of existing RL setups.
Revised: As such, our approach can be applied to a wide range of existing RL setups.

Revised: Several works have incorporated auxiliary loses which results in representations which can generalize.

Revised: Jaderberg et al.

Revised: (2016) considered pseudo reward functions which helps to generalize effectively across different Atari games.

Revised: In this work, we propose to use the consistency loss for improving the dynamics model in the context of reinforcement learning.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We designed our experiments to answer the following questions:
Revised: We designed our experiments to answer the following questions:

Original: • How does the proposed Consistent Dynamics model compares against the state-of-the-art approaches for both observation space models and state space models in terms of sample complexity and asymptotic performance?
Revised: • How does the proposed Consistent Dynamics model compares against the state-of-the-art approaches for both observation space models and state space models in terms of sample complexity and asymptotic performance?

Original: • Does adding the consistency constraint actually results in better dynamics model?
Revised: • Does adding the consistency constraint actually results in better dynamics model?

Original: All the experiments are performed using 3 random seeds.
Revised: All the experiments are performed using 3 random seeds.

Original: We consider different baselines for the the observation space model and the latent space models and describe them in the subsequent sections.
Revised: We consider different baselines for the the observation space model and the latent space models and describe them in the subsequent sections.

Original: 5. O BSERVATION S PACE M ODELS
Revised: 5. O BSERVATION S PACE M ODELS

Original: We use the hybrid model-based and model-free ( Mb-Mf ) algorithm
Revised: We use the hybrid model-based and model-free ( Mb-Mf ) algorithm

Original: (Nagabandi et al., 2017) as the baseline model for the observation space models.
Revised: (Nagabandi et al., 2017) as the baseline model for the observation space models.

Original: In this setup, the policy and the dynamics model are learnt jointly.
Revised: In this setup, the policy and the dynamics model are learnt jointly.

Original: The implementation details for these models have been described in the appendix 7.1.1.
Revised: The implementation details for these models have been described in the appendix 7.1.1.

Original: We quantify the advantage of using consistency constraint by considering 4 classical Mujoco environments from RLLab (Duan et al., 2016):
Revised: We quantify the advantage of using consistency constraint by considering 4 classical Mujoco environments from RLLab (Duan et al., 2016):

Original: Ant ( S ∈ R 41 , A ∈ R 8 ), Humanoid ( S ∈ R 142 ,
Revised: Ant ( S ∈ R 41 , A ∈ R 8 ), Humanoid ( S ∈ R 142 ,

Original: A ∈ R 21 ), Half-Cheetah ( S ∈ R 23 , A ∈ R 6 ) and Swimmer ( S ∈ R 17 , A ∈ R 3 ).
Revised: A ∈ R 21 ), Half-Cheetah ( S ∈ R 23 , A ∈ R 6 ) and Swimmer ( S ∈ R 17 , A ∈ R 3 ).

Original: For computing the consistency loss, the learner’s dynamics model is unrolled for k = 20 steps.
Revised: For computing the consistency loss, the learner’s dynamics model is unrolled for k = 20 steps.

Original: The imagined state transitions and the actual state transitions are encoded into ﬁxed length real vectors using GRU Cho et al.
Revised: The imagined state transitions and the actual state transitions are encoded into ﬁxed length real vectors using GRU Cho et al.

Original: We consider the effect of changing the unrolling length k as part of ablation studies.
Revised: We consider the effect of changing the unrolling length k as part of ablation studies.

Original: 5.1. A VERAGE E PISODIC R ETURN
Revised: 5.1. A VERAGE E PISODIC R ETURN

Original: The average episodic return (and the average discounted episodic return) is a good estimate of the effectiveness of the jointly trained dynamics model and policy.
Revised: The average episodic return (and the average discounted episodic return) is a good estimate of the effectiveness of the jointly trained dynamics model and policy.

Original: To show that consistency constraint helps in learning a more powerful policy and a better dynamics model, we compare the average episodic rewards for the baseline Mb-Mf model (which does not use consistency loss) and the proposed consistent dynamics model (which does use the consistency loss).
Revised: To show that consistency constraint helps in learning a more powerful policy and a better dynamics model, we compare the average episodic rewards for the baseline Mb-Mf model (which does not use consistency loss) and the proposed consistent dynamics model (which does use the consistency loss).

Original: We expect that using consistency would either lead to higher rewards or would enable the agent to achieve the same level of rewards (as no-consistency case) but in fewer updates.
Revised: We expect that using consistency would either lead to higher rewards or would enable the agent to achieve the same level of rewards (as no-consistency case) but in fewer updates.

Original: Figure 2 compares the average episodic returns for the agents trained with and without consistency.
Revised: Figure 2 compares the average episodic returns for the agents trained with and without consistency.

Original: We observe that using consistency helps to learn a better policy in fewer updates for all the four environments.
Revised: We observe that using consistency helps to learn a better policy in fewer updates for all the four environments.

Original: A similar trend is obtained for the average discounted returns (as shown in ﬁgurein appendix 7.2.1).
Revised: A similar trend is obtained for the average discounted returns (as shown in ﬁgurein appendix 7.2.1).

Original: Since we are learning both the policy and the model of the environment at the same time, these results indicate that using the consistency constraint helps to jointly learn a more powerful policy and a better dynamics model.
Revised: Since we are learning both the policy and the model of the environment at the same time, these results indicate that using the consistency constraint helps to jointly learn a more powerful policy and a better dynamics model.

Original: [Figure]
Revised: [Figure]

Original: 5.1. E FFECT OF CHANGING k
Revised: 5.1. E FFECT OF CHANGING k

Original: During the open-loop setup, the dynamics model is unrolled for k timesteps.
Revised: During the open-loop setup, the dynamics model is unrolled for k timesteps.

Original: The choice of k could be a critical hyper-parameter for controlling the effect of consistency constraint.
Revised: The choice of k could be a critical hyper-parameter for controlling the effect of consistency constraint.

Original: We study the effect of changing k (during training) on the average episodic return for the Ant and
Revised: We study the effect of changing k (during training) on the average episodic return for the Ant and

Original: Humanoid tasks, by training the agents with k ∈ { 5 , 20 } .
Revised: Humanoid tasks, by training the agents with k ∈ { 5 , 20 } .

Original: As an ablation, we also include the case of training the policy without using a model, in a fully model-free fashion.
Revised: As an ablation, we also include the case of training the policy without using a model, in a fully model-free fashion.

Original: We would expect that a smaller value of k would push the average episodic return of the consistent dynamics model closer to the Mb-Mf case.
Revised: We would expect that a smaller value of k would push the average episodic return of the consistent dynamics model closer to the Mb-Mf case.

Original: Figure 3 (Left) shows that a higher value of k ( k = 20 ) leads to better returns for both tasks. 
Revised: Figure 3 (Left) shows that a higher value of k ( k = 20 ) leads to better returns for both tasks. [Figure]

Original: 5. S TATE S PACE M ODELS
Revised: 5. S TATE S PACE M ODELS

Original: State space models are useful in scenarios where the observation space is high-dimensional and possibly redundant (eg. pixels-space observations).
Revised: State space models are useful in scenarios where the observation space is high-dimensional and possibly redundant (eg. pixels-space observations).

Original: In such cases, the state space models may be[Figure] used to learn the model dynamics in a condensed latent space.
Revised: In such cases, the state space models may be used to learn the model dynamics in a condensed latent space.

Original: These setups are more challenging than the observation space setup as here the agent also needs to learn an encoder and a decoder.
Revised: These setups are more challenging than the observation space setup as here the agent also needs to learn an encoder and a decoder.

Original: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as the baseline state space model.
Revised: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as the baseline state space model.

Original: We train an expert policy for sampling high-reward trajectories from the environment.
Revised: We train an expert policy for sampling high-reward trajectories from the environment.

Original: The details about the training setup are described in Appendix 7.1.2.
Revised: The details about the training setup are described in Appendix 7.1.2.

Original: In the Learning to Query agent, the trajectories are used to train the policy π φ using imitation learning and the dynamics model by maximum likelihood.
Revised: In the Learning to Query agent, the trajectories are used to train the policy π φ using imitation learning and the dynamics model by maximum likelihood.

Original: We consider 3 continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016): Half-Cheetah, Fetch-Push (Plappert et al., 2018) and Reacher.
Revised: We consider 3 continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016): Half-Cheetah, Fetch-Push (Plappert et al., 2018) and Reacher.

Original: During open loop, the dynamics model is unrolled for k = 10 steps for Half-Cheetah and k = 5 for Fetch-Push  Reacher.
Revised: During open loop, the dynamics model is unrolled for k = 10 steps for Half-Cheetah and k = 5 for Fetch-Push and Reacher.

Original: 5.2. E VALUATING D YNAMICS M ODELS
Revised: 5.2. E VALUATING D YNAMICS M ODELS

Original: We want to show that the consistency constraint helps to learn a better dynamics model of the environment.
Revised: We want to show that the consistency constraint helps to learn a better dynamics model of the environment.

Original: Since we learn a dynamics model over the states, we also need to jointly learn an observation model (decoder, see appendix 7.1.2) conditioned on the states.
Revised: Since we learn a dynamics model over the states, we also need to jointly learn an observation model (decoder, see appendix 7.1.2) conditioned on the states.

Original: We can then compute the log-likelihood of trajectories in the real environment (sampled with the expert policy) under this observation model.
Revised: We can then compute the log-likelihood of trajectories in the real environment (sampled with the expert policy) under this observation model.

Original: We compare the log-likelihoods corresponding to these observations for the Learning to Query agent (trained without the consistency loss) and Consistent Dynamics agent (trained with the consistency loss).
Revised: We compare the log-likelihoods corresponding to these observations for the Learning to Query agent (trained without the consistency loss) and Consistent Dynamics agent (trained with the consistency loss).

Original: We expect that the Consistent Dynamics agent would achieve a higher log likelihood.
Revised: We expect that the Consistent Dynamics agent would achieve a higher log likelihood.

Original: Figure 4 shows that in terms of imagination log likelihood, the Consistent Dynamics agent outperforms the Learning to Query baseline agent for all the 3 environments indicating that the agent learns a more powerful dynamics model of the environment.
Revised: Figure 4 shows that in terms of imagination log likelihood, the Consistent Dynamics agent outperforms the Learning to Query baseline agent for all the 3 environments indicating that the agent learns a more powerful dynamics model of the environment.

Original: Note that in case of Fetch-Push and
Revised: Note that in case of Fetch-Push and

Original: Reacher, we see improvements in the log-likelihood, even though the dynamics model is unrolled for just 5 steps.
Revised: Reacher, we see improvements in the log-likelihood, even though the dynamics model is unrolled for just 5 steps.

Original: 5.2. L EARNING THE POLICY BY I MITATION L EARNING
Revised: 5.2. L EARNING THE POLICY BY I MITATION L EARNING

Original: For the state-space models, we used the expert trajectories to train our policy π φ using imitation learning.
Revised: For the state-space models, we used the expert trajectories to train our policy π φ using imitation learning.

Original: To show that the consistency constraint helps to learn a more powerful policy, we compare the imitation learning loss for the Learning to Query agent (trained without the consistency loss) and Consistent Dynamics agent (trained with the consistency loss).
Revised: To show that the consistency constraint helps to learn a more powerful policy, we compare the imitation learning loss for the Learning to Query agent (trained without the consistency loss) and Consistent Dynamics agent (trained with the consistency loss).

Original: C ONCLUSION

Original: In this paper, we formulate a way to ensure consistency between the predictions of a dynamics model and the real observations from the environment thus allowing the agent to learn powerful policies, as well as better dynamics models.

Original: The learning agent, in parallel, (i) builds a model of the environment and (ii) engages in an interaction with the environment.

Original: This results in two sequences of state transitions: one in the real environment where the agent actually performs actions

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t
Revised: li

Original: [Figure]
Revised: [Figure]

Original: i
Revised: In this paper

Original: [Figure]
Revised: [Figure]

Original: li
Revised: we formulate a way to ensure consistency between the predictions of a dynamics model and the real observations from the environment thus allowing the agent to learn powerful policies, as well as better dynamics models.

Original: [Figure]
Revised: The learning agent, in parallel, (i) builds a model of the environment and (ii) engages in an interaction with the environment. This results in two sequences of state transitions: one in the real environment where the agent actually performs actions and other in the agent’s dynamics model (or the “world”) where it imagines taking actions, and hallucinates the state transitions.

Original:  from each other.
Revised: We apply an auxiliary loss which encourages the behaviour of state transitions across the two sequences to be indistinguishable from each other.

Original: We evaluate our proposed approach for both observation space models, and state space models and show that the agent learnsmore a  powerful policy and a better generative model.
Revised: We evaluate our proposed approach for both observation space models, and state space models and show that the agent learns a more powerful policy and a better generative model.

Original: Future work would consider how these 2 interaction pathways could lead to more targeted exploration.
Revised: Future work would consider how these 2 interaction pathways could lead to more targeted exploration.

Original: Furthermore, having more ﬂexibility over the length over which we unroll the model could allow the agent to take these decisions over multiple timescales.
Revised: Furthermore, having more ﬂexibility over the length over which we unroll the model could allow the agent to take these decisions over multiple timescales

Original:  A PPENDIX 7. E NVIRONMENT M ODEL 7.1. O BSERVATION S PACE M ODEL
Revised:  A PPENDIX 7. E NVIRONMENT M ODEL 7.1. O BSERVATION S PACE M ODEL

Original: We use the experimental setup, environments and the hybrid model-based and model-free (Mb-Mf) algorithm as described in (Nagabandi et al., 2017) 1 .
Revised: We use the experimental setup, environments and the hybrid model-based and model-free (Mb-Mf) algorithm as described in (Nagabandi et al., 2017) 1 .

Original: We consider two training scenarios: training a model-based learning agent with and without the consistency constraint.
Revised: We consider two training scenarios: training a model-based learning agent with and without the consistency constraint.

Original: The consistency constraint is applied by unrolling the model for multiple steps using the observations predicted by the learner’s dynamics model (closed-loop setup).
Revised: The consistency constraint is applied by unrolling the model for multiple steps using the observations predicted by the learner’s dynamics model (closed-loop setup).

Original: We train an on-policy RL algorithm for Cheetah, Humanoid,
Revised: We train an on-policy RL algorithm for Cheetah, Humanoid,

Original: Ant and Swimmer tasks from RLLab (Duan et al., 2016) control suite.
Revised: Ant and Swimmer tasks from RLLab (Duan et al., 2016) control suite.

Original: We report both the average discounted and average un-discounted reward obtained by the learner in the two cases: with and without the use of consistency constraint.
Revised: We report both the average discounted and average un-discounted reward obtained by the learner in the two cases: with and without the use of consistency constraint.

Original: The model and policy architectures for the observation space models are as follows:
Revised: The model and policy architectures for the observation space models are as follows:

Original: Transition Model :
Revised: Transition Model :

Original: The transition model ˆ f θ ( s t , a t ) has a Gaussian distribution with diagonal covariance, where the mean and covariance are parametrized by MLPs (Schulman et al., 2015a), which maps an observation vector s t and an action vector a t to a vector µ which speciﬁes a distribution over observation space.
Revised: The transition model ˆ f θ ( s t , a t ) has a Gaussian distribution with diagonal covariance, where the mean and covariance are parametrized by MLPs (Schulman et al., 2015a), which maps an observation vector s t and an action vector a t to a vector µ which speciﬁes a distribution over observation space.

Original: During training, the log likelihood p ( s | µ ) is maximized and state-representations can be sampled from p ( s | µ ) .
Revised: During training, the log likelihood p ( s | µ ) is maximized and state-representations can be sampled from p ( s | µ ) .

Original: Policy :
Revised: Policy :

Original: The learner’s policy ˆ π φ ( s t ) is also a Gaussian MLP which maps an observation vector s to a vector µ policy which speciﬁes a distribution over action space.
Revised: The learner’s policy ˆ π φ ( s t ) is also a Gaussian MLP which maps an observation vector s to a vector µ policy which speciﬁes a distribution over action space.

Original: Like before, the log-likelihood p ( a | µ ) is maximized and actions can be sampled from p ( a | µ ) .
Revised: Like before, the log-likelihood p ( a | µ ) is maximized and actions can be sampled from p ( a | µ ) .

Original: Learner’s policy and the dynamics model are implemented as Gaussian policies with MLPs as function approximations, and are trained using TRPO (Schulman et al., 2015a).
Revised: Learner’s policy and the dynamics model are implemented as Gaussian policies with MLPs as function approximations, and are trained using TRPO (Schulman et al., 2015a).

Original: Following the hybrid
Revised: Following the hybrid

Original: Mb-Mf approach(Nagabandi et al., 2017), we normalize the states and actions.
Revised: Mb-Mf approach(Nagabandi et al., 2017), we normalize the states and actions.

Original: The dynamics model is trained to predict the change in state ∆ s t as it can be difﬁcult to learn the state transition function when the states s t and s t
Revised: The dynamics model is trained to predict the change in state ∆ s t as it can be difﬁcult to learn the state transition function when the states s t and s t

Original: +1 are very similar and the action a t has a small effect on the output.
Revised: +1 are very similar and the action a t has a small effect on the output.

Original: 7.1. S TATE S PACE M ODEL
Revised: 7.1. S TATE S PACE M ODEL

Original: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.
Revised: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.

Original: The model and policy architecture for the state space models is as follows:
Revised: The model and policy architecture for the state space models is as follows:

Original: Encoder :
Revised: Encoder :

Original: The learner encodes the pixel-space observations ( 64 × 64 × 3 ) from the environment into state-space observations ( 256 dimensional vectors) with a convolutional encoder (4 convolutional layers with 4 × 4 kernels, stride 2 and 64 channels).
Revised: The learner encodes the pixel-space observations ( 64 × 64 × 3 ) from the environment into state-space observations ( 256 dimensional vectors) with a convolutional encoder (4 convolutional layers with 4 × 4 kernels, stride 2 and 64 channels).

Original: To model the velocity information, a stack of the latest 4 frames is used as the observation.
Revised: To model the velocity information, a stack of the latest 4 frames is used as the observation.

Original: The pixel-space observation at time t − 1 is denoted as o t − 1 , and is encoded into state-space observation s t − 1 . 2.
Revised: The pixel-space observation at time t − 1 is denoted as o t − 1 , and is encoded into state-space observation s t − 1 . 2.

Original: Transition Model :
Revised: Transition Model :

Original: The transition model is a Long Short-Term Memory model (LSTM, Hochreiter & Schmidhuber, 1997), that predicts the transitions in the state space.
Revised: The transition model is a Long Short-Term Memory model (LSTM, Hochreiter & Schmidhuber, 1997), that predicts the transitions in the state space.

Original: For every time-step t , latent variables z t are introduced, whose distribution is a function of previous state-space observation s t − 1 and previous action a t − 1 .
Revised: For every time-step t , latent variables z t are introduced, whose distribution is a function of previous state-space observation s t − 1 and previous action a t − 1 .

Original: ie z t ∼ p ( z t | s t − 1 , a t − 1 ) .
Revised: ie z t ∼ p ( z t | s t − 1 , a t − 1 ) .

Original: The output of the transition model is then a deterministic function of z t , s t − 1 , and a t − 1 .
Revised: The output of the transition model is then a deterministic function of z t , s t − 1 , and a t − 1 .

Original: ie s t = f ( z t , s t − 1 , a t − 1 ) .
Revised: ie s t = f ( z t , s t − 1 , a t − 1 ) .

Original: Stochastic Decoder : The learner can decode the state-space observations back into the pixel-space observations by use of stochastic convolutional decoder.
Revised: Stochastic Decoder : The learner can decode the state-space observations back into the pixel-space observations by use of stochastic convolutional decoder.

Original: The decoder takes as input the current state-space observation s t and the current latent variable z t and generates the current observation-space distribution from which the learner could sample an observation.
Revised: The decoder takes as input the current state-space observation s t and the current latent variable z t and generates the current observation-space distribution from which the learner could sample an observation.

Original: ie o t +1 ∼ p ( o t +1 | s t , z t ) .
Revised: ie o t +1 ∼ p ( o t +1 | s t , z t ) .

Original: This observation model is Gaussian, with a diagonal covariance.
Revised: This observation model is Gaussian, with a diagonal covariance.

Original: In the closed-loop trajectory, when the learner cannot interact with the environment, the latent variables are sampled from the prior distribution p ( z t | s t − 1 , a t − 1 ) .
Revised: In the closed-loop trajectory, when the learner cannot interact with the environment, the latent variables are sampled from the prior distribution p ( z t | s t − 1 , a t − 1 ) .

Original: The latent variables are sampled from Normal distributions with diagonal covariance matrices.
Revised: The latent variables are sampled from Normal distributions with diagonal covariance matrices.

Original: Since we cannot compute the loglikelihood L ( θ ) in a close form for the latent variable models, we minimize the evidence lower bound ELBO ( p posterior ) ≤ L ( θ ) .
Revised: Since we cannot compute the loglikelihood L ( θ ) in a close form for the latent variable models, we minimize the evidence lower bound ELBO ( p posterior ) ≤ L ( θ ) .

Original: As discussed previously, the consistency constraint is applied between the open-loop and closed-loop predictions with the aim of making their behaviour as similar as possible.
Revised: As discussed previously, the consistency constraint is applied between the open-loop and closed-loop predictions with the aim of making their behaviour as similar as possible.

Original: Figure 7 shows a graphical representation of the open-loop and close-loop pathways in the state-space model.
Revised: Figure 7 shows a graphical representation of the open-loop and close-loop pathways in the state-space model.

Original: Expert policy Having access to some policy trained on a large number of experience is required to sample high-quality trajectories with pixel-observations.
Revised: Expert policy Having access to some policy trained on a large number of experience is required to sample high-quality trajectories with pixel-observations.

Original: To train these expert policies, we used policy-based methods such as Proximal Policy Optimization (PPO, Schulman et al., 2017) for the half-cheetah and reacher environments, or Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG with HER, Andrychowicz et al., 2017) for the pushing task.
Revised: To train these expert policies, we used policy-based methods such as Proximal Policy Optimization (PPO, Schulman et al., 2017) for the half-cheetah and reacher environments, or Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG with HER, Andrychowicz et al., 2017) for the pushing task.

Original: The architectures and hyper-parameters used are similar to the ones given by the Baselines library (Dhariwal et al., 2017).
Revised: The architectures and hyper-parameters used are similar to the ones given by the Baselines library (Dhariwal et al., 2017).

Original: Note that these expert policies were trained on the state representation of the agents (ie.
Revised: Note that these expert policies were trained on the state representation of the agents (ie.

Original: the positions and velocities of their joints), while the trajectories were generated with pixel-observations captured from a view external to the agent.
Revised: the positions and velocities of their joints), while the trajectories were generated with pixel-observations captured from a view external to the agent.

Original: 7. R ESULTS 7.2. O BSERVATION S PACE M ODELS
Revised: 7. R ESULTS 7.2. O BSERVATION S PACE M ODELS

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: p i s
Revised: p i s

Original: [Figure]
Revised: [Figure]

Original: d i c R e t u r n
Revised: d i c R e t u r n

Original: [Figure]
Revised: [Figure]