Original: L EARNING POWERFUL POLICIES AND BETTER GENERATIVE MODELS BY INTERACTION
Revised: L EARNING POWERFUL POLICIES AND BETTER DYNAMICS MODELS BY ENCOURAGING CONSISTENCY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Model-based reinforcement learning approaches have the promise of being sample efﬁcient.
Revised: Model-based reinforcement learning approaches have the promise of being sample efﬁcient.

Original: Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning.
Revised: Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning.

Original: There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment.
Revised: There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment.

Original: Interaction with the environment allows humans to carry out experiments : taking actions that help uncover true casual relationships whichin turn can be used for building better dynamics models.
Revised: Interaction with the environment allows humans to carry out experiments : taking actions that help uncover true casual relationships which can be used for building better dynamics models.

Original: Analogously, we would expect such interaction to be helpful for a learning agent while it learns to model the dynamics of the environment.
Revised: Analogously, we would expect such interaction to be helpful for a learning agent while learning to model the environment dynamics.

Original: In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by actually performing actions in the real world) and what it hallucinates (by imagining to have taken actions in the environment).
Revised: In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by acting in the real world) and what it imagines (by acting in the “learned” world).

Original: Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.
Revised: Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement Learning consists of two fundamental problems: learning and planning .
Revised: Reinforcement Learning consists of two fundamental problems: learning and planning .

Original: Learning comprises of improving the agent’s current policy by interacting with the environment while planning involves improving the policy without interacting with the environment.
Revised: Learning comprises of improving the agent’s current policy by interacting with the environment while planning involves improving the policy without interacting with the environment.

Original: These problems evolve into the dichotomy of model-free methods (which primarily rely on learning ) and modelbased methods (which primarily rely on planning ).
Revised: These problems evolve into the dichotomy of model-free methods (which primarily rely on learning ) and modelbased methods (which primarily rely on planning ).

Original: Recently, model-free methods have shown many successes, such as learning to play Atari games with pixel observations (Mnih et al., 2015b;
Revised: Recently, model-free methods have shown many successes, such as learning to play Atari games with pixel observations (Mnih et al., 2015b;

Original: Mnih et al., 2016) and learning complex motion skills from high dimensional inputs (Schulman et al., 2015a;b).
Revised: Mnih et al., 2016) and learning complex motion skills from high dimensional inputs (Schulman et al., 2015a;b).

Original: But their high sample complexity is a still a major criticism of the model-free approaches.
Revised: But their high sample complexity is a still a major criticism of the model-free approaches.

Original: In contrast, model-based reinforcement learning methods have been introduced in the literature where the goal is to improve the sample efﬁciency by learning a dynamics model of the environment.
Revised: In contrast, model-based reinforcement learning methods have been introduced in the literature where the goal is to improve the sample efﬁciency by learning a dynamics model of the environment.

Original: But model-based RL has several caveats.
Revised: But model-based RL has several caveats.

Original: If the policy takes the learner to an unexplored state in the environment, the learner’s model could make errors in estimating the environment dynamics, leading to sub-optimal behaviour.
Revised: If the policy takes the learner to an unexplored state in the environment, the learner’s model could make errors in estimating the environment dynamics, leading to sub-optimal behaviour.

Original: This problem is referred to as the model-bias problem (Deisenroth & Rasmussen, 2011).
Revised: This problem is referred to as the model-bias problem (Deisenroth & Rasmussen, 2011).

Original: In order to make prediction about the future, dynamics models are unrolled step by step which leads to the issue of “compounding errors” (Talvitie, 2014; Bengio et al., 2015; Lamb et al., 2016): an error in modeling the environment at time t affects the predicted observations at all subsequent time-steps and hence the dynamics model may no longer be grounded in reality.
Revised: In order to make prediction about the future, dynamics models are unrolled step by step which leads to the issue of “compounding errors” (Talvitie, 2014; Bengio et al., 2015; Lamb et al., 2016): an error in modeling the environment at time t affects the predicted observations at all subsequent time-steps.

Original: This problem is much more challenging for the environments where the agent observes high-dimensional image inputs and not compact state representations.
Revised: This problem is much more challenging for the environments where the agent observes high-dimensional image inputs and not compact state representations.

Original: On the other hand, model-free algorithms are not limited by the accuracy of the model, and therefore can achieve better ﬁnal performance by trial and error, though at the expense of much higher sample complexity.
Revised: On the other hand, modelfree algorithms are not limited by the accuracy of the model, and therefore can achieve better ﬁnal performance by trial and error, though at the expense of much higher sample complexity.

Original: In the model-based approaches, the dynamics model is usually trained with supervised learning techniques i.e just by observing the data .
Revised: In the model-based approaches, the dynamics model is usually trained with supervised learning techniques mostly just by observing the data .

Original: On the other hand, there’s enough evidence that humans learn thedynamics of the environment  not just by observing the environment but also by interacting with the environment (Cook et al., 2011; Daniels & Nemenman, 2015).
Revised: On the other hand, there’s enough evidence that humans learn the environment dynamics not just by observing the environment but also by interacting with the environment (Cook et al., 2011; Daniels & Nemenman, 2015).

Original: Interaction is useful as it allows the agent to carry out experiments in the real world to determine causality, which is clearly a desirable characteristic when building dynamics models.
Revised: Interaction is useful as it allows the agent to carry out experiments in the real world to determine causality, which is clearly a desirable characteristic when building dynamics models.

Original: This leads to an interesting possibility.
Revised: This leads to an interesting possibility.

Original: The agent could consider two possible pathways: (i) Taking actions in the real world to generate new observations and (ii) Imagining to take actions and hallucinating the new observations.
Revised: The agent could consider two possible pathways: (i) Taking actions in the real world to generate new observations and (ii) Imagining to take actions and predicting the new observations.

Original: Consider the humanoid robot from the MuJoCo environment (Mordatch et al., 2015).
Revised: Consider the humanoid robot from the MuJoCo environment (Mordatch et al., 2015).

Original: In the ﬁrst case, the humanoid agentactually takes a step into the  environment, observes the change in its position (and location), takes another step and so on.
Revised: In the ﬁrst case, the humanoid agent takes an action in the real environment, observes the change in its position (and location), takes another step and so on.

Original: In the second case, the agent imagines taking a step, hallucinates what the observation would look like, imagines taking another step and so on.
Revised: In the second case, the agent imagines taking a step, predicts what the observation would look like, imagines taking another step and so on.

Original: The ﬁrst case is a close-loop setup, where the humanoid observes the state of the world,and then takes an action, gets the true observation from the environment, which is used to choose the next action, and so on.
Revised: The ﬁrst case is a close-loop setup, where the humanoid observes the state of the world, takes an action, gets the true observation from the environment, which is used to choose the next action, and so on.

Original: The second case is a open-loop setup, where the agent imagines for multiple time steps into the future without interacting with the environment.
Revised: The second case is a open-loop setup, where the agent predicts subsequent states for multiple time steps into the future without interacting with the environment.

Original: The two cases have been summarized in Figure 1.
Revised: The two cases have been summarized in ﬁgure 1.

Original: As such, the two pathways may not to be “consistent” given the challenges in learning a multi-step dynamics model.
Revised: As such, the two pathways may not to be “consistent” given the challenges in learning a multi-step dynamics model.

Original: By “consistent”, we mean the behaviour of state transitions along the two paths should be indistinguishable.
Revised: By “consistent”, we mean the behaviour of state transitions along the two paths should be indistinguishable.

Original: Had they been consistent, the learner’s model would be grounded in reality (in terms of being consistent with observations from the environment over a long time horizon).
Revised: Had they been consistent, the learner’s model would be grounded in reality ie the predictions from the open loop would be similar to the predictions from the closed loop over a long time horizon.

Original: In this work, we propose to ensure consistency by using an auxiliary loss which explicitly seeks to match the generative behaviour (from open loop) and the observed behaviour (from closed loop) as closely as possible.
Revised: In this work, we propose to ensure consistency by using an auxiliary loss which explicitly seeks to match the generative behaviour (from open loop) and the observed behaviour (from closed loop) as closely as possible.

Original: More generally, we show that the proposed approach helps to simultaneously train more powerful policies as well as better generative models, by using a training objective that is not solely focused on predicting the next observation.
Revised: More generally, we show that the proposed approach helps to simultaneously train more powerful policies as well as better dynamics models, by using a training objective that is not solely focused on predicting the next observation.

Original: Our evaluation protocol consists of learning both observation-space models and state-space models.
Revised: Our evaluation protocol consists of learning both observation-space models and state-space models.

Original: We consider various continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016), and RLLab (Duan et al., 2016) and show that using the proposed auxiliary loss  helps in achieving better performance across tasks.
Revised: We consider various continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016), and RLLab (Duan et al., 2016) and show that using the proposed auxiliary loss consistently helps in achieving better performance across tasks.

Revised: We evaluate the proposed approach on the pixel-based cheetah domain from the OpenAI Gym suite (Brockman et al., 2016).

Revised: This domain is bit difﬁcult for the “baseline” state space models as one can only infer positions and not velocities from the images and hence this makes the task partially observable.

Revised: We compare the proposed model to the state of the art state space models (Buesing et al., 2018), and show that the proposed method consistently achieves better results.

Original:  E NVIRONMENT M ODEL
Revised:  P RELIMARIES

Revised: A ﬁnite time Markov decision process M is generally deﬁned by the tuple ( S , A , f, R, γ ) . Here, A the action space, S is the set of states, f ( s t +1 | s t , a t ) the transition distribution, r : S × A → R is the reward function and γ the discount factor.

Revised: We deﬁne the return as the sum of rewards r ( s t , a t ) along a trajectory τ :=

Revised: ( s 0 , a 0 , ..., s T − 1 , a T − 1 , s T ) , here T refers to the effective horizon of the process.

Revised: The goal of reinforcement learning is to ﬁnd a policy that maximizes the expected return.

Revised: Model-based RL methods learn the dynamics model from the observed transitions.

Revised: This is usually done with a function approximator usually parameterized as a neural network ˆ f φ ( s t +1 | s t , a t ) .

Revised: In such case, the parameters φ of the dynamics model are optimized to maximize the log-likelihood of the state transition distribution.

Revised:  E NVIRONMENT M ODEL

Original: Consider a learning agent which is training to optimize a reward signal r in a given environment.
Revised: Consider a learning agent which is training to optimize a reward signal r in a given environment.

Original: At a given timestep t , the agent is in some state s t ∈ S .
Revised: At a given timestep t , the agent is in some state s t ∈ S .

Original: It takes an action a t ∈ A according to its policy a t ∼ π t ( a t | s t ) , receives a reward r t (from the environment) and transitions to a new state s t +1 .
Revised: It takes an action a t ∈ A according to its policy a t ∼ π t ( a t | s t ) , receives a reward r t (from the environment) and transitions to a new state s t +1 .

Original: The agent is trying to maximize its expected reward and has two pathways for improving its behaviour
Revised: The agent is trying to maximize its expected reward and has two pathways for improving its behaviour:

Original: Close-loop path, where the learner interacts with the environment at every step.
Revised: Close-loop path: Here, the learner interacts with the environment at every step.

Original: The agent startsat time t in a state s t , chooses an action a t to perform (using its policy π t ),actually performs the  action, and receives a reward r t .
Revised: The agent starts in state s 0 and is in state s t at time t . It chooses an action a t to perform (using its policy π t ), performs the chosen action, and receives a reward r t .

Original: It thenagain observes the environment to obtain the new state s t +1 , uses this state to decide which action a t +2 to perform next.
Revised: It then observes the environment to obtain the new state s t +1 , uses this state to decide which action a t +2 to perform next and so on.

Original: Open-loop path, where the learner imagines interacting with the environment.
Revised: Open-loop path: Here, the learner predicts the future observations (or future belief state in case of state space models). The agent starts in state s 0 and in state s t at time t . Note

Original: The agent starts at time t ina state s t , chooses an action a t to perform (using its policy π t ), imagines that it has performed the action  and hallucinates the new state s imagt +1 of the environment.
Revised: that the agent ”imagines” itself to be in state s It and can not access the true state of the environment. It chooses an action a t to perform (using its policy π t ), performs the action in the “learners” world and imagines to transition to the new state s It +1 .

Original: This imagined state is used to choose action a t +1 to perform next.
Revised: Thus the current “imagined” state is used to predict the next “imagined” state.

Original: During these “imagined” roll-outs, the agent does not interact with the environment but interacts with its “imagined” version of the environment which we callas its dynamics model or the learner’s “world”.
Revised: During these “imagined” roll-outs, the agent does not interact with the environment but interacts with its “imagined” version of the environment which we call its dynamics model or the learner’s “world”.

Original: As an alternative, the agent could use both the pathways simultaneously.
Revised: As an alternative, the agent could use both the pathways simultaneously.

Original: The agent could, in parallel, (i) build a model of the environment and (ii) engage inan interaction with the environment as shown in Figure 1.
Revised: The agent could, in parallel, (i) Build a model of the environment and (ii) Engage in interaction with the environment as shown in Figure 1.

Original: This could allow the agent to learn the environment dynamics not just by only observing the environment by also interacting with it.

Original: We propose to make the two pathways consistent with each other so as to ensure that the predictions from the learner’s dynamics model are grounded in the observations from the environment.
Revised: We propose to make the two pathways consistent with each other so as to ensure that the predictions from the learner’s dynamics model are grounded in the observations from the environment.

Original: We show that such a “consistency constraint” helps the agent to learn a powerful policy and a better dynamics model for the environment.
Revised: We show that such a “consistency constraint” helps the agent to learn a powerful policy and a better dynamics model of the environment.

Original: 2. C ONSISTENCY C ONSTRAINT
Revised: 3. C ONSISTENCY C ONSTRAINT

Original: We want the generative behaviour (from the open loop) to be consistent with the observed behaviour (from the close loop).
Revised: We want the “imagined” behaviour (from the open loop) to be consistent with the observed behaviour (from the close loop).

Original: We want to make sure that the prediction from the learner’s dynamics model (or the “world”) are similar to the  observations from the environment.
Revised: Thus making sure that the predictions from the learner’s dynamics model (or the “world”) are similar to the actual observations from the environment.

Original: The learner’s dynamics model could either be in observation space (pixelspace or feature space) or it could be in state space.
Revised: The learner’s dynamics model could either be in observation space (pixel space) or it could be in state space.

Original: State space models are generally more efﬁcient as they model dynamics at some higher level of abstraction.
Revised: State space models are generally more efﬁcient as they model dynamics at some higher level of abstraction.

Original: In the case of state space models, the learner predicts transitions in the state space (representing the learner’s “world”). In this case, the learner ﬁrst encode the actual observations from the environment into the state space of the learner and then impose the consistency constraint in the  state space.
Revised: In the case of state space models, the learner predicts transitions in the state space by ﬁrst encoding the actual observation from the environment into the state space of the learner and then imposing the consistency constraint in the (learned) state space.

Original: When we want to apply the consistency constraint in the open loop setup, we need to obtain the corresponding state transitions in the environment.
Revised: In order to apply the consistency constraint in the open loop setup, we need to obtain the corresponding state transitions in the environment.

Revised: At a given timestep t , the learner is in some environment state s t while it imagines to be in state s It .

Revised: It takes an action a t according to its policy a t ∼ π t ( a t

Original: For each “imagined” action, a t , we perform that action in thereal environment  to obtain the new environment state.{+ˆ f θ and imagine to transition to the state s It +
+}
Revised: | s t ) . Now the learner can make transition in two ways. It could execute the action in the environment and transition to state s t +1 (as governed by the dynamics of the environment). Alternatively, it could execute the action in the “learned” dynamics environment {+ˆ f θ and imagine to transition to the state s It +
+}

Original: This environment state is only used for computing the consistency loss and is not used by the agent while imagining the subsequent state transitions.
Revised: = ˆ f θ ( s It , a t ) . Note that the state s t is not used by the learner’s dynamics model when making state transitions during the open-loop setup.

Original: Once we have the imagined transitions and the actual transitions, several possibilities exist for imposing the “consistency constraint”:
Revised: Many possibilities exist for imposing the “consistency constraint”. In a simple setup, we could enforce the per step output from the open and the closed loops to be similar.

Original: Minimize the prediction error between the future predictions as a result of running thelearner’s dynamics model  and  the  real observation coming from the environment.
Revised: The downside of this approach is that it encourages the dynamics model to mimic each and every detail of the environment irrespective of its utility in predicting the state transitions (Lamb et al., 2016). Hence, we encode the state transitions (during both open-loop and closed-loop) into ﬁxed-size real vectors using recurrent networks and enforce the output of the recurrent networks to be similar in the two cases. Encoding the sequence can be seen as abstracting out the per-step state transitions into how the dynamics of the environment evolve over time.

Original: Here, the  consistency loss would encourage the per-step predictions  from the learner’s “world” and environment to be similar to each other.
Revised: This way, we do not focus on mimicking each state but the high level dynamics of the state transitions. This consistency loss encourages the dynamics model to only focus on information that makes the multi-step predictions (from the open-loop) indistinguishable from the actual future observations from the environment. This is shown in ﬁgure 1. Once we have the predicted state transitions and the real state transitions, we could impose the “consistency constraint” in several ways.

Original: Encode theopen-loop prediction sequence (from the learner’s imagination) and the closedloop prediction sequence (from the environment) into ﬁxed size representations using RNNs.
Revised: In order to keep our setup simple, we minimize the prediction error ( L 2 error) between the encoding of predicted future observations as coming from the learner’s dynamics model (during open-loop) and the encoding of the future observations as coming from the environment (during closed loop). Let us assume that the agent started in state s 0 and that a 0: T − 1 denote the sequence of actions that the agent takes in the environment from time t = 0 to T − 1 . Similarly, s 1: T denotes the sequence of states that the agent transitions through. Alternatively, the agent could have “imagined” a trajectory of state transitions by performing the actions a 0: T − 1 in the learner’s dynamics model.

Original: A GAN discriminator could be trained on the output of  the  RNN to predict if the trajectory corresponds to  the learner’s “world” or the environment.
Revised: This would result in the sequence of states s I 1: T . The consistency loss is computed as follows: [Equation] where (cid:107)(cid:107) denotes the L2 norm and RNN denotes the GRU model used to encode the sequence of state transitions into a ﬁxed length vector.

Original: The dynamics function is trained with an auxiliary consistency loss which encourages it to fool the discriminator.
Revised: The agent which is trained with the consistency constraint is referred to as the consistent dynamics agent.

Original: This setup has previously
Revised: The overall loss for such a learning agent can be written as follows:

Original: also
Revised: [Equation]

Original: beein used in
Revised: where θ refers to the parameters of the agent’s transition model ˆ f and φ refers to the parameters of the agent’s policy π . The ﬁrst component of the loss function, l rl ( θ, φ ) , corresponds to the RL objective i.e maximizing expected reward and is referred to as the RL loss.

Original: Professor Forcing (Lamb et al., 2016).
Revised: The second component of the loss, l cc ( θ, φ ) , corresponds to the loss associated with the consistency constraint and is referred to as consistency loss. α is a hyper-parameter to scale the consistency loss component with respect to the RL loss.

Original: 2. O BSERVATION S PACE M ODEL
Revised: 3. O BSERVATION S PACE M ODEL

Original: For the observation space models, we represent the environment as a Markov Decision Process  with an unknown state transition function f : S ×
Revised: For the observation space models, we represent the environment as a Markov Decision Process M with an unknown state transition function f : S ×

Original: A → S .
Revised: A → S .

Original: Given a starting state s t ∈ S , the agent learns a policy function π t to choose an action a t ∈ A and a dynamics model ˆ f to predict the next state s t + given a state-action pair ( s t , a t ). 
Revised: Given a starting state s t ∈ S , the agent learns a policy function π t to choose an action a t ∈ A and a dynamics model ˆ f to predict the next state s t +1 given a state-action pair ( s t , a t ). We use the hybrid model-based and model-free (Mb-Mf) algorithm

Revised: (Nagabandi et al., 2017) as the baseline to design and learn the transition function and the policy.

Revised: They propose to use a trained, deep neural network based dynamics model to initialize a model free learning agent to combine the sample efﬁciency of model-based approaches with the high task-speciﬁc performance of model-free methods.

Original: Both the transition function and the policy are parameterized using neural networks (Gaussian Policies) as ˆ f θ ( s t , a t ) and π φ ( s t ) where θ and φ denote the parameters of the dynamics model and the policy respectively.
Revised: Both the transition function and the policy are parameterized using neural networks (Gaussian Policies) as ˆ f θ ( s t , a t ) and π φ ( s t ) where θ and φ denote the parameters of the dynamics model and the policy respectively.

Original: The details about model and policy implementation are provided in the appendix 6.1.1.
Revised: The details about model and policy implementation are provided in the appendix 7.1.1.

Original: In the close loop setup, the agent starts in a state s  t ,  chooses an action a t ∼ π t ( a t | s t ) , receives a reward r t and observes the next state s t +1 which it uses to choose the next action a t +1 .
Revised: In the close loop setup, the agent starts in a state s 0 . At any time t , it is in state s t , it chooses an action a t ∼ π t ( a t | s t ) , receives a reward r t and observes the next state s t +1 which it uses to choose the next action a t +1 .

Original: In the open loop setup, the agent starts in a state s  t ,  chooses an a t ∼ π t ( s t ) , imagines the next state s imagt +1 which it uses to choose the next action a t +1 ∼ π t ( s imagt ) .{+It +
 =+}
Revised: In the open loop setup, the agent starts in a state s 0 . At any time t , it is in state s t , while it imagines to be in state s It . It chooses an a t ∼ π t ( s t ) , imagines the next state s {+It +
 =+}

Original: External to the agent, we simulate the actions a t  in the environment to obtain the next environment state s t +1 .
Revised: ˆ f ( s It , a t ) . Simultaneously, the action a t is simulated in the environment to obtain the next environment state s t +1 .

Original: Note that the learning agent does not use these environment observations while it is “imagining” to perform actions.

Original: These environment states are needed to ensure consistency between the learner’s imagination and the state transitions grounded in the real environment.
Revised: These environment states are needed to ensure consistency between the learner’s imagination and actual state transitions.

Original: Let a imag 1: T denote the sequence of actions that the agent imagines to have taken from time t = 1 to T . Then, s imag 2: T +1 denotes the sequence of states that the learner hallucinates as a consequence of taking actions a imag 1: T . Similarly, s 2: T +1 denote the sequence of states that would have the learner would have visited in the environment had he actually taken those actions in the environment (instead of just imagining them). By the means of consistency constraint, we want to make the behaviour of sequence s 2: T +1 indistinguishable from s imag 2: T +1 .
Revised: As described in equation 1, we encode the two state transition sequences into ﬁxed length vectors using recurrent models and then minimize the L norm between them.

Original: In the empirical evaluation, we compare the average rewards obtained by the agent and the average model loss with and without the consistency constraint and show that our proposed consistency constraint helps to learn a more powerful policy and a better dynamics model.

Original: 2. S TATE S PACE M ODEL
Revised: 3. S TATE S PACE M ODEL

Original: In case the observation space is high dimensional, as in case of pixel-wise observations(from high dimensional image data), state space models may be used to model the dynamics of the environment.
Revised: In case the observation space is high dimensional, as in case of pixel-wise observations(from high dimensional image data), state space models may be used to model the dynamics of the environment.

Original: These models can be computationally more efﬁcient than the observation space models (as in pixel space) as they make predictions at a higher level of abstraction and learn a compact representation of the observation.
Revised: These models can be computationally more efﬁcient than the observation space models (as in pixel space) as they make predictions at a higher level of abstraction and learn a compact representation of the observation.

Original: Further, it may be easier to model the environment dynamics in the latent space as compared to the high dimensional pixel space.
Revised: Further, it may be easier to model the environment dynamics in the latent space as compared to the high dimensional pixel space.

Revised: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.

Original: Consider a learning agent operating in an environment that produces an observation o t at every time-step t .
Revised: Consider a learning agent operating in an environment that produces an observation o t at every time-step t .

Original: These observations can be high-dimensional and highly redundant (for modelling the dynamics of the environment).
Revised: These observations can be high-dimensional and highly redundant (for modelling the dynamics of the environment).

Original: The agent learns to encode these observations ( o t ) into compact state-space representations ( s t ) using an encoder e .
Revised: The agent learns to encode these observations ( o t ) into compact state-space representations ( s t ) using an encoder e .

Original: The agent learns a policy function πt to choose actions a t ∼ π ( a t
Revised: The agent learns a policy function π to choose actions a t ∼ π ( a t

Original: | s t ) .
Revised: | s t ) .

Original: The environment dynamics is given by an unknown observation transition function f : O × A → O and the agent aims to learn the model dynamics in state-space representation using a state transition function ˆ f .
Revised: The environment dynamics is given by an unknown observation transition function f : O × A → O and the agent aims to learn the model dynamics in state-space representation using a state transition function ˆ f .

Original: Both the policy and state transition functions are parameterized using neural networks as π φ and ˆ f θ where φ and θ represents the parameters of the policy and the transition function respectively.
Revised: Both the policy and state transition functions are parameterized using neural networks as π φ and ˆ f θ where φ and θ represents the parameters of the policy and the transition function respectively.

Revised: A latent variable z t is introduced per timestep to introduce stochasticity in state transition function.

Original: The observation space decoding o t +1 can be obtained from the pixel space encoding as o t +1 ∼ p ( o t +1 | s t , z t ) .
Revised: The observation space decoding o t +1 can be obtained from the pixel space encoding as o t +1 ∼ p ( o t +1 | s t , z t ) .

Original: We now describe the steps in the closed loop and open loop setup.
Revised: We now describe the steps in the closed loop and open loop setup.

Original: Close Loop: 
Revised: Close Loop: The agent starts in some state s 0 and receives an observation o 1 from the environment.

Original: The agent is in a state s t − 1 and receives an observation o t from the environment.
Revised: At time t , the agent is in a state s t − 1 and receives an observation o t from the environment.

Original: z t ∼ q ( z t | e ( o t ) , s t − 1 , a t − 1 ) 3.
Revised: z t ∼ q ( z t | e ( o t ) , s t − 1 , a t − 1 ) 3.

Original: Transition to a new state, s t
Revised: Transition to a new state, s t

Original: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.
Revised: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.

Original: Choose an action a t =
Revised: Choose an action a t =

Original: π ( a t | s t ) 5.
Revised: π ( a t | s t ) 5.

Original: Decode the state s t into observation o t +1 ∼ p ( o t +1 | s t , z t ) Open Loop: 
Revised: Decode the state s t into observation o t +1 ∼ p ( o t +1 | s t , z t ) Open Loop: The agent starts in some state s 0 .

Original: The agent is in a state s t − 1 .2.
Revised: At time t , the agent is in a state s t − 1 .

Original: z t ∼ p ( z t | s t − 1 , a t − 1 ) 3.
Revised: z t ∼ p ( z t | s t − 1 , a t − 1 ) 3.

Original: Transition to a new state, s t
Revised: Transition to a new state, s t

Original: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.
Revised: = ˆ f θ ( z t , s t − 1 , a t − 1 ) 4.

Original: Choose an action a t =
Revised: Choose an action a t =

Original: π ( a t | s t )
Revised: π ( a t | s t )

Original: The open loop setup for the state space models is quite similar to the case of observation space models.
Revised: The open loop setup for the state space models is quite similar to the case of observation space models.

Original: The agent starts ina state s t , chooses an a t=
Revised: At time t , the agent is in state s t , chooses an a t

Original:  πt ( a t | s t ) , imagines the next state s imagt + (using the transition function) which it uses to choose the next action a t +1 = π t ( s imagt ) .
Revised: = π ( a t | s t ) , transitions to the next state s It + (using the learner’s dynamics function).

Original: External to the agent, we simulate the actions a t  in the  environment to obtain the next environment observation o t +1 .
Revised: Simultaneously, the action a t is simulated in the external environment to obtain the next environment observation o t +1 .

Original: These environment observations are then encoded into the latent state and are needed to ensure consistency between the learner’s imagination and the  state transitions in the real environment.
Revised: These environment observations are then encoded into the latent state and are needed to ensure consistency between the learner’s imagined state transition and the actual state transitions in the real environment.

Original: s imag 2: T+1 denotes the sequence of states that the learner hallucinates and s 2: T +1 denote the sequence ofencoded environment observations that the learner would have received from the environment had he actually taken actions in the environment (instead of just imagining them).
Revised: s I 1: T denotes the sequence of states that the agent imagines and o 1: T denotes the sequence of observations that the agent obtains from the environment. These observations are encoded into the state space to yield a sequence of encoded environment observations s 1: T .

Original: Like the previous case, we want to make the behaviour of sequence s 2: T+1 indistinguishable from s imag 2: T+1 .
Revised: We want to make the behaviour of sequence s 1: T indistinguishable from s I 1: T .

Revised: We follow the same approach as observation space models where we encode the two state transition sequences into ﬁxed length vectors using recurrent models and then minimize the L2 norm between them (as described in equation 1).

Original: The agent is trained by imitation learning using trajectories sampled using an expert policy.
Revised: The agent is trained by imitation learning using trajectories sampled using an expert policy.

Original: The details about the model and policy implementation are provided in the appendix 6.1.2.
Revised: The details about the model and policy implementation are provided in the appendix 7.1.2.

Original: In the empirical evaluation, we compare the imitation learning loss and the model log-likelihood with and without the  consistency constraint.
Revised: Since most of the latent models with stochastic dynamics are trained with one step ahead predictions, they tend to suffer from inconsistent predictions when predicting multiple time steps into the future. On the other hand, stochasticity is also important to capture long term dependencies. By using the proposed consistency loss in the latent space, we can enforce that the multi-step predictions be grounded in the observations from the actual environment.

Original: We show that our proposed consistency constraint helps to learn a strong policy and a better dynamics model.
Revised: Hence, using the proposed consistency loss to improve the long term predictions (as shown empirically) can also be seen as a regularizer.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Model based RL A large majority of the literature in policy search relies on model-free methods, where no prior knowledge of the environment is required to ﬁnd an optimal policy, through either policy improvement (value-based methods, Rummery & Niranjan (1994); Mnih et al.
Revised: Model based RL A large majority of the literature in policy search relies on model-free methods, where no prior knowledge of the environment is required to ﬁnd an optimal policy, through either policy improvement (value-based methods, Rummery & Niranjan (1994); Mnih et al.

Original: (2015a)), or direct policy optimization (policy gradient methods, Mnih et al.
Revised: (2015a)), or direct policy optimization (policy gradient methods, Mnih et al.

Original: (2016); Schulman et al.
Revised: (2016); Schulman et al.

Original: (2015a)).
Revised: (2015a)).

Original: Although this is conceptually simple, these algorithms have a high sample complexity.
Revised: Although this is conceptually simple, these algorithms have a high sample complexity.

Original: To improve their sample-efﬁciency, one can learn a model of the environment alongside the policy, to sample experience from.
Revised: To improve their sample-efﬁciency, one can learn a model of the environment alongside the policy, to sample experience from.

Original: PILCO (Deisenroth & Rasmussen, 2011) is a model-based method that learns a probabilistic model of the dynamics of the environment, and incorporates the uncertainty provided by the model for planning on long-term horizons.
Revised: PILCO (Deisenroth & Rasmussen, 2011) is a model-based method that learns a probabilistic model of the dynamics of the environment, and incorporates the uncertainty provided by the model for planning on long-term horizons.

Original: This model of the dynamics induces a bias on the policy search though.
Revised: This model of the dynamics induces a bias on the policy search though.

Original: Previous work has tried to address the model-bias issue of model-based methods, by having a way to characterize the uncertainty of the models, and by learning a more robust policy (Deisenroth & Rasmussen, 2011;
Revised: Previous work has tried to address the model-bias issue of model-based methods, by having a way to characterize the uncertainty of the models, and by learning a more robust policy (Deisenroth & Rasmussen, 2011;

Original: Rajeswaran et al., 2016;
Revised: Rajeswaran et al., 2016;

Original: Lim et al., 2013).
Revised: Lim et al., 2013).

Original: Model Predictive Control (MPC, Lenz et al., 2015) has also been proposed in the literature to account for imperfect models by re-planning at each step, but it suffers from a high computational cost.
Revised: Model Predictive Control (MPC, Lenz et al., 2015) has also been proposed in the literature to account for imperfect models by re-planning at each step, but it suffers from a high computational cost.

Original: There is no sharp separation between model-free and model-based reinforcement learning, and often model-based methods are used in conjunction with model-free algorithms.
Revised: There is no sharp separation between model-free and model-based reinforcement learning, and often model-based methods are used in conjunction with model-free algorithms.

Original: One of the earliest example of this interaction is the classic Dyna algorithm (Sutton, 1991), which takes advantage of the model of the environment to generate simulated experiences, which get included in the training data of a model-free algorithm (like Q-learning, with Dyna-Q).
Revised: One of the earliest example of this interaction is the classic Dyna algorithm (Sutton, 1991), which takes advantage of the model of the environment to generate simulated experiences, which get included in the training data of a model-free algorithm (like Q-learning, with Dyna-Q).

Original: Extensions of Dyna have been proposed (Silver et al., 2008; Sutton et al., 2012), including deep neural-networks as function approximations.
Revised: Extensions of Dyna have been proposed (Silver et al., 2008; Sutton et al., 2012), including deep neural-networks as function approximations.

Original: Recently, the Model-assisted Bootstrapped DDPG (MA-DDPG, Kalweit & Boedecker, 2017) was proposed to incorporate model-based rollouts into a Deep Deterministic Policy Gradient method.
Revised: Recently, the Model-assisted Bootstrapped DDPG (MA-DDPG, Kalweit & Boedecker, 2017) was proposed to incorporate model-based rollouts into a Deep Deterministic Policy Gradient method.

Original: Recently, (Weber et al., 2017) used a predictive model in Imagination-Augmented Agents to provide additional context to a policy network.
Revised: Recently, (Weber et al., 2017) used a predictive model in Imagination-Augmented Agents to provide additional context to a policy network.

Original: Off-policy learning There are different approaches which combine on-policy learning algorithms with off-policy samples.
Revised: Off-policy learning There are different approaches which combine on-policy learning algorithms with off-policy samples.

Original: Recent examples of this approach include the interpolated policy gradient (Gu et al.)
Revised: Recent examples of this approach include the interpolated policy gradient (Gu et al.)

Original: , PGQ (O’Donoghue et al., 2016) and ACER (Wang et al., 2016), which combine policy gradient learning with ideas from off-policy learning, and a methodology inspired by Q-learning.
Revised: , PGQ (O’Donoghue et al., 2016) and ACER (Wang et al., 2016), which combine policy gradient learning with ideas from off-policy learning, and a methodology inspired by Q-learning.

Original: While we can incorporate experience from a behaviour policy to learn both the model of the environment as well as the policy (see Section 4.2.2), our method remains orthogonal to these works.
Revised: While we can incorporate experience from a behaviour policy to learn both the model of the environment as well as the policy (see Section 5.2.2), our method remains orthogonal to these works.

Original: We propose to ensure consistency between the open-loop and the closed-loop pathways as a means to learn a stronger policy, and better dynamics model.
Revised: We propose to ensure consistency between the open-loop and the closed-loop pathways as a means to learn a stronger policy, and better dynamics model.

Original: As such, our approach can be applied to a wide range of existing RL setups.
Revised: As such, our approach can be applied to a wide range of existing RL setups.

Original:   R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: The learning agent has two pathways for improving its behavior: (i) the close loop path, where it interacts with the environment by taking actions and observing the state transitions and (ii) the open loop path, where it imagines taking actions andhallucinates the state transitions that could happen. We show that ensuring “consistency” between what the learner predicts from the environment (with the dynamics model) and  the actual observations from the environment helps to:
Revised: We designed our experiments to answer the following questions: • How does the proposed Consistent Dynamics model compares against the state-of-the-art approaches for both observation space models and state space models in terms of sample complexity and asymptotic performance? • Does adding the consistency constraint actually results in better dynamics model?

Original: Learn a more powerful policy: we quantify the effect of the consistency constraint on the the policy by comparing the average episodic reward for observation space models, and the imitation learning loss for the state space models.
Revised: All the experiments are performed using 3 random seeds.

Original: Learn a better dynamics model: we evaluate the effect of theconsistency constraint on the dynamics model by comparing the  modelloss, with andwithout consistency, for the observation space models, and  the log-likelihood of trajectories from the real environment, under the model we learn.
Revised: We consider different baselines for the the observation space model and the latent space models and describe them in the subsequent sections.

Original: 4. O BSERVATION S PACE M ODELS
Revised: 5. O BSERVATION S PACE M ODELS

Original: We consider a model-based training approach (without the consistency constraint) as the baselinewhere the dynamics model and the policy  are learnt jointly.
Revised: We use the hybrid model-based and model-free ( Mb-Mf ) algorithm (Nagabandi et al., 2017) as the baseline model for the observation space models. In this setup, the policy and the dynamics model are learnt jointly.

Original: Details about the baselines are described in Appendix 6.1.1.
Revised: The implementation details for these models have been described in the appendix 7.1.1.

Original: We quantify the advantage of using consistency constraint by consideringclassical Mujoco environments from RLLab (Duan et al., 2016):
Revised: We quantify the advantage of using consistency constraint by considering 4 classical Mujoco environments from RLLab (Duan et al., 2016):

Original: Ant ( S ∈ R 41 , A ∈ R 8 ), Humanoid ( S ∈ R 142 , A ∈ R 21 ), Half-Cheetah ( S ∈ R 23 , A ∈ R 6 ) and Swimmer ( S ∈ R 17 , A ∈ R 3 ).
Revised: Ant ( S ∈ R 41 , A ∈ R 8 ), Humanoid ( S ∈ R 142 , A ∈ R 21 ), Half-Cheetah ( S ∈ R 23 , A ∈ R 6 ) and Swimmer ( S ∈ R 17 , A ∈ R 3 ).

Original: The batch size is ﬁxed to 50000 for all the cases. The dynamics model is unrolled for k = 20 steps for obtaining the sequence of imagined observations from the dynamics model, andactual observations from the environment (over which the consistency constraint is applied).
Revised: For computing the consistency loss, the learner’s dynamics model is unrolled for k = 20 steps. The imagined state transitions and the actual state transitions are encoded into ﬁxed length real vectors using GRU Cho et al.

Original: We describe the effect of changing the unrolling length in the subsequent sections.
Revised: We consider the effect of changing the unrolling length k as part of ablation studies.

Original: 4.1. A VERAGE E PISODIC R ETURN
Revised: 5.1. A VERAGE E PISODIC R ETURN

Original: The average episodic return (and the average discounted episodic return) is a good estimate of the effectiveness of the jointly trained dynamics model and policy.
Revised: The average episodic return (and the average discounted episodic return) is a good estimate of the effectiveness of the jointly trained dynamics model and policy.

Original: To show that consistency constraint helps in learning a more powerful policy and a better dynamics model, we compare the average episodic rewards for the cases where the agent is trained with and without consistency.
Revised: To show that consistency constraint helps in learning a more powerful policy and a better dynamics model, we compare the average episodic rewards for the baseline Mb-Mf model (which does not use consistency loss) and the proposed consistent dynamics model (which does use the consistency loss).

Original: We expect that using consistency would either lead to higher rewards or would enable the agent to achieve the same level of rewards (as no-consistency case) but in fewer updates.
Revised: We expect that using consistency would either lead to higher rewards or would enable the agent to achieve the same level of rewards (as no-consistency case) but in fewer updates.

Original: Figure 2 compares the average episodic returns for the agents trained with and without consistency.
Revised: Figure 2 compares the average episodic returns for the agents trained with and without consistency.

Original: We observe that using consistency helps to learn a better policy in fewer updates for all the 4 environments.
Revised: We observe that using consistency helps to learn a better policy in fewer updates for all the four environments.

Original: Similar trends are obtained with when we plot the average discounted returns (as shown in Figure 8 in Appendix 6.2.1).
Revised: A similar trend is obtained for the average discounted returns (as shown in ﬁgurein appendix 7.2.1).

Original: Since we are learning both the policy and the model of the environment at the same time, these results indicate that using the consistency constraint helps to jointly learn a more powerful policy and a better dynamics model.[Figure] i [Figure]
Revised: Since we are learning both the policy and the model of the environment at the same time, these results indicate that using the consistency constraint helps to jointly learn a more powerful policy and a better dynamics model.

Original: R

Original: [Figure]
Revised: [Figure]

Original: t [Figure] 4.1. E FFECT OF CHANGINGTHE OPEN - LOOP UNROLLING LENGTH k
Revised: 5.1. E FFECT OF CHANGING k

Original: During the open-loop setup, the dynamics model is unrolled for k timesteps.
Revised: During the open-loop setup, the dynamics model is unrolled for k timesteps.

Original: The choice of k could be a critical hyper-parameter for controlling the effect of consistency constraint.
Revised: The choice of k could be a critical hyper-parameter for controlling the effect of consistency constraint.

Original: In the extreme case, setting k = 1 is equivalent to a setup where the dynamics model is used for one-step predictions.

Original: We study the effect of changing k (during training) on the average episodic return for the Ant and
Revised: We study the effect of changing k (during training) on the average episodic return for the Ant and

Original: Humanoid tasks, by training the agents with k ∈ { 5 , 20 } .
Revised: Humanoid tasks, by training the agents with k ∈ { 5 , 20 } .

Original: As an ablation, we also include the case of training the policy without using a model, in a fully model-free fashion.
Revised: As an ablation, we also include the case of training the policy without using a model, in a fully model-free fashion.

Original: We would expect that a smaller value of k would push the average episodic return  closer to the no-consistency case.
Revised: We would expect that a smaller value of k would push the average episodic return of the consistent dynamics model closer to the Mb-Mf case.

Original: Figure(Left) shows that a higher value of k ( k = 20 ) leads to better returns for both tasks.[Figure]
Revised: Figure 3 (Left) shows that a higher value of k ( k = 20 ) leads to better returns for both tasks.

Original: We further explore the effect of different values of k on the model loss.

Original: This loss corresponds to an approximation (one-step ahead) of the log-likelihood of trajectories sampled from the true environment under our dynamics model.

Original: We expect a smaller value of k to have a lower onestep ahead prediction loss, compared to a larger value of k .

Original: The agent learns a high-level abstract representation of the states at the cost of low level details.

Original: The larger is the value of k , the longer is the prediction horizon for the agent, and so it is more likely to miss out on the low level details.

Original: Figure 3 (Right) shows that for one-step ahead predictions, the model with the consistency constraint and k = 5 obtains a lower loss than the one with k = 20 .

Original: 4. S TATE S PACE M ODELS
Revised: 5. S TATE S PACE M ODELS

Original: State space models are useful in scenarios where the observation space is high-dimensional and possibly redundant (eg. pixels-space observations).
Revised: State space models are useful in scenarios where the observation space is high-dimensional and possibly redundant (eg. pixels-space observations).

Original: In such cases, the state space models may be  used to learn the model dynamics in a condensed latent space.
Revised: In such cases, the state space models may be [Figure] used to learn the model dynamics in a condensed latent space.

Original: These setups are more challenging than the observation space setup as here the agent also needs to learn an encoder and a decoder.
Revised: These setups are more challenging than the observation space setup as here the agent also needs to learn an encoder and a decoder.

Revised: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as the baseline state space model.

Original: We train an expert policy for sampling high-reward trajectories from the environment.
Revised: We train an expert policy for sampling high-reward trajectories from the environment.

Original: The details about the training setup are described in Appendix 6.1.2.
Revised: The details about the training setup are described in Appendix 7.1.2.

Original: In the baseline agent, the trajectories are used to train the policy π φ using imitation learning and the dynamics model by maximum likelihood.
Revised: In the Learning to Query agent, the trajectories are used to train the policy π φ using imitation learning and the dynamics model by maximum likelihood.

Original: We consider 3 continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016): HalfCheetah, Fetch-Push (Plappert et al., 2018) and Reacher.
Revised: We consider 3 continuous control tasks from the OpenAI Gym suite (Brockman et al., 2016): Half-Cheetah, Fetch-Push (Plappert et al., 2018) and Reacher.

Original: The dynamics model is unrolled for k =steps for Half-Cheetah and k = 5 for Fetch-Pushand Reacher.
Revised: During open loop, the dynamics model is unrolled for k = 10 steps for Half-Cheetah and k = 5 for Fetch-Push Reacher.

Original: 4.2. E VALUATING D YNAMICS M ODELS
Revised: 5.2. E VALUATING D YNAMICS M ODELS

Original: We want to show that the consistency constraint helps to learn a better dynamics model of the environment.
Revised: We want to show that the consistency constraint helps to learn a better dynamics model of the environment.

Original: Since we learn a dynamics model over the states, we also need to jointly learn an observation model (decoder, see Appendix 6.1.2) conditioned on the states.
Revised: Since we learn a dynamics model over the states, we also need to jointly learn an observation model (decoder, see appendix 7.1.2) conditioned on the states.

Original: We can then compute the log-likelihood of trajectories in the real environment (sampled with the expert policy) under this observation model.
Revised: We can then compute the log-likelihood of trajectories in the real environment (sampled with the expert policy) under this observation model.

Original: We compare the log-likelihoods corresponding to these observations for the agents trained with and without the consistency loss.
Revised: We compare the log-likelihoods corresponding to these observations for the Learning to Query agent (trained without the consistency loss) and Consistent Dynamics agent (trained with the consistency loss).

Original: We expect that the consistency constrained agent would achieve a higher log likelihood.
Revised: We expect that the Consistent Dynamics agent would achieve a higher log likelihood.

Original: Figure 4 shows that for the consistency constrained agent, the dynamics model out-perform the  baseline in all the 3 cases, which indicates that the agent learns a more powerful dynamics model of the environment.
Revised: Figure 4 shows that in terms of imagination log likelihood, the Consistent Dynamics agent outperforms the Learning to Query baseline agent for all the 3 environments indicating that the agent learns a more powerful dynamics model of the environment.

Original: Note that in case of Fetch-Push and Reacher, we see improvements in the log-likelihood, even though the dynamics model is unrolled for just 5 steps.[Figure]
Revised: Note that in case of Fetch-Push and Reacher, we see improvements in the log-likelihood, even though the dynamics model is unrolled for just 5 steps.

Original: 4.2. L EARNING THE POLICY BY I MITATION L EARNING
Revised: 5.2. L EARNING THE POLICY BY I MITATION L EARNING

Original: For the state-space models [Figure] we used the expert trajectories to train our policy π φ using imitation learning.
Revised: For the state-space models, we used the expert trajectories to train our policy π φ using imitation learning.

Original: To show that the consistency constraint helps to learn a more powerful policy [Figure] we compare the imitation learning loss for the  agent trained with and without the consistency constraint. Figure 5 shows that for the consistency constrained agent, the imitation learning loss is smaller for 2 environments
Revised: To show that the consistency constraint helps to learn a more powerful policy, we compare the imitation learning loss for the Learning to Query agent (trained without the consistency loss) and Consistent Dynamics agent (trained with the consistency loss).

Original: [Figure]

Original: and on par with the loss of the agent trained without this constraint on the third environment.

Original: This indicates that using the consistency constraint helps the agent to learn a more powerful policy in some cases and equally-powerful policy in other cases

Original: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we formulate a way to ensure consistency between the predictions of a dynamics model and the real observations from the environment thus allowing the agent to learn powerful policies, as well as better dynamics models.
Revised: In this paper, we formulate a way to ensure consistency between the predictions of a dynamics model and the real observations from the environment thus allowing the agent to learn powerful policies, as well as better dynamics models.

Original: The learning agent, in parallel, (i) builds a model of the environment and (ii) engages in an interaction with the environment.
Revised: The learning agent, in parallel, (i) builds a model of the environment and (ii) engages in an interaction with the environment.

Original: This results in two sequences of state transitions: one in the real environment where the agent actually performs actions and other in the agent’s dynamics model (or the “world”) where it imagines taking actions, and hallucinates the state transitions.
Revised: This results in two sequences of state transitions: one in the real environment where the agent actually performs actions [Figure] [Figure]

Original: We apply an auxiliary loss which encourages the behaviour of state transitions across the two sequences to be indistinguishable from each other.
Revised: t [Figure] i [Figure]

Original:  We evaluate our proposed approach for both observation space models, and state space models and show that the agent learns more a powerful policy and a better generative model.
Revised: li [Figure] from each other. We evaluate our proposed approach for both observation space models, and state space models and show that the agent learns more a powerful policy and a better generative model.

Original: Future work would consider how these 2 interaction pathways could lead to more targeted exploration.
Revised: Future work would consider how these 2 interaction pathways could lead to more targeted exploration.

Original: Furthermore, having more ﬂexibility over the length over which we unroll the model could allow the agent to take these decisions over multiple timescales.
Revised: Furthermore, having more ﬂexibility over the length over which we unroll the model could allow the agent to take these decisions over multiple timescales.

Original:  A PPENDIX 6. E NVIRONMENT M ODEL 6.1. O BSERVATION S PACE M ODEL
Revised:  A PPENDIX 7. E NVIRONMENT M ODEL 7.1. O BSERVATION S PACE M ODEL

Original: We use the experimental setup and environments  as described in (Nagabandi et al., 2017) 1 .
Revised: We use the experimental setup, environments and the hybrid model-based and model-free (Mb-Mf) algorithm as described in (Nagabandi et al., 2017) 1 .

Original: We consider two training scenarios: training a model-based learning agent with and without the consistency constraint.
Revised: We consider two training scenarios: training a model-based learning agent with and without the consistency constraint.

Original: The consistency constraint is applied by unrolling the model for multiple steps using the observations predicted by the learner’s dynamics model (closed-loop setup).
Revised: The consistency constraint is applied by unrolling the model for multiple steps using the observations predicted by the learner’s dynamics model (closed-loop setup).

Original: We train an on-policy RL algorithm for Cheetah, Humanoid, Ant and Swimmer tasks from RLLab (Duan et al., 2016) control suite.
Revised: We train an on-policy RL algorithm for Cheetah, Humanoid, Ant and Swimmer tasks from RLLab (Duan et al., 2016) control suite.

Original: We report  the average  reward obtained by the learner in the two cases: with and without  consistency constraint.
Revised: We report both the average discounted and average un-discounted reward obtained by the learner in the two cases: with and without the use of consistency constraint.

Original: The model and policy architectures for the observation space models are as follows:
Revised: The model and policy architectures for the observation space models are as follows:

Original: Transition Model :
Revised: Transition Model :

Original: The transition model ˆ f θ ( s t , a t ) has a Gaussian distribution with diagonal covariance, where the mean and covariance are parametrized by MLPs (Schulman et al., 2015a), which maps an observation vector s t and an action vector a t to a vector µ which speciﬁes a distribution over observation space.
Revised: The transition model ˆ f θ ( s t , a t ) has a Gaussian distribution with diagonal covariance, where the mean and covariance are parametrized by MLPs (Schulman et al., 2015a), which maps an observation vector s t and an action vector a t to a vector µ which speciﬁes a distribution over observation space.

Original: During training, the log likelihood p ( s | µ ) is maximized and state-representations can be sampled from p ( s | µ ) .
Revised: During training, the log likelihood p ( s | µ ) is maximized and state-representations can be sampled from p ( s | µ ) .

Original: Policy :
Revised: Policy :

Original: The learner’s policy ˆ π φ ( s t ) is also a Gaussian MLP which maps an observation vector s to a vector µ policy which speciﬁes a distribution over action space.
Revised: The learner’s policy ˆ π φ ( s t ) is also a Gaussian MLP which maps an observation vector s to a vector µ policy which speciﬁes a distribution over action space.

Original: Like before, the log-likelihood p ( a | µ ) is maximized and actions can be sampled from p ( a | µ ) .
Revised: Like before, the log-likelihood p ( a | µ ) is maximized and actions can be sampled from p ( a | µ ) .

Original: Learner’s policy and the dynamics model are implemented as Gaussian policies with MLPs as func tion approximations, and are trained using TRPO (Schulman et al., 2015a).
Revised: Learner’s policy and the dynamics model are implemented as Gaussian policies with MLPs as function approximations, and are trained using TRPO (Schulman et al., 2015a).

Revised: Following the hybrid

Original: Following (Nagabandi et al., 2017), we normalize the states and actions.
Revised: Mb-Mf approach(Nagabandi et al., 2017), we normalize the states and actions.

Original: The dynamics model is trained to predict the change in state ∆ S t as it can be difﬁcult to learn the state transition function when the states s t and s t
Revised: The dynamics model is trained to predict the change in state ∆ s t as it can be difﬁcult to learn the state transition function when the states s t and s t

Original: +1 are very similar and the action a t has a small effect on the output.
Revised: +1 are very similar and the action a t has a small effect on the output.

Original: 6.1. S TATE S PACE M ODEL
Revised: 7.1. S TATE S PACE M ODEL

Revised: We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.

Original: The model and policy architecture for the state space models is as follows:
Revised: The model and policy architecture for the state space models is as follows:

Original: Encoder :
Revised: Encoder :

Original: The learner encodes the pixel-space observations ( 64 × 64 × 3 ) from the environment into state-space observations ( 256 dimensional vectors) with a convolutional encoder (4 convolutional layers with 4 × 4 kernels, stride 2 and 64 channels).
Revised: The learner encodes the pixel-space observations ( 64 × 64 × 3 ) from the environment into state-space observations ( 256 dimensional vectors) with a convolutional encoder (4 convolutional layers with 4 × 4 kernels, stride 2 and 64 channels).

Original: To model the velocity information, a stack of the latest 4 frames is used as the observation.
Revised: To model the velocity information, a stack of the latest 4 frames is used as the observation.

Original: The pixel-space observation at time t − 1 is denoted as o t − 1 , and is encoded into state-space observation s t − 1 . 2.
Revised: The pixel-space observation at time t − 1 is denoted as o t − 1 , and is encoded into state-space observation s t − 1 . 2.

Original: Transition Model :
Revised: Transition Model :

Original: The transition model is a Long Short-Term Memory model (LSTM, Hochreiter & Schmidhuber, 1997), that predicts the transitions in the state space.
Revised: The transition model is a Long Short-Term Memory model (LSTM, Hochreiter & Schmidhuber, 1997), that predicts the transitions in the state space.

Original: For every time-step t , latent variables z t are introduced, whose distribution is a function of previous state-space observation s t − 1 and previous action a t − 1 .
Revised: For every time-step t , latent variables z t are introduced, whose distribution is a function of previous state-space observation s t − 1 and previous action a t − 1 .

Original: ie z t ∼ p ( z t | s t − 1 , a t − 1 ) .
Revised: ie z t ∼ p ( z t | s t − 1 , a t − 1 ) .

Original: The output of the transition model is then a deterministic function of z t , s t − 1 , and a t − 1 .
Revised: The output of the transition model is then a deterministic function of z t , s t − 1 , and a t − 1 .

Original: ie s t = f ( z t , s t − 1 , a t − 1 ) .
Revised: ie s t = f ( z t , s t − 1 , a t − 1 ) .

Original: Stochastic Decoder : The learner can decode the state-space observations back into the pixel-space observations by use of stochastic convolutional decoder.
Revised: Stochastic Decoder : The learner can decode the state-space observations back into the pixel-space observations by use of stochastic convolutional decoder.

Original: The decoder takes as input the current state-space observation s t and the current latent variable z t and generates the current observation-space distribution from which the learner could sample an observation.
Revised: The decoder takes as input the current state-space observation s t and the current latent variable z t and generates the current observation-space distribution from which the learner could sample an observation.

Original: ie o t +1 ∼ p ( o t +1 | s t , z t ) .
Revised: ie o t +1 ∼ p ( o t +1 | s t , z t ) .

Original: This observation model is Gaussian, with a diagonal covariance.
Revised: This observation model is Gaussian, with a diagonal covariance.

Original: In the closed-loop trajectory, when the learner cannot interact with the environment the latent variables are sampled from the prior distribution p ( z t | s t − 1 , a t − 1 ) .
Revised: In the closed-loop trajectory, when the learner cannot interact with the environment, the latent variables are sampled from the prior distribution p ( z t | s t − 1 , a t − 1 ) .

Original: The latent variables are sampled from Normal distributions with diagonal covariance matrices.
Revised: The latent variables are sampled from Normal distributions with diagonal covariance matrices.

Original: Since we cannot compute the loglikelihood L ( θ ) in a close form for the latent variable models, we minimize the evidence lower bound ELBO ( p posterior ) ≤ L ( θ ) .
Revised: Since we cannot compute the loglikelihood L ( θ ) in a close form for the latent variable models, we minimize the evidence lower bound ELBO ( p posterior ) ≤ L ( θ ) .

Original: As discussed previously, the consistency constraint is applied between the open-loop and closed-loop predictions with the aim of making their behaviour as similar as possible.
Revised: As discussed previously, the consistency constraint is applied between the open-loop and closed-loop predictions with the aim of making their behaviour as similar as possible.

Original: Figure 7 shows a graphical representation of the open-loop and close-loop pathways in the state-space model.
Revised: Figure 7 shows a graphical representation of the open-loop and close-loop pathways in the state-space model.

Original: Expert policy Having access to some policy trained on a large number of experience is required to sample high-quality trajectories with pixel-observations.
Revised: Expert policy Having access to some policy trained on a large number of experience is required to sample high-quality trajectories with pixel-observations.

Original: To train these expert policies, we used policy-based methods such as Proximal Policy Optimization (PPO, Schulman et al., 2017) for the half-cheetah and reacher environments, or Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG with HER, Andrychowicz et al., 2017) for the pushing task.
Revised: To train these expert policies, we used policy-based methods such as Proximal Policy Optimization (PPO, Schulman et al., 2017) for the half-cheetah and reacher environments, or Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG with HER, Andrychowicz et al., 2017) for the pushing task.

Original: The architectures and hyper-parameters used are similar to the ones given by the Baselines library (Dhariwal et al., 2017).
Revised: The architectures and hyper-parameters used are similar to the ones given by the Baselines library (Dhariwal et al., 2017).

Original: Note that these expert policies were trained on the state representation of the agents (ie.
Revised: Note that these expert policies were trained on the state representation of the agents (ie.

Original: the positions and velocities of their joints), while the trajectories were generated with pixel-observations captured from a view external to the agent.
Revised: the positions and velocities of their joints), while the trajectories were generated with pixel-observations captured from a view external to the agent.

Original: 6. R ESULTS 6.2. O BSERVATION S PACE M ODELS
Revised: 7. R ESULTS 7.2. O BSERVATION S PACE M ODELS

Original: [Figure]
Revised: [Figure]

Original: v e r a g e
Revised: e

Original: D i s c o un t e d E p i s o d
Revised: [Figure]

Original:  i c R e t
Revised: p i s

Original: [Figure]
Revised: [Figure]

Original:  r n
Revised: d i c R e t u r n

Original: [Figure]
Revised: [Figure]