Original: M ODEL - BASED IMITATION LEARNING FROM STATE TRAJECTORIES
Revised: M ODEL - BASED IMITATION LEARNING FROM STATE TRAJECTORIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Imitation learning from demonstrations usually relies on learning a policy from trajectories of optimal states and actions.
Revised: Imitation learning from demonstrations usually relies on learning a policy from trajectories of optimal states and actions.

Original: However, in real life expert demonstrations, often the action information is missing and only state trajectories are available.
Revised: However, in real life expert demonstrations, often the action information is missing and only state trajectories are available.

Original: We present a model-based imitation learning method that can learn environment-speciﬁc optimal actions only from expert state trajectories.
Revised: We present a model-based imitation learning method that can learn environment-speciﬁc optimal actions only from expert state trajectories.

Original: Our proposed method starts with a model-free reinforcement learning algorithm with a heuristic reward signal to sample environment dynamics, which is then used to train the state-transition probability.
Revised: Our proposed method starts with a model-free reinforcement learning algorithm with a heuristic reward signal to sample environment dynamics, which is then used to train the state-transition probability.

Original: Subsequently, we learn the optimal actions from expert state trajectories by supervised learning, while back-propagating the error gradients through the modeled environment dynamics.
Revised: Subsequently, we learn the optimal actions from expert state trajectories by supervised learning, while back-propagating the error gradients through the modeled environment dynamics.

Original: Experimental evaluations show that our proposed method successfully achieves performance similar to (state, action) trajectory-based traditional imitation learning methods even in the absence of action information, with much fewer iterations compared to conventional model-free reinforcement learning methods.
Revised: Experimental evaluations show that our proposed method successfully achieves performance similar to (state, action) trajectory-based traditional imitation learning methods even in the absence of action information, with much fewer iterations compared to conventional model-free reinforcement learning methods.

Original: We also demonstrate that our method can learn to act from only video demonstrations of expert agent for simple games and can learn to achieve desired performance in less number of iterations.
Revised: We also demonstrate that our method can learn to act from only video demonstrations of expert agent for simple games and can learn to achieve desired performance in less number of iterations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning(RL) involves training an agent to learn a policy that accomplishes a certain task in an environment.
Revised: Reinforcement learning(RL) involves training an agent to learn a policy that accomplishes a certain task in an environment.

Original: The objective of reinforcement learning is to maximize the expected future reward Sutton & Barto (1998) from a guiding signal.
Revised: The objective of reinforcement learning is to maximize the expected future reward Sutton & Barto (1998) from a guiding signal.

Original: Mnih et al.
Revised: Mnih et al.

Original: (2015) showed that neural networks can be used to approximate state-action value functions used by an agent to perform discrete control based on a guiding reward.
Revised: (2015) showed that neural networks can be used to approximate state-action value functions used by an agent to perform discrete control based on a guiding reward.

Original: This was demonstrated in Atari games where the score was used as the reward signal.
Revised: This was demonstrated in Atari games where the score was used as the reward signal.

Original: Similarly, continuous control of robotics arm was achieved by Lillicrap et al.
Revised: Similarly, continuous control of robotics arm was achieved by Lillicrap et al.

Original: minimizing the distance between end-effector and target location.
Revised: minimizing the distance between end-effector and target location.

Original: Following these, other methods such as Schulman et al.
Revised: Following these, other methods such as Schulman et al.

Original: (2017; 2015) were proposed to improve the sample efﬁciency of modelfree algorithms with theoretical guarantees of policy improvement in each step.
Revised: (2017; 2015) were proposed to improve the sample efﬁciency of modelfree algorithms with theoretical guarantees of policy improvement in each step.

Original: These algorithms assume that a guiding reward signal is available for the agent to learn optimal behavior for a certain task.
Revised: These algorithms assume that a guiding reward signal is available for the agent to learn optimal behavior for a certain task.

Original: However, in most cases of natural learning, such guiding signal is not present and learning is performed by imitating an expert behavior.
Revised: However, in most cases of natural learning, such guiding signal is not present and learning is performed by imitating an expert behavior.

Original: Imitation learning involves copying the behavior of an expert agent to accomplish the desired task.
Revised: Imitation learning involves copying the behavior of an expert agent to accomplish the desired task.

Original: In the conventional imitation learning setting, a set of expert trajectories providing states and optimal actions τ = { s 0 , a 0 , s 1 , a 1 , ..., s n , a n ) performed by an expert agent π E are available but the reward (or cost function), r E ( s, a ) used to achieve the expert behavior is not available.
Revised: In the conventional imitation learning setting, a set of expert trajectories providing states and optimal actions τ = { s 0 , a 0 , s 1 , a 1 , ..., s n , a n ) performed by an expert agent π E are available but the reward (or cost function), r E ( s, a ) used to achieve the expert behavior is not available.

Original: The goal is to learn a new policy π , which imitates the expert behavior by maximizing the likelihood of given demonstration trajectories.
Revised: The goal is to learn a new policy π , which imitates the expert behavior by maximizing the likelihood of given demonstration trajectories.

Original: A straightforward way for imitation learning is to direct learn the optimal action to perform given the current state as proposed by Pomerleau (1991);
Revised: A straightforward way for imitation learning is to direct learn the optimal action to perform given the current state as proposed by Pomerleau (1991);

Original: Duan et al.
Revised: Duan et al.

Original: The policy π can learn to imitate the expert behavior by maximizing likelihood of the condition distribution of action given states p ( a | s ) .
Revised: The policy π can learn to imitate the expert behavior by maximizing likelihood of the condition distribution of action given states p ( a | s ) .

Original: This can be achieved by simply training a parameterized function (neural networks for instance) with state and action pairs from the expert trajectories.
Revised: This can be achieved by simply training a parameterized function (neural networks for instance) with state and action pairs from the expert trajectories.

Original: Since this involves end-to-end supervised learning, training is much more sample-efﬁcient compared to reinforcement learning and overcomes inherent problems in model-free methods such as credit assignment(Sutton (1984)).
Revised: Since this involves end-to-end supervised learning, training is much more sample-efﬁcient compared to reinforcement learning and overcomes inherent problems in model-free methods such as credit assignment(Sutton (1984)).

Original: However, since behavior cloning learns optimal action from a single state value only, it is unaware of the future state distribution the current action will produce.
Revised: However, since behavior cloning learns optimal action from a single state value only, it is unaware of the future state distribution the current action will produce.

Original: Thus, errors are compounded in the future states leading to undesired agent behavior as shown by Ross et al.
Revised: Thus, errors are compounded in the future states leading to undesired agent behavior as shown by Ross et al.

Original: ; Ross & Bagnell (2010).
Revised: ; Ross & Bagnell (2010).

Original: Therefore, numerous training samples are required for behavior cloning to reduce errors in action prediction required for satisfactory imitation learning.
Revised: Therefore, numerous training samples are required for behavior cloning to reduce errors in action prediction required for satisfactory imitation learning.

Original: The second approach to imitation learning involves setting up exploration in a Markov Decision
Revised: The second approach to imitation learning involves setting up exploration in a Markov Decision

Original: Process(MDP) setting.
Revised: Process(MDP) setting.

Original: The goal then is to recover a reward signal that best explains the expert trajectories.
Revised: The goal then is to recover a reward signal that best explains the expert trajectories.

Original: Ng & Russell (2000) ﬁrst introduced Inverse Reinforcement Learning(IRL), where the goal is to ﬁnd a reward signal ˆ r from the trajectories such that the expert is uniquely optimal.
Revised: Ng & Russell (2000) ﬁrst introduced Inverse Reinforcement Learning(IRL), where the goal is to ﬁnd a reward signal ˆ r from the trajectories such that the expert is uniquely optimal.

Original: After computing this estimated reward signal, usually, a model-free reinforcement learning performed to obtain the desired policy imitating the expert behavior by maximizing the expected discounted reward
Revised: After computing this estimated reward signal, usually, a model-free reinforcement learning performed to obtain the desired policy imitating the expert behavior by maximizing the expected discounted reward

Original: E π
Revised: E π

Original: ( (cid:80) t γ t ˆ r ( s t , a t )) .
Revised: ( (cid:80) t γ t ˆ r ( s t , a t )) .

Original: While this alleviates the problem of compounding errors as in behavior cloning, Ziebart et al.
Revised: While this alleviates the problem of compounding errors as in behavior cloning, Ziebart et al.

Original: (2008) showed that estimating a unique reward function from state and action trajectories is an ill-posed problem.
Revised: (2008) showed that estimating a unique reward function from state and action trajectories is an ill-posed problem.

Original: Following the success of Generative Adversarial Networks(GANs) (Goodfellow et al.
Revised: Following the success of Generative Adversarial Networks(GANs) (Goodfellow et al.

Original: in various ﬁelds of machine learning, adversarial learning has also been shown incorporated in the imitation learning framework.
Revised: in various ﬁelds of machine learning, adversarial learning has also been shown incorporated in the imitation learning framework.

Original: The recent work on Generative Adversarial Imitation Leaning or GAIL by Ho & Ermon (2016) showed that model-free reinforcement learning using the discriminator as a cost function can learn to imitate the expert agent with much less number of demonstrated trajectories compared to behavior cloning.
Revised: The recent work on Generative Adversarial Imitation Leaning or GAIL by Ho & Ermon (2016) showed that model-free reinforcement learning using the discriminator as a cost function can learn to imitate the expert agent with much less number of demonstrated trajectories compared to behavior cloning.

Original: Following the success of GAIL, there have extensions by Baram et al.
Revised: Following the success of GAIL, there have extensions by Baram et al.

Original: (2017) to model-based generative imitation learning using a differentiable dynamics model of the environment.
Revised: (2017) to model-based generative imitation learning using a differentiable dynamics model of the environment.

Original: Robust imitation policy strategies using a combination of variational autoencoders (Kingma & Ba (2015); Rezende et al.
Revised: Robust imitation policy strategies using a combination of variational autoencoders (Kingma & Ba (2015); Rezende et al.

Original: and GAIL has also been proposed by Wang et al.
Revised: and GAIL has also been proposed by Wang et al.

Original: The previous works assume that the expert trajectories consist of both action and state values from the optimal agent.
Revised: The previous works assume that the expert trajectories consist of both action and state values from the optimal agent.

Original: However, optimal actions are usually not available in real-world imitation learning.
Revised: However, optimal actions are usually not available in real-world imitation learning.

Original: For example, we often learn tasks like skipping, jump rope, gymnastics, etc. just by watching other expert humans perform the task.
Revised: For example, we often learn tasks like skipping, jump rope, gymnastics, etc. just by watching other expert humans perform the task.

Original: In this case, the optimal expert trajectories only consist of visual input, in other words, the consecutive states of the expert human with no action information.
Revised: In this case, the optimal expert trajectories only consist of visual input, in other words, the consecutive states of the expert human with no action information.

Original: We learn to jump rope by trying to reproduce actions that result in state trajectories similar to the state trajectories observed from the expert.
Revised: We learn to jump rope by trying to reproduce actions that result in state trajectories similar to the state trajectories observed from the expert.

Original: This requires exploring the environment in a structured fashion to learn the dynamics of the rope (for jump rope) which then enables executing optimal actions to imitate the expert behavior.
Revised: This requires exploring the environment in a structured fashion to learn the dynamics of the rope (for jump rope) which then enables executing optimal actions to imitate the expert behavior.

Original: The recent work of Liu et al.
Revised: The recent work of ?

Original: (2017) presents learning from observations only with focus to transferring skills learned from source domain to an unseen target domain, using rewards obtained by feature tracking for model-free reinforcement learning.
Revised: presents learning from observations only with focus to transferring skills learned from source domain to an unseen target domain, using rewards obtained by feature tracking for model-free reinforcement learning.

Original: Inspired by the above method of learning in humans, we present a principled way of learning to imitate an expert from state information only, with no action information available.
Revised: Inspired by the above method of learning in humans, we present a principled way of learning to imitate an expert from state information only, with no action information available.

Original: We ﬁrst learn a distribution of the next state from the current state trajectory, used to estimate a heuristic reward signal enabling model-free exploration.
Revised: We ﬁrst learn a distribution of the next state from the current state trajectory, used to estimate a heuristic reward signal enabling model-free exploration.

Original: The state, action and next states information from modelfree exploration is used to learn a dynamics model of the environment.
Revised: The state, action and next states information from modelfree exploration is used to learn a dynamics model of the environment.

Original: For the case of learning in humans, this is similar to performing actions for replicating the witnessed expert state trajectories, which in turn gives information about the dynamics of the environment.
Revised: For the case of learning in humans, this is similar to performing actions for replicating the witnessed expert state trajectories, which in turn gives information about the dynamics of the environment.

Original: Once this forward model is learned, we try to ﬁnd the action that maximizes the likelihood of next state.
Revised: Once this forward model is learned, we try to ﬁnd the action that maximizes the likelihood of next state.

Original: Since the forward model gives a function approximation for the environment dynamics, we can back propagate errors through it to perform model-based policy update by end to end supervised learning.
Revised: Since the forward model gives a function approximation for the environment dynamics, we can back propagate errors through it to perform model-based policy update by end to end supervised learning.

Original: We demonstrate that our proposed network can reach, with fewer iterations, the level close to an expert agent behavior (which is a pre-trained actor network or manually provided by humans), and compare it with reinforcement learning using a hand-crafted reward or a heuristics reward that is based on prediction error of next state learned from the optimal state trajectories of the expert.
Revised: We demonstrate that our proposed network can reach, with fewer iterations, the level close to an expert agent behavior (which is a pre-trained actor network or manually provided by humans), and compare it with reinforcement learning using a hand-crafted reward or a heuristics reward that is based on prediction error of next state learned from the optimal state trajectories of the expert.

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: We summarize the notations used in the paper in this section.
Revised: We summarize the notations used in the paper in this section.

Original: Consider a Markov Decision Process (MDP) denoted as ( S, A, P, r, ρ 0 , γ ) , where S is the ﬁnite set of states, A is the set of possible actions, P : S ×
Revised: Consider a Markov Decision Process (MDP) denoted as ( S, A, P, r, ρ 0 , γ ) , where S is the ﬁnite set of states, A is the set of possible actions, P : S ×

Original: A → S is the transition probability distribution and r : S ×
Revised: A → S is the transition probability distribution and r : S ×

Original: A → R be the reward signal from state and actions, ρ 

Revised: A → R be the reward signal from state and actions, ρ

Original: → R is the initial state distribution and γ ∈ (0 , 1) is the discount factor.
Revised: → R is the initial state distribution and γ ∈ (0 , 1) is the discount factor.

Original: Let π : S ×
Revised: Let π : S ×

Original: A → (0 , 1) be the policy that gives the conditional distribution of actions given current state, p ( a | s ) and R ( π ) =
Revised: A → (0 , 1) be the policy that gives the conditional distribution of actions given current state, p ( a | s ) and R ( π ) =

Original: E π [ (cid:80) t γ t r t ( s t , a t )] is the discounted reward associated with the policy.
Revised: E π [ (cid:80) t γ t r t ( s t , a t )] is the discounted reward associated with the policy.

Original: We consider expert trajectories consisting of only optimal state distribution without any action information, τ E = { s 0 , s 1 , ..., s n } .
Revised: We consider expert trajectories consisting of only optimal state distribution without any action information, τ E = { s 0 , s 1 , ..., s n } .

Original: The trajectories sampled from model-free exploration is denoted as, τ RL = { s 0 , a 0 , s 1 , a 1 , ..., s n , a n } .
Revised: The trajectories sampled from model-free exploration is denoted as, τ RL = { s 0 , a 0 , s 1 , a 1 , ..., s n , a n } .

Original: We use the terms dynamic model, state-transition probability and forward model interchangeably in the paper.
Revised: We use the terms dynamic model, state-transition probability and forward model interchangeably in the paper.

Original: f E ( s t , a t ) denotes the non-differentiable forward model of the environment and f ( s t , a t ) denotes the differentiable version which is learned during the proposed training procedure.
Revised: f E ( s t , a t ) denotes the non-differentiable forward model of the environment and f ( s t , a t ) denotes the differentiable version which is learned during the proposed training procedure.

Original: π mf denotes the model-free policy and π mb denotes the model-based policy network.
Revised: π mf denotes the model-free policy and π mb denotes the model-based policy network.

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: While most previous works use trajectories containing both state and action information to infer a policy that imitates the expert behavior, our problem statement is to imitate an expert from optimal state trajectories only.
Revised: While most previous works use trajectories containing both state and action information to infer a policy that imitates the expert behavior, our problem statement is to imitate an expert from optimal state trajectories only.

Original: This setting is common in many natural scenarios where only state information is available.
Revised: This setting is common in many natural scenarios where only state information is available.

Original: For example, humans learning to swim by seeing videos or observing other experts swimmers, have access to the sensory stream of information containing states only.
Revised: For example, humans learning to swim by seeing videos or observing other experts swimmers, have access to the sensory stream of information containing states only.

Original: The optimal actions are obtained by trying to replicate the state trajectories in the real environment via exploration.
Revised: The optimal actions are obtained by trying to replicate the state trajectories in the real environment via exploration.

Original: In this work, we learn a time series predictor of next state given the current state.
Revised: In this work, we learn a time series predictor of next state given the current state.

Original: Subsequently, we estimate a heuristic reward signal at each step, based on the difference of predicted next state by the time series model and the actual next state taken by the policy network to learn a model-free policy with exploration.
Revised: Subsequently, we estimate a heuristic reward signal at each step, based on the difference of predicted next state by the time series model and the actual next state taken by the policy network to learn a model-free policy with exploration.

Original: However, such heuristic methods suffer from the disadvantages of slow modelfree training to reach satisfactory policy and provide no guarantees on convergence.
Revised: However, such heuristic methods suffer from the disadvantages of slow modelfree training to reach satisfactory policy and provide no guarantees on convergence.

Original: Therefore we resort to model-based policy learning that learns a differentiable model of the environment dynamics used to learn policy by directly supervised learning.
Revised: Therefore we resort to model-based policy learning that learns a differentiable model of the environment dynamics used to learn policy by directly supervised learning.

Original: The proposed algorithm alternates between dynamic model parameters and policy parameters update using gradient propagation through the differentiable dynamics model.
Revised: The proposed algorithm alternates between dynamic model parameters and policy parameters update using gradient propagation through the differentiable dynamics model.

Original: We formalize this setup in the following sections.
Revised: We formalize this setup in the following sections.

Original: 2. M ODEL - FREE POLICY LEARNING VIA REWARD ESTIMATION
Revised: 2. M ODEL - FREE POLICY LEARNING VIA REWARD ESTIMATION

Original: Following the work in Inverse Reinforcement Learning(IRL), it is possible to frame the imitation learning problem in the MDP framework.
Revised: Following the work in Inverse Reinforcement Learning(IRL), it is possible to frame the imitation learning problem in the MDP framework.

Original: The imitation learning problem then reduces to ﬁnding a uniquely optimal reward signal that maximizes the likelihood of the observed demonstration trajectories.
Revised: The imitation learning problem then reduces to ﬁnding a uniquely optimal reward signal that maximizes the likelihood of the observed demonstration trajectories.

Original: However, ﬁnding a uniquely optimal reward function, even with optimal trajectories containing both state and action information, is an ill-posed problem.
Revised: However, ﬁnding a uniquely optimal reward function, even with optimal trajectories containing both state and action information, is an ill-posed problem.

Original: The ill-posed nature of optimal reward estimation is further accentuated in the absence of action information in expert trajectories.
Revised: The ill-posed nature of optimal reward estimation is further accentuated in the absence of action information in expert trajectories.

Original: As such, many solutions for parameterized families of reward estimation models maximizing the likelihood of only expert state trajectories might be sub-optimal to learn the desired task.
Revised: As such, many solutions for parameterized families of reward estimation models maximizing the likelihood of only expert state trajectories might be sub-optimal to learn the desired task.

Original: For model-free policy learning, we use a heuristic reward estimated by the error in next state prediction at each step.
Revised: For model-free policy learning, we use a heuristic reward estimated by the error in next state prediction at each step.

Original: We make an assumption that estimating a locally optimal reward maximizing the likelihood of next step predicting, is intuitively globally optimal for learning the desired task.
Revised: We make an assumption that estimating a locally optimal reward maximizing the likelihood of next step predicting, is intuitively globally optimal for learning the desired task.

Original: A straight-forward method to obtain such heuristic reward signal from the trajectories, τ s , is to learn a time series predictive model of the next state given the current state, p ( s t +

Revised: A straight-forward method to obtain such heuristic reward signal from the trajectories, τ s , is to learn a time series predictive model of the next state given the current state, p ( s t +

Original: | s 1: t ) from the expert state trajectories using time series modeling.
Revised: | s 1: t ) from the expert state trajectories using time series modeling.

Original: In our case, we use an exponential of the difference between predicted states and actual next state associated with the action predicted by the policy network.
Revised: In our case, we use an exponential of the difference between predicted states and actual next state associated with the action predicted by the policy network.

Original: [Equation]
Revised: [Equation]

Original: where k is a constant controlling gain of the reward signal and σ controls the sensitivity of reward to divergence from the predicted state.
Revised: where k is a constant controlling gain of the reward signal and σ controls the sensitivity of reward to divergence from the predicted state.

Original: This reward can be used for guiding any standard model-free reinforcement learning techniques to maximize the expected reward, R ( π mf ) =
Revised: This reward can be used for guiding any standard model-free reinforcement learning techniques to maximize the expected reward, R ( π mf ) =

Original: E π mf [ (cid:80) t γ t r t ( s t , a t )] .
Revised: E π mf [ (cid:80) t γ t r t ( s t , a t )] .

Original: We assume on the intuition that locally optimal heuristic reward estimation will be sufﬁcient to ensure global optimality for learning the desired task.
Revised: We assume on the intuition that locally optimal heuristic reward estimation will be sufﬁcient to ensure global optimality for learning the desired task.

Original: M ODEL - BASED IMITATION LEARNING FROM STATE TRAJECTORIES
Revised: M ODEL - BASED IMITATION LEARNING FROM STATE TRAJECTORIES

Original: Consider there are m sets of expert trajectory episode each consisting of T states, given as τ E = { s 0 , s 1 , ..., s n } , where n = mT .
Revised: Consider there are m sets of expert trajectory episode each consisting of T states, given as τ E = { s 0 , s 1 , ..., s n } , where n = mT .

Original: We assume that the trajectories in each episode are independent of each other.
Revised: We assume that the trajectories in each episode are independent of each other.

Original: For the imitation learning problem, we wish to imitate an expert agent π E from state trajectories τ E .
Revised: For the imitation learning problem, we wish to imitate an expert agent π E from state trajectories τ E .

Original: we formulate a maximum likelihood problem for state trajectories given a parameterized model of the agent policy, given as
Revised: we formulate a maximum likelihood problem for state trajectories given a parameterized model of the agent policy, given as

Original: [Equation]
Revised: [Equation]

Original: where θ represents the parameter of the model.
Revised: where θ represents the parameter of the model.

Original: We assume that the random variables state( s t ), action( a t ) and next state s t +1 form a directed graphical model as shown in ﬁgure 2 (b).
Revised: We assume that the random variables state( s t ), action( a t ) and next state s t +1 form a directed graphical model as shown in ﬁgure 2 (b).

Original: Following the natural dynamics of environments in reinforcement learning setting, we assume that control action for the agent a t is conditionally independent of other state and actions in the past given the current state.
Revised: Following the natural dynamics of environments in reinforcement learning setting, we assume that control action for the agent a t is conditionally independent of other state and actions in the past given the current state.

Original: Distribution of the next state is conditionally independent of the other past states given current state and action following the MDP setting.
Revised: Distribution of the next state is conditionally independent of the other past states given current state and action following the MDP setting.

Original: In this framework, we can frame the modelbased policy as an encoder network with action as the latent variable and the dynamics policy as the decoder network predicting the next state.
Revised: In this framework, we can frame the modelbased policy as an encoder network with action as the latent variable and the dynamics policy as the decoder network predicting the next state.

Original: The log-likelihood estimation loss, in this case, can be written as,
Revised: The log-likelihood estimation loss, in this case, can be written as,

Original: [Equation]
Revised: [Equation]

Original: where θ e , theta d are the encoder and decoder parameters respectively.
Revised: where θ e , theta d are the encoder and decoder parameters respectively.

Original: Learning can be performed using ( s t , s t +1 ) pairs from the expert trajectories by minimizing the above loss, sampling action values from the posteriori distribution p ( a t | s t ) using standard Markov Chain Monte Carlo (MCMC) methods or variational inference methods Kingma & Ba (2015); Rezende et al.
Revised: Learning can be performed using ( s t , s t +1 ) pairs from the expert trajectories by minimizing the above loss, sampling action values from the posteriori distribution p ( a t | s t ) using standard Markov Chain Monte Carlo (MCMC) methods or variational inference methods Kingma & Ba (2015); Rezende et al.

Original: However, the learned action from the encoder in this case will not mimic the actual control commands used to perform the desired task in the control environment.
Revised: However, the learned action from the encoder in this case will not mimic the actual control commands used to perform the desired task in the control environment.

Original: We propose a constrained minimization cost function which enforces the decoder network to mimic the environment dynamics of the agent.
Revised: We propose a constrained minimization cost function which enforces the decoder network to mimic the environment dynamics of the agent.

Original: The proposed cost function enforces that the decoders model minimizes the loss for dynamics model prediction of next state given the current state and action, whereas the composition of encoder over the decoder minimizes prediction loss of the next state given the current state.
Revised: The proposed cost function enforces that the decoders model minimizes the loss for dynamics model prediction of next state given the current state and action, whereas the composition of encoder over the decoder minimizes prediction loss of the next state given the current state.

Original: This loss function is given as,
Revised: This loss function is given as,

Original: [Equation]
Revised: [Equation]

Original: where we θ dyn are the parameters of dynamics model and θ mb are model-based policy network’s parameters.
Revised: where we θ dyn are the parameters of dynamics model and θ mb are model-based policy network’s parameters.

Original: Let us denote the ﬁrst term of the loss term in equation 4 as model-based policy loss, L mb and the second term is referred as dynamics model loss, L dyn .
Revised: Let us denote the ﬁrst term of the loss term in equation 4 as model-based policy loss, L mb and the second term is referred as dynamics model loss, L dyn .

Original: As shown in ﬁgure 1, we perform alternate minimizations on the proposed cost function and training on the encoder and decoder are performed on two separate datasets.
Revised: As shown in ﬁgure 1, we perform alternate minimizations on the proposed cost function and training on the encoder and decoder are performed on two separate datasets.

Original: Firstly, the dynamics model parameters are updated from the experience replay samples gathered during model-free exploration.
Revised: Firstly, the dynamics model parameters are updated from the experience replay samples gathered during model-free exploration.

Original: Subsequently, the updated dynamics model is used as the decoder network in the above formulation with ﬁxed weights while the model-based policy parameters are updated by the gradient, ∇ θ mb L mb .
Revised: Subsequently, the updated dynamics model is used as the decoder network in the above formulation with ﬁxed weights while the model-based policy parameters are updated by the gradient, ∇ θ mb L mb .

Original: This enforces the encoder network to act as a model-based policy that learns to predict the optimal action given the current state.
Revised: This enforces the encoder network to act as a model-based policy that learns to predict the optimal action given the current state.

Original: During implementations, a deterministic encoder and decoder are used and a single action is sampled from the posterior distribution during training.
Revised: During implementations, a deterministic encoder and decoder are used and a single action is sampled from the posterior distribution during training.

Original: 2.2. L EARNING DYNAMICS MODEL FROM MODEL - FREE SAMPLES
Revised: 2.2. L EARNING DYNAMICS MODEL FROM MODEL - FREE SAMPLES

Original: Since our expert trajectories only consist of sensory streams containing state information only, learning from the heuristic reward in a MDP setting can be slow and does not guarantee that the reward is optimal for the desired task.
Revised: Since our expert trajectories only consist of sensory streams containing state information only, learning from the heuristic reward in a MDP setting can be slow and does not guarantee that the reward is optimal for the desired task.

Original: Thus we resort to dynamics model based learning which can propagate error gradients with respect to action information as discussed in the above section.
Revised: Thus we resort to dynamics model based learning which can propagate error gradients with respect to action information as discussed in the above section.

Original: Consider an analogy of a robot learning to navigate through a maze of expert state information only.
Revised: Consider an analogy of a robot learning to navigate through a maze of expert state information only.

Original: It must ﬁrst learn the state transition model p ( s t +1 | s t , a t ) to navigate through the environment.
Revised: It must ﬁrst learn the state transition model p ( s t +1 | s t , a t ) to navigate through the environment.

Original: Once dynamics model is learned, it can obtain the best action that takes the current state in optimal state trajectories to the next state.
Revised: Once dynamics model is learned, it can obtain the best action that takes the current state in optimal state trajectories to the next state.

Original: Solving for the desired action at each state ( p ( a t | s t ) ) is a maximum likelihood problem from the expert state trajectories which can be solved by end-to-end supervised learning.
Revised: Solving for the desired action at each state ( p ( a t | s t ) ) is a maximum likelihood problem from the expert state trajectories which can be solved by end-to-end supervised learning.

Original: Let us assume we have a parameterized model for agent dynamics, given as s t
Revised: Let us assume we have a parameterized model for agent dynamics, given as s t

Original: +1 =
Revised: +1 =

Original: f
Revised: f

Original: ( s t , a t ; θ dyn ) , where θ dyn denotes parameters of the model.
Revised: ( s t , a t ; θ dyn ) , where θ dyn denotes parameters of the model.

Original: During model-free learning, we store the trajectories of ( s t , a t , s t +1 ) that were encountered during exploration by the agent.
Revised: During model-free learning, we store the trajectories of ( s t , a t , s t +1 ) that were encountered during exploration by the agent.

Original: Let us denotes the trajectories of these triplets as τ RL .
Revised: Let us denotes the trajectories of these triplets as τ RL .

Original: For continuous state spaces, the gradient for dynamic model parameters are given as
Revised: For continuous state spaces, the gradient for dynamic model parameters are given as

Original: [Equation]
Revised: [Equation]

Original: which is gradient on the mean squared error loss between model predicted and true next state.
Revised: which is gradient on the mean squared error loss between model predicted and true next state.

Original: For the discrete state space case, we can ﬁrst maximize the probability of next state given the current state and action using a categorical loss.
Revised: For the discrete state space case, we can ﬁrst maximize the probability of next state given the current state and action using a categorical loss.

Original: Any standard stochastic gradient descent algorithms can be used for the above optimizations.
Revised: Any standard stochastic gradient descent algorithms can be used for the above optimizations.

Original: However, we use neural networks as function approximators which are shown to have approximation capacity for arbitrary non-linear function, although its nonconvex nature of optimization does always guarantee a solution that is globally optimal.
Revised: However, we use neural networks as function approximators which are shown to have approximation capacity for arbitrary non-linear function, although its nonconvex nature of optimization does always guarantee a solution that is globally optimal.

Original: Recent techniques in stochastic gradient descent (Zeiler (2012); Kingma & Ba (2014) ) have alleviated this problem to a large extent.
Revised: Recent techniques in stochastic gradient descent (Zeiler (2012); Kingma & Ba (2014) ) have alleviated this problem to a large extent.

Original: If we assume the dynamics model is ideal in predicting the next state and there exists a unique action to reach the next state from the current state, then the proposed method is identical to behavior cloning, although true action information is not provided by the expert.
Revised: If we assume the dynamics model is ideal in predicting the next state and there exists a unique action to reach the next state from the current state, then the proposed method is identical to behavior cloning, although true action information is not provided by the expert.

Original: Therefore, the performance of this method is upper bounded by the performance of behavior cloning model which learns from the true action information between states.
Revised: Therefore, the performance of this method is upper bounded by the performance of behavior cloning model which learns from the true action information between states.

Original: 2. T RANSFORMING STATES BASED ON ACTION
Revised: 2. T RANSFORMING STATES BASED ON ACTION

Original: In our formulation so far, the entire next state of the agent is predicted by the next state predictor.
Revised: In our formulation so far, the entire next state of the agent is predicted by the next state predictor.

Original: However, we found that predicting a part of the state which is dependent on action gives better reward structure.
Revised: However, we found that predicting a part of the state which is dependent on action gives better reward structure.

Original: This is in line with the work of Pathak et al.
Revised: This is in line with the work of Pathak et al.

Original: (2017), where the authors predict φ ( s t ) as the latent representation of the neural network predicting the action from consecutive states, a t =
Revised: (2017), where the authors predict φ ( s t ) as the latent representation of the neural network predicting the action from consecutive states, a t =

Original: g ( φ ( s t ) , φ ( s t +1 )) .
Revised: g ( φ ( s t ) , φ ( s t +1 )) .

Original: This transformed state value, φ ( s t ) is also used as the input and output for the dynamics model.
Revised: This transformed state value, φ ( s t ) is also used as the input and output for the dynamics model.

Original: This transformation is beneﬁcial for two reasons: (i) It is difﬁcult to learn the dynamics model for high dimensional state information and thus ﬁrst projecting onto a low dimensional manifold to learn the dynamics model gives a more feasible learning framework.
Revised: This transformation is beneﬁcial for two reasons: (i) It is difﬁcult to learn the dynamics model for high dimensional state information and thus ﬁrst projecting onto a low dimensional manifold to learn the dynamics model gives a more feasible learning framework.

Original: (ii)
Revised: (ii)

Original: In case of transferring the learned dynamics model between different tasks that use the same environment, a common state input is required for the dynamics model, which can be achieved such transformation.
Revised: In case of transferring the learned dynamics model between different tasks that use the same environment, a common state input is required for the dynamics model, which can be achieved such transformation.

Original: In case of learning from videos, we use the agent position in the image frames as φ ( s t ) .
Revised: In case of learning from videos, we use the agent position in the image frames as φ ( s t ) .

Original: For the case of linked arm reacher, we will use phi ( s t ) as the joint angles and joint velocities.
Revised: For the case of linked arm reacher, we will use phi ( s t ) as the joint angles and joint velocities.

Original: 2. A LGORITHM
Revised: 2. A LGORITHM

Original: We now outline the algorithm based on the above discussed model-based policy learning framework.[Equation] log [Equation]
Revised: We now outline the algorithm based on the above discussed model-based policy learning framework.

Original: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: p

Original: [Equation]
Revised: [Equation]

Original: ; θ p do Sample trajectory from free policy k π kmf replay Updated the dynamics model  parameter using trajectories buffer [Equation]
Revised: Update model based parameter from expert trajectories with ﬁxed dynamics model.

Original: [Table]
Revised: [Table]

Original: 7: end for
Revised: end The above algorithm shows an iterative process where in each iteration we ﬁrst train a model-free algorithm using the heuristic reward function. This step is necessary because we collect a certain amount of system dynamics data, ( s t , a t , s t +1 ) , while training the model-free policy. Then, we train a system dynamics model using the above collected data.

Original: The  frequency of switching between model-free replay buffer collection and model-based update can be varied depending on the complexity of dynamics model.
Revised: The action policy is then trained using the system dynamics model in the model-based part, which constitutes one cycle of the training. Subsequently, we repeat this cycle again starting from the model-free part. With each iteration, we collect additional system dynamics data, which results in a more precise dynamics model, leading to accurate action policy parameter gradients for updating the model-based policy. The frequency of switching between model-free replay buffer collection and model-based update can be varied depending on the complexity of dynamics model.

Original: For state predictions from previous states, we used Long Short-Term Memory Network, proposed by Hochreiter & Schmidhuber (1997).
Revised: For state predictions from previous states, we used Long Short-Term Memory Network, proposed by Hochreiter & Schmidhuber (1997).

Original: For the policy and dynamics model, we use neural networks as function approximators.
Revised: For the policy and dynamics model, we use neural networks as function approximators.

Original: In this work, we assume that the state transformation φ ( . ) is manually speciﬁed in each experiments, although it is possible to learn such representation by learning a common transformation between states that predicts the action, a t =
Revised: In this work, we assume that the state transformation φ ( . ) is manually speciﬁed in each experiments, although it is possible to learn such representation by learning a common transformation between states that predicts the action, a t =

Original: g ( φ ( s t ) , φ ( s t +1 )) .
Revised: g ( φ ( s t ) , φ ( s t +1 )) .

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We perform experimental evaluations on three kinds of environment, (i) Robotics arm reacher in 2d, (ii) Simple 2d obstacle avoidance based reacher, (iii) Learning to play simple games from raw video demonstrations.
Revised: We perform experimental evaluations on three kinds of environment, (i) Robotics arm reacher in 2d, (ii) Simple 2d obstacle avoidance based reacher, (iii) Learning to play simple games from raw video demonstrations.

Original: In each experiment, we show speciﬁc strengths of the proposed algorithm as follows.
Revised: In each experiment, we show speciﬁc strengths of the proposed algorithm as follows.

Original: 3. R OBOTICS ARM REACHER IN 2D
Revised: 3. R OBOTICS ARM REACHER IN 2D

Original: We use roboschool reacher(OpenAI (2017); Brockman et al.
Revised: We use roboschool reacher(OpenAI (2017); Brockman et al.

Original: (2016) environment to simulate twolink robotic arm that can move in two-dimensional space.
Revised: (2016) environment to simulate twolink robotic arm that can move in two-dimensional space.

Original: The desired task is to learn reaching a given target from random starting conﬁgurations.
Revised: The desired task is to learn reaching a given target from random starting conﬁgurations.

Original: The arm is controlled using angular torque values of both joints.
Revised: The arm is controlled using angular torque values of both joints.

Original: We use state values consisting of angular position, angular velocity, the end effector location of the robotic link and the position of the target.
Revised: We use state values consisting of angular position, angular velocity, the end effector location of the robotic link and the position of the target.

Original: The robotic arm angles and angular velocities were used as φ ( s t ) , which is the portion of state dependent on action.
Revised: The robotic arm angles and angular velocities were used as φ ( s t ) , which is the portion of state dependent on action.

Original: In this experiment, we assume that the true reward function is known and in addition, we have some state trajectories from the expert policy.
Revised: In this experiment, we assume that the true reward function is known and in addition, we have some state trajectories from the expert policy.

Original: The goal is to leverage these existing state trajectories to learn a better policy in less number of steps.
Revised: The goal is to leverage these existing state trajectories to learn a better policy in less number of steps.

Original: Reward signal, consisting of distance potential, electricity cost and the penalty for stuck joint, which is the default reward speciﬁed for the environment, was used.
Revised: Reward signal, consisting of distance potential, electricity cost and the penalty for stuck joint, which is the default reward speciﬁed for the environment, was used.

Original: Speciﬁcally, we show 500 trajectories of optimal states each with 100 steps to learn model-based policy using the proposed method.
Revised: Speciﬁcally, we show 500 trajectories of optimal states each with 100 steps to learn model-based policy using the proposed method.

Original: We used neural networks with hidden layers containing (128 , 64) neurons for both model-based and model-free policies.
Revised: We used neural networks with hidden layers containing (128 , 64) neurons for both model-based and model-free policies.

Original: Training was performed over 2000 episodes using Deep Deterministic Policy Gradients(DDPG) proposed by Lillicrap et al.
Revised: Training was performed over 2000 episodes using Deep Deterministic Policy Gradients(DDPG) proposed by Lillicrap et al.

Original: Figure 3(a) shows the comparison of proposed method against the DDPG algorithm.
Revised: Figure 3(a) shows the comparison of proposed method against the DDPG algorithm.

Original: Our method learns the dynamics model from the model-free exploration, which quickly learns the simple environment dynamics, in this case, thereby learning an optimal policy imitating the state trajectories much faster than model-free training, which is shown in results.
Revised: Our method learns the dynamics model from the model-free exploration, which quickly learns the simple environment dynamics, in this case, thereby learning an optimal policy imitating the state trajectories much faster than model-free training, which is shown in results.

Original: However, we found this is due to a large number of state trajectories that are shown to the proposed method.
Revised: However, we found this is due to a large number of state trajectories that are shown to the proposed method.

Original: Since our performance is upper bounded by behavior cloning results, we share the same drawbacks of compounding errors and data-hungry policy learning.
Revised: Since our performance is upper bounded by behavior cloning results, we share the same drawbacks of compounding errors and data-hungry policy learning.

Original: 3. 2D O BSTACLE AVOIDER
Revised: 3. 2D O BSTACLE AVOIDER

Original: In this experiment, we demonstrate that proposed algorithm can be used for direct end-to-end supervised imitation learning on novel tasks without resorting to model-free reinforcement learning.
Revised: In this experiment, we demonstrate that proposed algorithm can be used for direct end-to-end supervised imitation learning on novel tasks without resorting to model-free reinforcement learning.

Original: We refer to this setup as one-shot imitation learning, which we demonstrate on a simple toy environment.
Revised: We refer to this setup as one-shot imitation learning, which we demonstrate on a simple toy environment.

Original: The environment is shown in ﬁgure 2(a).
Revised: The environment is shown in ﬁgure 2(a).

Original: The environment consists of an agent which can freely move in 2D space.
Revised: The environment consists of an agent which can freely move in 2D space.

Original: The agent is controlled by continuous position control where the action is the change in ( x, y ) agent position at each time step.
Revised: The agent is controlled by continuous position control where the action is the change in ( x, y ) agent position at each time step.

Original: The goal is to reach the target location while avoiding the obstacle.
Revised: The goal is to reach the target location while avoiding the obstacle.

Original: Initially, we train our algorithm on an environment to avoid a single obstacle while reaching the target.
Revised: Initially, we train our algorithm on an environment to avoid a single obstacle while reaching the target.

Original: We use state information as the absolute position of the agent, target and obstacle, the agent velocity and relative location of obstacle and target with respect to the agent.
Revised: We use state information as the absolute position of the agent, target and obstacle, the agent velocity and relative location of obstacle and target with respect to the agent.

Original: We use φ ( s t ) as the agent 2d position in the environment.
Revised: We use φ ( s t ) as the agent 2d position in the environment.

Original: For expert demonstration, we implemented a manually engineered policy that always avoids the obstacle.
Revised: For expert demonstration, we implemented a manually engineered policy that always avoids the obstacle.

Original: We used 1000 number of demonstrations containing only state trajectories to reach a target while avoiding a single obstacle.
Revised: We used 1000 number of demonstrations containing only state trajectories to reach a target while avoiding a single obstacle.

Original: Out of 1000 demonstrations, 800 are used for training and 200 for validation.
Revised: Out of 1000 demonstrations, 800 are used for training and 200 for validation.

Original: We ﬁrst learn the time series prediction of the next state, used to compute the heuristic reward based using prediction error as discussed in section 2.1.
Revised: We ﬁrst learn the time series prediction of the next state, used to compute the heuristic reward based using prediction error as discussed in section 2.1.

Original: For model-based policy, we use a
Revised: For model-based policy, we use a

Original: MLP with (64 , 64) hidden units.
Revised: MLP with (64 , 64) hidden units.

Original: We use the same policy network for both model-based and modelfree policy.
Revised: We use the same policy network for both model-based and modelfree policy.

Original: The dynamics model is also modeled as a neural network with a single hidden layer of 8 units for both state and action input.
Revised: The dynamics model is also modeled as a neural network with a single hidden layer of 8 units for both state and action input.

Original: We used a switching frequency of 5 between the model-free and model-based updates.
Revised: We used a switching frequency of 5 between the model-free and model-based updates.

Original: Using these setting for the proposed algorithm, we get a model-based policy and the dynamics model as output.
Revised: Using these setting for the proposed algorithm, we get a model-based policy and the dynamics model as output.

Original: Using the dynamics model obtained from training with demonstrations of single obstacle avoidance, we perform one-shot imitation learning to learn avoidance of two obstacles.
Revised: Using the dynamics model obtained from training with demonstrations of single obstacle avoidance, we perform one-shot imitation learning to learn avoidance of two obstacles.

Original: The algorithm is presented 500 samples of expert state trajectories for avoiding two obstacles.
Revised: The algorithm is presented 500 samples of expert state trajectories for avoiding two obstacles.

Original: The model-based policy in the new setting is learned by step 6 of the proposed algorithm 1 using the previously learned dynamics model.
Revised: The model-based policy in the new setting is learned by step 6 of the proposed algorithm 1 using the previously learned dynamics model.

Original: Although the state information for the policy networks might change due to an additional obstacle, since φ ( s t ) , which is the agent 2d location, remains same in both cases, we can perform one-shot imitation learning in this case.
Revised: Although the state information for the policy networks might change due to an additional obstacle, since φ ( s t ) , which is the agent 2d location, remains same in both cases, we can perform one-shot imitation learning in this case.

Original: We compare the results with respect to the expert policy and behavior cloning and report the average of test reward on 50 episodes.
Revised: We compare the results with respect to the expert policy and behavior cloning and report the average of test reward on 50 episodes.

Original: While the expert policy achieves average test reward of 3036 , and behavior cloning achieves 3939 and our imitation learning method gave a reward of 3805 .
Revised: While the expert policy achieves average test reward of 3036 , and behavior cloning achieves 3939 and our imitation learning method gave a reward of 3805 .

Original: This demonstrates that our proposed algorithm can be used for one-shot imitation in environments with same dynamics and can produce comparable results to behavior cloning which was trained from true actions.
Revised: This demonstrates that our proposed algorithm can be used for one-shot imitation in environments with same dynamics and can produce comparable results to behavior cloning which was trained from true actions.

Original: 3. L EARNING POLICIES FROM RAW VIDEOS
Revised: 3. L EARNING POLICIES FROM RAW VIDEOS

Original: In this experiment, we learn model-based control policies from raw pixels.
Revised: In this experiment, we learn model-based control policies from raw pixels.

Original: We use the python reimplementation of the game Flappy bird (Lau (2017)).
Revised: We use the python reimplementation of the game Flappy bird (Lau (2017)).

Original: In this environment, the agent has to hover through the pipes without collision by ﬂapping its wings.
Revised: In this environment, the agent has to hover through the pipes without collision by ﬂapping its wings.

Original: The environment has gravity and with each ﬂap, the agent receives an upward acceleration which sends it to an upward parabolic trajectory.
Revised: The environment has gravity and with each ﬂap, the agent receives an upward acceleration which sends it to an upward parabolic trajectory.

Original: We choose this environment due to its complicated dynamics and show that our proposed method can learn to model this dynamics.
Revised: We choose this environment due to its complicated dynamics and show that our proposed method can learn to model this dynamics.

Original: We learn action policies from raw videos of just 10 episodes each with 1000 steps.
Revised: We learn action policies from raw videos of just 10 episodes each with 1000 steps.

Original: The reward is assumed to be unknown in this case and we estimate the reward by the error in prediction of the next state as mentioned in section 2.1.
Revised: The reward is assumed to be unknown in this case and we estimate the reward by the error in prediction of the next state as mentioned in section 2.1.

Original: The control action is a single discrete command of whether to ﬂap the bird’s wings or not.
Revised: The control action is a single discrete command of whether to ﬂap the bird’s wings or not.

Original: We denotes this action space as { +1 , − 1 } .
Revised: We denotes this action space as { +1 , − 1 } .

Original: For state information at each step we use 4 consecutive frames resized to (80 × 80 × 4) .
Revised: For state information at each step we use 4 consecutive frames resized to (80 × 80 × 4) .

Original: We also assume that the absolute position of the bird’s location is available to us, which we use as φ ( s t ) .
Revised: We also assume that the absolute position of the bird’s location is available to us, which we use as φ ( s t ) .

Original: This can be also computed by learning a simple object detector from image frames.
Revised: This can be also computed by learning a simple object detector from image frames.

Original: For the next state predictor, we use an LSTM predictor that outputs the next position of the agent location given the sequence of states witnessed so far.
Revised: For the next state predictor, we use an LSTM predictor that outputs the next position of the agent location given the sequence of states witnessed so far.

Original: The model-free reward prediction step receives a reward signal based on the difference of the actual next state taken the policy, from the predicted next state by LSTM.
Revised: The model-free reward prediction step receives a reward signal based on the difference of the actual next state taken the policy, from the predicted next state by LSTM.

Original: This reward is used to train DQN(Mnih et al.
Revised: This reward is used to train DQN(Mnih et al.

Original: for model-free policy update which collects data to train the dynamics model, which in turn trains the model-based policy network.
Revised: for model-free policy update which collects data to train the dynamics model, which in turn trains the model-based policy network.

Original: We also train vanilla DQN using a standard reward, which in this case is 0.1 for each time step and +1 reward if the agent can successfully pass through a pipe.
Revised: We also train vanilla DQN using a standard reward, which in this case is 0.1 for each time step and +1 reward if the agent can successfully pass through a pipe.

Original: To compare the various methods, we used the reward used by vanilla DQN as a baseline for comparison.
Revised: To compare the various methods, we used the reward used by vanilla DQN as a baseline for comparison.

Original: For the model-based policy we used a convolutional neural network (CNN) with soft-max output, which is essentially a binary classiﬁcation task.
Revised: For the model-based policy we used a convolutional neural network (CNN) with soft-max output, which is essentially a binary classiﬁcation task.

Original: For the DQN model-free policy, the last layer predicts the q-values and therefore has linearly activation.
Revised: For the DQN model-free policy, the last layer predicts the q-values and therefore has linearly activation.

Original: In this case, we ﬁrst learn dynamics model, which is approximated as a multi layered perceptron (MLP) with single hidden layer of 16 units for both state and action inputs.
Revised: In this case, we ﬁrst learn dynamics model, which is approximated as a multi layered perceptron (MLP) with single hidden layer of 16 units for both state and action inputs.

Original: It learns to regress the next state from the current state and actions minimizing mean squared error loss.
Revised: It learns to regress the next state from the current state and actions minimizing mean squared error loss.

Original: After this, from the expert demonstration of state trajectory( τ E ), we ﬁnd the current optimal action that minimizes the next state a ∗ k = arg min a (cid:107) p ( s t +1 − f ( s t , a t ) (cid:107)| a t ∈{ 0 , 1 } , { s t +1 , s t }∼ τ E .
Revised: After this, from the expert demonstration of state trajectory( τ E ), we ﬁnd the current optimal action that minimizes the next state a [Equation]

Original:  The next step is to ﬁnd the modelbased policy by behavior cloning on the states ( s t ∼ τ E ) and current optimal actions, a ∗ k .
Revised: }∼ τ E . The next step is to ﬁnd the modelbased policy by behavior cloning on the states ( s t ∼ τ E ) and current optimal actions, a ∗ k .

Original: We found that, the number of +1(ﬂap) actions by the agent are far less frequent compared to the number of -1(no ﬂap) actions which cause an unbalance in distribution.
Revised: We found that, the number of +1(ﬂap) actions by the agent are far less frequent compared to the number of -1(no ﬂap) actions which cause an unbalance in distribution.

Original: Thus, we used class balancing techniques for learning both model based policy update and direct behavior cloning from true actions (baseline) results.
Revised: Thus, we used class balancing techniques for learning both model based policy update and direct behavior cloning from true actions (baseline) results.

Original: Figure 3(b) shows the comparison of proposed model-based method with estimated reward ( mb reward pred ), behavior cloning ( bc ), model-free RL using DQN with standard known reward ( dqn reward ) and with estimated reward ( dqn reward pred ).
Revised: Figure 3(b) shows the comparison of proposed model-based method with estimated reward ( mb reward pred ), behavior cloning ( bc ), model-free RL using DQN with standard known reward ( dqn reward ) and with estimated reward ( dqn reward pred ).

Original: It is to be noted that although original reward just provides a constant reward (of 0.1 and +1 bonus) without constant guidance at each step value, estimated reward provides a dense guidance signal which leads to faster convergence as shown in the comparison results.
Revised: It is to be noted that although original reward just provides a constant reward (of 0.1 and +1 bonus) without constant guidance at each step value, estimated reward provides a dense guidance signal which leads to faster convergence as shown in the comparison results.

Original: We take desired samples from the model-free training, based on prioritized sampling technique with regards to the estimated reward signal, to perform model-based policy update.
Revised: We take desired samples from the model-free training, based on prioritized sampling technique with regards to the estimated reward signal, to perform model-based policy update.

Original: We found that prioritized sampling was essential to learn a good dynamics model of the environment.
Revised: We found that prioritized sampling was essential to learn a good dynamics model of the environment.

Original: The results show that the model-based policy learns behavior close to optimal in much fewer steps compared to the model-free counterparts.
Revised: The results show that the model-based policy learns behavior close to optimal in much fewer steps compared to the model-free counterparts.

Original: However, its performance in upper bounded by the behavior cloning method which achieves an average test reward (over 20 iterations) of 49.62 .
Revised: However, its performance in upper bounded by the behavior cloning method which achieves an average test reward (over 20 iterations) of 49.62 .

Original: The expert score is 60.9 .
Revised: The expert score is 60.9 .

Original: Since DQN with estimated reward signal learns via exploration in a MDP setting, it can surpass the performance of behavior cloning since the number of expert demonstrations is limited in this case.
Revised: Since DQN with estimated reward signal learns via exploration in a MDP setting, it can surpass the performance of behavior cloning since the number of expert demonstrations is limited in this case.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: We presented a model-based imitation learning method that can learn to act from expert state trajectories in the absence of action information.
Revised: We presented a model-based imitation learning method that can learn to act from expert state trajectories in the absence of action information.

Original: Our method uses trajectories sampled from the modelfree policy exploration to train a dynamics model of the environment.
Revised: Our method uses trajectories sampled from the modelfree policy exploration to train a dynamics model of the environment.

Original: As model-free policy is enriched through time, the forward model can better approximate the actual environment dynamics, which leads to improved gradient ﬂow, leading to better model-based policy update which is trained in a supervised fashion from expert state trajectories.
Revised: As model-free policy is enriched through time, the forward model can better approximate the actual environment dynamics, which leads to improved gradient ﬂow, leading to better model-based policy update which is trained in a supervised fashion from expert state trajectories.

Original: In the ideal case, when dynamics model perfectly approximates the environment, our proposed method is equivalent to behavior cloning, even in the absence of action information.
Revised: In the ideal case, when dynamics model perfectly approximates the environment, our proposed method is equivalent to behavior cloning, even in the absence of action information.

Original: We demonstrate that the proposed method learns the desired policy in less number of iterations compared conventional model-free methods.
Revised: We demonstrate that the proposed method learns the desired policy in less number of iterations compared conventional model-free methods.

Original: We also show that once the dynamics model is trained it can be used to transfer learning for other tasks in a similar environment in an end-to-end supervised manner.
Revised: We also show that once the dynamics model is trained it can be used to transfer learning for other tasks in a similar environment in an end-to-end supervised manner.

Original: Future work includes tighter integration of the model-based learning and the model-free learning for higher data efﬁciency by sharing information (1) between the model-free policy π mf and the model-based policy π mb and (2) between the next state predictor p ( s t +1 | s t ) and the dynamics model p ( s t +1 | s t , a t ) and (3) improving the limitations of compounding errors and requirement of large number of demonstration, by adversarial training which can maximize likelihood of future state distributions as well.
Revised: Future work includes tighter integration of the model-based learning and the model-free learning for higher data efﬁciency by sharing information (1) between the model-free policy π mf and the model-based policy π mb and (2) between the next state predictor p ( s t +1 | s t ) and the dynamics model p ( s t +1 | s t , a t ) and (3) improving the limitations of compounding errors and requirement of large number of demonstration, by adversarial training which can maximize likelihood of future state distributions as well.