Original: M ODEL -P REDICTIVE P OLICY L EARNING WITH U NCERTAINTY R EGULARIZATION FOR D RIVING IN D ENSE T RAFFIC
Revised: M ODEL -P REDICTIVE P OLICY L EARNING WITH U NCERTAINTY R EGULARIZATION FOR D RIVING IN D ENSE T RAFFIC

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training.
Revised: Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training.

Original: We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on.
Revised: We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on.

Original: We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks.
Revised: We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks.

Original: We evaluate our approach using a large-scale observational dataset of driving behavior recorded from trafﬁc cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.
Revised: We evaluate our approach using a large-scale observational dataset of driving behavior recorded from trafﬁc cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, model-free reinforcement learning methods using deep neural network controllers have proven effective on a wide range of tasks, from playing video or text-based games (Mnih et al., 2015; 2016; Narasimhan et al., 2015) to learning algorithms (Zaremba et al., 2015) and complex locomotion tasks (Lillicrap et al., 2015;
Revised: In recent years model-free reinforcement learning methods using deep neural network controllers have proven effective on a wide range of tasks, from playing video or text-based games (Mnih et al., 2015; 2016; Narasimhan et al., 2015) to learning algorithms (Zaremba et al., 2015) and complex locomotion tasks (Lillicrap et al., 2015;

Original: Zhang et al., 2015).
Revised: Zhang et al., 2015).

Original: However, these methods often require a large number of interactions with the environment in order to learn.
Revised: However, these methods often require a large number of interactions with the environment in order to learn.

Original: While this is not a problem if the environment is simulated, it can limit the application of these methods in realistic environments where interactions with the environment are slow, expensive or potentially dangerous.
Revised: While this is not a problem if the environment is simulated, it can limit the application of these methods in realistic environments where interactions with the environment are slow, expensive or potentially dangerous.

Original: Building a simulator where the agent can safely try out policies without facing real consequences can mitigate this problem, but requires human engineering effort which increases with the complexity of the environment being modeled.
Revised: Building a simulator where the agent can safely try out policies without facing real consequences can mitigate this problem but requires human engineering effort which increases with the complexity of the environment being modeled.

Original: Model-based reinforcement learning approaches try to learn a model of the environment dynamics, and then use this model to plan actions or train a parameterized policy.
Revised: Model-based reinforcement learning approaches try to learn a model of the environment dynamics, and then use this model to plan actions or train a parameterized policy.

Original: A common setting is where an agent alternates between collecting experience by executing actions using its current policy or dynamics model, and then using these experiences to improve its dynamics model.
Revised: A common setting is where an agent alternates between collecting experience by executing actions using its current policy or dynamics model, and then using these experiences to improve its dynamics model.

Original: This approach has been shown empirically to signiﬁcantly reduce the required number of environment interactions needed to obtain an effective policy or planner (Atkeson & Santamaria, 1997;
Revised: This approach has been shown empirically to signiﬁcantly reduce the required number of environment interactions needed to obtain an effective policy or planner (Atkeson & Santamaria, 1997;

Original: Deisenroth &
Revised: Deisenroth &

Original: Rasmussen, 2011; Nagabandi et al., 2017; Chua et al., 2018).
Revised: Rasmussen, 2011; Nagabandi et al., 2017; Chua et al., 2018).

Original: Despite these improvements in sample complexity, there exist settings where even a single poor action executed by an agent in a real environment can have consequences which are not acceptable.
Revised: Despite these improvements in sample complexity, there exist settings where even a single poor action executed by an agent in a real environment can have consequences which are not acceptable.

Original: At the same time, with data collection becoming increasingly inexpensive, there are many settings where observational data of an environment is abundant.
Revised: At the same time, with data collection becoming increasingly inexpensive, there are many settings where observational data of an environment is abundant.

Original: This suggests a need for algorithms which can learn policies primarily from observational data, which can then perform well in a real environment.
Revised: This suggests a need for algo- rithms which can learn policies primarily from observational data, which can then perform well in a real environment.

Original: Autonomous driving is an example of such a setting: on one hand, trajectories of human drivers can be easily collected using trafﬁc cameras (Halkias & Colyar, 2006), resulting in an abundance of observational data; on the other hand, learning through interaction with the real environment is not a viable solution.
Revised: Autonomous driving is an example of such a setting: on one hand, trajectories of human drivers can be easily collected using trafﬁc cameras (Halkias & Colyar, 2006), resulting in an abundance of observational data; on the other hand, learning through interaction with the real environment is not a viable solution.

Original: However, learning policies from purely observational data is challenging because the data may only cover a small region of the space over which it is deﬁned.
Revised: However, learning policies from purely observational data is challenging because the data may only cover a small region of the space over which it is deﬁned.

Original: If the observational data consists of state-action pairs produced by an expert, one option is to use imitation learning (Pomerleau, 1991).
Revised: If the observational data consists of state-action pairs produced by an expert, one option is to use imitation learning (Pomerleau, 1991).

Original: [Figure]
Revised: [Figure]

Original: which cover a limited region the function’s domain. Models make arbitrary predictions However, this is well-known to suffer from a mismatch between the states seen at training and execution time (Ross & Bagnell, 2010), and may need to be augmented by querying an expert (Ross et al., 2011).
Revised: However, this is well-known to suffer from a mismatch between the states seen at training and execution time (Ross & Bagnell, 2010).

Original: Another option is to learn a dynamics model from observational data, and then use it to train a policy (Nguyen & Widrow, 1989).
Revised: Another option is to learn a dynamics model from observational data, and then use it to train a policy (Nguyen & Widrow, 1989).

Original: However, the dynamics model may make arbitrary predictions outside the domain it was trained on, which may wrongly be associated with low cost (or high reward) as shown in Figure 1.
Revised: However, the dynamics model may make arbitrary predictions outside the domain it was trained on, which may wrongly be associated with low cost (or high reward) as shown in Figure 1.

Original: The policy network may then exploit these errors in the dynamics model and produce actions which lead to wrongly optimistic states.
Revised: The policy network may then exploit these errors in the dynamics model and produce actions which lead to wrongly optimistic states.

Original: In the interactive setting, this problem is naturally self-correcting, since states where the model predictions are wrongly optimistic will be more likely to be experienced, and thus will correct the dynamics model.
Revised: In the interactive setting, this problem is naturally self-correcting, since states where the model predictions are wrongly optimistic will be more likely to be experienced, and thus will correct the dynamics model.

Original: However, the problem persists if the dataset of environment interactions which the model is trained on is ﬁxed.
Revised: However, the problem persists if the dataset of environment interactions which the model is trained on is ﬁxed.

Original: In this work, we propose to train a policy while explicitly penalizing the mismatch between the distribution of trajectories it induces and the one reﬂected in the training data.
Revised: In this work, we propose to train a policy while explicitly penalizing the mismatch between the distribution of trajectories it induces and the one reﬂected in the training data.

Original: We use a learned dynamics model which is unrolled for multiple time steps, and training a policy network to minimize a differentiable cost over this rolledout trajectory.
Revised: We use a learned dynamics model which is unrolled for multiple time steps, and train a policy network to minimize a differentiable cost over this rolled-out trajectory.

Original: This cost contains two terms: a policy cost which represents the objective the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on.
Revised: This cost contains two terms: a policy cost which represents the objective the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on.

Original: We measure this second cost by using the uncertainty of the dynamics model about its own predictions, calculated using dropout.
Revised: We measure this second cost by using the uncertainty of the dynamics model about its own predictions, calculated using dropout.

Original: We evaluate our approach in the context of learning policies to drive an autonomous car in dense trafﬁc, using a large-scale dataset of real-world driving trajectories which we also adapt into an environment for testing learned policies; both dataset and environment will be made public.
Revised: We apply our approach in the context of learning policies to drive an autonomous car in dense trafﬁc, using a large-scale dataset of real-world driving trajectories which we also adapt into an environment for testing learned policies 1 .

Original: We show that model-based control using this additional uncertainty regularizer substantially outperforms unregularized control, and enables learning good driving policies using only observational data with no environment interaction or additional labeling by an expert.
Revised: We show that model-based control using this additional uncertainty regularizer substantially outperforms unregularized control, and enables learning good driving policies using only observational data with no environment interaction or additional labeling by an expert.

Original: We also show how to effectively leverage an action-conditional stochastic forward model using a modiﬁed posterior distribution, which encourages the model to maintain sensitivity to input actions.
Revised: We also show how to effectively leverage an action-conditional stochastic forward model using a modiﬁed posterior distribution, which encourages the model to maintain sensitivity to input actions.

Original:  M ODEL -P REDICTIVE P OLICY L EARNING WITH U NCERTAINTY R EGULARIZATION
Revised:  M ODEL -P REDICTIVE P OLICY L EARNING WITH U NCERTAINTY R EGULARIZATION

Original: We assume we are given a dataset of observational data which consists of state-action pairs D
Revised: We assume we are given a dataset of observational data which consists of state-action pairs D

Original: = { ( s t , a t ) } t .
Revised: = { ( s t , a t ) } t .

Original: We ﬁrst describe our general approach, which consists of two steps: learning an actionconditional dynamics model using the collected observational data, and then using this model to train a fast, feedforward policy network which minimizes both a policy cost and an uncertainty cost.
Revised: We ﬁrst describe our general approach, which consists of two steps: learning an actionconditional dynamics model using the collected observational data, and then using this model to train a fast, feedforward policy network which minimizes both a policy cost and an uncertainty cost.

Original: 2. A CTION - CONDITIONAL F ORWARD M ODEL
Revised: 2. A CTION - CONDITIONAL F ORWARD M ODEL

Original: In this work, we consider recent approaches for stochastic prediction based on Variational Autoencoders (Kingma & Welling, 2013; Babaeizadeh et al., 2017; Denton & Fergus, 2018).
Revised: In this work, we consider recent approaches for stochastic prediction based on Variational Autoencoders (Kingma & Welling, 2013; Babaeizadeh et al., 2017; Denton & Fergus, 2018).

Original: The stochastic model f θ ( s 1: t , a t , z t ) takes as input a sequence of observed or previously predicted states s 1: t , an action a t , and a latent variable z t which represents the information about the next state s t +1 which is not a deterministic function of the input.
Revised: The stochastic model f θ ( s 1: t , a t , z t ) takes as input a sequence of observed or previously predicted states s 1: t , an action a t , and a latent variable z t which represents the information about the next state s t +1 which is not a deterministic function of the input.

Original: During training, latent variables are sampled from a distribution whose parameters are output by a posterior network q φ ( s 1: t , s t +1 ) conditioned on the past inputs and true targets.
Revised: During training, latent variables are sampled from a distribution whose parameters are output by a posterior network q φ ( s 1: t , s t +1 ) conditioned on the past inputs and true targets.

Original: This network is trained jointly with the rest of the model using the reparameterization trick, and a term is included in the loss to minimize the KL divergence between the posterior distribution and a ﬁxed prior p ( z ) , which in our case is an isotropic Gaussian.
Revised: This network is trained jointly with the rest of the model using the reparameterization trick, and a term is included in the loss to minimize the KL divergence between the posterior distribution and a ﬁxed prior p ( z ) , which in our case is an isotropic Gaussian.

Original: The per-sample loss used for training the stochastic model is given by:
Revised: The per-sample loss used for training the stochastic model is given by:

Original: [Equation]
Revised: [Equation]

Original: After training, different future predictions for a given sequence of frames can be generated by sampling different latent variables from the prior distribution.
Revised: After training, different future predictions for a given sequence of frames can be generated by sampling different latent variables from the prior distribution.

Original: [Figure] Recent models for stochastic video prediction (Babaeizadeh et al., 2017; Denton & Fergus, 2018) do not use their model for planning or training a policy network, and parameterize the posterior distribution over latent variables using a diagonal Gaussian.
Revised: Recent models for stochastic video prediction (Babaeizadeh et al., 2017; Denton & Fergus, 2018) do not use their model for planning or training a policy network, and parameterize the posterior distribution over latent variables using a diagonal Gaussian.

Original: In our case, we are training an actionconditional video prediction model which we will later use to train a policy.
Revised: In our case, we are training an actionconditional video prediction model which we will later use to train a policy.

Original: This leads to an additional requirement: it is important for the prediction model to accurately respond to input actions, and not use the latent variables to encode factors of variation in the outputs which are due to the actions.
Revised: This leads to an additional requirement: it is important for the prediction model to accurately respond to input actions, and not use the latent variables to encode factors of variation in the outputs which are due to the actions.

Original: To this end we propose to use a mixture of two Gaussians, with one component ﬁxed to the prior, as our posterior distribution:
Revised: To this end we propose to use a mixture of two Gaussians, with one component ﬁxed to the prior, as our posterior distribution:

Original: [Equation]
Revised: [Equation]

Original: Thisis related to the dropout approximating distribution described in (Gal & Ghahramani, 2016), and can be seen as applying a form of global dropout to the latent variables at training time 1 .
Revised: This can be seen as applying a form of global dropout to the latent variables at training time 2 .

Original: This forces the prediction model to extract as much information as possible from the input states and actions by making the latent variable independent of the output with some probability.
Revised: and forces the prediction model to extract as much information as possible from the input states and actions by making the latent variable independent of the output with some probability.

Original: In our experiments we will refer to this parameterization as z -dropout.
Revised: In our experiments we will refer to this parameterization as z -dropout.

Original: 2. T RAINING A P OLICY N ETWORK WITH U NCERTAINTY M INIMIZATION
Revised: 2. T RAINING A P OLICY N ETWORK WITH U NCERTAINTY M INIMIZATION

Original: Once the forward model is trained, we use it to train a parameterized policy network π ψ , which we assume to be stochastic.
Revised: Once the forward model is trained, we use it to train a parameterized policy network π ψ , which we assume to be stochastic.

Original: We ﬁrst sample an initial state sequence s 1: t from the training set, unroll the forward model over T time steps, and backpropagate gradients of a differentiable objective function with respect to the parameters of the policy network (shown in Figure 2).
Revised: We ﬁrst sample an initial state sequence s 1: t from the training set, unroll the forward model over T time steps, and backpropagate gradients of a differentiable objective function with respect to the parameters of the policy network (shown in Figure 2).

Original: During this process the weights of the forward model are ﬁxed, and only the weights of the policy network are optimized.
Revised: During this process the weights of the forward model are ﬁxed, and only the weights of the policy network are optimized.

Original: This objective function contains two terms: a policy cost C , which reﬂects the underlying objective the policy is trying to learn, and an uncertainty cost U , which reﬂects how close the predicted state induced by the policy network is to the manifold which the data D is drawn from.
Revised: This objective function contains two terms: a policy cost C , which reﬂects the underlying objective the policy is trying to learn, and an uncertainty cost U , which reﬂects how close the predicted state induced by the policy network is to the manifold which the data D is drawn from.

Original: Training the policy using a stochastic forward model involves solving the following problem, where latent variables are sampled from the prior and input into the forward model at every time step:argmin
Revised: Training the policy using a stochastic forward model involves solving the following problem, where latent variables are sampled from the prior and input into the forward model at every time step:

Original: [Equation]
Revised: [Equation]

Original: i =

Original: C (ˆ s t + i ) + λU (ˆ s t

Original: + i ) (cid:105) , such that:

Original: [Equation]

Original: [Figure]

Original: The uncertainty cost U is applied to states predicted by the forward model, and could reﬂect any measure of their likelihood under the distribution the training data is drawn from.
Revised: The uncertainty cost U is applied to states predicted by the forward model, and could reﬂect any measure of their likelihood under the distribution the training data is drawn from.

Original: We propose here a general form based on the uncertainty of the dynamics model, which is calculated using dropout.
Revised: We propose here a general form based on the uncertainty of the dynamics model, which is calculated using dropout.

Original: Intuitively, if the dynamics model is given a state-action pair from the same distribution as D (which it was trained on), it will have low uncertainty about its prediction.
Revised: Intuitively, if the dynamics model is given a state-action pair from the same distribution as D (which it was trained on), it will have low uncertainty about its prediction.

Original: If it is given a state-action pair which is outside this distribution, it will have high uncertainty.
Revised: If it is given a state-action pair which is outside this distribution, it will have high uncertainty.

Original:  Dropout (Hinton et al., 2012; Srivastava et al., 2014) is a regularization technique which consists of randomly setting hidden units in a neural network to zero with some probability.
Revised: [Figure] Dropout (Hinton et al., 2012; Srivastava et al., 2014) is a regularization technique which consists of randomly setting hidden units in a neural network to zero with some probability.

Original: The work of (Gal & Ghahramani, 2016) showed thata neural network trained with dropout is equivalent to an approximation of a probabilistic deep Gaussian Process model. A key result of this is that estimates of the neural network’s model uncertainty for a given input can be obtained by calculating the covariance of its outputs taken over multiple dropout masks.
Revised: The work of (Gal & Ghahramani, 2016) showed that estimates of a neural network’s model uncertainty for a given input can be obtained by calculating the covariance of its outputs taken over multiple dropout masks.

Original: We note that this uncertainty estimate is the composition of differentiable functions: each of the models induced by applying a different dropout mask is differentiable, as is the covariance operator.
Revised: We note that this uncertainty estimate is the composition of differentiable functions: each of the models induced by applying a different dropout mask is differentiable, as is the covariance operator.

Original: Furthermore, we can summarize the covariance matrix by taking its trace (which is equal to the sum of its eigenvalues, or equivalently the sum of the variances of the outputs across each dimension), which is also a differentiable operation.
Revised: Furthermore, we can summarize the covariance matrix by taking its trace (which is equal to the sum of its eigenvalues, or equivalently the sum of the variances of the outputs across each dimension), which is also a differentiable operation.

Original: This provides a scalar estimate of uncertainty which is differentiable with respect to the input.
Revised: This provides a scalar estimate of uncertainty which is differentiable with respect to the input.

Original: More precisely, let f θ 1 , ..., f θ K denote our prediction model with K different dropout masks applied to its hidden units (this can also be viewed as changing its weights).
Revised: More precisely, let f θ 1 , ..., f θ K denote our prediction model with K different dropout masks applied to its hidden units (this can also be viewed as changing its weights).

Original: We deﬁne our scalar measure of uncertainty U as follows:
Revised: We deﬁne our scalar measure of uncertainty U as follows:

Original: [Equation]
Revised: [Equation]

Original: where d is the dimensionality of the output.
Revised: where d is the dimensionality of the output.

Original: Minimizing this quantity with respect to actions encourages the policy network to produce actions which, when plugged into the forward model, will produce predictions which the forward model is conﬁdent about 2 .
Revised: Minimizing this quantity with respect to actions encourages the policy network to produce actions which, when plugged into the forward model, will produce predictions which the forward model is conﬁdent about 3 .

Original: A simple way to deﬁne U given an initial sequence of states s 1: t from D would be to set
Revised: A simple way to deﬁne U given an initial sequence of states s 1: t from D would be to set

Original: U (ˆ s t + k ) =
Revised: U (ˆ s t + k ) =

Original: (cid:107) ˆ s t
Revised: (cid:107) ˆ s t

Original: + k − s t + k (cid:107) 2 , which would encourage the policy network to output actions which lead to a similar trajectory as the one observed in the dataset.
Revised: + k − s t + k (cid:107) 2 , which would encourage the policy network to output actions which lead to a similar trajectory as the one observed in the dataset.

Original: This leads to a set of states which the model is presumably conﬁdent about, but may not be a trajectory which also satisﬁes the policy cost C unless the dataset D consists of expert trajectories.
Revised: This leads to a set of states which the model is presumably conﬁdent about, but may not be a trajectory which also satisﬁes the policy cost C unless the dataset D consists of expert trajectories.

Original: If this is the case, setting
Revised: If this is the case, setting

Original: C (ˆ s t
Revised: C (ˆ s t

Original: + i ) =
Revised: + i ) =

Original: U (ˆ s t + i ) =
Revised: U (ˆ s t + i ) =

Original: 12 (cid:107) ˆ s t
Revised: 12 (cid:107) ˆ s t

Original: + k − s t + k (cid:107) 2 gives a model-based imitation learning objective which simultaneously optimizes the policy cost and the uncertainty cost.
Revised: + k − s t + k (cid:107) 2 gives a model-based imitation learning objective which simultaneously optimizes the policy cost and the uncertainty cost.

Original: The problem then becomes:
Revised: The problem then becomes:

Original: [Equation]
Revised: [Equation]

Original:  such that:
Revised: , such that

Original: [Equation]
Revised: [Equation]

Original:  We call the ﬁrst approach MPUR , for Model-Predictive Policy with Uncertainty Regularization, and the second MPER , for Model-Predictive Policy with Expert Regularization.
Revised: [Figure] We call the ﬁrst approach MPUR , for Model-Predictive Policy with Uncertainty Regularization, and the second MPER , for Model-Predictive Policy with Expert Regularization.

Original: A key feature of both approaches is that we optimize the objective over T time steps, which is made possible by our learned dynamics model.
Revised: A key feature of both approaches is that we optimize the objective over T time steps, which is made possible by our learned dynamics model.

Original: This means that the actions will receive gradients from multiple time steps ahead, which will penalize actions which lead to large divergences from the training manifold further into the future, even if they only cause a small divergence at the next time step.
Revised: This means that the actions will receive gradients from multiple time steps ahead, which will penalize actions which lead to large divergences from the training manifold further into the future, even if they only cause a small divergence at the next time step.

Original: 2. R ELATIONSHIP TO B AYESIAN N EURAL N ETWORKS
Revised: 2. R ELATIONSHIP TO B AYESIAN N EURAL N ETWORKS

Original: Our MPUR approach can be viewed as training a Bayesian neural network (BNN) (Neal, 1995) with latent variables using variational inference (Jordan et al., 1999; Kingma & Welling, 2013).
Revised: Our MPUR approach can be viewed as training a Bayesian neural network (BNN) (Neal, 1995) with latent variables using variational inference (Jordan et al., 1999; Kingma & Welling, 2013).

Original: The distribution over model predictions for s t +1 is given by:
Revised: The distribution over model predictions for s t +1 is given by:

Original: [Equation]
Revised: [Equation]

Original: The distribution p ( θ, z |D ) reﬂects the posterior over model weights and latent variables given the data, and is intractable to evaluate.
Revised: The distribution p ( θ, z |D ) reﬂects the posterior over model weights and latent variables given the data, and is intractable to evaluate.

Original: We instead approximate it with the variational distribution q parameterized by η = { φ, θ ∗ } :
Revised: We instead approximate it with the variational distribution q parameterized by η = { φ, θ ∗ } :

Original: [Equation]
Revised: [Equation]

Original: Here q φ represents a distribution over latent variables represented using a posterior network with parameters φ , which could be a diagonal Gaussian or the mixture distribution described in Section 2.1.
Revised: Here q φ represents a distribution over latent variables represented using a posterior network with parameters φ , which could be a diagonal Gaussian or the mixture distribution described in Section 2.1.

Original: The distribution q θ ∗ is the dropout approximating distribution over forward model parameters described in (Gal & Ghahramani, 2016), a mixture of two Gaussians with one mean ﬁxed at zero.
Revised: The distribution q θ ∗ is the dropout approximating distribution over forward model parameters described in (Gal & Ghahramani, 2016), a mixture of two Gaussians with one mean ﬁxed at zero.

Original: We show in Appendix B that training the stochastic forward model with dropout by minimizing the loss function in Equation 1 is approximately minimizing the Kullback-Leibler divergence between this approximate posterior and the true posterior.
Revised: We show in Appendix B that training the stochastic forward model with dropout by minimizing the loss function in Equation 1 is approximately minimizing the Kullback-Leibler divergence between this approximate posterior and the true posterior.

Original: Once the forward model is trained, for a given input we can obtain an approximate distribution over outputs p
Revised: Once the forward model is trained, for a given input we can obtain an approximate distribution over outputs p

Original: (ˆ s t +1 | s 1: t , a ) induced by the approximate posterior by sampling different latent variables and dropout masks.
Revised: (ˆ s t +1 | s 1: t , a ) induced by the approximate posterior by sampling different latent variables and dropout masks.

Original: We now show that the covariance of the outputs
Revised: We now show that the covariance of the outputs

Original: ˆ s t +1 can be decomposed into a sum of two covariance matrices which represent the aleatoric and epistemic uncertainty, using a similar approach as (Depeweg et al., 2018).
Revised: ˆ s t +1 can be decomposed into a sum of two covariance matrices which represent the aleatoric and epistemic uncertainty, using a similar approach as (Depeweg et al., 2018).

Original: Using the conditional covariance formula we can write:
Revised: Using the conditional covariance formula we can write:

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is the covariance of the random vector E z [ˆ s t +1 | s 1: t , a, θ ] when θ ∼ q θ ∗ ( θ ) .
Revised: The ﬁrst term is the covariance of the random vector E z [ˆ s t +1 | s 1: t , a, θ ] when θ ∼ q θ ∗ ( θ ) .

Original: This term ignores any contribution to the variance from z and only considers the effect of θ .
Revised: This term ignores any contribution to the variance from z and only considers the effect of θ .

Original: As such it represents the epistemic uncertainty.
Revised: As such it represents the epistemic uncertainty.

Original: The second term represents the covariance of the predictions obtained by sampling different latent variables z ∼ p ( z ) averaged over different dropout masks, and ignores any contribution to the variance from θ .
Revised: The second term represents the covariance of the predictions obtained by sampling different latent variables z ∼ p ( z ) averaged over different dropout masks, and ignores any contribution to the variance from θ .

Original: As such it represents the aleatoric uncertainty.
Revised: As such it represents the aleatoric uncertainty.

Original: Our uncertainty penalty explicitly penalizes the trace of the ﬁrst matrix where the expectation over z is approximated by a single sample from the prior.
Revised: Our uncertainty penalty explicitly penalizes the trace of the ﬁrst matrix where the expectation over z is approximated by a single sample from the prior.

Original: Note also that the covariance matrix corresponding to the aleatoric uncertainty will change depending on the inputs.
Revised: Note also that the covariance matrix corresponding to the aleatoric uncertainty will change depending on the inputs.

Original: This allows our approach to handle heteroscedastic environments, where the aleatoric uncertainty will vary for different inputs.[Figure]
Revised: This allows our approach to handle heteroscedastic environments, where the aleatoric uncertainty will vary for different inputs.

Original:  D ATASET AND P LANNING E NVIRONMENT
Revised:  D ATASET AND P LANNING E NVIRONMENT

Original: We apply our approach to learn driving policies using a large-scale dataset of driving videos taken from trafﬁc cameras.
Revised: We apply our approach to learn driving policies using a large-scale dataset of driving videos taken from trafﬁc cameras.

Original: The Next Generation Simulation program’s Interstate 80 (NGSIM I-80) dataset (Halkias & Colyar, 2006) consists of 45 minutes of recordings from trafﬁc cameras mounted over a stretch of highway.
Revised: The Next Generation Simulation program’s Interstate 80 (NGSIM I-80) dataset (Halkias & Colyar, 2006) consists of 45 minutes of recordings from trafﬁc cameras mounted over a stretch of highway.

Original: The driver behavior is complex and includes sudden accelerations, lane changes and merges which are difﬁcult to predict; as such the dataset has high enviroment (or aleatoric) uncertainty.
Revised: The driver behavior is complex and includes sudden accelerations, lane changes and merges which are difﬁcult to predict; as such the dataset has high environment (or aleatoric) uncertainty.

Original: After recording, a viewpoint transformation is applied to rectify the perspective, and vehicles are identiﬁed and tracked throughout the video.
Revised: After recording, a viewpoint transformation is applied to rectify the perspective, and vehicles are identiﬁed and tracked throughout the video.

Original: This yields a total 5596 car trajectories, which we split into training ( 80% ), validation ( 10% ) and testing sets ( 10% ).
Revised: This yields a total 5596 car trajectories, which we split into training ( 80% ), validation ( 10% ) and testing sets ( 10% ).

Original: In all, the dataset contains approximately 2 million transitions.
Revised: In all, the dataset contains approximately 2 million transitions.

Original: We then applied additional preprocessing to obtain a state and action representation ( s t , a t ) for each car at each time step, suitable for learning an action-conditional predictive model.
Revised: We then applied additional preprocessing to obtain a state and action representation ( s t , a t ) for each car at each time step, suitable for learning an action-conditional predictive model.

Original: Our state representation s t consists of two components: an image i t representing the neighborhood of the car, and a vector u t representing its current position and velocity.
Revised: Our state representation s t consists of two components: an image i t representing the neighborhood of the car, and a vector u t representing its current position and velocity.

Original: The images i t are centered around the ego car and encode both the lane emplacements and the locations of other cars.
Revised: The images i t are centered around the ego car and encode both the lane emplacements and the locations of other cars.

Original: Each image has 3 channels: the ﬁrst (red) encodes the lane markings, the second (green) encodes the locations of neighboring cars, which are represented as rectangles reﬂecting the dimensions of each car, and the third channel (blue) represents the ego car, also scaled to the correct dimensions.
Revised: Each image has 3 channels: the ﬁrst (red) encodes the lane markings, the second (green) encodes the locations of neighboring cars, which are represented as rectangles reﬂecting the dimensions of each car, and the third channel (blue) represents the ego car, also scaled to the correct dimensions.

Original: This is summarized in Figure 4.
Revised: This is summarized in Figure 4.

Original: The action a t at a given time step consists of a 2-dimensional vector representing the acceleration/braking of the car and its change in steering angle.
Revised: The action a t at a given time step consists of a 2-dimensional vector representing the acceleration/braking of the car and its change in steering angle.

Original: We also deﬁne two cost functions which together make up the policy cost: a proximity cost which reﬂects how close the ego car is to neighboring cars, and a lane cost which reﬂects how much the ego car overlaps with lane markings.
Revised: We also deﬁne two cost functions which together make up the policy cost: a proximity cost which reﬂects how close the ego car is to neighboring cars, and a lane cost which reﬂects how much the ego car overlaps with lane markings.

Original: These are represented as a cost vector at each timestep, c t =
Revised: These are represented as a cost vector at each timestep, c t =

Original: ( C proximity ( s t ) , C lane ( s t )) .
Revised: ( C proximity ( s t ) , C lane ( s t )) .

Original: Full details can be found in Appendix A.
Revised: Full details can be found in Appendix A.

Original: We also adapted this dataset to be used as an environment to evaluate learned policies, with the same interface as OpenAI Gym (Brockman et al., 2016).
Revised: We also adapted this dataset to be used as an environment to evaluate learned policies, with the same interface as OpenAI Gym (Brockman et al., 2016).

Original: Choosing a policy for neighboring cars is challenging due to a cold-start problem: to accurately evaluate a learned policy, the other cars would need to follow human-like policies which would realistically react to the controlled car, which are not available.
Revised: Choosing a policy for neighboring cars is challenging due to a cold-start problem: to accurately evaluate a learned policy, the other cars would need to follow human-like policies which would realistically react to the controlled car, which are not available.

Original: We take the approach of letting all the other cars in the environment follow their trajectories from the dataset, while a single car is controlled by the policy we seek to evaluate.
Revised: We take the approach of letting all the other cars in the environment follow their trajectories from the dataset, while a single car is controlled by the policy we seek to evaluate.

Original: This approach avoids hand-designing a policy for the neighboring cars which would likely not reﬂect the diverse nature of human driving.
Revised: This approach avoids hand-designing a policy for the neighboring cars which would likely not reﬂect the diverse nature of human driving.

Original: The limitation is that the neighboring cars do not react to the controlled car, which likely makes the problem more difﬁcult as they do not try to avoid collisions.
Revised: The limitation is that the neighboring cars do not react to the controlled car, which likely makes the problem more difﬁcult as they do not try to avoid collisions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A number of authors have explored the use of learned, action-conditional forward models which are then used for planning, starting with classic works in the 90’s (Nguyen & Widrow, 1990; Schmidhuber, 1990; Jordan & Rumelhart, 1992), and more recently in the context of video games (Oh et al., 2015;
Revised: A number of authors have explored the use of learned, action-conditional forward models which are then used for planning, starting with classic works in the 90’s (Nguyen & Widrow, 1990; Schmidhuber, 1990; Jordan & Rumelhart, 1992), and more recently in the context of video games (Oh et al., 2015;

Original: Pascanu et al., 2017; Weber et al., 2017), robotics and continous control (Finn et al., 2016;
Revised: Pascanu et al., 2017; Weber et al., 2017), robotics and continous control (Finn et al., 2016;

Original: Agrawal et al., 2016; Nagabandi et al., 2017; Srinivas et al., 2018).
Revised: Agrawal et al., 2016; Nagabandi et al., 2017; Srinivas et al., 2018).

Original: Our approach to learning policies by backpropagating through a learned forward model is related to the early work of (Nguyen & Widrow, 1989) in the deterministic case, and the SVG framework of (Heess et al., 2015) in the stochastic case.
Revised: Our approach to learning policies by backpropagating through a learned forward model is related to the early work of (Nguyen & Widrow, 1989) in the deterministic case, and the SVG framework of (Heess et al., 2015) in the stochastic case.

Original: However, neither of these approaches incorporates a term penalizing the uncertainty of the forward model when training the policy network.
Revised: However, neither of these approaches incorporates a term penalizing the uncertainty of the forward model when training the policy network.

Original: The works of (McAllister & Rasmussen, 2016; Chua et al., 2018) also used model uncertainty estimates calculated using dropout in the context of model-based reinforcement learning, but used them for sampling trajectories during the forward prediction step.
Revised: The works of (McAllister & Rasmussen, 2016; Chua et al., 2018) also used model uncertainty estimates calculated using dropout in the context of model-based reinforcement learning, but used them for sampling trajectories during the forward prediction step.

Original: Namely, they applied different dropout masks to simulate different state trajectories which reﬂect the distribution over plausible models, which were then averaged to produce a cost estimate used to select an action.
Revised: Namely, they applied different dropout masks to simulate different state trajectories which reﬂect the distribution over plausible models, which were then averaged to produce a cost estimate used to select an action.

Original: Our model uncertainty penalty is related to the cost used in (Kahn et al., 2017), who used dropout and model ensembling to compute uncertainty estimates for a binary action-conditional collision detector for a ﬂying drone.
Revised: Our model uncertainty penalty is related to the cost used in (Kahn et al., 2017), who used dropout and model ensembling to compute uncertainty estimates for a binary action-conditional collision detector for a ﬂying drone.

Original: These estimates were then used to select actions out of a predeﬁned set which yielded a good tradeoff between speed, predicted chance of collision and uncertainty about the prediction.
Revised: These estimates were then used to select actions out of a predeﬁned set which yielded a good tradeoff between speed, predicted chance of collision and uncertainty about the prediction.

Original: In our work, we apply uncertainty estimates to the predicted high-dimensional states of a forward model at every time step, summarize them into a scalar, and backpropagate gradients through the unrolled forward model to then train a policy network by gradient descent.
Revised: In our work, we apply uncertainty estimates to the predicted high-dimensional states of a forward model at every time step, summarize them into a scalar, and backpropagate gradients through the unrolled forward model to then train a policy network by gradient descent.

Original: The work of (Depeweg et al., 2018) also proposed adding an uncertainty penalty when training paramaterized policies, but did so in the context of BNNs trained using α -divergences applied in low-dimensional settings, whereas we use variational autoencoders combined with dropout for high-dimensional video prediction.
Revised: The work of (Depeweg et al., 2018) also proposed adding an uncertainty penalty when training paramaterized policies, but did so in the context of BNNs trained using α -divergences applied in low-dimensional settings, whereas we use variational autoencoders combined with dropout for high-dimensional video prediction.

Original: α -BNNs can yield better uncertainty estimates than variational inference-based methods, which can underestimate model uncertainty by ﬁtting to a local mode of the exact posterior (Depeweg et al., 2016; Li & Gal, 2017).
Revised: α -BNNs can yield better uncertainty estimates than variational inference-based methods, which can underestimate model uncertainty by ﬁtting to a local mode of the exact posterior (Depeweg et al., 2016; Li & Gal, 2017).

Original: However, they also require computing multiple samples from the distribution over model weights when training the forward model, which increases memory requirements and limits scalability to high-dimensional settings such as the ones we consider here.
Revised: However, they also require computing multiple samples from the distribution over model weights when training the forward model, which increases memory requirements and limits scalability to high-dimensional settings such as the ones we consider here.

Original: The problem of covariate shift when executing a policy learned from observational data has been well-recognized in imitation learning. It was ﬁrst noted in the early work of (Pomerleau, 1991), and was shown in (Ross & Bagnell, 2010) to cause a regret bound which grows quadratically in the time horizon of the task.
Revised: The problem of covariate shift when executing a policy learned from observational data has been well-recognized in imitation learning (Pomerleau, 1991; Ross & Bagnell, 2010).

Original: The work of (Ross et al., 2011) proposed a method to efﬁciently use expert feedback if available, which has also been applied in the context of autonomous driving (Zhang & Cho, 2016).
Revised: The work of (Ross et al., 2011) proposed a method to efﬁciently use expert feedback (if available) to correct this shift, which has also been applied in the context of autonomous driving (Zhang & Cho, 2016).

Original: Our approach also addresses covariate shift, but does so without querying an expert.
Revised: Our approach also addresses covariate shift, but does so without querying an expert.

Original: Our MPER approach is related to the work of (Englert et al., 2013), who also performed imitation learning at the level of trajectories rather than individual actions.
Revised: Our MPER approach is related to the work of (Englert et al., 2013), who also performed imitation learning at the level of trajectories rather than individual actions.

Original: They did so in low-dimensional settings using Gaussian Processes, whereas our method uses an unrolled neural network representing the environment dynamics which can be applied to high-dimensional state representations. 
Revised: They did so in low-dimensional settings using Gaussian Processes, whereas our method uses an unrolled neural network representing the environment dynamics which can be applied to high-dimensional state representations. The

Original: The work of (Baram et al., 2017) also used a neural network dynamics model in the context of imitation learning, but did so in the interactive setting to minimize a loss produced by a discriminator network.
Revised: [Figure] work of (Baram et al., 2017) also used a neural network dynamics model in the context of imitation learning, but did so in the interactive setting to minimize a loss produced by a discriminator network.

Original: Several works have applied deep learning methods in the context of autonomous driving. The works of (Pomerleau, 1991; LeCun et al., 2006; Bojarski et al., 2016; Pan et al., 2017) used neural networks to control policies trained by imitation learning, while (Williams et al., 2017) learned models of the vehicle dynamics.
Revised: Several works have used deep learning models for autonomous driving, either to learn policies through imitation learning (Pomerleau, 1991; LeCun et al., 2006; Bojarski et al., 2016; Pan et al., 2017) or for modeling vehicle dynamics (Williams et al., 2017).

Original: These works focused on lane following or avoiding static obstacles in visually rich environments and did not consider settings with dense moving trafﬁc, which we focus on in this work.
Revised: These works focused on lane following or avoiding static obstacles in visually rich environments and did not consider settings with dense moving trafﬁc.

Original: The work of (Sadigh et al., 2016) developed a model of the interactions between the two drivers which was then used to plan actions in simple settings, using symbolic state representations.
Revised: The work of (Sadigh et al., 2016) developed a model of the interactions between the two drivers which was then used to plan actions in simple settings, using symbolic state representations.

Original: In our work, we consider the problem of learning driving policies in dense trafﬁc, using high-dimensional state representations which reﬂect the neighborhood of the ego car.[Figure]
Revised: In our work, we consider the problem of learning driving policies in dense trafﬁc, using high-dimensional state representations which reﬂect the neighborhood of the ego car.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now report experimental results.
Revised: We now report experimental results.

Original: We designed a deterministic and stochastic forward model to model the state and action representations described in Section 3, using convolutional layers to process the images i t and fully-connected layers to process the vectors u t and actions a t .
Revised: We designed a deterministic and stochastic forward model to model the state and action representations described in Section 3, using convolutional layers to process the images i t and fully-connected layers to process the vectors u t and actions a t .

Original: All model details can be found in Appendix C and training details can be found in Appendix D. Additional video results for the model predictions and learned policies can be found at the following URL: https://sites.google.com/view/model-predictive-driving/home .
Revised: All model details can be found in Appendix C and training details can be found in Appendix D. Code and additional video results for the model predictions and learned policies can be found at the following URL: https://sites.google.com/view/model-predictive-driving/home .

Original: 5. P REDICTION R ESULTS
Revised: 5. P REDICTION R ESULTS

Original: We ﬁrst generated predictions using both deterministic and stochastic forward models, shown in
Revised: We ﬁrst generated predictions using both deterministic and stochastic forward models, shown in

Original: Figure 5.
Revised: Figure 5.

Original: The deterministic model produces predictions which become increasingly blurry further into thefuture. Our stochastic model produces predictions which stay sharp far into the future.
Revised: The deterministic model produces predictions which become increasingly blurry, while the stochastic model produces predictions which stay sharp far into the future.

Original: By sampling different sequences of latent variables, different future scenarios are generated.
Revised: By sampling different sequences of latent variables, different future scenarios are generated.

Original: Note that the two sequences generated by the stochastic model are different from the ground truth future which occurs in the dataset.
Revised: Note that the two sequences generated by the stochastic model are different from the ground truth future which occurs in the dataset.

Original: This is normal as the future observed in the dataset is only one of many possible ones.
Revised: This is normal as the future observed in the dataset is only one of many possible ones.

Original: Additional video generations can be viewed at the URL.
Revised: Additional video generations can be viewed at the URL.

Original: 5. P OLICY E VALUATION R ESULTS
Revised: 5. P OLICY E VALUATION R ESULTS

Original: We evaluated policies using two measures: whether the controlled car reaches the end of the road segment without colliding into another car or driving off the road, and the distance travelled before the episode ends.
Revised: We evaluated policies using two measures: whether the controlled car reaches the end of the road segment without colliding into another car or driving off the road, and the distance travelled before the episode ends.

Original: Policies which collide quickly will travel shorter distances.
Revised: Policies which collide quickly will travel shorter distances.

Original: We compared our approach against several baselines which can also learn from observational data, which are described in Figure 6c.
Revised: We compared our approach against several baselines which can also learn from observational data, which are described in Figure 6c.

Original: Table 6a compares performance for the different methods, optimized over different rollout lengths.
Revised: Table 6a compares performance for the different methods, optimized over different rollout lengths.

Original: The 1-step imitation learner, SVG and VG all perform poorly,and do not beat the simple baseline of performing no action. Both MPUR and MPER signiﬁcantly outperform the other methods. Videos of the learned policies for both MPER and MPUR driving in the environment can be found at the URL. The policies learn effective behaviors such as braking, accelerating and turning to avoid other cars.
Revised: The 1-step imitation learner, SVG and VG all perform poorly,

Original: Figure 7 shows trajectories on the map for different methods.

Original: We see that the single-step imitation learner produces divergent trajectories which turn into other lanes, whereas the MPUR and MPER methods show trajectories which primarily stay within their lanes.

Original: MPUR becomes equivalent to VG in the deterministic setting if we remove the uncertainty penalty, and the large difference in performance shows that including this penalty is essential.

Original: Table 1 shows the average predicted policy cost and uncertainty cost of the two methods.

Original: VG produces much lower predicted policy cost, yet very high uncertainty cost.

Original: This indicates that the actions the policy

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Human
Revised: Human

Original: The human trajectories observed in the testing set, which are all collision-free.
Revised: The human trajectories observed in the testing set, which are all collision-free.

Original: No action
Revised: No action

Original: A policy which outputs an action of zero, maintaining constant speed and direction.
Revised: A policy which outputs an action of zero, maintaining constant speed and direction.

Original: 1-step IL
Revised: 1-step IL

Original: A policy network trained with single-step imitation learning.
Revised: A policy network trained with single-step imitation learning.

Original: SVG
Revised: SVG

Original: A policy network trained with stochastic value gradients.
Revised: A policy network trained with stochastic value gradients.

Original: This is the same setup as (Heess et al., 2015), with the difference that the agent does not interact 
Revised: This is the same setup as (Heess et al., 2015), with the difference that the agent does not interact with

Original: with the environment and learns from a ﬁxed observational dataset.
Revised: the environment and learns from a ﬁxed observational dataset.

Original: VG A policy trained with value gradients, using the deterministic forward model.
Revised: VG A policy trained with value gradients, using the deterministic forward model.

Original: This is similar to SVG, but does not involve latent variables.
Revised: This is similar to SVG, but does not involve latent variables.

Original: MPUR
Revised: MPUR

Original: A policy trained with MPUR , using a deterministic or stochastic model.
Revised: A policy trained with MPUR , using a deterministic or stochastic model.

Original: A cost term is included to penalize the uncertainty of the dynamics model.
Revised: A cost term is included to penalize the uncertainty of the dynamics model.

Original: MPER
Revised: MPER

Original: A policy trained with MPER , using a deterministic or stochastic model.
Revised: A policy trained with MPER , using a deterministic or stochastic model.

Original: The policy is trained to match expert trajectories from the training set.
Revised: The policy is trained to match expert trajectories from the training set.

Original: [Figure]
Revised: [Figure]

Revised: and do not beat the simple baseline of performing no action.

Revised: Both MPUR and MPER signiﬁcantly outperform the other methods.

Revised: Videos of the learned policies for both MPER and MPUR driving in the environment can be found at the URL.

Revised: The policies learn effective behaviors such as braking, accelerating and turning to avoid other cars.

Revised: Figure 7 shows trajectories on the map for different methods.

Revised: We see that the single-step imitation learner produces divergent trajectories which turn into other lanes, whereas the MPUR and MPER methods show trajectories which primarily stay within their lanes.

Revised: MPUR becomes equivalent to VG in the deterministic setting if we remove the uncertainty penalty, and the large difference in performance shows that including this penalty is essential.

Revised: Table 1 shows the average predicted policy cost and uncertainty cost of the two methods.

Revised: VG produces much

Original:  produces induce a distribution over states which the forward model is highly uncertain about.
Revised: [Figure] [Table] lower predicted policy cost, yet very high uncertainty cost. This indicates that the actions the policy produces induce a distribution over states which the forward model is highly uncertain about.

Original: The policy trained with MPUR produces higher policy cost estimates, but lower uncertainty cost, and performs much better when executed in the environment.
Revised: The policy trained with MPUR produces higher policy cost estimates, but lower uncertainty cost, and performs much better when executed in the environment.

Original: The stochastic model trained with a standard Gaussian posterior yields limited improvement over the deterministic model.
Revised: The stochastic model trained with a standard Gaussian posterior yields limited improvement over the deterministic model.

Original: However, the stochastic model trained with the z -dropout parameterization yields a signiﬁcant improvement.
Revised: However, the stochastic model trained with the z -dropout parameterization yields a signiﬁcant improvement.

Original: Comparisons of action-conditional predictions using both models can be seen at the URL.
Revised: Comparisons of action-conditional predictions using both models can be seen at the URL.

Original: Thestochastic model which uses the standard posterior is less responsive to input actions than the model trained with the modiﬁed posterior, which likely accounts for their difference in performance.This lack of responsiveness is especially pronounced when the model is given latent variables inferred by the posterior network, as opposed to latent variables which are [Figure] [Table]
Revised: The standard model is less responsive to input actions than the model trained with z -dropout, which likely accounts for their difference in performance.

Original: sampled from the prior. We also trained a policy network using MPUR with inferred rather than sampled latent variables, and observed a large drop in performance (see Appendix E). A possible explanation is that the stochastic forward model uses the latent variables to encodesome factors of variation of the output which are in fact due to the actions.The sequence of latent variables sampled from the prior are independent, which may cause these effects to partially cancel each other over the sequence.
Revised: We hypothesize that the standard model uses its latent variables to encode factors of variation in the output which are in fact due to the actions.

Original: However, the latent variables inferred from a sequence of states in the training set are highly dependent, and together they may explain away the effects of the actions.

Original: Our modiﬁed posterior distribution discourages the model from encoding action information in the latent variables, since factors of variation in the outputwhich are due to the actions cannot be encoded in the latent variables when they are sampled from the prior, and the loss can better be lowered by predicting them from the actions instead. 
Revised: Using z -dropout discourages this since information in the output cannot be encoded in the latent variables the times they are sampled from the prior, and the loss can better be lowered by predicting the outputs from the actions instead. Please see Appendix E for additional experiments and discussion of this phenomenon.

Original: Figure 6b shows the performance of MPUR and MPER for different rollout lengths.
Revised: Figure 6b shows the performance of MPUR and MPER for different rollout lengths.

Original: All methods see their performance improve dramatically as we increase the rollout length, which encourages the distribution of states the policy induces and the training distribution to match over longer time horizons.
Revised: All methods see their performance improve dramatically as we increase the rollout length, which encourages the distribution of states the policy induces and the training distribution to match over longer time horizons.

Original: We also see that the stochastic model with z -dropout outperforms the standard stochastic model as well as the deterministic model over most rollout lengths.
Revised: We also see that the stochastic model with z -dropout outperforms the standard stochastic model as well as the deterministic model over most rollout lengths.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, wehave proposed a general approach for learning policies from purely observational data.
Revised: In this work, we proposed a general approach for learning policies from purely observational data.

Original: The key elements are: i) a learned stochastic dynamics model, which is used to optimize a policy cost over multiple time steps, ii) an uncertainty term which penalizes the divergence of the trajectories induced by the policy from the manifold it was trained on, and iii) a modiﬁed posterior distribution which keeps the stochasticdynamics model responsive to input actions.
Revised: The key elements are: i) a learned stochastic dynamics model, which is used to optimize a policy cost over multiple time steps, ii) an uncertainty term which penalizes the divergence of the trajectories induced by the policy from the manifold it was trained on, and iii) a modiﬁed posterior distribution which keeps the stochastic model responsive to input actions.

Original: We have applied this approach to a large observational dataset of real-world trafﬁc recordings, and shown it can effectively learn policies for navigating in dense trafﬁc, which outperform other approaches which learn from observational data.
Revised: We have applied this approach to a large observational dataset of real-world trafﬁc recordings, and shown it can effectively learn policies for navigating in dense trafﬁc, which outperform other approaches which learn from observational data.

Original: However, there is still a sizeable gap between the performance of our learned policies and human performance.
Revised: However, there is still a sizeable gap between the performance of our learned policies and human performance.

Original: We release both our dataset and environment, and encourage further research in this area to help narrow this gap.
Revised: We release both our dataset and environment, and encourage further research in this area to help narrow this gap.

Original: We also believe this provides a useful setting for evaluating generative models in terms of their ability to produce good policies.
Revised: We also believe this provides a useful setting for evaluating generative models in terms of their ability to produce good policies.

Original: An interesting next step would be to optimize actions or policies to produce more complex and targeted behaviors, such as changing to a speciﬁed lane while avoiding other cars.

Original: Finally, our approach is general and could potentially be applied to many other settings where interactions with the environment are expensive or unfeasible, but observational data is plentiful.
Revised: Finally, our approach is general and could potentially be applied to many other settings where interactions with the environment are expensive or unfeasible, but observational data is plentiful.

Original: A D ATASET AND P LANNING E NVIRONMENT
Revised: A D ATASET AND P LANNING E NVIRONMENT

Original: To begin with we describe the details and preparation of the dataset and planning environment which we used, which are summarized in Figure 4.
Revised: To begin with we describe the details and preparation of the dataset and planning environment which we used, which are summarized in Figure 4.

Original: The Next Generation Simulation program’s Interstate 80 (NGSIM I-80) dataset (Halkias & Colyar, 2006) consists of 45 minutes of recordings made of a stretch of highway in the San Francisco Bay Area by cameras mounted on a 30-story building overlooking the highway.
Revised: The Next Generation Simulation program’s Interstate 80 (NGSIM I-80) dataset (Halkias & Colyar, 2006) consists of 45 minutes of recordings made of a stretch of highway in the San Francisco Bay Area by cameras mounted on a 30-story building overlooking the highway.

Original: The recorded area includes six freeway lanes (including a high-occupancy vehicle lane) and an onramp.
Revised: The recorded area includes six freeway lanes (including a high-occupancy vehicle lane) and an onramp.

Original: The driver behavior is complex and includes sudden accelerations, lane changes and merges which are difﬁcult to predict; as such the dataset has high aleatoric uncertainty.
Revised: The driver behavior is complex and includes sudden accelerations, lane changes and merges which are difﬁcult to predict; as such the dataset has high aleatoric uncertainty.

Original: There are three time segments, each of 15 minutes, taken at different times of day which capture the transition between uncongested and congested peak period conditions.
Revised: There are three time segments, each of 15 minutes, taken at different times of day which capture the transition between uncongested and congested peak period conditions.

Original: After recording, a viewpoint transformation is applied to rectify the perspective, and vehicles are identiﬁed and tracked throughout the video; additionally, their size is inferred.
Revised: After recording, a viewpoint transformation is applied to rectify the perspective, and vehicles are identiﬁed and tracked throughout the video; additionally, their size is inferred.

Original: This yields a total 5596 car trajectories, represented as sequences of coordinates { x t , y t } .
Revised: This yields a total 5596 car trajectories, represented as sequences of coordinates { x t , y t } .

Original: We split these trajectories into training ( 80% ), validation ( 10% ) and testing sets ( 10% ).
Revised: We split these trajectories into training ( 80% ), validation ( 10% ) and testing sets ( 10% ).

Original: We then applied additional preprocessing to obtain suitable representations for learning a predictive model.
Revised: We then applied additional preprocessing to obtain suitable representations for learning a predictive model.

Original: Speciﬁcally, we extracted the following: i) a state representation for each car at each time step s t , which encodes the necessary information to choose an action to take, ii) an action a t which represents the action of the driver, and iii) a cost c t , which associates a quality measure to each state.
Revised: Speciﬁcally, we extracted the following: i) a state representation for each car at each time step s t , which encodes the necessary information to choose an action to take, ii) an action a t which represents the action of the driver, and iii) a cost c t , which associates a quality measure to each state.

Original: We describe each of these below.
Revised: We describe each of these below.

Original: State representation : Our state representation consists of two components: an image representing the neighborhood of the car, and a vector representing its current position and velocity.
Revised: State representation : Our state representation consists of two components: an image representing the neighborhood of the car, and a vector representing its current position and velocity.

Original: For the images, we rendered images centered around each car which encoded both the lane emplacements and the locations of other cars.
Revised: For the images, we rendered images centered around each car which encoded both the lane emplacements and the locations of other cars.

Original: Each image has 3 channels: the ﬁrst (red) encodes the lane markings, the second (green) encodes the locations of neighboring cars, which are represented as rectangles reﬂecting the dimensions of each car, and the third channel (blue) represents the ego car, also scaled to the correct dimensions.
Revised: Each image has 3 channels: the ﬁrst (red) encodes the lane markings, the second (green) encodes the locations of neighboring cars, which are represented as rectangles reﬂecting the dimensions of each car, and the third channel (blue) represents the ego car, also scaled to the correct dimensions.

Original: All images have dimensions 3 × 117 × 24 , and are denoted by i t .
Revised: All images have dimensions 3 × 117 × 24 , and are denoted by i t .

Original: 3 Two examples are shown in Figure 8.
Revised: 4 Two examples are shown in Figure 8.

Original: We also computed vectors u t =
Revised: We also computed vectors u t =

Original: ( p t , ∆ p t ) , where p t = ( x t , y t ) is the position at time t and ∆ p t =
Revised: ( p t , ∆ p t ) , where p t = ( x t , y t ) is the position at time t and ∆ p t =

Original: ( x t +1 − x t , y t +1 − y t ) is the velocity.
Revised: ( x t +1 − x t , y t +1 − y t ) is the velocity.

Original: Action representation : Each action vector a t consists of two components: an acceleration (which can be positive or negative) which reﬂects the change in speed, and a change in angle.
Revised: Action representation : Each action vector a t consists of two components: an acceleration (which can be positive or negative) which reﬂects the change in speed, and a change in angle.

Original: The acceleration at a given time step is computed by taking the difference between two consecutive speeds, while the change in angle is computed by projecting the change in speed along its orthogonal direction: 
Revised: The acceleration at a given time step is computed by taking the difference between two consecutive speeds, while the change in angle is computed by projecting the change in speed along its orthogonal direction: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The higher the speed, the longer the safety distance required to maintain low cost
Revised: The higher the speed, the longer the safety distance required to maintain low cost.

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Cost : Our cost function has two terms: a proximity cost and a lane cost.
Revised: Cost : Our cost function has two terms: a proximity cost and a lane cost.

Original: The proximity cost reﬂects how close the ego car is to neighboring cars, and is computed using a mask in pixel space whose width is equal to the width of a lane and whose height depends on the speed of the car.
Revised: The proximity cost reﬂects how close the ego car is to neighboring cars, and is computed using a mask in pixel space whose width is equal to the width of a lane and whose height depends on the speed of the car.

Original: Two examples are shown in Figure 8.
Revised: Two examples are shown in Figure 8.

Original: This mask is pointwise multiplied with the green channel, and the maximum value is taken to produce a scalar cost.
Revised: This mask is pointwise multiplied with the green channel, and the maximum value is taken to produce a scalar cost.

Original: The lane cost uses a similar mask ﬁxed to the size of the car, and is similarly multiplied with the red channel, thus measuring the car’s overlap with the lane.
Revised: The lane cost uses a similar mask ﬁxed to the size of the car, and is similarly multiplied with the red channel, thus measuring the car’s overlap with the lane.

Original: Both of these operations are differentiable so that we can backpropagate gradients with respect to these costs through images predicted by a forward model.
Revised: Both of these operations are differentiable so that we can backpropagate gradients with respect to these costs through images predicted by a forward model.

Original: This preprocessing yields a set of state-action pairs ( s t , a t ) (with s t
Revised: This preprocessing yields a set of state-action pairs ( s t , a t ) (with s t

Original: = ( i t , u t ) ) for each car, which constitute the dataset we used for training our prediction model.
Revised: = ( i t , u t ) ) for each car, which constitute the dataset we used for training our prediction model.

Original: We then use the cost function to optimize action sequences at planning time, using different methods which we describe in Section 2.2.
Revised: We then use the cost function to optimize action sequences at planning time, using different methods which we describe in Section 2.2.

Original: We now describe how we adapted this dataset to be used as an environment to evaluate planning methods.
Revised: We now describe how we adapted this dataset to be used as an environment to evaluate planning methods.

Original: Building an environment for evaluating policies for autonomous driving is not obvious as it suffers from a cold-start problem.
Revised: Building an environment for evaluating policies for autonomous driving is not obvious as it suffers from a cold-start problem.

Original: Precisely measuring the performance of a given driving policy would require it to be evaluated in an environment where all other cars follow policies which accurately reﬂect human behavior.
Revised: Precisely measuring the performance of a given driving policy would require it to be evaluated in an environment where all other cars follow policies which accurately reﬂect human behavior.

Original: This involves reacting appropriately both to other cars in the environment as well as the car being controlled by the policy being evaluated.
Revised: This involves reacting appropriately both to other cars in the environment as well as the car being controlled by the policy being evaluated.

Original: However, constructing such an environment is not possible as it would require us to already have access to a policy which drives as humans do, which in some sense is our goal in the ﬁrst place.
Revised: However, constructing such an environment is not possible as it would require us to already have access to a policy which drives as humans do, which in some sense is our goal in the ﬁrst place.

Original: One could hand-code a driving policy to control the other cars in the environment, however is it not clear how to do so in a way which accurately reﬂects the diverse and often unpredictable nature of human driving.
Revised: One could hand-code a driving policy to control the other cars in the environment, however is it not clear how to do so in a way which accurately reﬂects the diverse and often unpredictable nature of human driving.

Original: We adopt a different approach where we let all other cars in the environment follow their trajectories in the dataset, while controlling one car with the policy we seek to evaluate.
Revised: We adopt a different approach where we let all other cars in the environment follow their trajectories in the dataset, while controlling one car with the policy we seek to evaluate.

Original: The trajectory of the controlled car is updated as a function of the actions output by the policy, while the trajectories of the other cars remain ﬁxed.
Revised: The trajectory of the controlled car is updated as a function of the actions output by the policy, while the trajectories of the other cars remain ﬁxed.

Original: If the controlled car collides with another car, this is recorded and observation = env.reset() while not done: [Figure] (observation)
Revised: If the controlled car collides with another car, this is recorded and observation = env.reset() while not done: action = policy(observation)

Original: [Figure]
Revised: [Figure]

Original: the episode ends.
Revised: the episode ends.

Original: This approach has the advantage that all other cars in the environment maintain behavior which is close to human-like.
Revised: This approach has the advantage that all other cars in the environment maintain behavior which is close to human-like.

Original: The one difference with true human behavior is that the other cars do not react to the car being controlled or try to avoid it, which may cause crashes which would not occur in real life.
Revised: The one difference with true human behavior is that the other cars do not react to the car being controlled or try to avoid it, which may cause crashes which would not occur in real life.

Original: The driving task is thus possibly made more challenging than in a true environment, which we believe is preferable to using a hand-coded policy.
Revised: The driving task is thus possibly made more challenging than in a true environment, which we believe is preferable to using a hand-coded policy.

Original: The interface is set up the same way as environments in OpenAI Gym (Brockman et al., 2016), and can be accessed with a few lines of Python code, as shown in Figure 9.
Revised: The interface is set up the same way as environments in OpenAI Gym (Brockman et al., 2016), and can be accessed with a few lines of Python code, as shown in Figure 9.

Original: B B AYESIAN N EURAL N ETWORK F ORMULATION D ETAILS
Revised: B B AYESIAN N EURAL N ETWORK F ORMULATION D ETAILS

Original: Our MPUR approach can be viewed as training a Bayesian neural network (BNN) (Neal, 1995) with latent variables using variational inference (Jordan et al., 1999; Kingma & Welling, 2013).
Revised: Our MPUR approach can be viewed as training a Bayesian neural network (BNN) (Neal, 1995) with latent variables using variational inference (Jordan et al., 1999; Kingma & Welling, 2013).

Original: The distribution over model predictions for s t +1 is given by:
Revised: The distribution over model predictions for s t +1 is given by:

Original: [Equation]
Revised: [Equation]

Original: The distribution p ( θ, z |D ) reﬂects the posterior over model weights and latent variables given the data, and is intractable to evaluate.
Revised: The distribution p ( θ, z |D ) reﬂects the posterior over model weights and latent variables given the data, and is intractable to evaluate.

Original: We instead approximate it with the variational distribution q parameterized by η :
Revised: We instead approximate it with the variational distribution q parameterized by η :

Original: [Equation]
Revised: [Equation]

Original: Here q φ represents a distribution over latent variables parameterized by φ , which could be a diagonal Gaussian or the mixture distribution described in Section 2.1.
Revised: Here q φ represents a distribution over latent variables parameterized by φ , which could be a diagonal Gaussian or the mixture distribution described in Section 2.1.

Original: The distribution q θ ∗ is the dropout approximating distribution over model parameters described in (Gal & Ghahramani, 2016)
Revised: The distribution q θ ∗ is the dropout approximating distribution over model parameters described in (Gal & Ghahramani, 2016)

Original: (Section 3.2 of Supplement).
Revised: (Section 3.2 of Supplement).

Original: This deﬁnes a mixture of two Gaussians with small variances over each row of each weight matrix in the forward model, with the mean of one Gaussian ﬁxed at zero.
Revised: This deﬁnes a mixture of two Gaussians with small variances over each row of each weight matrix in the forward model, with the mean of one Gaussian ﬁxed at zero.

Original: The parameters of this distribution are the model weights, and samples can be drawn by applying different dropout masks.
Revised: The parameters of this distribution are the model weights, and samples can be drawn by applying different dropout masks.

Original: The parameters of the variational distribution are thus η = { θ ∗ , φ } , and can be optimized by maximizing the evidence lower bound, which is equivalent to minimizing the
Revised: The parameters of the variational distribution are thus η = { θ ∗ , φ } , and can be optimized by maximizing the evidence lower bound, which is equivalent to minimizing the

Original: Kullback-Leibler divergence between the approximate posterior and true posterior:
Revised: Kullback-Leibler divergence between the approximate posterior and true posterior:

Original: [Equation]
Revised: [Equation]

Original: Here p 

Revised: Here p

Original: ( z, θ ) =
Revised: ( z, θ ) =

Original: p 0 ( z ) · p 0 ( θ ) represents a prior over latent variables and model parameters.
Revised: p 0 ( z ) · p 0 ( θ ) represents a prior over latent variables and model parameters.

Original: By applying the chain rule for KL divergences together with the fact that z and θ are independent, we obtain:
Revised: By applying the chain rule for KL divergences together with the fact that z and θ are independent, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Setting both Gaussians in p 0 ( θ ) to have zero mean, the second KL term becomes equivalent to scaled (cid:96) 2 regularization on the model parameters, and can be set arbitrarily small (Section 4.2 in
Revised: Setting both Gaussians in p 0 ( θ ) to have zero mean, the second KL term becomes equivalent to scaled (cid:96) 2 regularization on the model parameters, and can be set arbitrarily small (Section 4.2 in

Original: Supplement of (Gal & Ghahramani, 2016)).
Revised: Supplement of (Gal & Ghahramani, 2016)).

Original: Ignoring this term and approximating the integral in
Revised: Ignoring this term and approximating the integral in

Original: Equation 3 using a single sample, we obtain:
Revised: Equation 3 using a single sample, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where ¯ θ ∼ q θ ∗ ( θ ) and z ∼ q φ ( s 1: t , s t +1 )) .
Revised: where ¯ θ ∼ q θ ∗ ( θ ) and z ∼ q φ ( s 1: t , s t +1 )) .

Original: Assuming a diagonal Gaussian likelihood on the outputs with constant variance 1 /β , we can rewrite this as:
Revised: Assuming a diagonal Gaussian likelihood on the outputs with constant variance 1 /β , we can rewrite this as:

Original: [Equation]
Revised: [Equation]

Original: Multiplying by β does not change the maximum.
Revised: Multiplying by β does not change the maximum.

Original: We now see that maximizing this quantity is equivalent to minimizing our loss term in Equation 1, i.e. training a variational autoencoder with dropout.
Revised: We now see that maximizing this quantity is equivalent to minimizing our loss term in Equation 1, i.e. training a variational autoencoder with dropout.

Original: C M ODEL D ETAILS
Revised: C M ODEL D ETAILS

Original: The architecture of our forward model consists of four neural networks: a state encoder f enc , an action encoder f act , a decoder f dec , and the posterior network f φ .
Revised: The architecture of our forward model consists of four neural networks: a state encoder f enc , an action encoder f act , a decoder f dec , and the posterior network f φ .

Original: At every time step, the state encoder takes as input the concatenation of 20 previous states, each of which consists of a context image i t and a 4-dimensional vector u t encoding the car’s position and velocity.
Revised: At every time step, the state encoder takes as input the concatenation of 20 previous states, each of which consists of a context image i t and a 4-dimensional vector u t encoding the car’s position and velocity.

Original: The images i t − 20 , ..., i t are run through a 3-layer convolutional network with 64-128-256 feature maps, and the vectors u t − 20 , ..., u t are run through a 2-layer fully connected network with 256 hidden units, whose ﬁnal layers contain the same number of hidden units as the number of elements in the output of the convolutional network (we will call this number n H ).
Revised: The images i t − 20 , ..., i t are run through a 3-layer convolutional network with 64-128-256 feature maps, and the vectors u t − 20 , ..., u t are run through a 2-layer fully connected network with 256 hidden units, whose ﬁnal layers contain the same number of hidden units as the number of elements in the output of the convolutional network (we will call this number n H ).

Original: The posterior network takes the same input as the encoder network, as well as the the ground truth state s t +1 , and maps them to a distribution over latent variables, from which one sample z t is drawn.
Revised: The posterior network takes the same input as the encoder network, as well as the the ground truth state s t +1 , and maps them to a distribution over latent variables, from which one sample z t is drawn.

Original: This is then passed through an expansion layer which maps it to a representation of size n H .
Revised: This is then passed through an expansion layer which maps it to a representation of size n H .

Original: The action encoder, which is a 2-layer fully-connected network, takes as input a 2-dimensional action a t encoding the car’s acceleration and change in steering angle, and also maps it to a representation of size n H .
Revised: The action encoder, which is a 2-layer fully-connected network, takes as input a 2-dimensional action a t encoding the car’s acceleration and change in steering angle, and also maps it to a representation of size n H .

Original: The representations of the input states, latent variable, and action, which are all now the same size, are combined via addition.
Revised: The representations of the input states, latent variable, and action, which are all now the same size, are combined via addition.

Original: The result is then run through a deconvolutional network with 256-128-64 feature maps, which produces a prediction for the next image i t +1 , and a 2-layer fully-connected network (with 256 hidden units) which produces a prediction for the next state vector u t +1 .
Revised: The result is then run through a deconvolutional network with 256-128-64 feature maps, which produces a prediction for the next image i t +1 , and a 2-layer fully-connected network (with 256 hidden units) which produces a prediction for the next state vector u t +1 .

Original: These are illustrated in Figure C.
Revised: These are illustrated in Figure C.

Original: The speciﬁc updates of the stochastic forward model are given by:
Revised: The speciﬁc updates of the stochastic forward model are given by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The per-sample loss is given by:
Revised: The per-sample loss is given by:

Original: [Equation]
Revised: [Equation]

Original: [Equation] We also train a cost predictor which takes as input the states predicted by the forward model and produces a two-dimensional output (one output for the proximity cost, and one output for the lane cost).
Revised: We also train a cost predictor which takes as input the states predicted by the forward model and produces a two-dimensional output (one output for the proximity cost, and one output for the lane cost).

Original: This consists of a 3-layer encoder followed by a two-layer fully connected network with sigmoid non-linearities as the end to constrain the values between 0 and 1 .
Revised: This consists of a 3-layer encoder followed by a two-layer fully connected network with sigmoid non-linearities as the end to constrain the values between 0 and 1 .

Original: 2-layer MLP + 2-layer MLP 3-layer ConvNet + (a) f enc 2-layer MLP + 2-layer MLP 3-layer DeConvNet (b) f dec 2-layer MLP + 2-layer MLP 3-layer DeConvNet
Revised: 2-layer MLP + 2-layer MLP 3-layer ConvNet + (a) f enc 2-layer MLP + 2-layer MLP 3-layer DeConvNet (b) f dec 2-layer MLP + 2-layer MLP 3-layer DeConvNet

Original: [Figure]
Revised: [Figure]

Original: D T RAINING D ETAILS D. F ORWARD M ODEL
Revised: D T RAINING D ETAILS D. F ORWARD M ODEL

Original: We trained our prediction model in deterministic mode ( p = 0 ) for 200,000 updates, followed by another 200,000 updates in stochastic mode.
Revised: We trained our prediction model in deterministic mode ( p = 0 ) for 200,000 updates, followed by another 200,000 updates in stochastic mode.

Original: We save the model after training in deterministic mode and treat it as a deterministic baseline.
Revised: We save the model after training in deterministic mode and treat it as a deterministic baseline.

Original: Our model was trained using Adam (Kingma & Ba, 2014) with learning rate 0.0001 and minibatches of size 64, unrolled for 20 time steps, and with dropout ( p dropout = 0 . 1 ) at every layer, which was necessary for computing the epistemic uncertainty cost when training the policy network.
Revised: Our model was trained using Adam (Kingma & Ba, 2014) with learning rate 0.0001 and minibatches of size 64, unrolled for 20 time steps, and with dropout ( p dropout = 0 . 1 ) at every layer, which was necessary for computing the epistemic uncertainty cost when training the policy network.

Original: D. P OLICY M ODELS
Revised: D. P OLICY M ODELS

Original: All cars are initialized at the beginning of the road segment with the initial speed they were driving at in the dataset, and then are controlled by the policy being measured.
Revised: All cars are initialized at the beginning of the road segment with the initial speed they were driving at in the dataset, and then are controlled by the policy being measured.

Original: We only report performance for cars in the testing trajectories, which were not used when training the forward model or policy network.
Revised: We only report performance for cars in the testing trajectories, which were not used when training the forward model or policy network.

Original: All policy networks have the same architecture: a 3-layer ConvNet with feature maps of size 64128-256 (which takes 20 consecutive frames as input), followed by 3 fully-connected layers withhidden units each, with the last layer outputting the parameters of a 2D Gaussian distribution from which the action is sampled.
Revised: All policy networks have the same architecture: a 3-layer ConvNet with feature maps of size 64128-256 (which takes 20 consecutive frames as input), followed by 3 fully-connected layers withhidden units each, with the last layer outputting the parameters of a 2D Gaussian distribution from which the action is sampled.

Original: All policy networks are trained with Adam with learning rate 0.0001.
Revised: All policy networks are trained with Adam with learning rate 0.0001.

Original: The MPER and MPUR policies are trained by backpropagation through the unrolled forward model using the reparamaterization trick (Kingma & Welling, 2013).
Revised: The MPER and MPUR policies are trained by backpropagation through the unrolled forward model using the reparamaterization trick (Kingma & Welling, 2013).

Original: The single-step imitation learner is trained to directly minimize the negative log-likelihood of the ground truth action in the dataset under the parameters output by the policy network.
Revised: The single-step imitation learner is trained to directly minimize the negative log-likelihood of the ground truth action in the dataset under the parameters output by the policy network.

Original: All MPUR policies use a weighting of λ = 0 .for
Revised: All MPUR policies use a weighting of λ = 0 .for

Original: the uncertainty cost.
Revised: the uncertainty cost.

Original: Additionally, we detach gradients of predicted costs coming into the states, to prevent the policy from lowering the predicted cost (which is speed-dependent) by slowing down.
Revised: Additionally, we detach gradients of predicted costs coming into the states, to prevent the policy from lowering the predicted cost (which is speed-dependent) by slowing down.

Original: We found that not doing this can result in the policy slowing excessively, and then attempting to speed up only when another car gets close.
Revised: We found that not doing this can result in the policy slowing excessively, and then attempting to speed up only when another car gets close.

Original: We repeat the policy training with 3 random seeds for each method.
Revised: We repeat the policy training with 3 random seeds for each method.

Original: The policy cost which we minimize for VG, SVG and MPUR is given by:
Revised: The policy cost which we minimize for VG, SVG and MPUR is given by:

Original: [Equation]
Revised: [Equation]

Original: where C proximity and C lane are the proximity and lane costs described in Section 3.
Revised: where C proximity and C lane are the proximity and lane costs described in Section 3.

Original: This puts a higher priority on avoiding other cars while still encouraging the policy to stay within the lanes.
Revised: This puts a higher priority on avoiding other cars while still encouraging the policy to stay within the lanes.

Original: MPUR additionally minimizes U , the model uncertainty cost described in Section 2.2.
Revised: MPUR additionally minimizes U , the model uncertainty cost described in Section 2.2.

Original: When computing the uncertainty cost, to compensate for differences in baseline uncertainty across different rollout lengths, we normalize by the empirical mean and variance for every rollout length t of the forward model over the training set, to obtain µ tU and σ tU .
Revised: When computing the uncertainty cost, to compensate for differences in baseline uncertainty across different rollout lengths, we normalize by the empirical mean and variance for every rollout length t of the forward model over the training set, to obtain µ tU and σ tU .

Original: We then deﬁne our uncertainty cost as follows:
Revised: We then deﬁne our uncertainty cost as follows:

Original: [Equation]
Revised: [Equation]

Original: If the uncertainty estimate is lower than the mean uncertainty estimate on the training set for this rollout length, this loss will be zero.
Revised: If the uncertainty estimate is lower than the mean uncertainty estimate on the training set for this rollout length, this loss will be zero.

Original: These are cases where the model prediction is within normal uncertainty ranges.
Revised: These are cases where the model prediction is within normal uncertainty ranges.

Original: If the uncertainty estimate is higher, this loss exerts a pull to change the action so that the future state will be predicted with higher conﬁdence by the forward model.
Revised: If the uncertainty estimate is higher, this loss exerts a pull to change the action so that the future state will be predicted with higher conﬁdence by the forward model.

Original: E A CTION S ENSITIVITY E XPERIMENTS
Revised: E A CTION S ENSITIVITY E XPERIMENTS

Original: Here we report additional experiments measuring the effect of sampling from the prior compared to the posterior.
Revised: We found the lack of responsiveness of the stochastic forward model to be especially pronounced when it is given a sequence of latent variables inferred from the current training sequence by the posterior network, instead of a sequence of latent variables sampled from the prior (intuitively, using the inferred sequence corresponds to using the future from the dataset, while using a sampled sequence corresponds to a different future). One reason for this difference in responsiveness may be that in the ﬁrst case, the latent variables are highly dependent whereas in the second they are independent. If action information is encoded in the latent variables, the effects on the output may partially cancel each other out when the latents are independent. However, when they are highly dependent, together they may explain away the effects of the actions input to the forward model.

Revised: The table below shows the performance of MPUR policies learned using inferred and sampled latent variables.

Original: We seethat there is a large drop in performance when sampling latent variables from the posterior instead of the prior.
Revised: We see a large drop in performance when using the inferred latent variables.

Original: This is consistent with the videos at the URL, which show that the forward model is less sensitive to actions when the latent variables are sampled from the posterior instead of the prior.
Revised: This is consistent with the videos at the URL, which show that the forward model is less sensitive to actions when the latent variables are sampled from the posterior instead of the prior.

Original: Note that the z -dropout parameterization reduces this problem somewhat
Revised: Note that the z -dropout parameterization reduces this problem somewhat.

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: F M ODEL -F REE R ESULTS

Revised: Although the goal of this work is to learn policies with no environment interaction, for completeness we also report results of running Proximal Policy Optimization (PPO)

Revised: (Schulman et al., 2017), a state-of-the-art model-free algorithm which learns through interacting with its environment.

Revised: We used the OpenAI Baselines implementation (Brockman et al., 2016) with the same policy network architecture as for the other methods, and set the reward to be the negative policy cost deﬁned in equation 9.

Revised: We measure both ﬁnal performance and cumulative regret, using the success rate as a reward.

Revised: Cumulative regret is a measure often used in online learning which represents the difference between the agent’s accumulated reward and the accumulated reward which would have been obtained by following the optimal policy, which we take to be human performance here.

Revised: Speciﬁcally, the regret at epoch T is given by:

Revised: [Equation]

Revised: where R ∗ represents the reward obtained by following the optimal policy and R t is the reward obtained by following the policy at epoch t .

Revised: Unlike ﬁnal performance, regret also reﬂects poor decisions made during the learning process.

Revised: Results are shown in Figure 11.

Revised: PPO obtains slightly higher ﬁnal performance than MPUR, but also incurs higher regret as it executes poor policies in the environment during the early stages of learning.

Revised: In contrast, MPUR learns through observational data and already has a good policy in place when it begins interacting with the environment.

Revised: [Figure]

Revised: Success Rate

Revised: MPUR + z -dropout 171 .

Revised: 2 ± 4 .74 .

Revised: 8 ±

Revised: [Table]

Revised: [Figure]

Revised: PPO 181 .

Revised: [Figure]

Revised: ±

Revised: [Table]

Revised: [Table]

Revised: ±

Revised: [Table]

Revised: [Figure]