Original: D ROPOUT ’ S D REAM L AND : G ENERALIZATION FROM L EARNED S IMULATORS TO R EALITY
Revised: D ROPOUT ’ S D REAM L AND : G ENERALIZATION FROM L EARNED S IMULATORS TO R EALITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A World Model is a generative model used to simulate an environment.
Revised: A World Model is a generative model used to simulate an environment.

Original: World
Revised: World

Original: Models have proven capable of learning spatial and temporal representations of
Revised: Models have proven capable of learning spatial and temporal representations of

Original: Reinforcement Learning environments.
Revised: Reinforcement Learning environments.

Original: In some cases, a World Model offers an agent the opportunity to learn entirely inside of its own dream environment.
Revised: In some cases, a World Model offers an agent the opportunity to learn entirely inside of its own dream environment.

Original: In this work we explore improving the generalization capabilities from dream environments to reality (Dream2Real).
Revised: In this work we explore improving the generalization capabilities from dream environments to reality (Dream2Real).

Original: We present a general approach to improve a controller’s ability to transfer from a neural network dream environment to reality at little additional cost.
Revised: We present a general approach to improve a controller’s ability to transfer from a neural network dream environment to reality at little additional cost.

Original: These improvements are gained by drawing on inspiration from domain randomization, where the basic idea is to randomize as much of a simulator as possible without fundamentally changing the task at hand.
Revised: These improvements are gained by drawing on inspiration from domain randomization, where the basic idea is to randomize as much of a simulator as possible without fundamentally changing the task at hand.

Original: Generally, domain randomization assumes access to a pre-built simulator with conﬁgurable parameters but oftentimes this is not available.
Revised: Generally, domain randomization assumes access to a pre-built simulator with conﬁgurable parameters but oftentimes this is not available.

Original: By training the World Model using dropout, the dream environment is capable of creating a nearly inﬁnite number of different dream environments.
Revised: By training the World Model using dropout, the dream environment is capable of creating a nearly inﬁnite number of different dream environments.

Original: Our experimental results show that Dropout’s
Revised: Our experimental results show that Dropout’s

Original: Dream Land is an effective technique to bridge the reality gap between dream environments and reality.
Revised: Dream Land is an effective technique to bridge the reality gap between dream environments and reality.

Original: Furthermore, we additionally perform an extensive set of ablation studies.
Revised: Furthermore, we additionally perform an extensive set of ablation studies.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (Sutton & Barto, 2018) (RL) has experienced a ﬂurry of success in recent years, from learning to play Atari (Mnih et al., 2015) to achieving grandmaster level performance in StarCraft II (Vinyals et al., 2019).
Revised: Reinforcement learning (Sutton & Barto, 2018) (RL) has experienced a ﬂurry of success in recent years, from learning to play Atari (Mnih et al., 2015) to achieving grandmaster level performance in StarCraft II (Vinyals et al., 2019).

Original: However, in all these examples, the target environment is a simulator that can be directly trained in.
Revised: However, in all these examples, the target environment is a simulator that can be directly trained in.

Original: Reinforcement learning is often not a practical solution without a simulator of the environment.
Revised: Reinforcement learning is often not a practical solution without a simulator of the environment.

Original: Sometimes the target environment is expensive, dangerous, or even impossible to interact with.
Revised: Sometimes the target environment is expensive, dangerous, or even impossible to interact with.

Original: In these cases, the agent is trained in a simulated source environment.
Revised: In these cases, the agent is trained in a simulated source environment.

Original: Approaches that train an agent in a simulated environment with the hopes of generalization to the target environment experience a common problem referred to as the reality gap (Jakobi et al., 1995).
Revised: Approaches that train an agent in a simulated environment with the hopes of generalization to the target environment experience a common problem referred to as the reality gap (Jakobi et al., 1995).

Original: One approach to bridge the reality gap is domain randomization (Tobin et al., 2017).
Revised: One approach to bridge the reality gap is domain randomization (Tobin et al., 2017).

Original: The basic idea is that an agent which can perform well in an ensemble of simulations will also generalize to the real environment (Antonova et al., 2017; Tobin et al., 2017; Mordatch et al., 2015; Sadeghi & Levine, 2016).
Revised: The basic idea is that an agent which can perform well in an ensemble of simulations will also generalize to the real environment (Antonova et al., 2017; Tobin et al., 2017; Mordatch et al., 2015; Sadeghi & Levine, 2016).

Original: The ensemble of simulations is generally created by randomizing as much of the simulator as possible without fundamentally changing the task at hand.
Revised: The ensemble of simulations is generally created by randomizing as much of the simulator as possible without fundamentally changing the task at hand.

Original: Unfortunately, this approach is only applicable when a simulator is provided and the simulator is conﬁgurable.
Revised: Unfortunately, this approach is only applicable when a simulator is provided and the simulator is conﬁgurable.

Original: A recently growing ﬁeld, World Models (Ha & Schmidhuber, 2018), focuses on the side of this problem when the simulation does not exist.
Revised: A recently growing ﬁeld, World Models (Ha & Schmidhuber, 2018), focuses on the side of this problem when the simulation does not exist.

Original: World Models offer a general framework for optimizing controllers directly in learned simulated environments.
Revised: World Models offer a general framework for optimizing controllers directly in learned simulated environments.

Original: The learned dynamics model can be viewed as the agent’s dream environment.
Revised: The learned dynamics model can be viewed as the agent’s dream environment.

Original: This is an interesting area because it removes the need for an agent to operate in the target environment.
Revised: This is an interesting area because it removes the need for an agent to operate in the target environment.

Original: Some related approaches (Łukasz Kaiser et al., 2020;
Revised: Some related approaches (Łukasz Kaiser et al., 2020;

Original: Hafner et al., 2019; 2020; Sekar et al., 2020; Sutton, 1990) focus on an iterative version of the problem which enables the controller to continually interact with the target environment.
Revised: Hafner et al., 2019; 2020; Sekar et al., 2020; Sutton, 1990; Kurutach et al., 2018) focus on an adjacent problem which allows the controller to continually interact with the target environment.

Original: Despite the recent improvements (Łukasz Kaiser et al., 2020; Hafner et al., 2019; Sekar et al., 2020;
Revised: Despite the recent improvements (Łukasz Kaiser et al., 2020; Hafner et al., 2019; Sekar et al., 2020;

Original: Kim et al., 2020; Hafner et al., 2020) of World Models, none of them address the issue that World
Revised: Kim et al., 2020; Hafner et al., 2020) of World Models, none of them address the issue that World

Original: Models are susceptible to the reality gap.
Revised: Models are susceptible to the reality gap.

Original: The learned dream environment can be viewed as the source domain and the true environment as the target domain.
Revised: The learned dream environment can be viewed as the source domain and the true environment as the target domain.

Original: Whenever there are discrepancies between the source and target domains the reality gap can cause problems.
Revised: Whenever there are discrepancies between the source and target domains the reality gap can cause problems.

Original: Even though World Models suffer from the reality gap, none of the domain randomization approaches are directly applicable because the dream environment does not have easily conﬁgurable parameters.
Revised: Even though World Models suffer from the reality gap, none of the domain randomization approaches are directly applicable because the dream environment does not have easily conﬁgurable parameters.

Original: In this work we present Dropout’s Dream Land (DDL), a simple approach to bridge the reality gap from learned dream environments to reality.
Revised: In this work we present Dropout’s Dream Land (DDL), a simple approach to bridge the reality gap from learned dream environments to reality.

Original: Dropout’s Dream Land was inspired by the ﬁrst principles of domain randomization, namely, train a controller on a large set of different simulators which all adhere to the fundamental task of the target environment.
Revised: Dropout’s Dream Land was inspired by the ﬁrst principles of domain randomization, namely, train a controller on a large set of different simulators which all adhere to the fundamental task of the target environment.

Original: We are able to generate a nearly inﬁnite number of different simulators via the insight that dropout (Srivastava et al., 2014) can be understood as learning an ensemble of neural networks (Baldi & Sadowski, 2013).
Revised: We are able to generate a nearly inﬁnite number of different simulators via the insight that dropout (Srivastava et al., 2014) can be understood as learning an ensemble of neural networks (Baldi & Sadowski, 2013).

Original: Our empirical results demonstrate the advantage of Dropout’s Dream Land over baseline (Ha &
Revised: Our empirical results demonstrate the advantage of Dropout’s Dream Land over baseline (Ha &

Original: Schmidhuber, 2018; Kim et al., 2020) approaches.
Revised: Schmidhuber, 2018; Kim et al., 2020) approaches.

Original: Furthermore, we perform an extensive set of ablation studies which indicate the source of generalization improvements, requirements for the method to work, and when the method is most useful.
Revised: Furthermore, we perform an extensive set of ablation studies which indicate the source of generalization improvements, requirements for the method to work, and when the method is most useful.

Original:  R ELATED W ORKS 2. D ROPOUT
Revised:  R ELATED W ORKS 2. D ROPOUT

Original: Dropout (Srivastava et al., 2014) was introduced as a regularization technique for feedforward and convolutional neural networks.
Revised: Dropout (Srivastava et al., 2014) was introduced as a regularization technique for feedforward and convolutional neural networks.

Original: In its most general form, each unit is dropped with a probability p during the training process.
Revised: In its most general form, each unit is dropped with a probability p during the training process.

Original: Recurrent neural networks (RNNs) initially had issues beneﬁting from
Revised: Recurrent neural networks (RNNs) initially had issues beneﬁting from

Original: Dropout.
Revised: Dropout.

Original: Zaremba et al.
Revised: Zaremba et al.

Original: (2014) suggests not to apply dropout to the hidden state units of the RNN cell.
Revised: (2014) suggests not to apply dropout to the hidden state units of the RNN cell.

Original: Gal & Ghahramani (2016b) shortly after show that the mask can also be applied to the hidden state units, but the mask must be ﬁxed across the sequence during training.
Revised: Gal & Ghahramani (2016b) shortly after show that the mask can also be applied to the hidden state units, but the mask must be ﬁxed across the sequence during training.

Original: In this work, we follow the dropout approach from Gal & Ghahramani (2016b) when training the
Revised: In this work, we follow the dropout approach from Gal & Ghahramani (2016b) when training the

Original: RNN.
Revised: RNN.

Original: More formally, for each sequence, the boolean masks m xi , m xf , m xw , m xo , m hi , m hf , m hw , and m ho are sampled, then used in the following LSTM update:
Revised: More formally, for each sequence, the boolean masks m xi , m xf , m xw , m xo , m hi , m hf , m hw , and m ho are sampled, then used in the following LSTM update:

Original: [Equation]
Revised: [Equation]

Original: where x t , h t , and c t are the input, hidden state, and cell state, respectively, W xi , W xf , W xw ,
Revised: where x t , h t , and c t are the input, hidden state, and cell state, respectively, W xi , W xf , W xw ,

Original: W xo ∈ R d × r W hi , W hf , W hw ,
Revised: W xo ∈ R d × r W hi , W hf , W hw ,

Original: W ho ∈ R
Revised: W ho ∈ R

Original: d × d are the LSTM weight matrices, and b i , b f , b w , b o ∈ R
Revised: d × d are the LSTM weight matrices, and b i , b f , b w , b o ∈ R

Original: d are the LSTM biases.
Revised: d are the LSTM biases.

Original: The masks are ﬁxed for the entire sequence, but may differ between sequences in the mini-batch.
Revised: The masks are ﬁxed for the entire sequence, but may differ between sequences in the mini-batch.

Original: 2. D OMAIN R ANDOMIZATION
Revised: 2. D OMAIN R ANDOMIZATION

Original: The goal of domain randomization (Tobin et al., 2017; Sadeghi & Levine, 2016) is to create many different versions of the dynamics model with the hope that a policy generalizing to all versions of the dynamics model will do well on the true environment.
Revised: The goal of domain randomization (Tobin et al., 2017; Sadeghi & Levine, 2016) is to create many different versions of the dynamics model with the hope that a policy generalizing to all versions of the dynamics model will do well on the true environment.

Original: Figure 1 illustrates many simulated environments ( ˆ e j ) overlapping with the actual environment ( e ∗ ).
Revised: Figure 1 illustrates many simulated environments ( ˆ e j ) overlapping with the actual environment ( e ∗ ).

Original: Simulated environments are often far cheaper to operate in than the actual environment.
Revised: Simulated environments are often far cheaper to operate in than the actual environment.

Original: Hence, it is desirable to be able to perform the majority of interactions in the simulated environments.
Revised: Hence, it is desirable to be able to perform the majority of interactions in the simulated environments.

Original: Randomization has been applied on observations (e.g., lighting, textures) to perform robotic grasping (Tobin et al., 2017) and collision avoidance of drones (Sadeghi & Levine, 2016).
Revised: Randomization has been applied on observations (e.g., lighting, textures) to perform robotic grasping (Tobin et al., 2017) and collision avoidance of drones (Sadeghi & Levine, 2016).

Original: Randomization has also proven useful when applied to the underlying dynamics of simulators (Peng et al., 2018).
Revised: Randomization has also proven useful when applied to the underlying dynamics of simulators (Peng et al., 2018).

Original: Often, both the observations and simulation dynamics are randomized (Andrychowicz et al., 2020).
Revised: Often, both the observations and simulation dynamics are randomized (Andrychowicz et al., 2020).

Original: Domain randomization generally uses some pre-existing simulator which then injects randomness into speciﬁc aspects of the simulator (e.g., color textures, friction coefﬁcients).
Revised: Domain randomization generally uses some pre-existing simulator which then injects randomness into speciﬁc aspects of the simulator (e.g., color textures, friction coefﬁcients).

Original: Each of the simulated environments in Figure 1 can be thought of as a noisy sample of the pre-existing simulator.
Revised: Each of the simulated environments in Figure 1 can be thought of as a noisy sample of the pre-existing simulator.

Original: To the best of our knowledge, domain randomization has yet to be applied to entirely learned simulators.
Revised: To the best of our knowledge, domain randomization has yet to be applied to entirely learned simulators.

Original: 2. W ORLD M ODELS
Revised: 2. W ORLD M ODELS

Original: e ⇤ ˆ e 6 ˆ e 7 ˆ eˆ eˆ eˆ eˆ eˆ e The world model (Ha & Schmidhuber, 2018) has three modules trained separately: (i) vision module ( V ); (ii) dynamics module ( M ); and (iii) controller ( C ).
Revised: The world model (Ha & Schmidhuber, 2018) has three modules trained separately: (i) vision module ( V ); (ii) dynamics module ( M ); and (iii) controller ( C ).

Original: A high-level view is shown in Algorithm 1.
Revised: A high-level view is shown in Algorithm 1.

Original:  The vision module ( V ) is a variational autoencoder (VAE) (Kingma & Welling, 2013), which maps image observation ( o ) to a lower-dimensional representation z ∈ R n .
Revised: e ⇤ ˆ eˆ eˆ eˆ eˆ eˆ eˆ eˆ e The vision module ( V ) is a variational autoencoder (VAE) (Kingma & Welling, 2013), which maps image observation ( o ) to a lower-dimensional representation z ∈ R n .

Original: The dynamics model ( M ) is a mixture density network recurrent neural network (MDN-RNN)
Revised: The dynamics model ( M ) is a mixture density network recurrent neural network (MDN-RNN)

Original: (Ha & Schmidhuber, 2018; Graves, 2013).
Revised: (Ha & Schmidhuber, 2018; Graves, 2013).

Original: It is implemented as an LSTM followed by a fully-connected layer outputting parameters for a Gaussian mixture model with k components.
Revised: It is implemented as an LSTM followed by a fully-connected layer outputting parameters for a Gaussian mixture model with k components.

Original: Each feature has k different π parameters for the logits of multinomial distribution, and ( µ, σ ) parameters for the k components in the Gaussian mixture.
Revised: Each feature has k different π parameters for the logits of multinomial distribution, and ( µ, σ ) parameters for the k components in the Gaussian mixture.

Original: At each timestep, the MDN-RNN takes in the state z and action a as inputs and predicts π , µ , σ .
Revised: At each timestep, the MDN-RNN takes in the state z and action a as inputs and predicts π , µ , σ .

Original: To draw a sample from the MDN-RNN, we ﬁrst sample the multinomial distribution parameterized by π , which indexes which of the k normal distributions in the Gaussian mixture to sample from.
Revised: To draw a sample from the MDN-RNN, we ﬁrst sample the multinomial distribution parameterized by π , which indexes which of the k normal distributions in the Gaussian mixture to sample from.

Original: This is then repeated for each of the n features.
Revised: This is then repeated for each of the n features.

Original: Depending on the experiments, Ha & Schmidhuber (2018) also include an auxiliary head to the LSTM which predicts whether the episode terminates ( d ).
Revised: Depending on the experiments, Ha & Schmidhuber (2018) also include an auxiliary head to the LSTM which predicts whether the episode terminates ( d ).

Original: The controller ( C ) is responsible for deciding what actions to take.
Revised: The controller ( C ) is responsible for deciding what actions to take.

Original: It takes features produced by the encoder V and  model M as input (not the raw observations).
Revised: It takes features produced by the encoder V and dynamics model M as input (not the raw observations).

Original: The simple controller is a single-layer model which uses an evolutionary algorithm (CMA-ES (Hansen & Ostermeier, 2001)) to ﬁnd its parameters.
Revised: The simple controller is a single-layer model which uses an evolutionary algorithm (CMA-ES (Hansen & Ostermeier, 2001)) to ﬁnd its parameters.

Original: Depending on the problem setting, the controller ( C ) can either be optimized directly on the target environment ( e ∗ ) or on the dream environment ( ˆ e ).
Revised: Depending on the problem setting, the controller ( C ) can either be optimized directly on the target environment ( e ∗ ) or on the dream environment ( ˆ e ).

Original: This paper is focused on the case of optimizing exclusively in the dream environment.
Revised: This paper is focused on the case of optimizing exclusively in the dream environment.

Original:  D ROPOUT ’ S D REAM L AND
Revised:  D ROPOUT ’ S D REAM L AND

Original: In this work we introduce Dropout’s Dream Land (DDL).
Revised: In this work we introduce Dropout’s Dream Land (DDL).

Original: Dropout’s Dream Land is the ﬁrst work to offer a strategy to bridge the reality gap between learned neural network dynamics models and reality.
Revised: Dropout’s Dream Land is the ﬁrst work to offer a strategy to bridge the reality gap between learned neural network dynamics models and reality.

Original: Traditional domain randomization generates many different dynamics models by randomizing conﬁgurable parameters of a given simulation.
Revised: Traditional domain randomization generates many different dynamics models by randomizing conﬁgurable parameters of a given simulation.

Original: This approach does not apply to neural network dynamics models because they generally do not have conﬁgurable parameters (such as textures and friction coefﬁcients).
Revised: This approach does not apply to neural network dynamics models because they generally do not have conﬁgurable parameters (such as textures and friction coefﬁcients).

Original: In Dropout’s Dream Land, the controller can interact with billions 1 of dream environments, whereas previous works (Ha & Schmidhuber, 2018;
Revised: In Dropout’s Dream Land, the controller can interact with billions 1 of dream environments, whereas previous works (Ha & Schmidhuber, 2018;

Original: Kim et al., 2020) only use one dream environment.
Revised: Kim et al., 2020) only use one dream environment.

Original: A naive way to go about this would be to train a large collection of neural network world models.
Revised: A naive way to go about this would be to train a large collection of neural network world models.

Original: However, this would be computationally expensive.
Revised: However, this would be computationally expensive.

Original: To keep the computational cost low, we go about this by applying dropout to the  model in order to form different dynamics models.
Revised: To keep the computational cost low, we go about this by applying dropout to the dynamics model in order to form different dynamics models.

Original: Crucially, dropout is applied at both training time and inference time of M .
Revised: Crucially, dropout is applied at both training time and inference time of M .

Original: Each unique dropout mask applied to the  model M can be viewed as a different environment.
Revised: Each unique dropout mask applied to the dynamics model M can be viewed as a different environment.

Original: Similar to the spirit of domain randomization, an agent is expected to perform well in the real environment if it can perform well in all the different simulated environments.
Revised: Similar to the spirit of domain randomization, an agent is expected to perform well in the real environment if it can perform well in all the different simulated environments.

Original: 3. L EARNING THE D REAM E NVIRONMENT
Revised: 3. L EARNING THE D REAM E NVIRONMENT

Original: The Dropout’s Dream Land environments are built around the  model M .
Revised: The Dropout’s Dream Land environments are built around the dynamics model M .

Original: During training time the controller interactions are described by Figure 2.
Revised: During training time the controller interactions are described by Figure 2.

Original: In Figure 2 ˆ r , ˆ d , and ˆz are generated entirely by M .
Revised: In Figure 2 ˆ r , ˆ d , and ˆz are generated entirely by M .

Original: In this work M is an LSTM where x = [ z (cid:62) , a (cid:62) ] (cid:62) from equations (1)-(4). 
Revised: In this work M is an LSTM where x = [ z (cid:62) , a (cid:62) ] (cid:62) from equations (1)-(4). The

Original: The LSTM is followed by multiple heads for predictions of the latent state ( ˆz ), reward ( ˆ r ) and termination ( ˆ d ).
Revised: LSTM is followed by multiple heads for predictions of the latent state ( ˆz ), reward ( ˆ r ) and termination ( ˆ d ).

Original: The 
Revised: The reward and termination heads are simple fully-connected layers. Latent state prediction is [Equation] [Figure]

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: z h r d
Revised: z h r d

Original: C a e ⇤ o VM
Revised: C a e ⇤ o V

Original: The reward and termination heads are simple fully-connected layers.

Original: Latent state prediction is done with a MDN-RNN (Ha & Schmidhuber, 2018; Graves, 2013), but this could be replaced by any other neural network that supports dropout (e.g., GameGAN (Kim et al., 2020)).
Revised: M done with a MDN-RNN (Ha & Schmidhuber, 2018; Graves, 2013), but this could be replaced by any other neural network that supports dropout (e.g., GameGAN (Kim et al., 2020)).

Original: 3.1. L OSS F UNCTION
Revised: 3.1. L OSS F UNCTION

Original: The  model M jointly optimizes all three heads where the loss of a single transition is deﬁned as:
Revised: The dynamics model M jointly optimizes all three heads where the loss of a single transition is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Here, L z = − (cid:80) ni =1 log( (cid:80) kj =

Revised: Here, L z = − (cid:80) ni =1 log( (cid:80) kj =

Original: ˆ π
Revised: ˆ π

Original: i,j N ( z i | ˆ µ i,j , ˆ σ 2 i,j )) , where n is the size of the latent feature vector z , ˆ π i,j is the j th component probability for the i th feature, ˆ µ i,j , ˆ σ i,j are the corresponding mean and standard deviation, and L z is a mixture density loss for the latent state predictions.
Revised: i,j N ( z i | ˆ µ i,j , ˆ σ 2 i,j )) is a mixture density loss for the latent state predictions, where n is the size of the latent feature vector z , ˆ π i,j is the j th component probability for the i th feature, ˆ µ i,j , ˆ σ i,j are the corresponding mean and standard deviation.

Original: L r =
Revised: L r =

Original: ( r − ˆ r )is the square loss on rewards, where r and ˆ r are the true and estimated rewards, respectively.
Revised: ( r − ˆ r ) 2 is the square loss on rewards, where r and ˆ r are the true and estimated rewards, respectively.

Original: L d = − d log( ˆ d ) − (1 − d ) log(1 − ˆ d ) is the cross-entropy loss for termination prediction, where d and ˆ d are the true and estimated probabilities of the episode ending, respectively.
Revised: L d = − d log( ˆ d ) − (1 − d ) log(1 − ˆ d ) is the cross-entropy loss for termination prediction, where d and ˆ d are the true and estimated probabilities of the episode ending, respectively.

Original: Constants α d and α r in (5) are for trading off importance of the termination and reward objectives.
Revised: Constants α d and α r in (5) are for trading off importance of the termination and reward objectives.

Original: The loss ( L M ) is aggregated over each sequence and averaged across the mini-batch.
Revised: The loss ( L M ) is aggregated over each sequence and averaged across the mini-batch.

Original: [Equation] WITH D ROPOUT
Revised: 3.1. T RAINING D YNAMICS M ODEL M WITH D ROPOUT

Original: At training time of M , we apply dropout (Gal & Ghahramani, 2016b) to the LSTM to simulate different random environments.
Revised: At training time of M (Algorithm 1 Line 5), we apply dropout (Gal & Ghahramani, 2016b) to the LSTM to simulate different random environments.

Original: For each input and hidden unit, we ﬁrst sample a boolean indicator with probability p train .
Revised: For each input and hidden unit, we ﬁrst sample a boolean indicator with probability p train .

Original: If the indicator is 1, the corresponding input/hidden unit is masked.
Revised: If the indicator is 1, the corresponding input/hidden unit is masked.

Original: Masks m xi , m xf , m xw , m xo , m hi , m hf , m hw , and m ho are sampled independently (Equations (1)- (4)).
Revised: Masks m xi , m xf , m xw , m xo , m hi , m hf , m hw , and m ho are sampled independently (Equations (1)-(4)).

Original: When training the RNN, each mini-batch contains multiple sequences.
Revised: When training the RNN, each mini-batch contains multiple sequences.

Original: Each sequence uses an independently sampled dropout mask.
Revised: Each sequence uses an independently sampled dropout mask.

Original: We ﬁx the dropout mask for the entire sequence as this was previously found to be critically important (Gal & Ghahramani, 2016b).
Revised: We ﬁx the dropout mask for the entire sequence as this was previously found to be critically important (Gal & Ghahramani, 2016b).

Original: Training the RNN with many different dropout masks is critical in order to generate multiple different dynamics models.
Revised: Training the RNN with many different dropout masks is critical in order to generate multiple different dynamics models.

Original: At the core of domain randomization is the requirement that the randomizations do not fundamentally change the task.
Revised: At the core of domain randomization is the requirement that the randomizations do not fundamentally change the task.

Original: This constraint is violated if we do not train the RNN with dropout but apply dropout at inference time (explored further in Section 4.4).
Revised: This constraint is violated if we do not train the RNN with dropout but apply dropout at inference time (explored further in Section 4.4).

Original: After optimizing the  model M , we can use it to construct dream environments for controller training (Section 3.2).
Revised: After optimizing the dynamics model M , we can use it to construct dream environments for controller training (Section 3.2).

Original: In this work, we never sample masks to apply to the action ( a ).
Revised: In this work, we never sample masks to apply to the action ( a ).

Original: We do not zero out the action because in some environments this could imply the agent taking an action (ex: moving to the left).
Revised: We do not zero out the action because in some environments this could imply the agent taking an action (ex: moving to the left).

Original: This design choice could be changed depending on the environment, for example, when a zero’d action corresponds to a no-op or a sticky action.
Revised: This design choice could be changed depending on the environment, for example, when a zero’d action corresponds to a no-op or a sticky action.

Original: 3. T RAINING THE C ONTROLLER 3.2. I NTERACTING WITH D ROPOUT ’ S D REAM L AND
Revised: 3. T RAINING THE C ONTROLLER 3.2. I NTERACTING WITH D ROPOUT ’ S D REAM L AND

Original: Interactions with the dream environment  can be characterized as training time for the controller ( C ) and inference time of the  model ( M ).
Revised: Interactions with the dream environment (Algorithm 1 Line 8) can be characterized as training time for the controller ( C ) and inference time of the dynamics model ( M ).

Original: An episode begins by generating the initial latent state vector ˆz by either sampling from a standard normal distribution or sampling from the starting points of the observed trajectories used to train M (Ha & Schmidhuber, 2018).
Revised: An episode begins by generating the initial latent state vector ˆz by either sampling from a standard normal distribution or sampling from the starting points of the observed trajectories used to train M (Ha & Schmidhuber, 2018).

Original: The hidden cell ( c ) and state ( h ) vectors are initialized with zeros.
Revised: The hidden cell ( c ) and state ( h ) vectors are initialized with zeros.

Original: The controller ( C ) decides the action to take based on ˆz and h .
Revised: The controller ( C ) decides the action to take based on ˆz and h .

Original: In Figure 2 the controller also observes ˆ r and ˆ d but these are exclusively used for the optimization process of the controller.
Revised: In Figure 2 the controller also observes ˆ r and ˆ d but these are exclusively used for the optimization process of the controller.

Original: The controller then performs an action a on a dream environment.
Revised: The controller then performs an action a on a dream environment.

Original: A new dropout mask is sampled (with probability p infer ) and applied to M .
Revised: A new dropout mask is sampled (with probability p infer ) and applied to M .

Original: We refer to the masked  model as M j and the corresponding Dropout’s Dream Land environment as ˆ e j .
Revised: We refer to the masked dynamics model as M j and the corresponding Dropout’s Dream Land environment as ˆ e j .

Original: The current latent state ˆz and action a are concatenated, and passed to M j to perform a forward pass.
Revised: The current latent state ˆz and action a are concatenated, and passed to M j to perform a forward pass.

Original: The episode terminates based on a sample from a Bernoulli distribution parameterized by ˆ d .
Revised: The episode terminates based on a sample from a Bernoulli distribution parameterized by ˆ d .

Original: The dream environment then outputs the latent state, LSTM’s hidden state, reward, and whether the episode terminates.
Revised: The dream environment then outputs the latent state, LSTM’s hidden state, reward, and whether the episode terminates.

Original: It is crucial to apply dropout at inference time (of the dynamics model M ) in order to create different versions of the dream environment for the controller C .
Revised: It is crucial to apply dropout at inference time (of the dynamics model M ) in order to create different versions of the dream environment for the controller C .

Original: In addition to randomizing masks at every step, we will also consider several other variants of how to apply dropout in Sections 4.3 and 4.4.
Revised: In addition to randomizing masks at every step, we will also consider several other variants of how to apply dropout in Sections 4.3 and 4.4.

Original: DDL’s use of dropout is different from traditional applications of dropout.
Revised: DDL’s use of dropout is different from traditional applications of dropout.

Revised: In related works (Kahn et al., 2017) Monte-Carlo (MC) Dropout (Gal & Ghahramani, 2016a) has been used to approximate the mean and variance of output predictions from an ensemble.

Revised: We emphasize that DDL does not use MC Dropout.

Revised: The purpose of DDL’s approach to dropout is to generate many different versions of the dynamics model.

Revised: More explicitly, the controller is trained to maximize expected returns across many different dynamics models in the ensemble as opposed maximizing expected returns on the ensemble average.

Revised: Dropout has also traditionally been used as a model regularizer.

Original: Generally dropout is only applied at training time but not  inference time.
Revised: Dropout as a model regularizer is only applied at training time but not at inference time.

Original: The usual trade-off is lower test loss at the cost of higher training loss (Srivastava et al., 2014; Gal & Ghahramani, 2016b).
Revised: The usual trade-off is lower test loss at the cost of higher training loss (Srivastava et al., 2014; Gal & Ghahramani, 2016b).

Original: However, DDL’s ultimate goal is not lower test loss.
Revised: However, DDL’s ultimate goal is not lower test loss.

Original: In fact, we expect applying dropout at inference time will consistently make the test loss worse (this will be experimentally veriﬁed in Section 4.2).
Revised: In fact, we expect applying dropout at inference time will consistently make the test loss worse (this will be experimentally veriﬁed in Section 4.2).

Original: The ultimate goal is providing dream environments to a controller so that the optimal policy in Dropout’s Dream Land also maximizes expected returns in the target environment ( e ∗ ).
Revised: The ultimate goal is providing dream environments to a controller so that the optimal policy in Dropout’s Dream Land also maximizes expected returns in the target environment ( e ∗ ).

Original: 3.2. T RAINING WITH CMA-ES
Revised: 3.2. T RAINING WITH CMA-ES

Original: We follow the same controller optimization procedure as was done in World Models (Ha & Schmidhuber, 2018) and GameGAN (Kim et al., 2020) on their DoomTakeCover experiments.
Revised: We follow the same controller optimization procedure as was done in World Models (Ha & Schmidhuber, 2018) and GameGAN (Kim et al., 2020) on their DoomTakeCover experiments.

Original: We train the controller with CMA-ES (Hansen & Ostermeier, 2001).
Revised: We train the controller with CMA-ES (Hansen & Ostermeier, 2001).

Original: Each controller in the population (of size N pop ) reports their mean returns on a set of N trials episodes generated in Section 3.2.1.
Revised: Each controller in the population (of size N pop ) reports their mean returns on a set of N trials episodes generated in Section 3.2.1.

Original: As controllers in the population do not share an environment, the probability of controllers interacting with the same sequence of dropout masks is vanishingly small.
Revised: As controllers in the population do not share a dream environment, the probability of controllers interacting with the same sequence of dropout masks is vanishingly small.

Original: Let N max ep len be the maximum number of steps in an episode.
Revised: Let N max ep len be the maximum number of steps in an episode.

Original: In a single CMA-ES iteration, the population as a whole can interact with
Revised: In a single CMA-ES iteration, the population as a whole can interact with

Original: N pop × N trials × N max ep len different environments.
Revised: N pop × N trials × N max ep len different environments.

Original: In our experiments, N pop = 64 , N trials = 16 , and N max ep len is 1000 for CarRacing and 2100 for DoomTakeCover.
Revised: In our experiments, N pop = 64 , N trials = 16 , and N max ep len is 1000 for CarRacing and 2100 for DoomTakeCover.

Original: This potentially results in > 1000000 different environments at each generation.
Revised: This potentially results in > 1000000 different environments at each generation.

Original: 3.2. D REAM L EADER B OARD
Revised: 3.2. D REAM L EADER B OARD

Original: After every ﬁxed number of generations ( 25 in our experiments), the best controller in the population (which received the highest average returns across its respective N trials episodes) is selected for evaluation (Ha & Schmidhuber, 2018; Kim et al., 2020).
Revised: After every ﬁxed number of generations ( 25 in our experiments), the best controller in the population (which received the highest average returns across its respective N trials episodes) is selected for evaluation (Ha & Schmidhuber, 2018; Kim et al., 2020).

Original: This controller is evaluated for another
Revised: This controller is evaluated for another

Original: N pop × N trials episodes in the Dropout’s Dream Land environments.
Revised: N pop × N trials episodes in the Dropout’s Dream Land environments.

Original: The controller’s mean across
Revised: The controller’s mean across

Original: N pop × N trials trials is logged to the Dream Leader Board.
Revised: N pop × N trials trials is logged to the Dream Leader Board.

Original: After 2000 generations, the controller at the top of the Dream Leader Board is evaluated in the real environment.
Revised: After 2000 generations, the controller at the top of the Dream Leader Board is evaluated in the real environment.

Original: 3.2. I NTERACTING WITH THE R EAL E NVIRONMENT
Revised: 3.2. I NTERACTING WITH THE R EAL E NVIRONMENT

Original: In Figure 3 we illustrate the controller’s interaction with the real environment ( e ∗ ).
Revised: In Figure 3 we illustrate the controller’s interaction with the real environment ( e ∗ ).

Original: The controller only interacts with the target environment during testing.
Revised: The controller only interacts with the target environment during testing.

Original: These interactions are never used to modify parameters of the controller.
Revised: These interactions are never used to modify parameters of the controller.

Original: At test time r , d , and o are generated by the target environment ( e ∗ ) and z is the embedding of o from the VAE ( V ).
Revised: At test time r , d , and o are generated by the target environment ( e ∗ ) and z is the embedding of o from the VAE ( V ).

Original: The only use of M when interacting with the target environment is producing h as a feature for the controller.
Revised: The only use of M when interacting with the target environment is producing h as a feature for the controller.

Original: Interactions with e ∗ do not apply dropout to the input/hidden units of M .
Revised: Interactions with e ∗ do not apply dropout to the input/hidden units of M .

Original: The purpose of applying dropout when training (Sections 3.2.1 and 3.2.2) and selecting the controller (Section 3.2.3) was to  force the controller to do well on a variety of environments at the cost of  model accuracy (detailed experimental results are in Section 4.2).
Revised: The purpose of applying dropout when training (Sections 3.2.1 and 3.2.2) and selecting the controller (Section 3.2.3) was to [Table] [Table] force the controller to do well on a variety of environments at the cost of dynamics model accuracy (detailed experimental results are in Section 4.2).

Original: At the time of interaction with e ∗ the controller’s parameters are frozen and thus there is no beneﬁt to trading  model accuracy.
Revised: At the time of interaction with e ∗ the controller’s parameters are frozen and thus there is no beneﬁt to trading dynamics model accuracy.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we perform experiments on the DoomTakeCover-v0 (Paquette, 2017) and CarRacingv0 (Klimov, 2016) environments from OpenAI Gym (Brockman et al., 2016).These have also been [Table]
Revised: In this section, we perform experiments on the DoomTakeCover-v0 (Paquette, 2017) and CarRacingv0 (Klimov, 2016) environments from OpenAI Gym (Brockman et al., 2016).

Original: [Table] used in related works (Kim et al., 2020; Ha & Schmidhuber, 2018).
Revised: These have also been used in related works (Kim et al., 2020; Ha & Schmidhuber, 2018).

Original: Architecture details of V , M , and C are in Appendix A.1.
Revised: Architecture details of V , M , and C are in Appendix A.1.

Original: DoomTakeCover is a control task in which the goal is to dodge ﬁreballs for as long as possible.
Revised: DoomTakeCover is a control task in which the goal is to dodge ﬁreballs for as long as possible.

Original: The controller receives a reward of +1 for every step it is alive.
Revised: The controller receives a reward of +1 for every step it is alive.

Original: The maximum number of frames is limited to 2100 .
Revised: The maximum number of frames is limited to 2100 .

Original: CarRacing is a continuous control task to learn from pixels.
Revised: CarRacing is a continuous control task to learn from pixels.

Original: The race track is split up into “tiles”.
Revised: The race track is split up into “tiles”.

Original: The goal is to make it all the way around the track (i.e., crossing every tile).
Revised: The goal is to make it all the way around the track (i.e., crossing every tile).

Original: We terminate an episode when all tiles are crossed or when the number of steps exceeds 1000.
Revised: We terminate an episode when all tiles are crossed or when the number of steps exceeds 1000.

Original: Let N tiles be the total number of tiles.
Revised: Let N tiles be the total number of tiles.

Original: The simulator (Klimov, 2016) deﬁnes the reward r t at each timestep as 100 N tiles − 0 .
Revised: The simulator (Klimov, 2016) deﬁnes the reward r t at each timestep as 100 N tiles − 0 .

Original: 1 if a new tile is crossed, and − 0 .
Revised: 1 if a new tile is crossed, and − 0 .

Original: 1 otherwise.
Revised: 1 otherwise.

Original: The number of tiles is not explicitly set by the simulator.
Revised: The number of tiles is not explicitly set by the simulator.

Original: We generated 10000 tracks and observed that the number of tiles in the track appears to follow a normal distribution with mean 289 .
Revised: We generated 10000 tracks and observed that the number of tiles in the track appears to follow a normal distribution with mean 289 .

Original: To simplify the reward function, we ﬁx N tiles to 289 in the randomly generated tracks, and call the modiﬁed environment CarRacingFixedN.
Revised: To simplify the reward function, we ﬁx N tiles to 289 in the randomly generated tracks, and call the modiﬁed environment CarRacingFixedN.

Original: For all experiments the controller is trained exclusively in the dream environment (Section 3.2.1) for 2000 generations.
Revised: For all experiments the controller is trained exclusively in the dream environment (Section 3.2.1) for 2000 generations.

Original: The controller only interacts with the target environments for testing (Section 3.2.4).
Revised: The controller only interacts with the target environments for testing (Section 3.2.4).

Original: The target environment is never used to update parameters of the controller.
Revised: The target environment is never used to update parameters of the controller.

Original: Broadly speaking, our experiments are focused on either evaluating the  model ( M ) or the controller ( C ).
Revised: Broadly speaking, our experiments are focused on either evaluating the dynamics model ( M ) or the controller ( C ).

Original: Accuracy of the  model is evaluated against a training and testing set of trajectories (Appendix A.2).
Revised: Accuracy of the dynamics model is evaluated against a training and testing set of trajectories (Appendix A.2).

Original: The controller is evaluated by returns in the dream and real environments (Appendix A.2).
Revised: The controller is evaluated by returns in the dream and real environments (Appendix A.2).

Original: 4. C OMPARISON WITH B ASELINES
Revised: 4. C OMPARISON WITH B ASELINES

Original: We compare the controller trained in Dropout’s Dream Land (DDL) with the baseline World Models (WM) controller and a random policy.
Revised: We compare the controller trained in Dropout’s Dream Land (DDL) with the baseline World Models (WM) controller and a random policy.

Original: On the Doom environment we also compare with
Revised: On the Doom environment we also compare with

Original: GameGAN (Kim et al., 2020) and Action-LSTM (Chiappa et al., 2017) 2 .
Revised: GameGAN (Kim et al., 2020) and Action-LSTM (Chiappa et al., 2017) 2 .

Original: All controllers are trained entirely in dream environments.
Revised: All controllers are trained entirely in dream environments.

Original: Results on the dream and real environments are in Tables 1 and 2.
Revised: Results on the dream and real environments are in Tables 1 and 2.

Revised: The CarRacing-v0 results appear different from those found in World Models (Ha & Schmidhuber, 2018) because we are not performing the same experiment.

Revised: In this paper we train the controller entirely in the dream environment and only interact with the real environment during testing.

Revised: In

Revised: World Models (Ha & Schmidhuber, 2018)

Revised: the controller was trained directly in the CarRacing-v environment.

Original: In Table 1 we observe that DDL offers performance improvements over all the baseline approaches.
Revised: In Table 1 we observe that DDL offers performance improvements over all the baseline approaches.

Original: The DoomTakeCover-v0 returns from DDL are lower than the returns reported by the temperatureregulated variant in Ha & Schmidhuber (2018).
Revised: The DoomTakeCover-v0 returns from DDL are lower than the returns reported by the temperatureregulated variant in Ha & Schmidhuber (2018).

Original: Even though the temperature-regulated variant increases uncertainty of the dream environment it is still only capable of creating one dream environment.
Revised: Even though the temperature-regulated variant increases uncertainty of the dream environment it is still only capable of creating one dream environment.

Original: Furthermore, we emphasize that adjusting temperature is only useful for a limited set of dynamics models.
Revised: Furthermore, we emphasize that adjusting temperature is only useful for a limited set of dynamics models.

Original: For example, it would not be straightforward to apply temperature to any dynamics model which does not produce a probability density function (ex: GameGAN); whereas  the DDL approach of generating many different dynamics models is useful to any learned neural network dynamics model.
Revised: For example, it would not be straightforward to apply temperature to any dynamics model which does not produce a probability density function (ex: GameGAN); whereas [Figure] [Figure] the DDL approach of generating many different dynamics models is useful to any learned neural network dynamics model.

Original: In Table 2 we observe that DDL achieves a signiﬁcant performance boost over WM in both the real CarRacing-v0 and CarRacingFixedN environment variants.
Revised: In Table 2 we observe that DDL achieves a signiﬁcant performance boost over WM in both the real CarRacing-v0 and CarRacingFixedN environment variants.

Original: This is because the WM dream environment was easier for the controller to exploit and overﬁt to errors between the simulator and reality.
Revised: This is because the WM dream environment was easier for the controller to exploit and overﬁt to errors between the simulator and reality.

Original: Forcing the controller to succeed in many different dropout environments makes it difﬁcult to overﬁt to discrepancies between the dream environment and reality.
Revised: Forcing the controller to succeed in many different dropout environments makes it difﬁcult to overﬁt to discrepancies between the dream environment and reality.

Original: 4. D OES D ROPOUT A FFECT A CCURACY OF THE RNN?
Revised: 4. D OES D ROPOUT A FFECT A CCURACY OF THE RNN?

Original: In this experiment, we compare the losses in (5) achieved by a standard RNN and an RNN with dropout ( p train = 0 . 05 ).
Revised: In this experiment, we compare the losses in (5) achieved by a standard RNN and an RNN with dropout ( p train = 0 . 05 ).

Original: The same training and test sets described in Appendix A.2 are used.
Revised: The same training and test sets described in Appendix A.2 are used.

Original: [Figure] [Figure] Standard use cases of dropout generally observe a larger training loss but lower test loss relative to the same model trained without dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016a).
Revised: Standard use cases of dropout generally observe a larger training loss but lower test loss relative to the same model trained without dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016a).

Original: In
Revised: In

Original: Table 3, we do not observe any immediate performance improvements of the RNN trained with dropout.
Revised: Table 3, we do not observe any immediate performance improvements of the RNN trained with dropout.

Original: In fact, we observe worse results on the test set.
Revised: In fact, we observe worse results on the test set.

Original: It is possible that the combination of the
Revised: It is possible that the combination of the

Original: RNN’s small size and dropout are causing the  model to underﬁt.
Revised: RNN’s small size and dropout are causing the dynamics model to underﬁt.

Original: The poor performance of both DDL RNNs indicates a clear conclusion about the results from Tables 1 and 2.
Revised: The poor performance of both DDL RNNs indicates a clear conclusion about the results from Tables 1 and 2.

Original: The improved performance of DDL relative to World Models comes from forcing the controller to operate in many different environments and not from a single more accurate  model M .
Revised: The improved performance of DDL relative to World Models comes from forcing the controller to operate in many different environments and not from a single more accurate dynamics model M .

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: 4. D OES THE I NFERENCE D ROPOUT R ATE A FFECT D REAM 2R EAL G ENERALIZATION ?
Revised: 4. D OES THE I NFERENCE D ROPOUT R ATE A FFECT D REAM 2R EAL G ENERALIZATION ?

Original: It is uncommon to evaluate test loss while still applying dropout.
Revised: It is uncommon to evaluate test loss while still applying dropout.

Original: In Dropout’s Dream Land this is important because we still apply dropout at inference time of the RNN.
Revised: In Dropout’s Dream Land this is important because we still apply dropout at inference time of the RNN.

Original: In this experiment we evaluate the relationship between the inference dropout rate ( p infer ), model loss, and returns in the target environment.
Revised: In this experiment we evaluate the relationship between the inference dropout rate ( p infer ), model loss, and returns in the target environment.

Original: Model loss is measured by the loss in (5) on the test sets for varying levels of inference dropout ( p infer ).
Revised: Model loss is measured by the loss in (5) on the test sets for varying levels of inference dropout ( p infer ).

Original: Returns in the target environment are reported based on the best controller (Section 3.2.3) trained with varying levels of inference dropout.
Revised: Returns in the target environment are reported based on the best controller (Section 3.2.3) trained with varying levels of inference dropout.

Original: Model loss results are shown in Figure 4.
Revised: Model loss results are shown in Figure 4.

Original: As expected, as the inference dropout rate is increased our model loss increases.
Revised: As expected, as the inference dropout rate is increased our model loss increases.

Original: However, we also observe in Figure 5 that increasing the inference dropout rate improves generalization to the target environment.
Revised: However, we also observe in Figure 5 that increasing the inference dropout rate improves generalization to the target environment.

Original: We believe that the increase in returns comes from an increase in capacity to distort the dynamics model.
Revised: We believe that the increase in returns comes from an increase in capacity to distort the dynamics model.

Original: Figures 4 and 5 suggest that we can trade accuracy of the dream environment for better generalization on the target environment.
Revised: Figures 4 and 5 suggest that we can trade accuracy of the dream environment for better generalization on the target environment.

Original: However, this should only be useful up to the point where the task at hand is fundamentally changed .
Revised: However, this should only be useful up to the point where the task at hand is fundamentally changed .

Original: Figuresuggests this point is somewhere between 0 .
Revised: Figuresuggests this point is somewhere between 0 .

Original: 1 and 0 .
Revised: 1 and 0 .

Original: 2 for p infer , though we suspect in practice this will be highly dependent on network architecture and the environment.
Revised: 2 for p infer , though we suspect in practice this will be highly dependent on network architecture and the environment.

Original: In Figure 5 we observe relatively weak returns on the real CarRacingFixedN environment when the inference dropout rate is zero.
Revised: In Figure 5 we observe relatively weak returns on the real CarRacingFixedN environment when the inference dropout rate is zero.

Original: Recall from Table 3 that the dropout variant has a much higher test loss than the non-dropout variant on CarRacingFixedN.
Revised: Recall from Table 3 that the dropout variant has a much higher test loss than the non-dropout variant on CarRacingFixedN.

Original: This means that when p infer = 0 the single environment DDL is able to create is relatively inaccurate.
Revised: This means that when p infer = 0 the single environment DDL is able to create is relatively inaccurate.

Original: It is easier for the controller to exploit any discrepancies between the dream environment and target environment because only a single dream environment exists.
Revised: It is easier for the controller to exploit any discrepancies between the dream environment and target environment because only a single dream environment exists.

Original: However, as we increase the inference dropout rate it becomes harder for the controller to exploit the dynamics model, suggesting that DDL is especially useful when it is difﬁcult to learn an accurate World Model.
Revised: However, as we increase the inference dropout rate it becomes harder for the controller to exploit the dynamics model, suggesting that DDL is especially useful when it is difﬁcult to learn an accurate World Model.

Original: 4. W HEN S HOULD D ROPOUT M ASKS BE R ANDOMIZED D URING C ONTROLLER T RAINING ?
Revised: 4. W HEN S HOULD D ROPOUT M ASKS BE R ANDOMIZED D URING C ONTROLLER T RAINING ?

Original: In this ablation study we evaluate when the dropout mask should be randomized during training of C .
Revised: In this ablation study we evaluate when the dropout mask should be randomized during training of C .

Original: We consider two possible approaches of when to randomize the masks.
Revised: We consider two possible approaches of when to randomize the masks.

Original: The ﬁrst case only randomizes the mask at the beginning of an episode ( episode randomization ).
Revised: The ﬁrst case only randomizes the mask at the beginning of an episode ( episode randomization ).

Original: The second case samples a new dropout mask at every step ( step randomization ).
Revised: The second case samples a new dropout mask at every step ( step randomization ).

Original: We also consider if it is effective to only apply dropout at inference time but not during RNN training (i.e., p infer > 0 , p train = 0 ).
Revised: We also consider if it is effective to only apply dropout at inference time but not during RNN training (i.e., p infer > 0 , p train = 0 ).

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: As can be seen, randomizing the mask at each step offers better returns on both real environments.
Revised: As can be seen, randomizing the mask at each step offers better returns on both real environments.

Original: Better returns in the real environment when applying step randomization comes from the fact that the controller is exposed to a much larger number ( > 1000 × ) of dream environments.
Revised: Better returns in the real environment when applying step randomization comes from the fact that the controller is exposed to a much larger number ( > 1000 × ) of dream environments.

Original: We also observe that applying step randomization without training the  model with dropout yields a weak policy on the real environment.
Revised: We also observe that applying step randomization without training the dynamics model with dropout yields a weak policy on the real environment.

Original: This is due to the randomization fundamentally changing the task.
Revised: This is due to the randomization fundamentally changing the task.

Original: Training the RNN with dropout ensures that at inference time the masked RNN is meaningful.
Revised: Training the RNN with dropout ensures that at inference time the masked RNN is meaningful.

Original: [Table]
Revised: [Table]

Original: 4. D OES THE N UMBER OF D REAM E NVIRONMENTS A FFECT R ETURNS ?
Revised: 4. D OES THE N UMBER OF D REAM E NVIRONMENTS A FFECT R ETURNS ?

Original: number of dropout masks in

Original: the real and dream CarRacingFixedN environment.

Original: Motivated by a similar experiment from Cobbe et al.
Revised: Motivated by a similar experiment from Cobbe et al.

Original: (2019), we measure the relationship between the number of inference dropout masks available to the dream environments and returns of the controller in the dream and real CarRacingFixedN environments.
Revised: (2019), we measure the relationship between the number of inference dropout masks available to the dream environments and returns of the controller in the dream and real CarRacingFixedN environments.

Original: Limiting the number of masks is the same as controlling for the number of possible environments.
Revised: Limiting the number of masks is the same as controlling for the number of possible environments.

Original: At the beginning of controller training we sample m sets 3 of masks according to p infer = 0 .
Revised: At the beginning of controller training we sample m sets 3 of masks according to p infer = 0 .

Original: 1 , yielding m dream environments.
Revised: 1 , yielding m dream environments.

Original: While optimizing the controller (Sections 3.2.2 and 3.2.3) we randomize the dream environment at every step by uniformly sampling from the m possible dream environments ( ˆ e j where 0 ≤ j < m ).
Revised: While optimizing the controller (Sections 3.2.2 and 3.2.3) we randomize the dream environment at every step by uniformly sampling from the m possible dream environments ( ˆ e j where 0 ≤ j < m ).

Original: Increasing m forces the controller to be capable of maximizing expected returns across a larger number of dream environments.
Revised: Increasing m forces the controller to be capable of maximizing expected returns across a larger number of dream environments.

Original: As can be seen from Figure 6, interestingly, a relatively small number of masks offers substantial improvements to the controller’s ability to generalize to the target environment.
Revised: As can be seen from Figure 6, interestingly, a relatively small number of masks offers substantial improvements to the controller’s ability to generalize to the target environment.

Original: We also notice that using more masks does not necessarily imply better generalization.
Revised: We also notice that using more masks does not necessarily imply better generalization.

Original: This suggests that there exists a smarter way to sample masks, aligned with similar observations in domain randomization (Mehta et al., 2020).
Revised: This suggests that there exists a smarter way to sample masks, aligned with similar observations in domain randomization (Mehta et al., 2020).

Original: C ONCLUSION
Revised: 4. I S D ROPOUT ’ S D REAM L AND M ORE THAN A R EGULARIZER ?

Revised: In this experiment we compare Dropout’s Dream Land with three regularization methods. First, we consider applying the standard use case of dropout ( 0 < p train < 1 and p infer = 0 ). Second, we consider a noisy variant of M , where every z adds a small amount of noise ( (cid:15) ∼ N (0 , 10 − 8 ) ).

Revised: Third, we apply L2 regularization to the weights (with L2 penalty λ

Revised: = 0 .

Revised: 001 ) during the training process of M .

Revised: In Table 5, we do not observe any of the standard regularization methods (L2, Dropout, Noise) to improve World Models generalization from dream environments to target environments.

Revised: L2 Regularized World Models and Dropout World Models can both be viewed as regularizers on M .

Revised: Noisy

Revised: World Models can be viewed as a regularizer on the controller C .

Revised: The strong returns on the real environment by DDL suggest that it is doing more than standard regularization techniques.

Revised: [Table]

Revised: 4. C OMPARISON TO E XPLICIT E NSEMBLE M ETHODS

Revised: In this experiment we compare Dropout’s Dream Land with two other approaches for randomizing the dynamics of the dream environment.

Revised: We consider using an explicit ensemble of a population of dynamics models.

Revised: Each environment in the population was trained with a different initialization and different mini-batches.

Revised: With the population of World Models we train a controller with Step Randomization and a controller with Episode Randomization.

Revised: Note that the training cost of dynamics models and RAM requirements at inference time scale linearly with the population size.

Revised: Due to the large computational cost we consider a population size of 2 .

Revised: In Table 6, we observe that neither Population World Models (PWM) Step Randomization or

Revised: Episode Randomization help close the Dream2Sim gap.

Revised: Episode Randomization does not help because the controller is forced to understand the hidden state ( h ) representation of every M in the population.

Revised: Step Randomization performs even worse than Episode Randomization because ontop of the previously state limitations, each of the dynamics models in the population is also forced to be compatible with the hidden state ( h ) representation of all other dynamics models in the population.

Revised: Even though we use a modest population size, PWM is plagued with issues that will not be ﬁxed by a larger population size.

Revised: DDL does not suffer from any of the previously stated issues and is also computationally cheaper because only one M must be trained as opposed to an entire population.

Revised: [Table]

Revised:  C ONCLUSION

Original: In this work we introduce an approach to improve controller generalization from dream environments to reality at little cost.
Revised: In this work we introduce an approach to improve controller generalization from dream environments to reality at little cost.

Original: To the best of our knowledge this is the ﬁrst work to bridge the reality gap between learned simulators and reality.
Revised: To the best of our knowledge this is the ﬁrst work to bridge the reality gap between learned simulators and reality.

Original: Future direction for this work could be modifying the dynamics model parameters in a targeted manner (Wang et al., 2019; 2020; Such et al., 2019).
Revised: Future direction for this work could be modifying the dynamics model parameters in a targeted manner (Wang et al., 2019; 2020; Such et al., 2019).

Original: This simple approach to generating different versions of a model could also be useful in committee-based methods (Settles, 2009; Sekar et al., 2020).
Revised: This simple approach to generating different versions of a model could also be useful in committee-based methods (Settles, 2009; Sekar et al., 2020).

Original: A A PPENDIX A. A RCHITECTURE D ETAILS
Revised: A A PPENDIX A. A RCHITECTURE D ETAILS

Original: We follow the same architecture setup as World Models.
Revised: We follow the same architecture setup as World Models.

Original: We adopt the following notation (Kim et al., 2020) to describe the VAE architecture.
Revised: We adopt the following notation (Kim et al., 2020) to describe the VAE architecture.

Original: Conv2D
Revised: Conv2D

Original: (a, b c) : 2D
Revised: (a, b c) : 2D

Original: Convolution layer with output channel size a , kernel size b , and stride c
Revised: Convolution layer with output channel size a , kernel size b , and stride c

Original: All use valid padding and relu activations.
Revised: All use valid padding and relu activations.

Original: T.Conv2D(a, b, c) :
Revised: T.Conv2D(a, b, c) :

Original: Transposed 2D-Convolution layer with output channel size a , kernel size b , stride c .
Revised: Transposed 2D-Convolution layer with output channel size a , kernel size b , stride c .

Original: The ﬁnal layer uses a sigmoid activation but every other layer uses relu activations.
Revised: The ﬁnal layer uses a sigmoid activation but every other layer uses relu activations.

Original: LSTM(a) : LSTM layer with a units.
Revised: LSTM(a) : LSTM layer with a units.

Original: Dense(a) : Fully Connected layer with output size a followed by a relu activation.
Revised: Dense(a) : Fully Connected layer with output size a followed by a relu activation.

Original: Linear(
Revised: Linear(

Original: a) : Linear layer with output size a
Revised: a) : Linear layer with output size a

Original: Reshape(a)
Revised: Reshape(a)

Original: : Reshape input to output size
Revised: : Reshape input to output size

Original: a .
Revised: a .

Original: [Table]
Revised: [Table]

Original: Conv2D(32, 4, 2) Conv2D(32, 4, 2)
Revised: Conv2D(32, 4, 2) Conv2D(32, 4, 2)

Original: Conv2D(64, 4, 2) Conv2D(64, 4, 2)
Revised: Conv2D(64, 4, 2) Conv2D(64, 4, 2)

Original: Conv2D(128, 4, 2) Conv2D(128, 4, 2)
Revised: Conv2D(128, 4, 2) Conv2D(128, 4, 2)

Original: Conv2D(256, 4, 2) Conv2D(256, 4, 2)
Revised: Conv2D(256, 4, 2) Conv2D(256, 4, 2)

Original: Reshape(1024)
Revised: Reshape(1024)

Original: Reshape(1024)
Revised: Reshape(1024)

Original: Linear(32), Linear(32)
Revised: Linear(32), Linear(32)

Original: Linear(64), Linear(64)
Revised: Linear(64), Linear(64)

Original: Dense(1024)
Revised: Dense(1024)

Original: Dense(1024)
Revised: Dense(1024)

Original: Reshape(1, 1, 1024) Reshape(1, 1, 1024)
Revised: Reshape(1, 1, 1024) Reshape(1, 1, 1024)

Original: T.Conv2D(128, 5, 2) T.Conv2D(128, 5, 2)
Revised: T.Conv2D(128, 5, 2) T.Conv2D(128, 5, 2)

Original: T.Conv2D(64, 5, 2) T.Conv2D(64, 5, 2)
Revised: T.Conv2D(64, 5, 2) T.Conv2D(64, 5, 2)

Original: T.Conv2D(32, 6, 2) T.Conv2D(32, 6, 2)
Revised: T.Conv2D(32, 6, 2) T.Conv2D(32, 6, 2)

Original: T.Conv2D(3, 6, 2)
Revised: T.Conv2D(3, 6, 2)

Original: T.Conv2D
Revised: T.Conv2D

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. E NVIRONMENT D ETAILS
Revised: A. E NVIRONMENT D ETAILS

Original: Means and standard deviations of returns achieved by the best controller (Section 3.2.3) in the target environment (Section 3.2.4) are reported based on 100 trials for CarRacing and 1000 trials for DoomTakeCover.
Revised: Means and standard deviations of returns achieved by the best controller (Section 3.2.3) in the target environment (Section 3.2.4) are reported based on 100 trials for CarRacing and 1000 trials for DoomTakeCover.

Original: 4 Returns in the dream environment are reported based on 1024 trials (Section 3.2.3) for both CarRacing and DoomTakeCover.
Revised: 4 Returns in the dream environment are reported based on 1024 trials (Section 3.2.3) for both CarRacing and DoomTakeCover.

Original: DoomTakeCover Environment For all tasks on this environment, we collect a training set oftrajectories and a test set of 100 trajectories.
Revised: DoomTakeCover Environment For all tasks on this environment, we collect a training set oftrajectories and a test set of 100 trajectories.

Original: A trajectory is a sequence of state ( z ), action ( a ), reward ( r ), and termination ( d ) tuples.
Revised: A trajectory is a sequence of state ( z ), action ( a ), reward ( r ), and termination ( d ) tuples.

Original: Both datasets are generated according to a random policy.
Revised: Both datasets are generated according to a random policy.

Original: Following the same convention as World Models (Ha & Schmidhuber, 2018), on the DoomTakeCover environment we concatenate z , h , and c as input to the controller.
Revised: Following the same convention as World Models (Ha & Schmidhuber, 2018), on the DoomTakeCover environment we concatenate z , h , and c as input to the controller.

Original: In (5), we set α d = 1 and α r =because the Doom reward function is determined entirely based off whether the controller lives or dies.
Revised: In (5), we set α d = 1 and α r =because the Doom reward function is determined entirely based off whether the controller lives or dies.

Original: CarRacing Environment For all tasks on this environment, the training set contains 5000 trajectories and the test set contains 100 trajectories.
Revised: CarRacing Environment For all tasks on this environment, the training set contains 5000 trajectories and the test set contains 100 trajectories.

Original: Both datasets are collected by following an expert policy with probability 0 .
Revised: Both datasets are collected by following an expert policy with probability 0 .

Original: 9 , and a random policy with probability 0 .
Revised: 9 , and a random policy with probability 0 .

Original: The expert policy was trained directly on the CarRacing-v0 environment and received an average return of 885 ± 63 acrosstrials.
Revised: The expert policy was trained directly on the CarRacing-v0 environment and received an average return of 885 ± 63 acrosstrials.

Original: In comparison, the performance of the random policy is − 53 ± 41 .
Revised: In comparison, the performance of the random policy is − 53 ± 41 .

Original: This is similar to the setup in GameGAN (Kim et al., 2020) on the Pacman environment which also used an expert policy.
Revised: This is similar to the setup in GameGAN (Kim et al., 2020) on the Pacman environment which also used an expert policy.

Original: For this environment, we set α d = α r = 1 in (5).
Revised: For this environment, we set α d = α r = 1 in (5).