BLAST: Latent Dynamics Models from Bootstrapping
Abstract
State-of-the-art world models such as DreamerV2 [15] have signiﬁcantly improved the capabilities of model-based reinforcement learning.
However, these approaches typically rely on a reconstruction loss to shape their latent representations, which is known to fail in environments with high ﬁdelity visual observations [8].
Previous work has found that when learning latent dynamics models without a reconstruction loss by using only the signal provided by the reward, the performance can also drop dramatically [1].
We present a simple set of modiﬁcation to DreamerV2 to remove its reliance on reconstruction inspired by the recent self-supervised learning method
Bootstrap Your Own Latent [10].
The combination of adding a stop-gradient to the posterior, using a powerful auto-regressive model for the prior, and using a slowly updating target encoder, which we call BLAST, allows the world model to learn from signals present in both the reward and observations, improving efﬁciency on our tested environment as well as being signiﬁcantly more robust to visual distractors.
Introduction
Many environments have simple dynamics and yet require millions or billions of potentially expensive agent interactions to be solved using deep reinforcement learning (DRL) algorithms.
One promising way to improve the sample efﬁciency of DRL is by learning a model of the environment.
Then learning a policy within this model.
Model-based RL (MBRL) is particularly efﬁcient when learning a world model is relatively easy compared to learning the policy.
Luckily, the state of the art in generative modeling is constantly improving, and as new techniques are discovered, these can bring signiﬁcant improvements to our ability to learn world models for MBRL, making it a viable approach for learning a growing set of complex tasks
With recent progress in generative modeling, MBRL can solve control tasks from pixels and master several Atari games.
However, MBRL still struggles to solve problems with highly complex observations.
This is largely because most MBRL techniques today rely on reconstruction to learn their models, where the world model is trained to predict future frames at a pixel level.
This makes
MBRL susceptible to changes in how an environment is rendered.
While an environment may have fundamentally simple dynamics, it can be rendered with high ﬁdelity textures, in 3D, or with a natural video playing in the background.
All of these changes have been shown to cripple the performance of
MBRL since a large amount of modeling capacity is taken up by modeling features that are irrelevant to the control task.
DreamerV2 [15], a state-of-the-art MBRL agent that learns a discrete world model, uses reconstruction to form its representations and thus suffers from this problem.
The authors describe a modiﬁcation that doesn’t use a reconstruction loss, where it simply learns a latent representation that can predict rewards and the distribution of the next latent state.
While learning a latent dynamics model this way should, in theory [9], be sufﬁcient, empirical studies have shown that across multiple tasks, the signal from predicting future observations is critical and without it, performance suffers dramatically [1, 15].
In our work, we show a simple way to regain this performance without using reconstruction or even contrastive [20, 22] losses.
Additionally, our method of leveraging image signals for representation learning without reconstruction is signiﬁcantly more robust to distractors than DreamerV2 and can perform well even when a video is displayed in the background of the environment.
Our method, which we call BLAST, consists of the following changes on top of DreamerV2:
• While DreamerV2 experiments with weighing the gradients that go to the prior and posterior in the KL loss, we show that by completely stopping the gradient from going to the posterior
the KL loss encourages representations that are informative of the predictable features of the observations.
• Since our method does not update the posterior to be predictable by the prior, we choose to use an auto-regressive prior [32] to better ﬁt the latent distribution.
• Inspired by similarities to recent advancements in self-supervised learning, we use techniques from Bootstrap Your Own Latent (BYOL) [10] such as a slowly updating target network for prior targets to stabilize training and using batch normalization [16] in the encoder .
In order to evaluate our method, we run experiments on a highly-customizable grid world environment as well as continuous control experiments on the DeepMind Control Suite (DMC) [30].
In our grid world experiments, we evaluate the performance of our method on several rendering modiﬁcations, including environments with a small agent sprite as well as video backgrounds.
We show empirically that DreamerV2 performs poorly on these environments while BLAST can learn a strong world model.
On DMC, we ﬁnd that BLAST can match the performance of several other MBRL agents without relying on reconstruction or contrastive losses.
We conduct an extensive ablation study on the proposed changes and ﬁnd that while all of the changes improve performance, batch normalization makes a substantial difference in several environments.
Overall, BLAST represents a simple set of changes to DreamerV2 that enables practical reconstruction-free MBRL.
BLAST: Bootstrapped LAtents for Simulating Trajectories 2.1 Problem Formulation
In reinforcement learning, an agent acts in an unknown environment and optimizes its actions to increase its reward.
In this work, we assume the agent is interacting with a POMDP, or partially observable Markov decision process.
A POMDP is deﬁned by a tuple ( S, A, T, R, γ, Ω , O ) .
S is a set of states; A is a set of actions; the transition probabilities T : S ×
A × S → [0 , 1] deﬁne the probability of the environment transitioning from state s to state s (cid:48) given that the agent acts with action a ; the reward function R : S ×
A → R deterministically maps a state-action transition to a real number; 0 ≤ γ ≤ 1 is the discount factor, which controls how much an agent should value rewards sooner rather than later.
Since a POMDP is partially observable, it also includes Ω , a set of observations, and O : S × O → [0 , 1] , a set of conditional probabilities deﬁning how likely a speciﬁc observation is for an underlying state.
The agent only receives observations; the true state of the system is unknown and can only be inferred using the observation and action history, the set of which we will refer to as belief states B .
An agent acts in the POMDP with a policy π : B ×
A → [0 , 1] , which determines the probability that it takes action a while in belief state b .
1 ^ z zhr
KL 2 ^ z zhr
KL 1 ^ z zh
KL ^ ^ r 1 ^ aooo
DV z 1 ^ (1) z
1 ^ (2) z
1 ^ (3) z
1 ^ (4)
BLAST
The expected return of a policy is denoted:
[Equation]
(cid:34)(cid:88)
[Equation]
A fundamental challenge in RL is that we do not have access to the ground truth environment dynamics.
Instead, an agent interacts with the POMDP by taking actions and receiving rewards.
Since this interaction can be costly in environments that are expensive to simulate or in real life,
MBRL is seen as a promising approach to improving sample efﬁciency, since it may be faster to learn to model the dynamics of the POMDP than to learn a policy through interaction.
2.2 Modeling Dynamics in a Latent Space
As in prior works [12, 14, 13, 15], we wish to learn a latent dynamics model , where a model is trained to predict the future conditioned on actions in a compact latent space rather than in observation space.
By predicting the future in this latent space, a latent dynamics model avoids the memory and compute intensive task of having to predict each frame when simulating the environment, allowing thousands of imagined trajectories to be simulated in parallel on a GPU.
A policy can also be learned directly on this latent state, potentially accelerating policy learning since the policy does not have to directly deal with high dimensional visual inputs.
Most prior works use reconstruction to form the latent representation.
While the observations contain important signals that can accelerate learning, precisely reconstructing image observations has many disadvantages.
Losses in pixel space (e.g. mean squared error) typically do not place the most weight on reconstructing the most “important” elements in the frame.
World models learned using reconstruction can fail to learn when features important to acting are represented by only a small percentage of pixels (e.g. the ball in Pong).
In high ﬁdelity or real world scenes where most pixels are taken up by uncontrollable elements like the clouds or a billboard, reconstruction is not a feasible approach.
2.3 Dreamer without Reconstruction
In this work, we build off of DreamerV2, a state-of-the-art MBRL algorithm.
In Dreamer [13, 15], a latent dynamics model called a Recurrent State-Space Model (RSSM)
[13] is learned.
This model consists of:
[Equation]
The world model is trained by minimizing the following loss, which corresponds to an ELBO of a hidden Markov model conditioned on the action sequence, making the RSSM a sequential VAE [15, 18]:
[Equation]
In the original work [13, 14, 15], various representation learning ablations were done, including the removal of the image log loss.
However, without image gradients to help form representations,
Dreamer fails to learn an accurate world model.
While this naive reward-only version of Dreamer might not work well in practice, there have been several works that learn representations in Dreamer without reconstruction, primarily through contrastive learning.
The original Dreamer paper [14] used contrastive learning to learn a representation that has maximal mutual information with the encoded observation.
Temporal Predictive Coding (TPC) [20] proposes to augment this loss with a contrastive approach that operates between timesteps, encouraging representations of predictable elements.
While Dreamer with contrastive representations did not achieve performance on par with reconstruction-based representations, TPC is competitive with Dreamer with reconstruction on several tasks.
Theoretically, TPC learns representations that are sufﬁcient for control, but it is unclear whether the loss is still a lower bound on the log probability of the data.
In practice, TPC requires carefully balancing four separate losses to avoid collapsed representations in practice.
2.3.1 Learning Bootstrapped Representations
In order to learn a world model with neither reconstruction nor contrastive losses, we note that there are two ways to optimize the KL loss in DreamerV2.
We can change the latent representations from the past to be more informative of future representations (forward) or we can change our future representation to be more predictable by the past (reverse).
DreamerV2 [15] introduces an additional hyperparameter called the KL balance, where a parameter α is used to scale the gradients going into the prior and posterior in the KL loss.
In experiments on Atari, DreamerV2 uses a KL balance of α = 0 . 8 , which updates both the posterior and prior distributions.
In this section, we argue that by completely stopping the gradients that update the future representation ( α = 1 ), we are able to learn representations not only from the reward signal, but from bootstrapping off of the information present in the embeddings of future observations.
By only allowing gradients to ﬂow into the prior and not the posterior, we recover a representation learning algorithm similar to BYOL [10], a self-supervised learning algorithm that has achieved strong performance on benchmarks without the use of the negative samples that are necessary for contrastive learning.
Intuitively, even the representation produced by our encoder at initialization has some information about the observation it is encoding.
By predicting this future representation, the past representations are encouraged to contain information that can help with the prediction task, which in turn increases the amount of information that is present in the representation.
In order to stabilize training, we use techniques from BYOL.
Primarily, we employ a slowly updating target encoder with parameters ξ to produce the posterior, resulting in the following loss:
[Equation]
The target encoder’s parameters ξ are updated as an exponential moving average of θ :
[Equation]
2.3.2 Auto-Regressive Prediction of Latent Representations
While BYOL uses continuous representations and predicts the mean, we would like to use our prediction model to make accurate predictions about future latent states, even in stochastic environments where they may be multi-modal.
Like in DreamerV2, [15], we use k discrete categorical variables for each latent representation.
Unlike DreamerV2, an independent prior will likely not be able to accurately ﬁt the distribution of future latents since we do not update the encoder to be predictable when we use a KL balance of α = 1 .
Therefore, for the prior p θ ( z t | h t ) we use a powerful auto-regressive model so that we can accurately capture the joint distribution of these variables.
Since the prior predicts the distribution of future observations encoded by the target representation encoder, during training we set the inputs to the prior model to come from both the online encoder and the target encoder.
2.3.3 Batch Normalization
It has been noted that while batch normalization [16] is not strictly necessary to make BYOL work [26].
However, others have proposed that batch normalization plays a similar role in BYOL as negative samples in contrastive learning.
This stems from the fact that both batch normalization and negative samples effectively push representations within a batch apart from each other [7].
In our experiments, we found that batch normalization can dramatically improve stability and performance.
Experiments
We run experiments to evaluate the following hypotheses:
• (H1)
The combination of proposed changes (BLAST) enable the learning of a strong world model without the use of reconstruction.
• (H2) BLAST is more robust to changes in environment rendering compared to DreamerV2.
• (H3)
The most effective combination of the proposed changes is the combination of all ﬁve, which we call BLAST.
3.1 Environment
We primarily run experiments on a modiﬁed grid world environment as well as the DeepMind Control
Suite (DMC) [30].
We choose to use the dynamic obstacles environment in gym-minigrid [4].
In this environment, three obstacles randomly move to adjacent tiles each time-step and the agent must navigate around them to reach a goal without colliding with any obstacles.
This environment was primarily chosen since it is easy to modify while retaining characteristics that make it a good test-bed for testing world models, such as environment stochasticity and sparse reward.
We also chose to evaluate on DMC to allow comparison to prior work which tested on this benchmark.
3.1.1 Environment Rendering Modiﬁcations
We modiﬁed the base environment MiniGrid-Dynamic-Obstacles-6x6-v0 in several ways to evaluate robustness to different rendering styles.
While prior works primarily evaluate robustness to videos or images placed in the backgrounds of visual environments, we want to evaluate robustness to other scenarios that could potentially harm reconstruction-based agents in order to evaluate (H2).
Examples observations from these environments are shown in Figure 2.
• color_direction :
A different color is used to represent each of the possible directions that the agent could be facing.
Scenario: uncertainty in reconstruction can be confused with another possible observation.
• smaller_agent : The agent is rendered as a triangle using one quarter of the pixels as in the original environment.
Scenario: important elements have a low weight in the reconstruction loss.
• random_frames : The background is set to a random frame of a ﬁxed video at each timestep.
Scenario: environment observations have temporally uncorrelated and uncontrollable elements in the background.
• video : The background is set to a random frame of a ﬁxed video upon resetting the environment.
The video is then displayed in order in subsequent steps.
Scenario: environment observations have temporally correlated and uncontrollable elements in the background.
For all experiments, we used a ﬁxed horizon of 100 time-steps.
The return is the number of times the agent reaches the green square in the episode minus the number of times it collides with an obstacle.
For video background we use a video drawn from the the kinetics400 [17] driving class across all experiments.
3.2 Experimental Setup
We ran all experiments using code forked from the ofﬁcial DreamerV2 [15] implementation.
The code was modiﬁed to allow the use of batch normalization, a separate target encoder with weights set to an exponentially moving average of online weights, and an autoregressive prior within the RSSM.
This allows us to conﬁgure DreamerV2 with any subset of our proposed changes and fairly compare the different conﬁgurations.
We opted to ﬁx hyperparameters for individual environments rather than individually tune each conﬁguration of the agent since we found that generally, hyperparameters that worked well on the base DreamerV2 agent also worked well on BLAST.
More details about hyperparameter and how to reproduce our experiments can be found in appendix A.1.
For our plots, we plot the mean over 5 seeds for grid world environments and 6 seeds 2 for DMC tasks.
Error bars represent standard deviation.
3.3 Summary of Modiﬁcations
• (-recon)
We remove the reconstruction loss from DreamerV2 by setting its weight to zero.
• (+SG) We add a stop gradient to the posterior in the KL loss by setting a KL balance of α = 1 . • (+EMA)
We use an exponential moving average for the encoder when we compute the posterior in the KL loss.
• (+BN)
We add batch normalization to the encoder.
• (+AR)
We use an autoregressive prior in the RSSM to better ﬁt the discrete latent distribution.
3.4 Grid World Experiments
Figure 3 shows how performance differs on the grid world environments when the proposed modiﬁcations are added to DreamerV2 to create BLAST.
DreamerV2 works well on the unmodiﬁed
Time Steps (10 4 )
R e t u r n s unmodiﬁed
Time Steps (10 4 )   color direction
Time Steps (10 4 )   smaller agent
Time Steps (10 4 )   random frames
Time Steps (10 4 )   video
BLAST (Ours)
DreamerV
[Figure]
environment but struggles to learn in the presence of most of the rendering modiﬁcations.
BLAST is able to achieve strong performance on all of the rendering modiﬁcations, conﬁrming our hypothesis that BLAST would be more robust to adversarial rendering (H2).
We found that only BLAST (+AR) and BLAST without an autoregressive prior (+BN) consistently performed well.
We note that there are some environments such as color_direction where the use of batch normalization is vital to achieve strong performance.
We also display training curves in Figure 7 in the appendix, which shows that BLAST can be signiﬁcantly more sample efﬁcient than DreamerV2 with the same hyperparameters.
We believe this is primarily due to not needing to train a decoder.
3.5 DeepMind Control Suite
Our experiments on the modiﬁed grid world environments conﬁrm that BLAST is signiﬁcantly more robust to rendering changes than DreamerV2.
In order to compare with prior state of the art model based RL algorithms, we ran continuous control experiments on several standard DMC environments.
Figure 4 shows that BLAST can achieve performance that is essentially on par with vanilla DreamerV and Dreamer.
Surprisingly, we found that batch normalization had an even larger effect on these environments, with several tasks achieving practically zero return until batch normalization was added.
We also found that the autoregressive prior did not have a signiﬁcant effect on performance in
DMC environments, likely due to the deterministic nature of the environment.
Figure 4 also shows that BLAST achieves comparable performance to TPC, both in terms of asymptotic performance as well as sample complexity.
Despite using neither reconstruction nor contrastive losses, BLAST achieves performance that is on par with existing state-of-the-art MBRL approaches on DMC tasks.
[Figure]
r e t u r n
[Figure]
TPC
Related Work
Self-supervised learning In recent years, contrastive methods have largely overtaken their unsupervised counterparts as the state-of-the-art technique for visual representations learning [3, 23].
These methods learn their representation using pairs of views on the same underlying data.
These pairs come in two ﬂavors: positive pairs where the views are of the same underlying datum and negative pairs where they are not.
Contrastive methods produce a latent space where positive pairs converge and negative pairs spread apart.
With the correct setup, this technique can be shown to optimize a lower bound on the mutual information between the input images and the latent space [23].
Recent work in the self-supervised learning literature has shown that the negative samples that characterize contrastive learning are unnecessary to learn good representations.
Like contrastive learning approaches, these works employ matching latent representations.
They differentiate by disposing of negative pairs and thus remove the need to store a bank of negative samples and function well with lower batch sizes [2, 10].
Latent dynamics models Traditional MBRL methods work best on low dimensional state representations [13, 5].
One of the most successful technique for scaling MBRL has been the use of latent dynamics models.
In a latent dynamics model, high-dimensional observations are encoded into a compact latent state.
Then these latent states are used to predict future latent states, rewards and discount factors.
This process allows for training to proceed entirely within the latent dynamics model [12, 14, 13, 15].
These models generally achieve this accuracy by using variational methods that are primarily or entirely driven by a reconstruction loss.
In theory, this reconstruction loss ensures that all the information contained in the high-dimensional observations ﬁnds its way into the latent states and in theory helps to ensure the world model’s accuracy.
Robust world models Though effective for simple environments, latent dynamics models that rely on reconstruction can fail to learn in more realistic settings [8, 22, 20, 15].
They can be distracted by irrelevant information in their inputs, like video backgrounds, and have issues learning to model small or fast-moving objects [8, 22, 20, 33].
Several papers attempt to alleviate these issues.
Paster et al.
[25] learns a latent world model using inverse dynamics models to model only controllable features in an environment, but the method is primarily used on goal-directed tasks.
Fu et al.
[8] attempts to solve the problem of distracting background imaginary by learning a partitioned state space using an adversarial objective.
Their objective tries to ensure that the reward is only recoverable from half of the state space.
The method can handle distracting backgrounds.
However, it does not handle cases where the environment combines relevant and irrelevant information in a way that a simple pixel mask cannot partition.
Perhaps most similar to our work is that of Nguyen et al.
[20], Okada and Taniguchi [22] and Ma et al.
These works attempt to learn world models purely using contrastive losses.
Nguyen et al.
[20] ﬁnds improved performance over DreamerV1 on the DeepMind Control Suite [30].
In addition to this, despite not explicitly discouraging irrelevant information from entering the latent representations, the authors ﬁnd substantially improved performance on video background environments.
Similarly, Okada and Taniguchi [22] ﬁnds that their model sees improved results over
DreamerV1 in environments that require the agent to perceive ﬁne details.
While these works learn representations that can avoid being distracted and resolve environments with ﬁne detail, they still require negative samples.
BLAST does not.
Learning from just rewards Another successful method of learning world models stems from the
Value Prediction Networks and MuZero [29, 21, 27].
These methods learn their environment model by predicting just rewards, values and actions derived by planning.
While these MBRL methods have proven spectacularly successful [27], they cannot handle stochastic or partially observable environments and are computationally expensive to train since they do not make use of the training signal provided by observations [24, 28].
Recent work has attempted to address these issues in MuZero by augmenting the method with pre-trained embedding that tries to represent the environment’s stochasticity [24].
However, this method again relies on reconstruction to produce these embeddings in the form of a Vector Quantised-Variational AutoEncoder [31].
Finally, there are several works that attempt to address the problem of representation learning with a focus on predicting future latent representations and bisimulation metrics [6].
While Zhang et al.
[34], Gelada et al.
[9], Guo et al.
[11] learn dynamics models, they are only used for representation learning and never for planning.
Discussion
We present BLAST, a modiﬁcation of DreamerV2 which adds a stop-gradient to the posterior, a powerful auto-regressive prior, and a slowly updating target network to learn representations without using reconstruction.
BLAST performs far better than DreamerV2 on a range of differently rendered grid world environments, including ones with a video embedded in the background where DreamerV fails entirely to learn.
We empirically justify our modiﬁcations and show evidence that adding the stop-gradient encourages the latent representations to contain more information about the observation.
While BLAST shows strong performance in our grid world environments as well as on continuous control tasks, we acknowledge several limitations and opportunities for future work.
The primary limitation of this work is the need for an expressive prior such as an autoregressive model in stochastic environments.
Since autoregressive models are considerably slower than the independent prior used in DreamerV2, our model takes longer to run simulations.
We also believe that more work should be done to differentiate self-predictive approaches to representation learning such as BLAST to contrastive approaches in order to provide a stronger recommendation to practitioners looking to use reconstruction-free world models.
Acknowledgements
We gratefully acknowledge funding from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs Program, and Microsoft Research.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute for Artiﬁcial Intelligence ( www.vectorinstitute.ai/partners ).
A Appendix A.1 Experimental Details
For all of our experiments we built on top of the open source implementation of DreamerV provided by the authors at https://github.com/danijar/dreamerv2.
Hyperparameters for the grid world experiments were optimized for maximum performance for DreamerV2 across the tested environments.
The hyperparameters with the greatest effect on DreamerV2’s performance were the dimension of the latent state rssm.stoch and the scale of the reward loss loss_scales.reward .
We found that BLAST generally worked well for these same hyperparameters and therefore we ﬁxed the hyperparameters across all of our conﬁgurations.
For DMC experiments, we used the default hyperparameters for DreamerV2 on DMC and similarly kept them ﬁxed across all of the tasks and conﬁgurations.
A.2 Hyperparameters
Gym-MiniGrid prefill 10, train_every steps 5e pred_discount
False discount γ 0.
rssm.hidden rssm.deter rssm.stoch rssm.discrete actor_grad reinforce loss_scales.kl 0.
loss_scales.reward 100.
reward_clamp tanh kl.balance 0.
DMC prefill train_every steps 1e pred_discount
False discount γ 0.
rssm.hidden rssm.deter rssm.stoch rssm.discrete actor_grad dynamics loss_scales.kl 1. loss_scales.reward 1.
reward_clamp none kl.balance 0.
BLAST kl.balance 1.
encoder.norm bn rssm.obs_out_norm bn target_ema 0.
target_input_p 0.
rssm.ar_steps same as rssm.stoch use_target_encoder true
[Figure]
A.3 Learning Representations with Only Latent Prediction
Due to its similarity to self-supervised learning algorithms like BYOL [10]
[Figure]
we hypothesize that the signal from predicting future latents may be sufﬁcient for learning representations in BLAST.
To test this
[Figure]
we stopped the gradients from the reward and discount prediction heads of BLAST so that the world model and observation representation would only be updated with the KL loss.
As demonstrated in Figure
[Figure]
BLAST, with no reward or reconstruction gradients
[Figure]
still manages to learn a surprisingly strong policy on the unmodiﬁed and random frames environment, empirically showing that the latent prediction loss alone can be enough to learn a world model
[Figure]
A.4 Training Curves
[Figure]
grid world environments.
[Figure]
A.5 Visual Reconstruction with Different Representations
In order to visually inspect the quality of the representations learned by DreamerV2 without reconstruction
[Figure]
we trained a reconstruction network on top of the learned representations.
Figure 8 shows that even on an environment modiﬁed with a video playing in the background, BLAST improves the quality of representations and allows for a clearer reconstruction
[Figure]
r ec o n s tr u c t i o n o r i g i n a l i m ag e unmodiﬁed
Ours) r ec o n s tr u c t i o n o r
i
g
i n a l i m ag e unmodiﬁed & -recon r ec o n s tr u c t i o n o r i g i n a l i m ag e video & BLAST (Ours) r ec o n s tr u c t i o n o r i g
i n a l i m ag
e
[Figure] [Figure]
dmc_cartpole_swingup
DV 6 seeds -recon 6 seeds +SG 6 seeds +EMA 5 seeds +BN 4 seeds +AR 5 seeds dmc_cheetah_run
DV 6 seeds -recon 6 seeds +SG 3 seeds +EMA 4 seeds +BN 5 seeds +AR 6 seeds dmc_cup_catch DV 6 seeds -recon 6 seeds +SG 2 seeds +EMA 6 seeds +BN 6 seeds +AR 6 seeds dmc_hopper_hop
DV 6 seeds -recon 6 seeds +SG 6 seeds +EMA 6 seeds +BN 6 seeds +AR 4 seeds dmc_pendulum_swingup DV 6 seeds -recon 6 seeds +SG 6 seeds +EMA 6 seeds +BN 6 seeds +AR 4 seeds dmc_walker_run
DV 6 seeds -recon 6 seeds +SG 4 seeds +EMA 2 seeds +BN 5 seeds +AR 5 seeds