S YNTHETIC E XPERIENCE R EPLAY
A BSTRACT
A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results.
In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay , whereby a dataset of past experiences is used to train a policy or value function.
However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited.
Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training.
In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (S YNTH ER), a diffusion-based approach to arbitrarily upsample an agent’s collected experience.
We show that S YNTH ER is an effective method for training RL agents across offline and online settings.
In offline settings, we observe drastic improvements both when upsampling small offline datasets and when training larger networks with additional synthetic data.
Furthermore, S YNTH ER enables online agents to train with a much higher update-to-data ratio than before, leading to a large increase in sample efficiency, without any algorithmic changes .
We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data.
I NTRODUCTION
In the past decade, the combination of large datasets (Deng et al., 2009; Schuhmann et al., 2022) and ever deeper neural networks (Krizhevsky et al., 2012; He et al., 2015; Vaswani et al., 2017;
Devlin et al., 2018) has led to a series of more generally capable models (Radford et al., 2019;
Brown et al., 2020; Ramesh et al., 2022).
In reinforcement learning (RL, Sutton & Barto (2018)), agents typically learn online from their own experience.
Thus, to leverage sufficiently rich datasets,
RL agents typically make use of experience replay (Mnih et al., 2015; Fedus et al., 2020), where training takes place on a dataset of recent experiences.
However, this experience is typically limited, [Figure] Data Augmentation Methods [Figure]
[Figure]
unless an agent is distributed over many workers which requires both expensive computational cost and sufficiently fast simulation (Espeholt et al., 2018; Kapturowski et al., 2019).
Another approach for leveraging broad datasets for training RL policies is offline RL (Agarwal et al., 2020; Levine et al., 2020), whereby behaviors may be distilled from previously collected data either via behavior cloning (Schaal, 1996), off-policy learning (Kumar et al., 2020; Fujimoto & Gu, 2021) or model-based methods (Yu et al., 2020; Kidambi et al., 2020; Lu et al., 2022a).
Offline data can also significantly bootstrap online learning (Hester et al., 2017; Wagenmaker & Pacchiano, 2022;
Ball et al., 2023), however, it is a significant challenge to apply these methods when there is a mismatch between offline data and online environment.
Thus, many of the successes rely on toy domains with transfer from specific behaviors in a simple proprioceptive environment.
Whilst strong results have been observed in re-using prior data in RL, appropriate data for particular behaviors may simply not exist and thus this approach falls short in generality.
We thus consider an alternative approach—rather than passively reusing data, we seek to leverage tremendous progress in generative modeling to generate a large quantity of new, synthetic data.
While prior work has considered upsampling online RL data with VAEs or GANs (Huang et al., 2017; Imre, 2021; Ludjen, 2021), given the dramatic progress in generative models in recent years, we propose making use of diffusion generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Karras et al., 2022), which unlocks significant new capabilities.
Our approach, which we call Synthetic Experience Replay , or S YNTH ER, is conceptually simple, whereby given a limited initial dataset, we can arbitrarily upsample the data for an agent to subsequently use as if it was real experience.
Therefore, in this paper, we seek to answer a simple question: Can the latest generative models replace or augment traditional datasets in reinforcement learning?
To answer this, we consider the following settings: offline RL where we entirely replace the original data with data produced by a generative model, and online RL where we upsample recent experiences to broaden the training data available to the agent.
In both cases, S YNTH ER leads to a drastic improvement in training with synthetic data, obtaining performance comparable to that of agents trained with substantially more real data.
Furthermore in certain offline settings, the additional data enables effective training of larger policy and value networks, resulting in higher performance by alleviating the representational bottleneck.
We thus believe this paper presents sufficient evidence that given additional scale, our approach could enable entirely new efficient training strategies for RL agents.
To summarize, the contributions of this paper are:
• We propose S YNTH ER in Section 3, a diffusion-based approach that allows one to generate synthetic experiences and thus arbitrarily upsample data for any reinforcement learning algorithm utilizing experience replay.
• We validate the synthetic data generated by S YNTH ER in offline settings in Section 4.1, presenting the first generative approach to show parity with real data on the standard D4RL offline datasets with a wide variety of algorithms.
Furthermore, we observe considerable improvements from upsampling for small offline datasets in Section 4.1.1 and scaling up network sizes in Section 4.1.2.
• We show how S YNTH ER can arbitrarily upsample an online agent’s training data in Section 4.2 by continually training the diffusion model.
This allows us to significantly increase an agent’s update-to-data (UTD) ratio matching the efficiency of specially designed dataefficient algorithms without any algorithmic changes .
B ACKGROUND
In this section, we introduce the (online and offline) reinforcement learning framework and diffusion generative models.
2. R EINFORCEMENT L EARNING
We model the environment as a Markov Decision Process (MDP, Sutton & Barto (2018)), defined as a tuple M =
( S , A , P, R, ρ 0 , γ ) , where S and A denote the state and action spaces respectively,
P ( s ′ | s, a ) the transition dynamics, R ( s, a ) the reward function, ρ 0 the initial state distribution, and γ ∈ (0 , 1) the discount factor.
The goal in reinforcement learning is to optimize a policy π ( a | s ) that maximizes the expected discounted return E π,P,ρ
[ (cid:80) ∞ t =
γ t R ( s t , a t )] .
2. O FFLINE R EINFORCEMENT L EARNING
In offline RL (Levine et al., 2020), the policy is not deployed in the environment until test time.
Instead, the algorithm only has access to a static dataset D env = { ( s t , a t , r t , s t +1 ) }
Tt =1 , collected by one or more behavioral policies π b .
We refer to the distribution from which D env was sampled as the behavioral distribution (Yu et al., 2020).
In some of the environments we consider, the environment may be finite horizon or have early termination.
For example, in the Hopper environments in the standard D4RL datasets (Fu et al., 2020), the episode ends if the robot falls over.
For these datasets, the transition tuple also contains a terminal flag d t which is typically represented as a 0-1 binary variable where d t = 1 indicates the episode ended early at timestep t and d t = 0 otherwise.
2. D IFFUSION M ODELS
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a class of generative models inspired by non-equilibrium thermodynamics that learn to iteratively reverse a forward noising process and generate samples from noise.
Given a data distribution p ( x ) with standard deviation σ data , we consider noised distributions p ( x ; σ ) obtained by adding i.i.d.
Gaussian noise of standard deviation σ to the base distribution.
The forward noising process is defined by a sequence of noised distributions following a fixed noise schedule σ 0 = σ max > σ 1 > · · · > σ N = 0 .
When σ max ≫ σ data , the final noised distribution p ( x ; σ max ) is essentially indistinguishable from random noise.
Karras et al.
(2022) consider a probability-flow ODE with a corresponding continuous noise schedule σ ( t ) that maintains the desired distribution p as x evolves through time given by Equation (1).
[Equation]
where the dot indicates a time derivative and ∇ x log p ( x ; σ ( t )) is the score function (Hyv¨arinen, 2005), which points towards the data at a given noise level.
Infinitesimal forward or backward steps of this ODE either nudge a sample away or towards the data.
Karras et al.
(2022) consider training a denoiser D θ ( x ; σ ) on an L2 denoising objective:
[Equation]
and then use the connection between score-matching and denoising (Vincent, 2011) to obtain ∇ x log p ( x ; σ ) =
( D θ ( x ; σ ) − x ) /σ 2 .
We may then apply an ODE (or SDE as a generalization of Equation (1)) solver to reverse the forward process.
In this paper, we train our diffusion models to approximate the online or offline behavioral distribution.
S YNTHETIC E XPERIENCE R EPLAY
In this section, we introduce Synthetic Experience Replay (S YNTH ER), our approach to upsampling an agent’s collected experience using diffusion.
We begin by describing the simpler training process used for offline reinforcement learning and then how that may be adapted to the online setting by continually training the diffusion model.
3. O FFLINE S YNTH ER
For offline reinforcement learning, we take the data distribution of the diffusion model p ( x ) to simply be the offline behavioral distribution.
For the proprioceptive environments we consider, the full transition is low-dimensional compared with typical pixel-based applications of diffusion.
Therefore, the network architecture is an important design choice and we elect to use a residual MLP denoising (Tolstikhin et al., 2021)
network, with detailed hyperparameters given in Appendix B.
We validate our diffusion model in Figure 6 in Appendix A on the D4RL datasets by showing that the synthetic data closely matches the original data when comparing the marginal distribution over each dimension.
This uses the same denoising network we use in our main evaluation in Section 4.
Furthermore, in Appendix C.3, we comprehensively show that the quality of the samples from the diffusion model is significantly better than with prior generative models such as VAEs (Kingma &
Welling, 2014) and GANs (Goodfellow et al., 2014).
An immediate advantage of sampling data from a generative model is compression.
In Table 1, we compare the memory requirements of S YNTH ER and the original data by the number of 32-bit floating point numbers used by each for some sample D4RL (Fu et al., 2020) datasets.
For the original data, this simply scales linearly with the size of the dataset.
On other hand, S YNTH ER amortizes this in the number of parameters in the denoising network, resulting in a high level of dataset compression, at the cost of sampling speed.
This property was also noted in the continual learning literature with generative models summarizing previous tasks (Shin et al., 2017).
As we discuss in Appendix B.2, sampling is fast with 100K transitions taking around 90 seconds to generate.
[Table]
3. O NLINE S YNTH ER
S YNTH ER may be used to upsample an online agent’s experiences by continually training the diffusion model on new experiences.
We provide pseudocode for how to incorporate S YNTH ER into any online replay-based RL agent in Algorithm 1 and visualize this in Figure 2.
Concretely, a diffusion model is periodically updated on the real transitions and then used to populate a second synthetic buffer.
The agent may then be trained on a mixture of real and synthetic data sampled with ratio r .
For the results in Section 4.2, we simply set r = 0 .
5 analogously to Ball et al.
The synthetic replay buffer may also be configured with a finite capacity to prevent overly stale data.
[Table]
E MPIRICAL E VALUATION
In our empirical evaluation
[Table]
we evaluate S YNTH ER across a wide variety of offline and online settings.
We first validate our approach on offline RL, where we entirely replace the original data, and further show large benefits from upsampling small offline datasets.
Next, we show that S YNTH ER leads to large improvements in sample efficiency in online RL where we upsample recent experiences to broaden the training data available to the agent.
4. O FFLINE D4RL E VALUATION
We first verify that synthetic samples from S YNTH ER faithfully model the underlying distribution from the original D4RL (Fu et al., 2020) datasets.
To do this, we evaluate S YNTH ER in combination with 3 distinct SOTA offline RL algorithms: TD3+BC (Fujimoto & Gu (2021), explicit policy regularization), IQL (Kostrikov et al.
(2022), expectile regression), and EDAC (An et al.
(2021), uncertainty-based regularization) on an extensive selection of D4RL datasets.
We consider the MuJoCo (Todorov et al., 2012) locomotion (halfcheetah, walker2d, and hopper) and maze2d environments.
For these experiments, we use a residual MLP denoising network with full hyperparameters given in Appendix B. All datasets share the same training hyperparameters, with some larger datasets using a wider network.
For every dataset, we upsample the original dataset to 5M samples ; we justify this choice in Appendix C.1.
We show the final performance in Table 2.
Our results show that we achieve at least parity for all groups of environments and algorithms as highlighted in the table, regardless of the precise details of each algorithm .
We note significant improvements to maze2d environments, which are close to the ‘best’ performance as reported in
CORL (Tarasov et al., 2022) (i.e., the best iteration during offline training) rather than the final performance.
We hypothesize this improvement is largely due to increased data from S YNTH ER, which leads to less overfitting and increased stability.
For the locomotion datasets, we largely reproduce the original results, which we attribute to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large.
However, as detailed in Table 1, S YNTH ER allows the effective size of the dataset to be compressed significantly.
4.1. U PSAMPLING FOR S MALL D ATASETS
Next, we investigate the benefit of S YNTH ER for small offline datasets and compare it to canonical ‘explicit’ data augmentation approaches (Laskin et al., 2020; Ball et al., 2021).
Concretely, we wish to understand whether S YNTH ER generalizes and generates synthetic samples that improve
[Figure]
policy learning compared with explicitly augmenting the data with hand-designed inductive biases.
We focus on the walker2d (medium, medium-replay/mixed, medium-expert) datasets in D4RL and uniformly subsample each at the transition level.
We subsample each dataset proportional to the original dataset size so that the subsampled datasets approximately range from 20K to 200K samples.
As in Section 4.1, we then use S YNTH ER to upsample each dataset to 5M transitions.
Our denoising network uses the same hyperparameters as for the original evaluation in Section 4.1.
In Figure 3, we can see that for all datasets, S YNTH ER leads to a significant gain in performance and vastly improves on explicit data augmentation approaches.
For explicit data augmentation, we select the overall most effective augmentation scheme from Laskin et al.
(2020) (adding Gaussian noise of the form ϵ ∼ N (0 , 0 . 1) ).
Notably, with S YNTH ER we can achieve close to the original levels of performance on the walker2d-medium-expert datasets starting from only 3% of the original data .
In Figure 1a, we methodically compare across both additive and multiplicative versions of RAD, as well as dynamics augmentation (Ball et al., 2021) on the 15% reduced walker medium-replay dataset.
[Figure]
Why is S YNTH ER better than explicit augmentation?
To provide intuition into the efficacy of S YNTH ER over canonical explicit augmentation approaches, we compare the data generated by S YNTH ER to that generated by the bestperforming data augmentation approach in Figure 1a, namely additive noise.
We wish to evaluate two properties: 1) How diverse is the data?
2) How accurate is the data for the purposes of learning policies?
To measure diversity, we measure the minimum L2 distance of each datapoint from the dataset, which allows us to see how far the upsampled data is from the original data.
To measure the validity of the data, we follow Lu et al.
(2022a)
and measure the MSE between the reward and next state proposed by S YNTH ER with the true next state and action defined by the simulator.
We then plot both these values in a joint scatter plot to compare how they vary with respect to each other.
For this, we compare specifically on the reduced 15% subset of walker2d medium-replay as in Figure 1a.
As we see in Figure 4, S YNTH ER generates a significantly wider marginal distribution over the distance from the dataset, and generally produces samples that are further away from the dataset than explicit augmentations.
Remarkably, however, we see that these samples are far more consistent under the true environment dynamics.
Thus, S YNTH ER generates samples that have significantly lower dynamics MSE than explicit augmentations, even for datapoints that are far away from the training data.
This implies that a high level of generalization has been achieved by the S YNTH ER model, resulting in the ability to generate novel, diverse, yet dynamically accurate data that can be used by policies to improve performance.
4.1. S CALING N ETWORK S IZE
A further benefit we observe from S YNTH ER on the TD3+BC algorithm is that upsampled data can enable scaling of the policy and value networks leading to improved performance.
As is typical for
RL algorithms, TD3+BC uses a small value and policy network with 2 layers and width of 256, and a batch size of 256.
We consider increasing the size of both networks to be 3 layers and width(approximately 6x more parameters), and the batch size to 1024 to better make use of the upsampled data in Table 3.
We observe a large overall improvement of 12.2% for the locomotion datasets when using a larger network with synthetic data (Larger Network + S YNTH ER).
Notably when using the original data (Larger Network + Original Data), the larger network performs the same as the baseline.
This suggests that the bottleneck in the algorithm lies in the representation capability of the neural network and synthetic samples from S YNTH ER enables effective training of the larger network .
This could alleviate the data requirements for scaling laws in reinforcement learning (Adaptive Agent Team et al., 2023; Hilton et al., 2023).
However, for the IQL and EDAC algorithms, we did not observe an improvement by increasing the network size which suggests that the bottleneck there lies in the data or algorithm rather than the architecture.
[Table]
O NLINE E VALUATION
Finally
[Table]
we show that S YNTH ER can effectively upsample an online agent’s continually collected experiences on 3 environments from the DeepMind Control Suite (DMC, Tunyasuvunakool et al.
(cheetah-run, quadruped-walk
[Table]
and reacher-hard) and 3 environments the OpenAI
Gym
Suite (Brockman et al., 2016) (walker2d, halfcheetah
[Table]
and hopper).
We choose the base algorithm to be SAC (Haarnoja et al., 2018), a popular off-policy entropy-regularized algorithm [Table] and benchmark against a SOTA data-efficient variant of itself, REDQ (Chen et al., 2021).
REDQ uses an ensemble of 10 Q-functions and computes target values across a randomized subset of them during training.
By default, SAC uses an update-to-data ratio of 1 (1 update for each transition collected); the modifications to SAC in REDQ enable this to be raised to 20.
Our method, ‘SAC (S YNTH ER)’, augments the training data by generating 1M new samples for every 10K real samples collected and samples them with a ratio r = 0 .
We then match REDQ and train with a UTD ratio of 20.
We evaluate our algorithms over 200K online steps for the DMC environments and 100K for OpenAI Gym.
In Figure 5, we see that SAC (S YNTH ER) matches or outperforms REDQ on the majority of the environments with particularly strong results on the quadruped-walk and halfcheetah-v2 environ-
[Figure]
ments.
This is particularly notable as D’Oro et al.
(2023) found that UTD=20 on average decreased performance for SAC compared with the default value of 1, attributable to issues with overestimation and overfitting (Chen et al., 2021;
Li et al., 2023).
We aggregate the final performance on the environments in Figure 1b, normalizing the DMC returns following Lu et al.
(2022b) and OpenAI returns as in D4RL.
Moreover, due to the fast speed of training our diffusion models and fewer Qnetworks, our approach is slightly faster than REDQ based on wall-clock time, whilst also requiring fewer algorithmic design choices, such as large ensembles and random subsetting.
Full details on run-time are given in Appendix D.2.
R ELATED W ORK
Whilst generative training data has been explored in reinforcement learning; in general, synthetic data has not previously performed as well as real data on standard RL benchmarks.
Generative Training Data.
Imre (2021); Ludjen (2021) had both considered using VAEs and GANs to generate synthetic data for online reinforcement learning.
However, we note that both works failed to match the original performance on simple environments such as CartPole—this is likely due to the use of a weaker class of generative models which we explore in Appendix C.3.
Huang et al.
(2017) considered using GAN samples to pre-train an RL policy, observing a modest improvement in sample efficiency for CartPole.
Yu et al.
; Chen et al.
(2023) consider augmenting the image observations of robotic control data using a text-guided diffusion model whilst maintaining the same action.
This differs from our approach which models the entire transition and can synthesize novel action and reward labels .
Outside of reinforcement learning, He et al.
(2022) consider generative training data for image classification and pre-training.
They also find that synthetic data is useful for data-scarce settings which are especially prevalent in reinforcement learning.
Sehwag et al.
(2022) consider generative training data to improve adversarial robustness in image classification.
In continual learning, “generative replay” (Shin et al., 2017) has been considered to compress examples from past tasks to prevent forgetting.
Generative Modeling in RL.
Prior work in diffusion modeling for offline RL has largely sought to supplant traditional reinforcement learning with “upside-down RL” (Schmidhuber, 2019).
Diffuser (Janner et al., 2022) models long sequences of transitions or full episodes and can bias the whole trajectory with guidance towards high reward or a particular goal.
It then takes the first action and re-plans by receding horizon control.
Decision Diffuser (Ajay et al., 2022) similarly operates at the sequence level but instead uses conditional guidance on rewards and goals.
(Du et al., 2023) present a similar trajectory-based algorithm for visual data.
In contrast, S YNTH ER operates at the transition level and seeks to be readily compatible with existing reinforcement learning algorithms.
Pearce et al.
(2023) consider a diffusion-based approach to behavioral cloning, whereby a stateconditional diffusion model may be used to sample actions that imitate prior data.
Azar et al.
Li et al.
(2020) provide theoretical sample complexity bounds for model-based reinforcement learning given access to a generative model.
Model-Based Reinforcement Learning.
We note the parallels between the synthetic data we generate and model-based reinforcement learning (Janner et al., 2019;
Yu et al., 2020;
Lu et al., 2022a); model-based methods generate synthetic samples by rolling out from previously seen states.
Two key differences to our method are: S YNTH ER synthesizes new experiences without the need to start from a real state and the generated experiences are distributed exactly according to the data, rather than subject to compounding model errors due to inaccuracies and policy exploitation.
Furthermore, S YNTH ER is an orthogonal approach and could in fact be combined with model-based RL by sampling initial states to rollout from, which could lead to increased diversity.
C ONCLUSION
In this paper, we proposed S YNTH ER, a powerful and general method for upsampling agent experiences in any reinforcement learning algorithm using experience replay.
We integrated S YNTH ER with ease on four distinct algorithms, each fine-tuned for its own use case, with no algorithmic modification .
Our results show the potential of synthetic training data when combined with modern diffusion models.
In offline reinforcement learning, S YNTH ER allows training from extremely small datasets, scaling up policy and value networks, and high levels of data compression.
In online reinforcement learning, the additional data allows online agents to use higher update-to-data ratios leading to increased sample efficiency.
We believe that scaling S YNTH ER to more settings would unlock exciting new capabilities for RL agents.
S YNTH ER could readily be extended to n -step formulations of experience replay by simply expanding the input space of the diffusion model.
Furthermore, as diffusion models have to date been largely applied in the visual domain, we expect the findings of this paper to transfer to an even greater extent in the visual setting.
One could leverage the generalization capability of image-based diffusion models to synthesize novel views and configurations of a pixel-based environment.
Finally, by leveraging guidance for diffusion models (Ho & Salimans, 2021), the generated synthetic data could be biased towards certain modes, resulting in new, transferable, and composable sampling strategies for RL algorithms.
S UPPLEMENTARY M ATERIAL A D ATA M ODELING
In this section, we provide further details for our data modeling.
Our diffusion model generates full environment transitions i.e., a concatenation of states, actions, rewards, next states, and terminals where they are present.
For the purposes of modeling, we normalize each continuous dimension (non-terminal) to have 0 mean and 1 std.
We visualize the marginal distributions over the state, action, and reward dimensions on the standard halfcheetah medium-replay dataset in Figure 6 and observe that the synthetic samples accurately match the high-level statistics of the original dataset.
We note the difficulties of appropriately modeling the terminal variable which is a binary variable compared to the rest of the dimensions which are continuous for the environments we investigate.
This is particularly challenging for “expert” datasets where early termination is rare.
For example, walker2d-expert only has ≈ 0 .
0001% terminals.
In practice, we find it sufficient to leave the terminals un-normalized and round them to 0 or 1 by thresholding the continuous diffusion samples in the middle at 0.5.
A cleaner treatment of this variable could be achieved by leveraging work on diffusion with categorical variables (Hoogeboom et al., 2021).
B H YPERPARAMETERS
The formulation of diffusion we use in our paper is the Elucidated Diffusion Model (EDM, Karras et al.
We parametrize the denoising network D θ as a simple residual MLP.
B. D ENOISING N ETWORK
Our denoising network D θ is an MLP with skip connections from the previous layer as in Tolstikhin et al.
Thus each layer has the form given in Equation (3).
[Equation]
The hyperparameters are listed in Table 4.
The noise level of the diffusion process is encoded by a
Random Fourier Feature (Rahimi & Recht, 2007) embedding.
The base size of the network uses a width of 1024 and depth of 6 and thus has ≈ 6 M parameters.
We adjust the batch size for training based on dataset size.
For online training and offline datasets with fewer than 1 million samples (medium-replay datasets) we use a batch size of 256, and 1024 otherwise.
For the following offline datasets, we observe more performant samples by increasing the width up to 2048: halfcheetah medium-expert, hopper medium, and hopper medium-expert.
This raises the network parameters to ≈ 25 M, which remains less parameters than the original data as in Table 1.
We provide ablations on the depth and type of network used in Table 6.
[Table]
B. E LUCIDATED D IFFUSION M ODEL
For the diffusion sampling process, we use the stochastic SDE sampler of Karras et al.
(2022) with the default hyperparameters used for the ImageNet, given in Table 5.
We use a higher number of diffusion timesteps at 128 for improved sample fidelity.
We use the implementation at https:// github.com/lucidrains/denoising-diffusion-pytorch which is released under an Apache license.
[Table]
The diffusion model is fast to train, taking approximately 17 minutes for 100K training steps on a standard V100 GPU.
It takes approximately 90 seconds to generate 100K samples with 128 diffusion timesteps.
C S YNTH ER A BLATIONS
We consider ablations on the number of generated samples and type of denoiser used for our offline evaluation in Section 4.1.
C. S IZE OF U PSAMPLED D ATASET
[Figure]
Size of Upsampled Dataset Ablations
[Figure]
In our main offline evaluation in Section 4.1, we upsample each dataset to 5M.
We investigate this choice for the walker medium-replay dataset in Figure 7 and choose 10 levels log-uniformly from the range [50K, 5M].
Similarly to He et al.
(2022), we find that performance gains with synthetic data eventually saturate and that 5M is a reasonable heuristic for all our offline datasets.
C. N ETWORK A BLATIONS
We ablate the hyperparameters of the denoising network, comparing 3 settings of depth from { 2 , 4 , 6 } and analyze the importance of skip connections.
The remaining hyperparameters follow Appendix B.1.
We choose the hopper medium-expert dataset as it is a large dataset of 2M.
As we can see in Table 6, we see a positive benefit from the increased depth and skip connections which leads to our final choice in Table 4.
[Table]
C. C HOICE OF G ENERATIVE M ODEL
In this section, we contrast our choice of diffusion models over prior generative models such as
VAEs (Kingma & Welling, 2014) and GANs (Goodfellow et al., 2014).
For these ablations, we use the state-of-the-art Tabular VAE (TVAE) and Conditional Tabular GAN (CTGAN) models proposed by Xu et al.
for low-dimensional data with the default hyperparameters.
We evaluate S YNTH ER and the prior generative models on the halfcheetah medium-replay dataset as in Appendix A.
As proposed in Patki et al.
(2016), we compare the following two high-level statistics on the synthetic data.
• Marginal: (Number between 0-1)
Aggregate Kolmogorov-Smirnov ( Massey Jr 1951) statistic, measuring the maximum distance between cumulative distribution functions, for each individual dimension of the data.
• Correlation: (Number between 0-1) Aggregate pairwise Pearson rank correlation (Fieller et al., 1957) between the dimensions of the data.
as well as down-stream offline RL performance using the synthetic data in Table 7.
We see that the diffusion model is more faithful to the original data and thus the synthetic data leads to substantially higher returns on both the TD3+BC and IQL algorithms.
Thus, we hypothesize a large part of the failure of prior methods (Imre, 2021; Ludjen, 2021) was due to a weaker generative model.
[Table]
D RL I MPLEMENTATION
For the algorithms in the offline RL evaluation in Section 4.1, we use the ‘Clean Offline Reinforcement Learning’ (CORL, Tarasov et al.
(2022)) codebase.
We take the final performance they report for the baseline offline evaluation.
Their code can be found at https://github.com/ tinkoff-ai/CORL and is released under an Apache license.
For the online evaluation, we consider Soft Actor-Critic (SAC, Haarnoja et al.
and use the implementation from the REDQ (Chen et al., 2021) codebase.
This may be found at https:// github.com/watchernyu/REDQ and is released under an MIT license.
We use the ‘dmcgym’ wrapper for the DeepMind Control Suite (Tunyasuvunakool et al., 2020).
This may be found at https://github.com/ikostrikov/dmcgym and is released under an MIT license.
D. D ATA A UGMENTATION H YPERPARAMETERS
For the data augmentation schemes we visualize in Figure 1a, we define:
Additive Noise (Laskin et al., 2020): adding ϵ ∼ N (0 , 0 . 1) to s t and s t
+1 .
Multiplicative Noise (Laskin et al., 2020): multiplying s t and s t +1 by single number ϵ ∼ Unif ([0 . 8 , 1 . 2]) .
Dynamics Noise (Ball et al., 2021): multiplying the next state delta s t +1 − s t by ϵ ∼ Unif ([0 . 5 , 1 . 5])
so that s t +
=
s t + ϵ · ( s t +1 − s t ) .
D. O NLINE R UNNING T IMES
Our online implementation in Section 4.2 uses the default training hyperparameters in Appendix B. to train the diffusion model every 10K online steps, and generates 1M new transitions each time.
On the 200K DMC experiments, ‘SAC (SynthER)’ takes ≈ 21 .
1 hours compared to ≈ 22 .
7 hours with
REDQ on a V100 GPU.
We expect that this may be heavily sped-up with early stopping on the diffusion training steps, and leave this to future work.