Paper: Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR-2024)

Reviewer comment (Experiments):
The work has a limited range of algorithms, which the authors did not state as a weakness.

Author response:
Thank you for pointing out the additional limitation with respect to our RL algorithm choices – we’ll include it in our revisions. 

Different algorithms may be more or less capable of learning from augmented experience. We do think that it would be interesting to conduct another empirical study investigating how different algorithmic advances (e.g. multiple target networks [1], entropy regularization [2], n-step returns [3], etc.) may affect learning from augmented data. Several recent works have shown that n-step returns are critical to data efficient RL when learning with only observed data [4,5], and we hypothesize this may also be the case when learning with augmented data.

Reviewer comment (Experiments):
How were the algorithms used in the study chosen?

Author response:
We initially wanted to use TD3 throughout the paper; TD3 often performs much better than DDPG, and is much less expensive to run than SAC. However, we found that TD3 performed noticeably worse than DDPG on all Panda tasks. In fact,  Gallouedec et. al [6] (the creators of Panda robot tasks) made this same observation. Thus, we decided to use DDPG for Panda tasks and TD3 for Goal2D as well as all MuJoCo tasks in Appendix F.

Reviewer comment (Evaluation):
Is there intuition about how the differences in the tasks affect the results? For example, in figure 3 the "x2" and "x4" results for policy data and augmented data are similar, but this is not true in figure 1.

Author response:
In the Goal2D task, we do see that additional augmented data improves data efficiency more than additional policy-generated data. In Panda tasks, augmented data is either just as good or slightly worse. We believe this difference arises because Goal2D is much simpler than the Panda tasks. 

The *absolute* improvement to data efficiency provided by DA is not a critical point in our analysis; we sought to understand how much different aspects of DA contribute to the observed *relative improvements* in data efficiency.

Reviewer comment (Evaluation):
The empirical findings in the paper convey weak messages. While the authors provide some useful (yet somewhat intuitive) insights on using data augmentation in RL, there are no effective usages presented for practical application. It would be better to clarify how their findings can be used in practice.

Author response:
## **Core messages of our empirical analysis:**

In our submission, we provide two practical guidelines:

1. **Data augmentation (DA) should focus less on generating reward signal and more on increasing state-action coverage.** In our experiments, most – and sometimes all – of the benefits of a DAF can be explained by an increase in state-action coverage alone, and increasing reward density beyond a small threshold typically provides no further benefit. This guideline contrasts with the core motivation of HER and its follow-up works [1, 2] which focus on generating augmented reward signal to improve data efficiency.
2. **When it is possible to generate a large amount of augmented data, use this data to decrease the replay ratio** (rather than to increase the batch size used for updates.)

We’re pleased that our second guideline was clear, and we hope our response clarifies the first guideline. In our revisions, we will better emphasize these guidelines.

Reviewer comment (Evaluation):
In the reviewer’s understanding, the most useful practical message from the work is to consider tuning the augmented replay ratio. It would be helpful to provide practical strategies to use data augmentation effectively, based on the empirical findings.

Author response:
## **Core messages of our empirical analysis:**

In our submission, we provide two practical guidelines:

1. **Data augmentation (DA) should focus less on generating reward signal and more on increasing state-action coverage.** In our experiments, most – and sometimes all – of the benefits of a DAF can be explained by an increase in state-action coverage alone, and increasing reward density beyond a small threshold typically provides no further benefit. This guideline contrasts with the core motivation of HER and its follow-up works [1, 2] which focus on generating augmented reward signal to improve data efficiency.
2. **When it is possible to generate a large amount of augmented data, use this data to decrease the replay ratio** (rather than to increase the batch size used for updates.)

We’re pleased that our second guideline was clear, and we hope our response clarifies the first guideline. In our revisions, we will better emphasize these guidelines.

Reviewer comment (Novelty):
There exists a line of prior works studying the impact of data augmentation in RL and ways to maximize its efficacy, e.g., [A, B, C]. It has been observed that maximally using data augmentation in RL training can sometimes be worse than not using augmentation at all. This coincides with the paper’s major finding (that a low augmented replay ratio is better), although the prior works focused on vision-based RL, which is different from the setting considered in this paper. It is necessary to discuss differences and coincidences between this work and existing ones.

Author response:
## **Differences between our work and prior works studying visual augmentations:**

This is a great question that we can clarify with a few minor additions to the related work section. There are a few fundamental differences between our work and the related works you’ve provided:

1. Most prior works – including all three you have listed – focused on introducing new types of data augmentation functions (DAFs) or frameworks and demonstrating that they can boost the data efficiency of RL. To the best of our knowledge, our work is the first to systematically investigate which aspects of DA are most responsible for observed improvements in data efficiency.

2. Raileanu et. al [3] and Ko & Ok [4] use augmented data for auxiliary tasks, while we focus on using augmented data for model-free  updates. Moreover, these auxiliary tasks aim to make the policy and value function invariant to augmentation, i.e. $\pi( \cdot | s) = \pi( \cdot | \tilde s)$ and $V(s) = V(\tilde s)$, and cannot be used with the dynamics-invariant DAFs we considered. For example, the optimal action generally changes if we translate the agent or goal, so it is undesirable to have $\pi( \cdot | s) = \pi( \cdot | \tilde s)$.

3. As you noted, all three works you listed consider visual augmentations that only transform the agent’s state and generate augmented data with the same semantic meaning as the original data. As such, they study DA for representation learning. In contrast, the dynamics-invariant DAFs we consider generate augmented data with a new semantic meaning and thus study DA to improve exploration.

Reviewer comment (Novelty):
The proposed framework to study the effectiveness of data augmentation seems straightforwardly designed. It would be helpful to clarify the technical novelty of the framework.

Author response:
## **Novelty of our framework:**

The core novelty of our framework is that it permits fine-grained control over many hyperparameters relevant to DA, allowing us to systematically study different aspects of DA.

Crucially, our framework guarantees that the agent uses the same ratio of augmented to observed data (or update ratio) in each update by sampling observed and augmented data from separate replay buffers. Existing DA frameworks [*e.g.*, 5,6] and applications of DA [*e.g.*, 7, 8] sample data from a shared replay buffer containing both augmented and observed data, so the update ratio can change across updates. Moreover, with a shared replay buffer, the update ratio and the augmentation ratio (the number of augmented samples generated per observed sample) are entangled; increasing the augmentation ratio increases the update ratio.

Given that DA is so widely studied and that most prior work focuses on improving over baselines rather than understanding why certain methods perform well, we hope the analysis enabled by our framework is the first of many which move us towards the development of further practical DA guidelines.

We hope our response clarifies our work's novelties with respect to the existing DA literature. If you have further questions, we are more than happy to discuss them!

Reviewer comment (Experiments):
Do the findings hold if data augmentation is used for more than just augmenting data, e.g., for policy distillation or representation learning? Other uses of data augmentation may extract its full potential in RL and yield somewhat different observations.

Author response:
## **Do the findings hold when DA is used for policy distillation or representation learning?**

Our work focuses on understanding the benefit of integrating dynamics-invariant augmented data into model-free RL updates. Analogous studies focusing on policy distillation and representation learning are interesting directions for future work that we intend to investigate, but are beyond the scope of this analysis. We note that such studies will likely need to focus on different aspects of DA than the ones we considered in our work. 

In particular, prior DA works focusing on policy distillation [4] and representation learning [3] are designed to work with DAFs that generate augmented data with the same semantic meaning as the original data, such as visual augmentations. This point has three implications:

1. Because these augmentations produce data which could never be observed through environment interaction (e.g. an agent would never receive a cropped, recolored, and rotated image from the environment), they primarily aid representation learning rather than exploration. 
2. It may be imprecise to say that these augmentations increase coverage, because the observed and augmented data have the same semantic meaning. 
3. Because the original and augmented observation have the same semantic meaning, the augmented reward will often be the same as the original reward. Thus, the concept of  “reward density” does not apply to these augmentations.

Having said this, the role of the augmented replay ratio can and should be studied for visual augmentation. Researchers have become increasingly interested in how the replay ratio affects learning [9-12], and lowering the replay ratio via augmentation may yield substantial improvements in data efficiency on visual tasks.

Reviewer comment (Experiments):
In machine learning, choosing hyperparameters is critical. The reviewer wants to ensure that every hyperparameter has been optimized for each scenario.

Author response:
## **Hyperparameter tuning:**

Appendix G in the supplemental material lists hyperparameters and training details for all experiments. For Panda tasks, we use the same hyperparameters used by Plappert et. al [13], the creators of the Fetch robot tasks analogous to the Panda robot tasks we consider in our work. Gallouedec et. al [14] (creators of Panda robot tasks) also use these hyperparameters. Plappert et. al found these hyperparameters by performing a sweep over many different hyperparameter combinations described in Appendix B of [13].

We modified two hyperparameters in our replay ratio experiments (also noted in Appendix G). These experiments use a larger split of augmented data in each update, and we observed that these modifications improved performance in all tasks for both augmented and non-augmented agents.

Again, please let us know if our response clarifies your comments. We would be happy to discuss any follow-up questions you may have!