Paper: Revisiting Data Augmentation in Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Presentation):
**Excessive Reference to Appendix:** The frequent deference to the appendix, especially in sections 5.2 and 6, detracts from the paper's flow. Readers are forced into continuous back-and-forth toggling, making the narrative harder to follow.

Author response:
We thank the reviewer to have read our paper so carefully.  
We admit that it is distracting to frequently check the appendix and we apologize for this.  
As the reviewer may understand, due to the page limit, all the information cannot be included in the main text.  
We tried to maintain the coherence of our presentation as much as possible. The references to the appendix mainly provides the theoretical proofs, further empirical evidence, additional experiments, and other minor details.  
We believe that they can be skipped in a first reading.

Reviewer comment (Evaluation):
**Comparison with RAD and SVEA:** In the original paper, SVEA, which does not employ data augmentation on the target, significantly outperforms RAD. A comment or analysis from the authors on this discrepancy would have been informative.

Author response:
Before we compare RAD and SVEA, we need to clarify that SVEA actually uses random shift in calculating the targets and only avoids those complex image transformations in calculating the targets.  
Although it is removed from their public repository in the github commit (https://github.com/nicklashansen/dmcontrol-generalization-benchmark/commit/35a1eed87f72d3c9881c425fd83604c756849e56), we checked with the authors by email to confirm that the results in the paper were generated by applying random shift in calculating the targets.  

Now, if we compare the generalization ability evaluated in color-hard background, we should observe that RAD (random shift) $<$ RAD (random conv) $<$ SVEA (random conv).  
The first inequality is decided by using random conv as the image transformation.  
The second inequality is caused by avoiding using random conv in calculating the target.

Reviewer comment (Presentation):
**Bolding in Tables:** The rationale behind the use of bolding in tables (specifically tables 4 and 5) needs clarification.

Author response:
Thank you for your constructive suggestions.  
We initially used bold to highlight the values that are commented in the text.  
We understand this formatting may lead to some confusion.  
We have modified our paper according to the suggestions:  
- For a clear presentation, we removed bolding in Tables 1, 4, and 5 and added more explanations in the captions.  
- We included more discussions in the Limitation section and explicitly referred to it in the Conclusion section in the main text.  
- We added the missing legends for Figure 8.

Reviewer comment (Presentation):
**Placement of Limitations:** Limitations are located in the appendix and should be explicitly referenced in the main body.

Author response:
Thank you for your constructive suggestions.  
We initially used bold to highlight the values that are commented in the text.  
We understand this formatting may lead to some confusion.  
We have modified our paper according to the suggestions:  
- For a clear presentation, we removed bolding in Tables 1, 4, and 5 and added more explanations in the captions.  
- We included more discussions in the Limitation section and explicitly referred to it in the Conclusion section in the main text.  
- We added the missing legends for Figure 8.

Reviewer comment (Evaluation):
Based on the data from Table 2, RAD+ exhibits lower variance on the critic loss compared to DrQ. Yet, it underperforms DrQ in terms of overall results. Given that both methods utilize the SAC algorithm, this seems to counter the claim made by the authors that reducing critic loss variance leads to more stabilized training. Can the authors provide insights or explanations for this observation?

Author response:
In Figure 2, RAD+ is better than DrQ at the beginning and becomes worse later on.  
In general, if everything else is kept the same, a lower variance of the critic loss can lead to a more stable training and a better performance.  
However, the performance of an algorithm is decided by many factors.  
For instance, for hard tasks such as walker\_run, we observe that RAD and DrQ are not so stable.  
The instability of this environment might be the key cause for this result.

Reviewer comment (Experiments):
My understanding is that the only discernible difference between the authors' method and DrQ+KL is the introduction of the tangent prop regularization. Drawing from Table 1, it appears that this regularization is the primary factor that reduces the critic loss variance, rather than the authors' generic algorithm. Could the authors comment on the role and impact of tangent prop in this context?

Author response:
Tangent prop regularization is derived by enforcing the derivative of the critic with respect to the image transformation parameters to be zero over the whole set of parameters.  
From this derivation, we can easily see that the tangent prop regularization term explicitly enforces the invariance in the critic and it is important in reducing the variance of the critic.  
However, we can not omit the variance reduction from using more augmented samples, which is also part of the generic algorithm.

Reviewer comment (Evaluation):
The authors assert that employing random convolution in SVEA doesn't induce significant variance. However, the empirical results from Table 5 seem to suggest otherwise. Could the authors clarify?

Author response:
We answered this question in point 3 (answer to W3) above.

Reviewer comment (Theory):
What drives the intuition that the KL term assists in reducing the variance of the target critic?

Author response:
The KL term explicitly enforces the invariance in the actor.  
When calculating the target Q-values for an augmented next state, we need to sample from the current policy to choose the next action for this augmented next state.  
If invariance with respect to the image transformations approximately holds, the next action will be sampled from a similar distribution over actions, which leads to a smaller variance in the target values.  
Under the extreme case where the invariance in the actor has been satisfied, the next action will be sampled from exactly the same distribution.

Reviewer comment (Experiments):
For a more comprehensive analysis, would it be feasible for the authors to include SVEA in the experiments of the initial experimental setup?

Author response:
The new results for evaluating SVEA in the initial experimental setup are updated in Figure 3 and Figure 9 in the newest version.  
It is not surprising that using SVEA with random overlay increases the generalization ability of the agent at the cost of sample efficiency.  
Especially in environments like reacher, the target goal might not be clear in the image when random overlay is applied.  
We also updated our paper to include the new results.

Reviewer comment (Experiments):
The experimental section of the paper is somewhat lacking despite the thorough theoretical derivation. It compares the performance with the previous statistically trained model, SVEA, providing detailed experimental data and theoretical analysis, but there is a lack of in-depth analysis of the shortcomings of the previous algorithms and the advantages of the proposed algorithm. Moreover, the algorithm should be further compared with more methods and applied in various domains to validate its generalizability.

Author response:
To the best of our knowledge, most recently-proposed techniques for improving generalization ability are actually orthogonal to our analysis and could be combined with our  proposed algorithm.
However, if we have missed any  relevant work, we would appreciate if you could provide some specific references and we would try our best to add some additional comparative experimental results in our final version.
As for the limitations of our proposed algorithm, please check point 2 in our Meta Reply.

Reviewer comment (Evaluation):
The paper discusses varying impacts of different complex image augmentation techniques on the invariance of target Q-values and uses cosine similarity from encoder outputs for evaluation. While the solution of performing more updates at each training step is proposed to handle difficult transformations like random convolution or random rotation, the paper does not address potential overfitting issues under these specific conditions or how to handle them.

Author response:
We did not observe this overfitting issue when we updated more.
To verify this point, we ran some additional experiments to provide further evidence for it.
First, we can see that the training scores are similar to the evaluation scores when we update more, as shown in the figure (evaluation_training_score_for_reviewer_9PSY.png) in our new supplementary material.
We also ran the experiments of training the agent with more iterations when more updates are applied per iteration and the result is shown in the figure (More_iterations_and_more_updates_for_reviewer_9PSY.jpeg) in our new supplementary material. 
In this experiment, we trained the agent with more iterations and more updates per iterations while we did not observe an overfitting issue.

Reviewer comment (Novelty):
The work in this paper seems to be equivalent to adding an explicit regularization to implicit regularization, is it equivalent to DrQ combined with DrAC in terms of functionality?

Author response:
Please check point 1 in our Meta Reply.
    
Roughly speaking, our method follows the idea from DrQ to apply invariant image transformations in both the training of the actor and critic, complements it with a KL regularization term in the actor loss and introduces a new regularization term called tangent prop in the critic loss.
However, there are three key differences between our method and the simple combination of DrQ and DrAC:
- Our method introduces a new regularization term called tangent prop which is novel in DRL.
- The KL regularization term in our method is not exactly the KL regularization term used in DrAC. 
The KL regularization term $D_{KL}[\pi_\theta(\cdot|s),\pi_\theta(\cdot|f_\mu(s))]$ used in DrAC only augments the state on one side while the KL regularization term $D_{KL}[\pi_\theta(\cdot|f_\eta(s),\pi_\theta(\cdot|f_\mu(s))]$ in our method augments the states on both sides.
The regularization term in our method is somehow equivalent to using an averaged policy as the target, which may be preferred.
The detailed analysis is presented at the end of Section 4.2 and the experimental comparisons between these two KL regularizations are shown in Figure 2. The two lines which are respectively labeled as KL\_fix\_target and KL correspond to the KL from DrAC and the KL in our method.
- Our method is compatible with using more than one type of image transformation in the training which is required to achieve state-of-the-art performance in both sample efficiency and generalization ability. 
Moreover, we also provide an analysis and suggest a criterion for the usage of complex image transformations.

Reviewer comment (Theory):
Please check that the two terms in the KL divergence in Eq. 8, and there seems to be an ambiguity with Eq. 4 and Eq. 9.

Author response:
There is actually no ambiguity here. 
The KL term in Equation 8 is different from the KL terms in Equation 4 and Equation 9, as listed below:
- Only the state in one side of the KL term in Equation 8 is augmented while the states in both sides of the KL term in Equation 4/Equation 9 are augmented.
The differences between them are explained in our answer 1(b) to W1 above.
- The directions of the KL regularization terms are different.
In Equation 8, the stop gradient is attached to the $\theta$ on the right hand side.
Considering that KL is asymmetric, it is necessary to discuss the difference which is presented in the second paragraph below Proposition 4.1.
In conclusion, we found that using the stop gradient in the first term is better and including an explicit KL term in the actor loss is preferred.

Reviewer comment (Theory):
How to prove that the distributions of $\nu$ and $\mu$ in Eq. 11 and Eq. 12 are the distributions of $\hat{\nu}$ and $\hat{\mu}$ satisfying Lemma 1?

Author response:
Note that in Eqns. 11 and 12, the distributions of $\nu$ and $\mu$ are not restricted. They are usually set to be uniform distributions, but other distributions could be chosen depending on the application domain.
    
Lemma 1 simply states that for any distributions $\nu$, implicit regularization in the critic with specifically chosen $\hat\nu$ and $\hat\mu$ (see Proof of Lemma 1 for details) is equivalent to explicit regularization in the critic with $\nu$.

Reviewer comment (Experiments):
How to show that "even using complex image transformations such as random convolution in the target". We did not find an experiment in the paper that verifies this conclusion.

Author response:
We apologize for the ambiguity here. The experimental results for this point are shown in Appendix F.  
We modified our paper to explicitly indicate that all observations about using complex image transformations in calculating the targets are presented in Appendix F.
Moreover, we clarified the descriptions of Tables 4 and 5.

Reviewer comment (Reproducibility):
I'm curious about the method of determining $ \alpha_i$, which seems to be missing the reason(s) in the paper. Does it have a specific value for each augmentation in the set, or does it choose an average weight?

Author response:
We simply use an average weight for different image transformations, as it was done in SVEA.

Reviewer comment (Experiments):
In Figure 2, the batch size of DrQ reproduced in the results is 256 instead of 512 in the original DrQ. The scores of DrQ will decrease under some environments when using a smaller batch size. Compared with the official results of DrQ (https://github.com/denisyarats/drq), some results of DrQ shown in Figure 9 are lower. Such as ball_in_catch, walker_walk, walk_run. Therefore, to make a fair comparison, the author should completely use the hyperparameter settings in DrQ to reproduce, or use the scores in DrQ.

Author response:
We ran the experiments of DrQ and our method in the environments listed in the question with batch size of 512:
|  Envs  | DrQ(batch_size=256) | DrQ(batch_size=512) | Ours(batch_size=256) | Ours(batch_size=512)
|-----------------|-------------------|----------------|----------------|----------------|
| Ball in cup catch | 947.38+-41.76  |962.08+-19.54 |974.06+-5.09|969.96+-8.30
| Walker walk | 591.25+-433.38 | 740.67+-343.23 |925.62+-35.03 |903.11+-56.54
| Walker run | 378.29+-184.58 | 350.71+-210.66 |496.06+-26.33 |569.17+-63.75

We can see that
1. Our method is insensitive to the choice of batch size.
2. Without retuning the hyperparameter, our method always matches or outperforms DrQ in these environments with these two choices of batch size.
3. Considering the modest computational resources we have and the huge increase in computational cost due to usage of batch size = 512, it is fair to compare sample efficiency of all the methods with the batch size of 256.

Please let us know if you have any other comments or concerns!

Reviewer comment (Evaluation):
More insights into the limitations and potential failures of the proposed method should be discussed. This would provide a more balanced perspective and help readers better understand the practical considerations when applying the proposed approach.

Author response:
1. **W1:** *More insights into the limitations and potential failures of the proposed method should be discussed.*
    
Please check point 2 in our Meta Reply.

Reviewer comment (Experiments):
Further analysis and comparisons with a wider range of existing techniques should be conducted to showcase the advantages and limitations of the proposed method in different scenarios. This would provide a more comprehensive view of its effectiveness and contribution to the field.

Author response:
2. **W2:** *Further analysis and comparisons with a wider range of existing techniques should be conducted to showcase the advantages and limitations of the proposed method in different scenarios.*

Please check point 3 in our Meta Reply.