Paper: Replay across Experiments: A Natural Extension of Off-Policy RL (ICLR-2024)

Reviewer comment (Experiments):
What motivated the reuse of previous experimental data? There is no explicit explanation for this.

Author response:
We demonstrate how reusing previous data improves performance across a range of diverse domains and algorithms. Related work has demonstrated a similar effect to some degree albeit with additional algorithmic complexities that render them difficult to implement. Our focus in this work is on analyzing an underlying simple core idea more directly. With the increasing proliferation of offline datasets for robotics domains, RaE is an effective tool that can be incorporated into existing algorithms and fits naturally into existing RL experimentation workflows.

Reviewer comment (Evaluation):
The results in Table 1 are interesting. However, the reasons for the observed trends are not clearly explained.

Author response:
Table 1 presents an analysis of choosing different mixing ratios when running RaE with smaller datasets (10,000 and 100,000 episodes). Importantly, for all the main results shown in Figure 3 and 4 a fixed 50-50 ratio works well across domains. However, when the amount of data is restricted, using more online data is beneficial and prevents overfitting to a narrow distribution of offline trajectories. We have clarified this point in the description of Table 1.

Reviewer comment (Presentation):
In Fig 1, the size of the ‘Data’ blocks is increasing, but is not immediately noticeable. It would be better to exaggerate the increase in block size to bring a reader’s attention to the data accumulation mechanism.

Author response:
Thank you for the suggestion. We have updated the paper with a new version of the figure that highlights the increasing block size of the data on each iteration. Please let us know if further improvements can be made.

Reviewer comment (Evaluation):
It is not clear to me if "Total online steps" in the figures includes the steps from prior experiments or not, so I'm concerned about the fairness of the comparison between RaE and baselines. If the authors can clarify this point then I may be willing to raise my score.

Author response:
We have updated Figure 4 to include the data from the offline dataset. The relative performance of RaE against the baselines remains unchanged and importantly, RaE continues to show improved asymptotic performance on challenging domains.  We hope this addresses your original comment and will help improve your original score. If there are further questions, please let us know.

Reviewer comment (Writing):
I'm a bit confused about the difference between Random Weight Resetting and RaE. The authors write that "Reloading data for an experiment restart implicitly involves resetting network weights", which makes me think that RaE and Random Weight Resetting are very similar. However, this clearly isn't the case since RaE performs better. Can the authors clarify the difference?

Author response:
When restarting an experiment with RaE, in addition to resetting the weights of the network, the data distribution throughout learning utilizes a fixed ratio of mixing between offline and online data. In contrast, `Random resets’ only reload network weights and then use data from the replay buffer for learning in which older data is overwritten with time. In addition, RaE is more practical to implement since a period to reset does not have to be chosen and fixed a-priori; experiments can be restarted and data mixed naturally during the course of experimentation with repeated iteration.

Reviewer comment (Experiments):
Across all figures does "Total online steps" include the steps from prior experiments for RaE? If not, I'm concerned that it may not be a fair comparison.

Author response:
We have updated Figure 4 to include the data from the offline dataset. The relative performance of RaE against the baselines remains unchanged and importantly, RaE continues to show improved asymptotic performance on challenging domains.  We hope this addresses your original comment and will help improve your original score. If there are further questions, please let us know.

Reviewer comment (Experiments):
The authors write, "At the beginning of each training run, policy and value-function are re-initialized in line with stand-alone experiments". I'm curious if re-initializing vs not re-initializing at the start of each training run makes a difference in performance?

Author response:
This is a very good point. The choice of re-initializing weights is orthogonal to RaE and can be incorporated on top of RaE. Figure 7 of Appendix C.2 highlights this by comparing the performance of RaE with and without reloading weights on the `Locomotion’ domains. We find reinitializing weights can improve learning speed without hurting asymptotic performance.

Reviewer comment (Novelty):
The main insight of the work, while interesting, is a small contribution. As the authors note in the background section, other methods already use data stored from previous experiments, so the only novelty here is that *data is not discarded*. There is no theoretical analysis in the paper. Without significant novelty or theory, the paper depends solely on its empirical results.

Author response:
* “The main insight of the work, while interesting, is a small contribution.”

We respectfully disagree on the scale of contribution of our work. As you note, we present a highly-effective, simple method that can easily be introduced to RL methods and provide a comprehensive evaluation showing good performance compared to several strong baselines across a breadth of domains. As we discuss in the main text, other works that use a similar idea do so in an incidental manner, and commonly focus on additional algorithmic changes that add considerable complexity. By focusing on the core approach and combining its simplicity with a strong empirical contribution we believe our work can translate to practical real world impact across a range of RL domains.

Reviewer comment (Evaluation):
It is unclear that replaying offline data from a long time ago is as beneficial as the authors claim. Table 1 seems to indicate that performance improves almost monotonically as the proportion of online data increases to 90%. The authors claim that “as more data becomes available, a lower ratio [of 70-80%] works better,” but this only happens occasionally, and the performance improvement is small (about 2-5%). Without any measure of significance provided, it does not seem that more than 10% offline data actually helps much.

Author response:
* 90% online data works best in Table 1

The results in Table 1 discuss how best to adapt RaE to regimes where data is sparse. As you point out, using more online data is a reasonable strategy in this setting. However note that mixing some offline data is preferable to learning purely offline and all the results in Figure 3 and 4 show an improvement in asymptotic performance with a simple 50/50 ratio of mixing data.
Additionally, we have updated the paper to include an additional result in the Appendix C.3 that illustrates the effect of mixing different data ratios when using the full dataset (of 4e5 episodes). This result represents many project settings and shows that when more data is available, a mixing ratio of 50 to 70 percent of online data works better than a mix with too much online data.

* “It is not clear to me that mixing offline data is always helpful against pure online agent”

We show that across a range of diverse environments and algorithms a 50-50 ratio of offline and online mixing improves asymptotic performance when simply reloading data from a previously completed run. We would like to clarify though that we do not claim that mixing offline data is always helpful against a pure online agent. In fact as the ablation in Table 1 shows, with very little data (10,000 episodes), a 50-50 mixing ratio works less well than a purely online agent. In such cases, we find a higher ratio of online data can improve performance over the online run. The performances in Table 1 are represented as a percentage compared to training entirely online in that setup.

Reviewer comment (Presentation):
In-text citations are not rendered correctly on the last line of page 1.

Author response:
* Minor edits

Thank you for pointing these out to us. We have made corrections to the text.

Reviewer comment (Evaluation):
What is meant by “Accumulated Reward” in the y-axes of Figures 3, 4? Is this the undiscounted episode return?

Author response:
* Accumulated Reward

Yes, by accumulated reward we mean the undiscounted episode return achieved by the agent. We have updated the captions of Figures 3 and 4 to reflect this.

Reviewer comment (Writing):
What is meant by “from scratch to convergence” in Table 1’s caption? Does that mean that exactly 100% refers to the final performance obtained by a pure online-data agent?

Author response:
* Table 1 caption ‘from scratch to convergence’

The results in Table 1 present the performance of RaE with small subsets of data as a percentage of learning from scratch. By ‘from scratch to convergence’, we want to highlight that the agent performance being compared against is not at the 10,000 or 100,000 episode mark but with an online agent trained until it converges.

* Concerned that datasets used in ablations are smaller than in the main experiment: ablation would be stronger if it closely matches the main experiment setup. Why is a 100% online-data mix not compared in this study? 

Thanks for the comment. The follow-up ablation in Figure 8a shows the effect of data mixing with the exact experimental setup as described in the main experiments. We have updated this figure and corresponding text to include the 100% online performance as well as per your suggestion. 

In addition, we are also running the ablation in Figure 8a on the `Locomotion (Soccer) Vision` task.  If space allows, we will try to incorporate this into the main body of the paper. These results may not be ready before the end of the review process but we will try our best to update them in time. 

The purpose of the original ablation was to investigate how RaE performs with different data mixing ratios under various regimes where less data is available. We chose to report this as a percentage of the performance achieved when training entirely online for ease of understanding. To clarify by 100% score relative to `from scratch to convergence’ we mean an agent trained entirely online (i.e. a 100% online-data mix) with the same experimental setup as the other methods in the ablation, until it converges. With these new results and clarifications, we hope that you will consider increasing your score and are happy to discuss further.

Reviewer comment (Experiments):
Aside from RaE, another natural baseline for this across-experiment setting would be to use behaviour cloning (or distillation) on the previously trained agents. Have you considered doing so?

Author response:
* “Additional baseline: BC and distillation to previous agent.”  

When fine-tuning, we consider two baseline agents trained with the offline data: one that uses CRR and another that uses BC. We then select the best performing variant among these to finetune. We believe this addresses your point but are happy to clarify and run another baseline (time permitting) if required.

Reviewer comment (Evaluation):
As I mentioned in strengths section, I think training across-experiments could be relevant for practical purposes. How do you see it being used in a scientific context? It could be difficult to fairly assess algorithms if the data they have access to depends on the sequence of previous experiments done. i.e. an advantage of discarding previous data is that algorithms start on equal ground in different papers, allowing fair comparisons.

Author response:
* RaE in the context of the scientific method for reproducible and comparable results.  

This is a very good point; thank you for highlighting it. Reproducibility in RL is already a challenging open problem [1]. If RaE is more widely adopted, it may add another dimension to this issue: subtle changes in the ordering of data when mixing may lead to changes in final performance. One solution to this could be to standardize and open source offline data sources for benchmark domains (e.g. RL Unplugged) and define deterministic orderings when reading from the data using, for example, fixed random seeds. We have included a brief discussion on this point in the Discussion section (Section 5).

Reviewer comment (Evaluation):
I'm a bit surprised that doing some offline learning first is so detrimental to the policy (Fig. 4, finetuning and AWAC). Intuitively, I would guess that there should be a jumpstart in the performance due to the additional offline training. Could you clarify this? Do you have any hypotheses why these methods don't perform very well here?

Author response:
* “Why does offline training work so poorly?”  

In Figure 4, fine-tuning and AWAC learn on the Humanoid Run domain but perform poorly  on the Manipulator domains of RL Unplugged. Humanoid run uses a dense reward signal that incentivizes the agent for moving forward. In contrast, the Manipulator domains provide sparse rewards that are triggered in stages when a robotic arm moves to a peg (or ball) and inserts it into a hole. Learning offline from limited data in sparse reward settings is particularly challenging which is why we see a strong benefit of using RaE to mix in online data to improve asymptotic performance.  Additionally offline learning may also overfit to the limited set of trajectories in the dataset, making further improvement through fine-tuning difficult.We have added a sentence to Section 3.3 to highlight this.

Reviewer comment (Experiments):
Currently, the experiments that use RaE are quite similar to simply using resets but at the end of each experimental run. It would be interesting to see some experiments where the same algorithm was not used in consecutive experiments or, at least, changing hyperparameters. This could better simulate the development process of an RL algorithm. As a suggestion, I would be curious to see if hyperparameter optimization could be made easier. Since RL agents can be sometimes hard to run initially on a new problem, we might be able to more easily identify which hyperparameter settings are promising by giving the agent additional data.

Author response:
* “.. more easily identify which hyperparameter settings are promising by giving the agent additional data.”  

Yes, the faster iteration time when using RaE could be useful to search for better hyperparameters to use in new settings from scratch. However, it is important to note that hyperparameters that are tuned in a setting that reuses data may not directly transfer over to settings where data is not reused. For example, learning rates and optimizer parameters may not be directly transferable.