Paper: Intelligent Switching for Reset-Free RL (ICLR-2024)

Reviewer comment (Theory):
The paper has no significant theoretical results.

Author response:
**[No Significant Theoretical Results and lack of Convergence Proofs]**
Although we do not provide a proof of the convergence properties of RISC, we do provide a discussion of the role of bootstrapping in the reset-free RL setting in Section 4.1. Reset-Free RL thus far has been a fairly empirical field, with several prior works [1,2,3,4] being accepted to ICLR and comparable conferences without theoretical results. While it would be useful, developing such theory falls outside the scope of this work.

Reviewer comment (Theory):
The convergence of the proposed method is not guaranteed.

Author response:
**[No Significant Theoretical Results and lack of Convergence Proofs]**
Although we do not provide a proof of the convergence properties of RISC, we do provide a discussion of the role of bootstrapping in the reset-free RL setting in Section 4.1. Reset-Free RL thus far has been a fairly empirical field, with several prior works [1,2,3,4] being accepted to ICLR and comparable conferences without theoretical results. While it would be useful, developing such theory falls outside the scope of this work.

Reviewer comment (Experiments):
Too many hyperparameters to tune, e.g., minimum trajectory length, conservative factor...

Author response:
**[Too many hyperparameters]**
We agree that adding hyperparameters adds complexity to the method, but we would like to point out: (1) Our algorithm is fairly robust to hyperparameter changes, and our performance does not degrade drastically even when completely removing a hyperparameter from the method. To demonstrate this, we have added a hyperparameter sensitivity analysis in Figure 10 (Appendix F, new figure) and Figure 11 (Appendix F, new figure) showing the results when only optimizing for 1 or 2 of the 3 parameters. In both cases, the perturbation/removal of hyperparameters reduced performance, but not significantly. (2) Having those hyperparameters is necessary to maintain the flexibility of our method. Unlike our work, several prior methods only work on sparse reward environments with demonstrations, and the ones that also work on dense reward environments without demonstrations, we outperform. Thus, for the cost of the extra hyperparameters, we provide added coverage on different environments.

Reviewer comment (Theory):
Could the author explain more about how the method makes the learning stable?

Author response:
**[How the method makes learning more stable]**
The timeout-nonterminal bootstrapping strategy makes learning more stable, as it computes consistent targets for a given state regardless of whether a trajectory ended there. With the other strategy, if a transition is sampled from the replay, and it represents the end of the trajectory but not a terminal state, it still gets assigned a target value of 0. If this transition also exists in the replay buffer as not the end of a trajectory, it gets assigned a target value based on the next state in the transition. Removing this discrepancy by using the timeout-nonterminal bootstrapping strategy leads to more stable learning.

Reviewer comment (Novelty):
I suppose the work will benefit the reward-free RL (Yang, Q., & Spaan, M. T. (2023). CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration. AAAI-2023). Could the author elaborate a bit on how to use the ideas in reward-free RL for exploration?

Author response:
**[Relation to CEM and Reward-Free RL]**
The ideas from CEM and Reward-Free RL seem complementary to the ideas of our work and potentially useful in adapting Reset-Free RL methods to unsafe environments with irreversible states. Our method allows the agent to explore areas it hasn’t mastered yet, and the ideas from CEM can be used to guide the agent and perform safe exploration.

Reviewer comment (Experiments):
One important argument this paper makes is that adding bootstrapping for time-out non-terminal states is important, although it is theoretically well motivated, I think it would be better to see some practical motivations, especially why it is important under the reset-free setting. For example, are value estimations very different?

Author response:
**[Practical motivations for timeout non-terminal bootstrapping]**
The timeout-nonterminal bootstrapping strategy makes learning more stable, as it computes consistent targets for a given state regardless of whether a trajectory ended there. With the other strategy, if a transition is sampled from the replay, and it represents the end of the trajectory but not a terminal state, it still gets assigned a target value of 0. If this transition also exists in the replay buffer as not the end of a trajectory, it gets assigned a target value based on the next state in the transition. Removing this discrepancy by using the timeout-nonterminal bootstrapping strategy leads to more stable learning.

Reviewer comment (Experiments):
The paper introduces several method-specific hyper-parameters such as M, m, and β. It would be valuable to discuss the method's sensitivity and robustness to these hyper-parameters.

Author response:
**[Hyperparameter Sensitivity Analysis]**
Our algorithm is fairly robust to hyperparameter changes, and our performance does not degrade drastically when completely removing a hyperparameter from the method. To demonstrate this, we have added a hyperparameter sensitivity analysis in Figure 10 (Appendix F, new figure), and Figure 11 (Appendix F, new figure) showing the results when only optimizing for 1 or 2 of the 3 parameters. In both cases, the perturbation/removal of hyperparameters reduced performance, but not significantly.

Reviewer comment (Reproducibility):
Will code be available?

Author response:
**[Will code be available?]**
Yes, we will release code with the final version of the paper.

Reviewer comment (Experiments):
Are there any correlations between adding bootstrapping and adding an intelligent switch? For example, will switch be **more** useful when bootstrapping is added?

Author response:
**[Are there any correlations between adding bootstrapping and adding an intelligent switch]**
The timeout-nonterminal bootstrapping strategy is necessary in order to do more aggressive switching. Since intelligent switching introduces more trajectory ends into the replay buffer, not having timeout-nonterminal bootstrapping could likely result in even more instability. This can be seen with the ablation experiments in section 5.3, where just the intelligent switching isn’t able to learn well, but adding bootstrapping and the switching results in the best agent.

Reviewer comment (Experiments):
If I understand correctly, a separate critic success function was trained. Why didn't the authors use the critic from SAC (Soft Actor-Critic) instead of training a new one?

Author response:
**[Why not use the SAC critic instead of training a separate success critic?]**
We trained a separate success critic because the competency metric we wanted to estimate doesn’t always correspond to the discounted return that the SAC critic is estimating (e.g., when using dense rewards or reward scaling).

Reviewer comment (Experiments):
Bootstrapping for time-out non-terminal states should always be performed, do you have any intuitions on is it more important in the reset-free setting, since in episodic RL setting, people generally just ignore it?

Author response:
**[Is bootstrapping timeout-nonterminal states more important in the reset-free setting]** 
We expect timeout-nonterminal bootstrapping to likely be more important for the reset-free setting. There is both empirical and theoretical evidence for this. Based on the analysis in section 4.1, we expect that having the correct timeout-nonterminal bootstrapping strategy helps stabilize learning in the reset-free setting. Since the state distribution for a reset-free agent can be very different from the forward policy’s state distribution, stabilizing the targets through the timeout-nonterminal bootstrapping is necessary to learn. In standard RL, because the state distribution more closely matches the distribution of the forward policy, the targets will likely automatically be more stable even with the timeout-terminal strategy (although it’s been shown that doing timeout-nonterminal bootstrapping can still help). Empirically, the community has only recently started to shift to timeout-nonterminal bootstrapping for episodic tasks, and still saw much success with the old strategy. In reset-free RL, several tasks cannot be learned at all or are learned suboptimally when using the timeout-terminal strategy.

Reviewer comment (Writing):
At the beginning of section 5.1, the paper discusses RC and RISC. Could you please elaborate more on RC and the difference between these two agents? Does RC also use bootstrapping?

Author response:
**[Difference between RC and RISC]**
RC and RISC both use bootstrapping and the same base DQN agent. The only difference is the switching mechanism. The RISC agent tries to switch when it’s entering zones where it has high Q-values, while the RC agent takes a more traditional curriculum approach of trying to switch back when encountering states it has very low Q-values for. The experiment is to demonstrate the point that ideas behind things such as reverse curriculums and the way they are used in episodic settings might not transfer well to reset-free settings.

Reviewer comment (Evaluation):
Analysis on Fig.3 (the last paragraph of section 5.1), you mentioned the RISC agent tends to visit areas where Q-values are low. But in the third/fourth column of Fig.3, it seems like the RC agent also tends to visit areas where Q-values are low? In the third column, the RISC agent actually gathers lots of data in the first room where Q-values are already quite high.

Author response:
**[RC vs RISC analysis in Figure 3]**
This experiment was just a qualitative example of the trend that occurs with RC and RISC agents. Column 3 is a slight outlier in showing the RC agent having better exploration than the RISC agent, but you can also see that the level of exploration that the RC agent reaches in column 4 already happens in the RISC agent by column 2. In Column 4, the RISC agent almost exclusively explores the region of the state space that it hasn’t learned yet.

Reviewer comment (Evaluation):
There’s a performance drop on sawyer peg tasks, do you have any intuitions on the decrease of performance?

Author response:
**[Performance drop on Sawyer Peg]**
Near the end of training, one of our seeds for that task diverged in performance. It’s not clear why that happened.

Reviewer comment (Novelty):
There's a recent work called IBC (Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum) from ICML 2023. While it is relatively new, it may be worth discussing it in the related work section without the need for experimental comparisons.

Author response:
**[Discussion of IBC]**
This is quite a relevant work to our paper, as they also try to create curriculums for both the forward and reset agents. Thank you for pointing it out, we will add it to our related work section.

Reviewer comment (Theory):
The paper does not provide a thorough analysis of the theoretical properties of RISC, such as convergence guarantees.

Author response:
**[The paper does not provide a thorough analysis of the theoretical properties of RISC]**: While developing such theory would be useful, Reset-Free RL thus far has been a fairly empirical field, with several prior works [1,2,3,4] being accepted to ICLR and comparable conferences without theoretical results, and developing such theory falls outside the scope of this work. Although we do not provide a proof of the convergence properties of RISC, we do provide a discussion of the role of bootstrapping in the reset-free RL setting in Section 4.1.

Reviewer comment (Experiments):
The experiments are limited to a small set of environments, and it is unclear how RISC would perform on more complex tasks or in other domains.

Author response:
**[Experiments limited to small set of environments]**
We perform experiments across 3 sparse reward robotic manipulation with demonstrations (tabletop, sawyer door, sawyer peg), 1 dense reward robotic navigation without demonstrations (minitaur), and 1 sparse reward gridworld task without demonstrations (4rooms). 
Recently published works in the Reset-Free RL have a less diverse evaluation than our work, such as MEDAL [4] (tabletop, sawyer door, sawyer peg) and VapRL [2] (tabletop, sawyer door, hand manipulation), as these are all sparse reward tasks with demonstrations. In fact, we evaluate on every publicly available environment in the EARL benchmark other than Franka Kitchen, which no reset-free method has made any progress on.

Reviewer comment (Experiments):
How does RISC compare to other reset-free RL algorithms in terms of sample efficiency and generalization?

Author response:
**[Comparison of RISC to other reset-free algorithms in terms of sample efficiency and generalization]**
We refer the reviewer to Figures 4 and 5 (main paper, existing figures) for a summary of RISC’s performance in terms of generalization. Our algorithm has the best performance on two of the evaluated environments, the second best performance on the other two, and across all environments, has the highest mean and interquartile mean. We also have added Figure 7 (Appendix D, new figure) to highlight our method’s gains in sample efficiency compared to prior work. The figure shows the area under the curve of the learning curves, and shows that our method is significantly more sample efficient than the other baselines. Thank you for this helpful suggestion.

Reviewer comment (Theory):
Can RISC be extended to handle environments with irreversible states, where the agent could get stuck?

Author response:
**[Can RISC be extended to handle environments with irreversible states, where the agent could get stuck?]**
Yes, we believe that RISC could be extended to handle environments with irreversible states. It would likely involve making the exploration more conservative, perhaps incorporating ideas from  the risk-averse RL literature [5]. An alternative approach would be to incorporate prior knowledge about the environment and unsafe states into the switching algorithm through the use of large pretrained models.