Paper: Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency (ICLR-2024)

Reviewer comment (Theory):
The assumption of resetting from demonstrations is a strong one and only applicable in simulation environments. While this study proposes an efficient way to leverage one or more demonstrations through the reverse curriculum, I still consider it to be a strong assumption.

Author response:
We agree that this is a strong assumption, as is the case for many other papers in our research area focusing on robot learning ([1], [2]). We argue however that work on sim2real is orthogonal to our research and thus investigating the most efficient algorithms in simulation is of great value. Papers like [1] have shown using sim2real can still achieve some fairly dexterous capabilities on real robots despite training only in simulation. Additionally, another use case that arises from using simulation is having an algorithm that trains much faster and uses less demos in simulation can help generate more diverse simulation data to be used with limited real world data to help scale up training data for large-scale robotics models.

Reviewer comment (Novelty):
The study may appear to be not significantly different from consecutively performing Jump-Start RL and PLR.

Author response:
In general our reverse curriculum is fairly different to Jump-Start RL (JSRL). JSRL suffers from exploration problems in sparse reward settings. We alleviate some significant exploration problems by efficiently leveraging state resets. Moreover, we construct a unique per-demo curriculum and introduce a few techniques like dynamic time limits that JSRL does not have which enable faster training as shown in ablations.

Reviewer comment (Experiments):
While it compares to various baselines, many of them seem to be algorithms that do not assume state resets or use curricula.

Author response:
JSRL is the only baseline that uses some form of reverse curriculum with comparable results (they test on proper benchmarks that have code). We do cite several past methods that use reverse curricula and state reset but they do not have code and often test on simple tasks (e.g. maze navigation) outside of robotics that also have no code. As an alternative, we perform an ablation testing the use of a uniform state reset that methods like [2] use in section 5.2 (Table 1) instead of the reverse curriculum RFCL uses. Across ManiSkill2 tasks, uniform state reset is incapable of reaching high success rates within 1M environment interactions when evaluating on initial states $s\_{i, 0}$ in the 5 demonstrations given. Finally, methods like [3] use curriculums but assume access to human-engineered curriculums which is a stronger assumption so they are hard to compare fairly.

We have only found one open-sourced baseline that claims to use state reset which is the one by Nair et. al. [2]; however we could not get their code working correctly as upon further inspection, their code does not use state resets. We will run our RFCL method on the environments Nair et. al. [2] test on and compare against their written results obtained via state resets and report here. [1] also uses state reset and has some code but is not optimizing to be sample efficient as it uses PPO so it would be unfair to compare against (in addition to relying on a number of human designed heuristics).

For now, to compare against [2], you can look at the PickCube results for RFCL in Fig. 5 which is a very similar environment to the Pick and Place task [2] uses. The main difference is our task has a larger action space as we can control the pose of the end-effector (necessary to help grasp rotated cubes) whereas in [2] you are fixed to controlling just the position of the end-effector to grasp cubes with fixed rotation. RFCL with just 5 demos is capable of achieving a high success rate after 250k samples whereas [2] needs about 500k+ samples despite using 100 demonstrations and solving a easier task.

[1] Chen et. al, “Sequential Dexterity” CoRL, 2023

[2] Nair et. al, “Overcoming exploration in reinforcement learning with demonstrations” ICRA, 2018

[3] Li et. al, “Towards practical multi-object manipulation using relational reinforcement learning” ICRA, 2020

Reviewer comment (Presentation):
In Figure 6, it seems that 'reverse' and 'forward' are only applicable to 'reverse-forward,' and not to 'none' and 'forward only,' which might be confusing.

Author response:
This is actually the correct interpretation. We are simply marking when the reverse and forward curriculum are active. We have added an extra bracket under the second row indicating that the forward curriculum is constantly running for that ablation.

Reviewer comment (Reproducibility):
Which algorithms among the compared baselines require the State Reset assumption?

Author response:
JSRL is the only baseline that uses some form of reverse curriculum with comparable results (they test on proper benchmarks that have code). We do cite several past methods that use reverse curricula and state reset but they do not have code and often test on simple tasks (e.g. maze navigation) outside of robotics that also have no code. As an alternative, we perform an ablation testing the use of a uniform state reset that methods like [2] use in section 5.2 (Table 1) instead of the reverse curriculum RFCL uses. Across ManiSkill2 tasks, uniform state reset is incapable of reaching high success rates within 1M environment interactions when evaluating on initial states $s\_{i, 0}$ in the 5 demonstrations given. Finally, methods like [3] use curriculums but assume access to human-engineered curriculums which is a stronger assumption so they are hard to compare fairly.

We have only found one open-sourced baseline that claims to use state reset which is the one by Nair et. al. [2]; however we could not get their code working correctly as upon further inspection, their code does not use state resets. We will run our RFCL method on the environments Nair et. al. [2] test on and compare against their written results obtained via state resets and report here. [1] also uses state reset and has some code but is not optimizing to be sample efficient as it uses PPO so it would be unfair to compare against (in addition to relying on a number of human designed heuristics).

For now, to compare against [2], you can look at the PickCube results for RFCL in Fig. 5 which is a very similar environment to the Pick and Place task [2] uses. The main difference is our task has a larger action space as we can control the pose of the end-effector (necessary to help grasp rotated cubes) whereas in [2] you are fixed to controlling just the position of the end-effector to grasp cubes with fixed rotation. RFCL with just 5 demos is capable of achieving a high success rate after 250k samples whereas [2] needs about 500k+ samples despite using 100 demonstrations and solving a easier task.

[1] Chen et. al, “Sequential Dexterity” CoRL, 2023

[2] Nair et. al, “Overcoming exploration in reinforcement learning with demonstrations” ICRA, 2018

[3] Li et. al, “Towards practical multi-object manipulation using relational reinforcement learning” ICRA, 2020

Reviewer comment (Evaluation):
If possible, could you please explain the reason for the initial high performance of RLPD in the stick pull task in Figure 4, followed by a drop?

Author response:
Unfortunately it is difficult to say why this has occurred, although unlikely, maybe some data corrupted when uploading training results. We are currently rerunning RLPD with the same 5 seeds plus 5 more seeds on the stick pull task and will re-record results, it might also just be some strange noise. Overall however the conclusions remain the same about RFCL's performance relative to RLPD.

Reviewer comment (Writing):
Please do not use the word 'extreme'; you used it several places throughout the paper.

Author response:
Revised!

Reviewer comment (Theory):
Regarding the statement "In practice, the observations used by the policy πθ may be different from the actual environment state but for simplicity in this paper state also refers to observation.": So you don't consider noisy feedback or POMDPs? This should be mentioned clearly in the introduction. As the method shows significant improvement in learning curves, it is vital to indicate when/where we expect them to hold and where they would fail.

Author response:
In our revision we have clarified in the introduction that we only benchmark on fully observable environments (as also done by benchmarked baselines).

Reviewer comment (Writing):
Minor comment: "As a result, a curriculum for each demonstration is necessary as opposed to a curriculum constructed from all demonstrations as done in prior work in order to ensure noisy information arising from the multi-modality of demonstrations do not impact the reverse curriculum of each demonstration as much." This is too long a sentence.

Author response:
Revised!

Reviewer comment (Writing):
You use \phi before you introduce it, and I didn't get what it's supposed to mean.

Author response:
We have revised the paper to clarify that $\phi$ is a hyperparameter indicating the ratio of the demonstration length to the episode horizon. A larger value would then mean shorter episodes relative to how long the demonstrator took, and vice versa.

Reviewer comment (Writing):
Regarding the statement "In this manner, initial states that sometimes receive return are prioritized the most, then initial states that receive no return, then initial states that are consistently receiving return.": This is not a very clear sentence. Do you want to use 'than' instead?

Author response:
Thanks for pointing this out, we have made a revision to make it more clear.

Reviewer comment (Experiments):
It would be nice to discuss why the methods compared against were chosen out of all the possible RL algorithms out there (maybe in an appendix?). Would the others be unsuitable for the task (e.g., on-policy, not suitable for demonstrations, etc.)?

Author response:
This is a good point, we add to appendix B.5 how baselines were chosen and why some were not chosen. Generally we chose off-policy learning-from-demonstrations baselines as they are the most sample-efficient on our benchmarks and could not include some baselines because they do not have any code or tests on open-sourced tasks. We are actively trying to compare against at least one more baseline that uses state resets as we describe in response to reviewer gfCC.

Reviewer comment (Experiments):
What if there is no demonstration? Or intentionally forget about demonstration but just design curriculum that directly moves the reset/initial states away from goal? If this paper's assumption is that in the end, the initial/reset states should be able to cover the entire space anyway, then whether there is demonstrations provided should not matter: all starting states have to be explored sooner or later. I didn't see an ablation experiment to compare against the setting where no demonstration is available.

Author response:
Thank you for raising this comment. We believe there may be a misunderstanding of the initial state distribution and what is inside it. In our paper, the initial state distribution $\mathcal{S}\_{\text{init}}$ refers to the distribution of states the environment resets to (e.g. reset robot to a rest position, place objects in random locations not near the goal locations). Critically, the **demonstrations provide states along a successful trajectory which are states outside of the initial state distribution**. Without a demonstration to reset states to, we have a very difficult exploration problem as the benchmarked tasks have high dimensional observation and action spaces, making it nearly impossible under sparse reward to explore to states near the goal and get reward. 

It is not possible to sample a state near the goal without using a demonstration, extra human-engineered code, or infeasible amounts of exploration. Designing a curriculum by hand would be time-consuming especially for these high dimensional tasks and human demonstrations (e.g. the demos in the Adroit benchmark) are easier to collect and need less expertise. That being said, we think it is interesting future work to explore how one can use simple human written curriculums (e.g. via language description) to accelerate policy learning.

Running any RL algorithm without using demonstrations should achieve 0 success rate under the same compute budgets for the majority of tasks benchmarked due to the exploration problem under sparse reward. We will run no-demonstration experiments on some tasks with and without the forward curriculum and post results here.

Reviewer comment (Experiments):
In Figure 5, do all experiments have the same reset/initial state distribution (cover all possible states) at the end of their curricula? If so, it's so hard to understand why the more demonstration there is, the faster it can be trained.

Author response:
After the reverse curriculum the policy can achieve high success rates if we initialize from the start of those demonstrations $\{s\_{i, 0}\}$, the first state of each demonstration $i$. Note that all $\{s\_{i, 0}\}$ are usually from $\mathcal{S}\_{\text{init}}$. With more demonstrations, the policy after reverse curriculum is more generalized across different initial states, meaning it can train faster as it does not need to spend as much time during the forward curriculum learning to generalize. The improved generalization can be seen in Fig. 5, the success rate curves are already trending up before the reverse curriculum completes (marked by the vertical dashed gray line) in higher-demonstration settings.

Furthermore, more demonstrations means more ground-truth data of hard to reach states (e.g. peg inside the hole for the peg insertion task) with observation, action, and sparse reward labels to train with, which in general for any learning from demonstrations algorithm (e.g. DAPG, RLPD) will improve the performance.

Reviewer comment (Novelty):
It seems that RFCL without "forward curriculum" works almost the same as RFCL with "forward curriculum" in most tasks (Fig 5). This again questions the necessity of having a second stage, as mentioned earlier. Without the two-stage setting, the novelty of this work is negatively impacted.

Author response:
In particular, we emphasize that with a forward curriculum in RFCL we can achieve better demonstration efficiency. This is seen clearly by the blue line (RFCL) outperforming the green line (RFCL w/o forward curriculum) for settings with less demonstrations. For PegInsertionSide, with 10 demos or less the forward curriculum has much better success rates. For PickCube, the same occurs on the 1 demo ablation. When there are enough demonstrations, the reverse curriculum trained policy has much better generalization and thus the forward curriculum provides less noticeable (if any) benefits. The improved generalization of the reverse curriculum training is evidenced by the success rate curves already increasing before the dashed gray line in Fig. 5 (when the reverse curriculum completes).

To further emphasize the importance of combining reverse and forward curricula, we will add some additional demo ablations comparing with and without forward curriculum results on some hard tasks (like Plug Charger) and post results here (and add to appendix due to space limitations).

Reviewer comment (Experiments):
I'm not sure if the tasks selected in this paper are suitable for the proposed method: If the robot arms are moved by a positional controller, then the initial/reset position/states of the robot arm does not matter, because the positional controller should guide the robot arm/end effector to converge to the goal position anyway regardless of initial states. Not much exploration is needed. So if this is the case, then the selected task cannot fully evaluate the potential of the proposed method.

Author response:
Thanks for asking about the controller, we revised appendix B.1 to include details on controllers. We use an end-effector delta pose or delta position controller (depending on what the benchmark provides), but most initial state randomness does not stem from the robot's joints being randomized. Regarding the appropriateness of task, there is a significant amount of exploration among initial states required as the initial state distribution contains randomizations of other properties of the tasks, such as the object shape in PegInsertionSide, object poses, and the goal locations (all tasks randomize goal, ManiSkill2 randomizes the most). For visuals of some of the initial state distribution, see Appendix B.2 / check the website section https://reverseforward-cl.github.io/#task-visuals-a.

Robot arms are not moved by an absolute position controller (e.g. predicting absolute position, pose, or joint angles). While it could be, this is not the standard adopted in these benchmarks and used by other algorithms.

We hope we are able to address all your concerns. Considering our response, would you be able to raise your score? We would be happy to discuss any further issues / questions you may have.

Reviewer comment (Experiments):
If the robot arms are not moved by a positional controller, then why not do so?

Author response:
Thanks for asking about the controller, we revised appendix B.1 to include details on controllers. We use an end-effector delta pose or delta position controller (depending on what the benchmark provides), but most initial state randomness does not stem from the robot's joints being randomized. Regarding the appropriateness of task, there is a significant amount of exploration among initial states required as the initial state distribution contains randomizations of other properties of the tasks, such as the object shape in PegInsertionSide, object poses, and the goal locations (all tasks randomize goal, ManiSkill2 randomizes the most). For visuals of some of the initial state distribution, see Appendix B.2 / check the website section https://reverseforward-cl.github.io/#task-visuals-a.

Robot arms are not moved by an absolute position controller (e.g. predicting absolute position, pose, or joint angles). While it could be, this is not the standard adopted in these benchmarks and used by other algorithms.

We hope we are able to address all your concerns. Considering our response, would you be able to raise your score? We would be happy to discuss any further issues / questions you may have.