Paper: RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
The main weakness of this paper is that I am suspicious of the results due to some confusion, and I request that the authors answer the questions below to clarify.

Author response:
Dear Reviewer 48gS,

We would like to express our gratitude for your insightful feedback and acknowledgment of our work. We have carefully considered your questions and provided clarifications below, which will be incorporated into the paper to enhance clarity. We believe that these modifications and clarifications address the weakness you point out about confusion in the paper. Please let us know if all issues have been addressed, or if any further issues remain, as we would be happy to fix these!

Reviewer comment (Experiments):
If the expert intervenes for 5 steps, would all 5 transitions be labeled -1? Wouldn’t this result in actions taken by the expert being labeled -1 too?

Author response:
Expert actions won’t be labeled as -1 when an expert intervenes for 5 steps, only the state and action pair immediately before the intervention begins is labeled -1, this was originally in algorihtm 2 box. The intuition here is that the -1 reward is assigned to the transition (state-action tuple) that most immediately resulted in the intervention. Our theoretical analysis in Section 6 shows that this leads to a good policy (that approximately maximizes the true reward) in theory, and our empirical evaluation shows this in practice. Expert actions are labeled as 0, similar to non-intervention transitions.

Reviewer comment (Reproducibility):
Could the else statement in line 7 of Algorithm 2 set the reward to the RL reward signal if it were available?

Author response:
Yes, that could be a straightforward extension as you suggested. Though in our paper we focused on learning entirely from intervention feedback without knowledge of the ground truth reward.  It’s reasonable to assume the sample efficiency of RLIF would be further improved if it could also access the true task reward.

Reviewer comment (Theory):
There is confusion about the sentence: “the performance of DAgger-like algorithms will be subject to the suboptimality of the experts, while our method can reach good performance even with suboptimal experts by learning from the expert’s decision of when to intervene.“ If no task reward is provided, how could the policy end up outperforming the expert?

Author response:
A crucial assumption for RLIF to work is that the intervention signal itself should contain task-relevant information. For example, in the driving example in the introduction, it is easier for a safety driver to intervene when the car does something bad than it is for them to carry out a perfectly optimal sequence of intervention actions – the choice of when to intervene itself often carries a lot of information. In Section 5.1, we provide one example of an expert intervention model based on Q-values that would carry considerable information in this choice. we also ablate the performance of RLIF  w.r.t. Q functions containing different level of task information; as you hypothesized, the least task-relevant information Q function has, the worse the performance, as in Appendix A.6.  Of course, there is no free lunch, and our method is sensitive to the particular intervention strategy the human uses, as evidenced by the fact that the naive random intervention baseline in our experiments performs much worse. However, this also indicates that our method is effective at deriving a learning signal from the intervention strategy, and the real-world experiments illustrate that real humans do provide interventions that are informative.

Reviewer comment (Evaluation):
How do you end up with a 110% expert if the expert level is with respect to an expert trained to near optimality?

Author response:
The statement about a 110% expert level refers to the expert's evaluation normalized score. In D4RL locomotion environments, we additionally learn expert policies from a curated dataset, so it can be possible that learned expert achieve normalized scores above 100.  And we label them as greater than 100% in expert levels. This was also explained Section 5.2 - results and discussions. We’ll add a more detailed explanation in the final paper.

Reviewer comment (Experiments):
It seems unfair that DAgger variants are given 100 episodes to learn from while RLIF gets 1 million. It is also unrealistic to expect that we could have 1 million episodes in any real-world setting. It is unclear how RLIF would perform if also restricted to a more realistic 10–100 episodes.

Author response:
Thank you for the observation, this is indeed an oversight on our side! 
We would like to clarify that, for RLIF it’s 1M environment steps, so it is 5000 episodes.
We chose 500 episodes for DAgger because the performance of DAgger converged at this point and stopped changing. To show this, we also present results for the Adroit Pen environment for 5000 episodes below, it’s indeed not doing better than training for 500 episodes. 
Additionally, RLIF converges around 500 episodes or earlier most of the time as seen in Appendix A.5.  

|Adroit Pen          | HG-DAgger 5000 episodes| HG-DAgger 500 episodes|
|--------------------|---------------------------|---------------------------|
| 90% Expert         | 76.47                     | 73.47    |
| 40% Expert         | 51.93                     | 60   |
| 8% Expert          | 28.07                     | 28.53  |


Additionally, in our real-world robot experiments in Figure 5, DAgger and RLIF were both trained for fewer than 10 iterations, which amounts to less than an hour of wall clock time. We found RLIF performs better than DAgger in these complex vision-based real-world robotic tasks.

Reviewer comment (Experiments):
In experiments, do you warm start the dataset for RLIF? If so, with what reward? And does this pose an unfair advantage to your algorithm?

Author response:
RLIF's replay buffer is warm-started with expert demonstrations labeled with zero reward, ensuring a fair comparison with DAgger-based methods, which also use the same offline dataset.

Reviewer comment (Experiments):
The paper only compares with DAgger in supervised learning setups, even though DAgger is often used in RL and should have been included as a baseline in such configuration.

Author response:
We compare to DAgger under the same assumptions as our method -- namely, for both DAgger and RLIF, no ground truth reward function is provided, and the algorithm performs rollouts from the current policy with interventions provided by an expert. Could you elaborate on what kind of comparison you would like to see here? We are not aware of any prior work that uses DAgger in RL, but it's entirely possible we missed something (or misunderstood your comment), so if you can provide a reference to a relevant method we should have compared with, we can attempt to include such an evaluation.

Reviewer comment (Theory):
The proposed technique seems to reduce to a heuristic to define a reward function.

Author response:
Our method does not require the expert to define a reward function, but rather to provide interventions. While the way this is used to induce a reward function in RL, that is not a heuristic: we show that this leads to a reward signal that allows our method to recover a policy that is close to the optimal policy (under some assumptions).

Reviewer comment (Novelty):
What is the advantage of the proposed paper versus using DAgger in RL?

Author response:
We are not sure what "using DAgger in RL" means -- could you provide a reference of a method that does this? The advantage of our method over DAgger is illustrated in the empirical comparisons. For example, Table 1 shows our method is 2-3x better than DAgger.

Reviewer comment (Experiments):
The author should include more baselines in Table 1, where the mentioned methods only contain DAgger. For example, some IRL methods could also be applied in the same settings.

Author response:
>Regarding baselines: 

our aim is to develop a reinforcement learning method that can operate under the same assumptions as interactive imitation learning (e.g., DAgger), hence we compare to DAgger, HG-DAgger, and BC. While IRL methods could in principle be adapted to this problem setting also, this would be an awkward fit, as to our knowledge IRL methods do not use interventions.
However, we did conduct new experiments using IQL as an offline RL baseline across all simulation benchmarks, and the results are detailed in Table 1.

Reviewer comment (Experiments):
Does this method require high frequencies to intervene? Can you do some ablations for this? If RLIF cannot work under high frequent intervenes, it can be hard to apply in more complicated real-world scenarios.

Author response:
>Regarding your question about intervention frequencies, 

our real-world robot experiments with vision-based peg insertion and deformable object manipulation (in Section 5) already demonstrated the effectiveness of our method under both sparse and dense interventions, i.e., the user intervenes only a few times, but each intervention length is long vs an user intervenes multiple times for a short time during a rollout. In practical terms, the choice between sparse and dense interventions is problem-dependent, varying according to the specific needs of each scenario. 
We have also included a plot of the average intervention rate over environment steps in Figure 3, illustrating that at the onset of training, the intervention rate tends to be higher, gradually decreasing as the policy improves towards the end of the training process.

Reviewer comment (Novelty):
I am wondering what the difference is between RLHF and your work RLIF when you choose the Space Mouse to give signals by hand. No related works are mentioned and no experiments are conducted for this. For example, what about using some foundation models to give signals?

Author response:
>RLIF vs RLHF:

Thank you for your suggestion! RLHF indeed can be relevant to our work, we’ll add some related works in the final version.

Reviewer comment (Theory):
Though the theoretical analysis does not answer the key problem, how the level of sub-optimality results in what sample complexity of the offline RL agent. The reviewer still enjoys reading the analysis framework proposed here. The reviewer may miss some key contents, but would the authors explain more in the proposed Corollary 6.7 regarding the above key problem?

Author response:
To answer your question regarding the theoretical results:

> how the level of sub-optimality results in what sample complexity of the offline RL agent. The reviewer still enjoys reading the analysis framework proposed here.

Corollary 6.7 provides a sample complexity bound of learning a policy using the intervention reward $\tilde{r}$. Since the intervention strategy is generated by the expert $\pi^{\exp}$, which eventually affects the suboptimality gap in a linear relationship w.r.t. $\Delta_{ref}$, the visitation distribution gap between the reference policy $\pi^{ref}$ and the optimal policy $\pi^{\star}$.

Reviewer comment (Experiments):
There is no offline-RL baselines provided. Is it because that the true reward function is unknown in the evaluation results? Otherwise, please add at least 1 offlineRL baseline in all the simulation based experiments.

Author response:
> Concerning your question about offline RL baselines

First, RLIF in principle can be integrated with any off-policy RL algorithm. We chose RLPD because of its good sample efficiency. That said, per your suggestion, we conducted additional experiments incorporating IQL as an offline RL baseline across all simulation benchmarks. Please find the detailed results in Table 1. It's worth noting that, in all our experiments, true rewards were not provided, as our algorithm is designed to learn an optimal policy with only intervention feedback.

Upon examination, the results from the IQL experiments indicate a performance that lags behind the outcomes achieved using the online algorithm RLPD. This observation aligns with our expectations because it’s slow for IQL to explore new actions.

Thank you for your continued engagement with our work, and we remain open to any further questions or suggestions you may have.

Table 1: IQL Baseline

|                        | IQL with Random Interventions | IQL with Value Based Interventions |
|------------------------|------------------------------|------------------------------------|
| **adroit-pen**         |                              |                                    |
| Expert Levels          | 68.83                        | 42.5                               |
| 90% Expert             | 25.92                        | 46.58                              |
| 40% Expert             | 12.08                        | 40.67                              |
| 8% Expert              | 35.61                        | 43.25                              |
|                        | 80.22                        | 20.37                              |
| **locomotion-walker2d**|                              |                                    |
| 110 Expert             | 56.33                        | 47.97                              |
| 70 Expert              | 14.7                         | 47.02                              |
| 20 Expert              | 50.42                        | 38.45                              |
|                        | 37.91                        | 26.37                              |
| **locomotion-hopper**   |                              |                                    |
| 110 Expert             | 27.88                        | 28.98                              |
| 40 Expert              | 16.64                        | 25.32                              |
| 15 Expert              | 27                           | 26.89                              |