Paper: Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR-2024)

Reviewer comment (Writing):
The necessary details for understanding equation (12) are not clearly presented.

Author response:
***A:*** Thanks for asking this question. We have updated the Eq. 12 in the revised version to address this confusion. Essentially, we first compute $k^* = argmax_{k\in[{|\Pi^E|}]}\hat{V}^k_{\mu}(s)$ to get the index $k^*$ of the policy with the highest value. If the estimated standard deviation for the value ($\sigma_{k^*}(s)$) is larger than a threshold, we use the estimated value for the learner policy as $\hat{f}^+(s)$. If not we use $\hat{f}^+(s) = \hat{V}^{k^*}_{\mu}(s)$.

Reviewer comment (Reproducibility):
The value ensemble is central to the proposed method, yet there are only two sentences at the bottom of page 5 discussing how the value ensemble is trained.

Author response:
***A:*** Thanks for asking the question. We add a more detail about the value ensemble network including how it is trained in the Appendix D.5 Hyperparameter and architectures. Please check our updated PDF file for more details.

Reviewer comment (Novelty):
The theory section borrows from MAMBA, which is natural since these methods are closely related, but it would have been clearer to emphasize the points of departure from MAMBA more and perhaps move the parts that are more closely shared to the appendix.

Author response:
***A:*** We appreciate your feedback! We agree with the reviewer that there are similarities with MAMBA and that the paper can be clearer regarding how RPI differs. Despite these similarities, we respectfully note that our theory is not directly borrowed from MAMBA. Different from MAMBA, our approach considers an oracle set that includes the learner policy, which we call *extended oracle set*. With this change, the (extended) oracle set that the algorithm work with is not static anymore (due to the improvement of learner's policy). This changes the theoretical analysis in a non-trivial manner. While Proposition 5.1 is marked as "adapted from MAMBA," our bound differs from MAMBA's for this reason. We updated the main paper to make this departure from MAMBA clear.  

***A:*** We agree that section 4 has similarity to MAMBA, as both share the same problem setting. The main difference arises from lifting the static oracle set into our dynamic extended oracle set. However, to formally state the problem and establish our algorithm and analysis, we find it necessary to rigorously set up the notations and keep the work self-contained, which unavoidably overlaps with MAPS/MAMBA as a special case. In fact, many of the notations also inherit from MAMBA due to the shared problem setting, and we have moved most of the background discussion to `Appendix B1` (“Additional algorithms for learning from multiple oracles”) for clarity. In addition, to make it clearer, we added a footnote in section 4 where we explicitly state, "When we exclude the learner’s policy from the extended oracle set, this reduces to the max-aggregation policy, which was used in MAMBA (Cheng et al., 2020)" to elucidate the connection with existing work (Please find details in the revised version.). Starting from Section 5, our approach clearly departs from MAMBA as it really focus on selecting between learner's policy and oracle policies, as well as the robust update in policy gradient.

Reviewer comment (Novelty):
The definition of policy gradient with advantage is cited for Schulman, but there are prior works considering policy gradients with advantage estimation, e.g., [2].

Author response:
***A:*** Thank you for catching this! We added the following references to the paper.  

Sutton et al., Policy Gradient Methods for Reinforcement Learning with Function Approximation. JMLR, 1999  

Greensmith et al., Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. NeurIPS, 2004

Reviewer comment (Experiments):
Computing the update requires evaluating the value ensembles for all of the oracles. Compared to a normal RL algorithm, e.g., PPO, this means number of oracles times the size of ensemble more forward passes. Depending on the number of oracles and ensemble members, this may equate to a significantly more expensive loss evaluation. In addition to the missing details about the ensemble size and number of oracles, it would be good to include some wall-time comparisons between the different methods.

Author response:
***A:*** Thank you for the suggestion! We have added Table 4 in Appendix D.6 "Computing Infrastructure and Wall-Time Comparison" that provides the wall-time comparison that you requested.

Reviewer comment (Novelty):
Including a thorough discussion about the relationship of the proposed method and MAMBA as well as limitations would make the paper easier to understand.

Author response:
***A:*** Thank you for your advice! We have added a detailed paragraph in Appendix A.1 (additional notes on related work) where we discuss the relationship between the proposed method and MAMBA, as well as their limitations (see Appendix B.2 limitations of the prior art). Please see the updated PDF. In the meantime, we created the following table in hopes that it addresses your questions/concerns.  

|         | RPI    | MAPS [ICML'23] | MAMBA [Neurips'20] |
|--------      |--------     |--------       |  --------   |
| Objective| Perform active IL and RL for robust policy improvement from multiple suboptimal oracles. Addressed the limitation of MAPS and MAMBA with regard to the robustness.  | Improve the sample efficiency of MAMBA by active policy selection (For MAP-SE, it also proposed active state exploration)| Perform online imitation learning from multiple suboptimal oracles. |
| Contribution | (1) Proposed the $\text{max}^+$ framework for robust learning (2) Proposed $\text{max}^+$-aggregation baseline  (3) Robust policy improvement (RPI) algorithm (4) Robust Active Policy Selection strategy (RAPS)  (5) Robust Policy Gradient (RPG) (6) Provided theoretical analysis. | MAPS (1) Active policy selection (APS) (2) Provided theoretical analysis. | (1) Proposed max-aggregation baseline. (2) Novel algorithm MAMBA, max-aggregated baseline. (3) Regret-based performance guarantee for MAMBA.
| Experiments |(1) Cheetah (Dense) (2) Walker-walk (Dense) (3) Cartpole (Dense) (4) Pendulum (Sparse) (5) Faucet-open (Dense) (6) Faucet-open (Sparse) (7) Drawer-close (Dense) (8) Drawer-close (Sparse) (9) Button-press (Sparse) (10)Button-press (Dense) (11) Window-close (Sparse) (12) Window-close (Dense)| (1)Cheetah (Dense)  (2)Walker-walk (Dense) (3)Cartpole (Dense) (4)Pendulum (Sparse)| (1) DIP (Dense) (2) Cheetah (Dense) (3) Ant (Dense) (4) Cartpole  (Dense)|
| Baseline  | (1)MAPS (2)MAMBA (3)Max-aggregation (4)LOKI-variant (5)PPO-GAE (6)Best oracle| (1)MAMBA (2)PPO-GAE (3)Best oracle | (1)PG-GAE (2)Best oracle|
| Limitation  |    lacking robustness guarantee against the optimal pure RL policy (a limitation shared by MAPS/MAMBA, although RPI demonstrates superior empirically performance).   | Not robust to the oracle quality (such as pure adversarial oracle setting)   | (1) No active oracle selection (Sample inefficiency)  (2) Not robust to the oracle quality (such as pure adversarial oracle setting) |

Reviewer comment (Theory):
Regarding equation (12), what does it mean when the "when"-clause equates to false?

Author response:
***A:*** Thanks for asking this question. We have updated the Eq. 12 in the revised version to address this confusion. Essentially, we first compute $k^* = argmax_{k\in[{|\Pi^E|}]}\hat{V}^k_{\mu}(s)$ to get the index $k^*$ of the policy with the highest value. If the estimated standard deviation for the value ($\sigma_{k^*}(s)$) is larger than a threshold, we use the estimated value for the learner policy as $\hat{f}^+(s)$. If not we use $\hat{f}^+(s) = \hat{V}^{k^*}_{\mu}(s)$.

Reviewer comment (Writing):
Regarding equation (12), what does $\hat{V}_{\mu}^{|\Pi^{\varepsilon}|}$ mean?

Author response:
***A:***  $\hat{V}_{\mu}^{|\Pi^E|}$ refers to the estimated value function of the learner policy, where $|\Pi^E|$ denotes the index of the extended policy, which is the learner policy.  

To make it clearer, we update it to $\hat{V}_{\mu}^{\pi_n}$. Please check Eq. 12 in revised version for details.

Reviewer comment (Reproducibility):
How many members are there in the value-function ensemble?

Author response:
***A:*** We followed the state-of-the-art MAPS for value-function ensemble numbers and set it to five. Please check the Appendix D.5 Hyperparameters and architectures for more details.

Reviewer comment (Experiments):
How many oracles does each experiment have?

Author response:
***A:*** In each experiment, we have three diversified oracles. Please see the updated Table 3 in Appendix D.5 and the caption of Figure 2.

Reviewer comment (Experiments):
Why are the replay buffers for the oracles so small? The oracle policies do not change, so the data does not become more off-policy as the training proceeds.

Author response:
***A:*** We set the size of replay buffer to $19,200$. Indeed, the oracle policies are fixed and increasing replay buffer seem to have a positive effect, however, in our preliminary experiments (not reported on the paper) we figured that further increasing the replay buffer does not affect the performance.

Reviewer comment (Reproducibility):
How many timesteps are sampled for value-function pre-training?

Author response:
***A:*** In order to pretrain a value function for each oracle, we collected 8 episodes (it amounts to 8k steps for DMControl, 2.4k steps for MetaWorld) per oracle. For each training step of the RPI algorithm, the environment is queried for 4 episodes (roll-in + roll-out), followed by 2,048 steps (rolling out the learner's policy).

Reviewer comment (Evaluation):
How would the learning curves look if the value-function pre-training steps were included on the x-axis?

Author response:
***A:*** In order to pretrain a value function for each oracle, we collected 8 episodes (it amounts to 8k steps for DMControl, 2.4k steps for MetaWorld) per oracle. For each training step of the RPI algorithm, the environment is queried for 4 episodes (roll-in + roll-out), followed by 2,048 steps (rolling out the learner's policy).

Reviewer comment (Experiments):
How would the following naive baseline perform? Initialize from the expert policy, and then finetune from there with RL. It seems like this baseline could perform quite well in the proposed experimental setup, given that PPO-GAE is one of the baselines and performs quite well.

Author response:
***A:*** We appreciate the suggestion! This baseline is indeed intriguing, and we have implemented and tested a similar one called MAPS-ALL (see Appendix E.1 that compares RPI against baselines with a data advantage). In MAPS-ALL, the learner policy has access to the initial pretrain transitions given by rolling-out the oracles, which is approximately twice the size accessible to the RPI algorithm. We initialize the learner policy using this extended offline data. We observe that such initialization enhances performance in the Pendulum environment (Figure 5 (c)). However, despite using significantly less data, RPI still outperforms MAPS-ALL in Walker-walk and Cartpole, and this difference is statistically significant.  

In addition, we followed the approach of previous works, such as MAPS and MAMBA, in using the best-so-far plot. However, we observed that MAPS-ALL often does not perform as well as expected. The primary challenge arises from "catastrophic forget" or "performance drop", which, despite not being reflected in the plot, occurs when transitioning from offline training to online fine-tuning. This challenge is a common issue in offline-to-online reinforcement learning. The prevailing method to address catastrophic forgetting involves adopting offline pessimism to online pessimism, but this often leads to suboptimal policies. Another approach is addressing catastrophic forgetting through online reinforcement learning with offline data. In contrast to the offline-to-online approach, MAPS, our RPI, demonstrates greater robustness and stability.

Reviewer comment (Novelty):
The prior work section is rather brief with a total of 6 citations. I am not familiar with the exact sub-field but this seems rather little given the literature is several years old.

Author response:
***A:*** We had originally focused the discussion on methods we felt to be most relevant to our work, while providing additional references in Appendix A "Selective Comparison Against Related Work". Following the reviewer's advice, however, we have extended the related work section with a discussion of additional methods.

Reviewer comment (Novelty):
My first main concern is related to the contextualization of various parts of the paper to previous work. I think it could be made clearer which parts are inspired by previous work. For instance, the ideas of UCB value selection and multi-step advantage over several value functions seem already present in previous work but this is not immediately apparent from the current manuscript. I understand that the present manuscript defines these things with the inclusion of an additional learner policy, which requires the development of novel machinery. However, I think it would be good to give credit to the general formulations in previous work where appropriate.

Author response:
***A:*** Thank you for the advice! We agree that the paper would benefit from a discussion that better places RPI in the context of existing work. We created the following table in an effort to clarify the relationship between RPI and what we believe are the most relevant existing methods. Additionally, we added a detailed paragraph to Appendix A.1 that discusses related work.



|         | RPI    | MAPS [ICML'23] | MAMBA [Neurips'20] |
|--------      |--------     |--------       |  --------   |
| Objective| Perform active IL and RL for robust policy improvement from multiple suboptimal oracles. Addressed the limitation of MAPS and MAMBA with regard to the robustness.  | Improve the sample efficiency of MAMBA by active policy selection (For MAP-SE, it also proposed active state exploration)| Perform online imitation learning from multiple suboptimal oracles. |
| Contribution | (1) Proposed the $\text{max}^+$ framework for robust learning (2) Proposed $\text{max}^+$-aggregation baseline  (3) Robust policy improvement (RPI) algorithm (4) Robust Active Policy Selection strategy (RAPS)  (5) Robust Policy Gradient (RPG) (6) Provided theoretical analysis. | MAPS (1) Active policy selection (APS) (2) Provided theoretical analysis. | (1) Proposed max-aggregation baseline. (2) Novel algorithm MAMBA, max-aggregated baseline. (3) Regret-based performance guarantee for MAMBA.
| Experiments |(1) Cheetah (Dense) (2) Walker-walk (Dense) (3) Cartpole (Dense) (4) Pendulum (Sparse) (5) Faucet-open (Dense) (6) Faucet-open (Sparse) (7) Drawer-close (Dense) (8) Drawer-close (Sparse) (9) Button-press (Sparse) (10)Button-press (Dense) (11) Window-close (Sparse) (12) Window-close (Dense)| (1)Cheetah (Dense)  (2)Walker-walk (Dense) (3)Cartpole (Dense) (4)Pendulum (Sparse)| (1) DIP (Dense) (2) Cheetah (Dense) (3) Ant (Dense) (4) Cartpole  (Dense)|
| Baseline  | (1)MAPS (2)MAMBA (3)Max-aggregation (4)LOKI-variant (5)PPO-GAE (6)Best oracle| (1)MAMBA (2)PPO-GAE (3)Best oracle | (1)PG-GAE (2)Best oracle|
| Limitation  |    lacking robustness guarantee against the optimal pure RL policy (a limitation shared by MAPS/MAMBA, although RPI demonstrates superior empirically performance).   | Not robust to the oracle quality (such as pure adversarial oracle setting)   | (1) No active oracle selection (Sample inefficiency)  (2) Not robust to the oracle quality (such as pure adversarial oracle setting) |

Reviewer comment (Novelty):
This might be a me thing but the term robust policy gradient has been used several times in the literature before. It might make sense to consider a less generic name for the algorithm. Examples:  
Xuezhou Zhang et al. Robust Policy Gradient against Strong Data Corruption. ICML 2021.  
Yue Wang and Shaofeng Zou. Policy Gradient Method For Robust Reinforcement Learning. ICML 2022.

Author response:
***A:*** Thanks for the advice! We are happy to use a more specific term and welcome any suggestions that you might have.

Reviewer comment (Novelty):
I’m being a little nitpicky here but the policy gradient theorem with baseline is definitely older than 2015 (P6 section 6.2). It might be good to cite relevant prior work here.

Author response:
***A:*** The reviewer is correct. Thank you for pointing this out. We have updated the text to include citations to the following papers.

Sutton et al., Policy Gradient Methods for Reinforcement Learning with Function Approximation. JMLR, 1999

Greensmith et al., Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. NeurIPS, 2004

Reviewer comment (Novelty):
The policy improvement of perfect knowledge is similar to making ensembles of several imitated policies, and the theoretical analysis of the method seems a little redundant.

Author response:
***A:*** We understand that our approach and ensembles of imitated policies may seem similar, but our active RL and IL strategy introduces novel technical challenges. Importantly, we do *not* assume "perfect knowledge" of the experts---unlike standard ensemble imitation, our approach requires state-wise differentiation to identify the most effective oracle in any given context (state). Moreover, a crucial aspect of our methodology is the active decision-making process, i.e., choosing between imitating the best policy or focusing on self-improvement. This aspect is particularly critical in scenarios where imitating a suboptimal policy can be detrimental, a challenge not addressed by existing ensemble-based approaches. Our learner policy is designed to determine which expert policy to follow based on the state, further differentiating our approach from traditional ensembles.

Reviewer comment (Novelty):
The exploration in RPI is uniformly random sampling (line 3 of Alg.1), which seems trivial.

Author response:
***A:*** Please note that we are not claiming the uniform sampling of the roll-in horizon $t_e$ in Line 3 of Alg. 1 as novelty of this work. Rather, we focus on the robust *policy selection* and robust *policy gradient* algorithm, which ensures active exploration of RPI.

Selecting an oracle to imitate uniformly at random from the learner’s state distribution is a standard procedure in online imitation learning. As the reviewer correctly points out, this strategy, employed in several classic online IL algorithms like Dagger (Ross et al., 2011), Aggravate (Sun et al., 2017), THOR (Sun et al.,2018), MAMBA (Cheng et al.,2020), and MAPS (Liu et al.,2023), serves as a common practice for conducting imitation learning based on the learner's distribution, ensuring comparability with existing methodologies.

Reviewer comment (Experiments):
The experimental setting is not clear and sufficient, particularly in terms of details such as how many demonstrations are used for imitation learning, how many online interactions are used for reinforcement learning, what "100 training step" means on the x-axis of the curves, and whether the baselines use the same examples and online interactions.

Author response:
***A:*** Thank you for raising these questions. We would like to note that most of these experimental details were presented in the appendix (D.3 setup D.4 implementation details and D.5 hyperparameter table) and in the source code---both included in the supplemental materials---for reproducibility of our results. In terms of your specific questions:
* We will explain the details of each training iteration as follows. For each training iteration of the RPI algorithm, the environment is queried for 4 episodes (roll-in + roll-out; this amounts to 4k steps for DMControl suite, 1.2k steps for MetaWorld tasks), followed by 2,048 steps (only rolling out the learner's policy). Thus, each training iteration involves 6,048 steps of environment interactions for DMControl suite (or 3,248 steps of interactions for MetaWorld).
* To elaborate on the learner's policy update within a single training iteration, we will give a little bit of implementation details: 1. We first prepare a set of transitions. This comes from the roll-in (see Alg. 1 line 4) and full roll-out (see Alg. 1 line 6) of the learner policy. 2. We then create a set of mini-batches (with size 128) by grouping the transitions. 3. We iteratively perform gradient updates on the mini-batches for 2 epochs (i.e., going over all of the mini-batches twice). This entire process makes up a single training step.
* All baselines were subjected to an equal number of environment interaction steps to maintain consistency, except for a minor difference in PPO-GAE (please see Appendix D.3 Setup). 

We have updated the text in the main paper and appendix to make sure that the experimental details are clear.

Reviewer comment (Experiments):
The selected tasks are easy. In meta-world, button-press and window-close are not considered hard tasks; it is unclear why more challenging tasks like bin-picking and door-open were not included. In DMC, harder tasks such as hopper or humanoid could have been considered.

Author response:
***A:*** Please note that the tasks considered were variants of the standard setting, tailored towards the evaluation of imitation learning with multiple blackbox oracles:

* Button-press and window-close are originally dense-reward environments (see Appendix E.2 Meta-world experiments (dense reward)). However, we converted the environment to have a sparse reward, where a reward is given only when the window is closed or a button is pressed. Thus, most of the time, the learner doesn't receive any feedback. Compared to the original dense reward environment, this is significantly harder learning problem. 

* We chose these tasks to facilitate comparisons with the most relevant work in online imitation learning as they are a superset of the standard baselines for this domain. Please refer to the following table, which compares tasks across methods, and kindly check our updated PDF for more details.


|         | RPI    | MAPS [ICML'23] | MAMBA [Neurips'20] |
|--------      |--------     |--------       |  --------   |
| Experiments |(1) Cheetah (Dense) (2) Walker-walk (Dense) (3) Cartpole (Dense) (4) Pendulum (Sparse) (5) Faucet-open (Dense) (6) Faucet-open (Sparse) (7) Drawer-close (Dense) (8) Drawer-close (Sparse) (9) Button-press (Sparse) (10)Button-press (Dense) (11) Window-close (Sparse) (12) Window-close (Dense)| (1)Cheetah (Dense)  (2)Walker-walk (Dense) (3)Cartpole (Dense) (4)Pendulum (Sparse)| (1) DIP (Dense) (2) Cheetah (Dense) (3) Ant (Dense) (4) Cartpole  (Dense)|

Reviewer comment (Experiments):
There is a lack of baselines. Except for MAPS, all the baselines are from before 2020, while there are more recent works that could have been included.

Author response:
***A:*** To the best of our knowledge, there are only a few existing approaches that focus on online imitation learining from multiple black-box oracles. We included the state-of-the-art MAPS [2023], along with all baselines used to evaluate MAPS, such as MAMBA [2020]. Additionally, we implemented several variants of the proposed algorithm to provide a comprehensive evaluation. To provide some further context, please refer to `Appendix Table 2` for a detailed comparison to 17 other relevant methods, which explicitly points out the key differences in problem settings.

We believe we have included a sufficient set of relevant baselines for the problem setting that we consider, but would appreciate the reviewer's suggestion for any other relevant baselines. 


|         | RPI    | MAPS [ICML'23] | MAMBA [Neurips'20] |
|--------      |--------     |--------       |  --------   |
| Baseline  | (1)MAPS (2)MAMBA (3)Max-aggregation (AggreVaTed-variant for muliple oracles) (4)LOKI-variant (5)PPO-GAE (6)Best oracle| (1)MAMBA (2)PPO-GAE (3)Best oracle | (1)PG-GAE (2)Best oracle (AggreVatD(single oracle setting))|