Paper: Adversarial Imitation Learning via Boosting (ICLR-2024)

Reviewer comment (Experiments):
While the paper compares AILBoost with existing off-policy AIL algorithms such as DAC, ValueDICE, and IQ-Learn, it would be beneficial to include a comparison with more state-of-the-art IL methods as well. This would provide a more comprehensive evaluation and demonstrate how AILBoost performs relative to the best-performing IL algorithms in the field.

Author response:
__[W1] More SOTA IL algorithms__

We believe that we have covered a good range of SOTA off-policy, IL algorithms. Could the reviewer point us to any additional algorithms they would like to see?

Reviewer comment (Theory):
The paper lacks a detailed theoretical analysis of the AILBoost algorithm. While the authors provide some intuition and connections to boosting algorithms, a more rigorous theoretical analysis would strengthen the paper's claims. Specifically, providing formal proofs or guarantees of convergence, optimality, or sample complexity would enhance the theoretical foundation of AILBoost.

Author response:
__[W2] Lack of Theoretical analysis__

Thank you for this feedback. We agree with the reviewer that additional, in-depth analysis would only strengthen our work, but with this work we aimed to investigate and experimentally instantiate an algorithm that has the same principled off-policy treatment as ValueDICE and the practical scalability of DAC. Thus we focused on doing a thorough evaluation across the DeepMind Control Suite.

Reviewer comment (Experiments):
The paper does not include ablation studies to analyze the individual components or design choices of AILBoost. By systematically removing or modifying specific components of the algorithm and evaluating their impact on performance, the authors could gain a deeper understanding of the contributions of each component and provide insights into their importance.

Author response:
__[W4] Lack of Ablation__

Thank you for this critique. The practical instantiation of our algorithm is very close to DAC with the main difference being the boosted discriminator update. By comparing to DAC we felt that this was both ablating the most important contribution of AILBoost when compared to baselines. As for the ablation of the mixing parameter, we did not find this parameter to be so sensitive. Note that we used the same parameters across all experiments in a given task suite.

Reviewer comment (Theory):
The "weak learner" presented in this study doesn't conform to the traditional definition of weak learnability, which could be misleading. Contrary to AdaBoost, if the learner's base policy class fails to match the performance of the expert, then even mixing them at the initial state won't achieve expert-level performance.

Author response:
We would first like to clarify that AILBoost is a gradient boosting [1] algorithm rather than one closer to adaboost. Moreover, could the reviewer elaborate on the unrealizable case? As [2] showed, our RL procedure could be viewed as boosted policy search.

[1] Boosting Algorithms as Gradient Descent, Mason 1999

[2] Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search, Scherrer and Geist 2014

Reviewer comment (Experiments):
Why using weighted mixing instead of using the DAgger[1] style average mixing?

Author response:
We do not consider the mixing performed in Dagger because we do not assume access to an online interactive expert. Instead, in our setting, we only have access to expert demonstration data. We are performing weighted mixing of weak learners, not a weighted mixture between a weak learner(s) and an expert, because we do not have access to an expert policy. More specifically, we are learning a boosted policy in the state-action occupancy measure space as detailed in section 2 of [1].

[1] Provably Efficient Maximum Entropy Exploration, Hazan et al. 2019

Reviewer comment (Theory):
Is the smooth loss condition in Section 4.1 hard to satisfy? Is the specific condition that \(d^{\pi}\)'s state action distribution covers the expert's state action distribution? Could you point out the exact theorem that makes this claim?

Author response:
Please see [1], section 1.1 for their informal main theorem. For a more detailed discussion, please see Section 4 of [1].

[1] Provably Efficient Maximum Entropy Exploration, Hazan et al. 2019

Reviewer comment (Writing):
Can I get a definition for "off-policy"? I believe the proposed algorithm is still "on-policy" as it compares the current mixed policy state distribution to the expert's.

Author response:
The underlying reinforcement learning algorithm that our boosting algorithm builds on top of is soft actor-critic (SAC). SAC is an off-policy algorithm because it keeps around a replay buffer of past experiences to update the policy (which weren’t collected from the current/most recent policy), vs. on-policy algorithms that strictly update the policy from samples gathered from the current policy. In our algorithm, we keep around a replay buffer of past weak learners’ experiences that we weight to update our mixture policy, making our algorithm off-policy.

Reviewer comment (Experiments):
It would be interesting to see experiments where the learner is truly weaker than the expert, such that the expert is nonrealizable. Is weak learnability in imitation learning sufficient for near-expert performance?

Author response:
It would be interesting to see experiments where the learner is truly weaker than the expert. We will leave this investigation for future work.

Reviewer comment (Theory):
In Algorithm 2, why do we calculate \(\pi_{t+1}\) without the expert dataset? Is this practical? Any further justification?

Author response:
In algorithm 2, we update the weak learner $\pi_{t+1}$ with our off-policy RL algorithm using the reward function from the discriminator. The discriminator reward function captures the difference between the mixture policy and the expert data. The new weak learner is trained to minimize this difference, essentially making the mixture policy more similar to the expert dataset.

Reviewer comment (Theory):
Why in Algorithm 2, line 10, is the replay buffer given uniform weight for all samples? Any further justification?

Author response:
Our boosting algorithm modifies the discriminator updates to address the fact that the original DAC does not compute the density ratio – which is needed to minimize the original divergence that we care about. For updating our weak learner, we follow the original soft actor-critic (SAC) procedure which uses uniform weighting when sampling data from the replay buffer [1]. 

[1] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja et al. 2018

Reviewer comment (Writing):
More succinct Abstract and Introduction are needed. I think they are quite dense in its current form. One example I could think is adding a small figure to describe AILBoost's contribution, but this is not a mandatory comment to follow.

Author response:
We would like to thank the reviewers for the feedback on the presentation of our paper. We agree that focusing on the paper’s main contribution and removing additional details about existing work could increase the readability of the abstract and introduction. We will address these concerns in the final version of the paper.

Reviewer comment (Theory):
There are two different forms of reverse KLD; (1) Donsker-Varadhan dual form from ValueDICE (2) Variational form from f-divergence GAIL. In AILBoost, the second form was used. Do we have any reason for not using the first form?

Author response:
DAC optimizes the same divergence that AIRL [1] minimizes, which is the reverse KL (please see Section 4.1 of [2]). Taking recommendations from both [2] and [3], we chose to minimize the functional form for the reverse KL for our adversarial IL setting. Finally, we independently confirmed that we can reproduce DAC’s results with this functional form of the reverse KL.

[1] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, Fu et al. 2018

[2] A Divergence Minimization Perspective on Imitation Learning Methods, Ghasemipour et al. 2019

[3] Imitation Learning as f-Divergence Minimization, Ke et al. 2020

Reviewer comment (Experiments):
The idea of using weighted replay buffer and boosting seems applicable to general AIL frameworks, although authors applied this only to the variational form of reverse KLD. Can we apply this idea to DAC, and if possible, can we see the performance for those cases? One reason I'm asking is the training procedure that combines DAC with boosting may be simpler than the one with rev KLD.

Author response:
We agree with the reviewer that our idea of incorporating a weighted replay buffer and boosting can be extended to any divergence minimization imitation learning algorithm. Note that DAC uses the reverse KL divergence in their algorithm, so we performed our experiments using this divergence for a fair comparison and thus we are improving DAC through boosting.

Reviewer comment (Writing):
In page 7, line 1, "we always warm start from $\pi_t$". ---> What does this mean? Also, Appendix A doesn't appear in the manuscript.

Author response:
When learning a weak learner in our boosting algorithm, we could either randomly initialize our weak learner or initialize our weak learner from our previous policy. On page 7, line 1, we are stating that our weak learners are initialized to the previous policy that was learned.

Reviewer comment (Experiments):
AILBoost's computational complexity increases as $t$ increases in Algorithm 1. Can you please compare training times among DAC, ValueDICE and AILBoost?

Author response:
In terms of training time, the computational complexity remains the same – we still sample the same from the replay buffer for discriminator training (albeit weighted) and we only update the ensemble via SAC on the latest policy. The key difference is that we roll out with our ensemble, which doesn’t incur any computational overhead. We do make a point that there will be a slight increase in memory cost, which we found manageable at the end of Section 4.

Reviewer comment (Presentation):
Figure 2,3,4's file sizes seem to be too large, which I think end up the heavy file size (34MB) of pdf file and makes text loading slow. Can we make those figures' sizes smaller?

Author response:
Thank you for pointing out the issues with figure 2,3,4. We will make sure to address these issues for the final version.