Paper: Adversarial Imitation Learning from Visual Observations using Latent Information (ICLR-2024)

Reviewer comment (Presentation):
The proposed algorithm combines the adversarial imitation loss with gradient penalties from DAC [1] with the off-policy algorithm, stacking and data-augmentation strategy from DrQv2 [2]. While both these papers are cited in text, the way the methodology is introduced in Section 5 never makes these connections explicit. As a consequence, I feel the way the algorithm is presented can be quite misleading to an unfamiliar reader. Hence, I believe changing Section 5 to clarify which components come from DAC, which ones come from DrQv2 and that the novelty lies in *combining* them, would be extremely important before this work can be published.

Author response:
We have updated the beginning of Section 5 to cite all the literature inspiring our algorithm, including DAC and DrQ v2, in order to improve clarity and transparency. 

We want to emphasize that the combination of the different elements from DAC and DrQ v2 is well-motivated by the novel theoretical analysis in Section 4, which is instrumental for the formulation of our algorithm and of other possible variants.

Reviewer comment (Novelty):
I found the novelty of the theoretical analysis and methodology to be quite limited. While I believe this is not a mandatory aspect for a good paper, especially if the resulting algorithm is effective, I found the quality of the empirical evaluation insufficient to make such assessment (see point 3).

Author response:
Thank you for the opportunity to further discuss this point. We believe that important insights come both from our theoretical analysis and our experiments. 

To the best of our knowledge, theoretical analysis of the imitation from observations problem in a partially observable environment does not appear in previous work. The analysis in Section 4 sheds light on the assumptions needed between expert and agent in order to perform visual imitation from observations, and motivates the idea of latent variable inference from high dimensional observations. Note that this is in contrast with PatchAIL in [a] where imitation is directly performed on the pixel space. 

The Experiments section shows the practical implications of performing imitation on a lower dimensional space rather than on the pixel space. This results in significant improvements in computational efficiency, as emphasized by the wall-clock time ratio in Table 2. 

[a] Liu, Minghuan, et al. "Visual Imitation Learning with Patch Rewards." The Eleventh International Conference on Learning Representations. 2023.

Reviewer comment (Evaluation):
There are several aspects of the evaluation that left me unsatisfied with its quality. First, the comparison with PatchAIL-W and VMAIL is only carried out on six tasks from three environments from the DeepMind Control (DMC) suite, while the comparison with DrQv2 and Dreamer-v2 is only carried out in three tasks from a single environment. I would have appreciated seeing a wider variety (e.g., including other complex environments from DMC such as quadruped/jaco arm and from alternative benchmarks e.g., car racing, claw rotate as considered in VMAIL). Furthermore, the current ablation seems very much limited as it could consider studying the effect on performance of many additional design choices (e.g. spectral norm v gradient penalty for Lipshitzeness, number of stacked frames, type of data augmentation...). Additionally, I think that reporting results also for a simple behavior cloning baseline with the same data-augmentation/architecture/optimization would help understand the contribution from the introduced IRL methodology. Most worryingly, however, when comparing LAlfO with Dreamerv2 and DrQv2 the performance of the baselines is considerably lower than what is reported in prior work (e.g. see [2]). Even after 10x10^6 million steps, the gains from incorporating expert demonstrations seem marginal at best (if any) when using the results from DrQv2. I would really appreciate it if the authors could clarify this inconsistency (also given that DrQv2 shares the data that produced their reported learning curves).

Author response:
The goal of our Experiments section is to answer questions (1)-(4) as introduced at the beginning of Section 6 and not be exhaustive. We agree with the reviewer that our paper could benefit from additional experiments, but we do not think that the main message of the section will change if more environments or tasks are added. 

Moreover, in terms of quantity and difficulty of the tested environments, our paper is aligned with the convention established by the previous literature. Specifically, for the comparison with PatchAIL-W [a] and VMAIL [b], we test 3 easy tasks and 3 medium tasks from the DMC suite. In [a], 4 easy tasks and 2 medium tasks from the DMC suite are tested. In [b], 5 environments are tested.

For the comparison with DrQ v2 and Dreamer v2, we carry out experiments only in tasks which are considered hard to solve. In these types of tasks, leveraging expert videos makes more sense and the impact of our solution can be better appreciated as DrQ v2 and Dreamer v2 are quite inefficient.

[b] Rafailov, Rafael, et al. "Visual adversarial imitation learning using variational models." Advances in Neural Information Processing Systems 34 (2021): 3016-3028.

Reviewer comment (Evaluation):
Again, related to the evaluation Section, I find some of the claims to be quite misleading. For example, in connection to the humanoid results the authors state "we solve these tasks by using only 10^7 interactions." However, the reported performance on 2/3 tasks (walk and run) is still extremely low, and I would refrain from referring to any of these tasks as solved. Furthermore, I think to make the comparison fairer I would have also appreciated seeing results for DrQv2/Dreamerv2 adding the expert demonstrations to their respective replay buffer.

Author response:
Thank you for pointing this out. We have updated the wording to clarify this claim in the updated version of the paper.

The main goal of these experiments is to show how our algorithm can leverage videos of experts to improve the efficiency of standard RL from pixels (see question (4) at the beginning of the Experiments section). This is the reasoning behind the proposed comparison in Figure 4. 

We also want to emphasize that we only assume access to visual observations of the expert in our algorithm and not direct knowledge of expert states, actions, and rewards since such information cannot be easily obtained when using videos of the expert. This condition rules out several offline RL techniques which might be used in order to leverage expert data in DrQ v2 and Dreamer v2.

Reviewer comment (Novelty):
I believe the visual imitation problem setting described is a special simpler case of the visual third-person/observational imitation learning setting tackled by prior methods [3 as cited, 4, 5]. Yet, in contrast to what is stated in Related Work ("All of the aforementioned works consider fully observable environments"), this line of work also deals with visual observation. Hence, I believe there should be a clearer explicit connection.

Author response:
Thank you for sharing these papers with us. We have added them to our Related Work section and clarified the connections with our work in the updated version of the paper.

Reviewer comment (Experiments):
Where can I find detail regarding the expert data (is it taken from a standard benchmark? Was it collected with any particular protocol?) I cannot find this important information in the main text.

Author response:
The expert policies are obtained by training DDPG in a fully observable setting as described in the caption of Table 2. The full set of pre-trained policies can be downloaded from https://figshare.com/s/22de566de2229068fb75 . This information is available in the README of our code repository, and we have updated the caption of Table 2 to make this clear in the main text.

Reviewer comment (Experiments):
Can the authors provide learning curves for an increased number of steps for the Humanoid tasks, if available? (or at least also show the DrQv2 learning curves for the full 3x10^7 steps, shared in their repository to provide a reference for Figure 4)

Author response:
At the moment, we can provide curves for all the experiments as shown in the paper. We are now running RL+LAIfO for the full 3x10^7 steps and we will also provide these curves as soon as they are ready.

Reviewer comment (Evaluation):
The results presented in Table 2 do not seem to be statistically significant. I think it is misleading to highlight the best final performance in bold despite overlapping confidence intervals.

Author response:
We understand the concern raised by the reviewer. We chose to highlight the best final performance in bold based on conventions established in previous literature (refer to [1,2] for instance). 

[1] Yang, Chao, et al. "Imitation learning from observations by minimizing inverse dynamics disagreement." Advances in neural information processing systems 32 (2019).

[2] Liu, Minghuan, et al. "Visual Imitation Learning with Patch Rewards." The Eleventh International Conference on Learning Representations. 2023

Reviewer comment (Experiments):
The experiments in the imitation learning from demonstration setting are not fair as none of the baseline makes use of the expert demonstrations. It would be better to compare to methods that focus on this problem setting.

Author response:
The main goal of Figure 4 and of the paragraph entitled “Improving RL using expert videos” is to show how our algorithm can leverage videos of experts to improve the efficiency of standard RL from pixels (see question (4) at the beginning of the Experiments section). As a result, in our view it makes sense to propose the comparison as in Figure 4. We also want to emphasize that we only assume access to visual observations of the expert in our algorithm and not direct knowledge of expert states, actions, and rewards since such information cannot be easily obtained when using videos of the expert. 

In all the remaining paragraphs within the Experiments section, our algorithm and the baselines are compared in the same setting, i.e., either visual imitation learning or visual imitation from observations.

Reviewer comment (Novelty):
While I think that the method is novel, it also very straightforward and simple. While I do believe that simple methods are good, I could not get many new insights from the paper (the theorems are also relatively straightforward variations of previous theorems that bound suboptimality based on TV distance in IL and RL).

Author response:
Thank you for the opportunity to further discuss this point. We also agree that simple methods are good, and we believe that interesting insights come both from the Theoretical Analysis and the Experiments sections. 

The Theoretical Analysis section sheds light on the assumptions needed between expert and agent in order to perform visual imitation from observations, and motivates the idea of latent variable inference from high dimensional observations. Note that this is in contrast with PatchAIL where imitation is directly performed on the pixel space. 

The Experiments section shows the practical implications of performing imitation on a lower dimensional space rather than on the pixel space. It is also clear from the Experiments section that our straightforward approach to latent variable estimation leads to strong performance in practice, without requiring more complicated inference procedures.

Reviewer comment (Theory):
How can adding a reward objective to the imitation learning objective be justified? Can the detrimental effects of one objective on the other be bounded in some way?

Author response:
By assumption, we consider both the expert and the agent acting on the same POMDP. As a result, in this specific case, the reward objective represents the reward, possibly sparse, which is maximized by the expert. The imitation learning objective, instead, provides a dense reward to the learning agent. Adding the imitation learning objective to this sparse reward yields the improved efficiency illustrated in Figure 4.

We believe that considering the impact of suboptimal data and/or relaxing the aforementioned assumption represent interesting directions for future work.