Paper: Learning From Multi-Expert Demonstrations: A Multi-Objective Inverse Reinforcement Learning Approach (ICLR-2024)

Reviewer comment (Novelty):
The novelty of this paper is not clear with respect to a string of prior works that utilize a very similar approach (inferring a common reward function to learn from heterogeneous demonstrations). Could you clarify how your work differs from the work below?  
1. Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation (https://arxiv.org/abs/2001.00503)  
2. Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations (https://arxiv.org/abs/2209.11908)  
3. InfoGAIL

Author response:
* **Could you clarify how your work differs from the work below?**

1. Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation [1]
    
    The core idea of MSRD is connecting $n$ separate IRL through a regularization term, thereby recovering a robust task reward. The biggest problem is it completely ignores the computational cost because it still needs to run IRL $n$ times, which produces $n+1$ independent reward networks and $n$ independent policy networks. Not to mention, at the end of each iteration, it still need to run TRPO to determine $\pi_i$.
    
2. Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations [2]

    Although FLAIR slightly reduces the computation loss through policy mixture, it still needs to run IRL multiple times when a new expert is unique enough. In contrast, Our work adopts a single model architecture, as stated in the abstract. The core idea of our work is sharing knowledge within a single model. The reason behind this is that these heterogeneous experts are engaging in the same task, with their only differences lying at the preference level.

3. InfoGAIL [3]

    Although InfoGAIL and our work share a similar motivation in addressing the heterogeneity of experts, InfoGAIL introduces a latent variable that can only be used to disentangle trajectories. In contrast, our work can recover a common reward that has transferability to different preferences. Besides, InfoGAIL is constrained by the limitations of IL, such as the inability to adapt to environmental changes and excessive reliance on the quantity and quality of experts.

Reviewer comment (Experiments):
Further, as learning from heterogeneous demonstrators has been studied for several years, it would be beneficial to compare against a framework that attempts to accomplish this goal. It would be expected that a baseline like GAIL will underperform the proposed approach.

Author response:
* **Further, as learning from heterogeneous demonstrators has been studied for several years, it would be beneficial to compare against a framework that attempts to accomplish this goal.**

In the updated version, we have included IQ-Learn [4], the base model of our work, as another baseline to further demonstrate the improvements. Please refer to appendix A.1 "Does MOIQ really improves?"

Reviewer comment (Experiments):
It isn't clear how the preferences were designed in Section 5 and how the preferences result in different behaviors qualitatively.

Author response:
* **It isn't clear how the preferences were designed in Section 5 and how the preferences result in different behaviors qualitatively.**

As we focus on the setting of multi-expert (heterogeneous) in this work, it would have very limited use picking similar preferences. The preferences designed in 5.1 aim to accentuate heterogeneities among experts as much as possible while taking different dimensions of the reward function into account.

Reviewer comment (Theory):
Is it valid in practice if we assume we know the number of multiple experts which produced the whole demonstration?

Author response:
This assumption is relatively fair as long as we know the indices of experts. It's also used in ILEED [1]. If the assumption is removed, meaning we don't know both the number and preferences of experts (as preferences can be treated as an identifier), then two possible options may be considered:
1. Treat every trajectories heterogeneous initially.
2. Apply a preliminary clustering algorithm to trajectories.

Reviewer comment (Theory):
Is it valid in practice if we assume each multiple experts produced equal contribution to produce the whole given demonstration (since the formulation)?

Author response:
In our work, we assume that the experts are equally optimal. If that's not the case, it would require more meta-information from experts, such as the relative ranking or importance weight among experts, which may not be easy to acquire.

Reviewer comment (Experiments):
In related work, two methods of MOIRL are introduced. Should these be encompassed in the baseline algorithm?

Author response:
For MODIRL [3], we didn't found the realeased official code. For ILEED, they did release the official code for discrete action space. We have reached out to them to ask for the code applicable for continuous action space, and the reply we received was *there is no working implementation that we can easily pull*. However, they did provide us with detailed instructions to implement it ourselves, and there are still some obstacles to overcome. As an alternative option, we have included IQ-Learn [5], the base model of our work, as another baseline to further demonstrate the improvements. Please refer to appendix A.1 "Does MOIQ really improves?"

Reviewer comment (Experiments):
Can the proposed method applicable to other algorithm than SAC framework?

Author response:
Indeed, the concept of the *common reward* can be done in various approaches. The key is how you enforce the constraint of common reward. In section 4.1, we demonstrate that the common reward can be found by using consensus ADMM. Subsequently, we run PPO [4] to find the corresponding policy.

Reviewer comment (Theory):
Can we use MOIQ for discrete case?

Author response:
Sure. It works in discrete case with some appropriate modifications on SAC [2]. E.g., The output of actor in SAC is originally a Gaussian distribution. It should be modified to output a probability of each action.

Reviewer comment (Evaluation):
In Table 1, how MOIQ sometimes outperform expert?

Author response:
It's one of the benefits when considering multiple experts. Having a highly correlated estimated reward function allows MOIQ to find the best model that connects experts. It has the potential to outperform the original expert. This is something that cannot be accomplished with single-expert.

Reviewer comment (Novelty):
The related works are quite limited. A more thorough review should be presented.

Author response:
After carefully reading through the valuable feedbacks from reviewers, we have provided a more comprehensive review in our related works.

Reviewer comment (Experiments):
The experimental results, while valuable, would benefit from a more comprehensive analysis. Currently, the authors only compare their proposed algorithms to Generative Adversarial Imitation Learning (GAIL), and it's worth noting that in the Mo-walker and Mo-Ant environments, the Average Return of MOIQ performs worse than GAIL. Additional analyses are necessary to provide a more complete evaluation of these experiments.

Author response:
In the updated version, we have included IQ-Learn [1], the base model of our work, as another baseline to further demonstrate the improvements. Please refer to appendix A.1 "Does MOIQ really improves?"

Reviewer comment (Writing):
The author's use of the variable 'w' is somewhat ambiguous as it is defined multiple times in the paper. It is unclear whether 'w' represents the preference of the expert or a d-dimensional probability vector. Clarification is needed on this aspect.

Author response:
Thanks for pointing out this misleading definition, we have corrected it in our updated version. The confusion may come from the word "probability". We refer to $d$-dimensional probability vector as a vector with $d$ non-negative entries that adds up to $1$. Furthermore, we use it to denote the preferences of experts.

Reviewer comment (Evaluation):
It would be valuable if the authors could provide a more in-depth explanation regarding the instability observed in the transfer experiments, particularly in the Mo-HalfCheetah and Mo-Walker environments.

Author response:
As described in section 5.1, the experts in Mujoco are trained from scratch for 0.5M steps with their own scalarized rewards, computed by eq (1). They are expected to acquire corresponding amounts in each dimension of return according to their preferences, like the orange points in DST and Mo-Ant in Fig 3(b). However, the trained experts didn't go the way we expected, resulting in a limited differences among experts. Therefore, it influenced the transferability in our model.

Reviewer comment (Writing):
Could the author please illustrate the distinction between the common reward function and the ground-truth reward function?

Author response:
In this work, you can view them as same things. More specifically, we try to emphasize our differences with traditional IRL. In traditional IRL, we can still recover one reward function for an expert. If there are $n$ experts, then we can recover $n$ reward functions by repeating IRL $n$ times for every expert's demonstration. However, these $n$ reward functions can be greatly different. That's why we emphasize common reward function in our MOIRL work.

Reviewer comment (Reproducibility):
Annotating the preference vector $\omega$ can be cumbersome, especially when dealing with a large expert demonstration dataset collected from multiple human expert sources. This manual annotation process may become a bottleneck, particularly when preferences vary significantly among the human experts.

Author response:
We agree that it could be a bottleneck of our work. Currently, we're working on enabling our model to learn preferences. One of the approaches we have tried is finding the one that maximizes the expected value of the estimated cumulated return during training, expressed as $\arg\max_\omega \mathbb{E}_{\tau_E}[\sum _{t=0}^N\gamma^t\omega^T \cdot \hat{R}(s_t,a_t)]$. However, this approach didn't yield the desired results, as it would allocate the entire budget to the dimension with the higher estimated cumulated return, which seems to be unreasonable. We'd love to hear your opinions and suggestions to help us enhance our work.

Reviewer comment (Theory):
The proposed method primarily addresses the challenge of demonstration diversity stemming from different preferences among experts, but it does not explicitly tackle the diversity arising from multi-modality or stochastic behavior within the same preference category. For instance, some experts might have the same preference but choose different paths, such as passing by a tree on the left or the right.

Author response:
The conducted experiments indeed involve experts with multi-modality. As stated in section 5.1, we adopt the optimal stochastic policy as experts in discrete DST environment. The results are shown in Figure 1.

Reviewer comment (Experiments):
The choice of the baseline method appears to be relatively weak. While multi-expert inverse reinforcement learning algorithms are limited, the authors could have selected more advanced IRL methods as baselines and trained a model for each preference individually. This would have provided a better understanding of the proposed method's capabilities by allowing for a more robust comparison.

Author response:
In the updated version, we have included IQ-Learn [1], the base model of our work, as another baseline to further demonstrate the improvements. Please refer to appendix A.1 "Does MOIQ really improves?"