Paper: Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations (ICLR-2024)

Reviewer comment (Reproducibility):
It would be great if the authors could provide code since a benchmark is only useful if the code is available.

Author response:
>It would be great if the authors could provide code […]

We have included the code base as a .zip file in the supplementary material. Additionally, we have provided a link to an anonymous GitHub repository, accessible here: https://github.com/d3iltest/Temporary_D3IL.

Reviewer comment (Theory):
The fact that different behaviors must be enumerated ahead of time is a large limitation of the proposed behavior-level entropy measure.

Author response:
We acknowledge the reviewer's concern about the limitation imposed by the prerequisite to enumerate different behaviors in advance for the proposed behavior-level entropy measure. However, quantifying diverse behavior is a challenging problem, for which, to the best of our knowledge, exists no approach that circumvents this requirement. Moreover, there exists no task suite that provides any metric for quantifying diverse behaviors as highlighted in Table 1 of the paper. In light of this, we believe that our work makes a valuable contribution to the field.

Reviewer comment (Evaluation):
The KL-divergence usage is a bit unmotivated, as the paper assumes the demonstration dataset is always uniformly distributed among all behaviors. In cases where the demonstration dataset is not uniformly distributed, it is unclear if KL divergence is the right measure to use. For example, a good learning method may be expected to have low KL, but if the demo dataset is skewed, a policy that is uniform across behaviors (even with higher KL than exactly matching the skewed distribution) may be preferable.

Author response:
We agree with the reviewer that directly introducing the entropy improves the paper’s clarity and readability. We thank the reviewer for their assessment and updated a revised version of our manuscript that does not introduce the KL divergence. These changes are colored in magenta and can be found on page 3.

Reviewer comment (Theory):
The entire discussion about KL-divergence in the paper seems pointless, and it would be more straightforward to just use the behavior entropy definition instead.

Author response:
We agree with the reviewer that directly introducing the entropy improves the paper’s clarity and readability. We thank the reviewer for their assessment and updated a revised version of our manuscript that does not introduce the KL divergence. These changes are colored in magenta and can be found on page 3.

Reviewer comment (Evaluation):
The paper focuses on measuring a certain quantity rather than arguing why it matters, and it would have been beneficial to include some argumentation on the importance or desirability of high behavior diversity.

Author response:
We thank the reviewer for their suggestion to enhance the motivation for policies capable of learning diverse behavior.  In response, we have incorporated additional points and references into the introduction of our paper. These changes are colored in magenta and can be found on page 1. 

Additionally, we summarized these points below:

1. **Improving Generalization:** If the learned policy overfits to a specific set of demonstrated behaviors, it may not generalize well to new situations. By exposing the model to diverse behaviors, the risk of overfitting is reduced, and the learned policy is more likely to capture the underlying principles of the task rather than memorizing specific trajectories [1, 2, 3].
2. **Enhancing Skill Transfer:** Learning diverse behaviors facilitates better skill transfer across different but related tasks. If the agent can imitate a wide range of behaviors, it is more likely to possess a set of skills that can be applied to various tasks, making it a more versatile and capable learner [1, 3].
3. **Unpredictability in competitive Games:** Predicting an opponent’s strategy in competitive games, such as table tennis, becomes much harder if the adversary has a diverse set of skills [4].

Reviewer comment (Experiments):
Could the authors comment on where conditional imitation learning falls into the picture? For example, in the pushing task, if the 4 behaviors are known ahead of time, one could condition the policy on a 1-hot vector indicating the desired behavior (“push X1 to Y1”, “push X2 to Y2”, etc.). This would allow a deterministic policy to achieve any of the behaviors assuming a perfect learning method. What is the argument for why it is not possible or not desirable to provide additional context to the policy to specify how the task should be performed?

Author response:
The primary argument against adopting goal-conditioned imitation learning (GCIL) [5, 6, 7] lies in the labor-intensive nature of context provision. Integrating contextual information often entails human annotation, a resource-intensive and challenging process to scale, particularly with extensive, unannotated datasets. We believe that the findings from our research contribute to mitigating this dependence on additional labeling efforts by highlighting methods that can learn diverse behaviors from datasets without human annotations.

Reviewer comment (Reproducibility):
A benchmark paper is useless without the environment codes and the data, which is absent from the supplementary materials. This is a major negative for this paper because we are being asked to judge it without being able to understand how easy it may be to run new algorithms against this benchmark.

Author response:
We have included the codebase as a .zip file in the supplementary material. Furthermore, we have supplied a link to an anonymous GitHub repository, which can be accessed here: https://github.com/d3iltest/Temporary_D3IL

Reviewer comment (Experiments):
Another primary criticism is that all five environments are very simple tabletop environments, and thus the complexity of the algorithm needed to solve that may not be quite high. A better benchmark would involve multiple kinds of environments, involving 2D/3D environments, potentially with different intractable elements.

Author response:
We appreciate the reviewer's consideration and understand the concern about the simplicity of the tabletop environments. As Reviewer DMDV shared a similar concern, we kindly guide the reviewer to our reply to Reviewer DMVD. In that response, we addressed the concerns and pointed to Appendix D of the revised version of the paper which elaborates on the new tasks we plan to introduce in the camera-ready version.

Reviewer comment (Evaluation):
One critical component missing from the evaluation is the required forward pass time or control frequency of the algorithms, which to my understanding is one of the largest disadvantages of diffusion-based models.

Author response:
We acknowledge the importance of providing forward pass/inference times to offer a more comprehensive understanding of the strengths and weaknesses of the different methods. Consequently, we have revised our manuscript and incorporated this information. You can find the results in Appendix B, highlighted in green.