Paper: Select to Perfect: Imitating desired behavior from large multi-agent data (ICLR-2024)

Reviewer comment (Evaluation):
For the results, it is necessary to provide some reference for a better understanding of the performance, e.g., reporting the results using the ground truth identification, or the results using the shapely value or other credit assignment methods.

Author response:
Thank you for this remark. We would first like to clarify that, for the human datasets, a ground-truth identification does not exist, as no ground-truth cluster assignments are given for the human participants in this dataset. For the simulated datasets, we can observe in Figure 4 that the EV estimation error is very low if 30% of groups are observed, hence the estimated EVs are very close to ground-truth EVs. The results for EV2BC in Table 3 assume that 30% of groups are observed and hence very closely match the performance using ground-truth identification of agents. We will add the exact results for ground-truth identification, but for now focused on the additional Starcraft experiments.

Unfortunately, it is impossible to compute a Shapley Value baseline in the given scenarios, as Shapley Values cannot be computed for fixed-size games. However, if Shapley Values could be computed (in the absence of group size constraints), then the ordering of agents by Exchange Values is *identical* to the ordering of agents by Shapley Values (see section 4.1 "Relationship between Shapley Value and Exchange Value" for details). Hence, in the absence of group size constraints, using Shapley Values to guide imitation learning will result in identical results as when using Exchange Values. We have also empirically verified this in the Tragedy of Commons environment. We are unaware of other credit-assignment methods that apply to our problem setting but will include a comparison if pointed to such works.

Reviewer comment (Evaluation):
How do you justify the choice of the DVF for each domain? How do you ensure that the DVF is aligned with the desired behavior and does not have any unintended consequences or biases?

Author response:
We would like to point out that in the discussion of AI Alignment, a distinction between normative and technical alignment is often made [A]. Our work addresses solely the technical aspect of alignment, and it does not try to answer the question of which norms AI systems should align with. Specifically, we do not try to answer the question of what the correct Desired Value function (DVF) for a given problem is. The DVFs that we chose for the different environments are simply inspired by previous literature. A possible option for setting DVFs is to crowdsource rankings of collective outcomes from human annotators [B]. 

Citations:  
[A] “Artificial Intelligence, Values and Alignment”, Gabriel, Minds and Machines, 2020  
[B] "Human-centred mechanism design with Democratic AI", Koster et al., Nature human behaviour, 2022

Reviewer comment (Experiments):
How scalable would this method be to let's say a dataset of multi-agent driving scenes? It seems to me like scalability is an issue here, specifically due to clustering. This brings me to an important point, the weaknesses of the proposed approach should be addressed.

Author response:
To underline the scalability of our method we now added experiments in the Starcraft multi-agent environment (SMAC, [A]), which we picked as it is probably the most widely adopted benchmark in the multi-agent community [C, D] and has various predefined scenarios and benchmarks (unlike multi-agent driving). It is partially observable, involves long-term planning, requires strong coordination, and is heterogeneous (as also described by Reviewer floZ). We find that also here, our method outperforms the baselines by a substantial margin, underlining the applicability of our method to diverse settings. Please consider the general response for details. 

We would like to point the reviewer to the conclusion, which already addressed three main limitations of our work. We have now added that we see the application of our method to even larger multi-agent environments, such as driving, as a relevant direction for future research.

Reviewer comment (Experiments):
I'm still left confused by the differences between behavioural clustering and EV-clustering. I understand the differences in the approaches, but the ablation study seems to point to behavioural clustering being more stable in low-data regiments. I see that the ablation study says to look at section 5.1 to show why behavioural clustering is not sufficient by itself, but I do not see the supporting results.

Author response:
Thank you for this remark; we now discuss this matter in more detail in Section 5.1 in the main paper. We first recall that behavior clustering only considers low-level behavior data (actions taken by individual agents), while EV-clustering only considers collective outcomes (DVF scores assigned to collective trajectories). In general, EV-clustering should be preferred to behavior clustering, for the following reason. Consider a case where two agents contribute equally to the DVF, in which case both agents should be assigned equal EVs, but both agents do show very different low-level behavior. When applying EV-Clustering, both agents would be assigned to the same cluster and would hence correctly be assigned the same EV. However, when applying behavior clustering, both agents would be assigned to different clusters, which would potentially result in erroneously assigning different EVs to each agent (and would likely result in higher-variance EV estimates overall). EV-clustering may have high variance if only very few collective outcomes are observed for some agents, in which case we can combine it with behavior clustering.

Reviewer comment (Experiments):
I'm a little confused about some experimental details. Specifically, the number of agents in the datasets, and the exact composition of the dataset is unclear. Can you clarify the composition of the $D^\text{adv}$? Are there really two different types of agents, but $n=100$ agents?

Author response:
Yes, the $D^{adv}$ dataset contains 100 agents but only two different types of agents (some behaving optimally, some behaving adversarially.) In contrast, the $D^{\lambda}$ dataset contains 100 different agents, where $\lambda$ is sampled from a uniform distribution for each agent. We chose to also have 100 agents in $D^{adv}$, to make it equal in size to $D^{\lambda}$.

Reviewer comment (Presentation):
In my opinion, the paper would be better organized by moving more results from the appendix into the main paper. One simple way of improvement is to move Figure 3 to the appendix and add the ablation study on the EV-clustering vs. Behavior clustering.

Author response:
Thank you for this suggestion. We moved Figure 3 to the Appendix. As we also have to accommodate the additional Starcraft experiments in the main paper, we were unable to move Figure 6 to the main paper, but we now discuss the findings from the ablation study, as well as the pros and cons of both clustering methods, in more detail in Section 5.1.

Reviewer comment (Experiments):
Only two environments are studied, while there are many environments that can further highlight the real-world value of the proposed method, e.g. public traffic.

Author response:
> Limited Environments: Only two environments are studied, while there are many environments that can further highlight the real-world value of the proposed method, e.g. public traffic.

To underline the scalability of our method we now added experiments in the Starcraft multi-agent environment (SMAC, [A]), which we picked as it is probably the most widely adopted benchmark in the multi-agent community [C, D] and has various predefined scenarios and benchmarks (unlike multi-agent driving).
SMAC involves long-term planning, requires strong coordination, and is heterogeneous (as also described by Reviewer floZ). We find that also here, our method outperforms the baselines by a substantial margin, underlining the applicability of our method to diverse settings. Please consider the general response for details.

Reviewer comment (Theory):
Shapley Values are known to have good properties, e.g. symmetry, dummy (zero value for null players), additivity etc, which make it interpretable, appealing, and useful. It would be interesting to see analysis of Exchange Values with respect to these properties.

Author response:
> Lack of more theoretical analysis of the properties of EV
Shapley Values are known to have good properties, e.g. symmetry, dummy (zero value for null players), additivity, etc, which make it interpretable, appealing, and useful. It would be interesting to see an analysis of Exchange Values with respect to these properties.

Thank you for this remark. The Shapley Value satisfies four main axioms: Dummy, Efficiency, Symmetry, and Linearity. We have now further clarified the relationship between Shapley Values and Exchange Values in the main paper, and added further clarifications about the different axioms. In Appendix A.1 we provide the following proofs:
- for the relationship between Shapley Values and Exchange Values
- that the Exchange Value satisfies the symmetry and linearity axioms
- that the Exchange Value satisfies appropriately modified versions of the dummy and efficiency axioms.

Reviewer comment (Evaluation):
More detailed analysis can be added to the main text with respect to how different EV values can be connected to various behavior patterns. In particular, the $\lambda$ values in both "Tragedy of Commons" and "Overcooked" can be linked to the estimated EVs to validate the method.

Author response:
> Lack of interpretation of EV: More detailed analysis can be added to the main text with respect to how different EV values can be connected to various behavior patterns. In particular, I think the $\lambda$
 values in both "Tragedy of Commons" and "Overcooked" can be linked to the estimated EVs to validate the method.

Thank you for this remark. Such plots are given in the appendix (due to lack of space), and we now improved their descriptions and referencing in the main paper. Figure 6 shows a plot of the relation of $\lambda$ and EV in the Overcooked environments, and Figure 8 shows the relation between EVs and different strategies in Tragedy of Commons. The strong correlation in Figure 6, as well as the visibly correct ordering of agents in Figure 8, provide insights into the relation between EVs and different strategies. This verifies that EVs are adequate estimators for individual contributions in both environments.

Reviewer comment (Experiments):
Although this is the first work tackling this specific problem setup, it would be interesting to see whether Shapley values could be similarly useful for guiding imitation learning in multi-agent datasets when the group size constraint is absent (which should be possible in many cases, e.g. the Tragedy of the Commons).

Author response:
> Lack of baseline: I understand that this is the first work tackling this specific problem setup, but I'm interested to see whether Shapley values could be similarly useful for guiding imitation learning in multi-agent datasets when the group size constraint is absent (which should be possible in many cases, e.g. the Tragedy of the Commons).

Thank you for this remark; we now further clarified this in the main paper. As described and shown in Section 4.1, if Shapley Values can be computed (in the absence of group size constraints), then the ordering of agents by Exchange Values is **identical** to the ordering of agents by Shapley Values. Hence, in the absence of group size constraints, using Shapley Values to guide imitation learning will result in identical results as using Exchange Values. We also empirically verified this in the Tragedy of Commons environment.

Reviewer comment (Writing):
Although the paper is generally enjoyable to read, many sentences throughout the article are a bit repetitive and convoluted.

Author response:
> Writing: Although I in general enjoyed reading the paper, I still find many sentences throughout the article a bit repetitive and convoluted.

We appreciate this feedback. We have made several improvements, also following other reviewers' suggestions, and welcome any further suggestions.

Reviewer comment (Writing):
What does the $m$ in line 220 denote? Is it a fixed value or can it take multiple possible values (since $m \in M$)? Why must $k \geq m$? Should $k$ change when $m$ takes a different value?

Author response:
> What does the m in line 220 denote? Is it a fixed value or can it take multiple possible values (since $m \in M$). Why must $k \geq m$? Should k change when m takes a different value?

We apologize for the misunderstanding. This is a typo that we have now fixed. It now reads $k \leq n$, where n is the number of agents and k is the number of clusters.

Reviewer comment (Theory):
**Assumption of Consistent Agent Behavior**: The framework assumes that individual agents behave similarly across multiple trajectories. This assumption may not always hold true in complex, dynamic environments where agent behavior can vary significantly based on context.

Author response:
Our method can be extended to situations where agents’ behavior differs across multiple trajectories by splitting up the data of an individual agent into different sub-agents and treating each sub-agent as its own agent. This extension of our work does not require additional methodological contributions, and we see it as an interesting direction for future work, as also outlined in our conclusion.

Reviewer comment (Experiments):
**Utilization of Undesired Behavior Data**: The paper points out that further research could explore how to utilize data on undesired behavior more effectively, such as developing policies that are explicitly constrained to avoid undesirable actions.

Author response:
Generally, data on undesired behavior could be used to disincentivize the policy from imitating undesired behavior or to gain further understanding of the environment dynamics. While some work exists in this field [B], this constitutes a highly interesting direction for future work.