Paper: Quantifying Zero-shot Coordination Capability with Behavior Preferring Partners (ICLR-2024)

Reviewer comment (Theory):
Though the authors claim that a "diversity-complete" set may be intractable for complex environments, the proposed methodology (generation and selection) fails to well adhere to the exact definition in the desiderata (Section 3.2) with the lack of revealing the gap with a real "diversity-complete" set.

Author response:
**Meeting the Desiderata**. We acknowledge that our evaluation workflow is approximating the 'diversity-complete' by meeting the desiderata as much as possible. 
- Reasonable: We restrict the reward space to indicate reasonable behaviors by adding the original task reward and filtering sabotaging agents after generating candidates.
- Skill-style diverse: We traverse the reward space for diverse fully trained behavior-preferring agents and select the representative set by BR-Div.
- Skill-level diverse: We collect the earlier checkpoints of the selected fully trained evaluation partners to increase the skill-level diversity.

We highlight how our evaluation workflow meets the desiderata in Sec. 4.2 of the update paper.

**Gap with "diversity-complete"**. First, we need to claim that we cannot get a real "diversity-complete" set of evaluation partners for most tasks. We reveal the gap of our generated partners with a real "diversity-complete" set as follows:
- Events are designed based on human knowledge and may not cover all possible reasonable behaviors.
- We can only obtain approximate BRs.
- Style levels of different checkpoints may not cover all skill levels.

Though our evaluation partners still have a gap with a real "diversity-complete", we are the first to propose the concept of ideal evaluation partners, i.e., "diversity-complete" and our evaluation workflow makes a significant improvement over previous evaluation methods in providing a fair and comprehensive evaluation.

Reviewer comment (Evaluation):
The proposed metric and evaluation workflow lack necessary discussions with previous approaches (e.g., methods in Table 1).

Author response:
We first clarify that we list the previous evaluation methods in the first column and the works that utilize them in the second column in Table 1. For the previous evaluation methods, we have already discussed their defects in Sec. 3.2 and Appendix C.

Defects of previous evaluation partners:
- Human Players: The human players in ZSC problem are 'perfect' candidates for evaluation because human are strictly qualified as unseen partners. While evaluating with human players are expensive, time-consuming and unrepeatable. We need a more efficient evaluation method as a supplement for human evaluation. 
- Human Proxy Agents: Human proxy agents in overcooked environment do not account for human behaviors [5], which shows that using human proxy agents does not represent the diversity of human.
- Trained Self-play Agents: Train self-play agents are similar to the agents used to train the ego agent and are not diverse, as shown in Figure 2.
- Rule-based Specialist: Manually building expert rules is difficult to implement in complex environments and may not meet diversity requirements. We point out that designing a set of events is much easier than designing a policy.
- Random Agents: The diversity of the random initializations cannot be ensured and random initializations lack of high level performance.

Defects of previous metric:
- Cross-play with Trained Adapted Agents: Some works compare their ZSC methods by cross-playing the agents trained by other ZSC methods and their ego agents. This results include self-play performance and does not completely reflect the capabilities of ZSC. On the other hand, excluding the self-play results leads to potential unfairness.

The major difference is that **previous evaluation partners and metrics are unable to provide a comprehensive and fair evaluation due to the evaluation partners are not diverse or the metric is unfair. Instead, our evaluation workflow provide a comprehensive and fair measurement of ZSC capability by assembling reasonable evaluation partners with diverse skill-styles and skill-levels and by proposing BR-Prox to assess the generalization capability and improvement potential.**

We have highlighted the differences in Sec. 2 of the update paper. We also remark that we are the first to systematically investigate the construction of evaluation partners and the measurement of ZSC capability. Our aim is to ensure a more fair and comprehensive assessment of ZSC algorithms, addressing the current gaps in evaluation methodologies.

Reviewer comment (Writing):
The presentation of this paper is generally obscure. The authors involve a sort of techniques during the evaluation workflow but do not well explain the necessity and details (e.g., how to represent the behavior feature of a policy and reasons to involve event-based rewards).

Author response:
**Event-based Reward**. To clarify, we explain our motivation for introducing event-based rewards in Sec. 4.2. We expect partners to have diverse and reasonable behaviors, while the behavior space is intractably large. Inspired by [2] in which human preferences can be regarded as event-centric and modeled as event-based reward functions, we apply the event-based reward shaping method to encourage behavior discovery and generate behavior-preferring partners. The event-based reward space can be formulated as follows:
$$
\mathcal{R}=\\{r _{\bf{w}}| r _{\bf{w}}(s _{t}, \bf{a} _{t}) = r + \phi(s _{t}, \bf{a} _{t})^{T}\bf{w}, \bf{w} \in \mathbb{R}^{m}, \|\bf{w}\| _{\infty} \leq B _{\text{max}}, \sum _{i} \mathbb{1}(\bf w _{i} \neq 0) \leq C _{\text{max}}\\}
$$
The hyper-paramters $B _{\text{max}}$ and $C _{\text{max}}$ are used to restrict the reward space to indicate reasonable behaviors. $\phi(s _t,\bf{a} _{t})$ is the indicator of pre-defined events.

**Policy Behavior Feature**. The policy behavior features that are used to figure out BR-Div depend on the environment and the task at hand, which are related to the skills being tested. For simplicity, we count the occurrence of the pre-defined events, i.e., $\mathbb{E}[\sum _{t=1}^{T}\phi(s _t,\bf{a} _t) ]$, as the policy behavior feature.


**$\bf{w}$ Design**. Given the above formulation of reward space $\mathcal{R}$, we have restricted the space of $\bf{w}$ to indicate only reasonable combinations of events. $\mathcal{R}$ includes preferences that do not cause the game to be disrupted or unable to cooperate with. **Then we traverse the restricted reward space, i.e., the space of $\bf{w}$**. An event-based reward function $r _{w}$ indicates a behavior preference since $r _{w}$ encourages the preferred events (behaviors). The agent equipped with this event-based reward function is trained with an agent equipped with the reward according to the given task. The two-agent team approximates an NE in a two-player Markov game, which models the scenario in which a task-reward-maximizing agent collaborates with another agent or a human with its own preferences to complete the given task. Therefore, we get an agent preferring the behaviors presented by $r _{w}$ after approximating the NE.



*An example of Overcooked*

In Overcooked, we use $B _{\text{max}} = 20$, $C _{\text{max}}=3$ and generate up to 194 candidates and select up to 30 evaluation partners. The generated candidates are excluded if they cannot complete a delivery when cooperating with their BRs. The pre-defined events are listed as follows:
| Events | Weights |
| --- | --- |
| Put an onion or a dish or a soup onto the counter  | 0  |
| Pickup an onion or a dish or a soup from the counter | 0 |
| Pickup an onion from the onion dispenser | -20,0,10  |
| Pickup a dish from the dish dispenser  | -20,0,10 |
| Pickup a soup | -20,0,5,10 |
| Place an ingredient into the pot | -20,0,3,10 |
| Deliver a soup | -20,0 |
| Stay | -0.1,0,0.1 |
| Movement | 0 |
| Order Reward | 0.1,1 |

We also provide the pseudocode for using these events to show that one can make a moderate effort to implement the event-based rewards.
```python=
state = current state of the game
action = action of the agent whose events are collected
reward = 0
for e_i in range(NUM_EVENTS):
    if True == EVENT_JUDGER(e_i, state, action):
        reward += EVENT_WEIGHTS[e_i]
```
The details are also present in Appendix B.1.

Reviewer comment (Evaluation):
The evaluated baselines seem not distinguishable in the evaluation setting while the authors do not provide further insights of what kinds of approaches are generally useful on ZSC.

Author response:
It is important to highlight that this 'undistinguishability' is, in fact, one of our significant findings. The observation that various baselines appear to be similar in performance within our evaluation framework is not a limitation but rather a crucial insight. This finding suggests that current evaluation layouts may not be sufficiently challenging or diverse to distinguish different ZSC methods. We argue that this discovery underscores the need for more complex and sophisticated evaluation environments and mechanisms in ZSC research.

We categorized the current ZSC methods into two classes:
- co-play: a two-stage framework that pretrains a population via self-play and then trains the ego agent with the pre-trained population
- evolution: the ego agent is trained with an evolving population of partners.

We point out that the most popular methods are mostly co-play methods, and under our evaluation workflow, co-play methods, especially MEP and HSP, are generally useful in ZSC. The conclusion is based on the results aggregated from all layouts and all population sizes, as shown in right of Figure 5 in the main text and Table 4 in the appendix.

As a future work, we plan to introduce complex mechanisms into the Overcooked environment and apply our evaluation workflow to more environments to promote the development of ZSC methods.

Reviewer comment (Experiments):
The evaluation workflow is restricted to a two-agent form while previous ZSC methods can generalize to multi-agent settings.

Author response:
To the best of our knowledge, current zero-shot coordination research (including FCP [1], TrajeDi [8], MEP [4], COLE [5], and HSP [2]) primarily involves theoretical frameworks and practical algorithms designed for two-player cooperative games. The most commonly used environments, such as Overcooked [6] and Hanabi [7], are also predominantly used in two-agent settings. While it is true that some ZSC research (including Other-Play [9], Any-Play [10]) can theoretically extend to n-player scenarios, the limitation of computational resources makes them evaluate the algorithms using 2-player settings in the experiments.

Also, as discussed in previous ZSC research, the settings of more than two agents can be easily reduced to two-agent settings. It is not the key challenge of the ZSC problem. In the context of n agents collaborating, we can consider these n-1 partners to cooperate as a joint policy and the left agent as the other policy. Thus, we can train the behavior of preferring joint policies and their best responses in the same way, maintaining the effectiveness of our evaluation workflow.

We strongly believe in the value of extending the evaluation environment to n-players, and we believe this will be the focus of our future work. We will consider implementing multi-agent setting environments and evaluations to contribute to promoting ZSC research in n-player settings. We are implementing our evaluation workflow in the 5-vs-5 Google Research Football [11], and we will update the additional results if time permits.

Reviewer comment (Evaluation):
How can we extend the evaluation manner to multi-agent settings?

Author response:
To the best of our knowledge, current zero-shot coordination research (including FCP [1], TrajeDi [8], MEP [4], COLE [5], and HSP [2]) primarily involves theoretical frameworks and practical algorithms designed for two-player cooperative games. The most commonly used environments, such as Overcooked [6] and Hanabi [7], are also predominantly used in two-agent settings. While it is true that some ZSC research (including Other-Play [9], Any-Play [10]) can theoretically extend to n-player scenarios, the limitation of computational resources makes them evaluate the algorithms using 2-player settings in the experiments.

Also, as discussed in previous ZSC research, the settings of more than two agents can be easily reduced to two-agent settings. It is not the key challenge of the ZSC problem. In the context of n agents collaborating, we can consider these n-1 partners to cooperate as a joint policy and the left agent as the other policy. Thus, we can train the behavior of preferring joint policies and their best responses in the same way, maintaining the effectiveness of our evaluation workflow.

We strongly believe in the value of extending the evaluation environment to n-players, and we believe this will be the focus of our future work. We will consider implementing multi-agent setting environments and evaluations to contribute to promoting ZSC research in n-player settings. We are implementing our evaluation workflow in the 5-vs-5 Google Research Football [11], and we will update the additional results if time permits.

Reviewer comment (Presentation):
In Figure 2, how are the high-level behaviors visualized? What is the meaning of different data points in Figure 2?

Author response:
To get Figure 2, we collect the occurrence of the designed events by counting the triggered events alongside the episodes and representing each trajectory as a vector containing the normalized times of event occurrences. Then we reduce the vectors into 2-dimension through principal components analysis (PCA [3]). Finally we plot the 2-dimension vectors in Figure 2. **Briefly, each data point means the normlized event occurrences of a trajectory.**

Reviewer comment (Evaluation):
In Figure3, why does the population diversity first rise and then drop with the population size increasing? As near 0 values mean linear correlation, why is the diversity low in a small population size?

Author response:
Recall that we define the population diversity as $\operatorname{PD}(\\{\pi _{i}\\} _{i=1}^{M}) = \operatorname{det}(\bf{K})$, where $\bf{K} _{ij} = \theta _{i}\cdot\theta _{j}$ and each element in $\bf{K}$ is normalized into \[0,1\]. Thus, population diversity is a determinant. The determinant of a matrix represents the "volume" of the matrix but is also related to the dimension of the matrix.

Therefore, the population diversity metric is related to both the diversity and the size of the population. Strictly speaking, we should only compare the points on the two curves with the same population size. The population diversity of a small population is low because it is the determinant of a small and normalized matrix. 

Figure 3 illustrates that sampling based on BR-Div reaches higher population diversity than that based on P-Div at the same population size before the population diversity diminishes, which means BR-Div is more effective in constructing a diverse population from a set of policies.

Reviewer comment (Writing):
Does the P-Div metric mean $\text{PD}$ on $\pi_i$ instead of $BR(\pi_i)$?

Author response:
Yes, and P-Div is first mentioned in Sec. 4.1, "according to the partners population diversity (P-Div)". We further add the definition that $\operatorname{P-Div}(\\{\pi _{i}\\} _{i=1}^{M}) = \operatorname{PD}(\\{\pi _{i}\\} _{i=1}^{M})$ to make it clear.

Reviewer comment (Evaluation):
In Section 5.2, how do Figure 4 and 5 show "increasing population size contributes to the improvement of performance under the condition that the diversity of the population is also grown".

Author response:
Thank you for your suggestion to improve our representation. We have updated the explanation in the revised paper. 

Specifically, in Figures 4 and 5, some algorithms get performance improvement with increasing population sizes in some layouts, which is exactly why we conclude that "increasing population size contributes to the improvement of performance under the condition that the diversity of the population is also grown". In the following, we explain what the condition is from two aspects.
- Some ZSC methods lack an explicit mechanism to promote population diversity, including FCP and COLE. Thus, the performance of FCP and COLE does not benefit from increasing the population size from 24 to 36.
- On the other hand, ZSC methods with explicit mechanisms to promote population diversity may sometimes fail to expand population diversity in some layouts. We provide a case study in the Counter Circ. and Forced Coord. layouts to show that the reason is existing ZSC methods may fail to effectively produce enough diverse, high-performing agents in some layouts.

Reviewer comment (Theory):
How do the event-based rewards contribute to behavior preferring policies? Specifically, how do the method design and adjust $w$? How is this approach related to proposed "skill-level diversity"?

Author response:
**Event-based Reward**. To clarify, we explain our motivation for introducing event-based rewards in Sec. 4.2. We expect partners to have diverse and reasonable behaviors, while the behavior space is intractably large. Inspired by [2] in which human preferences can be regarded as event-centric and modeled as event-based reward functions, we apply the event-based reward shaping method to encourage behavior discovery and generate behavior-preferring partners. The event-based reward space can be formulated as follows:
$$
\mathcal{R}=\\{r _{\bf{w}}| r _{\bf{w}}(s _{t}, \bf{a} _{t}) = r + \phi(s _{t}, \bf{a} _{t})^{T}\bf{w}, \bf{w} \in \mathbb{R}^{m}, \|\bf{w}\| _{\infty} \leq B _{\text{max}}, \sum _{i} \mathbb{1}(\bf w _{i} \neq 0) \leq C _{\text{max}}\\}
$$
The hyper-paramters $B _{\text{max}}$ and $C _{\text{max}}$ are used to restrict the reward space to indicate reasonable behaviors. $\phi(s _t,\bf{a} _{t})$ is the indicator of pre-defined events.

**Policy Behavior Feature**. The policy behavior features that are used to figure out BR-Div depend on the environment and the task at hand, which are related to the skills being tested. For simplicity, we count the occurrence of the pre-defined events, i.e., $\mathbb{E}[\sum _{t=1}^{T}\phi(s _t,\bf{a} _t) ]$, as the policy behavior feature.


**$\bf{w}$ Design**. Given the above formulation of reward space $\mathcal{R}$, we have restricted the space of $\bf{w}$ to indicate only reasonable combinations of events. $\mathcal{R}$ includes preferences that do not cause the game to be disrupted or unable to cooperate with. **Then we traverse the restricted reward space, i.e., the space of $\bf{w}$**. An event-based reward function $r _{w}$ indicates a behavior preference since $r _{w}$ encourages the preferred events (behaviors). The agent equipped with this event-based reward function is trained with an agent equipped with the reward according to the given task. The two-agent team approximates an NE in a two-player Markov game, which models the scenario in which a task-reward-maximizing agent collaborates with another agent or a human with its own preferences to complete the given task. Therefore, we get an agent preferring the behaviors presented by $r _{w}$ after approximating the NE.



*An example of Overcooked*

In Overcooked, we use $B _{\text{max}} = 20$, $C _{\text{max}}=3$ and generate up to 194 candidates and select up to 30 evaluation partners. The generated candidates are excluded if they cannot complete a delivery when cooperating with their BRs. The pre-defined events are listed as follows:
| Events | Weights |
| --- | --- |
| Put an onion or a dish or a soup onto the counter  | 0  |
| Pickup an onion or a dish or a soup from the counter | 0 |
| Pickup an onion from the onion dispenser | -20,0,10  |
| Pickup a dish from the dish dispenser  | -20,0,10 |
| Pickup a soup | -20,0,5,10 |
| Place an ingredient into the pot | -20,0,3,10 |
| Deliver a soup | -20,0 |
| Stay | -0.1,0,0.1 |
| Movement | 0 |
| Order Reward | 0.1,1 |

We also provide the pseudocode for using these events to show that one can make a moderate effort to implement the event-based rewards.
```python=
state = current state of the game
action = action of the agent whose events are collected
reward = 0
for e_i in range(NUM_EVENTS):
    if True == EVENT_JUDGER(e_i, state, action):
        reward += EVENT_WEIGHTS[e_i]
```
The details are also present in Appendix B.1.

Reviewer comment (Writing):
How can we represent the behavior feature $\theta$ of a policy?

Author response:
**Policy Behavior Feature**. The policy behavior features that are used to figure out BR-Div depend on the environment and the task at hand, which are related to the skills being tested. For simplicity, we count the occurrence of the pre-defined events, i.e., $\mathbb{E}[\sum _{t=1}^{T}\phi(s _t,\bf{a} _t) ]$, as the policy behavior feature.

Reviewer comment (Experiments):
A significant limitation is the absence of a crucial baseline, specifically HSP. Since the paper's approach to training evaluation candidates and best response policies closely mirrors that of HSP, it should not pose a substantial challenge to also train an ego agent for HSP. I would consider increasing my score if this limitation is addressed.

Author response:
Thank you for suggesting that HSP be included in the Overcooked evaluation. The HSP algorithm has been implemented and included in the comparison. We have updated the HSP results in figures 4, 5, 10, and 11. We present the performance rank under our evaluation workflow as follows for your convenience. Table 4 of Appendix B.3 contains the rankings as well.

| Method \ Rank | 1 | 2 | 3 | 4 | 5 | 6 |
| --- | --- | --- | --- | --- | --- | --- |
| SP     | 0.0 | 5.56 | 11.11 | 5.56 | 5.56 | 72.22 |
| FCP  |  5.56 | 16.67 | 44.44 | 5.56 | 22.22 | 5.56  |
| MEP  |  38.89 | 22.22 | 22.22 | 5.56 | 11.11 | 0.0  |
| TrajDi | 11.11 | 16.67 | 16.67 | 33.33 | 5.56 | 16.67 | 
| COLE   | 0.0 | 11.11 | 0.0 | 27.78 | 55.56 | 5.56   |
| HSP | 44.44 | 27.78 | 5.56 | 22.22 | 0.0 | 0.0|

The table shows the percentage of ranks that ZSC methods achieve; for example, HSP ranks first at 44.44 percent of all layouts and population sizes. The results show that HSP outperforms the other algorithms evaluated.

Reviewer comment (Evaluation):
In figure 3, how would the population diversity of selection with population diversity lower than that of selection with BR-Div? This seems to be counterintuitive as it is expected to find the subset with largest population diversity if P-Div is used as the selection metric.

Author response:
BR-Div is a novel diversity metric for population diversity in the ZSC domain, as we demonstrate. We argue that the diversity of a set of cooperative agents' best responses should be used to measure population diversity.

The insight is that the ego agent strives to emulate the BRs to any partner in the training population. As a result, **an ego agent with higher ZSC capability emulates more BRs**. To evaluate the ZSC capability, we should measure the ego agent's ability to emulate a large number of BRs, i.e., expose the ego agent to partners with a large number of BRs.

We verify the effectiveness of BR-Div in Figure 2, Figure 3, Figure 8 and Figure 9, both qualitatively and quantitatively. **A simple and intuitive explanation of why BR-Div is more effective than P-Div is that two different partners may respond to similar BRs or even the same BR.** This phenomenon has also been observed in recent works [1,2].

Reviewer comment (Experiments):
Please explain the difference among different layouts in more details. Are there other kinds of events except conflicts should be considered to influence the performance of different methods in the introduced layouts?

Author response:
When we talk about the baseline's performance, we need to note that we first found the issue of not enough discrimination in the layouts of the original overcooked environment. Our evaluation workflow reflects the defects of previous evaluation environments' layouts. Appendix A contains a detailed description of the layouts. We will introduce them briefly below.
- Forced Coordination: Because the two players are in separate, non-overlapping sections of the kitchen, they must cooperate.
- Counter Circuit: Players must coordinate their movements and make use of the middle counter.
- Coordination Ring: Players must coordinate their movements in this ring.
- Bothway Coordination: allows both left and right agents access to onions and pots, providing them with more policy space and cooperation styles.
- Blocked Corridor: A U-shaped corridor forces the agents to work together to reduce the time it takes to prepare the soup. This layout includes a variety of coordination styles, such as using spare counters and coordinating agent movement.
- Asymmetric Advantages and Asymmetric Coordination: Two agents have individual advantages in completing some tasks, and they must coordinate their advantages in order to maximize their performance.

We know that other things, like the layout size, the number of possible coordination conventions, and the skill levels of the agents, may have an effect on how well baselines do (see Sec. 5.1 for differences in performance with evaluation partners of different skill levels). However, we discovered that **the resource and physical collisions between players are the most influential factors in evaluating ZSC capability, which are also two of the main reasons for the other factors mentioned above**. When conflicts exist in layouts, the need for coordination capabilities is much greater than when they do not. This is due to the fact that the strategy is more influenced by other players at this time, such as the preferences of other players' strategies and the coordination capabilities of other players who are in conflict.

We hope that the added results can address the limitations of our paper and that our explanation can address your concerns. We are looking forward to your reply.

Reviewer comment (Evaluation):
The main concern is the practicality of the proposed evaluation method. It evolves complicated steps to construct such evaluation agents and prepare their approximate BRs. The idea of computing the BR diversity is straightforward, but it is not easy to really use such a metric in practice. And the huge implementation effort would definitely reduce its impact on the community.

Author response:
**We have clarified that the steps, including designing event-based rewards, generating evaluation partners, and training approximate BRs, are practical in the above replies.** 

We point out that in the rising ZSC community, Overcooked is the most popular and most used for algorithm evaluation, while the community lacks a fair and comprehensive evaluation method. Our evaluation workflow is a strategic enhancement that promises to deliver a fair and comprehensive evaluation of the ZSC algorithm for the current tasks at hand. **Understanding the potential complexity of this implementation, we open-source our code and train agent policies to make a conscious effort to alleviate the burden on the following researchers.**

It's crucial to emphasize that our method is far from impractical in complex environments like robotics. **In scenarios where the algorithm under evaluation successfully trains an agent capable of completing the task, incorporating our evaluation process is entirely feasible and beneficial.** The idea of an event-based reward adds the chance for human involvement within the set parameters of evaluation, giving you more precise control over the situations in which the algorithm is evaluated. For instance, if the objective is to enhance an agent's adaptability to various human reaction speeds, we can effectively design event-based rewards to create evaluation partners with differing response times. Moreover, building a reliable evaluation method is a prerequisite for the vigorous development of the ZSC community. As a result, even though we acknowledge that any new evaluation method will inevitably require some implementation work, we think that the potential advantages of using our framework outweigh the practical difficulties. By providing our implementation and offering ready-to-use components, we aim to enhance the accessibility and impact of our evaluation method within the ZSC research community.

Reviewer comment (Experiments):
The design of reward space is crucial for the proposed evaluation partners. However, it is clearly discussed, at least I didn’t find it yet in the main text. What’s more, it is hard to say the resulting partners would show expected reasonable behaviors.

Author response:
We have presented our reward-space design in Sec. 4.2, and we further give the details of reward-space in Overcooked in Sec. 5. Specifically, the event-based reward space can be formulated as follows:
$$
\mathcal{R}=\\{r _{\bf{w}}| r _{\bf{w}}(s _{t}, \bf{a} _{t}) = r + \phi(s _{t}, \bf{a} _{t})^{T}\bf{w}, \bf{w} \in \mathbb{R}^{m}, \|\bf{w}\| _{\infty} \leq B _{\text{max}}, \sum _{i} \mathbb{1}(\bf w _{i} \neq 0) \leq C _{\text{max}}\\}
$$
The hyper-paramters $B _{\text{max}}$ and $C _{\text{max}}$ are used to restrict the reward space to indicate reasonable behaviors. $\phi(s _t,\bf{a} _{t})$ is the indicator of pre-defined events.

Event-based rewards have been widely used in lots of complex tasks, including robotics [2], large-scale games [3,4] and product management [5]. **We acknowledge that event-based rewards cannot promise to generate all the expected behaviors, while we can indeed get diverse reasonable behaviors after traversing the designed reward space in Overcooked.** 

**We provide visualizations of learned behaviors in Figures 8 and 9 to show that we indeed learned diverse, reasonable behaviors.**

Moreover, we give an example of how to design the reward space.

### An example of Overcooked
In Overcooked, we use $B_{\text{max}} = 20$, $C_{\text{max}}=3$ and generate up to 194 candidates and select up to 30 evaluation partners. The generated candidates are excluded if they cannot complete a delivery when cooperating with their BRs. The pre-defined events are listed as follows:
| Events | Weights |
| --- | --- |
| Put an onion or a dish or a soup onto the counter  | 0  |
| Pickup an onion or a dish or a soup from the counter | 0 |
| Pickup an onion from the onion dispenser | -20,0,10  |
| Pickup a dish from the dish dispenser  | -20,0,10 |
| Pickup a soup | -20,0,5,10 |
| Place an ingredient into the pot | -20,0,3,10 |
| Deliver a soup | -20,0 |
| Stay | -0.1,0,0.1 |
| Movement | 0 |
| Order Reward | 0.1,1 |

We also provide the pseudocode for using these events to show that one can make a moderate effort to implement the event-based rewards.
```python=
state = current state of the game
action = action of the agent whose events are collected
reward = 0
for e_i in range(NUM_EVENTS):
    if True == EVENT_JUDGER(e_i, state, action):
        reward += EVENT_WEIGHTS[e_i]
```
The details are also present in Appendix B.1.

Reviewer comment (Reproducibility):
In addition to the evaluation agents, the proposed method requires users to compute the approximate BRs. In simple tasks like Overcooked, it could be fine. However, it could be hard to obtain in complex robot tasks.

Author response:
**Computing approximate BR.** We have detailed discussed why training approximate BRs is practical for complex tasks in the common response. Our evaluation workflow includes two kinds of approximate best responses. We further explain how we compute the approximate BR here. 

The first ones are obtained when approximating the Nash Equilibrium (NE) of the two-player Markov game constructed with event-based rewards. As described in the paragraph Event-based Behavior Preferring Partners of Sec. 4.2, the two agents receiving an event-based reward and the original task reward, respectively, construct a two-player Markov game. Then, each agent works on their own to maximize their own rewards using the Proximal Policy Optimization (PPO) algorithm. This converges to the NE of this two-player Markov game because the event-based reward still encourages the agents to work together [1]. Then we obtain an approximate NE, which consists of two policies that approximate the best responses to each other. Thus, we obtain a fully trained behavior-preferring partner and its approximate BR. **It won't be more difficult than training the ZSC methods in this case because approximating the NE is like training a self-play agent-pair**.

The second ones are the approximate best responses to earlier checkpoints of the fully trained behavior preferring agents. We train an approximate BR by performing the PPO algorithm to maximize the original task reward with the partner as one of those earlier checkpoints. **The complexity of this case will be lower than training a self-play agent pair since one of the agents is fixed.**

Therefore, we would like to alleviate the concern raised about the feasibility of obtaining Approx. BRs in more complex tasks, such as robotics. We argue that training partner agents and Approx. BRs could be more practical than intuitively thought and more convenient and practical compared to frequently recruiting humans for evaluations. 

If an algorithm is capable of training two robots to collaboratively solve complex tasks, i.e., at least capable of training a self-play agent pair to solve complex tasks, **the extension of this capability to approximate an NE or train strong responses to fixed partners should be viewed as practically achievable**. The creation of these kinds of algorithms naturally shows that they can handle the computational and training problems that come with making approximate BRs.

Moreover, it's important to understand that we only approximate the BRs instead of finding the exact BRs in our framework. The approximate BRs serve primarily as strong responses for each evaluation partner. The implementation and computation requirements for approximating BRs are much easier than finding exact BRs. With feasible implementation and computation costs, approximate BRs benefit the ZSC evaluation by supporting the BR-Div and providing a fair and robust measurement for ZSC performance.

Reviewer comment (Reproducibility):
How to compute the approximate BR, $\widehat{BR}(\pi_{\omega})$ ?

Author response:
**Computing approximate BR.** We have detailed discussed why training approximate BRs is practical for complex tasks in the common response. Our evaluation workflow includes two kinds of approximate best responses. We further explain how we compute the approximate BR here. 

The first ones are obtained when approximating the Nash Equilibrium (NE) of the two-player Markov game constructed with event-based rewards. As described in the paragraph Event-based Behavior Preferring Partners of Sec. 4.2, the two agents receiving an event-based reward and the original task reward, respectively, construct a two-player Markov game. Then, each agent works on their own to maximize their own rewards using the Proximal Policy Optimization (PPO) algorithm. This converges to the NE of this two-player Markov game because the event-based reward still encourages the agents to work together [1]. Then we obtain an approximate NE, which consists of two policies that approximate the best responses to each other. Thus, we obtain a fully trained behavior-preferring partner and its approximate BR. **It won't be more difficult than training the ZSC methods in this case because approximating the NE is like training a self-play agent-pair**.

The second ones are the approximate best responses to earlier checkpoints of the fully trained behavior preferring agents. We train an approximate BR by performing the PPO algorithm to maximize the original task reward with the partner as one of those earlier checkpoints. **The complexity of this case will be lower than training a self-play agent pair since one of the agents is fixed.**

Therefore, we would like to alleviate the concern raised about the feasibility of obtaining Approx. BRs in more complex tasks, such as robotics. We argue that training partner agents and Approx. BRs could be more practical than intuitively thought and more convenient and practical compared to frequently recruiting humans for evaluations. 

If an algorithm is capable of training two robots to collaboratively solve complex tasks, i.e., at least capable of training a self-play agent pair to solve complex tasks, **the extension of this capability to approximate an NE or train strong responses to fixed partners should be viewed as practically achievable**. The creation of these kinds of algorithms naturally shows that they can handle the computational and training problems that come with making approximate BRs.

Moreover, it's important to understand that we only approximate the BRs instead of finding the exact BRs in our framework. The approximate BRs serve primarily as strong responses for each evaluation partner. The implementation and computation requirements for approximating BRs are much easier than finding exact BRs. With feasible implementation and computation costs, approximate BRs benefit the ZSC evaluation by supporting the BR-Div and providing a fair and robust measurement for ZSC performance.

Reviewer comment (Experiments):
How to design the reward space?

Author response:
We have presented our reward-space design in Sec. 4.2, and we further give the details of reward-space in Overcooked in Sec. 5. Specifically, the event-based reward space can be formulated as follows:
$$
\mathcal{R}=\\{r _{\bf{w}}| r _{\bf{w}}(s _{t}, \bf{a} _{t}) = r + \phi(s _{t}, \bf{a} _{t})^{T}\bf{w}, \bf{w} \in \mathbb{R}^{m}, \|\bf{w}\| _{\infty} \leq B _{\text{max}}, \sum _{i} \mathbb{1}(\bf w _{i} \neq 0) \leq C _{\text{max}}\\}
$$
The hyper-paramters $B _{\text{max}}$ and $C _{\text{max}}$ are used to restrict the reward space to indicate reasonable behaviors. $\phi(s _t,\bf{a} _{t})$ is the indicator of pre-defined events.

Event-based rewards have been widely used in lots of complex tasks, including robotics [2], large-scale games [3,4] and product management [5]. **We acknowledge that event-based rewards cannot promise to generate all the expected behaviors, while we can indeed get diverse reasonable behaviors after traversing the designed reward space in Overcooked.** 

**We provide visualizations of learned behaviors in Figures 8 and 9 to show that we indeed learned diverse, reasonable behaviors.**

Moreover, we give an example of how to design the reward space.

### An example of Overcooked
In Overcooked, we use $B_{\text{max}} = 20$, $C_{\text{max}}=3$ and generate up to 194 candidates and select up to 30 evaluation partners. The generated candidates are excluded if they cannot complete a delivery when cooperating with their BRs. The pre-defined events are listed as follows:
| Events | Weights |
| --- | --- |
| Put an onion or a dish or a soup onto the counter  | 0  |
| Pickup an onion or a dish or a soup from the counter | 0 |
| Pickup an onion from the onion dispenser | -20,0,10  |
| Pickup a dish from the dish dispenser  | -20,0,10 |
| Pickup a soup | -20,0,5,10 |
| Place an ingredient into the pot | -20,0,3,10 |
| Deliver a soup | -20,0 |
| Stay | -0.1,0,0.1 |
| Movement | 0 |
| Order Reward | 0.1,1 |

We also provide the pseudocode for using these events to show that one can make a moderate effort to implement the event-based rewards.
```python=
state = current state of the game
action = action of the agent whose events are collected
reward = 0
for e_i in range(NUM_EVENTS):
    if True == EVENT_JUDGER(e_i, state, action):
        reward += EVENT_WEIGHTS[e_i]
```
The details are also present in Appendix B.1.

Reviewer comment (Experiments):
If all of these evaluation agents can be constructed, why not use them to train the ego agents?

Author response:
To clarify, our primary focus is on developing an evaluation methodology with a keen emphasis on the fairness of comparisons. **The key challenge of the ZSC problem is generating ego agents that can cooperate with unknown partners.** Thus, we generate a set of diverse evaluation partners, evaluate the agents with these evaluation partners, and measure the ZSC performance with BR-Prox. The problem of how to train ZSC agents is beyond our scope.

We discuss this problem in two cases:
- Under our evaluation workflow, training an ego agent using the evaluation partners and evaluating the trained ego agent with the same set of evaluation partners is similar to using the test dataset as the training data in the supervised learning algorithms, which is unreasonable.
- However, if you mean training the ego agent with a set of diverse partners constructed using our method and evaluating the trained ego agent with another set of constructed agents, we have added similar experiments to the updated paper. HSP is an algorithm that constructs partners to simulate human behaviors and trains the ego agent with those constructed partners. We train the HSP ego agents with unselected evaluation partner candidates and evaluate the HSP ego agents with the selected evaluation partners.

We hope our reply addresses your concerns about practicality, and we are glad to discuss further if you have any other suggestions or questions.

Reviewer comment (Writing):
Some parts of the paper are not very clear. For example, the motivation for introducing the event-based rewards is not clear.

Author response:
Thank you for suggesting potential improvements to the presentation. We have elaborated on some technical details in the updated paper.

Specifically, we explain our motivation for introducing event-based rewards in Sec. 4.2. We expect partners to have diverse and reasonable behaviors, while the behavior space is intractably large. Inspired by [1] in which human preferences can be regarded as event-centric and modeled as event-based reward functions, we apply the event-based reward shaping method to encourage behavior discovery and generate behavior preferring partners.

Reviewer comment (Reproducibility):
In practical implementation, the method requires humans to manually define some triggered events so as to derive diverse behaviors, which is difficult to obtain in complex tasks.

Author response:
We have detailedly discussed the practicality of our evaluation method in the common response. Specifically, we argue that the extra human effort is not as heavy as intuitively thought when applying our evaluation workflow to a new task. Defining some triggered events in complex tasks is very similar to conventional feature and reward engineering, which is inevitable when exploring new tasks. Moreover, event-based reward shaping has already been applied in both complex robot tasks [2] and large multi-agent games [3,4]. So defining events may not cost a lot of extra effort and is general to the ZSC domain. Besides, there indeed exist works that learn human behaviors without assuming a set of events [5] and we leave making these methods general to the ZSC domain as future work.

We additionally give an example of the event-based reward design in Overcooked to explain the feasibility of designing event-based rewards. The pre-defined events are listed as follows:

| Events | Weights |
| --- | --- |
| Put an onion or a dish or a soup onto the counter  | 0  |
| Pickup an onion or a dish or a soup from the counter | 0 |
| Pickup an onion from the onion dispenser | -20,0,10  |
| Pickup a dish from the dish dispenser  | -20,0,10 |
| Pickup a soup | -20,0,5,10 |
| Place an ingredient into the pot | -20,0,3,10 |
| Deliver a soup | -20,0 |
| Stay | -0.1,0,0.1 |
| Movement | 0 |
| Order Reward | 0.1,1 |

We also provide the pseudocode for using these events to show that one can make a moderate effort to implement the event-based rewards.
```python=
state = current state of the game
action = action of the agent whose events are collected
reward = 0
for e_i in range(NUM_EVENTS):
    if True == EVENT_JUDGER(e_i, state, action):
        reward += EVENT_WEIGHTS[e_i]
```
The details are also present in Appendix B.1.

Reviewer comment (Experiments):
Comparisons with previous baselines listed in Table 1 may be missing.

Author response:
We do not fully understand whether you mean comparison between our evaluation workflow and the previous evaluation methods or comparison among the ZSC algorithms.

**Comparisons with Previous Evaluation Methods**
We detailedly compare the previous evaluation methods and our evaluation workflow in the common response, which reveals that previous evaluation partners and metrics cannot provide a comprehensive and fair evaluation and that our evaluation workflow has advantages in measuring the ZSC capability.

We first clarify that we list the previous evaluation methods in the first column and the works that utilize them in the second column in Table 1. 

**For the previous evaluation methods**, we have already discussed their defects in Sec. 3.2 and Appendix C.  

Defects of previous evaluation partners:
- Human Players: The human players in ZSC problem are 'perfect' candidates for evaluation because human are strictly qualified as unseen partners. While evaluating with human players are expensive, time-consuming and unrepeatable. We need a more efficient evaluation method as a supplement for human evaluation. 
- Human Proxy Agents: Human proxy agents in overcooked environment do not account for human behaviors [5], which shows that using human proxy agents does not represent the diversity of human.
- Trained Self-play Agents: Train self-play agents are similar to the agents used to train the ego agent and are not diverse, as shown in Figure 2.
- Rule-based Specialist: Manually building expert rules is difficult to implement in complex environments and may not meet diversity requirements. We point out that designing a set of events is much easier than designing a policy.
- Random Agents: The diversity of the random initializations cannot be ensured and random initializations lack of high level performance.

Defects of previous metric:
- Cross-play with Trained Adapted Agents: Some works compare their ZSC methods by cross-playing the agents trained by other ZSC methods and their ego agents. This results include self-play performance and does not completely reflect the capabilities of ZSC. On the other hand, excluding the self-play results leads to potential unfairness.

We have highlighted the differences in Sec. 2 of the update paper. We also remark that we are the first to systematically investigate the construction of evaluation partners and the measurement of ZSC capability.

**Comparison among ZSC algorithms**
To improve the reliability of our evaluation results, we have reevaluated five representative ZSC algorithms: SP, FCP, MEP, TrajeDi, and COLE. We also reevaluate the HSP [1] algorithm in Overcooked and add the results to the revised paper.

We hope that our explanations and clarifications have addressed your concerns and shed light on the insights behind our methodology and its potential impact. We invite further discussion and are open to any additional questions or suggestions you may have.

**Reference**

[1] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. "Learning zero-shot cooperation with humans, assuming humans are biased". ICLR. 2023.

[2] Brohan, Anthony, et al. "Do as i can, not as i say: Grounding language in robotic affordances." CoRL. 2023.

[3] Kurach, Karol, et al. "Google research football: A novel reinforcement learning environment." AAAI. 2020.

[4] Deheng Ye, et al. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. AAAI. 2020.

[5] Shah, Rohin, et al. "On the feasibility of learning, rather than assuming, human biases for reward inference." ICML. 2019.