Paper: A Cooperative-Game-Theoretical Model for Ad Hoc Teamwork (ICLR-2024)

Reviewer comment (Theory):
In the definitions of $\mathcal A_{\mathcal N}$ and $\Theta_{\mathcal N}$, the reviewer thinks an $\exists$ quantifier should be used instead of a $\forall$, as otherwise both sets are empty since there is no joint action that simultaneously belongs to $\mathcal A_{\mathcal N_t}$ for all ${\mathcal N_t}$.

Author response:
**I think $\mathcal{A}\_{\mathcal{N}}$ and $\Theta\_{\mathcal{N}}$ should have an $\exists$ quantifier in their definitions, not a $\forall$ --- otherwise, they're both empty sets, since there is not a joint action that is simultaneously in $\mathcal{A}\_{\mathcal{N}\_{t}}$ for all $\mathcal{N}\_{t}$.**

    *Reply:* Thank you for your suggestion. The original expression of $\mathcal{A}\_{\mathcal{N}}$ and $\Theta\_{\mathcal{N}}$ are actually confused. In our view, it could be better to express them as $\mathcal{A}\_{\scriptscriptstyle\mathcal{N}} = \bigcup\_{\mathcal{N}\_{t} \in \mathbb{P}(\mathcal{N})} \{a \vert a \in \mathcal{A}\_{{\scriptscriptstyle \mathcal{N}}\_{t}} \}$ and $\Theta\_{\scriptscriptstyle\mathcal{N}} = \bigcup\_{\mathcal{N}\_{t} \in \mathbb{P}(\mathcal{N})} \{\theta \vert \theta \in \Theta^{{\scriptscriptstyle |\mathcal{N}\_{t}|}} \}$. We are happy to hear from your feedback.

Reviewer comment (Presentation):
In the formulation in Section 2.2, it is unclear how the team $\mathcal N_t$ evolves over time. This seems to be formalized in the next subsection, but if so, there should be a forward pointer.

Author response:
**In the formulation in Sec 2.2, it is unclear how the team $\mathcal{N}_{t}$ evolves with time. It seems to me that this is formalized in the next subsection, but if so there should be a forward pointer.**

    *Reply:* Thank you for your suggestion. However, $\mathcal{N}\_{t}$ in Section 2.2 was defined by the GPL paper. We agree with you that it is unclear how $\mathcal{N}\_{t}$ evolves. This is the reason why we redefine the dynamics thouroughly in our own theoretical model in the next subsection. We believe you have already discovered some benefits and novelties of our work. To respect the GPL paper and distinguish our work from theirs, we decide to keep the current expression.

Reviewer comment (Theory):
In Theorem 1, it should be made clear exactly what "maximizing the social welfare under the grand coalition" means. The reviewer interprets it as finding a joint policy $\pi : \mathcal S \times \mathbb P(\mathcal N) \ni (s_t, \mathcal N_t) \mapsto a_t \in \mathcal A_{\mathcal N_t}$ that, for every pair $(s_t, \mathcal N_t)$, selects $a_t$ to maximize the local social welfare $\sum_{i \in \mathcal N_t} R_i(s_t, a_t)$. This should be formally stated.

Author response:
**In Theorem 1 it should be made clear exactly what "maximizing the social welfare under the grand coalition" means. I am interpreting it as finding a joint policy, that is, a map $\pi: \mathcal{S} \times \mathbb{P}(\mathcal{N}) \in (s\_{t}, \mathcal{N}\_{t}) \mapsto a\_{t} \in \mathcal{A}\_{\mathcal{N}\_{t}}$ that, for every pair $(s\_{t}, \mathcal{N}\_{t})$, selects $a\_{t}$ to maximize the local social welfare $\sum\_{j \in \mathcal{N}\_{t}} R_{j}(s\_{t}, a\_{t})$. Is that correct? In any case this should be formally stated.**

    *Reply:* Thanks for your suggestion. Your understanding about the result of Theorem 1 for fitting the problem we would like to solve (open ad hoc teamwork) is correct. However, the result from Theorem 1 is general for any preference value profile in CAG. Note that in the original version of CAG, it does not consider action space, while in our theory we extend the definition of preference value with action as an decision variable (i.e. a realization of preference value). Nevertheless, we agree with you that the statement should be linked to our problem more directly to make it better comprehensive. For this reason, we have added a corollary to comprehend the result of Theorem 1 in our case in the revised paper.

Reviewer comment (Theory):
The paper appears to adopt a single-agent perspective, with a single learner $i \in \mathcal N$ and all other agents’ policies fixed. The reviewer questions how social welfare can be optimized in Theorem 1 under this setup, especially if the other agents act such that learner $i$ alone cannot achieve social welfare optimality.

Author response:
**It seems like the paper is adopting a single-agent perspective, where there is a single learner $i \in \mathcal{N}$ and all other agents' policies are held fixed (is this correct?). But then how can we optimize social welfare, required by Theorem 1, if we only control the single learner $i$? In particular, what if the other agents are acting in such a way that learner $i$ alone cannot achieve social welfare optimality?**

    *Reply:* Thanks for your really thoughtful questions. Yes, in the common setting of ad hoc teamwork, we can only control one agent to collaborate with other teammates on the fly. To make the theoretical analysis simpler, we here assume all other agents' policies are held fixed (i.e., to keep the environment stationary to the learner we control). However, this work can be extended to scenarios where other agents' policies would vary, for which the analysis would become complicated due to the non-stationary envrionment. 
    
    Although we only control the learner $i$, it can affect other agents' decision, since other agents are able to respond the learner's action with regarding the learner's behavior as partial input of their policies. As a result, other agents' individual preference reward as an evaluation of their policies would accordingly change. 
    
    The final question is really insightful. Since the basic assumption of ad hoc teamwork is that all agents are assumed to have a common goal, so the situation you consider is out of scope. Nevertheless, if the situation you state really happens (that is an open question), then for the moment we can only say it could be extended to control a group of agents, where at least one agent is able to influence other uncontrollable agents. We hope our response addresses your concerns.

Reviewer comment (Writing):
Can you please elaborate on what do types represent since all teammates always want to collaborate? This is not explained in the text.

Author response:
The types here is for emphasizing the heterogeneity of agents. Even if all teammates always want to collaborate, a group of different types of agents may collaborate in different manners. Thereby, in this paper we apply affinity graph with learnable weights (capture variant collaborative manner) to depict this property.

Reviewer comment (Presentation):
Can you please highlight the main theorem that addresses the motivating question at the bottom of page and top of page 2? I admit that I got lost in some parts of the theoretical presentation.

Author response:
Sorry for making you lost owing to our presentation. For convenience, we will reply following the layout of the revised paper. The answer to the motivating question is constituted of the holistic theory. It starts from the introduction of CAG with the affinity graph and the solution concept for solving the weights of the affinity graph called strict core stability (see Definition 1). Then, we define the affinity graph as a star graph to better describe the scenario of open ad hoc teamwork (see Definition 2). After that, we derive a method to find the solution of weights (see Theorem 1 and the derived Eq. (4)) and an inductive bias for easily finding the weights (see Theorem 2 and Proposition 2). Finally, we show the feasibility of solving Eq. (4) under the open team scenario (see Lemma 1 and Theorem 3). In summary, our theory answers the question that why introducing and learning a graph is necessary and feasible to solve open ad hoc teamwork from the theoretical perspective rather than the pure intuition in the GPL paper, so the method derived from our theory is reliable.

Reviewer comment (Presentation):
An algorithm environment for CA-GPL would have aided the reader to appreciate the novelties over GPL.

Author response:
What do you mean by an algorithm environment? Could you please give some more information about it? We have added an achitecture of CA-GPL to illustrate the novelty over GPL in the revised paper as Figure 1 shows.

Reviewer comment (Writing):
Example of complicated notation — what is $|-i|$? Is it the cardinality of the set of all agents other than $i$? If yes, then first, this is never explained, and second why not simply write $N_t-1$?

Author response:
Yes, thanks for the suggestion. We do not write it as $N_{t} - 1$ is for the conciseness in expression.

Reviewer comment (Writing):
Example of complicated notation — what is $b_j$ in the bottom of page 2? Is this used later on?

Author response:
$b_{j}$ was originally used to indicate that the singleton coalition value is a constant value greater than or equal to zero. However, we agree with your suggestion and modify it as $v_{j}(\mathcal{C}) = v_{j}(\{j\}) \geq 0$ in the revised paper.

Reviewer comment (Writing):
Example of complicated notation — what is $a_{t,j}$ below equation (1)? Should this be $a_{t,-i}$?

Author response:
Yes, you are correct. We have fixed it in the revised paper. Thanks for the careful inspection.

Reviewer comment (Writing):
Social welfare is used in Theorem 1, but it has not been defined before.

Author response:
We now refine the contents in this paragraph and social welfare is defined in the revised paper.

Reviewer comment (Theory):
It seems to me that the definition of the joint policy $\pi_{t,-i} :\mathcal{S}\times \Theta_N \to \Delta(\mathcal{A}_N)$ allows for correlated policies. However, this contradicts the assumption that permeats the text, that teammates don't interact at all with each other. Is this concern valid?

Author response:
Thanks for the really good question. The correlated policy is not allowed in our work. The definition here is the one we refer to from the origianl GPL paper. If combined with our specified definition in Section 3.1 such that the corresponding coalitional action decided by $\pi_{t, {\scriptscriptstyle \mathcal{C}}} = \times_{j \in {\mathcal{C}}} \ \pi_{j}$, this ambiguity can be addressed. For instance, let $\mathcal{C} = -i$, then $\pi_{t,-i} = \times_{j \in -i} \ \pi_{j}$.

Reviewer comment (Theory):
Theorem 1 is hard to parse for me. If the grand coalition is strict core stable (with respect to what valuations?), then what does it mean to show that it is strict core stable? Apologies if my confusion is not justified.

Author response:
We believe any concern is necessary to be addressed before publication. To highlight the valuation of strict core stability, we have specified it in an exclusive definition as Definition 1 in the revised paper. We say that a blocking coalition $\mathcal{C}$ weakly blocks a coalition structure $\mathcal{CS}$ if every agent $j \in \mathcal{C}$ weekly prefers $\mathcal{C}$ to $\mathcal{CS}(j)$ and there exists at least one agent $j' \in \mathcal{C}$ who strictly prefers $\mathcal{C}$ to $\mathcal{CS}(j)$. A coalition structure admitting no weakly blocking coalition $\mathcal{C} \subseteq \mathcal{N}$, exhibits the strict core stability.  
    
    The implication of Theorem 1 is as follows: If there exist weights ($w_{jj'}$) to guarantee the strict core stability of the grand coalition, then maximizing social welfare is an approach to find the solution. Our previous description of Theorem 1 does not capture our intension well and make confusion to you. We are so sorry about it. In the revised paper, we have removed "the grand coalition is assumed to be strict core stable" without changing the purpose of Theorem 1, to avoid confusion.
    
    Theorem 2 mainly focuses on confining the space of $w_{jj'}$ (as a inductive bias) to facilitate the search of solution. Note that even if the condition in Theorem 2 can guarantee the existence of a grand coalition as a strict core stability solution, we still need to find the appropriate weights (i.e. in long term in the paper for tractability) that satisfy the global reward (which describes the goal of a collaborative task). This justifies the necessity of the result of Theorem 1. We clarify this in the revised paper. We hope our explanation can address your concerns.

Reviewer comment (Evaluation):
Regarding the experiments: In Table 1, we see that the stability metric worsens for CA-GPL in both scenarios as training progresses (especially for the 4 teammates). Why is this? Also, in the figures, we see that the curves keep increasing (they have not converged). Is this a valid concern? And would it make sense to provide larger frames?

Author response:
This is really a good question. In general, this verifies the effectiveness of our method on any arbitrary agent-type composition (avoiding the possibility of overfitting). The main reason could be that our star graph architecture with inductive bias on utilities ($Q_{ij} \geq 0, Q_{i} \geq 0, Q_{j} \geq 0$) can capture the relation between the learner and teammates well, without the noisy estimation of relations between teammates that may happen in GPL (which is equipped with a fully connected graph, without any inductive bias on learning utilities). About the curves keeping increasing, it is difficult to be controlled during experiments. Although we agree that this might be resolved with larger frames, due to the page limit it is not easy to provide that in the revised paper.

Reviewer comment (Writing):
There are many typos or difficult to understand sentences — p8: "the relationship among ad hoc teammates is weak to" -> shouldn't it be the opposite here, i.e., strong relationship?

Author response:
We rephrase the sentence as "The comparatively good performance of CA-GPL-FCG could be due to that there still exists relationship among ad hoc teammates in some environments, but not strong enough, so its performance is still worse than CA-GPL" in the revised paper. This is what we would like to express.

Reviewer comment (Theory):
In many instances of the paper, assumptions are introduced without justification and they are not properly highlighted (just a "There exists..."). As an example, the existence of an affinity graph is not guaranteed in general as far as I know.

Author response:
Thanks for the interesting comment. "There exists ..." is a type of words to given a boudary, based on which we can analyze the results following logics. As for the existence of affinity graph, we now add discussion on the boundary in the revised paper (See Section 3.2 **Affinity Graph for Ad Hoc Teamwork**). We sincerely hope you could double check it.

Reviewer comment (Presentation):
Many different probability distributions are introduced ($P_E$, $P_O$, $\mathcal T$, ...) in Section 3.1, making the dependencies across variables really obscure. Figure 5 from the appendix and a clearer text may help.

Author response:
Thanks for your comment. In our view, different probabilistic distributions defined here does not obscure the dependencies across probabilities. Instead, it clarifies the dependencies, since compared with expressing the transition function as a whole probability distribution in SBG, it points out the position of intervention (control) during the whole ad hoc teamwork process, following the view of causal inference. Nonetheless, we agree that a figure could help understanding, we have redrawn Figure 5 in the last version and it is now Figure 4 in the revised paper.

Reviewer comment (Writing):
Lack of an example to clarify the settings and its peculiarities.

Author response:
Thanks for the insightful comment. This paper is not like the purpose of popular papers that capture a weakness that a prior work cannot solve and propose a method that can improve the weakness. The research question (motivation) of this paper is that the powerful GPL framework does not have a theoretical understanding of why the incorporation of graph can help. For this reason, we propose to incorporate CAG and surely its theory to mitigate this issue. More importantly, the incorporation of CAG can lead to the possibility the further investigation into more diverse situations (e.g., friend and enermies both exist). Although GPL seemly can address almost every situations, the further improvement of this architecture is a big problem. The main reason is that it does not consider any boudary from theoretical perspective. **However, the question is that we do not know whether the design is the best to any specific situation.** For example, in this paper, we propose to constrain the range of individual and pairwise Q-values to be greater than or equal to zero, according to our theory. Similarly, we can follow the route of the various and well studied research of cooperative game theory to analyze other scenarios (e.g. friend and foe).

Reviewer comment (Writing):
The term "Preference Q-Value" is unneeded, as it is just a Q value over the action space of the agent.

Author response:
Thanks for this interesting point. We use the term "Preference Q-Value" is to credit and respect the work of CAG and cooperative game theory as well as to distinguish our work from others, since the Q-value is derived by the theory over preference value which is the terminology in cooperative game theory. **If following your logic, it is also unnecessary to define Q-value since it is just a function.** We sincerely hope you could reconsider your statement.

Reviewer comment (Writing):
The text between Theorem 2 and Proposition 2 is unreadable and I could not understand it even reading it multiple times. Please rephrase it and provide proper space for the mathematical equations used.

Author response:
Thanks for the good point. We agree with your view and have rephrased and rearranged the texts in the revised version of paper. Please check it and give feedback to us that whether it is easier to be understood now.

Reviewer comment (Writing):
The GPL framework is never defined, and it could really help to highlight the contributions of the paper.

Author response:
Thanks for the useful comment. **Actually, we have defined GPL in details in the paper. Due to the page limits, we leave it in Appendix.** We sincerely hope you can double check it. We now set a pointer to it in the section of practical implementation in the revised paper for clarity.

Reviewer comment (Theory):
The usefulness of the theoretical contributions is debatable. In particular, the whole coalitional affinity games (CAG) framework seems like a very elaborate way of justifying the star-shaped Q-value computation novelty introduced. However, such a conclusion derives rather directly from the star-shaped interaction graph between agents, that is assumed.

Author response:
Thanks for the thoughtful comment. First, we would like to solve ad hoc teamwork, where the basic requirement is that an agent needs to act to cooperate with other agents (teammates). As we mentioned in the paper (see Section 3.1), the star graph is adequate to solve this problem. We do not agree that more general the result is, more exciting is the result, which should be discussed based on the situation or problem we would like to solve. Additionally, we have also mentioned in Proposition 4 (see Appendix E in the revised paper) that even with the fully connected graph, there still exists a representation of affinity weights to satisfy symmetry, individual rationality and the grand coalition exhibiting strict core stability. **This means that our theory is not limited to the star graph.** 
    
    Finally, we **do not** agree that the introduction of CAG makes the formalism uselessly complex. As we mentioned before, the introduction of CAG is only the first step to formalize the graph-based method for ad hoc teamwork (which is also beneficial to the validity of GPL). Then, referring to the complete study in cooperative game theory (see Section Conclusion), some more interesting scenarios such as a situation involving both teammates and enermies can be deeply investigated.

Reviewer comment (Theory):
Up to Theorem 2, the theoretical contributions are just saying that we need to optimize the learner's utility, which coincides with the social welfare thanks to the assumption of a star-shaped graph. Given the context at hand, this result is banal and the introduction of CAG actually makes the formalism uselessly complex.

Author response:
Thanks for the thoughtful comment. First, we would like to solve ad hoc teamwork, where the basic requirement is that an agent needs to act to cooperate with other agents (teammates). As we mentioned in the paper (see Section 3.1), the star graph is adequate to solve this problem. We do not agree that more general the result is, more exciting is the result, which should be discussed based on the situation or problem we would like to solve. Additionally, we have also mentioned in Proposition 4 (see Appendix E in the revised paper) that even with the fully connected graph, there still exists a representation of affinity weights to satisfy symmetry, individual rationality and the grand coalition exhibiting strict core stability. **This means that our theory is not limited to the star graph.** 
    
    Finally, we **do not** agree that the introduction of CAG makes the formalism uselessly complex. As we mentioned before, the introduction of CAG is only the first step to formalize the graph-based method for ad hoc teamwork (which is also beneficial to the validity of GPL). Then, referring to the complete study in cooperative game theory (see Section Conclusion), some more interesting scenarios such as a situation involving both teammates and enermies can be deeply investigated.

Reviewer comment (Presentation):
The final architecture developed for the CAG-GPL is not described in detail, nor properly compared with GPL. A picture is needed, as in the GPL paper.

Author response:
Thanks for the useful suggestion. We have added a figure to describe the architecture in Figure 1 in the revised paper.

Reviewer comment (Evaluation):
In Figure 1 it seems that CAG-GPL has really similar performance to GPL.

Author response:
Thanks for the useful comment. The similar performance in Figure 1 is mainly due to the simplicity of the scenario (with at most 2 teammates), where the difference in architecture between the fully connected graph (GPL) and the star graph (our method) is limited. For the worst case, GPL would at most overestimate one more edge than CA-GPL. **We belive that this instead verifies the necessity to develop a theory over the star graph as an evidence to support our response to Question 7 above.**

Reviewer comment (Presentation):
Figure 3 is unreadable: the legend's colors do not match with the plot, and in the Q-value plot there is written "learner" even on the rows not related to the learner.

Author response:
Thanks for the careful inspection. We have fixed this legend color mismatch typo in the revised paper. Additionally, we are sorry that our initial figure caption and the main text may not have provided a clear explanation. The Q-value plot you referred to presents the Preference-Q (Pref-Q) with the actions illustrated along the x-axis. To clarify, the last row of each heatmap indicates the learner's preference Q-value, while the previous rows depict teammates' Pref-Q, given teammates' actual actions. **Besides, the symbol "tri\_up" denoted as 'learner' marks the current action taken by the learner. By incorporating the learner’s actions into the Q-value plot, our aim is to demonstrate the alignment between the Pref-Q and cooperative behaviors (shown in the top trajectory visulization), even within a variable team setting.** In the heat map, the redder the color, the greater the Q value. It is observable that the learner's actions, marked by "tri_up", are consistently aligned with the areas of highest Q values. Nevertheless, the preference Q-value cannot always perfectly align with the learner's actions due to the learner's inaccurate estimation of teammates' policies. **The most successful alignments between trajectories and Pref-Q justify the efficacy of Pref-Q to encourage collaboration in open ad hoc teamwork and, thus, verify the reasonableness of our theoretical model.** Due to the limitation of page, we have moved Figure 3 to Appendix G.2. Additionally, we have provided a more detailed caption to highlight the interesting findings more effectively. Thanks again for your valuable questions.

Reviewer comment (Novelty):
Considering that the idea of combining open SBG and CAG is quite natural, and so does the RL formulation, the contribution of the work will concentrate on its practical performance.

Author response:
Thanks for your very critical comments. However, we would like to argue that the combination of SBG and CAG is not trivial or natural. The primary reason is that SBG is actually following the **non-cooperative game theory** framework, while CAG is under the **cooperative game theory**. These are totally different concepts, where SBG follows the solution concept of Nash equilibrium, whereas OSB-CAG or CAG follows the solution concept of strict core stability. In Section 4, we show the equivalence between these two concepts, then it establishes the connection between SBG and OSB-CAG. Therefore, it not only makes the GPL as an implementation of our framework, but also understands GPL from a theoretical perspective. Note that the original GPL only intuitively (empirically) extends the framework from SBG with a graph to handle open team settings without any theoretical understanding. **More importantly, according to the theory from CAG, we propose to set the range of individual utilities and pairwise utilities as greater than or equal to zero, which was never mentioned in GPL.**

Reviewer comment (Evaluation):
The difference of the outperformance seems marginal.

Author response:
Thanks for your comments again. If possible, could you please mention the concreate cases where our method only outperforms GPL marginally? In our understanding, CAG-GPL (our method) ourperforms GPL in every scenario, and with a large margin in most of scenarios, which is also recognized by Reviewer eRVD.  

We respect your understanding of "marginal" and "no margins", but we do not agree with this. What we can emphasize is that the primary algorithm to be compared is GPL. Other baselines are variants of our algorithms to show the correctness of outputs of our theoretical framework.