Paper: A Case Study for the Behaviors of Generalists and Specialists in Competitive Games (ICLR-2024)

Reviewer comment (Writing):
Sec 2.1: What is a "population", and what does it mean for a population to be "heterogeneous"?

Author response:
$\textbf{Q1:}$ What is a “population”, and what does it mean for a population to be “heterogeneous”?

$\textbf{A1:}$ A population refers to a set of agents that are trained simultaneously in a population-based training framework. A population is considered heterogeneous if the agents have different attributes, such as height, weight, etc.

Reviewer comment (Writing):
Sec 2.2: What is "gradient exchange"?

Author response:
$\textbf{Q2:}$ What is “gradient exchange”?

$\textbf{A2:}$ Gradient exchange is a technique that allows agents to share their gradients with each other during training. This can facilitate exploration, diversity, and coordination among the agents.

Reviewer comment (Writing):
Sec 2.2.1: I understand x and y to stand in for a^i and a^ii respectively. Is this accurate? In any case, this should be stated.

Author response:
$\textbf{Q3:}$ I understand x and y to stand in for a^i and a^ii respectively. Is this accurate? In any case this should be stated.

$\textbf{A3:}$ Yes, the reviewer is correct. We will make clarification in paper's revision.

Reviewer comment (Writing):
Sec 3: "Generalist" should be defined formally.

Author response:
$\textbf{Q4:}$ “Generalist” should be defined formally

$\textbf{A4:}$ A generalist is a policy that controls a population of agents that the training objective is such that the policy can be applied to a heterogeneous agent population. We have added a formal definition of generalist in the paper.

Reviewer comment (Writing):
It is unclear whether I should be considering a partially-observable game or a fully-observable game. The preliminaries suggest partial observability, but, for example, policies are conditioned on states, which suggests full observability.

Author response:
$\textbf{Q5:}$ Should I be considering a partially-observable game, or a fully-observable game? The prelims suggest partial observability, but e.g. policies are conditioned on states, which suggests full observability

$\textbf{A5:}$ We consider both partially-observable and fully-observable games in our paper. During competition, individual agents only observe their individual observations. After a competition, for example rock paper scissors, all agents see what the other agents have done. Liu et al. 2022's proposed condition neural net of NeuPL takes advantage of this by letting all agents jointly optimize on all agent’s experience. In other words, during training the fully observable experiences are used to train all agents. We have made this distinction more clear in the paper.

Reviewer comment (Theory):
My understanding is that the setting is a two-player zero-sum game. In that setting, a player should never correlate its action with the opposing player, i.e., the mutual information (8) should always be 0 in equilibrium. Why, then, is it interesting to consider (or optimize) the mutual information?

Author response:
$\textbf{Q6:}$ My understanding is that the setting is a two-player zero-sum game. In that setting, a player should never correlate its action with the opposing player, i.e., the mutual information (8) should always be 0 in equilibrium. Why, then, is it interesting to consider (or, indeed, to optimize) the mutual information?

$\textbf{A6:} We agree that in a two-player zero-sum game, the mutual information between the actions of the players should be zero in equilibrium. However, we are interested in the problem setting of population learning via self-play and policy gradient optimization. In this context, Eq. (3) is the result of (x and y) versus (x’ and y’), where (x’ and y’) are (x and y)’s opponent. As proposed by Liu et al 2022, NeuPL can make the two policies’ optimization more efficient by jointly optimizing them under a single conditional policy (conditioned on x and y’s id). The neural architecture of NeuPL is a single conditioned policy that jointly optimizes a population of agents. In Liu et al 2022’s paper, they showed that there is a Skill Transfer effect that makes agents’ learning more efficient. Our paper also finds the training would be more efficient, but at NeuPL’s convergence, it would not be optimal. Optimal here means there is still room for improvement for the individual agents to further improve.

Reviewer comment (Writing):
The writing of this paper is very unclear and the notation is not precise so it is very hard to follow what is the exact setting and main contribution of the paper.

Author response:
We apologize for the unclear writing and notation in the paper. We will revise the paper to improve the clarity and precision of the presentation.

Reviewer comment (Reproducibility):
The code is not provided for reproduction.

Author response:
We will also provide the code for reproduction upon acceptance.

Reviewer comment (Theory):
In the PRELIMINARY section, the problem is set as a zero-sum game, why in Eq. (3) the problem is for two policies to jointly maximize a single objective?

Author response:
A1: The problem setting we are tackling is that of population learning via self-play and policy gradient optimization. In this context, Eq. (3) is the result of (x and y) versus (x’ and y’), where (x’ and y’) are (x and y)’s opponent. As proposed by Liu et al 2022, NeuPL can make the two policies’ optimization more efficient by jointly optimizing them under a single conditional policy (conditioned on x and y’s id).

Reviewer comment (Theory):
Also in Eq. (3), why are two policies share the same parameter \theta_\tau? Shouldn’t they have their own different parameters?

Author response:
A2: The neural architecture of NeuPL is a single conditioned policy that jointly optimizes a population of agents. In Liu et al 2022’s paper, they showed that there is a Skill Transfer effect that makes agents’ learning more efficient. Our paper also finds the training would be more efficient, but at NeuPL’s convergence, it would not be optimal. Optimal here means there is still room for improvement for the individual agents to further improve.

Reviewer comment (Writing):
In the first line of PRELIMINARY, N={1,2,…,N} is bad notation.

Author response:
A3: We agree that this notation is confusing and we will change it to N={1,2,…,n}.

Reviewer comment (Writing):
Right after Eq. (1), the definition of Q(s,a) and V(s, id) are the same?

Author response:
A4: We agree that the definition ought to be different. Q(s,a) is the state-action value function, which is the expected return starting from state s and taking action a. V(s, id) is the state value function for agent id, which is the expected return starting from state s and following the policy of agent id. We will make this distinction more clear in the paper.

Reviewer comment (Writing):
In Eq. (6), what is the definition of T? What is the policy gradient integral over?

Author response:
A6: T is the total duration of training. Policy gradient integral over the change of parameter (over the training duration).

Reviewer comment (Writing):
Similarly, there seems to be a typo in Equation 10 (another key result). In addition, the terms in this equation have not been explained immediately following this equation, making it hard to understand the intuition.

Author response:
We apologize for the typo in Equation 10. The * has been replaced with a dot product symbol ($\centerdot$) in the revised paper. Equation 10 illustrates agent k's optimization of behavior to gain an advantage relative to a population of Generalists. The term $V(s,id^{k}) \centerdot \mathop{\mathbb{H}}(a^{g}, a^{k})$ represents the relation of agent k's expected value and the entropy of the joint action distribution of agent k and the Generalists. The term $Q(s,a^{k},id^{k}) \centerdot - \mathbf{\mathop{\mathbb{H}}(a^{g}, a^{k})$ represents the expected value of agent k's specific state and action, weighted by the negative entropy of the joint action distribution. We have added an explanation of these terms immediately after Equation 10 in the revised paper.

Reviewer comment (Writing):
Page 5, equation (9): Could you please give or recall the definition of $w(i;\dots;N+1)$?

Author response:
The weight of the Interaction Information term 'w' is defined as the expected weighting over a set of random variables. In the context of Multiagent Policy Gradient, this expected weighting would be the expected advantage function of (Q-V) for all agents jointly optimized under a single conditional neural net.

Reviewer comment (Writing):
Page 6, definition of $J$: Could you please clarify how $\psi^k$ in the left-hand side affects the right-hand side? Also, is the $*$ a typo and, if not, what is its meaning?

Author response:
The term '$\psi^k$' influences the right-hand side by altering the policy action distribution of '$a^k$', in terms of the action probability distribution entropy $\mathbb{H}$. The full equation is quite extensive, but its explicit form is shown in equation 15 as '$\pi_{\psi^k}(a^k|a^g,s,id^k) log(\pi_{\psi^k}(a^k|a^g,s,id^k)$'. The * in the right-hand side is a multiplication sign. We have changed it to $\centerdot$ as dot product in the paper revision to make it clearer.