Paper: Open-Ended Learning in General-Sum Games: The Role of Diversity in Correlated Equilibrium (ICLR-2024)

Reviewer comment (Writing):
In the diversity metric definitions (3) and (4), the relationship between the definition $\text{Div}(\cdot)$ and the function $f(\cdot)$ used later in the paper is unclear. It appears that the diversity metric $\text{Div}(\cdot)$ is intended to affect how new policies are introduced (i.e., to affect the best-response oracle picking the $\pi_i'$), whereas the rest of the paper concerns the meta-solver. This relationship is crucial because $f$ is used liberally in the rest of the paper.

Author response:
I apologize for any confusion in our explanation at this point. The meanings of $\text{Div}(⋅)$ and $\textbf{f}(⋅)$ are indeed consistent. The difference lies in the usage: $\text{Div}(⋅)$ is employed in the algorithm's pseudocode to represent the matrix formed by the diversities of all policies for all players, while $\textbf{f}(⋅)$ is used in the equilibrium definition, signifying that diversity is a function with policies as variables.

Reviewer comment (Writing):
"the isolation diversity term is meaningless" What do the authors mean by this? Could they define this precisely?

Author response:
As mentioned in the common response, some diversity definitions focus on the difference between an individual policy and all other policies within the population. Therefore, "the isolation diversity" refers to the scenario where we adhere to this definition and calculate diversity only for a specific policy within the population, without computing the diversity for all policies. This isolated diversity can only reflect the differences between the corresponding policy and all other policies in the population, failing to capture the overall diversity of the entire population. Hence, it is meaningless. We appreciate your suggestion, and we do acknowledge the need to provide an explanation for these points in the paper.

Reviewer comment (Experiments):
Why is NE not included as a baseline meta-strategy solver?

Author response:
For multi-player games, Nash Equilibrium (NE) is not unique, and its computational cost is substantial. Therefore, it is not suitable as a baseline meta-strategy solver, which is why JPSRO utilizes (C)CE instead of NE. However, in the case of 2-player games, the reviewer's suggestion is reasonable, and indeed, we should consider NE as a baseline meta-strategy solver for 2-player games.

Reviewer comment (Experiments):
How does the ablation study perform on the other games?

Author response:
We will supplement the ablation study performed on the other games in our paper.

Reviewer comment (Theory):
"Given that diversity has the ... and being computationally tractable." Diversity is not generally guaranteed to be tractable.

Author response:
To our knowledge, diversity is commonly defined in computationally tractable forms, such as vector norms, matrix norms, and distribution divergences.

Reviewer comment (Evaluation):
How do the methods compare if you're interested in finding an equilibrium as opposed to the maximum welfare equilibrium?

Author response:
In game theory, there is a widespread interest in equilibria that maximize the total payoff for all players, known as the maximum welfare equilibrium. This represents one of the fundamental questions in game theory research. Therefore, in our work, we have also chosen the maximum welfare equilibrium as an evaluation metric, as it provides an objective measure for assessing outcomes.

Reviewer comment (Novelty):
Some of the citations are confusing, for example Correlated Equilibrium is credited by two papers that came out after 2020.

Author response:
Thank you for your suggestion. We will correct this section accordingly.

Reviewer comment (Experiments):
The explanation of the metrics, equilibrium concept, and algorithm implies applicability to both correlated and coarse correlated equilibria, but the experiments use only CCE versions and measure the CCE gap. The reviewer asks whether the experiments can also be performed for CE.

Author response:
>The explanation of the metrics and equilibrium concept and algorithm implies that they can be used for both correlated as well as coarse correlated equilibria. However, the experiments seem to all use CCE versions and measure the CCE gap. Can the experiments be performed for CE as well?

The experiments be performed for CE, but the computational cost of CE is significantly higher than that of CCE (Correlated Equilibrium with Diversity). Due to time constraints, we did not present experimental results for CE in the paper. However, we will take your suggestion seriously and supplement this part of experimental results.

Reviewer comment (Writing):
The paper repeatedly states the goal of assigning higher sampling probabilities to "more diverse policies." However, it is unclear what it means for a policy to be "more diverse" than another, since diversity is typically defined for a set of policies, not an individual policy.

Author response:
>Throughout the text, it's stated that we want to assign higher sampling probabilities to "more diverse policies". But what does it mean for a policy to be "more diverse" than another? Isn't diversity defined for a set of policies, not for an individual policy?

Yes, diversity is usually defined for a set of policies. However, as I mentioned in the common response, some diversity measures define the difference between an individual policy and the other policies in the population. These definitions can be used to assess whether one policy is more diverse than another. Here, we want to convey that if a policy $\pi_a$ is more diverse than another policy $\pi_b$, then its difference from the entire population is larger than the difference of $\pi_b$ from the entire population.

Reviewer comment (Writing):
The (C)CE in Equation 8 is unclear. Specifically, when this refers to CCE, what is \(A_i\)?

Author response:
>I don't understand the (C)CE in Equation 8. When this is for CCE, what is $A_i$?

For CCE, the definition of A_i is given by $A_i(\pi^′_i , \pi) = u_i(\pi^′_i , \pi_{−i}) − u_i(\pi)$, and detailed information is provided in Section 2.2.

Reviewer comment (Theory):
Section 3.2 states that "calculating entropy can be challenging," but it is unclear why, since it should be \(O(|N| \times |\Pi|)\), where \(|\Pi|\) is the size of each player's population.

Author response:
>Section 3.2 says that "calculating entropy can be challenging" -- why is that? Isn't it just $O(|N|×|\Pi|)$ where $|\Pi|$ is the size of each player's population?

Sorry, we should be more rigorous here. What we want to express is that compared to Gini impurity, which is a quadratic function, the calculation of entropy involves a logarithmic function, which will become slower when frequent calculations are performed in large games.

Reviewer comment (Experiments):
First-price Sealed-bid Auction (FPSBA) is described as a multi-player game, and the paper is about multi-player games, but the experiment is only in a 2-player FPSBA.

Author response:
>First-price Sealed-bid Auction (FPSBA) is described as a multi-player game. This paper is about multi-player games. Then why is the experiment only in a 2-player FPSBA?

Thank you for your reminder. Indeed, we should provide the experimental results for 3-player FPSBA. Due to the increased complexity of 3-player FPSBA and the need for additional time resources, these results were not included in our paper. We will supplement the relevant experiments.

Reviewer comment (Experiments):
Blotto is a 2-player zero-sum game. The reviewer asks whether it can be compared to normal PSRO with a Nash Equilibrium meta-solver.

Author response:
>Blotto is a 2-player zero-sum game. Can we compare to normal PSRO with a Nash Equilibrium meta-solver?

Yes, your suggestion is valid. We should indeed compare with normal PSRO using a Nash Equilibrium meta-solver. At that time, our focus was primarily on meta-solvers suitable for multiplayer games, and we overlooked the normal Nash Equilibrium meta-solver.

Reviewer comment (Experiments):
Can you be more precise about the hyper-parameter tuning process of $k$? For example, when $k$ is very small (which it needs to be to guarantee uniqueness), the solution found is not optimizing for diversity but for maximizing the convex objective $\sigma^T \sigma$. It is unclear whether the success of low exploitability is really due to optimizing for diversity.

Author response:
Thank you for suggesting potential improvements to the presentation. We are glad that the reviewer considers the proposed method and the experiments here interesting.

>Can you be a bit more precise about the hyper-parameter tuning process of $\kappa$? For example when it is very small (which you need to be small enough to guarantee uniqueness) then the solution you find is not optimizing for diversity but for maximizing the convex objective $\sigma^\top\sigma$. So it is not clear to me whether the success of low exploitability is really due to optimizing for diversity.

The optimization objective of MG(C)CE is $\sigma^\top\sigma$, and it can be viewed as a scenario of D(C)CE when $\kappa$ is very small. The experimental results of MGCCE have been demonstrated in Figure 1, providing a comparison with the results of DCCE.

Reviewer comment (Experiments):
In terms of apples-to-apples comparisons to other techniques, did you try to examine what would happen if a similar fine-tuning process was explored for the other competing techniques?

Author response:
Thank you for suggesting potential improvements to the presentation. We are glad that the reviewer considers the proposed method and the experiments here interesting.

>Also in terms of an apple to apple comparisons to other techniques did you try to examine what would happen if a similar fine-tuning was explored for the other competing techniques?

To the best of our knowledge, there is currently no technology similar to ours. That is, no one has employed the diversity measure to define an equilibrium in a manner similar to our approach. The baselines we use also do not involve such operations. Therefore, it is challenging to conduct similar fine-tuning for other technologies at this point.