Paper: Welfare Diplomacy: Benchmarking Language Model Cooperation (ICLR-2024)

Reviewer comment (Theory):
It is challenging to attain Pareto-efficient equilibria, and how to achieve optimal Nash welfare remains unclear.

Author response:
“Pareto-efficient equilibria are often not stable, and there may be various factors that can lead to deviations from the equilibrium, such as imperfect information, externalities, or strategic behavior. These deviations can disrupt the equilibrium and lead to a new outcome that is not Pareto-efficient.”, “It is challenging to attain Pareto-efficient equilibria, and how to achieve optimal Nash welfare remains unclear.”, “Despite the existence of multiple Pareto-efficient Nash Equilibria (NEs), they often display instability, particularly in complex or realistic scenarios. How can we effectively tackle this challenge?” → We fully agree with the remarks about the instability of Pareto optimal NE, and the difficulty of attaining such profiles. There are a few things to note about this:
    1.  One hope is that having benchmarks like ours will facilitate research that makes it more likely that we can construct agents that do play Pareto optimal Nash equilibria.
    2.  We believe there is a need for both more robust solution concepts (e.g., performance against diverse distributions of agents) and the development of environments that are more realistic among the dimensions you mention (e.g., exhibiting noise and private information). We’ve added some remarks about this to our Conclusion, in red.
    3.  Pareto-optimal Nash equilibrium is a standard operationalization of rational cooperation, and so this is a limitation shared by much of the literature on game theory and multi-agent systems.

Reviewer comment (Writing):
As this paper aims to enhance societal safety by aiding researchers in the development and evaluation of multi-agent AI systems, could you please provide examples that illustrate the potential benefits of using benchmarks in real-world scenarios?

Author response:
“As this paper aims to enhance societal safety by aiding researchers in the development and evaluation of multi-agent AI systems, could you please provide examples that illustrate the potential benefits of using benchmarks in real-world scenarios?” → It is true that there are often gaps between benchmarks and real-world applications (e.g., https://aclanthology.org/2021.acl-long.81/). At the same time, it is difficult to improve systems if we do not know what to measure. We aim to push forward the work of measuring cooperative capabilities; we do not think that our Welfare Diplomacy benchmark in itself will inform a particular real-world application. That said, here are some examples of benchmarks that inform real-world scenarios: Claude 2’s system card includes evaluations on benchmarks (e.g., BBQA) which informed deployment (https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf); Anthropic’s “responsible scaling policy” requires evaluations for safety (https://www.anthropic.com/index/anthropics-responsible-scaling-policy); OpenAI spent six months after the training of GPT-4 to assess safety concerns, partially informed by benchmarks (https://cdn.openai.com/papers/gpt-4-system-card.pdf).

Reviewer comment (Theory):
Despite the existence of multiple Pareto-efficient Nash Equilibria (NEs), they often display instability, particularly in complex or realistic scenarios. How can we effectively tackle this challenge?

Author response:
“Pareto-efficient equilibria are often not stable, and there may be various factors that can lead to deviations from the equilibrium, such as imperfect information, externalities, or strategic behavior. These deviations can disrupt the equilibrium and lead to a new outcome that is not Pareto-efficient.”, “It is challenging to attain Pareto-efficient equilibria, and how to achieve optimal Nash welfare remains unclear.”, “Despite the existence of multiple Pareto-efficient Nash Equilibria (NEs), they often display instability, particularly in complex or realistic scenarios. How can we effectively tackle this challenge?” → We fully agree with the remarks about the instability of Pareto optimal NE, and the difficulty of attaining such profiles. There are a few things to note about this:
    1.  One hope is that having benchmarks like ours will facilitate research that makes it more likely that we can construct agents that do play Pareto optimal Nash equilibria.
    2.  We believe there is a need for both more robust solution concepts (e.g., performance against diverse distributions of agents) and the development of environments that are more realistic among the dimensions you mention (e.g., exhibiting noise and private information). We’ve added some remarks about this to our Conclusion, in red.
    3.  Pareto-optimal Nash equilibrium is a standard operationalization of rational cooperation, and so this is a limitation shared by much of the literature on game theory and multi-agent systems.

Reviewer comment (Theory):
Some arguments regarding the motivations of welfare diplomacy lack rigor and may be questionable. It has been repeatedly claimed in the paper that "While Standard Diplomacy (SD) has features that make it interesting as an environment for cooperative AI research, it is zero-sum and incentivizes the development of cooperation-undermining capabilities" and `In contrast to SD, WD is general-sum'. However, it has been pointed out in [1] that "In Diplomacy, seven players... coordinate their actions to both cooperate and compete with each other," suggesting that standard diplomacy is not necessarily a zero-sum game. If standard diplomacy were indeed zero-sum, cooperation would not be involved, similar to chess and heads-up poker.

Author response:
“suggesting that standard diplomacy is not necessarily a zero-sum game. If standard diplomacy were indeed zero-sum, cooperation would not be involved, similar to chess and heads-up poker.” → Yes, it is true that Standard Diplomacy also involves cooperation (between subsets of players). When we say that it is a "zero-sum game", we mean this in the technical sense: The sum of players' scores is the same for any policy profile (because a player's utility is the indicator of whether they won). Although SD can involve cooperation, the fact that there is a single winner means that it has a few limitations as a benchmark for cooperation. In particular, it does not exhibit opportunities for (global) rational cooperation (where “global” means between all players) and skilled play is not differentially (globally) cooperative, as we discuss in Section 2.1.

Reviewer comment (Theory):
The theoretical results are hard to interpret. It would be helpful to clarify the meaning of $\pi^k$ as a NE and how Theorem 1 relates to the main claim.

Author response:
“The theoretical results are hard to interpret. It would be helpful to clarify the meaning of  as a NE and how Theorem 1 relates to the main claim.” → We agree that the relevance of Theorem 1 to our main claims should be clarified. We have now edited the paragraph directly after Theorem 1 (see red text), explicitly pointing out the connection to our criterion (A), that there should exist opportunities for global, rational cooperation. In particular, Theorem 1 shows that, in the toy environment, there are Nash equilibria that Pareto-dominate others, and that these Pareto-dominating equilibria require more cooperative capabilities. We also already discussed the connection to Theorem 1 in Section 2.2; we have now added a forward reference to Theorem 1 to make this clear.

Reviewer comment (Evaluation):
The metric "basic proficiency" is defined as the mean of "the rate of model outputs that are valid JSON and thus able to be parsed without error, the rate of submitted orders that are valid possible orders, and the fraction of global SCs owned by any player and not left neutral." While the first two components are understandable, it is unclear why the fraction of global SCs should be considered as an aspect of 'basic proficiency'. This seems more like a metric about social welfare.

Author response:
“I don't understand why the fraction of global SCs should be considered as an aspect of 'basic proficiency'. To me, it is more like a metric about social welfare.” → This metric is about using all the available resources on the board. It’s true that some partitions of the board might involve agreements to leave certain SCs neutral, but in practice, we observed that less capable or worse-prompted models would just not try much to capture free SCs. We wanted a component of the metric to check for this possibility.

Reviewer comment (Experiments):
The author can try to include more experiment results and ablation studies such as prompt sensitivity, hyperparameter effects, etc.

Author response:
“more experiment results and ablation studies such as prompt sensitivity, hyperparameter effects, etc.” → See Appendices B, C. We have also added a small prompt ablation experiment on GPT-4 in Figure 8 of section C.5 in addition to the existing larger prompt ablation on Claude 1.2 in Figure 7.

Reviewer comment (Experiments):
The author should try to incorporate human-LLM mixed experiments to see how human engagement can influence LLM performance.

Author response:
“human-LLM mixed experiments to see how human engagement can influence LLM performance” → The suggestion is interesting! We are excited about pursuing this direction in future work; for this work, we were financially constrained and had to narrow our focus.

Reviewer comment (Evaluation):
Some human analysis of LLM's policy should be conducted to better understand LLM's performance.

Author response:
“Some human analysis of LLM's policy” → We have already included such analysis in Appendix D, E. We have now also added additional analysis of WDAgent(M) for each of the models M we looked at in Appendix C.4.

Reviewer comment (Writing):
There exist several typos in the paper, e.g. in the first line of section 3 certain NEs certain NEs.

Author response:
“There exist several typos in the paper, e.g. in the first line of section 3 certain NEs certain NEs.” → Thank you for pointing that out! We went through it once more and corrected typos.

Reviewer comment (Theory):
Can the authors elaborate more on how the theoretical analysis simplifies the real WD game in section 3.2.1 and theorem 1 and what is the gap between the theoretical analysis and the real WD game?

Author response:
“Can the authors elaborate more on how the theoretical analysis simplifies the real WD game in section 3.2.1 and theorem 1 and what is the gap between the theoretical analysis and the real WD game?” → We added a description of how the toy version of WD differs from the real version (see new text at the beginning of Section G in red). In short, our toy version has fewer states and more symmetries between players - which makes it much more tractable to analyze - and is centered around a bargaining problem that captures the core challenge of WD.

Reviewer comment (Writing):
I recommend the author slightly modify the title as the cooperation discussed in the paper is the cooperation in a general-sum game. It can help distinguish itself from the fully cooperative setting.

Author response:
“I recommend the author slightly modify the title” → To our knowledge, 'cooperation' is commonly understood as applying to both purely cooperative and mixed-motive games. For example, this is how it is meant in 'cooperative AI' (https://arxiv.org/abs/2012.08630), a subfield we consider our paper to be a part of. In combination with the fact that making the title more precise might require making it too long, we currently think it's best to stick with the present title.

Reviewer comment (Evaluation):
It seems most models, except advanced LLMs like GPT-4, cannot have policies that are significantly better than the random baseline, what could be the reasons?

Author response:
“It seems most models, except advanced LLMs like GPT-4, cannot have policies that are significantly better than the random baseline, what could be the reasons?” → Thank you for prompting us to look at this more closely. As part of the newly added Appendix C.4 (in red), we have provided commentary on some of the qualitative features of LLM policies that determine their performance. (Briefly: Claude-1.2 gets very low root mean Nash welfare because in several games, some players hold their units throughout the game and therefore get 0 Welfare Points. GPT-3.5 and Llama-2 each tend not to acquire many more SCs, but do sometimes disband units and therefore acquire some WPs.)