Paper: Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game (ICLR-2024)

Reviewer comment (Novelty):
The approach mainly combines prompt engineering with reinforcement learning (RL), specifically tailored for the Werewolf game. It's unclear how this would inspire or be applicable to other tasks.

Author response:
**W1: The approach mainly combines prompt engineering with RL, specifically tailored for the Werewolf game. It's unclear how this would inspire or be applicable to other tasks.**

Our approach has three components: (1) deductive reasoning, (2) diverse action generation, and (3) population-based RL training. We believe our framework of using these three components is general enough to be adapted to other social deduction games. The main changes would be specific prompt designs but the general framework stays the same.
Specifically, while some prompts used in the first component are carefully designed for the Werewolf game, the prompts for generating diverse actions, and the population-based reinforcement learning method in the second and third components are general to be applied in other tasks like social deduction games (e.g., *The Resistance: Avalon* [1]) and board games with negotiations (e.g., *Diplomacy* [2]). 

Take *The Resistance: Avalon* as an example. To apply our method in this game, we need to specialize the deduction template in the first component and maybe use additional prompting techniques like [1]. We also need to change the prompt in the third component to generate the agent pool for this game. Then our method can be applied without changing the diverse action generation and RL training components.
We have added a discussion on our method's application to other tasks in Appendix H.1 of our revised paper.

[1] Wang, Shenzhi, et al. "Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation." arXiv preprint arXiv:2310.01320 (2023).

[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

Reviewer comment (Novelty):
The paper does not clearly justify the need for using RL for action selection. There are alternative methods, such as in-context learning. What is the added benefit of the extra training cost incurred by using RL?

Author response:
**W2: The paper does not clearly justify the need for using RL for action selection. What is the added benefit of the extra training cost incurred by using RL?**

The motivation for using RL is that LLM alone cannot produce the optimal action distribution and is likely to be exploited by adversarial opponents in zero-sum games. By contrast, RL can learn the optimal action distribution and produce less exploitable policies. To intuitively show the benefit of using RL, we compare the action distributions of agents with and without RL policy in three cases and the discussion can be found in Section 5.4 of our revised paper. 

Consider the case when it is the first night and the Werewolves need to decide who to kill. The agent without RL policy has a high probability to choose player_0 (0.38) and a low probability to choose player_5 (0.01). An adversarial Doctor can take advantage of this pattern by always saving player_0 and achieve a 0.38 success rate. In comparison, our agent with RL policy produces a uniform distribution and no Doctor can achieve a success rate higher than 0.14. Please see Section 5.4 and Figure 4 in our revised paper for a detailed analysis on two more cases.

Reviewer comment (Experiments):
The paper lacks explanations on how credit assignment is handled in a multi-agent setting. Additionally, it does not specify the reward structure. The impact of the hyperparameter 'N', which represents the number of generated actions, on the results is also not discussed.

Author response:
**W3: The paper lacks explanations on how credit assignment is handled in a multi-agent setting. Additionally, it does not specify the reward structure. The impact of the hyperparameter 'N', which represents the number of generated actions, on the results is also not discussed.**

We have added descriptions of reward design and credit assignment in Appendix D.2 of our revised paper. 
* Winning reward: all winners +100, all losers -100.
* Werewolf killing reward: if the Werewolves successfully kill a player at night, the Werewolves +5, the Villagers (Seer, Doctor, Villagers) -5.
* Seer seeing reward: if the Seer successfully identifies a Werewolf at night, the Werewolves -2, the Seer +2.
* Doctor saving reward: if the Doctor successfully saves a player at night, the Werewolves -5, the Doctor +5.
* Voting result reward: 
    * If a Werewolf is voted out, the Werewolves -5, the Villagers (Seer, Doctor, Villagers) -5.
    * If a non-Werewolf is voted out, the Werewolves +5, the Villagers -5.
* Individual voting reward: 
    * If the current player votes for a Werewolf, the Werewolves -1, the current player +1.
    * If the current player votes for a non-Werewolf, the Werewolves +1, the current player -1.
    * If the current player chooses not to vote, no additional reward for any player.

We also added an ablation on hyperparameter N = 2, 3, 4, 5 in Appendix E.6. The table below shows that increasing N will produce more actions, but the diversity increased by newly added action is small (distance < 0.2) when N = 4, 5. This is because sometimes the number of possible actions is small. For example, when there are only 2 Werewolves and 3 other players alive (a common situation on the second night), the Werewolves only have 3 possible actions and N = 4, 5 will not introduce diversity in the action candidates.

|          N          |  2   |  3   |  4   |  5   |
|:-------------------:|:----:|:----:|:----:|:----:|
| increased diversity | 0.35 | 0.37 | 0.23 | 0.16 |

Reviewer comment (Novelty):
The motivational claims about the advantages of this benchmark are tenuous. "Prior work on LLM-based ... grounded in credible information ... to make wrong decisions." This claim isn't supported, and/or is making a weaker statement about the fragility of all single-agent RL not just LLMs.

Author response:
We would like to clarify that our work focuses on LLM-based agents and aims to improve their strategic reasoning ability. Comparing to existing environments for LLM-based agents, the Werewolf game has the following two challenges.

1. decision-making with deceptive information. We didn't cite papers to support our claim on "Prior work on LLM-based agents …" because there is little work that evaluates these agents in deceptive environments by the time of our submission. This claim is validated by our own results in Figure 3 and Table 1 where our agent with deductive reasoning achieves stronger results than the vanilla LLM-based agent. We have also added [1] (which was published after our submission) to support the claim in our revised paper.
2. strategic play in mixed cooperative-competitive games. Most works on LLM-based agents focus on single-agent or cooperative tasks. The mixed cooperative-competitive property of Werewolf is a key difference from existing environments like ALFWorld. 

For the claim that "it is impossible to achieve strong play in Werewolf without communication", we would like to emphasize that communication is critical in Werewolf to share information and deduce hidden roles. Without communication, there is no way for the Villagers to know who is the Werewolf. To validate this claim, we added agents w.o. communication as baselines and the table below shows that no communication greatly hurts the performance. The result has been added in Appendix E.4 of our revised paper.

|       Win Rate       |  Vanilla | Concurrent |  Atomic  |   Ours   |
|:--------------------:|:--------:|:----------:|:--------:|:--------:|
| **w. communication** | **0.16** |  **0.23**  | **0.26** | **0.30** |
|  w.o. communication  |   0.05   |    0.04    |   0.05   |   0.06   |

[1] Wang, Shenzhi, et al. "Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation." arXiv preprint arXiv:2310.01320 (2023).

Reviewer comment (Experiments):
A claim is made that it is impossible to achieve strong play in Werewolf without communication. This claim could be evaluated by playing with non-language/simple policies, which would additionally build out a set of baselines. There may be a non-communication equilibrium that works OK (à la Hanabi conventions).

Author response:
The agents without communication are not trained. We use the same agents as described in Section 5.1 and set their statements to be the empty string. Please note that the "Vanilla" agent and "Concurrent" agent are LLM-based agents and do not require training, only the "Atomic" agent and our agent can be trained.

We agree with the reviewer that conventions can make communication unnecessary in some environments like Hanabi and Avalon, but in the Werewolf game, communication is indispensable because it is the only way to share information, especially before the first voting phase. Even if the agents have established a prior convention, they still need to take public action to convey the information. Without communication, the agents will have no opportunity to take any public actions before the first voting phase and lose a crucial source of information. In this case, convention is of no use and the best a Villager can do is to randomly vote for a player or do not vote.

Reviewer comment (Experiments):
Hidden role games are analogous to ad hoc teamwork and opponent-policy belief/likelihood modeling, and these are not discussed nor used as potential baselines.

Author response:
We agree that hidden role games are related to ad hoc teamwork in general, and our claim that "once the teammates are correctly identified, collaboration with them is relatively easy" should be limited to the Werewolf game. To validate this claim in the Werewolf game, we let the Villagers know the identity of their teammates, and let the "Vanilla" agent cooperate with the four different agents in Section 5.1. All combinations achieve 100% win rates. This result shows that even the simplest "Vanilla" agents can cooperate with different teammates as long as they know who are their teammates in the Werewolf game.

Reviewer comment (Theory):
All problems/concepts of diversity are punted to just asking the LLM to be diverse, with no guarantees of diversity or notions of what kind of diversity.

Author response:
The objective of our diverse action generation component is to generate a set of strategically different actions. Although there are many ways to define what is diversity, they may not align with human's intuition and produce undesired behaviors. In fact, the best way to decide if the actions are diverse is to use human evaluation. In our method, LLM is used as a human proxy to decide what actions are diverse in the Werewolf game and we find simply asking the LLM to "propose strategically different actions" is effective and produces desired results. We have added two qualitative examples in Appendix C.6 of our revised paper.

To quantitatively validate this claim, given an action set $A$, we can define the increased diversity introduced by a new action $a$ as $\mathrm{div}(a, A) = \mathrm{min}_{a'\in A} \|e(a) - e(a')|$, i.e., the minimum Euclidean distance between its embedding and existing actions' embeddings. A larger value means that the new action $a$ is different from any actions in the set $A$. As a reference, the distance between “kill player_i” and “kill player_j” is about 0.36 when $i \neq j$. We consider $N = 2, 3, 4, 5$ and evaluated the increased diversity introduced by the last action. As shown in the table below, our method produces actions that are different from existing actions. The increased diversity is smaller when $N = 4, 5$ because in some cases the number of available actions is smaller than 4 or 5. We use $N = 3$ in our implementation and this result has been added in Appendix E.6 of our revised paper.
|          N          |  2   |  3   |  4   |  5   |
|:-------------------:|:----:|:----:|:----:|:----:|
| increased diversity | 0.35 | 0.37 | 0.23 | 0.16 |

Reviewer comment (Experiments):
The SelfPlay algorithm isn't well described and takes many changes from existing algorithms without analyzing their impact.

Author response:
We have added a section on the implementation details of population-based training in Appendix D.3 of our revised paper. We also added an ablation to compare our agent with a self-play agent without using the agent pool (trained with itself and its checkpoints). The table below shows that our agent achieves higher win rates. In general, adding more agents with different styles to the agent pool will make the final agent more robust. This result has been added in Appendix E.2 of our revised paper.

|      Win Rate     | Self-play | PBT (ours) |
|:-----------------:|:---------:|:----------:|
|  as the Villagers |    0.25   |  **0.30**  |
| as the Werewolves |    0.68   |  **0.70**  |

Reviewer comment (Experiments):
Particularly in the SelfPlay algorithm, the population is seeded with a pool of policies biased with prompts based on "predefined personalities". It would be good to understand what role this population, and subset(s), plays in the success of the algorithm.

Author response:
Due to the limited time for rebuttal, only one self-play run has been completed and the other two self-play runs are still under training. We will update the error bar as soon as the training is completed.

To support the claim that "In general, adding more agents with different styles to the agent pool will make the final agent more robust", we would like to add some further discussions and references. [1] investigates the geometrical properties of over 20 real-world games like Tic-Tac-Toe, Go, StarCraft II and finds that a diverse population of opponents with different play styles is important for improvement in policy strength. Other work like [2, 3] also shows that introducing diversity to the population of PSRO will improve the final performance. Another example is the famous work of AlphaStar [4], which also adds policies with different styles to its population for optimal performance. Our method draws inspiration from these works and we believe the same principle applies to the Werewolf game.

[1] Czarnecki, Wojciech M., et al. "Real world games look like spinning tops." Advances in Neural Information Processing Systems 33 (2020): 17443-17454.

[2] Balduzzi, David, et al. "Open-ended learning in symmetric zero-sum games." International Conference on Machine Learning. PMLR, 2019.

[3] Liu, Xiangyu, et al. "Towards unifying behavioral and response diversity for open-ended learning in zero-sum games." Advances in Neural Information Processing Systems 34 (2021): 941-952.

[4] Vinyals, Oriol, et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." Nature 575.7782 (2019): 350-354.

Reviewer comment (Experiments):
Are all of the four attributes (reasoning, role, confidence, and reference) generated by the deduction LLM necessary? Is there any data on ablations of this information?

Author response:
We added an ablation on the four attributes by gradually adding role, confidence, evidence, and reasoning and comparing their win rates against our agents with all four attributes. The table below shows that all attributes improve the performance and are necessary. This result has been added in Appendix E.3 of our revised paper.

|      Win Rate     | role | + confidence (c) | + c + evidence (e) | + c + e + reasoning |
|:-----------------:|:----:|:----------------:|:------------------:|:-------------------:|
|  as the Villagers | 0.18 |       0.20       |        0.21        |       **0.23**      |
| as the Werewolves | 0.55 |       0.56       |        0.58        |       **0.61**      |

Reviewer comment (Theory):
Why are reliability and confidence separate and somehow being treated as additive/substitutive? This feels a bit awkward and unintuitive.

Author response:
We would like to clarify that reliability is determined by confidence.
* If a player is deduced to be a Werewolf, then their reliability = 11 - confidence. In other words, if a player is deduced to be a Werewolf with high confidence, then we should not trust their statement and their reliability should be low.
* If a player is deduced to be a Seer, Doctor, or Villager, then their reliability = confidence. That is, if we think a player is likely to be a non-Werewolf, then we should trust their statements.

Reviewer comment (Novelty):
At this level of agent complexity in LLM work, we're essentially at a cognitive architecture with short-term and long-term memory. This is worth considering in implementations, baselines, and related work.

Author response:
We agree with the reviewer that there are potential ways to further improve the performance of our agent by using better prompting techniques or agent architecture. However, the main focus of this work is to combine LLM with RL to build strategic language agents, and we pay less attention to prompt design and agent architecture. Moreover, in the Werewolf games, the whole game history can fit in the LLM's context window (4k) most of the time, so we defer the Werewolf-specific short-term and long-term memory design to future work.

Reviewer comment (Experiments):
Why is self-attention used on the action embeddings? A much more natural approach is just to learn a Q value function.

Author response:
The key goal of the population-based RL training component is to learn a policy to select the best action from the candidates generated by the diverse action generation component. Using a self-attention policy or a Q value function are implementation choices, both ways are possible. We did try to learn a Q function in our early attempt but found the performance improvement was not significant. This may be because value-based methods could have unsatisfying results in multi-agent environments with diverse behaviors [5]. Therefore, we turned to the current self-attention implementation and used PPO to train the policy.

[5] Fu, Wei, et al. "Revisiting some common practices in cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2206.07505 (2022).

Reviewer comment (Theory):
Using a Q value function would be more flexible because it could consider infinite candidate actions.

Author response:
The key goal of the population-based RL training component is to learn a policy to select the best action from the candidates generated by the diverse action generation component. Using a self-attention policy or a Q value function are implementation choices, both ways are possible. We did try to learn a Q function in our early attempt but found the performance improvement was not significant. This may be because value-based methods could have unsatisfying results in multi-agent environments with diverse behaviors [5]. Therefore, we turned to the current self-attention implementation and used PPO to train the policy.

[5] Fu, Wei, et al. "Revisiting some common practices in cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2206.07505 (2022).

Reviewer comment (Presentation):
Please include error bars in the figures for Exp 5.1.

Author response:
We have updated Figure 3 to report both mean and std. For LLM-based agents like "Vanilla" and "Concurrent", the randomness only comes from the LLM. For methods with RL policy like "Atomic" and "Ours", we added two more runs with different seeds and the randomness comes from both the LLM and the RL policy.

Reviewer comment (Evaluation):
Cross play is a coarse performance measurement, especially in a team game where each player plays the same policy across the team.

Author response:
We agree with the reviewer that regret or exploitability would be a better metric in theory. However, in practice, computing regret or exploitability requires finding the best response to a fixed policy, and it is impossible to get the exact best response in complex n-player games like Werewolf. Instead, cross-play and human evaluation are commonly used to evaluate performance [1, 2].

The agent within the same team uses the same method. For example, the first row means the Villagers are all Vanilla agents, and the first column means the Werewolves are all Vanilla agents.

[1] Jaderberg, Max, et al. "Human-level performance in 3D multiplayer games with population-based reinforcement learning." Science 364.6443 (2019): 859-865.

[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

Reviewer comment (Evaluation):
A stronger metric than cross play would be regret, finding the worst performing case for your method across all possible agent types.

Author response:
We agree with the reviewer that it is possible to compute the approximate estimates of regret or exploitability by training an approximate best response (BR). However, it is still impossible to know how precise the approximate BR is, and comparing approximate results with no precision guarantee is not a rigorous way for evaluation. In other words, we should compute the $\varepsilon$-approximate BRs of different agents under the same $\varepsilon$, but the value of $\varepsilon$ is generally unknown or not guaranteed in practice. Some games like poker may have specialized methods to estimate the approximate regret with guarantee in $\varepsilon$, but is unknown how to guarantee a $\varepsilon$-approximate BR in more general games like the Werewolf game. Therefore, we do not use approximate regret or exploitability in our evaluation.

Reviewer comment (Experiments):
Was an experiment done with heterogeneous teams in Exp 5.1?

Author response:
The agent within the same team uses the same method. For example, the first row means the Villagers are all Vanilla agents, and the first column means the Werewolves are all Vanilla agents.

Reviewer comment (Evaluation):
In Exp 5.2, showing the first game performance and the cumulative game performance as separate metrics would be insightful. Performance as a function of number of previously played games would be even better.

Author response:
We have added results on the win rate in each round in Appendix E.8 of our revised paper. As shown in Table , the win rate of human players gradually increases as they played more games against other agents, while stay relatively unchanged against our agent. This shows our agent is more robust to adversarial opponents.

Reviewer comment (Experiments):
In Exp 5.2, how do you separate 80 people into the 16 evaluation categories? You mention dividing people into 4 groups of 20; how are they distributed into the categories?

Author response:
As we mentioned in the paper, each participant played with 6 AI agents of the same type. We divided the 80 participants into 4 groups, each group corresponding to one AI agent type. In addition, participants in each group are divided into two roles, the Villagers or the Werewolves. So effectively the 80 participants are divided into 8 categories. 
Human players are divided randomly and each category has 10 players. The other 8 values in Table 1 is produced by replacing the human player with our agent and playing agaisnt the same other AI agents. These data are used to compare the performance of human players and our agents.

Reviewer comment (Experiments):
Did you try running the experiment just without diversity?

Author response:
For agent just w.o. diversity, the LLM only produces one action, and there is no need to use the RL policy to select from the action set. Therefore it is the same as the agent w.o. training and diversity.

Reviewer comment (Evaluation):
In Exp 5.3, include error bars, as it's hard to tell if there is meaningful improvement without them.

Author response:
we have updated Table 2 to report both mean and std.

Reviewer comment (Reproducibility):
More implementation details are needed for the multi-agent RL algorithm to fully assess and reproduce the approach.

Author response:
**W1: More implementation details are needed for the multi-agent RL algorithm.**

We have added a section on the implementation details in Appendix D of our revised paper. The section includes detailed discussions on the attention policy architecture, environment reward, MARL hyperparameters, and population-based training setup. We are willing to provide more explanations if any details are not clear.

Reviewer comment (Experiments):
Additional rigorous evaluations of the MARL method would strengthen the results, such as multiple training runs and assessing cross-population transfer.

Author response:
**W2: Additional evaluation of the MARL method would strengthen the results, such as multiple training runs and assessing cross-population transfer.**

We have added two more runs to train the RL policy and report the mean and std in Figure 3 of our revised paper. The result remains the same: our agent achieves the best performance. For cross-population transfer, we are not sure about the meaning of this evaluation. We are happy to add this evaluation if you could provide more explanation.

Reviewer comment (Experiments):
What are the generalization settings that the models are tested for? Can the model generalize to new forms of the game? What are the limits? Is the train and test distribution the same?

Author response:
**W4: What are the generalization settings that the models are tested for? Can the model generalize to new forms of the game? What are the limits?**

The methods are tested for generalization to unknown teammates and opponents. During training, our agent does not know the teammates or opponents in test time and only plays with its past version and agents in the predefined agent pool. In test time, our agent is evaluated with players that are not seen in training like human players and other baselines.

For generalization to new forms of the Werewolf game, the first two components in our method (deductive reasoning and diverse action generation) can be directly generalized to new settings like changing the number of players, adding new roles, and changing the winning conditions, with slight changes in the prompt. 

The RL training method can also be directly used in different settings, but the policy needs to be retrained for a new setting to achieve the best performance. We expect an RL policy trained under one specific game setting to generalize to similar settings like adding or removing one player, but we do not expect it to generalize to games with significant changes like changing the winning conditions. To evaluate the transferability of a trained RL policy to slightly different game settings, we considered a 6-player and an 8-player Werewolf game and compared agents with and without the RL policy trained on the 7-player game. The table below shows the RL policy can generalize to the 6-player and 8-player settings and improve the performance. We have added the results and discussions in Appendix H.2 of our revised paper.

|        Win Rate        | 6-player | 8-player |
|:----------------------:|:--------:|:--------:|
|  w.o. 7-player policy  |   0.18   |   0.27   |
| **w. 7-player policy** | **0.23** | **0.30** |

Reviewer comment (Experiments):
Can the LLM itself be used as a reward model as in [1, 2] to choose actions?

Author response:
**W5: Can the LLM itself be used as a reward model as in [1, 2] to choose actions?**

The two mentioned works use LLMs in very different ways and we will discuss them separately. [1] does not use LLM as a reward model. Instead, it directly uses LLM as an agent and proposes several prompting techniques to improve its strategic reasoning ability. Their method requires recursive searches over the finite game tree and is not applicable in our setting because the action space in Werewolf is prohibitively large if not infinite (the agent can say anything in discussion).

[2] uses LLM as a reward model to solve the reward design problem when the reward is hard to specify. The example in their paper is to generate a reward function for a "versatile" negotiator. However, the reward design in our setting is clear and simple: the winners get +100 reward and the losers get -100 reward. We do use simple reward shaping as described in Appendix D.2, but the designs are straightforward and there is no need or advantage to use LLM for reward design.

[1] Gandhi, Kanishk, Dorsa Sadigh, and Noah D. Goodman. "Strategic Reasoning with Language Models." arXiv preprint arXiv:2305.19165 (2023).

[2] Kwon, Minae, et al. "Reward design with language models." arXiv preprint arXiv:2303.00001 (2023).

Reviewer comment (Novelty):
Analogously a discussion on an alternative method where an LLM is used as a reward model to train a separate agent is needed, what are the advantages of an LLM agent?

Author response:
**W6: Analogously a discussion on an alternative method where an LLM is used as a reward model to train a separate agent is needed, what are the advantages of an LLM agent?**

First, the pretrained LLMs can be directly used in the agent for natural language understanding and communication, which is critical in the Werewolf game. By contrast, training a separate agent from scratch for natural language communication would require a huge amount of data, even when an LLM is used as a reward model.

Second, as discussed in the previous question, the reward in Werewolf is simple and clear. There is no need to use LLM for reward design.

Reviewer comment (Novelty):
The similarities to prior work like Cicero diminish the novelty claims. The key difference seems to be using language for reward estimation.

Author response:
**W7: The similarities to prior work like Cicero diminish the novelty claims. The key difference seems to be using language for reward estimation.**

We would like to clarify that our method is very different from Cicero. 
* Cicero first uses the RL policy to choose from the game-predefined action set and then uses the selected actions as input for LLM to generate action-conditioned dialogue.
* Our method first uses LLM to propose diverse action candidates and then uses the RL policy to select from these actions.

The difference comes from the fact that Diplomacy is a board game with predefined actions. By contrast, the actions in the discussion phase of Werewolf can be any natural language and can not be predefined. 

We also implemented the "Atomic" baseline which is inspired by Cicero and compared it with our agent. Results in Figure 3 show that our agent achieves better performance.

Reviewer comment (Evaluation):
Providing quantitative results for emergent behaviors would substantiate that the examples shown are representative and not cherry-picked.

Author response:
**W10: Quantitive results for emergent behaviors.**

We have added results on the average occurrence of four emergent behaviors and compared our agent with three other agents. The result in Figure 7 of our revised paper shows that our agent produces complex behaviors like bluffing and sacrificing more often than other agents.

Reviewer comment (Evaluation):
I would love to see more of an analysis of the failure cases.

Author response:
**W11: Analysis of the failure cases.**

One failure case is Werewolf agents could unintentionally reveal their true identity in discussion. This is because the Werewolf agents need to first reason as a Werewolf and then pretend to be a non-Werewolf in discussion. This mismatch in their thoughts and words could make the LLM get confused and accidentally speak out their thought as a Werewolf. Fortunately, the probability of such a failure case is significantly reduced by generating multiple diverse actions and using an RL policy to choose the best one. Please see Appendix H.3 in our revised paper for a detailed analysis of more failure cases.

Reviewer comment (Experiments):
The experimental scope of this paper is limited by a limited number of tests and a narrow selection of baseline comparisons, which may not verify that the results are better than other current prompt engineering-based methods.

Author response:
**W1: A limited number of tests and a narrow selection of baseline comparisons.**

To the best of our knowledge, there is no prompting technique specialized for social deduction games like Werewolf other than the included concurrent work [1] by the time of our submission. Therefore, we used a vanilla LLM agent and the concurrent work as our baselines.

For comparison with general prompting techniques, we added CoT [2], ReAct [3] as our baselines and compared them with our agent using deductive reasoning only (no diverse action generation and RL training). The table below shows our prompting design achieves the highest win rates both as the Villagers and as the Werewolves. This result has been added in Appendix E.1 in our revised paper.

|      Win Rate     | ReAct |  CoT | Ours with deductive reasoning only |
|:-----------------:|:-----:|:----:|:----------------------------------:|
|  as the Villagers |  0.16 | 0.15 |              **0.23**              |
| as the Werewolves |  0.54 | 0.56 |              **0.61**              |


In addition, we would like to emphasize that prompt design is just one of our three components. The second component (diverse action generation) and third component (population-based RL training) use LLM with RL and can be combined with any new prompting techniques.

[1] Xu, Yuzhuang, et al. "Exploring large language models for communication games: An empirical study on werewolf." arXiv preprint arXiv:2309.04658 (2023).

[2] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837.

[3] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).

Reviewer comment (Experiments):
The description of the experimental part is not detailed enough and some details are missing.

Author response:
**W2: Description of the experimental part is not detailed enough and some details are missing.**

We have added a section on the implementation details in Appendix D of our revised paper. The section includes detailed discussions on the attention policy architecture, environment reward, RL training hyperparameters, population-based training setup. We are willing to provide more explanations if any details are not clear.

Reviewer comment (Experiments):
In this particular task, RL requires both strong language understanding capabilities, to comprehend the intentions behind all possible actions, and the ability to solve reasoning tasks. This dual demand can potentially result in low learning efficiency or pose challenges in the learning processes.

Author response:
**W3: RL requires both strong language understanding capabilities to comprehend the intentions behind all possible actions, and the ability to solve reasoning tasks. This dual demand can potentially result in low learning efficiency or pose challenges in the learning processes**

We would like to clarify that the strong language understanding capability comes from the LLM. The action embedding produced by LLM can be regarded as a compact representation of the intention in the semantic space. The RL policy directly uses these embeddings as input and only needs to learn the optimal action distribution to improve the strategic reasoning ability. The RL policy in our experiment is trained for 50M environment steps using 4 RTX4090 GPUs and does not have challenges in learning efficiency. Please see Appendix D of our revised paper for more training details.

Reviewer comment (Experiments):
Among the results for human players, the LLM-based agent has a slight advantage in winning rate compared to human players. But important details are missing, like how do humans take input? Voice or text? Have you considered more realistic scenarios, such as expressions and tones in conversations?

Author response:
**Q1: Details for human evaluation experiments.**

Human players take the same text input as our agent, which includes their ID and hidden role, their Werewolf teammate (if any), their secret night actions (if any), the announcement, the discussions, and the voting results. Please see Appendix E.7 in our revised paper for an example input for human players. 

Currently, we only use text as input because Werewolf is a language game and the words of players contain the most important information. It would be an interesting future direction to consider more input like the expression and tones.

Reviewer comment (Evaluation):
In the zero-shot transfer section table 2, while the unified RL policy keeps a similar win rate across different LLM models, this is primarily due to the foundational capabilities of the combined LLMs ensuring a balanced action space. However, this doesn’t prove the RL policy’s potential applicability to various reasoning tasks. Can the author provide more evidence to verify this, such as transferability between different games?

Author response:
**Q2: The zero-shot transfer experiments doesn't prove the RL policy's potential applicability to various reasoning tasks. Can the author provide more evidence to verify this, such as transferability between different games?**

We would like to clarify that the zero-shot transfer experiment aims to show the trained RL policy's applicability to unseen LLMs, not different tasks or games. The results show that the RL policy trained on one LLM can be directly combined with other LLMs to improve their performance in the Werewolf game. Since the RL policy is trained only under this specific game, it cannot be transferred to different games. However, we believe our framework itself can be adapted to other social deduction games, with the deductive deduction component reasoning about game history, the diverse action generation component generating multiple actions, and the population-based RL training component choosing from the generated actions in a strategic fashion.

Reviewer comment (Experiments):
Does the author consider changes in the same game but different settings? For example, increasing the number of players from seven people to eight? Or adjust the player’s role in the game? Can the proposed framework handle this situation? Does the RL policy need to be collected and trained again?

Author response:
**Q3: Does the author consider changes in the same game but different settings?**

For generalization to new forms of the Werewolf game, the first two components in our method (deductive reasoning and diverse action generation) can be directly generalized to new settings like changing the number of players, adding new roles, and changing the winning conditions, with only slight changes in the prompt.

The RL training method can also be directly used in different settings, but the policy needs to be retrained for a new setting to achieve the best performance. We expect an RL policy trained under one specific game setting to generalize to similar settings like adding or removing one player, but we do not expect it to generalize to games with significant changes like changing the winning conditions. To evaluate the transferability of a trained RL policy to slightly different game settings, we considered a 6-player and an 8-player Werewolf game and compared agents with and without the RL policy trained on the 7-player game. The table below shows the RL policy can generalize to the 6-player and 8-player settings and improve the performance. We have added the results and discussions in Appendix H.2 of our revised paper.

|        Win Rate        | 6-player | 8-player |
|:----------------------:|:--------:|:--------:|
|  w.o. 7-player policy  |   0.18   |   0.27   |
| **w. 7-player policy** | **0.23** | **0.30** |

Reviewer comment (Experiments):
Does the selection of the RL policy raise concerns about consistency? For instance, if a player is assigned the role of a werewolf and chooses to lie and impersonate a different identity during the daytime conversation on the second day, should the lies told in subsequent days’ statements be consistent?

Author response:
**Q4: Does the selection of the RL policy raise concerns about consistency?**

The consistency in agents' behavior is achieved by both the LLM and the RL policy. The past actions and statements are used as input for the LLM, therefore, the LLM knows its previous behavior and most of the proposed action candidates are consistent. However, it is possible that the LLM produces inconsistent statements. In this case, the inconsistent statement is not a good action and the RL policy learns to assign it a low probability to avoid sampling it. Please see Appendix H.3 in our revised paper for more discussions on potential failure cases.