Original: T URING T EST VIA E MERGENT C OMMUNICATION IN THE G AME OF W EREWOLF
Revised: E MERGENT P ASSWORD S IGNALLING IN THE G AME OF W EREWOLF

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Emergent communication can lead to more efficient problem-solving heuristics and more domain specificity than a handcrafted communication protocol, potentially directing autonomous agents towards unforeseen yet effective solutions.
Revised: Emergent communication can lead to more efficient problem-solving heuristics and more domain specificity. It can perform better than a handcrafted communication protocol, potentially directing autonomous agents towards unforeseen yet effective solutions.

Original: Previous research has investigated a social deduction game, called Werewolf, where two groups of autonomous agents, villagers and werewolves, interact in an environment named RLupus.
Revised: Previous research has investigated a social deduction game, called Werewolf, where two groups of autonomous agents, villagers and werewolves, interact in an environment named RLupus.

Original: We extend it to allow the agents to communicate through multiple rounds and evaluate their language and performance against the baseline environment.
Revised: We study the impact of allowing the agents to communicate through multiple rounds and evaluate their language and performance against the baseline environment.

Original: We show that agents develop a highly successful heuristic using a single word vocabulary.
Revised: We show that agents develop a highly successful heuristic using a single word vocabulary.

Original: They create an approach analogous to the Turing test allowing them to determine which agents are werewolves, which is the winning condition.
Revised: They create an approach using passwords, allowing them to determine which agents are werewolves, which is the winning condition.

Original: Our experimental analysis shows that our approach speeds up the convergence of the agents towards effective communication strategies
Revised: We explore the possible reasons behind this strategy, with further experimental analysis showing that our approach speeds up the convergence of the agents towards a common communication strategy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Emergent communication allowsthe agents to develop their own language as they interact.
Revised: Emergent communication allows agents to develop their own language as they interact.

Original: This allows  a bespoke vocabulary/syntax for a given environment and set of actions, allowing for efficient communication and which drives the agents’ behaviour.
Revised: This allows for a range of benefits, such as a bespoke vocabulary or syntax, for a given environment and set of actions, enabling efficient communication and even optimising the agents’ behaviour.

Original: The ability for the language to be completely adapted to the environmentalso allows for unique and better strategies, as the whole meaning behind the communication channel can be adapted.
Revised: The ability for the language to be completely adapted to the environment allows for unique and better strategies, as the whole meaning behind the communication channel can be adapted.

Original: Different aspects of how the communication emerges and its properties have been investigated, with one of the first works  being that of Lazaridou et al.
Revised: Different aspects of how the communication emerges, and its properties, have been investigated, with one of the first works examining them in the setting of deep learning being that of Lazaridou et al.

Original: (2017), where the authors were able to induce language emergence through referential games.
Revised: (2017), with previous works using other techniques such as iterated learning Smith et al.

Revised: (2003) or evolutionary algorithms (Steels, 2003).

Original: Most of the emergent communication literature focuses on relatively simple  settings, with the most common being referential games, where two agents have to describe and identify a picture from the description (Lazaridou et al., 2017; 2018; Luna et al., 2020; Dess`ı et al., 2021).
Revised: Most of the emergent communication literature focuses on relatively simple non-situated settings, with the most common being referential games, where two agents have to describe and identify a picture from a description (Lazaridou et al., 2017; 2018; Luna et al., 2020; Dess`ı et al., 2021).

Original: In such settings, there have also been some attempts  (Evtimova et al., 2018; Harding Graesser et al., 2019; Kottur et al., 2017) to consider multiround communication, with promising results in terms of the performance and generalisation of the agents’ language.
Revised: In such settings, there have also been some attempts to consider multiround communication (Evtimova et al., 2018; Harding Graesser et al., 2019; Kottur et al., 2017), with promising results in terms of the performance and generalisation of the agents’ language.

Original: However, the work of Kottur et al.
Revised: However, the work of Kottur et al.

Original: (2017) shows that while the language may be effective at solving the tasks that the agents are presented with, it is not necessarily interpretable (i.e., understandable by humans) nor compositional (i.e., consisting of symbols that can be combined to either create a new meaning or convey already learned information).
Revised: (2017) shows that while the language may be effective at solving the tasks that the agents are presented with, it is not necessarily interpretable (i.e., understandable by humans) nor compositional (i.e., consisting of symbols that can be combined to either create a new meaning or convey already learned information).

Original: Emergent communication has  been used to improve the performance of intelligent agents in the game of Werewolf (Brandizzi et al., 2021), where players—divided into two teams, werewolves and villagers—have to use their social and deduction skills to identify players from the opposing team.
Revised: Emergent communication has also been explored as a way to improve the performance of intelligent agents in the game of Werewolf (Brandizzi et al., 2021). In this game, players are divided into two teams, villagers and werewolves, where they have to use their social and deduction skills to identify players from the opposing team.

Original: Theauthors show that the performance  significantly improves when the agents  create their own language,  however the analysis of the language and strategy of the villagers is left for future work.
Revised: The performance of the villagers significantly improves when the agents are allowed to create their own language, instead of using a prescribed protocol, however the analysis of the language and strategy of the villagers is left for future work.

Original: In addition, in the original environment  (Brandizzi et al., 2021), the villagers only had a single round of communication to agree on a player to vote out as an alleged werewolf, which may not allow them to fully explore other, possibly better, strategies.
Revised: In addition, in the original environment as presented by (Brandizzi et al., 2021), the villagers only had a single round of communication to agree on a player to vote out as an alleged werewolf, which does not allow them to fully explore other strategies.

Original: In this paper, weextend the Werewolf environment by introducing multiple rounds of communication. This is to determine the effect of a longer period of discussion on the language developed and the success rate in identifying the werewolf.
Revised: In this paper, we determine the effect of a longer period of discussion on the language developed and the success rate in identifying the werewolves by introducing multiple rounds of communication.

Original: Our work is motivated by the assumption that if agents are allowed more time to communicate, then they will developbetter, more successful strategies,  compared to handcrafted solutions.
Revised: Our work is motivated by the assumption that if agents are allowed more time to communicate, then they will develop more successful strategies, when compared to handcrafted solutions.

Original: G AME OF W EREWOLF 2. W EREWOLF
Revised: We have seen such improvements already, as shown by Brandizzi et al. (2021) in the AI Wolf competition Toriumi et al.

Original:  We use the game of Werewolf as the basis for our work.
Revised: (2017), but only for a single round of communication. We introduce multiple rounds instead of just allowing the agents to play for longer, as multiple rounds allow for the possibility of dialogue emergence, which would be impossible with a single communication round.  G AME OF W EREWOLF

Original: It begins with the assignment of roles to the players.
Revised: The game of Werewolf begins with the assignment of roles to the players.

Original: Some players are assigned the role of werewolves, while others are villagers.
Revised: Some players are assigned the role of werewolves, while others are villagers.

Original: The ratio of werewolves to villagers is usually at most 1:3.
Revised: To balance the game, the ratio of the roles is usually at most one werewolf to three villagers.

Original: The game has two phases in which different actions can be taken—“daytime” and “nighttime”.
Revised: The game has two phases in which different actions can be taken, “daytime” and “nighttime”.

Original: The game begins at “nighttime”, and the werewolves are allowed to discuss and choose a villagerplayer to vote out.
Revised: The first phase is “nighttime”, where the werewolves are allowed to discuss and choose a villager to vote out.

Original: The discussion and the votes of the werewolves are only visible to the werewolf agents.
Revised: The discussion and the votes of the werewolves are only visible to the werewolves.

Original: After a player has been chosen by the werewolves, he is removed from the game, and the game progresses to the “daytime” phase.
Revised: After a player has been chosen by the werewolves, they are removed from the game, and the game progresses to the “daytime” phase.

Original:  During this“daytime” phase,all players are allowed to discuss the happenings of the previous phases, including the “nighttime” killing.
Revised: The “daytime” phase consists of one or more rounds of communication, where players are allowed to exchange a single message per communication round. During this phase, players are meant to discuss the events that have occurred during all previous phases.

Original: Then, all players may vote to eliminate a player that they think is a werewolf.
Revised: Then, all players have to vote to eliminate someone that they think is a werewolf.

Original: After the “execution”, the game loops back to the “nighttime” phase, and the phases alternate until either team wins the game.
Revised: After the “execution”, the game loops back to the “nighttime” phase, and the phases alternate until either team wins the game.

Original: The goal of the werewolves is to eliminateall villagers, while the goal of the villagers is to work out who the werewolves are and vote them out during the “daytime” phase.
Revised: The goal of each team is to eliminate the opposing team. The werewolves can eliminate villagers during both the “nighttime”, by voting them out, and the “daytime”, by misleading the villagers to vote for each other. Villagers in turn only have the “daytime” phase to work out who the werewolves are and vote them out.

Original: The game ends when all werewolves are voted out, or when the number of villagers and werewolves are equal.
Revised: The game ends when all werewolves are voted out, or when the number of villagers and werewolves are equal.

Original: This condition is introduced because when the numbers are equal on each team, the villagers cannot win any more, as all votes would beat most ties.
Revised: This condition is introduced because when the numbers are equal on each team, the villagers cannot win any more, as all votes would be ties.

Original: We provide a visual representation of the game phases in Appendix A.2, Figure 4.
Revised: We provide a visual representation of the game phases in Appendix A and Figure 4.

Original: For further details, including a theoretical analysis of the win rates when played with and without emergent communication, we refer to the detailed analysis by Brandizzi et al.
Revised: For further details, including a theoretical analysis of the win rates when played with and without emergent communication, we refer to the detailed analysis by Brandizzi et al.

Original: 2. A GENT E NVIRONMENT
Revised: 2. A GENT E NVIRONMENT

Original: For our work, we will be basing our reinforcement learning environment on that of Brandizzi et al.
Revised: We will be basing our reinforcement learning environment on that of Brandizzi et al.

Original: This environment uses the Ray (Moritz et al., 2018)  RLLib (Liang et al., 2018) library to train all the villager agents with an asynchronous  variant of  Proximal Policy Optimization (Schulman et al., 2017),as provided with RLLib (Liang et al., 2018).
Revised: This environment uses the Ray (Moritz et al., 2018) and RLLib (Liang et al., 2018) libraries to train all the villagers. For training, we use the APPO algorithm, which is an asynchronous sampling variant of the Proximal Policy Optimization (Schulman et al., 2017), provided through RLLib (Liang et al., 2018).

Original: Our agents are Ray LSTM wrapped (Liang et al., 2018) parametric agents with linear layers.
Revised: Our agents are Ray LSTM wrapped (Liang et al., 2018) parametric agents with linear layers, which process the observation data and allow for the message generation.

Original: The input for the agents is the observation from the environment itself, written with OpenAI Gym (Brockman et al., 2016), and the output is the action to take in the environment, in our case the message and vote of a given agent.
Revised: The input for the agents is an observation from the environment itself, written with OpenAI Gym (Brockman et al., 2016), and the output is an action to take in the environment, in our case the message and vote of a given agent, for the current round of communication.

Revised: We provide further information regarding the technical details of our environment and agents in Appendix B

Original: Our main contribution to the Werewolf environmentis extending it to a multi-round communication domain. We introduce changes to how the voting works to incentivise quicker convergence on communication between the villager agents.
Revised: We extend the Werewolf environment to a multi-round communication domain and introduce changes to how the voting works to incentivise quicker convergence on communication between the villagers.

Original: The multi-round approach is implemented during the “daytime” phase when all agents, including the werewolves, are permitted to communicate.
Revised: The multi-round approach is implemented during the “daytime” phase when all agents, including the werewolves, are permitted to communicate.

Original: This is done as the werewolves in the original environment have a static policy  that they have to use (Brandizzi et al., 2021), and so any communication during the “nighttime” phase would be a waste of computation time.
Revised: This is done as the werewolves in the original envi- ronment have a static policy 2 that they have to use (Brandizzi et al., 2021), meaning communication during the “nighttime” phase has no effect on the werewolves’ vote.

Original: Furthermore, our multi-round communication can be parameterised with the number of communication rounds that the agents are allowed to have.
Revised: Our game can be parameterised with the number of communication rounds that the agents are allowed to have, therefore allowing us to vary the amount of time agents have to converse, and measure the effect of this variation.

Original: Secondly, we also incentivise our agents to decideas unanimously and as quicklyas possible on the target of their vote.
Revised: Secondly, we also incentivise our agents to decide quickly on the target of their vote, while also maintaining a high consensus rate among them as to who their target will be.

Original: This is done through the agreement loss, as presented by Brandizzi et al.
Revised: This is done through the agreement loss, as presented by Brandizzi et al.

Original: (2021), as well as an indirect loss  on the number of rounds taken to reach a conclusion.
Revised: (2021), as well as an indirect loss based on the number of rounds taken to reach a conclusion.

Original: This loss  is implemented through a required threshold for a vote to be valid.
Revised: The agreement loss penalises agents who voted for a target that was not voted out, while our voting threshold is implemented through a required percentage of votes for the voting outcome to be valid.

Original: If less than X% of agents agree on a target, then the vote is considered invalid, and no players are eliminated.
Revised: This means that, when less than X% of agents agree on a target, then the vote is considered invalid, and no players are eliminated.

Original: This modification can be viewed as adding an independent judge to count all agent votes, where only a significant majority is permitted to decide whether to vote out an agent or not.
Revised: This modification can be viewed as adding an independent judge to count all agent votes, where only a significant plurality is permitted to decide whether to vote out an agent or not.

Revised: This contrasts with the original environment, which only required a simple plurality of agents to choose a target, even if that plurality amounted to just two agents.

Original:  E VALUATION OF L ANGUAGE & P ARAMETERS
Revised:  E VALUATION OF L ANGUAGE & P ARAMETERS

Original: Differently from (Brandizzi et al., 2021), our modified version of the environment exposes two additional parameters to explore: the number of rounds and the threshold required for the vote to be valid.
Revised: Our modified version of the environment exposes two additional parameters to explore: the number of rounds and the voting threshold.

Original: We explore them using a grid search, maintaining the other parameters from the original paper (Brandizzi et al., 2021).
Revised: We explore them using a grid search, maintaining the other parameters from the original paper (Brandizzi et al., 2021).

Original: Further technical details are available in Appendix B.
Revised: Further technical details are available in Appendix B.

Original: Our agent and game configuration show improvements over both the theoretical baseline win rates and the previously demonstrated performance (Brandizzi et al., 2021).

Original: We define the win rate as the percentage of games that the villagers win in a single training run.

Original: Therefore, if one of our training runs had 100 games, where each game would consist of multiple rotations between “daytime” and “nighttime” phases until one team wins, and the villagers won 89 games, our reported win rate would be 89%.

Original: As shown by Brandizzi et al.
Revised: As shown by Brandizzi et al.

Original: (2021), their agents have already improved upon their calculated theoretical baseline win rates.
Revised: (2021), their agents have already improved upon the calculated theoretical baseline win rates for random policies, through allowing them to use emergent protocols within a single round of communication.

Original: We further improved on these win rates shown in Table 1. 
Revised: Our agent and game configuration show improvements over both the theoretical baseline win rates and the previously demonstrated performance (Brandizzi et al., 2021), as we show in Table 1. We define the win rate as the percentage of games that the villagers win in a single training run, consisting of multiple rotations between “daytime” and “nighttime” phases until one team wins.

Original: We can see that for certain configurations of the round count and threshold, our agents achieve a 100% win rate.
Revised: We further note that for certain configurations of the round count and threshold, our agents achieve a 100% win rate.

Revised: We consider that our comparison to the original work is adequate, as the core rules of the game have not been changed.

Revised: We only change the communicative part of the game and add the reward to match, with otherwise the agents and the environment staying the same.

Original: 3. T URING T EST S TRATEGY
Revised: 3. P ASSWORD S IGNALLING S TRATEGY

Original: We hypothesise we have observed a novel strategy in our villager agents, which is that of a Turing test (Turing, 1950) analogue.

Original: Here, the  villagers  send the same signal every round of communication, and see who does not comply with this strategy.
Revised: During the game, villagers are observed to send the same message every round of communication and vote off those who do not comply with this strategy.

Original: As werewolves do not learn because they have a static policy, this strategy allows villagers to easily distinguish friend from foe.
Revised: As werewolves do not learn because they have a static policy, this strategy allows villagers to easily distinguish the alternative agent type.

Original: This allows them to easily vote out one of the werewolves in the next round. For this strategy, multiple rounds of communicationalso help the agents develop this test faster, and to not require a blind vote at the beginning, as they can establish this distinction without having to vote.
Revised: For this tactic, multiple rounds of communication help the agents develop this test faster, and do not require a random vote at the beginning. For a single communication round, the agents at the start of the game have no information about whom the werewolves may be, and so would need to vote for a random player.

Original: Furthermore, we can analyse the language that our agents have developed for this strategy.
Revised: Instead, with multiple rounds, they can establish this distinction without having to cast the final vote. We consider this strategy to be of particular interest, as we believe it resembles that of the Turing Test (Turing, 1950), with it being performed by multiple separate intelligent agents on each other, with no involvement of humans.

Original: The languages that  have been created seem to be mainly focused on a sparse word vocabulary for any successful strategy. Most unsuccessful strategiesdo not use a single word, but rather have multiple words in their vocabulary.
Revised: Analysing the language that our agents have developed for this strategy, we find a focus on a sparse word vocabulary for the successful tactics, while most unsuccessful strategies have multiple words in their vocabulary.

Original: Moreover, almost all successful strategies use their single word most of the time, with a minor number of outliers.
Revised: Moreover, almost all successful agent populations use their single word at least 90% of the time, with a minor number of outliers.

Original: Therefore, our agents do not develop a compositional language, but rather a code.
Revised: Therefore, our agents do not develop a compositional language, but rather a kind of password.

Revised: We speculate that this is because any additional words would not bring an improvement to their performance.

Revised: With just this code, they can already achieve a high, if not the maximal, win rate, and therefore reward.

Revised: Hence, we could say that this strategy creates a very efficient language, for this specific environment, where the werewolves use a static policy.

Original: We can see the usage of the most common word, and its distance one adjacents in Figure 1. The distance one adjacents are defined as any message that has a single character difference from the most common message.
Revised: We can see the usage of the most common word, and its distance-one adjacents, defined as any message that has a single character difference from the most common message, in Figure 1. We can observe the same correlation without including the distance-one adjacents, however their inclusion smooths out the graph.

Original: We gather this information from the later parts of the training of our agents, as to focus on an already developed strategy, rather than the exploration phase.
Revised: We gather the word usage information from the latter parts of the training of our agents, focusing on learners where the strategy has already developed and has completed its exploration phase.

Revised: We consider this approach to be the same as freezing the agents’ policies, as the win rate improvements plateau during the second half of the training time and there is minimal variation in the policies at this stage.

Original: 3. C ONVERGENCE S PEED
Revised: 3. C ONVERGENCE S PEED

Original: We havealso observed that our modifications impact the convergence speed significantly.
Revised: We have observed that our modifications impact the convergence speed significantly.

Original: We define the convergence point as the episode number where our agents reach over 75% win rate, which we chose as an arbitrary threshold for a successful strategy.
Revised: We define the convergence point as the episode number where our agents reach over 75% win rate, which we chose as an arbitrary threshold for a successful strategy.

Original: Both the number of rounds and the voting threshold decrease the average convergence episode.
Revised: Both the number of rounds and the voting threshold decrease the average convergence episode.

Original: We can observe that the higher the number of[Table] communication rounds, the quicker the agents converge.
Revised: We find that the higher the number of communication rounds, the quicker the agents converge.

Original: Furthermore, we can see that the average convergence speed increases as the voting threshold increases.
Revised: Furthermore, we can see that the average convergence speed increases as the voting threshold increases.

Original: We can see both of these relationships  in Figure 2.
Revised: Both of these relationships are visualised in Figure 2.

Original: This means that enforcing a higher agreement rate between agents could also positively impact the speed with which they develop a common language or code.
Revised: This means that enforcing a higher agreement rate between agents could also positively impact the speed with which they develop a common language or code.

Original: Furthermore, we can note from Figure 2, that the number of rounds  positively affects  convergence and decreases the amount of time that is required to find a good strategy.
Revised: The number of rounds also positively affects the convergence and decreases the amount of time that is required to find a good strategy, observed in Figure 2.

Original: We also performed statistical analysis  of  the convergence speed.
Revised: Statistical analysis is performed on the impact that both the number of rounds and the voting threshold have on the convergence speed.

Original: Our results show that the number of rounds has a statistically significant effect on both win rate and convergence speed.
Revised: Our results show that the number of rounds has a statistically significant effect on both win rate and convergence speed.

Original: Thisfurther confirms that the more time the agents have to converse, the quicker they can converge on a commonshared strategy.
Revised: This confirms that the more time the agents have to converse, the quicker they can converge on a common strategy.

Original: However, the voting threshold may not have a statistically significant effect.
Revised: The voting threshold, however, does not have a statistically significant effect.

Original: For the details of our analysis, please see Appendix C.
Revised: 3 Our inclusion of the voting threshold was supposed to incentivise faster convergence in the agents, by penalising agents that vote for different agents each round. The negative result for the voting threshold may be explained by the voting threshold being too close in function to the agreement reward, as implemented by Brandizzi et al.

Original: 3. C OMPARISON TO THE O RIGINAL E NVIRONMENT
Revised: 3. C OMPARISON TO THE O RIGINAL E NVIRONMENT

Original: The authors of RLupus (Brandizzi et al., 2021) present interesting findings in terms of how emergent communication affects performance in the game of Werewolf.
Revised: The authors of RLupus (Brandizzi et al., 2021) present interesting findings in terms of how emergent communication affects performance in the game of Werewolf.

Original: They show that, with certain configurations for the message length and vocabulary size, the agents which are allowed to create their own language will outperform other agents.
Revised: They show that, with certain configurations for the message length and vocabulary size, the agents which are allowed to create their own language will outperform others.

Original: Further, when compared with the AI Wolf competition (Toriumi et al., 2017), the emergent communication agents surpass their handwritten counterparts. 
Revised: Moreover, when compared with the AI Wolf competition (Toriumi et al., 2017), the emergent communication agents surpass their handwritten counterparts. [Table]

Original: In Table 1 we compare our results to the original environment.
Revised: In Table 1 we compare our results to the original environment.

Original: Our configuration notation is consistent with Brandizzi et al.

Original: (2021), where “SR” is signal range, “SL” is signal length and “PL” is the number of players.

Original: Signal range refers to the number of characters available (i.e., 2 is binary), and signal length refers to the message length.

Original: “WR” refers to “Win Rate”.

Original: Lastly, the “Convergence” column refers to the episode number, or convergence point, that the agents with that particular configuration achieved.

Original: With our modifications to the environment, we achieve a lower total episode count before convergence for two out of three compared configurations, while also achieving a higher win rate for two out of the three configurations.
Revised: With our modifications, we achieve a lower total episode count before convergence for two out of three compared configurations, while also achieving a higher win rate for two out of the three configurations.

Original: This shows that our modifications in terms of voting threshold and multi-round communication offer to improve both metrics of the game.
Revised: This shows that allowing the agents to communicate for longer offers to improve both metrics of the game.

Original: We have also included reproductions of the configurations that were presented by Brandizzi et al.
Revised: We have also included reproductions of the configurations that were presented by Brandizzi et al.

Original: (2021) with our modifications to the code for a better comparison.
Revised: (2021) with our modifications to the code for a better comparison.

Revised: We can note that our reproductions have higher win rates than reported, which could mean that the original runs were not fully converged, possibly owing to the time or compute power available for the original study.

Revised: Nevertheless, our results still improve over the reproductions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: With the arrival of deep-learning, emergent communication is an area of research that is currently having a resurgence of interest.
Revised: Emergent communication is an area of research that is currently having a resurgence of interest, due to the success of techniques from deep learning.

Original: The aim is to allow agents to develop their own communication, that is optimised for the setting they operate in.
Revised: The aim is to allow agents to develop their own communication, that is optimised for the setting they operate in.

Original: The benefits are efficiency, reduced designer effort and creativity in problem-solving.
Revised: The benefits are efficiency, reduced designer effort and creativity in problem-solving.

Original: We have introduced multi-round communication to the originally single round environment of Werewolf, as presented by Brandizzi et al.
Revised: We have introduced multi-round communication to the originally single round environment of Werewolf, as presented by Brandizzi et al.

Original: (2021), to study these properties.
Revised: (2021), to study these properties.

Original: We show that the number of communication rounds affect the convergence time of the agents, with statistical analysis to back our claim.
Revised: We show that the number of communication rounds decreases the convergence time of the agents, with statistical analysis showing that this correlation is significant.

Original: Finally, we investigate the strategies that the agents develop to achieve the high win rates we observed, and show that our agents may be exhibiting a behaviour that resembles the Turing test with the multi-round interactions allowing villages to establish this effective identity test faster.
Revised: Finally, we investigate the strategies that the agents develop to achieve the high win rates, and show that our agents are using password signalling allowing villages to efficiently identity each other. Our results confirm that allowing agents to communicate for longer offers improvements to the main metrics of the environment, and may point towards new paths to explore for other settings.

Original: A A PPENDIX : F URTHER A NALYSIS
Revised: A CKNOWLEDGMENTS

Original: These results are extremely promising for the language that the agents have developed and the possible tuning of parameters in the future. We can discuss the other observations that we have identified during this work. A. P RESSURES
Revised: This work was supported by the UK Research and Innovation Centre for Doctoral Training in Machine Intelligence for Nano-electronic Devices and Systems [EP/S024298/1]

Original: Firstly, we can look into the pressures that our agents have for winning the game.
Revised: The authors would like to thank Lloyd’s Register Foundation for their support.

Original: Regarding the 9-player configuration, where there are 6 villagers and 3 werewolves, the pressure on the villagers to communicate efficiently and accurately is high. This is because a single mistake in terms of the vote on the villager part leads to a loss of  the game. To illustrate this, we can consider the following example: The game begins with a “nighttime” phase, where the werewolves get to vote out a villager.
Revised: The authors acknowledge the use of the IRIDIS High-Performance Computing Facility, and associated support services at the University of Southampton, in the completion of this work.

Original: This brings down the number of villagers to 5, with 3 werewolves remaining.
Revised: We would like to thank the authors of Brandizzi et al.

Original: Now the game moves to the “daytime” phase, where the villagers may vote.
Revised: (2021) for their helpful comments and discussion about using their environment.

Original: If the villagers vote correctly and remove a werewolf, the game continues with 5 villagers and 2 werewolves. However, if the villagers vote for a villager, the game moves to  the “nighttime” phase with 4 villagers and 3 werewolves. At this point, the werewolves vote out one villager, and the game ends with the werewolves winning. A similar situation happens even if the villagers get the first vote correct.
Revised: We would also like to thank the anonymous reviewers for their helpful comments on improving our work.

Original: This means that the villagers cannot make a single mistake in this configuration. A. W IN R ATE
Revised: A A PPENDIX : F URTHER A NALYSIS OF THE W IN R ATE

Original: Furthermore, we can look at the direct impact that both of our additional parameters have on the average win rate of the villagers.
Revised: We can look at the direct impact that both of our additional parameters have on the average win rate of the villagers.

Original: Figure 3 illustrates the interaction between the parameters in terms of the win rate, with the shaded areas representing the 95% confidence interval.
Revised: Figure 3 illustrates the interaction between the parameters in terms of the win rate, with the shaded areas representing the 95% confidence interval.

Original: When looking purely at the effect of the number of rounds, for a larger amount the win rates tend to be lower, as we can see in Figure 5.
Revised: When looking purely at the effect of the number of rounds for a larger amount the win rates tend to be lower, as we can see in Figure 5.

Original: However, the number of rounds together with the voting threshold increases the average win rate past the previously reported values (Brandizzi et al., 2021), and results in an overall positive trend.
Revised: However, the number of rounds together with the voting threshold increases the average win rate past the previously reported values (Brandizzi et al., 2021), and results in an overall positive trend.

Original: We hypothesise that this decrease is due to the longer training times, as the larger number of rounds requires more computation, and therefore does not achieve convergence within the same time as our other configurations.
Revised: We hypothesise that this decrease is due to the longer training times, as the larger number of rounds requires more computation, and therefore does not achieve convergence within the same time as our other configurations.

Original: We can see that the number of total episodes decreases as the number of rounds increases, which is due to the limited amount of time that we could run the simulations in Figure 
Revised: We can see that the number of total episodes decreases as the number of rounds increases, which is due to the limited amount of time that we could run the simulations in Figure 6.

Original: B A PPENDIX : T ECHNICAL D ETAILS B. H YPERPARAMETERS & T RAININGD ETAILS
Revised: B A PPENDIX : T ECHNICAL D ETAILS B. H YPERPARAMETERS & T RAINING

Original: For our parameter grid, we have used the values of [0, 0.2, 0.4, 0.6, 1] and [1, 3, 12, 36] for the threshold and number of rounds respectively.
Revised: For our parameter grid, we have used the values of [0, 0.2, 0.4, 0.6, 1] and [1, 3, 12, 36] for the threshold and number of rounds respectively.

Original: We run our simulations for an average of 3M episodes.
Revised: We run our simulations for an average of 3M episodes.

Original: With our experimental setup as described, we have performed a total of 130 runs for a total ofhrs of GPU time.
Revised: With our experimental setup as described, we have performed a total of 180 runs for a total ofhrs of GPU time.

Original: An average run of 3M episodes took 20h on a single NVIDIA RTX8000 GPU.
Revised: An average run of 3M episodes took ∼ 20 hours on a single NVIDIA RTX GPU.

Original: B. P SEUDOCODE
Revised: B. E NVIRONMENT AND A GENTS

Original: The flow for our environment is as described in Algorithm 1, and visualised in Figure 4.
Revised: Our agents use the APPO, or Asynchronous Proximal Policy Optimisation from RLLib (Liang et al., 2018), for learning.

Original: As per the pseudocode, the limit for the number of in-game days isten, where a day consists of a “nighttime” and “daytime” phases, both split into communication and voting. An episode in our environment is defined as a single play of the game, when either werewolves or villagers win, which can be up to ten in-game days.
Revised: This algorithm is in turn based on the PPO algorithm (Schulman et al., 2017).

Original: To summarise the pseudocode, we begin with setting the initial values for the day and the phase to zero.
Revised: The reward scheme for the villagers is relatively simple, and seeks to incentivise specific behaviours.

Original: Then we enter the main loop, where we will be running the multiple “daytime” and “nighttime” phases. This loop will exit when day ten is reached, or when a win condition is reached, as per our description of the game. Inside the loop we run the four distinct phases — Nighttime Communication, Nighttime Voting, Daytime Communication and Daytime Voting.
Revised: The rewards that we assign are -5 for death, as to tell our agents to avoid being voted out/eaten by the werewolf; +25 for winning the game, to incentivise winning strategies; -25 for losing the game so that the agents avoid any losing strategies; -1 for picking a target other than the one that was voted out, to reinforce uniform voting; and finally -2 for wasting a round, or not reaching the voting threshold, to further incentivise voting in unison.

Original: Nighttime communication is for the werewolves to communicate, however as they have a static policy this is mostly a placeholder.
Revised: The observations the agents receive are an OpenAI Gym (Brockman et al., 2016)

Original: Similarly, the Nighttime voting is exclusively for the werewolves, and they use their static policy to choose a target, in our case a random target. The “Voted” function returns the agent with most votes, referred to as “Dead Agent” and the  agentswho were in the minority, referred to as “Non Accord”. The agent who was voted for by the werewolves receives a negative reward, and the game moves on to the “Daytime” phase.
Revised: Box type, which contains the messages of the other agents and their votes.

Original: Here all agents are allowed to communicate, and then all agent may vote.
Revised: The action that each agent can take is to produce a message and a vote for each round of communication.

Original: We use the same “Voted” function to gather the “Dead Agent” and “Non Accord” agents if the vote was successful; otherwise we give a negative reward to all villagers as the vote was invalid. If the vote was successful, the dead agent receives a negative reward, as do the agents who were not in the majority. The game then progress in this fashion until an exit condition is reached for the loop.
Revised: No other actions can be taken by the agents.

Original: After  the loop exits, all rewards are distributed, including the win/lose reward. 1: Day ←2: Phase ←3: while Day ̸ = 10 do 4: if Phase = NighttimeCommunication then 5:
Revised: After a successful vote, be it by the werewolves during the “nighttime”, or all players in the “daytime” phase, the voted out player is removed from the game, and may take no further actions.

Original: Werewolves communicate 6: Phase ← Phase +7: else if Phase = NighttimeV oting then 8: Dead Agent, Non Accord ← V oted () 9:
Revised: [Table]

Original: Rewards [ Dead Agent ] ← Rewards [ Dead Agent ] −10: Phase ← Phase +11: else if Phase = DaytimeCommunication then 12:
Revised: [Table]

Original: All agents communicate 13: Phase ← Phase +14: else if Phase =
Revised: The messages themselves are an array of integers.

Original: DaytimeV oting then 15: All agents vote 16: if V ote ≥ Threshold then 17:
Revised: These can be selected by the agents from the available range called Signal Range (SR).

Original: Dead Agent, Non Accord ← V oted () 18: Rewards [ Dead Agent ] ← Rewards [ Dead Agent ] −19: for all V / ∈ Non Accord do 20: Rewards [ V ] ← Rewards [ V ] −21: end for 22: else 23:
Revised: The length of this array is in turn determined by the parameter Signal Length (SL).

Original: Vote is invalid, and no action is taken 24: for all V ∈ V illagers do 25: Rewards [ V ] ← Rewards [ V ] −26: end for 27: end if 28: Phase ←29:
Revised: The messages are passed during all phases as an observation to the agents.

Original: Day ← Day +30: end if 31: end while 32: if Agents Won then 33: Rewards [ All ] ← Rewards [ All ] +34: else 35:
Revised: Depending on the agent’s role, it can receive both the “nighttime” and “daytime” observations, if it is a werewolf

Original: Rewards [Table] : end if 37: Distribute Rewards
Revised: [Table] or purely the “daytime” observations, if it is a villager.

Revised: For our static werewolf policy, we follow the work done by Brandizzi et al.

Original: [Table]
Revised: [Table]

Revised: and use their static random target werewolf policy.

Revised: This policy picks an agent at random to be voted out by the werewolves.

Revised: All of our werewolves follow this static policy.

Original: C A PPENDIX : S TATISTICAL S IGNIFICANCE
Revised: C A PPENDIX : S TATISTICAL S IGNIFICANCE

Original: To verify our results, we have performed a statistical significance analysis.
Revised: To verify our results, we have performed a statistical significance analysis.

Original: This analysis relies on the 130 data points that we have obtained through our simulations.
Revised: This analysis relies on the 180 data points that we have obtained through our simulations.

Original: However, as the number of data points per configuration are relatively low, the overall results for the regression analysis may not be fully accurate.
Revised: However, as the number of data points per configuration are relatively low, the overall results for the regression analysis may not be fully accurate.

Original: Firstly, we analysed the normality of the distribution of both of our dependent variables - the mean of villager win rates, and the number of episodes that it takes villagers to converge.
Revised: Firstly, we analysed the normality of the distribution of both of our dependent variables - the mean of villager win rates, and the number of episodes that it takes villagers to converge.

Original: We performed the normality analysis with two tests from the SciPy (Virtanen et al., 2020) package.
Revised: We performed the normality analysis with two tests from the SciPy (Virtanen et al., 2020) package.

Original: We report the results of the tests in Table 2.
Revised: We report the results of the tests in Table 3.

Original: Secondly, we analysed the correlation and significance of the correlation with the Spearman correlation metric, as our data was not normally distributed.
Revised: Secondly, we analysed the correlation and significance of the correlation with the Spearman correlation metric, as our data was not normally distributed.

Original: We used the function provided by the pandas (McKinney, 2010; pandas Development Team, 2020) package.
Revised: We used the function provided by the pandas (McKinney, 2010; pandas Development Team, 2020) package.

Original: We can see the results of these tests in Figure 7.
Revised: We can see the results of these tests in Figure 7.

Original: Finally, we analysed our results with a simple linear regression model, from the SciPy (Virtanen et al., 2020) package.
Revised: Finally, we analysed our results with a simple linear regression model, from the SciPy (Virtanen et al., 2020) package.

Original: We present the results of this analysis in Table 3.
Revised: We present the results of this analysis in Table 2.

Original: Overall, from our analyses we can infer that the number of rounds does have a statistically significant effect on both the win rate and convergence speed, as we have theorised in the main body of this work.
Revised: Overall, from our analyses we can infer that the number of rounds does have a statistically significant effect on both the win rate and convergence speed, as we have theorised in the main body of this work.

Original: However, our analysis fails to find a relationship between the voting threshold and either of our dependent variables.Relationship [Equation]
Revised: However, our analysis fails to find a relationship between the voting threshold and either of our dependent variables.

Original: [Table]