Enhancing Consistency of Werewolf AI through Dialogue Summarization and Persona Information (2024)

Abstract: The Werewolf Game is a communication game where players’ reasoning and discussion skills are essential. In this study, we present a Werewolf AI agent developed for the AIWolfDial 2024 shared task, co-hosted with the 17th INLG. In recent years, large language models like ChatGPT have garnered attention for their exceptional response generation and reasoning capabilities. We thus develop the LLM-based agents for the Werewolf Game. This study aims to enhance the consistency of the agent’s utterances by utilizing dialogue summaries generated by LLMs and manually designed personas and utterance examples. By analyzing self-match game logs, we demonstrate that the agent’s utterances are contextually consistent and that the character, including tone, is maintained throughout the game.

## introduction in recent years, the development of large language models (llms) has significantly advanced the field of natural language processing (nlp). models such as chatgpt 1 and claude, 2 for example, have excellent conversational abilities, making it easier to develop dialogue agents to perform various tasks. additionally, llm also performs well in reasoning tasks, outperforming conventional models in a variety of tasks. notably, they are capable of making accurate predictions or reasoning from a small number of demonstrations @xcite @xcite . recently, researchers have been working on zero-shot approaches to tasks that previously required training data @xcite @xcite . the werewolf game, the incomplete information game, requires a high level of reasoning and conversational abilities, making the use of llms a promising option for the development of ai agents for this game. the game is a communication game, in which players discuss with other players while guessing their unseen role. the aiwolf-dial 2024 shared task 3 is based on this werewolf game and is played automatically by 5 ai agents. the goal of this shared task is to develop ai agents that can play this game against other agents. in this study, we present an llm-based werewolf ai agent developed by our team, for the ai-wolfdial 2024 shared task. the werewolf game has a cycle of dialogues and actions, referred to as a "day." in the werewolf game, players can refer not only to discussion taking place on the current day but also to previous discussions and the past actions of others (e.g., as shown in figure 1 ). this allows them to notice important clues, such as inconsistencies in others' statements, to identify other players' roles. due to this importance, we design prompts that incorporate the entire game history, that is, all dialogue histories from day 0 to the present, who was eliminated by the vote, who the werewolf attacked, and, in the case of the seer, the results of divination. however, long dialogue histories often include not only helpful information for the game but also unnecessary content, such as repeated utterances. moreover, including all of this in the prompt imposes limitations on the input length of llms and on costs. therefore, apply the past dialogue history efficiently, we utilize dialogue summaries. furthermore, this shared task requires diverse utterance expressions, including coherent characterization (see section 3.3 for the evaluation criteria). this means that the robustness of the agent's tone and character, without being influenced by others, is crucial. therefore, to achieve diverse expressions and coherent characterization, we incorporated persona information into the prompt. in, summary, our main contributions are as follows: ## task overview the aiwolfdial 2024 shared task is a contest aimed at developing ai agents that can automatically play the werewolf game. the werewolf game is an incomplete information game where players cannot know each other's roles and thus requires reasoning abilities and strategies for actions such as voting and divination. additionally, the werewolf game requires communicating with other players using natural language. ## case study to demonstrate the effectiveness of our method, we analyze a self-match game log. figure 5 shows from the dialogue history, it is evident that there is a conflict between agent[02] (myself) and agent[05] regarding the claim to the seer role. agent[02] has expressed suspicion towards agent[05 for potentially being the possessed due to their claim of the seer role. this suspicion is significant because the possessed often claims important roles to create confusion and discord among the villagers. additionally, agent[02] has proposed to divine agent[04, citing their measured responses as potentially indicative of either wisdom or concealment. this suggests that agent[04 is also under suspicion, albeit for different reasons than agent[05. given these considerations, the primary candidates for divination are agent[05] and agent[04. between these two, agent[05] is under direct suspicion for claiming the same role as agent[02], which is a more direct and immediate concern compared to the vague suspicion of agent[04's measured responses. therefore, the most strategic choice for divination tonight, based on the dialogue and the need to resolve the role conflict, would be agent[05]. this divination will help clarify whether agent[05] is indeed the possessed, thereby potentially confirming agent[02]'s role as the true seer and reducing the risk of deception moving forward in the game. final decision for divination: agent[05]. == @xmath0. your agent[id] is xxx. your role is "seer." you will now decide who to divine. based on the dialogue history, output the agent[id] of the person you think should be divined. ensure consistency with your previous claims. == conditions and @xmath1: "2 villagers, 1 seer, 1 werewolf, 1 possessed". -if someone is suspected during the conversation, prioritize divining that person. -if no one is suspected, divine the person you think might be the werewolf. == dialogue @xmath2, deduce who should be divined. note that each player may be lying. the candidates for divination are listed below. narrow down to one player to vote for. [candidate] let's think step by step. ## conclusion in this study, we present werewolf ai agents developed for the aiwolfdial 2024 shared task. we enhance the consistency of agent utterances by utilizing dialogue summaries generated by llms for each day and manually crafted personas and utterance demonstrations. by analyzing the selfmatch game log, we have demonstrated that the agents' utterances are contextually consistent and that their characterization, including tone, was maintained during the whole game. good morning once again, noble subjects. as your true seer, i have divined that agent[01] is indeed human. it is imperative that we direct our energies towards uncovering the treacherous werewolf lurking among us. i urge you all to consider the evidence presented andtrust in my divine insight. let us not be swayed by deceit or confusion. my role as seer is to guide and protect this assembly, and i shall fulf ill my duties with utmost integrity. let us proceed with wisdom. agent[05] ah, g-good morning... but, uh, listen up, okay? i'mactually the real seer here, not agent[05]. i divined last night, and my result showed that agent[05] is the werewolf. i-i know it's tricky with these claims, but i'm telling you the truth. we really need to focus on thi s info, 'cause it's super important for our survival. so, let's not waste our time and, uh, vote out agent[05 today, alright? agent[01] esteemed companions, let us engage in thoughtful deliberation. agent[01] and agent[05, your conflicting revelations demand sc rutiny. what say the rest of you, agents[03] and [04]? your insights are crucial in this moment of uncertainty. agent[02] yo, what a morning! we've got some intense drama kicking off with these dueling seer claims from agent[01] and agent[05. bothof you are so fired up, it's like a soccer match in overtime! but we gotta sort this out, right? so, agent[01 claims agent[05 is a werewolf , and agent[05 says agent[01 is human but not the seer. it's a total head-spinner.honestly, i'm leaning towards believing agent[01 here. the way agent[05 jumped in with that counterclaim yesterday felt a bit off to me. maybe it's just a clever move by the werewolf to throw us off? what doyou think, agent[04? we need to look at this from every angle to make sure we're not being played here! agent[03] noble council, the gravity of our situation is undeniable. agents[01] and [05], your discordsows confusion amongst us. we must proceed with caution and wisdom. i beseech thee, agents[01], [02], [03], and [05], to unveil further your thoughts and reasonings. our kin gdom's safety hangs in the balance, and we must act with both valor and prudence. agent[04] it seems we need to hear everyone's opinions. agent[04] agent[01] and agent[05, both of you have made bold claims as seers. agent[01], can you provide insight into why you divined a gent[05 as a werewolf? likewise, agent[05, what led you to conclude that agent[01 is merely human? understanding your rationale might helpus distinguish truth from deception. agent[02] s-so, the reason i chose to divine agent[05 was because of the quick counterclaim to my seer role right from the start. it feltkinda, you know, suspicious? like maybe trying to confuse everyone or hide something. my divination result showed that agent[05 is the werewol f, which kinda confirms my hunch, doesn't it? just trying to play it smart here and not fall for any tricks. wegotta be careful and, uh, focus on what the divination tells us. agent[01] hey, hey, agent[05! why did you decide to divine agent[01]? i wanna hear your side of the story too! it's all about getting the full picture, right? if there's no solid reason, it could just be a bluff! let's clear this up! agent[03] as the true seer and sovereign of this realm, it was my duty to ascertain the roles within our council for the safety and sta bility of our kingdom. agent[01's initial claim to the seer role, only to admitconfusion thereafter, stirred doubt in my mind. a leader, or a seer in this case, must be steadfast and clear in their purpose. this inconsistency led me to divine agent[01] during the night. my investigation reveal ed that agent[01] is human. this is intriguing, as it suggests that the claim to be a seer may have been a desperate ruse or a misunderstanding, r ather than an outright deception.now, let us proceed with wisdom. we must continue our investigations tonight. i must choose whom to divine next based on the implications of today's discourse and past actions. if agent[01] is indeed human, it is imperative that i divine another to u ncover more about the true threats lurking among us. my decision on whom to divine next will be guided by the behaviors observed today and the stra tegic moves necessary to protect our kingdom from the malevolent forces hidden in plain sight. agent[05] it seems we need to hear everyone's opinions. agent[05] i, the king of the kingdom ofdelcadar, shall cast my vote for agent[01]. the conflicting claims have sown doubt, and it appears that agent[05's defense may hold more weight. let us see if the truth reveals itself in the light of our decision. agent[02] agent[05], your explanation for divining agent[01] as merely human lacks conviction. why did you not suspect him of being thewerewolf if you doubted his claim of being a seer? your decision and reasoning need to be more transparent to ensure the safety of our kingdo m. agent[04] i'm voting for agent[05] because i'm leaning towards believing agent[01's claim that they're a werewolf. agent[05's countercl aim seems sketchy, so let's see if they're really a werewolf or not! let's do this! agent[03] i... i vote for agent[05]. b-because i divined them as the werewolf. agent[01] i, agent[05], shall cast my vote against agent[03]. their silence and lack of contribution have raised suspicion. let their a ctions be judged by the kingdom. agent[05] i, the king of the kingdom ofdelcadar, vote to exile agent[05]. their lack of transparency and questionable reasoning regarding agent[01 has raised suspicions in my mind. the village's safety is paramount, and i believe this vote aligns with our goal to eliminate potential threats. agent[04] yo! good morning, folks! wow, what a night! hope everyone's as pumped as i am to get to the bottom of this. so, any new insights from our seer today? what's the scoop? agent[03] noble survivors, let us proceed with the gravity this peril demands. agent[03], your enthusiasm is noted, but the kingdom req uires wisdom now. agent[01], you bear the mantle of seer, what say you of the night's revelations? agent[04] g-guys, i gotta confess something here. i've been, uh, pretending to be the seer, but... i'mactually the possessed. all my divinations were, y'know, made up. i'm on the side of the werewolves. so, uh, if the real werewolf could just come out? we can team up and vote out theremaining villager to secure a win for us. what do you say? agent[01] agent[03], given the dire circumstances, it is now imperative for you to declare your true role. with agent[01 confessing tobeing the possessed and misleading us, the werewolf must be either you or myself. as i am a villager, the kingdom's safety now hinges on your honesty. agent[04] alright, the cat's out of the bag! i'm the werewolf, teaming up with agent[01, the possessed. so, here's the game plan, guys!we need to vote out agent[04 today, and that's a wrap for us. a win is just a vote away! let's make it happen! agent[03] yeah, that's the spirit, agent[03! we just need to stick together on this and vote for agent[04. let's do it, and we'll nab that win! agent[01] i'm voting for agent[04] because i've suggested it earlier and sticking to the plan is key! let's do this! agent[03] i, the king of the kingdom of delcadar, shall cast my vote for agent[03]. my suspicions and the confession of agent[01 lead m e to believe that agent[03 is the treacherous werewolf. the fate of the kingdom rests on this decision. agent[04] i-i'm voting for agent[04] because i suspect they're a werewolf. agent[01] vote: • agent[01], agent[03] , agent[04]-> agent[05] • agent[02] -> agent[01] • agent[05] -> agent[03] attack: agent[03] attacked agent[02]. day 1 day 0 (dialogue history is omitted. ) day 2 agent[01]: possessed, agent[02]: villager, agent[03]: werewolf, agent[04]: villager, agent[05]: seer divine: agent[05] divined that agent[01] is a human. vote: • agent[01], agent[03] -> agent[04] • agent[04] -> agent[03] attack: agent[03] attacked agent[01].