AIW olf D ial 2023: Summary of Natural Language Division of 5th International AIW olf Contest (2023)

Abstract: We held our 5th annual AIWolf international contest to automatically play the Werewolf game “Mafia”, where players try finding liars via conversations, aiming at promoting developments in creating agents of more natural conversations in higher level, such as longer contexts, personal relationships, semantics, pragmatics, and logics, revealing the capabilities and limits of the generative AIs. In our Natural Language Division of the contest, we had six Japanese speaking agents from five teams, and three English speaking agents, to mutually run games. By using the game logs, We performed human subjective evaluations and detailed log analysis. We found that the entire system performance has largely improved over the previous year, due to the recent advantages of the LLMs. However, it is not perfect at all yet; the generated talks are sometimes inconsistent with the game actions, it is still doubtful that the agents could infer roles by logics rather than superficial utterance generations. It is not explicitly observed in this log but it would be still difficult to make an agent telling a lie, pretend as a villager but it has an opposite goal inside. Our future work includes to reveal the capability of the LLMs, whether they can make the duality of the “liar”, in other words, holding a “true” and a “false” circumstances of the agent at the same time, even holding what these circumstances look like from other agents.

## introduction recent achievements of generation models, e.g. chatgpt @xcite , are gathering greater * correspondence to kano@kanolab.net attentions. however, it is not fully investigated whether such a huge language model can sufficiently handle coherent responses, longer contexts, common grounds, and logics. our shared task, aiwolfdial 2023, is an international open contest for automatic players of the conversation game "mafia", which requires players not just to communicate but to infer, persuade, deceive other players via coherent logical conversations, while having the role-playing non-task-oriented chats as well. aiwolfdial 2023 is one of the inlg 2023 generation challenges for this year. we believe that this contest reveals not just achievements but also current issues in the recent huge language models, showing directions of next breakthrough in this area. we propose to employ this werewolf game as a novel way of evaluations for dialog systems. while studies of dialog systems are very hot topics recently, they are still insufficient to make natural conversations with consistent context, or with complex sentences. one of the fundamental issues is a lack of an appropriate evaluation. because the werewolf game forces players to deceive, persuade, and detect lies, neither inconsistent nor vague response are evaluated as "unnatural", losing in the game. our werewolf game competition and evaluation could be new interesting evaluation criteria for dialog systems, but also for imperfect information game theories. in addition, the werewolf game allows any conversation, so the game includes both task-oriented and non-task-oriented conversations. we have been holding an annual series of competition to automatically play the werewolf game since 2014 @xcite , as the aiwolf project 1 . our competitions were linked with other conferences such as the competitions in ieee conference on games (cog), anac (automated negotiating agents competition) @xcite in international joint conference on artificial intelligence (ijcai), computer entertainment developers conference (cedec), etc., in addition to our aiwolfdial 2019 workshop at inlg 2019 @xcite . these mean that our contests attract interests from communities of many areas including dialog system, language generation, task-and non-task-oriented conversations, imperfect information game, human-agent interactions, and game ai. we have been providing two divisions in the contests: the protocol division and the natural language division. the protocol division uses our original aiwolf protocol which is designed for simplified language specific to the werewolf game player agents. in the natural language division, player agents should communicate in the natural languages (english or japanese). the natural language division is simple and natural goal of our project, but very difficult due to its underlying complexity of human intellectual issues. we focus on this natural language division in this report. in the natural language division of our contest, we ask participants to make self-match games as preliminary matches, and mutual-match games as final matches. agents should connect to our server to match, i.e. participants can run their systems in their own servers even if they require large computational resources. the game logs are evaluated by human subjective evaluations. eight agents of seven teams participated in this aiwolfdial 2023 shared task. because our games are held by five players, we held a mutual match 1 http://aiwolf.org/ game in the japanese language by six agents from five teams, and another mutual match game in the english language by three teams. in the following sections, we explain the game regulations of the aiwolf natural language division in section 2, detailed system designs for each agent in section 3, results of subjective evaluations in section 4.2 followed by discussions in section 5, finally conclude this paper in section 6. this paper is jointly written by the organizers and the participants, i.e. section 3 is written by each participant, the other sections are by the organizers, thus "we" stand for the organizers except for i.e. section 3. ## werewolf game and shared task settings we explain the rules of the werewolf game in this section. while there are many variation of the were-wolf game exists, we only explain the our aiwolfdial shared task setting in this paper. ## participant systems six agents from five teams participated our shared task in the japanese language, which agent names are am, chatwolf, k2b ara, k2b shigemura, kanolab, super il). three agents from three temas participated in the english language, which agent names are howlgpt, miv, and kanolab, where kanolab is their english version of their original japanese agent. most of the agents used chatgpt in their system, while its usage is different between the agents; chatwolf uses another llm, am employed a rule-based system. we, the organizers, provided a template agent code in java and python, in addition to the server codes. we describe each participant system in an alphabetical order in the following subsections. where "we" stands for the corresponding participants, only limited in this section. ## subjective evaluation results all of our shared task runs are in a five players werewolf games as described earlier. our shared task runs were performed in self-matches and mutual matches. the same five player agents play games in the self-matches; different five player agents play games in the mutual-matches. the shared task reviewers are required to perform subjective evaluations based on game logs of these matches. the game logs will be available from the our website 13 . ## conclusion and future work we held our 5th annual aiwolf international contest to automatically play the werewolf game "mafia", where players try finding liars via conversations, aiming at promoting developments in creating agents of more natural conversations in higher level, such as longer contexts, personal relationships, semantics, pragmatics, and logics. we performed human subjective evaluations and detailed log analysis. we found that the entire system performance has largely improved over the our future work includes to reveal the capability of the llms, whether they can make the duality of the "liar", in other words, holding a "true" and a "false" circumstances of the agent at the same time, even holding what these circumstances look like from other agents, further reflecting such observations of other agents. this would be possible by introducing the "whisper" feature which communicates with the werewolves only, employing more than five players in a game. another interesting demonstration would be to mix a human player with machine agents. currently the llm based agents talk longer time than humans to reply, sometimes minutes, thus acceleration of the agent system responses is a technical issue in future.