AIW olf D ial 2024: Summary of Natural Language Division of 6th International AIW olf Contest (2024)

Abstract: We held our 6th annual AIWolf international contest to automatically play the Werewolf game “Mafia”, where players try finding liars via conversations, aiming at promoting developments in creating agents of more natural conversations in higher level, such as longer contexts, personal relationships, semantics, pragmatics, and logics, revealing the capabilities and limits of the generative AIs. In our Natural Language Division of the contest, we had eight Japanese speaking agent teams, and five English speaking agents, to mutually run games. By using the game logs, we performed human subjective evaluations, win rates, and detailed log analysis. We found that the entire system performance has largely improved over the previous year, due to the recent advantages of the LLMs. There are several new ideas to improve the way using LLMs such as the summarization, characterization, and the logics outside LLMs, etc. However, it is not perfect at all yet; the generated talks are sometimes inconsistent with the game actions. Our future work includes to reveal the capability of the LLMs, whether they can make the duality of the “liar”, in other words, holding a “true” and a “false” circumstances of the agent at the same time, even holding what these circumstances look like from other agents.

## introduction recent achievements of generation models, e.g. chatgpt @xcite , are gathering greater attentions. however, it is not fully investigated whether such a huge language model can sufficiently handle coherent responses, longer contexts, * correspondence to kano@kanolab.net common grounds, and logics. our shared task, aiwolfdial 2024, is an international open contest for automatic players of the conversation game "mafia", which requires players not just to communicate but to infer, persuade, deceive other players via coherent logical conversations, while having the role-playing non-task-oriented chats as well. aiwolfdial2024 is one of the workshops of 17th international natural language generation conference @xcite . we believe that this contest reveals not just achievements but also current issues in the recent huge language models, showing directions of next breakthrough in this area. we propose to employ this werewolf game as a novel way of evaluations for dialog systems. while studies of dialog systems are very hot topics recently, they are still insufficient to make natural conversations with consistent context, or with complex sentences. one of the fundamental issues is a lack of an appropriate evaluation. because the werewolf game forces players to deceive, persuade, and detect lies, neither inconsistent nor vague re-sponse are evaluated as "unnatural", losing in the game. our werewolf game competition and evaluation could be new interesting evaluation criteria for dialog systems, but also for imperfect information game theories. in addition, the werewolf game allows any conversation, so the game includes both task-oriented and non-task-oriented conversations. we have been holding an annual series of competition to automatically play the werewolf game since 2014 @xcite , as the aiwolf project 1 . our competitions were linked with other conferences such as the competitions in ieee conference on games (cog), anac (automated negotiating agents competition) @xcite in international joint conference on artificial intelligence (ijcai), computer entertainment developers conference (cedec), etc., in addition to our aiwolfdial 2019 workshop at inlg 2019 @xcite and aiwolf-dial2023 at inlg 2023 @xcite . these mean that our contests attract interests from communities of many areas including dialog system, language generation, task-and non-task-oriented conversations, imperfect information game, humanagent interactions, and game ai. we have been providing two divisions in the contests: the protocol division and the natural language division. the protocol division uses our original aiwolf protocol which is designed for simplified language specific to the werewolf game player agents. in the natural language division, player agents should communicate in the natural languages (english or japanese). the natural language division is simple and natural goal of our project, but very difficult due to its underlying complexity of human intellectual issues. we focus on this natural language division in this report. in the natural language division of our contest, we ask participants to make self-match games as preliminary matches, and mutual-match games as final matches. agents should connect to our server to match, i.e. participants can run their systems in their own servers even if they require large computational resources. the game logs are evaluated by human subjective evaluations. eight agents (eight teams) participated in this ai-wolfdial 2024 shared task, where eight teams provided japanese language versions and five teams provided english language versions. because our games are held by five players, we held a mutual 1 http://aiwolf.org/ match game in the japanese language by eight agents from five teams, and another mutual match game in the english language by five teams. in the following sections, we explain the game regulations of the aiwolf natural language division in section 2, detailed system designs for each agent in section 3, results of subjective evaluations in section 4.1 followed by discussions in section 5, finally conclude this paper in section 6. this paper is jointly written by the organizers and the participants, i.e. section 3 is written by each participant, the other sections are by the organizers, thus "we" stand for the organizers except for i.e. section 3. ## werewolf game and shared task settings we explain the rules of the werewolf game in this section. while there are many variation of the werewolf game exists, we only explain the our aiwolfdial shared task setting in this paper. ## game server and participant systems eight agents from eight teams participated our shared task in the japanese language, which agent names are gptaku, hondanlp, is lab, kanolab, mille, satozaki, super il, and uec-il. five agents from five temas participated in the english language, which agent names are is lab, kanolab, satozaki, super il, and uec-il. most of the agents used chatgpt and other llms in their system, while its usage is different between the agents. we, the organizers, provided a template agent code in java and python, in addition to the server codes described in the following subsection. we describe each participant system in an alphabetical order in the following subsections, after the game server description. these participant system descriptions are based on the system descriptions and papers submitted by the participants. ## results all of our shared task runs are in a five players werewolf games as described earlier. our shared task runs were performed in self-matches and mutual matches. the same five player agents play games in the self-matches; different five player agents play games in the mutual-matches. the shared task reviewers are required to perform subjective evaluations based on game logs of these matches. we also calculated win rates in different aspects such as macro-averaged, micro-averaged, and rolewise, though the total number of the games are not so large which could make these statistics unreliable to some extent. the game logs will be available from the our website 2 . ## conclusion and future work we held our 5th annual aiwolf international contest to automatically play the werewolf game "mafia", where players try finding liars via conversations, aiming at promoting developments in creating agents of more natural conversations in higher level, such as longer contexts, personal relationships, semantics, pragmatics, and logics. we performed human subjective evaluations and detailed log analysis. we found that the entire system performance has largely improved over the previous year, due to the recent advantages of the llms. however, it is not perfect at all yet; the generated talks are sometimes inconsistent with the game actions, it is still doubtful that the agents could infer roles by logics rather than superficial utterance generations. it is not explicitly observed in this log but it would be still difficult to make an agent telling a lie, pretend as a villager but it has an opposite goal inside. our future work includes to reveal the capability of the llms, whether they can make the duality of the "liar", in other words, holding a "true" and a "false" circumstances of the agent at the same time, even holding what these circumstances look like from other agents, further reflecting such observations of other agents. this would be possible by introducing the "whisper" feature which communicates with the werewolves only, employing more than five players in a game. another interesting demonstration would be to mix a human player with machine agents. currently the llm based agents talk longer time than humans to reply, sometimes minutes, thus acceleration of the agent system responses is a technical issue in future.