Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games (2024)

Abstract: In this study, we explore the application of Large Language Models (LLMs) in Jubensha, a Chinese detective role-playing game and a novel area in Artificial Intelligence (AI) driven gaming. We introduce the first dataset specifically for Jubensha, including character scripts and game rules, to foster AI agent development in this complex narrative environment. Our work also presents a unique multi-agent interaction framework using LLMs, allowing AI agents to autonomously engage in Jubensha games. To evaluate the gaming performance of these AI agents, we developed novel methods measuring their mastery of case information and reasoning skills. Furthermore, we incorporated the latest advancements in prompting engineering to enhance the agents’ performance in information gathering, murderer identification, and logical reasoning. The experimental results validate the effectiveness of our proposed methods. This work aims to offer a novel perspective on understanding LLM capabilities and establish a new benchmark for evaluating large language model-based agents.

## introduction interactive role-playing games, where players unravel mysteries through strategic interactions and clue-based puzzles, have seen a significant rise in global popularity. tracing their origins to 19thcentury murder mystery novels and early 20thcentury party puzzles, these games have particularly flourished in china in recent years. in this context, they are referred to as "jubensha", "scripted murders" or 剧本杀 in chinese, and have experienced a remarkable surge in popularity since the late 2010s. influenced by western murder mystery games, jubensha games center around players who gather to identify the murderer by analyzing and interpreting the provided story scripts. the thrill for players to collect and interpret clues through intense social interaction and reasoning has made playing jubensha stand out as a notable cultural phenomenon. meanwhile, the field of artificial intelligence (ai) is reshaping the gaming landscape. from classic games like chess @xcite , @xcite , and poker @xcite to video games like @xcite , league of legends @xcite , and honor of kings (ye et al., 2020), ai contenders or collaborators have been integrated into these games. the recent surge in large language models (llms) has redirected research interest from conventional and video games to text-based games. significant examples include an llm deployed as agents in the strategic game @xcite , in the communicative game werewolf @xcite and @xcite @xcite , and in the adventure games such as @xcite and @xcite . however, the "jubensha game" remains an undeveloped field in terms of ai agents specifically tailored for its gameplay and evaluation. we believe there are several reasons. first, before the emergence of llms, understanding the character plots, role tasks, and game rules in the script of jubensha games was very challenging. not to mention the need for ai agents to engage in multi-round linguistic interactions, information gathering, and logical reasoning. second, currently there is no publicly available dataset specifically for jubensha games that researchers can use to develop and evaluate their agents. third, quantitatively and qualitatively automating the evaluation of ai agents' performance in jubensha games is also very difficult. the most commonly used win-rate metric in other game ais is of limited use in assessing the performance of jubensha game agents, i.e., win rate does not show the extent of the jubensha ai's mastery of case information or the level of its reasoning ability. in response to the above-mentioned challenges, we attempt to provide a solution in this work. this work focuses on constructing a multi-agent interaction framework in a "jubensha" game environment using large language models, and we have designed a set of methods to quantitatively and qualitatively assess the performance of large language model based (llm-based) agents. our contributions can be summarized as follows: interactive role-playing games interactive role-playing games (irpgs) provide immersive experiences in fictional settings, serving as a multidisciplinary research testbed in various fields @xcite @xcite . these games are categorized into single-player role-playing games (srpgs) and multi-player role-playing games (mrpgs), each with unique research opportunities. srpgs focus on character advancement through quests in various themes @xcite @xcite @xcite , while mrpgs highlight cooperation and confrontation between multiple characters @xcite @xcite @xcite @xcite @xcite @xcite . this work presents a dataset for mrpg jubensha, aimed at supporting studies on the development and evaluation of communicative and reasoning ai in an adversarial game setting. ## evaluating llm-based agents in jubensha games previous work has primarily employed metrics such as human-likeness and win rate to assess the performance of llm-based agents in games @xcite @xcite . these metrics either require substantial human involvement or likely leading to less reliable experimental conclusions due to the challenges in controlling variables (xu et al., 2023). 4 considering the unique characteristics of jubensha games, we have designed two tasks to quantitatively and qualitatively evaluate the performance of llm-based agents in jubensha games: factual question answering and inferential question answering. ## experiment in this section, we will briefly discuss the various llms used in this work. following this, we will present our experimental results, encompassing the evaluation of the information gathering and reasoning performance of llm-based agents. ## future work here we provide directions for future work: ## conclusion this work has explored the application of large language models in complex interactive environments, exemplified by the chinese detective role-playing game "jubensha". our research has yielded four main contributions: the creation of a specialized dataset tailored for the jubensha games, the design of a multi-agent interaction framework for the automatic conduct of jubensha games, the development of a set of quantitative and qualitative assessment methods to measure the information gathering and reasoning abilities of llm-based agents within the game, and the utilization of the latest prompting techniques to devise modules that enhance the performance of llm-based agents. we have empir-ically demonstrated that our designed multi-agent interaction framework and the prompting modules significantly improve upon the baseline in terms of information gathering, murderer identification, and reasoning capabilities. we believe this research will advance the community's knowledge of llmbased agents and offers a new perspective on evaluating the performance of llms in a complex, plot-driven, and adversarial reasoning game environment constrained by narrative contexts.