LLM -Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay (2024)

Abstract: This paper explores the open research problem of understanding the social behaviors of LLM-based agents. Using Avalon as a testbed, we employ system prompts to guide LLM agents in gameplay. While previous studies have touched on gameplay with LLM agents, research on their social behaviors is lacking. We propose a novel framework, tailored for Avalon, features a multi-agent system facilitating efficient communication and interaction. We evaluate its performance based on game success and analyze LLM agents’ social behaviors. Results affirm the framework’s effectiveness in creating adaptive agents and suggest LLM-based agents’ potential in navigating dynamic social interactions. By examining collaboration and confrontation behaviors, we offer insights into this field’s research and applications.

## introduction artificial intelligence (ai) agents @xcite exhibit human-like behaviors, from perceiving and analyzing the environment to decision-making and action-taking. advances in large language models (llms) @xcite @xcite offer new avenues for creating ai agents in complex environments, potentially simulating human society. various works @xcite @xcite simulate different aspects of human society. for instance, qian et al. @xcite simulate a software development company with agents representing diverse social identities. @xcite assign varied social roles to agents within a sandbox environment. however, prior studies mostly examine positive social behaviors like honesty and collaboration, leaving research on negative social behaviors of llm agents relatively scarce. previous research on human society has highlighted issues like misinformation and online conflicts, leading to efforts to address these problems @xcite @xcite . to delve deeper into the social behaviors of llm agents, we intend to comprehensively investigate both positive and negative aspects of their conduct. to achieve this, we employ avalon as the environment to illustrate collaboration and confrontation among agents. avalon, a representative social deduction game, assigns players hidden roles and divides them into opposing teams. throughout gameplay, players partake in discussions, debates, and strategic maneuvers. llm agents face a challenging task in winning the incomplete information game of avalon. they need to share and obtain information via communication and analysis, deducing other players' roles, building trust among allies, and deceiving opponents. success requires technical abilities like natural language understanding, incomplete information analysis, and strategy learning. additionally, social behaviors such as teamwork, persuasion, and camouflage are crucial for success in avalon gameplay. to investigate the llm-based agent society, we propose a novel framework for the agents to play avalon. specifically, we adopt chatgpt as the players and assign various roles to agents. we adopt system prompts to guide llm agents to play avalon automatically. following human's thinking methodology, we incorporate multiple modules, including memory storage and summarization, analysis and planning, game action and response generation, and experience learning. we utilize a competitive baseline approach @xcite , to elaborate the efficacy of our proposed framework. we also carefully ## background in our study, we chose avalon, also known as "the resistance", instead of werewolf as our environment. unlike werewolf, where players are gradually eliminated, avalon ensures that all players remain engaged throughout the game, promoting social cohesion. avalon accommodates 5 to 10 players, focusing on the 6-player variant herein. players receive secret roles in either the good or evil faction. the good faction includes merlin, percival, and loyal servants, while the evil faction comprises morgana and assassin. morgana and assassin know each other's identities, percival can identify merlin and morgana, and merlin recognizes all evil players. the game spans 3-5 rounds. players discuss and vote to form a quest team of 2-3 members. approval requires a majority vote; otherwise, leadership shifts. each round allows up to five voting cycles before the leader selects the team. quest success hinges on cards submitted by team members. good players submit success cards, while evil players can choose success or failure cards. a quest fails if it receives a failure card. the game concludes with victory for good players if three quests succeed, or for evil players if three quests fail. evil players can also win by correctly identifying merlin at the game's end. ## social behaviors of ai agents to evaluate if ai agents replicate human social behaviors in avalon, we conduct a thorough analysis. this involves assessing the agents' execution of teamwork, leadership, persuasion, camouflage, and confrontation through the frequency distribution in game logs from two sets of 10 consecutive games. ## conclusion this paper explores the social behaviors of llmbased agents in the avalon game. we introduce a multi-agent framework facilitating efficient communication and interaction. this framework includes memory, analysis, planning, action, and response modules capable of learning from experience. unlike prior studies, our research delves into the social dynamics of these agents in gameplay scenarios. our evaluation showcases the success of our framework in achieving winning strategies and the adaptability of llm agents in complex social interactions. future work involves optimizing our approach, exploring its applicability in diverse game environments, and further understanding llm agents' potential in dynamic social interactions.