Theory of Mind for Multi-Agent Collaboration via Large Language Models (2023)

Abstract: While Large Language Models (LLMs) have demonstrated impressive accomplishments in both reasoning and planning, their abilities in multi-agent collaborations remains largely unexplored. This study evaluates LLM-based agents in a multi-agent cooperative text game with Theory of Mind (ToM) inference tasks, comparing their performance with Multi-Agent Reinforcement Learning (MARL) and planning-based baselines. We observed evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. Our results reveal limitations in LLM-based agents’ planning optimization due to systematic failures in managing long-horizon contexts and hallucination about the task state. We explore the use of explicit belief state representations to mitigate these issues, finding that it enhances task performance and the accuracy of ToM inferences for LLM-based agents.

## introduction recent large language models (llms), such as @xcite , have demonstrated impressive competencies across a wide array of domains and tasks, ranging from mathematics to law, without the need for fine-tuning or special prompting @xcite . this advancement has significantly transformed the landscape of natural language processing (nlp) research. instead of developing domain-specific models for downstream applications, focus has shifted towards evaluating and harnessing llms' abilities to solve novel tasks. such a shift is consistent with the idea of studying machine behaviors, an interdisciplinary approach that expands the conventional bounds of computer science and integrates insights from diverse scientific fields @xcite . drawing inspiration from team science and group psychology @xcite , our study concentrates on collective machine behavior, evaluating llms' proficiency in multi-agent collaborations. there is ongoing debate regarding the intelligence levels of modern llms. while some argue that llms excel primarily in linguistic competence and struggle with cognitive abilities beyond language, known as functional competence, others demonstrate that llms can exhibit cognitive skills such as formal reasoning and world knowledge comprehension @xcite . motivated to explore this argument, we designed a text-based game to evaluate llms' ability in embodied interactions, including exploring unknown environments, maintaining beliefs about the world and collaborating with other agents, which is critical for natural social interactions and artificial general intelligence (agi). theory of mind, the capacity to reason about others' concealed mental states, is fundamental to human social interactions, collaborations, and communications @xcite . as llms increasingly participate in diverse social interactions with humans, their social intelligence is expected to improve for them to become effective collaborators @xcite . for instance, a proficient ai assistant should be able to infer a human's preferences based on previous experiences without needing to ask. recent studies have applied classic theory-of-mind tasks to several llms, concluding that current models (e.g., gpt-4) perform comparably to 9-yearold children @xcite . however, the research community has expressed doubts about the validity of text-based tom tests on machine intelligence @xcite . in response, our study proposes a novel evaluation of llms' high-order tom in interactive teamwork scenarios, encompassing dynamic belief state evolution and rich intent communication between multiple agents. the main contributions of this study include that we: ## multi-agent collaboration tasks to evaluate the capability of llm-based embodied agents, we design a multi-agent environment to simulate the collaborative and problem-solving dynamics of a search and rescue mission. ## llm-based embodied agents we chose to evaluate openai's latest chat completion models, namely gpt-3.5-turbo-0301 and gpt-4-0314, owing to their impressive performance in various benchmarks @xcite . these models are prompted to engage in a text-based game, with user inputs managed by the above-mentioned game interface. the llms functions as embodied agents interacting within the task environment. they are provided with the game's rules as context. for each round, the model is asked to choose actions and communicate messages, based on the current task state observations and past interaction history. interaction history between the llm-based agent and text game interface are maintained in the query text until it exceeds the maximum model input size. in our setup, all agents retain memory of the game rules and history from the previous two rounds, amounting to 4096 tokens. ## experiments we systematically ablate llm-based embodied agents and evaluate them in a collaborative task in teams of three. two modules are manipulated including llm models (i.e. gpt-4 or chatgpt) and belief representation (i.e. with or without belief state) resulting in a total of 4 experimental conditions. ## results ## discussions our study yields three primary insights. first, large language models (llms) demonstrate substantial planning and collaboration capabilities within our task scenarios. with suitable promptengineering, teams of llm-based agents perform comparably to state-of-the-art multi-agent reinforcement learning (marl) algorithms. this finding is particularly noteworthy given that marl agents receive extensive task-specific training with a centralized critic, while llm-based agents operate in a fully decentralized manner and undertake tasks in a zero-shot setting. despite prior research highlighting llms' limitations in generating actionable plans and interacting with the world, they perform reasonably well when placed in a team and tasked to process actions step-by-step. particularly, llms fine-tuned with reinforcement learning from human feedback demonstrate emergent social interaction skills in multi-agent collaborations, which might be similar to the collaborative and interactive settings in which human language is primarily learned and used @xcite . second, llms still fall short of being optimal planners or team players due to systematic failures, such as neglecting long-horizon contexts and making inaccurate assumptions about the task state (a.k.a hallucination). these flaws significantly hinder team collaborations as they can rapidly disseminate misinformation via communication, leading to widespread false beliefs. we attempted to mitigate these issues by allowing llm-based agents to maintain an explicit belief state about the world. our findings suggest that modern llms can update the given belief descriptions based on their observations, hinting at the potential emergence of advanced cognitive skills such as world knowledge understanding and situation modeling. moreover, belief state representations offer a structured framework that helps agents track key task-related information, leading to improved team performance. finally, our study indicates that the theory of mind (tom) capabilities of llms are still limited, particularly when evaluated within interactive teamwork scenarios that involve dynamic belief states and intensive communication. for context, while 5-year-old children can perform second-order tom inferences @xcite , adults don't consis-tently use this ability during communications due to the complexity and ambiguity of social interactions @xcite . thus, there's considerable work ahead for llms to develop a functional tom and interact naturally with humans. our study represents a preliminary effort to devise novel evaluation methods for llms' tom that go beyond traditional tests such as the sally-anne test. ## conclusions in this study, we assessed the ability of recent large language models (llms) to conduct embodied interactions in a team task. our results demonstrate that llm-based agents can handle complex multi-agent collaborative tasks at a level comparable with the state-of-the-art reinforcement learning algorithm. we also observed evidence of emergent collaborative behaviors and high-order theory of mind capabilities among llm-based agents. these findings confirm the potential intelligence of llms in formal reasoning, world knowledge, situation modeling and social interactions. furthermore, we discussed two systematic failures that limit the performance of llm-based agents and proposed a prompt-engineering method that mitigates these failures by incorporating an explicit belief state about world knowledge into the model input.