MA g IC : Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration (2024)

Abstract: Large Language Models (LLMs) have significantly advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities. As their applications expand into multi-agent environments, there arises a need for a comprehensive evaluation framework that captures LLMs’ reasoning, planning, collaboration, and other social abilities. This work introduces a novel competition-based benchmark framework specifically designed to assess LLMs within multi-agent settings, providing quantitative metrics to evaluate their judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality.We utilize two social deduction games alongside three game-theory scenarios to create diverse environments.Our frame is fortified with the probabilistic graphic modeling (PGM) method, enhancing the LLMs’ capabilities in navigating complex social and cognitive dimensions. We evaluate seven LLMs, quantitatively highlighting a significant capability gap of over threefold between the strongest, GPT o1, and the weakest, Llama-2-70B. It also confirms that our PGM enhancement boosts the abilities of all selected models by an average of 37%. Our data and code can be found here https://github.com/cathyxl/MAgIC.

## introduction large language models (llms), particularly chat-gpt and @xcite , have showcased impressive understanding and generation capabilities. beyond these fundamental abilities, llms also demonstrate promising capabilities in anthropic areas such as reasoning @xcite , planning @xcite , tool usage @xcite , and memorization @xcite . win rate area area figure 1 : the radar chart depicts llms' performance on 7 metrics, with "-t" for "-turbo" and "+p" for "+pgm". the bar chart displays the polygons' areas, and the red line indicates average game-winning rates. larger areas correlate with higher winning rates, validating the effectiveness of the proposed metrics for assessing llms' capabilities. for more information, refer to sec. 5. there is an increasing interest in investigating llms' behaviors as agents in single-or multipleagent systems. noteworthy examples include generative agents @xcite , camel @xcite , auto-gpt (richards, 2023), and voyager @xcite . meanwhile, quantitative assessment of llms as agents is crucial for their advancement. recent benchmarks, such as @xcite , evaluate llm-as-agent in multi-turn contexts, while concurrent work by @xcite tests them in games requiring reasoning and planning. however, these studies focus on understanding and reasoning in environments, overlooking true interaction capabilities in multi-agent systems. other research, including @xcite social aspects on coordination, and @xcite on bargaining, explores specific skills in multi-agent scenarios. while these studies provide useful insights into llms' certain capabilities, their scope is limited and lacks quantitative metrics. we have observed three key characteristics in interactive multi-agent systems. (1) agents in these systems often operate within the confines of their local perspectives. however, making wise decisions typically necessitates a good understanding of global information. to overcome this limitation, agents must adeptly discern contexts and reason about the roles or plans of other agents. (2) contexts are inherently dynamic in multi-agent systems due to the dependent nature of agents' decisions. success hinges on the ability to swiftly adapt strategies in response to evolving contexts. (3) collaboration and competition are inevitable when multiple agents try to solve tasks together. the ability to promote cooperation while preserving self-interest is often the ultimate goal of multi-agent systems. inspired by the above characteristics, we first propose a competition-based benchmark to evaluate the abilities of llms as agents by competing with a fixed type of llm. besides, seven quantitative metrics from the competitions are proposed to measure the essential capabilities of llms @xcite . we define these capabilities from four aspects: cognition, adaptability, rationality, and collaboration: (1) judgment and reasoning form the core cognition of agents, crucial for accurate information estimation in uncertain scenarios. judgment evaluates the ratio of the final correct decisions. reasoning measures the ability to logically analyze other agents' roles and strategy formulation, thus guiding agents to make correct decisions in uncertainty. in light of the essential abilities, we further propose a method to enhance llms as agents by integrating bayesian statistical foundations. this novel approach intertwines the probabilistic graphical model(pgm) @xcite with llms, thereby amplifying their capacity to comprehend intricate scenarios and enabling more informed and strategic decision-making in multiagent environments. in summary, our contributions are as follows: the results indicate that gpt o1, gpt-4, and gpt-3.5 remain the superior performers, followed by other commercial llms -palm 2, claude 2, and cohere. different large language models (llms) exhibit varying performance levels across different evaluation dimensions and possess distinct characteristics. for instance, gpt-o1 is more discernible with a good judgment score, gpt-4 tends to be more rational, whereas gpt-3.5 is generally more cooperative, as shown in figure 1 . ## pgm-aware agent in ai, bayesian methods embody symbolism, while large language models (llms) exemplify connectionism. despite their individual strengths, effectively combining these approaches remains a challenge. llms are proficient in complex language tasks but still struggle with ambiguous relationships and causal reasoning. this shortcoming is especially evident in multi-agent scenarios requiring complex inferential analysis. to address this, we propose integrating probabilistic graphical models (pgms), classic bayesian tools adept at depicting dependencies between random variables, to enhance llms' analytical and inferential capabilities. ## experiments in experiments, we make each challenger llm play with the same defender llms(gpt-4 as we used), and rank them by the wining rate. to reduce the randomness during game, we set the temperature of all participating llms as 0. all the code and data will be publicly released upon acceptance. ## discussion: generalization of benchmark beyond the scenarios mentioned in this paper, our benchmark can be generalized to more scenarios or tasks. in general, it evaluates agent behaviors in settings where the participants of a multi-agent system are usually local-viewed and need abilities involving cognition, adaptability, rationality, and collaboration to deduce the global information and make decisions, thus achieving the final goal. judgment evaluates an agent's ability to accurately assess unknown information, such as roles in games like 'chameleon' and 'undercover'. reasoning checks if an agent's perspective aligns with the actual and others' views, offering a nuanced understanding. self-awareness adapts to scenarios with undisclosed roles, while deception looks at how well an agent can influence others with false information. cooperation and coordination gauge the effectiveness of collaborative efforts, measuring agreement and the quality of proposals, respec-tively. lastly, we introduce rationality from game theory, defining it as the proportion of decisions that maximize an agent's outcomes. ## conclusion our research presents a benchmarking framework tailored for evaluating llms in multi-agent environments. this framework's incorporation of diverse scenarios has enabled a quantitative assessment of seven critical abilities for llms in multi-agent systems, including judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality. the integration of pgm enriches llms with structural reasoning ability in multi-agent scenarios. our quantitative analysis of 7 different multi-agent systems powered by various llms, including gpt-4-turbo, gpt-4, gpt-3.5-turbo, palm 2, claude 2, cohere, and llama2-70b, has revealed their capabilities' disparity. notably, gpt-4-turbo still emerged as the most capable, outperforming others by a threefold margin. moreover, the pgm enhancement amplifies the inherent abilities of these models by 37%. this shows our benchmark's effectiveness and pgm's potential to enhance llm capabilities.