LLMA rena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments (2024)

Abstract: Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings.

## introduction recent large language models (llms) have greatly promoted the progress of the field of natural language processing (nlp) due to their excellent zeroshot capabilities in various downstream tasks @xcite @xcite @xcite . llms not only excel in comprehending and generating intricate text but also demonstrate remarkable adaptability, even without training tailored to specific tasks @xcite . they possess the ability to analyze, strategize, and respond effectively to unfamiliar scenarios using minimal guiding prompts. this has opened new avenues for researchers to consider llms as autonomous agents, providing automated assistance in complex real-world tasks such as software development and knowledge integration @xcite . to better understand the skills needed by llm as an agent, researchers are now concentrating on creating and using various scenarios to assess llm's performance in specific settings @xcite @xcite @xcite . for instance, agentbench introduced eight distinct environments, such as operating systems and databases, to examine llm's proficiency in code generation and lateral thinking. similarly, smartplay tests llm in a gaming context as a solo agent to gauge its reasoning and strategic planning abilities. nonetheless, these benchmarks have their constraints. firstly, static datasets used in benchmarks like agentbench might lead to issues such as data leakage and overfitting, as llms could have already encountered this data during their pre-training phase @xcite . secondly, evaluation systems that focus solely on a single agent also have evident limitations, particularly in analyzing an agent's performance in intricate and dynamically interactive settings. such single-agent environments tend to concentrate on the interactions between the agent and its environment, thereby overlooking the complex dynamics of interactions that occur among multiple agents in a shared environment @xcite . to this end, we proposed llmarena, a dynamic evaluation benchmark specially designed for multi-agent interaction. as illustrated in figure 1, llmarena covers seven different types of dynamic, multi-agent game environments. for example, the texas hold'em poker game environment automatically generates a new hand for each game, and the complexity of each round's state increases exponentially. this dynamic nature of the game serves to mitigate the issue of data leakage. in the undercover environment, the llm agents need to point out the "undercover" during the communication process with other agents, which measures the llm agents' communication ability and opponent modeling ability when facing different agents. based on these environments, we can comprehensively evaluate llm's spatial comprehension, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration capabilities in the dynamic multi-agent environment. to more accurately evaluate llm's performance in these multi-agent environments, we adopted the trueskill™ @xcite scoring system, which can better help llmarena evaluate other metrics besides win rate, such as the relative skill level between agents, to provide a more comprehensive assessment under different scenarios and opponent configurations. we conduct an extensive experiment and human evaluation among 14 different sizes and types of llms. the results indicated significant potential for improvement in llm's teamwork and opponent modeling skills within multi-agent environments. based on these results, we hope llmarena can inspire future research focused on enhancing the core capabilities of llm in multi-agent settings to promote the widespread use of llm agents in real-world applications. ## benchmark detail in this section, we provide a detailed overview of llmarena's composition, which includes its seven distinct game environments, evaluation metrics, and the methodologies used for assessment. ## analysis in this section, we will conduct a thorough analysis of the experimental results, evaluating llms from seven key perspectives: spatial comprehension, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. we chose to experiment with 7 llms that demonstrate superior performance. ## conclusion in our study, we introduced llmarena, a benchmark designed to assess the diverse abilities of llm agents in dynamic, multi-agent environments. through comprehensive analysis of seven game environments that require necessary abilities for llm agents, our findings reveal that llms exhibit weaknesses in spatial reasoning, opponent modeling, and team collaboration. enhancing the performance of llm agents in dynamic multiagent settings remains an unresolved challenge. we anticipate that future researchers will utilize llmarena to conduct evaluations in a broader range of scenarios.