LLM s P ark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts (2025)

Abstract: As large language models (LLMs) advance across diverse tasks, the need for comprehensive evaluation beyond single metrics becomes increasingly important.To fully assess LLM intelligence, it is crucial to examine their interactive dynamics and strategic behaviors.We present LLMsPark, a game theory–based evaluation platform that measures LLMs’ decision-making strategies and social behaviors in classic game-theoretic settings, providing a multi-agent environment to explore strategic depth.Our system cross-evaluates 15 leading LLMs (both commercial and open-source) using leaderboard rankings and scoring mechanisms. Higher scores reflect stronger reasoning and strategic capabilities, revealing distinct behavioral patterns and performance differences across models.This work introduces a novel perspective for evaluating LLMs’ strategic intelligence, enriching existing benchmarks and broadening their assessment in interactive, game-theoretic scenarios.The benchmark and rankings are publicly available at https://llmsparks.github.io/.

## introduction with the rapid rise of large language models (llms) and large multimodal models (lmms), their performance on complex tasks such as code generation @xcite , recommender systems @xcite , and knowledge-intensive question answering has exceeded expectations, reshaping the landscape of natural language processing. beyond text, these models demonstrate broad applicability in text-guided generation and editing of images @xcite @xcite , 3d models @xcite @xcite @xcite , as well as audio and video understanding and editing @xcite @xcite @xcite . the release of @xcite and the rapid development of open-source models such as llama2 @xcite and chatglm2 @xcite have further accelerated this progress. in realworld applications-ranging from question answering @xcite , natural language inference @xcite , and text summarization @xcite to sentiment classification @xcite )-the performance of llms is now approaching, and in some cases rivaling, human-level abilities. they also exhibit strong competence in mathematical problem solving @xcite , logical reasoning @xcite , and even single-player games. however, evaluating the capabilities of llms remains a significant challenge. current evaluation paradigms are dominated by static benchmarks and large-scale knowledge-intensive datasets, such as mmlu @xcite , mtbench @xcite , chatbot arena @xcite , zhujiu @xcite , and ope-nai evals @xcite . while informative, these benchmarks primarily assess factual recall or task-specific performance, offering limited insights into interactive reasoning and adaptive behaviors. recent efforts have begun to extend evaluation into dynamic and agentic settings. for example, tools such as xagent @xcite and @xcite test llms in autonomous workflows, while social games are increasingly recognized as promising testbeds for examining ai decision-making and strategic behavior @xcite . llm-based agents have also been deployed in interactive environments, including generative social simulations @xcite , open-world games like minecraft @xcite , and multiagent role-playing games such as werewolf @xcite . recent studies have also evaluated llms in complex visual games such as smart- ## conclusion and future work in this study, we evaluated 15 llms across five representative games, including the prisoner's dilemma and who is spy, to analyze how parameter size and model design influence strategic performance. while models exhibited diverse strategic behaviors, key challenges remain in effective knowledge utilization and standardized evaluation. the proposed llmspark benchmark is modular and scalable, enabling seamless integration of new games and strategies. future work will focus on three directions: (1) enhancing models' ability to leverage historical game experience and incorporate human-like learning, (2) establishing consistent baseline methodologies for cross-game evaluation, and (3) reducing potential errors when generalizing from controlled theoretical settings to real-world applications. we also plan to expand the benchmark with additional games while maintaining adaptability to emerging llms and evolving strategies.