UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models (2024)

Abstract: Sequential decision-making refers to algorithms that take into account the dynamics of the environment, where early decisions affect subsequent decisions. With large language models (LLMs) demonstrating powerful capabilities between tasks, we can’t help but ask: Can Current LLMs Effectively Make Sequential Decisions? In order to answer this question, we propose the UNO Arena based on the card game UNO to evaluate the sequential decision-making capability of LLMs and explain in detail why we choose UNO. In UNO Arena, We evaluate the sequential decision-making capability of LLMs dynamically with novel metrics based Monte Carlo methods. We set up random players, DQN-based reinforcement learning players, and LLM players (e.g. GPT-4, Gemini-pro) for comparison testing. Furthermore, in order to improve the sequential decision-making capability of LLMs, we propose the TUTRI player, which can involves having LLMs reflect their own actions with the summary of game history and the game strategy. Numerous experiments demonstrate that the TUTRI player achieves a notable breakthrough in the performance of sequential decision-making compared to the vanilla LLM player.

## introduction in artificial intelligence, sequential decisionmaking refers to algorithms that take the dynamics of the world into consideration @xcite , and it can be described as a procedural approach to decision-making, or as a step by step decision theory. as a consequence, sequential decision-making has the intertemporal choice problem, where earlier decisions influences the later available choices @xcite . in recent years, large language models (llms) are gaining increasing popularity in both academia and industry, owing to their unprecedented performances in various applications @xcite , ranging from chatbots to medical diagnoses @xcite to robotics @xcite . from robots handling complex tasks @xcite to entrepreneurial action @xcite , sequential decision-making permeates diverse domains. hence, an interesting question arises: can current llms make sequential decisions effectively? to answer this question, we need to design a benchmark to evaluate the sequential decisionmaking ability of llms. however, evaluating llms' abilities is not trivial. many studies have been proposed to test llms' performances on either a large-scale static benchmark such as mmlu @xcite , or with a/b tests judged by humans @xcite . one common and evident limitation of these methods, however, is that the environment for llms to be tested is static @xcite , which can not reflect the domino effect in sequential decision-making. besides, data contamination @xcite @xcite , which means the inclusion of test data examples and labels in the pre-training data, also challenges the efficacy of these static benchmarks in differentiating model capabilities. unlike static evaluation, dynamic evaluation by treating llms as game-playing agents attracted more and more attention of researchers recently, such as beauty contests and private-value second price auctions @xcite , warewolf @xcite , avalon @xcite and leduc hold'em @xcite . however, current attempts do not account for sequential decision-making, and these games are either challenging to evaluate for intermediate results (such as werewolf) or have too few decision points per round (such as leduc hold'em). meanwhile, we should also note that studies of dynamically evaluating sequential decision-making capability in the reinforcement learning, such as games like go @xcite , dou di zhu @xcite , and mahjong @xcite . however, these games present an excessively large action space. for instance, in dou di zhu, players can use any combination of their cards in each round, posing significant challenges for current llms @xcite . considering the above aspects, we make the following efforts in this paper: first, we build the uno arena to dynamically evaluate the sequential decision-making capability of current llms. in the uno arena, we allow llms to participate as players in the uno game 2 , aiming to play all the cards in their hand as quickly as possible. compared to games like leduc hold'em, which have fewer moves per game, uno features an average of dozens of moves per game, making it an ideal testbed for sequential decisionmaking @xcite . additionally, unlike common games in the reinforcement learning, legal actions in the uno arena are limited, only including drawing cards, playing cards, selecting colors to convert, and choosing whether to challenge the wild draw four card. furthermore, to monitor the behaviours of llms in the uno arena, we propose some real-time quantitative evaluation metrics by leveraging the monte carlo method (kroese et al., 2014) as the reference, which provide a window to observe the intermediate results and various phenomenons (like domino effect) in llms' se-2 https://en.wikipedia.org/wiki/uno_(card_game) quential decision-making. second, based on the proposed uno arena, we set up a family of strong and representative players. in detail, we first build the random player, which makes decisions based on chance rather than a specific, consistent plan, without considering the game's current state or potential outcomes. then, we built heuristic players based on a conservative heuristic uno play strategy and implement the reinforcement learning based player, which leverages dqn @xcite to develop sophisticated strategies for playing uno. finally, to probing the capability of llms in sequential decisionmaking, we provides the task description and then prompt llms, like gpt-4 @xcite and gemini-pro @xcite , to generate their reasoning steps that lead to the final action. third, to unleash the fully potential capability of llms in sequential decision-making, we propose the tutri player with reflection mechanism @xcite , which can enable llms to analyze their own actions based on the game history and the game strategy. in detail, the proposed agent framework consists of two key reflection modules: the game history reflection module and the game strategy reflection module. in the game history reflection module, we provide the statistical data of game history and then prompt the llms to rethink their decision, which simulates the process of card memorization by humans when playing uno. in the game strategy reflection module, llms further take into account the game strategy, like saving wild draw four, and proceed to make the final decision, which simulates the use and adherence to strategies by humans when playing uno. in the experiment, we comprehensively evaluate some mainstream llms' ability of sequential decision-making, including gpt-3.5 (ope-nai, 2022), gpt-4 @xcite , geminipro @xcite , llama 2 @xcite , and chatglm3 @xcite . our experimental results show that among these llms, gpt-4 is the most effective sequential decision-maker. in summary, our contributions are as follows: evaluate llms dynamically with game: llms have presented increasingly emerging ability on game-playing (brookins and debacker, 2023; akata et al., 2023) in recent development and iterations. wang et al. (2023b) use the avalon, which contains elements of deception, to evaluate the capability of llms to recognize and handle deceptive information. gong et al. (2023) leverage the cuisine world and minecraft to assess the planning and emergency cooperation capabilities of llms. guo et al. (2024a) employ beauty contests and auction games to evaluate the rationality, strategic reasoning capability, and adherence to instructions of llms. xu et al. (2023) use the werewolf game to evaluate the capability of llms to infer player roles. despite evaluating with game becoming a popular trend, exploring into sequential decision-making capability is still of scarcity in current works. development of agent framework: llm agents have been perceived as a promising way to realiz-ing artificial general intelligence(agi) @xcite and recently have shown emergent abilities to execute various tasks in the complex environment @xcite . sillm @xcite merges large language models with synchronous machine translation, using policy decision agents and translation agents. llm-vectorizer @xcite uses multiple agents to generate vectorized code by leveraging large language models and test-based feedback. we tailor a special framework for uno in this paper, featuring self-refinement and iterative thinking. sequential decision-making capability: sequential decision-making refers to the process of making a series of decisions over time, where each decision may impact future choices and outcomes @xcite . though certain algorithms or reinforcement learning provide solutions for some sequential decision-making problems @xcite , llmbased sequential decision-making are only employed in limited field like recommendation @xcite . in our work, we utilize the uno game, which is not an easy one even for human @xcite , to explore the sequential decision-making ability of llms. with certain methods like integrating past experiences and expert advice or demonstrations @xcite , we make efforts to maximally leverage the decision making ability as possible in a sequential manner. ## the uno arena in this section, we first provide a brief overview of the version of uno we adopt in the subsection §3.1. then, we present the four different types of players in the uno arena in the subsection §3.2. next, we detail how to use monte carlo methods to determine whether a player has made an optimal decision in subsection §3.3. in the end, we introduce our evaluation metrics in subsection §3.4. ## experiments in this section, we first conduct preliminary experiments with vanilla llm players, rl players, and random players in subsection §4.1. then, we have multiple different llm-based vanilla players compete in uno arena to identify the best llm in subsection §4.2. next, we test the superiority of the tutri players compared to the vanilla llm players in subsection §4.3. finally, we perform ## conclusion in conclusion, llms possess the capability for sequential decision-making, as evidenced by the experimental results of llms playing the uno game. our proposed uno arena and unique evaluation metrics enable llms to compete with each other in the same uno arena game, thereby providing a better dynamic assessment of llms' sequential decision-making abilities. furthermore, we propose that the tutri player effectively addresses how to enhance llms' sequential decision-making abilities for better performance in playing uno arena. in the modified uno game, we simply set the game objective as to be the first player to clear out the hand. players play alternatively(2 players) or in circle manner(3 or more players) and strive to achieve the unique goal. it should be noted that if the cards in the deck are exhausted by players, the player with the fewest number of cards in hand wins, so there may be multiple winners in the same game.