Automatic Bug Detection in LLM -Powered Text-Based Games Using LLM s (2024)

Abstract: Advancements in large language models (LLMs) are revolutionizing interactive game design, enabling dynamic plotlines and interactions between players and non-player characters (NPCs). However, LLMs may exhibit flaws such as hallucinations, forgetfulness, or misinterpretations of prompts, causing logical inconsistencies and unexpected deviations from intended designs. Automated techniques for detecting such game bugs are still lacking. To address this, we propose a systematic LLM-based method for automatically identifying such bugs from player game logs, eliminating the need for collecting additional data such as post-play surveys. Applied to a text-based game DejaBoom!, our approach effectively identifies bugs inherent in LLM-powered interactive games, surpassing unstructured LLM-powered bug-catching methods and filling the gap in automated detection of logical and design flaws.

## introduction text-based computer games are complex, interactive simulations where the game state (location, scenes, and dialogue) is described through text, and players take actions and interact with nonplayer characters (npcs) within the game world to achieve an ultimate goal (côté et al., 2018). npcs are traditionally pre-scripted, limiting user inputs to text commands (mehta et al., 2022) and resulting in stiff interactions. more recently, advancements in llms have enabled a paradigm shift in textbased games, allowing for dynamic interactions between players and npcs foot_1 , with major companies foot_2 releasing llm-powered "character engines" foot_3 for adaptive, unscripted npcs with backstories, goals, emotions, and free-form conversations. these capabilities extend beyond npcs to include other game design aspects such as level generation (kumaran et al., 2023; taveekitworachai et al., 2023; tsai et al., 2023). the plotline-flexibility, npc adaptability, and text input freedom of llm-powered games, however, also creates opportunities for logical inconsistencies, hallucinations, and memory-loss mistakes. to ensure the playability of llm-powered games, automatic detection of these bugs is essential. particularly, in our work, we focus on logical and game balance bugs. logical bugs, per zheng et al. (2019), involve flaws in the logic implementation, resulting in unexpected outcomes without interface crashes, while game balance bugs are deviations from the designers' intentions, potentially making games overly challenging or easy (zheng et al., 2019). such bugs are rare in traditional games due to the predefined nature of both npc dialogues and plotlines. hence, most previous work on automated bug detection has focused on graphics bugs, crashes, or freezes (azizi and zaman, 2023; macklon et al., 2023; varvaressos et al., 2017; al-nassar et al., 2023), with little attention to logical and game balance bugs. moreover, traditional detection of these bugs relies mainly on sentiment analysis of player feedback surveys and game designer discretion (zheng et al., 2019). these surveys require extra effort to gather, but offer limited insights for complicated issues in games (su et al., 2020). to address this gap, we develop an llmpowered method to automatically detect player pain points and associated game logic and game balance bugs from game logs. unlike player surveys, which hinge on players' post-game recollection, game logs (example in fig 3a ), are automatically generated during gameplay, offering richer insights by capturing real-time player actions and dialogue within the game's progression. however, identify-ing game logic or balance bugs from logs of textbased games is an intricate task. it requires synthesizing multiple sources of information (game logic/goals and player progression), inferring intention from player action/utterance, and identifying subtle causal relationships between player behavior and game outcome. thus, this task remains challenging, even for human experts. we seek to leverage llm's language reasoning to address this challenge. feeding game logs directly into an llm, however, does not yield meaningful identification of logic and balance bugs due to the complexity of this task. we therefore develop a structured approach that leverages the game designer's intended gameplay to guide an llm in mapping diverse gameplay attempts to a trackable unified framework. this enables the aggregation of gameplay experiences across players for identifying potential game logic and balance bugs. our two main contributions are: ## dejaboom! llm-powered game we test our method on the recently published textbased mystery game "dejaboom!" (peng et al., 2024). this game utilizes a plot-constrained gpt-4 to generate all in-game text, including game state descriptions, outcomes of player actions, and npc dialogue, is llm-generated (côté et al., 2018). the game begins with the player waking up at home in a village, experiencing "deja vu" about an explosion. to solve the game, the player must obtain a bomb disposal kit and locate the bomb before diffusing it. the player must explore the village, interact with npcs and items to understand how to solve the mystery. some npcs and items can be useful for solving the mystery, but others are red herrings. the bomb detonates after a fixed number of gameplay steps (@xmath0, where each step is a single in-game player action/utterance, causing the player to relive the day until diffusing the bomb. they re-tain memory across explosions, but the game world and npcs are reset after each explosion. @xcite peng et al. (2024) released game logs for 28 players, including all player text inputs and game engine outputs during their gaming sessions. these logs include details of player actions and utterances, npc utterances, and current game state information such as location, inventory, and outcomes of player actions. @xcite additionally, they published poststudy survey responses from the players, offering additional insights into their experiences and feedback on the game. our automated bug detection procedure takes players' game logs and the designer's intended game logic as input, identifying bugs using an llm-powered two-stage procedure (fig 1 ). stage 1 uses a structured approach to prompt the llm to map various player attempts recorded in a game log to the designer's intended progression roadmap and extract information on the player's progression and gameplay experience into a standardized summary that is comparable across players and gameplay steps (section 3.1). stage 2 uses an llm to aggregate these summaries across players to quantify the difficulties of each game progression unit, identify common progression bottlenecks, and pinpoint possibly associated logic and game balance bugs (section 3.2). we use gpt-4 (openai, 2023), the current best of breed model, for all our llm needs. ## conclusion our work introduces a novel pipeline for automatically identifying logic and balance bugs behind player pain points from game logs of text-based adventure games. it is the first automated bug detection method for developing high quality, llmpowered games. by effectively extracting insights from rich content of game logs, it provides an objective and quantitative platform for bug identification, enhancing game design efficiency. future directions include testing this work on more complex text-based games and multimodal games. additionally, the output of our framework may also be fed directly into an llm-powered game to explore ways the game can automatically adjust without explicit game designer intervention.