A Framework for Exploring Player Perceptions of LLM -Generated Dialogue in Commercial Video Games (2023)

Abstract: The growing capabilities of large language models (LLMs) have inspired recent efforts to integrate LLM-generated dialogue into video games. However, evaluation remains a major challenge: how do we assess the player experience in a commercial game augmented with LLM-generated dialogue? To explore this question, we introduce a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games. We first extract dialogue from the widely-acclaimed role-playing game *Disco Elysium: The Final Cut*, which contains 1.1M words of dialogue spread across a complex graph of utterances where node reachability depends on game state (e.g., whether a certain item is held). Using this dataset, we have GPT-4 perform *dialogue infilling* to generate grounded utterances based on game state represented via code. In a statistically robust study of 28 players recruited from the r/DiscoyElysium subreddit, the LLM outputs are evaluated against the game designers’ writing via both preference judgments and free-form feedback using a web interface that recreates the game’s core conversation functionality. Overall, the game designers’ prose is significantly preferred to GPT-4 generations, with participants citing reasons such as improved logical flow and grounding with the game state. To spur more principled future research in this area, we release our web interface and tools to enable researchers to build upon our work. https://pl.aiwright.dev

## introduction dialogue in most narrative-driven video games has historically been static: players may choose from a small number of pre-written dialogue options that depend on the game's state (e.g., items held or goals achieved). the advent of large language models (llms) has inspired efforts to dynamically generate dialogue in video game environments, such as those by ai dungeon @xcite , inworld ai @xcite , and convai @xcite , which can potentially imbue games with endless variety. 2 however, evaluating the impacts of llm-generated dialogue on the player experience has yet to be tackled in a principled manner. in this paper, we directly evaluate the player experience by asking video gamers to interact with llm-generated dialogue injected into disco elysium: the final cut, a highly-acclaimed dialoguecentered video game @xcite . 3 to mitigate difficulties with evaluating open-ended text generation @xcite , we specifically examine a constrained dialogue generation task in which an llm must decide how to update a dialogue to match the corresponding game state. take for example a scene in which the game's protagonist (an amnesiac detective) is interrogating an uncooperative suspect. if the player decides to act like the suspect's friend to obtain more information, the variable seafort.deserter_sugg_you_are_buddies is set to true, and the corresponding line of dialogue is: you can tell me, here. it won't be *that* usable. continuing the example, we then prompt an llm to appropriately modify the dialogue when the variable seafort.deserter_i_am_also_communist is set to true. we evaluate the generated dialogue against the game writers' original line: you can tell a comrade. it won't be *that* usable. while the semantics of the utterance remains mostly unchanged, the player's assumed communist persona is reflected in the second dialogue. these dialogue options, and any associated nonplayer character (npc) responses, are defined us-ing a graph structure commonly referred to as a "dialogue tree" in the video game industry. crucially, our task setup does not expect an llm to generate all of the game's dialogue (as in e.g., ai dungeon), but rather provides the llm with human-written dialogue as input and tweaks it to fit various dynamic aspects of the game state. to condition on the game state, we devise a clever approach of encoding the graph structure as a mix of code and natural language, which also opens the possibility in future work of modifying the game state in response to a generated utterance. this constrained setup makes the task tractable to evaluate and also practically relevant to future llm-human collaborative game writing applications. can an llm understand enough about the game state to appropriately modify dialogue in a way that is logically and tonally consistent with the game state? more generally, how does llm-generated dialogue stack up with the award-winning dialogue written by disco elysium's designers, and what do video gamers think are the biggest issues with it? by choosing a popular roleplaying game with a dedicated following, we are more easily able to find participants familiar with the expected tone and lore needed to effectively assess our generated dialogue. we evaluate openai's state-of-the-art gpt-4 llm (openai, 2023) via a statistically robust user study, asking disco elysium fans to provide preference judgments and free-form feedback within an interface designed to mimic the game's dialogue engine. perhaps unsurprisingly, players strongly prefer the original dialogue (h) compared to llmgenerated dialogue (g), with participants citing reasons such as better logical consistency (h: 61% vs g: 28%) and flow (h: 67% vs g: 21%). however, participants note that gpt-4 begins to close the gap at providing interesting dialogue options (h: 47% vs g: 36%) that advance their goals (h: 57% vs g: 33%), though further work is required to ground the dialogue to the game state as 32% of generations rated by at least one player are deemed illogical upon reading the next utterance. to facilitate future research in player-centered video game dialogue generation, we release our annotation interface and tools to reproduce our dataset. ## related work commercial video games have become increasingly popular testbeds for neural approaches to grounded language @xcite and reinforcement learning @xcite . to the best of our knowledge, only the sandbox game minecraft has been explored as a testbed for interactive dialogue research @xcite , despite some commercial video game dialogue being explored in noninteractive settings @xcite . dialogue systems in roleplaying video games are task-oriented @xcite , where quests in the game act as tasks that the player must complete within the constraints of the game world. the static pre-written dialogue graphs are a form of finitestate dialogue management @xcite . using such a rigidly constrained dialogue management approach ensures the authorial intent of the game writers at the expense of more natural conversation flows. rather than upend these familiar techniques for more flexible approaches which have yet to gain traction @xcite , likely due to their complexity, we opt to augment the existing approaches actively in use in commericial video games. our work bridges the tightly scripted scenarios common to video games, with more natural speech that offers humans more agency over their interaction with virtual agents, leading to easier to design agent interactions useful for training simulations @xcite and tutoring @xcite . narrative-driven video games fall under the umbrella of interactive storytelling, which can take on many forms including tabletop roleplaying games like dungeons and dragons @xcite , choose your own adventure books @xcite , and interactive fiction @xcite . more broadly, graph representations have been used for finite-state dialogue management @xcite , and as knowledge bases @xcite for slot-filling approaches @xcite to task-oriented dialogue. ## the disco elysium dataset in disco elysium: the final cut, the player takes on the role of a down-on-his-luck detective in order to solve a murder mystery in a dystopian city. the majority of in-game interactions are in the form of dialogue, including interactions with not only other characters but also inanimate objects (e.g., a ceiling fan and a bathroom mirror from the first scene of the game), which makes the game well-suited for our experiments. additionally, the game's state is encoded in lua foot_3 via descriptively-named boolean variables and getter/setter functions that trigger on certain nodes of the dialogue graph. in this section, we describe how we extract and process the rich dialogue graph (figure 1 ) and game state from disco elysium for our constrained dialogue generation task. ## grounded dialogue infilling as prior text generation and dialogue research has clearly demonstrated @xcite , evaluating llm-generated dialogue is a daunting challenge. we examine a more constrained subtask in which an llm is given multiple lexically-similar human-written responses to a given utterance, each of which is slightly different from each other based on the game state. one of these responses is masked out, and an llm is asked to generate it based on cues from the game state (e.g., that communists have historically referred to one another as comrade). while this task is strictly easier than open-ended dialogue generation, we find that state-of-the-art llms still struggle to solve it. this section details the data filtering and preprocessing steps we performed, as well as the few-shot prompting strategy we use with gpt-4. ## results & analysis overall, our study reveals a strong preference for human-written dialogue over llm-generated dialogue. participants most commonly cite reasons for their preferences such as increased appropriateness, better match with their gameplay goals, and stylistic properties (figure 4 ). because assessing the generative capabilities of llms is confounded by the subjective nature of rating narrative quality @xcite , we also conduct a fine-grained analysis of free-form player justifications to uncover where gpt-4 succeeds and where it needs improvement. ## conclusion in this paper, we perform a user study of llmgenerated dialogue integrated into video games, hiring fans of disco elysium: the final cut to provide fine-grained insights about issues in this domain. we examine a constrained dialogue generation task, in which the game state is integrated into the llm prompt via lua scripts that encode game variables, functions, and dialogue. we develop a web interface that reproduces disco elysium's dialogue engine to conduct the evaluation. humanwritten dialogue is strongly preferred over gpt-4 generations for reasons such as improved logical flow, appropriateness, and tonal consistency. future work can build on our framework to consider user play style, faithfulness to the game state, and dynamically updating game state as important components of the modeling and evaluation process.