FIREBALL : A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information (2023)

Abstract: Dungeons & Dragons (D&D) is a tabletop roleplaying game with complex natural language interactions between players and hidden state information. Recent work has shown that large language models (LLMs) that have access to state information can generate higher quality game turns than LLMs that use dialog history alone. However, previous work used game state information that was heuristically created and was not a true gold standard game state. We present FIREBALL, a large dataset containing nearly 25,000 unique sessions from real D&D gameplay on Discord with true game state info. We recorded game play sessions of players who used the Avrae bot, which was developed to aid people in playing D&D online, capturing language, game commands and underlying game state information. We demonstrate that FIREBALL can improve natural language generation (NLG) by using Avrae state information, improving both automated metrics and human judgments of quality. Additionally, we show that LLMs can generate executable Avrae commands, particularly after finetuning.

## introduction dungeons & dragons (d&d) @xcite ) is a tabletop roleplaying game in which players assume the roles of characters in a fantasy adventure. play is conducted primarily through natural language, with players roleplaying as their characters and describing their actions. meanwhile another player, the dungeon master (dm), controls the fictional story world: setting obstacles, goals, and adventures, controlling monsters, and interpreting players' actions in the context of the rules of the game. although the dm makes a lot of the final decisions, the game is ultimately a collaborative storytelling experience. due to its use of natural language as actions, each individual player must maintain a personal ## related work previous papers have outlined the challenges of dungeons & dragons as an ai problem and examined various aspects of the game @xcite . subsequently, a number of datasets in the d&d space have been created @xcite @xcite @xcite , but these datasets either do not include game state information or include only an inexact game state, which lacks grounded, verified attributes, such as those included in our dataset. others have looked at using ai for subsets of d&d gameplay, such as generating spell descriptions @xcite or simulating combat @xcite . in addition to the gold-labelled game state, we include all of the attributes from the following papers within our fireball dataset. @xcite present a dataset pairing independent roleplay utterances with narrative descriptions of the 1 our fireball dataset is available here: https:// github.com/zhudotexe/fireball +2 15 +0 10 +3 17 +0 10 +2 14 +0 10 • +5 +0 • +6 +0 +2 +0 resistances -poison, fire advantage against poison +0 dex p +5 wis +0 int +2 str +0 cha p +3 int +2 wis +0 cha +0 int +2 wis +0 int p +5 wis +0 cha +0 cha +0 +0 +0 p +5 15 12 10 darkvision 60 ft. more recently, callison-burch et al. ( 2022 ) frame d&d as a dialogue challenge, approaching the challenge of generating cohesive and interesting gameplay utterances given the conversational history and a predicted game state. @xcite present a pilot study on understanding player intent and translating it to a high level game action, represented as an avrae command. their workshop paper examines similar tasks, but studies only 100 hand-annotated events. outside of the d&d domain, the task of translating natural language into a bash command @xcite or regex @xcite is similar to our utterance to command task, although these languages do not require conditioning on a context. for another natural language-centric game, diplomacy, @xcite demonstrate the importance of providing game state information to language models. ## playing d&d using avrae in dungeons & dragons, a group of players each create and play as a character. characters have classes (such as wizard or barbarian), fantasy races (such as elves, gnomes, and dragonborn), hit points (denoting their health), statistics that govern how well they can do certain actions, and invento-ries of items (armor, weapons, potions, etc). these game state elements are stored on a character sheet (figure 2 ). one player takes on the role of the dungeon master (dm). this special player creates the world in which the story is told, role plays all of the characters the other players interact with, and acts as arbiter of the rules of the game. there are two main modes of gameplay -incombat and out-of-combat-which have different styles of play. in-combat play simulates battles between characters and monsters, and involves turn taking and tracking stats. out-of-combat play is characterized by freeform collaborative storytelling. both elements of the game involves rolling dice to determine the success of players' actions (like attempting to attack a monster). the die's outcome is then modified with a stat that represents the character's skill in performing that particular action (e.g., +3 for acrobatics). if the dice roll plus the modifier was above a threshold that the rules or the dm determines (called a difficulty class, or dc), then the action succeeds. otherwise it fails. the dm then narrates the results. for example, if an attack hits a monster, the dm might narrate the player's blow and how the monster reacts, taking into account information like the attack's damage type and how many hits the monster has taken previously. traditionally, d&d is played in person with characters' stats written out on physical character sheets and monster stats referenced from books containing hundreds of prewritten "stat blocks". to track a stat that changes frequently like hit points, players and dms use paper and pencil or whiteboards, performing math in their head and writing over the previous value when it changes. some players also use maps and miniatures to track where characters and monsters are located relative to one another and to aid immersion in the game world. since the beginning of the pandemic, a large number of groups have moved online using tools like discord (a messaging program), virtual tabletops that simulate maps, and game state trackers like avrae rather than physical mediums. avrae is a discord bot that was designed to help people play d&d online. it allows players to import their character sheets, allows dms to access a database of monsters, and simulates dice rolls. during combat, avrae tracks the game state. this state contains detailed information including the list of participants in the battle, their stat blocks, their current hit points, and their available actions. avrae allows players to execute commands representing their characters' actions. it performs a simulated dice roll, adds the player's modifiers, and determines the success or failure of the roll. avrae then updates the game state, adjusting things like hit points, and turn tracking. a simplified example of interacting with avrae might look like the following example: player: filgo crouches down in the bush, loosing an arrow at the dire wolf charging towards him. ## dataset we worked with the developer of avrae to instrument it to collect game transcripts and log game state information. the data collection was approved by wizards of the coast, the game company that owns d&d and avrae, as well as by our institution's irb and the bot safety team at discord. any players who participated in our study provided their informed consent. ## state to narration task in this task, we want to generate a narrative utterance describing the effects of a player's actions, given all of the state changes since the start of the player's turn in combat. for example, in the scenario illustrated in figure 1 , a party is fighting a sea hag. on the cleric's turn, she attacks the hag with her mace, but misses. after seeing the result of her action, she narrates the miss, referencing the result as the hag dodging the attack. the full prompt for this example is available in appendix e. after our distillation and filter passes (sections 4.2-4.4), our dataset contains 43,000 aligned stateutterance pairs. to examine the importance of the game states provided in our dataset, we compare our results to methods that do not include the game state, such as dialog continuation @xcite and predicting the narration given only the command that was run @xcite . ## conclusions we have demonstrated how the fireball dataset can be used to predict game commands that correctly match a player's intent and generate cohesive and grounded narration from the results of a game action. our utterance to command model is capable of translating roleplay into game-specific actions and can aid novice users, reducing the amount of time players spend looking up documentation and allowing them to play the game more. our state to narration model can help inspire the dungeon master and take some of the cognitive load off of repetitive writing tasks, allowing them to focus on creating an enjoyable experience for the players. fireball opens the door to multiple exciting avenues of research, and we're excited to see how future work utilizes our unique dataset of state-augmented gameplay.