Storytelling with Dialogue: A Critical Role Dungeons and Dragons Dataset (2020)

Abstract: This paper describes the Critical Role Dungeons and Dragons Dataset (CRD3) and related analyses. Critical Role is an unscripted, live-streamed show where a fixed group of people play Dungeons and Dragons, an open-ended role-playing game. The dataset is collected from 159 Critical Role episodes transcribed to text dialogues, consisting of 398,682 turns. It also includes corresponding abstractive summaries collected from the Fandom wiki. The dataset is linguistically unique in that the narratives are generated entirely through player collaboration and spoken interaction. For each dialogue, there are a large number of turns, multiple abstractive summaries with varying levels of detail, and semantic ties to the previous dialogues. In addition, we provide a data augmentation method that produces 34,243 summary-dialogue chunk pairs to support current neural ML approaches, and we provide an abstractive summarization benchmark and evaluation.

## introduction artificial intelligence applied to human conversation remains an incredibly challenging task in computer science. task-oriented dialogues, which are more narrowly scoped and information dense than conversational dialogue, have been the focus of recent progress in dialogue understanding @xcite . a difficulty for hypothesis testing on non-task oriented dialogues is a lack of large datasets that are fully representative of the spontaneity and noise of real world conversation, especially in the areas of storytelling and narrative beyond long-form text or monologue. many potential dialogue processing tasks involve multi-speaker dialogues where narrative elements are conveyed through interaction between two or more speakers. these narrative elements can include changes in the states of narrative objects, sample dialogue chunk 0 travis: "i felt like i almost died and i had n't taken care of any of the shit that got me here in the first place . i was so worried about trying to learn about these new abilities that -i felt like i got distracted . i have people i want to find and things i want to remedy ." 1 marihsa: "yeah . how did jester do ? no offense , but she seems like she 's a little bit more willfully stronger than you are ." 2 travis: "i mean , fuck , it 's really disturbing . like , she came out of there like a little kettle of popcorn , just no problem . i mean -can i see jester ? is she nearby ?" 3 matt: "jester , are you nearby ?" 4 laura: "i 'm across the bar just fucking dancing alone . figure 1 : a tokenized dialogue chunk and the associated human written summary chunk after the text alignment process. jester, beau, and fjord are the aliases for laura, marisha, and travis respectively. ## related work the critical role dungeons and dragons dataset is a combination of story-telling dialogues structured around the game-play of dungeons and dragons and corresponding abstractive summarizations for each dialogue. as such, it can be compared to existing dialogue datasets and summarization datasets. ## scaling up the dialogue summaries the crd3 dataset can be applied to many tasks, but we find abstractive dialogue summarization the most compelling task to explore in this paper. due to the extensive length of the dialogues and summaries, and the frequent context switching and noise, we are presented with challenges that are poorly addressed by the current modeling and evaluation methods: we do not attempt to propose a solution for either the second or third challenges, as they are beyond the scope of this paper. instead, we address the first challenge by proposing a novel data augmentation method to dramatically scale up the number of available summary-dialogue turn sequence pairs. that outcome enables the community to start modeling and evaluation for the dialogue summarization task and we discuss initial benchmark results over this augmented set in section 5. ## conclusion and future work dialogue understanding and abstractive summarization remain both important and challenging problems for computational linguistics. in this paper, we contribute the critical role dungeons and dragons dataset (crd3), a linguistically rich dataset with dialogue extracted from the unscripted, livestreamed show critical role and long, abstractive summaries extracted from the critical role fandom wiki. we provide a data augmentation method to help the community start modeling and evaluation for the dialogue summarization task and discuss the initial modeling benchmark results. we find current paradigms in summarization modeling to have specific failures in capturing semantics and pragmatics, content selection, rewriting, and evaluation in the domain of long, story-telling dialogue. we hope crd3 offers useful, unique data for the community to further explore dialogue modeling and summarization. we also hope that the dataset can be added to in the future with multi-modal extractions, more granular annotations, and deeper mining of the wiki. recall error dialogue chunk 0 matt: "end of your turn, it's going to use two actions to do a wing attack, beating its wings, hitting every creature within 15 feet. you're out of range, actually, marisha. grog, i need you to make a dexterity saving throw." 1 travis: "i think i have advantage on this because of rage. i do. 21." 2 matt: "21? that unfortunately fails. you take 15 points of bludgeoning damage, and you're knocked prone. also, pike and vax, you both fail a death saving throw from the bludgeoning winds of the ice dragon's wings beating downward." aligned summary chunk 0 "scanlan takes a greater healing potion and moves towards vorugal. he hits him with a fireball." 1 "vorugal uses a wing attack against grog, hitting both vax and pike as well, losing a death save each." figure 7 : a (not tokenized) turn sequence and the associated human written summary chunk after the text alignment process. it is clear from the second sentence of the summary chunk, that the turn aligned turns are a subset of the the true turn sequence the summary chunk is referring to. in order to capture the turns referred to by the first sentence in the summary, we need to include the additional 29 preceding turns in the dialogue (which are treated as 29 false negatives). out of game dialogue chunk 0 orion: "ooh, like thai food." 1 liam: "i like indian." 2 matt: "ooh, indian is good." 3 ashley: "i really noticed-" 4 zac: "let them know not to order food." 5 liam: "don't, that's a terrible idea." 6 orion: "we just had a bunch of chicken." 7 marihsa: "oh you mean like right now? yeah, don't do it right now." 8 zac: "if you tell them what you want, all of a sudden i'll get a call, like, "your food is on the way!"" aligned summary chunk 0 "liam, matt, marisha, and taliesin like indian food." 1 "zac chimes in telling the chat not to order any more food right now." figure 8 : an out-of-game turn sequence and summary chunk. we find a single precision error in this alignment with orion mentioning thai food, which is not in this summary chunk.