STORIUM : A D ataset and E valuation P latform for M achine-in-the- L oop S tory G eneration (2020)

Abstract: Systems for story generation are asked to produce plausible and enjoyable stories given an input context. This task is underspecified, as a vast number of diverse stories can originate from a single input. The large output space makes it difficult to build and evaluate story generation models, as (1) existing datasets lack rich enough contexts to meaningfully guide models, and (2) existing evaluations (both crowdsourced and automatic) are unreliable for assessing long-form creative text. To address these issues, we introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our author-generated dataset contains 6K lengthy stories (125M tokens) with fine-grained natural language annotations (e.g., character goals and attributes) interspersed throughout each narrative, forming a robust source for guiding models. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them. Automatic metrics computed over these edits correlate well with both user ratings of generated stories and qualitative feedback from semi-structured user interviews. We release both the STORIUM dataset and evaluation platform to spur more principled research into story generation.

## introduction fiction writers express their creativity through both low-level linguistic choices and discourse-level sequencing of narrative elements (e.g., plot events and character development). unlike more constrained text generation tasks, such as translation or summarization, fiction writing allows for almost infinite creative freedom, which budding authors often find cognitively overwhelming @xcite . machine-in-the-loop storytelling @xcite , in which an author obtains automatically generated sentences or paragraphs when stuck with writer's block, lowers the barrier to entry for creative writing @xcite . to spur research in this area, we partner with sto-rium, 1 an online collaborative storytelling platform, to introduce a new dataset and evaluation methodology for story generation. the open-endedness of story writing does not just pose a barrier to humans-it also presents a challenge for building and evaluating computational models. prior work relies on datasets that are either too artificial to generalize to longform stories, such as the crowdsourced rocstories @xcite corpus, or too unconstrained, as in the r/writingprompts dataset @xcite , which pairs mediumlength stories with short prompts. furthermore, lack of standardized evaluation makes measuring progress difficult: most prior work evaluates outputs using a combination of simple automatic metrics not designed for long-form creative text generation (e.g., bleu and rouge against a single reference) and crowdsourced ratings @xcite @xcite that preclude evaluating long-form narratives. we address these limitations by (1) collecting a dataset of stories (section 2) containing finegrained structural annotations written in natural language, and (2) providing a platform for evaluating models in a machine-in-the-loop setting by allowing real storium authors to interact with the generated stories (section 4). our dataset contains nearly 6k longform stories (125m tokens) written by storium authors, each of which is broken into discourse-level scene entries annotated with narrative elements, such as character goals or abilities. ## storium dataset our storium dataset derives from an online collaborative storytelling community that provides rich metadata useful for guiding computational sto-2 https://storium.cs.umass.edu rytelling systems. in this section, we describe how the structural elements of storium stories fit together, and verify via an annotation task that this metadata indeed influences the text of the stories. finally, we use neural topic models to highlight the thematic content and narrative sequencing of storium. ## generating scene entries we focus our modeling efforts on generating scene entries, which are the smallest units of each story, because we want to evaluate the generated text on constraint segment types: intro character challenge card strength card prev entry entry title description figure 3 : an illustration of our segment embeddings and packing strategy. in addition to token and position embeddings, common to all transformer models, we employ compositional segment embeddings for conditioning on story metadata (e.g., deadly aim is the title of a strength card ). each metadata segment has linear constraints with associated priorities (e.g., @xmath0. the storium platform within a machine-in-theloop framework. 8 our method relies on fine-tuning a pretrained language model (gpt-2) on the sto-rium dataset using segment embeddings to differentiate each type of context. while gpt-2 has successfully been used as a state-of-the-art model for story generation @xcite , one crucial challenge is the length of the contexts: each entry in a story can condition on any narrative element that comes before it (e.g., previous entries, scenes, challenges). thus, the number of context tokens quickly grows larger than what is feasible to fit in gpu memory. another challenge lies in how to properly tune hyperparameters in a machine-in-the-loop setting, as it is infeasible to obtain human judgments for a huge number of configurations. the rest of this section fully specifies our model, a token-packing strategy to optimize use of the input context, and preliminary user-facing experiments that helped us decide on our final model hyperparameters. ## a machine-in-the-loop evaluation platform the inadequacies of existing human and automatic evaluation methods are a major roadblock for story generation research. automatic evaluations correlate weakly with human judgments @xcite , and these judgments are obtained from crowd workers who are not invested in the narratives they are assessing. these concerns are magnified with storium, as the story contexts are far too long for crowd workers to reliably evaluate (section 5). in this section, we propose an improved evaluation methodology by directly integrating our models onto the storium platform. this allows story authors to query a machine @xcite for suggestions during the process of writing their own stories. we develop a new evaluation metric, user story edit ratings (user), computed on top of the edits that storium users make to generated entries. finally, we provide experimental results that compare two configurations of our best model from section 3.2. ## analysis compared to existing work on story generation, the main novelty of our storium evaluation platform is that it enables authors to interact directly with model-generated text through their edits. in this section, we conduct experiments on our platform and analyze the edits by examining the correlation of user to likert scores. we explore linguistic properties of text that users preserve and also conduct a crowdsourced evaluation on amazon mechanical turk that demonstrates its unsuitability for this task. finally, we qualitatively describe feedback obtained from interviews with ten storium users who engaged with our models, which provides a roadmap for future work. top-k vs. nucleus sampling: using our platform (section 4), we evaluate our best model ( crowdsourced evaluation is unreliable: thus far, we have argued for our evaluation platform by claiming that crowdsourced methods are unsuitable for evaluating stories with complex and lengthy contexts. here, we measure fluency, coherence, relevance, and likability of our generated entries with a crowdsourced amazon mechanical turk task, to see if the results correspond to storium user ratings. designing this crowdsourced task is difficult, as we cannot show crowd workers the entire story context due to its length; we thus decide to show the same inputs that the model receives (section 3). we collect ratings of 100 examples per model, with three judgments per example. 14 feedback from user interviews: to better understand the strengths and weaknesses of our current model, we conduct semi-structured interviews with ten storium users. most were surprised with the overall fluency of our models. this partly explains the low correlation of fluency with user. relevance was mentioned by 9 out of 10 users as the number one area of improvement for our model, confirming our experimental results ( our work connects more closely to existing machine-in-the-loop storytelling work @xcite @xcite , in which systems work in concert with users to collaboratively author a narrative. much like the creative help platform of @xcite , we provide writing assistance by interactively generating continuations of storium stories. we improve over @xcite by evaluating a trained model (instead of a retrievalbased approach) with a large user population. finally, our storium evaluation takes a different approach to prior research that measures the quality of generated stories. @xcite train an automatic scorer on human annotations of overall story quality, relevance, and interestingness based on evaluation criteria from @xcite . @xcite consider a number of diversity related measures for automated evaluation of story generation systems by focusing on the gpt-2 small model, noting that quality assessments are still best measured through human evaluation. ## conclusion we introduce the storium dataset and evaluation platform for machine-in-the-loop story generation, built from an online collaborative storytelling community. storium contains 6k long stories annotated with structural metadata useful for conditioning language models. importantly, real storium authors evaluate model outputs by adding and removing text to create their own stories. we devise a metric on top of their edits that correlates strongly with judgments of the relevance of the generated text, which user interviews suggest is the most important area for improvement moving forward. our dataset and evaluation platform will be made publicly available to spur progress into story generation. when evaluating story continuations, we cannot compare against an a priori gold standard. rather, we consider the final published story a user generates to be the gold standard, and thus evaluate models by how much text the user retains. using rouge-l precision, which simply computes the ratio of the longest common subsequence (lcs) with the number of tokens in the generated text, we can measure this quantity. @xcite , rouge-l contains a subtle mismatch with expectations, as the lcs does not consider locality of matches -assigning equal weight to subsequences of the same length even when the distance between matched words differs. given a reference sequence x, the following two candidate sequences y 1 and y 2 produce the same rouge-l score (an underscore indicates a subsequence match):