Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models (2021)

Abstract: Story generation is an open-ended and subjective task, which poses a challenge for evaluating story generation models. We present Choose Your Own Adventure, a collaborative writing setup for pairwise model evaluation. Two models generate suggestions to people as they write a short story; we ask writers to choose one of the two suggestions, and we observe which model’s suggestions they prefer. The setup also allows further analysis based on the revisions people make to the suggestions. We show that these measures, combined with automatic metrics, provide an informative picture of the models’ performance, both in cases where the differences in generation methods are small (nucleus vs. top-k sampling) and large (GPT2 vs. Fusion models).

## introduction systems that automatically generate text suggestions to human authors have emerged as a new application of natural language generation models. evaluating such models, however, is challenging. typically, writers rate a single system's quality after some period of use, for example while authoring an entire story or poem (e.g., @xcite . a model's quality is measured using likert scale scores, sometimes combined with additional analysis, like the type or quantity of writer edits (e.g., @xcite . in contrast, a pairwise system evaluationwhere evaluators are given two suggestions at the same time and asked to choose between themwould allow researchers to compare generation models directly. comparative evaluations have been shown to produce more reliable and consistent results than likert-scale ratings @xcite , and they have been used to evaluate natural language generation systems for translation and dialogue @xcite . figure 1 : cyoa has a writer write a line of the story alone, and then two models generate suggestions for the next line. the writer chooses one (in this case, model1), edits it, and then adds it to the story. they repeat this process 5 times. cyoa collects writers' preferences between the two models, along with the humanauthored, machine-generated, and human-edited text, to evaluate the models. we propose choose your own adventure (cyoa), a protocol for pairwise evaluations of collaborative writing models, focusing on story generation. instead of scoring a single model, we compare two models. at fixed points during the writing process, each generates a suggestion, and writers choose one to continue their story (see fig. 1 ). the result is utterance-level feedback on which model's generated text writers prefer at that point in the story. along with the writer's revisions to the generated suggestions and comparisons between the generated and human-authored portions of the story, this evidence can help a researcher answer the following questions about their model: ## choose your own adventure cyoa evaluates a pair of story generation models by having people select and interact with text generated by each of the models as they write a story. both models generate suggestions for the writer at the same point in the story, and the writer must choose between the two suggestions, forcing a pairwise comparison of the two models. by having multiple people write stories with the two models, we can aggregate their preferences and interactions with the suggestions and analyze them to provide feedback on the two models. ## related work collaborative writing systems have been developed in domains like poetry @xcite , slogans @xcite , and stories @xcite @xcite . like storium @xcite , we focus on the potential to use these systems as evaluation platforms. however, we suggest using paired suggestions in collaborative writing systems to directly compare generation models. chateval @xcite collects human evaluations for paired chatbot utterances and @xcite for paired translations, but the generated text is static. by having writers interact with dynamically generated suggestions, collaborative writing systems reward helpful and robust generation models, underemphasized attributes in current evaluations @xcite . ## conclusion cyoa allows researchers to collect human and automatic evaluations for story generation models in a single collaborative writing task. the paired suggestions allow direct comparisons between two models, and automatic-metric comparisons among generated text, its revisions, and the human-authored portions provide additional insight. we expect cyoa evaluations to accelerate progress on applications for collaborative writing between humans and machines.