S tory W ars: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generation (2023)

Abstract: Collaborative stories, which are texts created through the collaborative efforts of multiple authors with different writing styles and intentions, pose unique challenges for NLP models. Understanding and generating such stories remains an underexplored area due to the lack of open-domain corpora. To address this, we introduce StoryWars, a new dataset of over 40,000 collaborative stories written by 9,400 different authors from an online platform. We design 12 task types, comprising 7 understanding and 5 generation task types, on {pasted macro ‘STORYWARS’}, deriving 101 diverse story-related tasks in total as a multi-task benchmark covering all fully-supervised, few-shot, and zero-shot scenarios. Furthermore, we present our instruction-tuned model, InstructStory, for the story tasks showing that instruction tuning, in addition to achieving superior results in zero-shot and few-shot scenarios, can also obtain the best performance on the fully-supervised tasks in StoryWars, establishing strong multi-task benchmark performances on StoryWars.

## introduction storytelling is crucial due to its vital role in human experience, history, and culture dating back to the earliest days of humanity. humans possess the unique storytelling ability to structure a sequence of events, whether factual, fictional or a mixture of both, and create a coherent narrative that conveys a big picture while also including intricate details. current story generation systems usually mimic this ability by starting with a plot then crafting the story. this can be done by linearly expanding @xcite @xcite or hierarchically developing @xcite @xcite @xcite the story based on the given plot. collaborative storytelling these are a series of letters found in my basement. not sure if this is a prank or something but here they are, a warning though, it's pretty weird. letter #1 greetings! you might not know me but i know you. i've been watching you. all the decisions you made (even the bad ones) and all the roads you took to get here, but all good things must come to an end. so, i'm here to wake you up! it's kind of the opposite of dying, in fact, you get to live again! only. you won't remember a thing. the friends you made, the people you loved, the treasures you acquired over your lifespan, all will be gone. your life will be sort of the same? i guess? it's basically your life all over again, except for one thing: i will be making your decisions for you this time letter #7 really? well at least this time your premature death was "out of this world". get it? your intellect was a little less than i expected though. honestly, you should learn to keep your helmet on in space. it is impossible to take a selfie, and plus there's no wifi. i don't know what you were thinking. well, i guess you were too busy partying to actually attend the astronaut training sessions. this time, i think i'll try something a little bit different. i don't think i'll even make you human. so get ready to say hello to your new self, the cat. signed, deathly ;) title: diaries of a reaper likes: 505, stars: 319, genres: is distinctly challenging because there is no predetermined plot or story outline of events. instead, collaborative stories are created through the collective efforts of multiple authors. each author contributes a section sequentially, while also attempting to express their own personal intentions within the context of the jointly crafted and jointly owned story. it is a more challenging problem as it requires not only the ability to generate text, but also the capability to understand the previous context and contributions written by other authors. large language models (llms) @xcite @xcite @xcite @xcite ) have demonstrated exceptional performance on various understanding and generation benchmarks, indicating their potential in addressing natural language processing (nlp) challenges related to collaborative storytelling. this prompts an intriguing question within the research community: how could llms synergize both their understanding and generation capabilities via multitask learning to address the challenges of collaborative storytelling? we present storywars, a dataset of over 40,000 stories gathered from an online collaborative storytelling platform foot_1 . figure 1 shows an example story in the storywars dataset. each story contains rich information including its title, genres given by the initial author, chapters written by different authors, and human ratings including stars and likes. each chapter was written by exactly one author and the previous author might leave a collaborative floor @xcite for the next author to continue. therefore, for a model to generate a continuing chapter, it needs to understand the preceding context, including the title, genres, and the writing styles and intentions of previous authors conveyed in the collaborative floor. due to the multitask nature of collaborative storytelling and the rich information of the story-wars, we design 12 task types, including both understanding and generation task types, as a multitask benchmark for an initial probe of collaborative storytelling. we follow the task definition from flan @xcite , where each task type contains multiple tasks. in the end, our benchmark contains 101 tasks in total, split such that it covers all fully-supervised, few-shot, and zeroshot learning application scenarios. it is important to note that prevailing multitask nlp benchmarks are either focusing on understanding (e.g. @xcite or generation (e.g. @xcite @xcite alone, or only a subset of the learning scenarios. to our knowledge, we are the first to propose a story benchmark that contains both understanding and generation in all three scenarios. large language models have been shown to not only be fully-supervised, few-shot, and zero-shot learners but also multitask ones. instruction tuning @xcite @xcite has been the state-of-the-art approach for zero-shot and few-shot scenarios. however, it has not yet been applied in the fully-supervised setting. we evaluated instruction tuning on the benchmark and we found that in addition to achieving state-ofthe-art results in zero-shot and few-shot scenarios, when combined with single-task fine-tuning, instruction tuning can surpass single-task fine-tuning alone, resulting in a consistent performance boost of 1.53 points on average for all tasks. our contributions are as follows: ## ethical considerations in section 3.1, we have discussed our procedures to identify and remove potential harmful content and user privacy information. however, it is important to also consider the broader ethical implications of using ai in collaborative storytelling. these include issues such as ensuring fair and unbiased representation, protecting data privacy, and preventing the use of ai-generated content for harmful purposes. for example, ai-generated stories or characters may perpetuate stereotypes or reinforce societal biases if they are trained on biased data. therefore, it is crucial to consider and address these ethical issues in order to create inclusive and responsible ai-generated stories that do not harm individuals or groups.