Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing (2023)

Abstract: We explore the idea of incorporating concepts from writing skills curricula into human-machine collaborative writing scenarios, focusing on adding writing modes as a control for text generation models. Using crowd-sourced workers, we annotate a corpus of narrative text paragraphs with writing mode labels. Classifiers trained on this data achieve an average accuracy of ~87% on held-out data. We fine-tune a set of large language models to condition on writing mode labels, and show that the generated text is recognized as belonging to the specified mode with high accuracy. To study the ability of writing modes to provide fine-grained control over generated text, we devise a novel turn-based text reconstruction game to evaluate the difference between the generated text and the author’s intention. We show that authors prefer text suggestions made by writing mode-controlled models on average 61.1% of the time, with satisfaction scores 0.5 higher on a 5-point ordinal scale. When evaluated by humans, stories generated via collaboration with writing mode-controlled models achieve high similarity with the professionally written target story. We conclude by identifying the most common mistakes found in the generated stories.

## introduction large-scale pre-trained language models (plms) have demonstrated a remarkable aptitude for generating text with an exceptional degree of fluency and structure @xcite , sparking renewed efforts to utilize them for the purpose of generating narrative fiction. recent work has explored various ways of controlling plms, using sentiment @xcite , style @xcite , and even character information @xcite , in an attempt to cater the generated text to an author's intentions. summary: a boy was walking along a path in the forest, when he came across a heap of leaves. ## fiction-writing mode dataset fiction-writing modes have long been proposed as a useful abstraction in the study of literature and creative writing @xcite , dating as far back as aristotle @xcite . while there is no consensus on the categorization of writing modes, most sources prefer to introduce at least three modes: (1) dialogue, direct quotation of characters speaking, (2) action, an account of a series of events, one after another, chronologically, and (3) description, a more detailed inspection of people, places, or things and their properties. these are the three major writing modes which are the focus of study in this paper. just as there is no agreement on how best to categorize writing modes, there is also no consensus on what text exhibits a particular mode. even a single sentence can exhibit multiple writing modes, in varying degrees. however, for the purpose of this work, we assume that each paragraph can be categorized as exhibiting a single writing mode. ## models we evaluate writing mode as a control factor on three different plm architectures: bart, @xcite , t5 @xcite , and gpt2 @xcite . all models have been used previously for text generation but differ in ways that may impact their ability to adhere to the conditioning information and the quality of the generated text. for instance, the comparatively larger size and contextual window of gpt2 has made it a common choice for story generation with long text @xcite @xcite , but smaller models like t5 show great controllability @xcite . we assess each of these three models, fine-tuning them to reconstruct paragraphs from the fride-xl dataset. for training conditional text generation models, we follow an established paradigm of summarization, conditioning, and reconstruction, as used by @xcite . first, each paragraph is summarized using an existing summarization model. here we use the narrative text summarization proposed in @xcite , and decode using beam search with a beam size of 5 as in that work. we then fine-tune a plm to reconstruct the original paragraph, conditioning on the summary. in this way, the summary acts as a semantic control: the trained model accepts user summaries and attempts to expand upon them to generate a longer paragraph, embellishing missing and less important details in a reasonable way. other forms of information can also be added to the summaries to function as additional controls. the conditioning factors provided to models are: for t5 and bart, the training methodology is straightforward: we concatenate the controlling information and use it as input to the encoder, training the decoder to generate the original paragraph. for gpt2, which has only a decoder, we concatenate the conditioning information as prompts. ## automatic evaluation in this section, we study the influence of model inputs (e.g., summaries, length, and writing modes) on the quality of text, and assess to what extent the writing modes of text can be controlled, as measured by automatic evaluation metrics. ## human evaluation in the preceding section, we used automatic measures to show consistent improvements in similarity and controllability when conditioning text generation models on writing modes. in this section, we assess whether these improvements translate into higher author satisfaction and story quality as judged by human participants. text reconstruction game to evaluate writing mode-controlled models within human-machine collaborative writing, we devise a story generation game (figure 2 ). using a web interface, participants are presented with a target story, drawn from fride-xl, and asked to reconstruct it through interaction with the trained models. each session utilizes two models, each using the same plm, one trained to condition on writing modes, and one that does not. at each round, (1) the author highlights a selection of text from the target story, and provides a text summary which serves as a rough sketch of the story described therein. constraints on the summary length and a time limit on each session force the author to extract essential elements of the text and prevent them from simply specifying the entire text as the summary. (2) the models then each return a sample of generated text, and (3) the author is asked to score each in terms of overall satisfaction (ordinal, 5-point scale), and choose one to continue the story. the source of the suggestions (model name) is not given to the participant, and they are presented in random order. the above process repeats until the author has attempted to recreate the entire target text. the author is then asked to provide feedback on overall satisfaction with the story and interface, the efficiency of using the interface over generating the text from scratch, and what errors were present in the generated text. our approach takes inspiration from previous human-in-the-loop evaluations @xcite where authors are asked to construct stories by collaborating with story generation models, but do so freely, without being provided any specific direction for how the story should unfold. therefore we cannot observe the author's intention, and it is difficult to ascertain how closely the generated stories match the author's original intention, or the extent to which bias may be affecting the model scores when satisfaction with the models is self-reported. our solution to these issues is to present the author with a target text they must attempt to reconstruct. the target text serves as a substitute for the author's intention, allowing us to more objectively measure the differences between the generated story and the target. while this design shifts the nature of interaction away from a more creative use case, we argue that increased awareness of the models' limitations provides a worthwhile trade-off when used strictly as a means of evaluation. using the unique properties of our proposed evaluation design, we evaluate: (1) the effectiveness of writing mode control when generating text as an author, (2) their effectiveness from the perspective of a reader, comparing generated stories with target stories in a blind study, and (3) the types of errors made where authors were unable to exert desired ## related work our work is based on prior research in computational modeling for story generation. early approaches to automatic story generation relied on graph-based planning and hand-crafted rules to structure narratives @xcite @xcite . more recent works generate stories by finetuning on large-scale plms @xcite to improve its fluency and incorporating structured knowledge such as planned events @xcite @xcite , summaries @xcite @xcite , and external knowledge @xcite @xcite to enhance its coherence and consistency. our story generation models are also finetuned on the large-scale plms to generate text following the given summaries. our work bears similarity to work on controllable story generation, which aims to control different attributes of stories such as the sentiments @xcite , genres @xcite , intention @xcite , and characters @xcite @xcite of stories. however, these attributes are largely unchanging throughout the story, while we focus on writing mode, a more dynamic attribute of text. thus our work is also more inherently related to interactive story generation where the author works closely with the model to craft text on a comparatively finer level of granularity (sentences or paragraphs). finally, our evaluation method is inspired by work on human-in-the-loop storytelling @xcite @xcite @xcite , where the authors are asked to work in concert with story generation models, curating their suggestions to craft the final story. human-in-theloop evaluations overcome many of the shortcomings of automatic evaluations, which capture rough statistics, but may be unaware of important errors in plot development and story continuity @xcite . by asking users to select between models' suggestions, we can instead gain a more accurate picture of which system improvements yield real benefits to a potential human-machine writing collaboration @xcite @xcite . our approach is similar to this, but the addition of a target text allows us to examine the difference between the generated and intended text, which we argue is a more important comparison when dealing specifically with understanding control. scenarios and showed that training models to condition on writing modes resulted in stories that were closer to targets. both the automatic and human evaluation shows that the writing modes of text are effectively controlled, authors prefer text suggestions made by writing mode-controlled models, and readers score stories to be more similar to targets in terms of both plot and style. to verify the hypothesis, we collected fride and fride-xl, datasets of narrative text annotated with writing modes, which we released to help facilitate further research in writing modes and fine-grained control for storytelling. in future work, we wish to apply reconstruction-based evaluation for other factors of human-machine storywriting, and incorporate a dynamic use of writing modes into fully automatic hierarchical story generation models.