Decoding Methods for Neural Narrative Generation (2021)

Abstract: Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. The task is similar to neural response generation for chatbots; however, innovations in response generation are often not applied to narrative generation, despite the similarity between these tasks. We aim to bridge this gap by applying and evaluating advances in decoding methods for neural response generation to neural narrative generation. In particular, we employ GPT-2 and perform ablations across nucleus sampling thresholds and diverse decoding hyperparameters—specifically, maximum mutual information—analyzing results over multiple criteria with automatic and human evaluation. We find that (1) nucleus sampling is generally best with thresholds between 0.7 and 0.9; (2) a maximum mutual information objective can improve the quality of generated stories; and (3) established automatic metrics do not correlate well with human judgments of narrative quality on any qualitative metric.

## introduction narrative generation (or story generation) is the task of generating a creative response given an input prompt. this output can be a story closure, a paragraph, or a structured story with multiple paragraphs. this input and output setup is similar to the response generation task of chatbots, as both tasks convert some variable-length sequential input from a user to an automatically generated variablelength sequential output. thus, the neural models and methods proposed to date for story generation and dialogue generation have been similar. however, as narrative generation is largely focused on coherence across long outputs, the strategies used in this subfield have evolved separately from those in chatbot response generation; the latter has been more concerned with generating interesting and diverse-and typically short-outputs. thus, while many beneficial techniques may have arisen from one domain, they are not often employed in the other. one decoding method, nucleus sampling @xcite , has recently been applied to narrative generation @xcite , but a thorough evaluation of its various p thresholds has not been performed with human judgments using narrative-specific criteria, as this can be time-and labor-intensive. also, recent advances in decoding methods for response generation-notably, the application of the maximum mutual information (mmi) objective @xcite -have resulted in more interesting dialog according to human evaluators @xcite ; nonetheless, this also has not been applied to narrative generation. indeed, the mmi objective has been confined to short-form and less openended generation tasks thus far. thus, we apply techniques from neural response generation to neural narrative generation in order to investigate the potential benefits-and pitfalls-of applying these methods in this underexplored domain. this study aims to connect research developments across tasks by sweeping various thresholds of nucleus sampling and the application of diverse decoding to generate more long-form creative outputs. we perform human and automatic evaluations of automatically generated stories in these settings in order to investigate the following phenomena: our preprocessing, training, generation, and analysis scripts are available publicly. 1 2 related work narrative generation tasks work on narrative generation is split between cloze tasks, open-ended generation, and guided generation. in a cloze task, a full story except for a final word, phrase, or sentence is given, and a model generates a completion. this could be cast as a short generation problemor, more commonly in this domain, a multiplechoice problem @xcite @xcite . open-ended generation is the task of generating long-form output conditioned on a prompt (figure 1 ). @xcite create a paired prompt and response dataset from the subreddit r/writingprompts foot_1 to train a sequence-tosequence "fusion model." @xcite extend @xcite , but use gpt-2 small and perform a top-k decoding parameter sweep. we focus on this open-ended narrative generation task in our investigation, but primarily focus on gpt-2 medium and on the effect of nucleus sampling thresholds and diverse decoding strengths on narrative quality. while @xcite similarly perform a hyperparameter search over sampling algorithms in a language generation setting, they perform human evaluations using a convincingness metric on a short-form news generation task; long-form narrative generation is not bound by realism (and may actually benefit from less realistic output), and thus requires different metrics and evaluation setups. guided generation is the middle ground of cloze and open-ended generation. the model is provided more context, such as characters, plot information, and potentially other information, and then generates a story based on all of the provided structural and semantic information @xcite . ## conclusions our results suggest that p values lower than those suggested for other tasks @xcite are ideal in narrative generation, and that small magnitudes of diverse decoding may produce better and more vivid stories. we also find that distinct-n and sentence-bert do not correlate well with any of our human perceptions of narrative quality, and that the quality-diversity trade-off is less strong in narrative generation than in other generation tasks. the latter finding is preliminary, though supported by @xcite , who find increases in both diversity and human scores with their proposed method. our findings aim to inform future efforts in the narrative generation domain by establishing future baselines given our recommended hyperparameters, and by facilitating further investigation of decoding objectives for better narrative generation. once @xcite is released for public use, it is very likely that this model will outperform gpt-2; thus, we encourage future work to investigate similar hyperparameters and sampling methods to see whether these trends are stable across model sizes.