A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020)

Abstract: Story generation, namely, generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we use multi-task learning, which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.

## introduction story generation is a strong indicator of machine understanding of natural language. it is often approached as selecting a sequence of events to form a story with a reasonable logic or plot. although existing generative models @xcite @xcite can generate stories with good local coherence, they are still struggling to plan a coherent plot and maintain a reasonable event sequence throughout the story, or they are often biased towards generating a limited set of stories with generic plots @xcite (e.g., i have a great time), even when using the powerful generative model openai's gpt-2 @xcite , as shown in pretrained gpt-2 has been shown to capture useful semantic and syntactic features @xcite , as demonstrated by state-of-theart performance on some generation tasks such as machine translation and text summarization @xcite . however, compared with such tasks whose source inputs have contained sufficient information to generate desired target texts, story generation is a typical openended generation task, where only very limited information is given in the input. as shown in this paper, we observe some severe issues when applying gpt-2 to generate reasonable stories, particularly commonsense stories from a limited beginning. these issues include repetition, logic conflicts, and lack of long-range coherence @xcite , as exemplified in ## error analysis although the proposed model outperforms the state-of-the-art baselines, it needs to be noted that there are still many unreasonable stories losing to other models in manual evaluation. therefore, we analyzed error types by manually checking all lost stories in pairwise comparisons between our model and two strong baselines including fusion and gpt-2 (fine-tune) to reveal the factors that affect the performance. the numbers of stories which lost to our model in logic are 114/102 of 200/200 in total for fusion/gpt-2 (fine-tune) respectively. and there are 111 stories of 400 generated by our model losing to these two baselines in logic. we manually annotated four types of error from the lost stories: repetition (repeating the same scenes), unrelated entities or events (with some wrong keywords but a reasonable main plot), conflicting logic (wrong causal relation or temporal order), and chaotic scenes (difficult to understand). the distribution of different error types is shown in we also present some typical cases by our model for each error type in ## conclusions and future work we present a knowledge-enhanced pretraining model with multi-task learning for commonsense story generation. the proposed framework leverages the implicit knowledge from deep pretrained language models as well as the explicit knowledge by post-training on external commonsense knowledge bases, which leads to better performance for commonsense story generation. besides, in order to further capture the causal and temporal dependencies between the sentences in a story, we employ an auxiliary classification task to distinguish true and auto-constructed fake stories. extensive experiments show that the proposed method can outperform strong baselines. further analysis demonstrates that the generated stories are more coherent and reasonable thanks to the use of commonsense knowledge and multi-task learning. as future work, it would be very interesting to make generative pretraining models have commonsense knowledge without any finetuning, namely, integrating the knowledge at the pretraining stage.