Plan- CVAE : A Planning-based Conditional Variational Autoencoder for Story Generation (2020)

Abstract: Story generation is a challenging task of automatically creating natural languages to describe a sequence of events, which requires outputting text with not only a consistent topic but also novel wordings. Although many approaches have been proposed and obvious progress has been made on this task, there is still a large room for improvement, especially for improving thematic consistency and wording diversity. To mitigate the gap between generated stories and those written by human writers, in this paper, we propose a planning-based conditional variational autoencoder, namely Plan-CVAE, which first plans a keyword sequence and then generates a story based on the keyword sequence. In our method, the keywords planning strategy is used to improve thematic consistency while the CVAE module allows enhancing wording diversity. Experimental results on a benchmark dataset confirm that our proposed method can generate stories with both thematic consistency and wording novelty, and outperforms state-of-the-art methods on both automatic metrics and human evaluations.

## introduction a narrative story is a sequence of sentences or words which describe a logically linked set of events @xcite . automatic story generation is a challenging task since it requires generating texts which satisfy not only thematic consistency but also wording diversity. despite that considerable efforts have been made in the past decades, the requirement of thematic consistency and wording diversity is still one of the main problems in the task of story generation. on the one hand, a well-composed story is supposed to contain sentences that are tightly connected with a given theme. to address this problem, most previous methods attempt to learn mid-level representations, such as events @xcite , prompts @xcite , keywords @xcite or actions @xcite , to guide the sentences generation. although these approaches have shown their encouraging effectiveness in improving the thematic consistency, most of them have no guarantee for the wording diversity. the main reason is that most of these methods are based on recurrent neural networks (rnns), which tend to be entrapped within local word co-occurrences and cannot explicitly model holistic properties of sentences such as topic @xcite @xcite . as a result, rnn tends to generate common words that appear frequently @xcite and this will lead to both high inter-and intra-story content repetition rates. on the other hand, a well-composed story also needs to contain vivid and diversified words. to address the issue of wording diversity, some studies have employed models based on variational autoencoder (vae) @xcite or conditional variational autoencoder (cvae) as a possible solution. it has been proved that, through learning distributed latent representation of the entire sentences, vae can capture global features such as topics and high-level syntactic properties, and thus can generate novel word sequences by preventing entrapping into local word co-occurrences @xcite . as a modification of vae, cvae introduces an extra condition to supervise the generating process and has been used in multiple text generation tasks, e.g., dialogue response generation @xcite , chinese poetry generation @xcite . recent researches @xcite in story generation task have confirmed that cvae can generate stories with novel wordings. despite the promising progress, how to keep thematic consistency while improving wording diversity is still a challenging problem, since these two requirements are to some extent mutually exclusive @xcite . specifically, consistent stories may limit the choice of words, while diversified wordings may lead to the risk of inconsistent themes. in this paper, we propose to conquer these two challenges simultaneously by leveraging the advantages of mid-level representations learning and the cvae model in improving wording novelty. specifically, we propose a planning-based cvae model, targeting to generate stories with both thematic consistency and wording diversity. our introduced method can be divided into two stages. in the planning stage, keyword extraction and expansion modules are used to generate keywords as sub-topics representations from the title, while in the generation stage, a cvae neural network module is employed to generate stories under the guidance of previously generated keywords. in our method, the planning strategy aims to improve the thematic consistency while the cvae module is expected to keep the wording diversity of the story. to evaluate our proposed method, we conduct experiments on a benchmark dataset, i.e., the rocstories corpus @xcite . experimental results demonstrate that our introduced method can generate stories that are more preferable for human annotators in terms of thematic consistency and wording diversity, and meanwhile outperforms state-of-the-art methods on automatic metrics. ## results and analysis ## conclusion in this paper, we proposed a planning-based conditional variational autoencoder model (plan-cvae) for story generation. our proposed method involves two stages. in the planning stage, the keywordextraction and keyword-expansion modules are used to generate keywords from the title. as for the generation stage, a cvae neural network module is employed to generate stories under the guidance of keywords. in our method, the planning strategy aims to improve the thematic consistency while the cvae module is expected to keep the wording diversity of the generated story. experimental results of both automatic and human evaluations on a benchmark dataset, i.e., rocstories corpus, show that our method performs better than existing methods on thematic consistency and wording diversity. the case study also confirms the effectiveness of our method.