Event Transition Planning for Open-ended Text Generation (2022)

Abstract: Open-ended text generation tasks, such as dialogue generation and story completion, require models to generate a coherent continuation given limited preceding context. The open-ended nature of these tasks brings new challenges to the neural auto-regressive text generators nowadays. Despite these neural models are good at producing human-like text, it is difficult for them to arrange causalities and relations between given facts and possible ensuing events. To bridge this gap, we propose a novel two-stage method which explicitly arranges the ensuing events in open-ended text generation. Our approach can be understood as a specially-trained coarse-to-fine algorithm, where an event transition planner provides a “coarse” plot skeleton and a text generator in the second stage refines the skeleton. Experiments on two open-ended text generation tasks demonstrate that our proposed method effectively improves the quality of the generated text, especially in coherence and diversity. We will release the codes to the community for further exploration.

## introduction with the fast development of large-scale pre-trained models, considerable progress has been made in improving the quality of machine generated text @xcite @xcite @xcite . today, machine learning models can do extremely well in generating text that looks human @xcite . the problem is still far from solved, however, as further reading of the machine-generated text often exposes defects such as self-contradiction and topic drifting @xcite @xcite ## methodology we focus on the conditional language modeling problem in open-ended text generation tasks. formally, given an input context x, models are required to generate a sentence y that is consistent with input context and not contradicts itself. in this work, we propose a two-stage model for the generation process. in the first stage, we extract the starting event sequence r x from the input context and employ the event transition planer to generates subsequent event transition path r y based on r x . in the second stage, the output text is generated from an auto-regressive model conditioning on the path and the preceding context x. figure 2 gives an overview of our coarse-to-fine framework for open-ended text generation. in a nutshell, we first fine-tune a gpt-2 on event transition sequences as an event planner (i.e., a conditional generative model for event paths). this fine-tuning involves event transition sequences extracted from both commonsense graphs and the training set. we then build a path-aware text generator with an event query layer specifically designed to refer to the planned path when generating the output. ## experiments we conduct experiments on two open-ended text generation tasks, dialogue generation and story completion, to answer the following questions: ## related work recent advances in pre-trained language models have resulted in impressive performances on open-domain text generation, such as story com-pletion @xcite @xcite , dialogue generation @xcite @xcite , question generation @xcite , and so on. for example, in dialogue generation, zhang et al. (2020b) design a trainable generative pretrained transformer by training an autoregressive language model on large-scale reddit contextresponse pairs with a maximum mutual information scoring function to improve diversity. goldfarb-tarrant et al. (2020) integrate semantic role labels and prompts into pre-trained bart (lewis et al., 2020) during fine-tuning for prompt based story telling. in this paper, we focus on story completion and dialogue generation and build a generative coarse-to-fine method to generate open-ended text with explicit event transition paths. despite the success of generative pre-trained language models on a series of open-ended text generation tasks, they still suffer in maintaining coherence throughout multiple sentences due to the left-to-right word-by-word generation style @xcite . to alleviate this problem, one research direction adopts coarse-to-fine progressive text generation @xcite . this generation paradigm has been studied in many text generation systems for specific tasks, such as datato-text generation @xcite , storytelling @xcite , and dialogue generation @xcite . our work adopts a generative event transition planner that is trained on a large amount of event transition paths, aiming to arrange the ensuing events in open-ended text generation. another research direction incorporates external entities to guide the open-ended text generation @xcite @xcite @xcite . ji et al. (2020) and xu et al. (2020b) retrieve entities from knowledge bases to control the generated content. however, the retrieval-based methods also suffer from the sparsity problem and the domain shift between external sources and downstream tasks @xcite . @xcite integrate entity relations into pre-trained language model via additional tuning on entity triples. even with such specialized learning, the resulted model still often stuck in logical errors or repeats pieces of narratives @xcite . this phenomenon demonstrates the need for an intact inductive bias on organizing event transition patterns for open-ended text generation. different from using event triples as additional training instances, our method explicitly maintains generative event transition paths to make the generation process more explainable and improve the coherence. ## conclusion in this paper, we propose a novel two-stage method to improve high-level consistency and diversity in open-ended text generation. we design a specialtrained event transition planner to explicitly arrange the ensuing events and introduce an event-pathaware text generator to exploit the event transition guidance for language generation. we investigate two open-ended text generation tasks, i.e., story completion and dialogue generation. thorough experiments demonstrate that the explicit arrangement of event transition path indeed facilitate models to generate more coherent and diverse text in open-ended scenery. besides, with the proposed event prompt and event query layer, our method could be extended to any other language models and open-ended generation tasks. a future line of investigation is to explore the effect of the proposed method on other open-ended tasks, such as commonsense question answering. yang xu. 2021. global divergence and local convergence of utterance semantic representations in dialogue. in proceedings of the society for computation in linguistics 2021, pages 116-124, online. association for computational linguistics. lili yao, nanyun peng, ralph m. weischedel, kevin knight, dongyan zhao, and rui yan. 2019. planand-write: towards better automatic storytelling. in the thirty-third aaai conference on artificial intelligence, aaai 2019, the thirty-first innovative applications of artificial intelligence conference, iaai 2019, the ninth aaai symposium on educational advances in artificial intelligence, eaai 2019, honolulu, hawaii, usa, january 27 -february 1, 2019, pages 7378-7385. aaai press. changlong yu, hongming zhang, yangqiu song, and wilfred ng. 2020. cocolm: complex commonsense enhanced language model. corr, abs/2012.15643. houyu zhang, zhenghao liu, chenyan xiong, and zhiyuan liu. 2020a. grounded conversation generation as guided traverses in commonsense knowledge graphs. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 2031-2043, online. association for computational linguistics. yizhe zhang, siqi sun, michel galley, yen-chun chen, chris brockett, xiang gao, jianfeng gao, jingjing liu, and bill dolan. 2020b. dialogpt : largescale generative pre-training for conversational response generation. in proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, pages 270-278, online. association for computational linguistics.