Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence (2021)

Abstract: Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing generation models (e.g., BART) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the decoder to capture the high-level semantics and discourse structures in the context beyond token-level co-occurrence. In this paper, we propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. To this end, we propose two pretraining objectives to learn the representations by predicting inter-sentence semantic similarity and distinguishing between normal and shuffled sentence orders. Extensive experiments show that our model can generate more coherent texts than state-of-the-art baselines.

## introduction the ability to generate coherent long texts plays an important role in many natural language generation (nlg) applications, particularly for openended language generation tasks such as story generation, namely generating a reasonable story from a prompt or a leading context. while existing generation models @xcite can generate texts with good intra-sentence coherence, it is still difficult to plan a coherent plot throughout the text, even when using the powerful pretrained models, as illustrated in figure 1 . pretrained generation models have shown stateof-the-art performance on various nlg tasks such as summarization and translation @xcite . however, such tasks figure 1 : story examples written by the fine-tuned bart model @xcite and a human writer given the same leading context from rocstories @xcite . the generated story by bart suffers from severe incoherence issue in spite of some related concepts (in bold). in comparison, the human writer can write a coherent story because they fully consider the context semantics and discourse relations (e.g., the temporal order) among the sentences. in this paper, we propose hint, a generation model equipped with high-level representations for long text generation. typical generative models usually train a left-to-right decoder by next word prediction based on the attention to all the prefix words. in order to encourage the model to capture high-level features, we extend the decoder to represent the prefix information at sentence level and discourse level, respectively, with special tokens which are inserted at the end of each sentence. to effectively learn the representations, we propose two pretraining objectives including: (a) semantic similarity prediction, which requires predicting the inter-sentence similarity using the sentencelevel representation, with the powerful sentence understanding model sentencebert @xcite as the teacher model; and (b) sentence order discrimination, which requires distinguishing between the normal and shuffled sentence orders using the discourse-level representation. the objectives are designed to help the decoder capture the semantics and discourse structure of the prefix, which can benefit modeling the longrange coherence when generating long texts. we summarize our contributions in two folds: i. we propose a generation model named hint for long text generation. hint derives high-level representations for each decoded sentence to model the long-range coherence. we adopt two pretraining objectives called similarity prediction and order discrimination to learn the representations at sentence level and discourse level, respectively. ## conclusion we present hint, a generation model for ation, which can represent the prefix information at sentence level and discourse level in the decoding process. we propose two pretraining objectives including inter-sentence similarity prediction and sentence order discrimination to learn the sentencelevel and discourse-level representations, respectively. extensive experiments demonstrate that hint can generate more coherent texts with related context and proper sentence orders than strong baselines. further analysis shows that hint has better ability of language modeling thanks to ability of modeling high-level coherence.