D isco DVT : G enerating Long Text with Discourse-Aware Discrete Variational Transformer (2021)

Abstract: Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware discrete variational Transformer to tackle the incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes the global structure of the text and then applies it to guide the generation process at each decoding step. To further embed discourse-aware information into the discrete latent representations, we introduce an auxiliary objective to model the discourse relations within the text. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.

## introduction generating passages that maintain long-range coherence is a long-standing problem in natural language generation @xcite . despite the recent advances of large pre-trained language generation models @xcite in various nlg tasks such as summarization and dialogue generation that target generating locally coherent texts which are relatively short, it is still challenging for pre-trained models to generate globally coherent passages spanning over dozens of sentences. global coherence in human texts is represented by the topic-maintenance and natural transition between viewpoints @xcite . as illustrated in figure 1 , discourse relations such as causal, temporal succession between contiguous text segments are commonly indicated by the highlighted discourse markers which bind collocated text segments into a global structure @xcite . although pre-trained language models are inspected to perform reasonably well in associating topic-related concepts, they can hardly arrange contents with well-structured discourses @xcite . in this work, we urge the revival of variational autoencoder (vae) with its global representation ability to tackle the incoherence issue in long text generation in the era of pre-trained language models. to represent texts with high-level structures, we propose to learn a latent variable sequence with each latent code abstracting a local text span. instead of the commonly used continuous latent variables, we resort to learn discrete latent codes that naturally correspond to interpretable categories in natural languages @xcite . for the latent codes to capture the explicit discourse structure of the texts as shown in figure 1 , we further design an auxiliary objective on the latent representations to model the discourse relations. we name the proposed model as discodvt, i.e., discourse-aware discrete variational transformer. the main idea is to learn a discrete latent variable sequence that summarizes the long text to reconstruct the original text by guiding the decoding process. the learning schema is shown in figure 2 (a). at the encoding phase, to capture the high-level structure of the text, we first use a bidirectional encoder to obtain contextualized token representations and then apply 1-dimensional convolutional neural networks (1d cnns) to abstract text segments at the temporal scale ( §3.2.1). to condense the continuous representations into categorical features, we map them into a one-hot categorical distribution over a fixed latent vocabulary, and obtain the discrete variable sequence ( §3.2.2). at the decoding phase, to apply the global discrete latent codes to guide the local text realization, the latent embeddings are first rescaled to the text length with transposed 1d cnns, and then added to the embedding layer of the decoder for step-wise control ( §3.2.3). for the latent codes to abstract the discourse structure of the text, we use explicit discourse relations from penn discourse treebank 2.0 (pdtb, @xcite and extract adjacent elementary discourse units (edus) from texts as shown in figure 1 and introduce an anxiliary objective to embed the relations into the latent representations. once the discrete latent codes are learned, we adopt an autoregressive transformer to model the prior distribution as a sequence transduction task ( §3.4). we summarize our contributions in three folds: ## ethics statement we observe that the proposed model may sometimes generate inaccurate or fictitious contents due to the systematic biases of model pre-training on the web corpora and the open-domain characteristics of the story generation datasets. we recommend the users to carefully examine the ethical ramifications of the generated contents in the realworld applications and demonstrations. ## conclusion we present discodvt, a discourse-aware discrete variational transformer for long text generation. discodvt learns a discrete variable sequence that summarizes the global structure of the text, which is then applied to guide the step-wise decoding process to maintain a coherent discourse structure. we further introduce a discourse-aware objective to the discrete latent representations to model discourse relations within the text. extensive experiments demonstrate that the discodvt can generate long texts with better long-range coherence with interpretable latent codes.