Long Text Generation with Topic-aware Discrete Latent Variable Model (2022)

Abstract: Generating coherent long texts is an important yet challenging task, particularly forthe open-ended generation. Prior work based on discrete latent codes focuses on the modeling of discourse relation, resulting in discrete codes only learning shallow semantics (Ji and Huang, 2021). A natural text always revolves around several related topics and the transition across them is natural and smooth.In this work, we investigate whether discrete latent codes can learn information of topics. To this end, we build a topic-aware latent code-guided text generation model. To encourage discrete codes to model information about topics, we propose a span-level bag-of-words training objective for the model. Automatic and manual evaluation experiments show that our method can generate more topic-relevant and coherent texts.

## introduction generating coherent long texts based on a shorter text input is an important yet challenging task. recent large-scale pre-trained language models (lms), e.g. gpt-2 @xcite and bart @xcite have shown state-of-the-art performance on various natural language generation (nlg) tasks such as summarization and dialog generation @xcite . although pre-trained language models can generate fluent texts, it is still challenging for them to generate coherent long-form texts. to address this, various text generation methods have been proposed to improve the coherence of generated texts. the first type of methods attempt to decompose the long text generation procedure into two stages: first producing a skeleton and then generating a text based on the skeleton, where the skeleton can be a set of keywords @xcite @xcite , events @xcite , etc. however, these methods more or less rely on highprecision automatic extraction tools. the second line of methods introduce latent variables to model high-level structures of texts @xcite @xcite . @xcite develop multi-level vae models @xcite for modeling paragraphlevel and sentence-level features. recently, @xcite have explored discrete latent variables @xcite to model intersentence relations and achieved impressive performance. however, through preliminary experiments, we have found that their methods often produced texts unrelated to the input prompts, as shown in intuitively, a long text includes a latent topic sequence, and using the topic sequence to guide the generation process can help to generate topicrelated and coherent texts. given the above discussion, we propose a topic-aware latent code-guided text generation model. the main idea is to learn discrete latent codes with topic-aware information. concretely, as shown in figure 1 , we first abstract a given long text to a discrete code sequence according to the fixed span length, and then it is spliced with the prompt and input into a pre-training language model to reconstruct the given text. to make the discrete latent codes capture information about topics foot_0 , we further propose a span-level bag-ofwords prediction auxiliary task to ensure that each code in the discrete sequence can reconstruct the bag-of-words of its corresponding text span. once discrete latent codes are learned, we use another auto-regressive transformer to fit the prior distribution of them. bart encoder y1, y2, ..., ym cnn cnn cnn cnn vector quantize z 1 z 2 z 3 z 4 bart encoder bart decoder sep y1, y2, ..., ym x z 1 , z 2 , z 3 , z 4 mlp z i ...... span-level bag-of-words loss we summarize our contributions as follows: ## our approach we formulate the long text generation task as a conditional generation problem, i.e., generating a long text @xmath0, y 2 , ..., y m ) given an short input prompt @xmath1, x 2 , ..., x n ), where m and n denote the number of tokens. a natural text can be segmented into successive several text spans, and each span implies a local topic. we suppose the text span length is l, 2 y can be segmented into a span sequence @xmath2, ..., s l ), where each s i consists of l tokens, @xmath3. our idea is to introduce discrete latent variables z to learn local topics of the text. following prior work @xcite , we optimize our model by maximizing the following the evidence lower bound (elbo, kingma and welling (2013)): the training process includes two stages. in the first training stage, we train the generator and posterior network to learn discrete latent codes of text ( §2.1). in the second stage, another auto-regressive model is used as prior network to model the prior distribution of discrete codes ( §2.2). during text generation, the prior network first predicts a discrete code sequence given the input prompt, which is then applied to guide text generation. ## conclusion in this paper, we have presented a topic-aware codeguided text generation model. the model firstly learns a discrete latent code sequence that abstracts the topic sequence of text, which is then applied to guide the generation of text. to encourage discrete codes to learn topical information, we further propose a span-level bag-of-words loss. experimental results show that our model can generate more relevant and coherent texts.