Coherent Long Text Generation by Contrastive Soft Prompt (2022)

Abstract: Improving the coherence of long text generation is an important but challenging task. Existing models still struggle to generate a logical and coherent sentence sequence. It is difficult for a model to plan long text generation and avoid generating incoherent texts from a high-level semantic perspective. We speculate that this is due to two factors: (1) current training methods mainly rely on maximum likelihood estimation computed from token-level probability prediction; (2) the role of incoherent texts has been largely under-explored, thus the noised generated texts with errors are out-of-distribution for the model. To address these issues, in this paper, we propose a Contrastive Soft Prompt (CSP) model for improving the coherence of long text generation. It learns text representations in the hidden space for better planning long text generation. To this end, it jointly learns to generate a text representation close to representations of coherent texts and away from incoherent ones, and then generate long text taking this representation as the soft prompt. We conduct experiments on two public story generation datasets, and experiment results show that our method can generate more coherent stories than the state-of-the-art model.

## introduction generating coherent long text plays a key role in many applications, e.g. news report generation @xcite , story generation @xcite ), text adventure games @xcite . taking story generation as an example, it requires the model to generate a reasonable story for a given prompt or a given leading context. in recent years, pre-trained language models @xcite have demonstrated their scalability to large-capability and datasets, becoming a de-facto standard for text ## method our task aims at generating a multi-sentence text @xmath0, y 2 , ..., y m ), given a text input @xmath1, x 2 , ..., x n ). figure 1 shows the structure of our proposed model. our proposed model consists of three parts, prompt generator, prompt posterior generator, and text generator. the prompt generator aims at generating a soft prompt that represents the hidden representation of the text to be generated. the prompt posterior generator is used to help train the prompt generator, which provides the hidden representations of positive and negative samples so that the prompt generator is trained to generate soft prompts close to positive samples and away from negative ones. the text generator generates the text using the soft prompt and the input. ## conclusion in this paper, we propose a contrastive soft prompt method for improving the coherence of long text generation. it learns long text representations in the hidden space for better planning long text generation. to this end, it jointly learns to generate a text representation close to representations of coherent texts and away from incoherent ones and generates long text taking this representation as the soft prompt. we conduct experiments on two public story generation datasets, and experimental results show that our method can generate more coherent stories than the state-of-the-art model. instruction 1. read the input (a leading sentence or prompt), and compare two stories. 2. select the story which is better with regard to fluency and coherence. where fluency measures linguistic quality while coherence focuses on logicality, e.g. causality and temporal relationship. these two aspects are evaluated independently. input: male had a roommate. text 1: male was a hard worker and a great student… text 2: he was very messy… 1 wins 2 wins tie 1 wins 2 wins tie fluency coherence the memory bank size is 65536. we set λ @xmath2.0, λ @xmath3.1 and λ @xmath4.1. we finetune the model on rocstories and writingprompts for 22000 steps respectively. we use adam optimizer, and the learning rate is set to 5e-5, no weight decay, and the batch size is 16. during generation, we use nucleus sampling with @xmath5.9, and the softmax temperature is 0.7. our model contains 234 million parameters. we run our experiments on one geforce rtx 3090, and it takes about 34 and 39 hours for training models on rocstories and writ-ingprompts datasets respectively.