Progressive Generation of Long Text with Pretrained Language Models (2021)

Abstract: Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. However, as our systematic examination reveals, it is still challenging for such models to generate coherent long passages of text (e.g., 1000 tokens), especially when the models are fine-tuned to the target domain on a small corpus. Previous planning-then-generation methods also fall short of producing such long text in various domains. To overcome the limitations, we propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution. Our method first produces domain-specific content keywords and then progressively refines them into complete passages in multiple stages. The simple design allows our approach to take advantage of pretrained LMs at each stage and effectively adapt to any target domain given only a small set of examples. We conduct a comprehensive empirical study with a broad set of evaluation metrics, and show that our approach significantly improves upon the fine-tuned large LMs and various planning-then-generation methods in terms of quality and sample efficiency. Human evaluation also validates that our model generations are more coherent.

## introduction generating coherent long text (e.g., 1000s of tokens) is useful in myriad applications of creating reports, essays, and other long-form content. yet the problem is particularly challenging as it demands models to capture global context, plan content, and produce local words in a consistent manner. prior studies on "long" text generation have typically limited to outputs of 50-200 tokens @xcite @xcite . 1 code available at https://github.com/ tanyuqian/progressive-generation figure 1 : results of large-scale lms (gpt-2 and bart) fine-tuned on 10k stories. coherence of text is evaluated by bert next sentence prediction (nsp) score, where x-axis is the position of the evaluated sentences in the passage. there is a significant gap in coherence between text by human and text by large-scale lms. our proposed progen instead generates more coherent samples close to human text. recent large-scale pretrained language models (lms), such as gpt-2 @xcite and bart @xcite , emerged as an impressive open-ended text generator capable of producing surprisingly fluent text. the massive lms are typically pretrained on large corpora of generic text once, and then fine-tuned with small domainspecific data. the latest work has mostly focused on the regime of relatively short text with low hundreds of tokens. for example, @xcite ; @xcite ; @xcite studied gpt-2 and bart generations with a maximum length ranging from 150 to 350 tokens. in this work, we study the problem of generating coherent, much longer passages of text (e.g., 1000 tokens). @xcite was reported to produce long essays, yet the results seem to need extensive human curations (e.g., marketmuse; gardian), and the system is not publicly available to adapt to arbitrary desired domains. in this work, we examine fine-tuning of largescale lms for domain-specific generation of extra-long text. we find that samples produced by gpt-2 fine-tuned on small domain-specific corpora exhibit various imperfections, including excessive repetitiveness and incoherence between sentences far apart. figure 1 measures the coherence of text generated by the fine-tuned gpt-2 w.r.t the bert next sentence prediction @xcite score. as the figure shows, gpt-2 models (regardless of the model size) exhibit a significant gap in the score compared with human text, hence falling short in generating coherent text. we hypothesize that the problem is mainly caused by the sequential generation order of the lms, which makes global content planning of the passage difficult, especially when the generated text is long and contains thousands of words. one could potentially adopt the recent planning-thengeneration or non-monotonic methods (sec 2), yet those methods either require specialized neural architectures that need costly retraining for each domain @xcite @xcite , or rely on dedicated intermediate content plans (e.g., summaries, srl labels) @xcite with limited flexibility and producing sub-optimal results as shown in our experiments. to overcome the limitations, we introduce a new method for progressive generation of text (pro-gen). we observe that generation of some words (e.g., stop words) does not require many contexts, while other words are decisive and have long-term impact on the whole content of the passage. motivated by this observation, our approach first produces a sequence of most informative words, then progressively refines the sequence by adding finergrained details in multiple stages, until completing a full passage. the generation at each stage is conditioning on the output of the preceding stage which provides anchors and steers the current generation (figure 2 ). the intermediate words produced at each stage are defined based on a simple tf-idf informativeness metric. the approach enjoys several core advantages: we conduct extensive empirical studies on the cnn news @xcite and writing-prompts @xcite corpora, evaluating various systems by a wide-range of automatic metrics as well as human judgement. results show that pro-gen achieves strongly improved performance by decomposing the generation into more progressive stages. our method produces diverse text passages of higher quality and coherence than a broad set of models, including fine-tuned gpt-2, bart, and other various planning-then-generation strategies. ## progressive generation of text one of the main challenges in generating long coherent passages is modeling long-range dependencies across the entire sequences (e.g., 1000 tokens). we propose a progressive generation approach that is conceptually simple yet effective. intuitively, progressive generation divides the complex problem of generating the full passage into a series of much easier steps of generating coarser-grained intermediate sequences. contrary to generating everything from left to right from scratch, our progressive generation allows the model to first plan globally and then shift attention to increasingly finer details, which results in more coherent text. figure 2 illustrates the generation process. ## conclusion we have proposed a new approach for domainspecific generation of long text passages in a progressive manner. our method is simple and efficient by fine-tuning large-scale off-the-shelf language models. we conduct extensive experiments using a variety of metrics and human studies. we demonstrate that our method outperforms a wide range of large pretrained lms with single-stage generation or prior planning-then-generation strategies, in terms of quality and coherence of the produced samples. the multi-stage generation also opens up new opportunities to enhance controllability of text generation, which we would love to explore in the future.