Open-ended Long Text Generation via Masked Language Modeling (2023)

Abstract: Pre-trained autoregressive (AR) language models such as BART and GPTs have dominated OPen-ended Long Text Generation (Open-LTG).However, the AR nature will decrease the inference efficiency along with the increase of generation length, which hinder their application in Open-LTG.To improve inference efficiency, we alternatively explore the potential of the pre-trained masked language models (MLMs) along with a representative iterative non-autoregressive (NAR) decoding strategy for Open-LTG.Our preliminary study shows that pre-trained MLMs can merely generate short text and will collapse for long text modeling. To enhance the long text generation capability of MLMs, we introduce two simple yet effective strategies for the iterative NAR model: dynamic sliding window attention (DSWA) and linear temperature decay (LTD). It can alleviate long-distance collapse problems and achieve longer text generation with a flexible trade-off between performance and inference speedup. Experiments on the storytelling and multi-paragraph opinionated article writing tasks show that pre-trained MLMs can achieve more than 3 \times \to 13 \times speedup with better performance than strong AR models.

## introduction pre-trained language models (plms) like bart @xcite and gpts @xcite @xcite have achieved remarkable progress in open-ltg. through modeling languages from left to right, they can autoregressively "create" fluent and grammatical content. with the further enhancement of planning strategies @xcite or high-level representation learning (guan et al., 2021a), pre-trained ar language models can achieve promising open-ltg. however, the low inference efficiency of ar impedes their usability in real-world applications. the high inference efficiency of nar models is at the sacrifice of output dependency modeling, in which each generation is executed in parallel @xcite . thus, nar models are mainly explored and utilized for text generation tasks with adequate input information to predict each output token of different positions and extra correlations to constrain the generation process, e.g., neural machine translation @xcite , summarization @xcite , sentence compression @xcite , dialogue generation @xcite , and constrained story-ending generation @xcite . to the best of our knowledge, none of the existing research explores open-ltg with nar models, particularly based on pre-trained mlms. we fill this gap by first conducting a preliminary study to calibrate the potential and limitations of a pre-trained mlm, i.e., roberta (liu et al., 2019) † , on two story generation corpora, i.e., rocstories (roc) @xcite and writingprompts (wp) @xcite . to achieve conditional generation, we simply use roberta as both the encoder and the decoder with mixed attention @xcite to achieve encoder-decoder cross-attention. through experiments, we found that: (1) pre-trained mlms can achieve competitive performance in the iterative nar fashion for open-ended short text generation (e.g., a paragraph with around 40 tokens), (2) pre-trained mlms fail to model open-ltg (with about 140 tokens on average), which will generate uninformative content with high-frequency and repeated tokens (e.g., "." and ","). furthermore, we offer three possible reasons for the attention mechanism of mlms and inference strategy to explain the collapse of the iterative nar model based on pre-trained mlms for the open-ltg scenario. inspired by the above observations, we introduce two improvement strategies: dynamic sliding window attention (dswa) and linear temperature decay strategy (ltd) to maintain more informative context content in the iterative nar generation. as a result, iterative nar models based on pre-trained mlms can achieve much longer text generation than the vanilla setting. experiments on two open-ltg tasks (i.e., storytelling and multi-paragraph opinionated article writing) with four widely-used datasets demonstrate that the pre-trained mlm can achieve better performance (bleu score, rouge score, bert score, and perplexity) than multiple strong ar models without extra post-training, structure modification, or using more model parameters. importantly, our approach can speed up the inference process due to non-autoregressive properties, making the pre-trained mlm as a promising candidate for the open-ltg community. the roberta base achieves more than 3 × → 13 × with better performance to the competitive bart. ## related work long text generation text generation tasks can be classified into two categories: directed generation and open-end generation. the directed generation @xcite @xcite for long text scenarios has long source than the target, which is also constrained by source sequence, e.g., neural machine translation and summarization. these tasks aim to solve the quadratic growth requirement of the memory and computational of the self-attention mechanism. the openended generation task @xcite @xcite @xcite desire to generate more freedom content and has recently become a promising research direction. previous works have explored multiple generation strategies to generate high-quality and fluent text, e.g., planning then generating @xcite @xcite @xcite and introducing external knowledge @xcite . although the above strategies enable the model to achieve significant advances, time-consuming is still a critical issue that hinders their usage in real-world applications @xcite . iterative non-autoregressive generation nonautoregressive (nar) model breaks the sequential dependencies from front to back for parallel text generation @xcite @xcite . furthermore, the iterative-based nar model @xcite @xcite can achieve comparable performance with the ar model. the typical cmlm model @xcite can generate fluent results conditioned on the predictions from the previous iteration instead of previous tokens: benefiting from this, the iterative nar model is more flexibly compared with the ar model, which can easily generate consistent and controllable text for each iteration step. to the best of our knowledge, the iterative nar model has never been used to solve open-ended generation. especially, we investigate its usability for the long text scenario, i.e., target lengths between 100 and 400, which is still under-explored in the directed generation tasks. ## preliminary study we first present the training and inference paradigm of utilizing the pre-trained mlms for open-ltg ( § 3.1), e.g., bert or roberta. then, we study the significant collapse problem in a long text generation scenario by conducting preliminary experiments on two datasets with different target lengths ( § 3.2). finally, we investigate the reason for the above issues with an exhaustive case study and exploration tests to motivate our method design ( § 3.3), where the model can generate text in nonautoregressive manner to speed up the inference. ## method in this section, we propose two simple yet effective strategies for attention mechanism and inference to mitigate the model collapse problems: dynamic sliding window attention (dswa) and linear temperature decay (ltd). these designs do not break the paradigm of mlm so that it can flexibly adapt to the pre-trained models. ## limitation although our nar approach can generate fluent and meaningful text, it inevitably suffers from the typical generation problems like in the ar fashion: