Non-Autoregressive Text Generation with Pre-trained Language Models (2021)

Abstract: Non-autoregressive generation (NAG) has recently attracted great attention due to its fast inference speed. However, the generation quality of existing NAG models still lags behind their autoregressive counterparts. In this work, we show that BERT can be employed as the backbone of a NAG model for a greatly improved performance. Additionally, we devise two mechanisms to alleviate the two common problems of vanilla NAG models: the inflexibility of prefixed output length and the conditional independence of individual token predictions. To further strengthen the speed advantage of the proposed model, we propose a new decoding strategy, ratio-first, for applications where the output lengths can be approximately estimated beforehand. For a comprehensive evaluation, we test the proposed model on three text generation tasks, including text summarization, sentence compression and machine translation. Experimental results show that our model significantly outperforms existing non-autoregressive baselines and achieves competitive performance with many strong autoregressive models. In addition, we also conduct extensive analysis experiments to reveal the effect of each proposed component.

## introduction autoregressive generation (ag) models achieve state-of-the-art performance on a wide range of text generation tasks, such as machine translation @xcite ) and text summarization @xcite . such models generate a token sequence in a left-to-right, token-by-token fashion. the prediction for the next token is conditioned on all previously generated tokens. this characteristic makes it impossible to parallelize the computational overhead for token predictions in different positions, which leads to a relatively high latency in inference. on the other hand, non-autoregressive generation (nag) models @xcite have emerged as a promising alternative due to their fast inference speed. nag models omit the sequential dependencies within the output-side sequence and predict tokens in all positions simultaneously once the output length has been determined beforehand. while nag models enjoy full parallelism and faster inference, the generation quality of nag models often lags behind their autoregressive counterparts. in this work, we explore the potential of largescale pre-trained language models for improving the performance of non-autoregressive generation. specifically, we utilize bert @xcite as the backbone for nag modelling and extend the architecture of bert with a crf output layer @xcite for better capturing the output-side dependencies. in addition, we analyze two significant limitations that nag models currently suffer from: (1) the inflexibility of prefixed output length, and (2) the conditional independence of individual token predictions. accordingly, we devise two solutions to these two problems. first, prior nag models require the output length to be determined before token generation, thus an extra module for output length prediction is always required. nevertheless, the most likely length from the prediction module is not necessarily the best-suited one for the token generation model. to this end, previous works @xcite usually rely on length-parallel decoding (lpd) @xcite for performance enhancement; that is, generating and re-ranking the results from different output length candidates. in this work, we propose a simple and elegant decoding mechanism that lets the model determine the output length on-the-fly. specifically, our model dynamically adjusts the output sequence length via emitting an @xcite token at any output position to indicate the ending of the generated sequence. therefore, we can avoid the additional efforts of output length prediction and results re-ranking. second, most existing nag models assume the token predictions in different positions are conditionally independent. as a consequence, they often tend to generate results that are ungrammatical with repetitions @xcite . to alleviate this problem, we propose a context-aware learning objective which impels the model to output different tokens at adjacent positions, thereby reducing the possibility of repetitive generation. furthermore, for tasks like text summarization, the output sequence (summary) is known to be shorter than the source sequence (article). in such cases, to further improve the model's inference efficiency, we introduce a new ratio-first decoding strategy. specifically, instead of performing inference on all source-side hidden states, ratio-first generates the result only based on a subset of source hidden states. the subset size is jointly determined by the source length t and a predefined ratio α that is set based on our prior knowledge from the data statistics. in the experiments, we show that ratio-first can significantly improve the inference speed while maintaining the generation quality. we evaluate the proposed model on three typical text generation tasks, including text summarization, sentence compression and machine translation. experimental results show that our model significantly outperforms many strong non-autoregressive baselines, and even performs competitively with several strong autoregressive models. in addition, we conduct extensive analysis experiments to study the effect of individual proposed components. in summary, our contributions are: (1) we propose a novel framework that utilizes bert for text generation under the non-autoregressive generation paradigm; (2) we propose a decoding mechanism that allows the model to dynamically determine the output length, and a new context-aware learning objective that reduces errors stemming from the output-side conditional independence assumption; ## background autoregressive generation (ag) models generate sequences based on a left-to-right factorization. as shown in figure 1 , given the source sequence x, the target sequence y with length t is generated via a chain of conditional probabilities based on the left-to-right sequential dependencies as: with this conditional independence assumption, nag models can fully parallelize their generation process, which significantly improves the inference speed. however, it has been shown that, the choice of the prespecified output length has a notable impact on the model's generation quality @xcite . in addition, the removal of output-side sequential dependency also causes the generation quality of nag models to be inferior to their autoregressive counterparts @xcite . ## proposed model in this section, we give a detailed explanation of the proposed model. first, we describe how to utilize bert as a non-autoregressive generation model. then we discuss the decoding mechanism which allows the model to determine the output length dynamically. finally, we introduce the new ratiofirst decoding strategy which further improves the model's decoding efficiency. ## learning due to the conditional independence approximation on output tokens, nag models often tend to generate repeated tokens @xcite . one way to alleviate this problem is to introduce implicit dependencies on the output side. in this work, we propose to use the unlikelihood formulation of @xcite in the context of nag, where we define the set of negative candidate as the surrounding tokens within a predefined context window c. formally, given the source sequence x and the target sequence y with length t , the proposed context-aware objective is defined as: ## related work non-autoregressive generation was first introduced by @xcite to reduce the inference latency in machine translation. recent works in this area have investigated ways to mitigate the tradeoff between the decoding speed and generation quality. gu et al. (2018) utilized fertility as latent variables for better translation performance. wang et al. (2019b) proposed two auxiliary objectives for better modelling the output states and solving the under-translation problem. to better model the intermediate alignments between source and target sides, ma et al. (2019) proposed a model based on the generative flow framework. ghazvininejad et al. (2019) proposed to use a masked language objective to train the nag model. during inference, starting from a fully masked sequence, the output is generated in an iterative refinement manner. recently, sun et al. (2019) proposed to incorporate a conditional random field into the decoder of a nag model for better modelling the outputside dependencies. our work is different from prior works in two aspects: (1) we directly utilize a pretrained language model (bert) to perform nonautoregressive generation; (2) our model can dynamically generate the output sequence without the need of prespecified output length. ## experiments we evaluate the proposed model on three typical text generation tasks: (1) text summarization; (2) sentence compression and (3) machine translation. ## conclusion in this work, we explored the potential of bert in various text generation tasks under the nag framework. to address problems from nag models previously having a prefixed output length, we devised a decoding mechanism which enables the model to determine the output length dynamically. to reduce errors stemming from the assumption of conditional independence of output tokens, we proposed a context-aware objective as well as using a crf decoding. furthermore, to maximize the inference speed advantage of our model, we introduced a ratio-first decoding strategy. we evaluated our model on three benchmark datasets and the results show that our model significantly outperforms many strong nag baselines and performs comparably to many strong ag models.