ELMER : A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation (2022)

Abstract: We study the text generation task under the approach of pre-trained language models (PLMs). Typically, an auto-regressive (AR) method is adopted for generating texts in a token-by-token manner. Despite many advantages of AR generation, it usually suffers from inefficient inference. Therefore, non-autoregressive (NAR) models are proposed to generate all target tokens simultaneously. However, NAR models usually generate texts of lower quality due to the absence of token dependency in the output text. In this paper, we propose ELMER: an efficient and effective PLM for NAR text generation to explicitly model the token dependency during NAR generation. By leveraging the early exit technique, ELMER enables the token generations at different layers, according to their prediction confidence (a more confident token will exit at a lower layer). Besides, we propose a novel pre-training objective, Layer Permutation Language Modeling, to pre-train ELMER by permuting the exit layer for each token in sequences. Experiments on three text generation tasks show that ELMER significantly outperforms NAR models and further narrows the performance gap with AR PLMs (ELMER (29.92) vs BART (30.61) ROUGE-L in XSUM) while achieving over 10 times inference speedup.

## introduction since the advant of gpt-2 @xcite , pre-trained language models (plms) have achieved state-of-the-art performance across text generation tasks, which aim to generate human-like texts on demand @xcite . these plms usually adopt an auto-regressive (ar) fashion to generate texts token-by-token: the next token is predicted based on all previously generated tokens. a major limitation of this approach is that it is hard to be parallelized for the inference * corresponding author process, thus leading to a relatively high inference latency @xcite . such a limitation prevents ar models from wide deployment in online realtime applications, such as query rewriting in search engines and online chat-bot. moreover, ar models are prone to suffering from the exposure bias problem since there is a gap between ar training and inference @xcite . these concerns have sparked extensive interests in non-autoregressive (nar) models for text generation @xcite . compared to ar models, nar models predict target tokens in all positions simultaneously and independently @xcite . this full parallelism leads to an efficient and low-latency inference process. however, the independence assumption prevents nar models from learning the dependency among target tokens, resulting in accuracy degradation @xcite . one widely-used solution to improve the nar generation quality is to iteratively refine outputs @xcite , which however leads to the loss in the speed-up advantage. in addition, many studies aim to learn the input-output mapping for more accurate generation via embedding mapping @xcite , latent alignment @xcite , and discrete variables @xcite . while easing the difficulty of nar generation to some extent, these methods still struggle for generating complex sentences. therefore, inspired by @xcite , we argue that the key to nar text generation is to enhance the learning of token dependency-each token should be generated depending on forward and backward generated tokens. in this paper, we propose elmer: an efficient and effective plm for nar text generation, to explicitly learn the bi-directional token dependency. typically, most nar models predict tokens simultaneously only at the last layer, thus making the token prediction unaware of tokens generated in other positions. to address this issue, we propose to generate tokens at different layers and the upperlayer token generation can depend on lower-layer generated tokens from both left and right. in this way, our model can explicitly learn the dependency between tokens from different layers while enjoying full parallelism in nar decoding, as shown in figure 1 . to this end, we propose to extend the early exit technique @xcite to nar text generation: if there is sufficient confidence to generate a token at a lower layer, the model is allowed to exit at this layer and make the prediction without passing through the upper layers. furthermore, instead of exiting at a fixed layer for a token, we aim to predict each token at different layers for learning diverse token dependencies in nar text generation. thus, inspired by xlnet @xcite , we further propose a novel pretraining objective based on early exit, i.e., layer permutation language modeling (lplm), to help elmer learn complex token dependencies. given a sequence, lplm will permute the exit layer (from 1 to the maximum layer) for each token and maximize the nar text generation probability w.r.t. all possible exit layer permutations of the sequence. through lplm, each token is able to exit at different layers and attend to all other tokens from both forward and backward positions. in this way, lplm could effectively capture diverse token dependencies from large-scale corpora. pre-trained with the general lplm, elmer can adapt to downstream text generation tasks and datasets by using specific early exit strategies. to the best of our knowledge, we are the first to introduce the idea of early exit to nar text generation. we fine-tune elmer on three popular text generation tasks. experiments show that elmer significantly improves the best nar models by +5.71 rouge-1 on xsum, +1.09 me-teor on squad v1.1, and +2.26 distinct-2 on personachat, and narrows the performance gap with auto-regressive plms (e.g., elmer (29.92) vs bart (30.61) rouge-l on xsum) while achieving over 10x faster inference. ## preliminaries generally, the goal of text generation is to model the conditional probability pr(y|x ), where @xmath0, . . . , x n ⟩ and @xmath1, . . . , y m ⟩ denote the input text and output text respectively and each consists of a sequence of tokens from a vocabulary v. there are three common generation paradigms to model the conditional probability pr(y|x ), i.e., autoregressive (ar), non-autoregressive (nar), and semi-nonautoregressive (semi-nar) generation. ar generation. ar generation models predict the output text based on a left-to-right factorization as: nar generation. in contrast to ar models, nar text generation models predict each token in output text simultaneously as follows, without modeling the forward or backward token dependency: semi-nar generation. semi-nar generation is formalized between ar and nar generation as: in this paper, we mainly focus on the nar approach, considering both effectiveness and efficiency for text generation models. ## approach our proposed nar text generation plm, elmer, is depicted in figure 1 . elmer aims to enhance the modeling of token dependency for nar models. with early exit, tokens exiting at different layers can build the bi-directional token dependency with each other. moreover, we design layer permutation language modeling (lplm) to pre-train elmer by permuting the exit layer for each token. next, we will describe each part in detail. layer 1 layer 2 layer 3 layer 4 𝑦 ! 𝑦 " 𝑦 # 𝑦 $ [mask] [mask] [mask] [mask] [mask] copy hidden state early exit ... layer 𝐿 ## experiments in this section, we detail the experimental setup and then highlight the main takeaways of our results. ## conclusion this paper presented an efficient and effective plm for nar text generation, called elmer. elmer introduced a token-level early exit mechanism to explicitly model the semantic dependency between target tokens during parallel decoding. moreover, we proposed a pre-training objective, layer permutation language modeling, to pre-train elmer on large-scale corpora by permuting the exit layer for each token in a sequence. experiments on text summarization, question generation, and dialogue generation demonstrate that our model can effectively improve the nar generation quality compared to highly competitive nar models and further narrow the performance gap with ar models while achieving higher inference efficiency. in future work, we will investigate the efficacy of utilizing the proposed model to iteratively generate texts and consider more early exit strategies in fine-tuning.