Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation (2023)

Abstract: Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation. However, in a wider range of text generation tasks, existing NAR models lack proper pre-training, making them still far behind the pre-trained autoregressive models. In this paper, we propose Pre-trained Directed Acyclic Transformer (PreDAT) and a novel pre-training task to promote prediction consistency in NAR generation. Experiments on five text generation tasks show that our PreDAT remarkably outperforms existing pre-trained NAR models (+4.2 score on average) and even achieves better results than pre-trained autoregressive baselines in n-gram-based metrics, along with 17 times speedup in throughput. Further analysis shows that PreDAT benefits from the unbiased prediction order that alleviates the error accumulation problem in autoregressive generation, which provides new insights into the advantages of NAR generation.1

## introduction pre-trained language models have been widely applied in text generation @xcite @xcite , which can effectively improve the performance of downstream generation tasks, especially in low-resource scenarios @xcite . most of these pre-trained language models are based on autoregressive (ar) generation, which produces high-quality texts by predicting each token one by one. however, such a sequential generation process suffers from high latency and low throughput in inference, thereby largely limiting the use of ar models in scenarios with real-time requirements. non-autoregressive (nar) generation is an alternative text generation paradigm @xcite . unlike sequential generation in ar models, nar models predict all tokens in parallel, which largely accelerates the decoding process. although early nar models suffer from serious quality degradation due to the independent token prediction, recent nar studies have made much progress on some generation tasks, such as machine translation @xcite @xcite . notably, @xcite propose directed acyclic transformer, which incorporates a directed acyclic graph to reduce the conflicts in capturing possible outputs, achieving a comparable translation quality to the ar models. despite the success of nar generation in machine translation, it is still challenging to apply nar models to a wider range of generation tasks, mainly due to the lack of appropriate pre-training. although some previous studies have explored pre-training methods such as directly fine-tuning bert for nar generation @xcite @xcite or pre-training nar models from scratch @xcite , their models still have a significant quality gap compared with ar ones. we argue that these methods do not fully exploit the characteristic of nar generation, thereby restricting downstream performance. specifically, we discuss two main issues: (1) previous pre-training tasks are ineffective in promoting sentence-level prediction consistency, making it hard for their models to predict a whole sentence simultaneously while preserving the fluency in downstream nar generation. (2) previous pre-training tasks fail to address the multi-modality problem @xcite , which has proved to be a fundamental and important challenge in training nar models @xcite . in this paper, we introduce predat, a pretrained directed acyclic transformer for highquality non-autoregressive text generation. we utilize the architecture of directed acyclic transformer and further propose a novel pre-training task, double-source text infilling (dsti), aiming to address the above issues in pre-trained nar models. specifically, dsti contains two steps: it corrupts a sentence and scatters the tokens into two sequences, which are fed into the encoder and decoder as two sources of information; then the model is trained to recover the corrupted fragments non-autoregressively. during the pre-training, our model predicts long sentence fragments (about 15 tokens) from nearby contexts, which promotes prediction consistency and bidirectional dependencies. moreover, dsti designs a strategy for creating pre-training data pairs that allow the output sequences to have flexible lengths, which well incorporates various alignment-based nar training objectives to alleviate the multi-modality problem @xcite @xcite . automatic evaluation shows that predat is effective and efficient on five text generation tasks. it remarkably outperforms previous pretrained nar models (+4.2 score on average) and even achieves better results than pre-trained ar baselines in n-gram-based metrics (+0.7 score on average), along with a 17x speedup in throughput. to our knowledge, predat is the first nar model that outperforms pre-trained ar models on various generation tasks in automatic evaluation. further ablation studies verify that our pretraining task designs, including the long fragment prediction and alignment-based training objectives, are crucial for success. to better understand the advantages and weaknesses of nar generation, we use automatic and manual methods to investigate the generated texts in downstream tasks. we find that predat can alleviate the error accumulation in ar generation and improve the relevance to the input, thereby leading to a better performance in n-gram-based metrics. however, we also find that nar models, including predat, are still weaker than ar models in preserving the consistency among generated tokens, leading to grammatical errors such as wrong word choices. we believe that these findings can provide novel insights for future nar studies. ## preliminaries: directed acyclic transformer directed acyclic transformer (dat, huang et al., 2022c; see also figure 1 ) is an nar model that effectively alleviates the multi-modality problem. it introduces a longer decoding sequence and an alignment-based objective to reduce the conflicts in capturing multiple possible outputs. specifically, given the input besides l dat , there are other alignment-based objectives that succeed in alleviating the multi-modality problem in nar generation, such as axe @xcite , oaxe @xcite , and ctc @xcite . in general, these objectives are also obtained by aligning the target y with the feature sequence v , thus denoted by l(v, y ). ## proposed method in this section, we introduce predat, pretrained directed acyclic transformer. we first propose the pre-training task (section 4.1) and then describe the fine-tuning and inference strategies (section 4.2). ## limitations although predat achieves a significant advancement in nar generation, it still faces the following limitations: squad1.1 passage: ## conclusion in this paper, we propose a pre-training task to promote sentence-level consistency and bidirectional dependencies for nar generation. we demonstrate that combining the state-of-the-art nar models with appropriate pre-training can lead to efficient and high-quality text generation on a wide range of tasks, where our predat largely outperforms previous nar pre-trained models in generation quality. we further show that, compared with ar models, predat alleviates error accumulation and enhances relevance to inputs, but still introduces non-negligible grammatical problems, thereby providing new insights into the strengths and weaknesses of nar generation.