Diffusion- NAT : Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation (2024)

Abstract: Recently, continuous diffusion models (CDM) have been introduced into non-autoregressive (NAR) text-to-text generation. However, the discrete nature of text increases the difficulty of CDM to generate coherent and fluent texts, and also causes the incompatibility problem between CDM and advanced NLP techniques, especially the popular pre-trained language models (PLMs).To solve it, we propose Diffusion-NAT, which introduces discrete diffusion models (DDM) into NAR text-to-text generation and integrates BART to improve the performance.By revising the decoding process of BART and the typical settings of DDM, we unify the inference process of BART and the denoising process of DDM into the same NAR masked tokens recovering task.In this way, DDM can rely on BART to perform denoising, which can benefit from both the rich pre-learned knowledge of BART and the iterative refining paradigm of DDM.Besides, we also propose the iterative self-prompting strategy to further improve the generation quality.Experimental results on 7 datasets show that our approach can outperform competitive NAR methods, and even surpass autoregressive methods.Our code and data are released at https://github.com/RUCAIBox/DiffusionNAT .

## introduction text-to-text generation @xcite is an essential task in natural language processing, which aims to generate human-like texts satisfying the task demand. to efficiently generate high-quality texts, nonautoregressive (nar) models @xcite are widely explored for text-to-text generation by predicting all tokens in the target text simultaneously, having a lower inference latency. to address these issues, we aim to develop a more effective approach to integrating diffusion models and plms for nar text-to-text generation. instead of using continuous diffusion, we utilize discrete diffusion @xcite ) for text generation, which performs denoising on discrete states (e.g., vocabulary) to recover the original tokens. it is more suitable for modeling discrete text data, making it feasible to develop more unified and compatible solutions to integrate diffusion models and well-trained plms for improving nar text generation. however, both discrete diffusion models and plms neither naturally fit with each other nor the nar text-to-text generation manner, making it hard to directly combine them for improving the nar generation quality. in this paper, we propose diffusion-nat, a selfprompting discrete diffusion model using plms for nar text-to-text generation. the core contribution lies in that we unify the inference process of plms and denoising process of discrete diffusion models into the same masked token recovering task in the nar manner. in this way, plms can play the role of the parameterized denoiser in discrete diffusion models, hence we can combine the merits of both diffusion models (using iterative refining generation) and plms (with rich semantic knowledge) for improving nar text generation. specifically, we select the seq2seq plm, bart @xcite as our backbone by revising its decoding process into the nar masked tokens recovering task. then, we adjust the typical discrete diffusion method to better fit the plm by adding mask tokens as noise, revising the learning objective and removing the time step embeddings. further, as our approach performs the denoising process fully based on the plm, we devise an iterative self-prompting strategy to guide the plm performing multi-turn deliberation and refinement on the intermediate generated results, to enhance the quality of the final output. to verify the effectiveness of our approach, we conduct extensive experiments on seven text-to-text generation datasets. experimental results show that our approach can outperform competitive nar text generation methods, e.g., improving the best nar models by +2.48 bleu-2 on personachat, +4.33 distinct-2 on dailydialog.our approach even surpasses state-of-the-art autoregressive plms, e.g., ours (62.68) v.s. bart (49.59) on bleu-2 in dailydialog, and our (44.2) v.s. bart (38.3) on rouge-l in msnews.besides, our approach also supports ddim @xcite for trading off the time cost and the generation quality during inference. by setting proper diffusion steps (e.g., 100 and 2), our approach can outperform competitive ar and nar models with similar inference latency, respectively. plms for text generation. pre-trained language models (plms) have shown remarkable performance in generating human-like texts @xcite . after pre-training, most existing plms @xcite are fine-tuned following the ar paradigm for text generation. in this way, they either reformulate generation tasks into the language model format (e.g., gpt @xcite ), or leverage the sequence-to-sequence manner to generate the text using an autoregressive decoder (e.g., bart @xcite ). however, as these plms only focus on fine-tuning under the ar paradigm, they can not be directly used for nar text generation. recently, bang @xcite and elmer @xcite rely on large-scale pre-training for improving the nar text generation. considering the pre-training cost, we aim to efficiently adapt bart into an effective nar model with diffusion models. diffusion models for text generation. diffusion models (dm) @xcite are a class of latent variable models that can progressively denoise a random gaussian noise into a data example. existing dms can be roughly categorized into continuous diffusion models @xcite @xcite and discrete diffusion models @xcite @xcite , which perform diffusion on continuous signals and discrete states, respectively. recently, dms have been utilized for text generation and have demonstrated superiority in controllable text generation tasks @xcite . for text-to-text generation tasks, existing works generally follow the continuous diffusion paradigm, and improve the performance by refining the model architecture @xcite , adding regularization @xcite and large-scale pre-training @xcite . in this work, we introduce discrete diffusion models into text-to-text generation tasks, and utilize a plm to improve it. ## preliminary problem statement. this work focuses on textto-text generation tasks using non-autoregressive (nar) models. generally, text-to-text generation tasks @xcite (e.g., dialog and summarization) can be formulated as modeling the conditional probability p (y |c), where @xmath0, c 2 , • • • , c m } and @xmath1, y 2 , • • • , y n } denote the input text and output text respectively, both consisting of a sequence of tokens from a vocabulary v. different from ar models with the left-to-right token-by-token generation manner, nar models @xcite predict all tokens of the output text y simultaneously, where each token y i is predicted only based on the input text c. thus, the conditional probability can be factorized as diffusion models. diffusion models (dm) @xcite ) sample an example from a data distribution p(x) by gradually denoising a random noise. typically, starting from a noise x t , the denoising process (also so-called reverse process) can be regarded as a markov process, where the noises at t -1, t -2, • • • , 0 steps are progressively predicted and removed to obtain the latent variables x t -1 , x t -2 , • • • , until reaching the final sample x 0 . conversely, given the sample x 0 , we can generate x 1 , x 2 , • • • , x t as a markov chain, denoted as the forward process: ## approach dialogue context: greetings, traveller from beyond the fog. ## conclusion in this paper, we proposed diffusion-nat, a selfprompting discrete diffusion model (ddm) using a plm bart for non-autoregressive (nar) text generation. in our approach, we unified the inference process of bart and the denoising process of ddm into the same masked tokens recovering task, to combine the merits of both the rich prelearned knowledge of bart and the iterative refining paradigm of ddm. concretely, we revised the decoding process of bart into the nar manner, and adapted the typical settings of ddm to better fit with bart, including markov transition matrix, training objective and time step embeddings. besides, we devised an iterative self-prompting strategy to guide the plm to deliberate and refine the intermediate generated results, to further improve the quality of final produced texts. extensive experiments on seven datasets have shown that our approach can outperform competitive nar and semi-nar models, and even surpass ar models.