Text Diffusion Model with Encoder-Decoder Transformers for Sequence-to-Sequence Generation (2024)

Abstract: The diffusion model, a new generative modeling paradigm, has achieved great success in image, audio, and video generation.However, considering the discrete categorical nature of the text, it is not trivial to extend continuous diffusion models to natural language. In this work, we propose SeqDiffuSeq, a text diffusion model, to approach sequence-to-sequence text generation with an encoder-decoder Transformer architecture.To improve the generation performance, SeqDiffuSeq is equipped with the self-conditioning technique and our newly proposed adaptive noise schedule technique. Self-conditioning enables SeqDiffuSeq to better use the predicted sequence information during the generation process.The adaptive noise schedule balances the difficulty of denoising across time steps at the token level.Experiment results illustrate the improved performance on five sequence-to-sequence generation tasks compared to other diffusion-based models regarding text quality and inference time.

## introduction generative modeling is drawing more attention in recent years of machine learning research due to the development of diffusion models @xcite . diffusion models define the forward process and the reverse process where the former gradually diffuses data to random noise while the latter recovers data from random noise iteratively, which have shown superior performance on synthesizing images @xcite , audios @xcite , and videos @xcite over other generative methods, such as generative adversarial network (gan) @xcite and normalizing flow @xcite . it is not trivial to extend diffusion models to the generation of natural languages. most of the existing diffusion models are applied to continuous feature space @xcite while texts are sequences of discrete categorical tokens. recently, research has explored categorical diffusion models in discrete space for text generation @xcite . there also exists research such as dif-fusionlm @xcite that applies continuous diffusion models to word embedding. however, these works only focus on unconditional and controlled text generation. sequence-to-sequence text generation is a fundamental natural language processing setting and covers various practical downstream tasks, such as dialogue @xcite and machine translation @xcite . in recent practice, researchers resort to auto-regressive (ar) @xcite or non-auto-regressive (nar) @xcite transformers for the tasks, and achieve good generation performance. using diffusion models, a recent work named diffuseq @xcite applies the diffusion-based method for sequenceto-sequence text generation. they deploy encoderonly transformers and partially define diffusion and denoising processes on output sequences. in this work, we explore diffusion models with encoder-decoder transformer architecture for sequence-to-sequence generation. we propose se-qdiffuseq which extends the continuous diffusion framework proposed in @xcite to sequence-to-sequence settings. we equip seqdiffuseq with the self-conditioning technique @xcite and our newly proposed adaptive noise schedule. self-conditioning helps the model better capture the information from former iterations during the generation. the proposed adaptive noise schedule learns a token-level noise schedule to better control the amount of noise injected and information recovered during the forward and reverse process @xcite . we conduct experiments on five generation tasks. results show that seqdiffuseq achieves compet-itive performance compared with ar and nar baselines in terms of generation quality and diversity. seqdiffuseq also shows improved generation performance and inference speed compared to text diffison model diffuseq. ablation studies demonstrate that our model can benefit from self-conditioning and adaptive noise schedule techniques, and both are complementary to each other in sequence-to-sequence settings. to summarize, the main contributions of this work are as follows: ## related work since the great success of diffusion models in vision @xcite @xcite , researchers have explored extending diffusion models to text generation. considering the discrete and categorical nature of texts, multinomial diffusion @xcite and d3pm @xcite are proposed for modeling categorical data. they define discrete diffusion models using discrete categorical transitions directly on texts. diffusionbert @xcite follows d3pm and introduces pre-trained models for language modeling. besides, recent research also explores converting texts into continuous features to adapt to diffusion models. bit diffusion @xcite encodes discrete data as binary bits and treats these binary bits as real number features. @xcite is proposed to build text diffusion models in continuous latent space. @xcite uses the word embedding space for continuous diffusion models and introduces auxiliary losses to enable joint learning of embedding and network parameters. following diffusionlm, recent research explores improving text generation quality @xcite , and diffuseq @xcite extends it to sequence-to-sequence settings. compared to dif-fuseq, we propose a different model architecture and self-conditioning and adaptive noise schedule techniques to improve sequence-to-sequence generation performance. noise schedules in diffusion models control the level of noise injected and the level of information recovered in the forward and reverse process respectively. previous research in vision and texts demonstrates that appropriate noise schedule design can improve the generation quality performance of diffusion models @xcite . concurrently, diffusion-bert @xcite proposes a spindle schedule for language modeling, and cdcd @xcite designs a learned noise schedule for language modeling and machine translation. different from both concurrent works, seqdiffuseq is proposed with a token-level noise schedule that balances the difficulty of denoising across time steps. @xcite proposes difformer and is orthogonal to our work. ## preliminary diffusion model is generally formulated by a designed forward diffusion process and a learnt reverse denoising process. in the forward diffusion process, samples gradually mix with random noise, while in the reverse denoising process, the random noise is gradually denoised to generate synthetic samples. we adopt the forward and reverse processes proposed in ddpm @xcite . for the forward process, given a sample z 0 from a real-world data distribution q(z 0 ). at each time step @xmath0, 2, for the reverse process, the diffusion model uses a learnt parameterized denoising distribution z t-1 ∼ p θ (z t-1 |z t ) to gradually recover samples from noise. the denoising distribution is parameterized by θ and is to fit the posterior distribution q(z t-1 |z t , z 0 ) of the forward process. with learnt denoising distribution p θ , a synthetic real-world sample z 0 can be generated from pure random noise z t step-by-step. ## approach in this section, we present the main design of our proposed seqdiffuseq for sequence-to-sequence language generation. the overview of seqdiffuseq is depicted in figure 1 . in the following sections, the input and output sequences are denoted as w x and w y respectively. for the i-th token in w y , the token is denoted as w i y , where 0 < @xmath1. in order to avoid lengthy notations, we omit the indices referring to different data samples. ## conclusion in this work, we explore to approach sequence-tosequence text generation with continuous diffusion models. we propose seqdiffuseq which uses an encoder-decoder transformers architecture to learn the denoising function. in order to improve text generation performance, the denoising function in seqdiffuseq is integrated with self-conditioning technique. seqdiffuseq also includes a newly proposed adaptive noise schedule which makes the denoising difficulty evenly distributed across all time steps and assigns exclusive noise schedules for tokens from different positional orders. through experiments, we illustrate the superior performance of seqdiffuseq in terms of generation quality and inference speed and provide insights into our proposed adaptive noise schedule technique.