Enable Fast Sampling for S eq2 S eq Text Diffusion (2024)

Abstract: Diffusion models exhibit promising capacity for generating high-quality text. However, owing to the curved nature of generation path, they necessitate traversing numerous steps to guarantee the text quality. In this paper, we propose an efficient model FMSeq, which utilizes flow matching to straighten the generation path, thereby enabling fast sampling for diffusion-based seq2seq text generation. Specifically, we construct transport flow only on the target sequences to adapt the diffusion-based model with flow matching. Furthermore, we explore different settings and identify target-parameterization, self-conditioning and time-difference as three effective techniques to improve the generation quality under a few steps. Experiments on four popular tasks demonstrate that FMSeq generates texts of comparable quality to the SOTA diffusion-based DiffuSeq in just 10 steps, achieving a 200-fold speedup.

## introduction sequence-to-sequence (seq2seq) text generation @xcite @xcite is a fundamental setting in nlp, encompassing a wide range of practical downstream tasks including machine translation @xcite ), text summarization @xcite , and conversational modeling @xcite . existing text generation methods predominantly utilize the autoregressive (ar) approach @xcite , which generates target tokens one by one. while this method captures sequential dependencies and hence produces high-quality texts, it is time-consuming. therefore, non-autoregressive (nar) generation methods have been proposed. these methods generate all tokens in parallel, significantly reducing inference latency @xcite . however, nar models generally underperform ar ones on text generation accuracy due to the issue of conditional total correlation @xcite . more recently, a new nar generative paradigm, diffusion models @xcite has exhibited remarkable generative performance across various domains @xcite @xcite and are increasingly gaining recognition in the field of nlp @xcite . diffuseq @xcite represents a pioneering effort in leveraging diffusion models for seq2seq text generation, demonstrating that diffusion model-based methods can achieve generation performance on par with ar models owing to their robust modeling capabilities. however, as a class of nar methods, diffusion models' sampling speed is not fast. to achieve a higher bleu score than gpt2-large @xcite , diffuseq requires 1000 sampling steps, which makes the number of tokens generated per second only half of gpt2-large. a recently proposed method, flow matching @xcite tries to speed up the sampling via straightening the generation path and has greatly improved the generation efficiency of diffusion-based models across multiple fields, including image generation @xcite @xcite , point cloud generation @xcite , audio synthesis @xcite and beyond @xcite . however, in the field of text generation, when the standard diffusion process of diffuseq is directly replaced with flow matching, the quality of the generated texts significantly declines, rendering the flow matching-based model unusable, which is evidenced in @xcite . in this paper, we find that the root of the above issue is the objective of recovering source sequences, which is not suitable for flow matching (see section 3). therefore, we eliminate this objective and construct transport flow only on the target sequences. then we design a combined loss to fit the transport flow and simultaneously keep the trainable embedding from collapsing. to further improve the generation quality of fmseq with a few steps, we explore different settings and identify targetparameterization, self-conditioning @xcite , and time-difference as three effective techniques. experiments on four popular tasks demonstrate that fmseq enables faster sampling compared to other models while generating texts of comparable quality. as shown in fig. 1 2 related work ## method we propose fmseq to utilize flow matching into seq2seq text generation, the workflow of which is shown in fig. 3 . ## conclusion in this paper, we propose fmseq, a novel seq2seq text generation model based on flow matching. we find that the objective of recovering the source sequences adopted in the diffusion-based model is unsuitable to flow matching and construct transport flow only on the target sequences. to fit the transport flow and simultaneously keep the trainable embedding from collapsing, we design a combined loss. to further improve the generation quality of fmseq under a few steps, we explore different settings and identify target-parameterization, selfconditioning, and time-difference as three effective techniques. experiments in four popular nlp tasks demonstrate that fmseq generates texts of comparable quality to diffuseq in just 10 steps, achieving a 200-fold speedup.