Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model (2024)

Abstract: Recently, Denoising Diffusion Probabilistic Models (DDPMs) have attained leading performances across a diverse range of generative tasks. However, in the field of speech synthesis, although DDPMs exhibit impressive performance, their prolonged training duration and substantial inference costs hinder practical deployment. Existing approaches primarily focus on enhancing inference speed, while approaches to accelerate training—a key factor in the costs associated with adding or customizing voices—often necessitate complex modifications to the model, compromising their universal applicability. To address the aforementioned challenges, we propose an inquiry: is it possible to enhance the training/inference speed and performance of DDPMs by modifying the speech signal itself? In this paper, we double the training and inference speed of Speech DDPMs by simply redirecting the generative target to the wavelet domain. This method not only achieves comparable or superior performance to the original model in speech synthesis tasks but also demonstrates its versatility. By investigating and utilizing different wavelet bases, our approach proves effective not just in speech synthesis, but also in speech enhancement.

## introduction recently, with the advancement of deep learning, generative models have made significant progress in various fields @xcite @xcite . particularly, the emergence of diffusion models has elevated the capabilities of deep generative models to a new level @xcite . in the field of speech processing, denoising diffusion probabilistic models (ddpms) not only exhibit astonishing performance in speech synthesis @xcite ; @xcite but also demonstrate commendable results in speech enhancement @xcite . however, despite the impressive results achieved by ddpms in the field of speech processing, the requirement to generate a guarantee of high sample quality -typically necessitating hundreds to thousands of denoising steps -results in training and inference speeds that are daunting in practical applications. given these issues, researchers from various fields have attempted different methods to improve diffusion models. in the realm of speech processing, existing approaches have endeavored to alter the model structure to accelerate the inference speed of speech synthesis @xcite , while others have experimented with changing training strategies to reduce the number of inference steps required for diffusion models in speech enhancement @xcite . these approaches primarily focus on enhancing the inference speed of speech diffusion models. however, in the field of speech synthesis, the industry frequently requires incorporating new voices to accommodate varied requirements. additionally, generative-based speech enhancement often demands tailoring models to distinct scenarios, which introduces practical limitations to the aforementioned methods in real-world applications. in the field of computer vision, researchers have attempted to accel- dwt approximation details diffusion model target matrix target matrix approximation details iwt target speech [batchsize,1,2x] [batchsize,1,x] [batchsize,2,x] [batchsize,2,x] [batchsize,1,x] [batchsize,2x] ## methodology in this section, the proposed method is illustrated using the cohen-daubechies-feauveau 5/3 wavelet as a case study @xcite . we first explain how we utilize wavelet transforms for compressing and parallel processing of speech sig-algorithm 1 wavelet diffwave training then, we delve into the specifics of accelerating speech synthesis and enhancement tasks. ## further study under the significant acceleration achieved by our method, we explore the potential for enhancing the quality of samples through wavelet transformation and further accelerating the training and sampling process of the diffusion model. ## conclusion in this paper, we have enhanced the speech diffusion model by transitioning its generation target to the wavelet domain, thereby doubling the model's training and inference speeds. we offer a new perspective on accelerating speech models by focusing on processing the signal itself rather than modifying the model. our approach has demonstrated model versatility and task adaptability across both speech enhancement and synthesis. through our research, we found that the coif1 wavelet is an excellent choice for scenarios requiring noise reduction without the need to preserve timbre, while the db2 wavelet is preferable when changes in timbre must be considered. for speech synthesis tasks, the haar wavelet offers simplicity and effectiveness, whereas the cdf53 wavelet excels at preserving information to the greatest extent. additionally, we designed two simple and easily integrable frontend modules. the first achieves better performance than the original model while doubling the speed. the second offers a performance comparable to the original while enabling an acceleration of more than five times.