F ast D iff 2: Revisiting and Incorporating GAN s and Diffusion Models in High-Fidelity Speech Synthesis (2023)

Abstract: Generative adversarial networks (GANs) and denoising diffusion probabilistic models (DDPMs) have recently achieved impressive performances in image and audio synthesis. After revisiting their success in conditional speech synthesis, we find that 1) GANs sacrifice sample diversity for quality and speed, 2) diffusion models exhibit outperformed sample quality and diversity at a high computational cost, where achieving high-quality, fast, and diverse speech synthesis challenges all neural synthesizers. In this work, we propose to converge advantages from GANs and diffusion models by incorporating both classes, introducing dual-empowered modeling perspectives: 1) FastDiff 2 (DiffGAN), a diffusion model whose denoising process is parametrized by conditional GANs, and the non-Gaussian denoising distribution makes it much more stable to implement the reverse process with large steps sizes; and 2) FastDiff 2 (GANDiff), a generative adversarial network whose forward process is constructed by multiple denoising diffusion iterations, which exhibits better sample diversity than traditional GANs. Experimental results show that both variants enjoy an efficient 4-step sampling process and demonstrate superior sample quality and diversity. Audio samples are available at https://RevisitSpeech.github.io/

## introduction speech synthesis has seen extraordinary progress with the recent development of deep generative models in machine learning @xcite @xcite @xcite . previous models @xcite generate waveforms autoregressively from mel-spectrograms yet suffer from slow inference speed. non-autoregressive methods @xcite @xcite have been designed to address this issue, they generate samples with extremely fast speed and achieve comparable voice quality with autoregressive models. among them, generative adversarial networks (gans) @xcite @xcite and denoising diffusion probabilistic models (ddpms) @xcite are two popular classes of deep generative models that have demonstrated surprisingly good results and dominated speech synthesis @xcite utilize local-variable convolution to capture different waveform intervals with adversarial learning. @xcite propose multireceptive field fusion (mrf) to model the periodic patterns matters. @xcite introduce a time-aware wavenet for conditional diffusion modeling. @xcite and @xcite utilize a noise predictor to learn a tight inference schedule for skipping denoising steps. despite their success in the high-fidelity generation, few studies have compared these two classes of deep generative models in conditional speech synthesis. in this work, we conduct a comprehensive study to revisit gans and diffusion models, and empirically demonstrate that: 1) gans tend to generate high-quality speeches but do not cover the whole distribution, which sacrifice sample diversity for quality and speed; and 2) diffusion models exhibit outperformed sample quality and diversity, buy they typically require a large number of iterative refinements. to this end, simultaneously achieving high-quality and diverse speech synthesis at a low computational cost has become an open problem for all neural synthesizers. in this work, we converge advantages from both classes by incorporating gans and diffusion models, introducing dual-empowered modeling perspectives for high-fidelity speech synthesis: 1) fast-diff 2 (diffgan): a diffusion model whose denoising process is parametrized by conditional gans, and the non-gaussian denoising distribution makes it much more stable to implement the reverse process with large step sizes; and 2) fastdiff 2 (gan-diff): a generative adversarial network whose forward process is constructed by multiple denoising diffusion iterations, which exhibits better sample diversity than traditional gans. experimental results show that both variants enjoy an effective 4-iter sampling process and demonstrate the outperformed sample quality and diversity. moreover, we show that both variants generalize well to the mel-spectrogram inversion of unseen speakers. the main contributions of this work are summarized as follows: ## background on speech synthesis with the development of deep generative models @xcite @xcite , speech synthesis technology has made rapid progress up to date. most models @xcite @xcite @xcite first convert input text or phoneme sequence into mel-spectrogram, and then transform it to waveform using a separately trained vocoder @xcite @xcite . in this work, we focus on designing the second-stage model that efficiently synthesizes high-fidelity waveforms from mel-spectrograms. neural vocoders require diverse receptive field patterns to catch audio dependencies, and thus previous models @xcite generate waveforms autoregressively from mel-spectrograms yet suffer from slow inference speed. in recent years, non-autoregressive methods @xcite @xcite have been designed to address this issue, which generates samples with extremely fast speed while achieving comparable voice quality with autoregressive models. below we mainly introduce two popular classes of deep generative models (diffusion models and gans) for conditional speech synthesis: ## preliminary study in image generation, superior sample diversity @xcite @xcite ) is a crucial reason for the diffusion model to produce high-quality samples even on the challenging dataset. due to the distinctive advantages of diversity and distribution coverage over gans, diffusion models have been demonstrated to generate realistic and vivid images, achieving the current state-of-the-art measured by fid. despite the comprehensive studies of gans and diffusion models for image generation, few have compared these two classes of deep generative models in speech synthesis, where an audio signal is different @xcite for its long-term dependencies, high sampling rate, and strong condition. in this section, we provide an empirical study and investigate the characteristic of both classes with close model capacity in speech. specifically, we evaluate the performance (including sample quality, speed, and diversity) and explore how distribution coverage impacts sample quality by auditory sensation. ## methods after revisiting gan and diffusion models for speech synthesis, we witness that 1) gans sacrifice sample diversity for better quality and speed, producing high-quality samples but not covering the whole distribution. 2) diffusion models exhibit outperformed sample quality and diversity, requiring iterative refinement at a high computational cost. in this section, we aim to converge advantages from both classes, introducing dualempowered modeling perspectives for high-fidelity, fast, and diverse speech synthesis. ## conclusion in this work, through revisiting two popular classes (diffusion models and gans) of deep generative models, we observed that 1) gans tended to generate samples but did not cover the whole distribution, and 2) diffusion models exhibited outperformed sample quality and diversity while requiring iterative refinement at a high computational cost. to achieve high-quality, fast and diverse speech synthesis, we converged advantages by incorporating gans and diffusion models, introducing dualempowered modeling perspectives: 1) fastdiff 2 (diffgan), a diffusion model whose denoising process was parametrized by conditional gans, and the non-gaussian denoising distribution made it much more stable to implement the reverse process with large step sizes; and 2) fastdiff 2 (gandiff): a generative adversarial network whose forward process was constructed by multiple denoising diffusion iterations, and it exhibited better mode coverage and sample diversity. experimental results showed that both variants enjoyed an efficient 4step sampling and demonstrated superior sample quality and diversity. we envisage that our work serve as a basis for future speech synthesis studies.