W ave FM : A High-Fidelity and Efficient Vocoder Based on Flow Matching (2025)

Abstract: Flow matching offers a robust and stable approach to training diffusion models. However, directly applying flow matching to neural vocoders can result in subpar audio quality. In this work, we present WaveFM, a reparameterized flow matching model for mel-spectrogram conditioned speech synthesis, designed to enhance both sample quality and generation speed for diffusion vocoders. Since mel-spectrograms represent the energy distribution of waveforms, WaveFM adopts a mel-conditioned prior distribution instead of a standard Gaussian prior to minimize unnecessary transportation costs during synthesis. Moreover, while most diffusion vocoders rely on a single loss function, we argue that incorporating auxiliary losses, including a refined multi-resolution STFT loss, can further improve audio quality. To speed up inference without degrading sample quality significantly, we introduce a tailored consistency distillation method for WaveFM. Experiment results demonstrate that our model achieves superior performance in both quality and efficiency compared to previous diffusion vocoders, while enabling waveform generation in a single inference step.

## introduction recent advancements in network architectures and training algorithms have greatly enhanced the ability of deep generative models to produce highfidelity audio in speech synthesis @xcite @xcite @xcite @xcite . the initial breakthrough came with the autoregressive generation of waveforms from mel-spectrograms @xcite , which provided high audio fidelity but suffered from slow inference speeds. to enable real-time high-fidelity speech synthesis, a variety of non-autoregressive models have been introduced, classified broadly into three categories: flow-based models, generative adversarial networks (gans), and diffusion models. flow-based models utilize invertible neural networks to generate waveforms from a selected prior distribution, such as a gaussian distribution, estimating log-likelihoods during training @xcite . while these intricately designed models maintain invertibility and evaluate determinants, this complexity limits their flexibility, and consequently the quality of the audio output. generative adversarial networks (gans) provide greater flexibility than flow-based models and can generate waveforms with high fidelity more efficiently @xcite @xcite @xcite . their success stems from the generators' large receptive fields and the discriminators' ability to detect noise across various scales and periods. for instance, @xcite introduced multi-scale discriminators, while @xcite developed a multi-receptive field (mrf) generator alongside multiple multi-period discriminators, leading to substantial improvements. moreover, @xcite further enhanced sample quality by utilizing the snake activation function and integrating the anti-aliased multi-periodicity (amp) composition module. denoising diffusion probabilistic models (ddpms) have recently gained significant popularity for their ability to transform a simple prior distribution into a complex ground truth distribution through a markov chain process @xcite @xcite . these models rely on a parameter-free noise-adding diffusion process to generate training data for the denoising generator, eliminating the need for auxiliary networks like discriminators or autodecoders during training. however, the inference phase of diffusion models tends to be time-consuming. to mitigate this issue, @xcite , @xcite , @xcite introduced several fast-sampling algorithms that speed up waveform generation, though with a minor compromise in sample quality. consistency models @xcite have been proposed to enhance the efficiency of diffusion models by directly predicting the endpoint of the probability flow ordinary differential equation (pf-ode) at each step, enabling single-step inference. these models surpass previous distillation approaches in image synthesis tasks and achieve higher distillation efficiency by aligning along ode trajectories, thereby avoiding the need to numerically solve the entire ode. in this study, we propose wavefm for the melspectrogram conditioned speech synthesis task. firstly, since the mel-spectrogram records the energy information in waveforms, an appropriately conditioned distribution can significantly improve sample quality. additionally, we adopt a reparameterized flow matching method that directly predicts the waveform, allowing us to apply several auxiliary losses to the original flow-matching loss to further enhance the model's performance. to better supervise the model's output waveforms regarding phase angles, we also incorporate a multi-resolution phase loss into our model. furthermore, gradient and laplacian operators are utilized on the real and generated spectrograms. minimizing the mean square losses associated with these operations enables the model to better learn edge details and structural patterns in the spectrograms. finally, we propose a tailored consistency distillation method for wavefm to further accelerate the model's inference speed while maintaining audio quality. the subjective and objective experiment results indicate that wavefm outperforms previous diffusion models in terms of sample quality and efficiency, and generalizes better on out-of-distribution musical mel-spectrograms. ## conclusion we propose wavefm, a high-fidelity vocoder for speech synthesis conditioned on mel-spectrograms. first, it utilizes the energy information from melspectrograms to generate a prior distribution with low variance. additionally, reparameterizing the original flow matching objective not only introduces periodic inductive bias into the neural network, but also enables the inclusion of auxiliary losses. specifically, we design a multi-resolution stft loss function to enhance sample quality for our model. finally, our consistency distillation algorithm allows wavefm to produce audio in one step without significantly sacrificing sample quality. together, these techniques greatly improve both the quality and efficiency of wavefm, with smos tests and automatic evaluators confirming that it performs competitively against previous diffusion models.