P ro SE : Diffusion Priors for Speech Enhancement (2025)

Abstract: Speech enhancement (SE) is the fundamental task of enhancing the clarity and quality of speech in the presence of non-stationary additive noise. While deterministic deep learning models have been commonly employed for SE, recent research indicates that generative models, such as denoising diffusion probabilistic models (DDPMs), have shown promise. However, different from speech generation, SE has a strong constraint to generate results in accordance with the underlying ground-truth signal. Additionally, for a wide variety of applications, SE systems need to be employed in real-time, and traditional diffusion models (DMs) requiring many iterations of a large model during inference are inefficient. To address these issues, we propose ProSE (diffusion-based Priors for SE), a novel methodology based on an alternative framework for applying diffusion models to SE. Specifically, we first apply DDPMs to generate priors in a latent space due to their powerful distribution mapping capabilities. The priors are then integrated into a transformer-based regression model for SE. The priors guide the regression model in the enhancement process. Since the diffusion process is applied to a compact latent space, the diffusion model takes fewer iterations than the traditional DM to obtain accurate estimations. Additionally, using a regression model for SE avoids the distortion issue caused by misaligned details generated by DMs. Comprehensive experiments show that ProSE achieves state-of-the-art performance on synthetic and real-world datasets using various metrics while consuming less computational costs.

## introduction speech enhancement (se) is a task that focuses on enhancing speech intelligibility and quality, especially when speech is degraded by non-stationary additive noise. se has important applications in various areas, including telecommunications (gay and benesty, 2012), medicine @xcite , and everyday communication @xcite . traditionally, deep learning models have been employed to establish a deterministic mapping from noisy to clean speech. although deterministic models have been considered superior in se, recent developments in generative models have shown promise and narrowed the performance gap @xcite @xcite . denoising diffusion probabilistic models (ddpms) have emerged as a powerful generative approach for realistic speech synthesis @xcite @xcite . specifically, ddpms are designed to denoise data iteratively by inverting the diffusion process. these models demonstrate that structured probabilistic diffusion approaches can effectively transform randomly sampled gaussian noise into complex target distributions, such as realistic speech or associated latent distributions, while avoiding the mode collapse and training instabilities commonly seen in gans @xcite . however, there are two major problems: (1) as a class of likelihood-based models, ddpms require many steps in large denoising models to generate precise details of the data distributions, which requires massive computational resources. unlike the speech synthesis tasks that generate each detail from scratch, se tasks are only required to remove additive noise. therefore, adopting the same extensive iterative process used for speech synthesis in se leads to inefficiencies and excessive computational costs. ## experimental setup datasets. following a wealth of prior work in se @xcite @xcite , we use the voicebank-demand (vbd) dataset to train and evaluate our models. the dataset comprises noisy speech recordings generated by mixing clean speech from the voicebank collection @xcite with noise from the de-mand dataset @xcite . it has 30 distinct speakers, with 28 speakers allocated for training and 2 for testing. clean samples were mixed with noise samples at four signal-to-noise ratios (snrs) during training ([0, 5, 10, 15] db) and testing ([2.5, 7.5, 12.5, 17.5] db). the training dataset comprised a total of 11,572 utterances, while the testing set included 824 utterances. additionally, to evaluate the generalization capabilities of our models trained on vbd, we also test them on chime-4 @xcite . chime-4 is baselines. we compare prose with recent open-sourced diffusion-based se methods dif-fuse @xcite , cdiffuse @xcite , sgmse @xcite , dr-diffuse @xcite and dose @xcite . details about baselines can be found in appendix f. ## discussion t for training and inference). we attribute this characteristic to the way we integrate dms into the prose architecture: precisely, we only apply dms in a compact latent space, and thus, the model does not need too many iterations to add details to the output, unlike speech synthesis. additionally, the latent generated, which is used as a prior for conditioning, only adds details for enhancement. thus, only a few iterations can lead to a stable prior. this also makes prose much more efficient than prior work, which employs dms to generate the entire enhanced speech. ## conclusion in this paper, we propose prose, a novel method for speech enhancement with diffusion models. specifically, prose makes a latent diffusion model generate a prior. this prior is then conditioned on a u-net style regression-based transformer network where the prior guides the network for speech enhancement. the regression-based method preserves the general distribution, while the prior feature generated by the diffusion model enhances the details of the enhanced speech. prose achieves sota performance on standard benchmarks in matched and unmatched scenarios.