Original: U NIVERSAL
Revised: U NIVERSAL

Original: S PEECH
Revised: S PEECH

Original: E NHANCEMENT
Revised: E NHANCEMENT

Original: WITH
Revised: WITH

Original: S COREBASED D IFFUSION
Revised: S COREBASED D IFFUSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings.
Revised: Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings.

Original: Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous.
Revised: Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous.

Original: In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time.
Revised: In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time.

Original: Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks.
Revised: Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks.

Original: We show that this approach signiﬁcantly outperforms the state of the art in a subjective test performed by expert listeners.
Revised: We show that this approach signiﬁcantly outperforms the state of the art in a subjective test performed by expert listeners.

Original: We also show that it achieves competitive objective scores with just 4–8 diffusion steps, despite not considering any particular strategy for fast sampling.
Revised: We also show that it achieves competitive objective scores with just 4–8 diffusion steps, despite not considering any particular strategy for fast sampling.

Original: We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.
Revised: We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Real-world recorded speech almost inevitably contains background noise, which can be unpleasant and prevent intelligibility.
Revised: Real-world recorded speech almost inevitably contains background noise, which can be unpleasant and prevent intelligibility.

Original: Removing background noise has traditionally been the objective of speech enhancement algorithms (Loizou, 2013).
Revised: Removing background noise has traditionally been the objective of speech enhancement algorithms (Loizou, 2013).

Original: Since the 1940s (Kolmogorov, 1941; Wiener, 1949), a myriad of denoising approaches based on ﬁltering have been proposed, with a focus on stationary noises.
Revised: Since the 1940s (Kolmogorov, 1941; Wiener, 1949), a myriad of denoising approaches based on ﬁltering have been proposed, with a focus on stationary noises.

Original: With the advent of deep learning, the task has been dominated by neural networks, often outperforming more classical algorithms and generalizing to multiple noise types (Lu et al., 2013;
Revised: With the advent of deep learning, the task has been dominated by neural networks, often outperforming more classical algorithms and generalizing to multiple noise types (Lu et al., 2013;

Original: Pascual et al., 2017; Rethage et al., 2018; Défossez et al., 2020; Fu et al., 2021).
Revised: Pascual et al., 2017; Rethage et al., 2018; Défossez et al., 2020; Fu et al., 2021).

Original: Besides recent progress, speech denoising still presents room for improvement, especially when dealing with distribution shift or real-world recordings.
Revised: Besides recent progress, speech denoising still presents room for improvement, especially when dealing with distribution shift or real-world recordings.

Original: Noise however is only one of the many potential disturbances that can be present in speech recordings.
Revised: Noise however is only one of the many potential disturbances that can be present in speech recordings.

Original: If recordings are performed in a closed room, reverberation is ubiquitous.
Revised: If recordings are performed in a closed room, reverberation is ubiquitous.

Original: With this in mind, a number of works have recently started to zoom out the focus in order to embrace more realistic situations and tackle noise and reverberation at the same time (Su et al., 2021; 2019; Polyak et al., 2021).
Revised: With this in mind, a number of works have recently started to zoom out the focus in order to embrace more realistic situations and tackle noise and reverberation at the same time (Su et al., 2021; 2019; Polyak et al., 2021).

Original: Some of these works adopt a generation or re-generation strategy (Maiti & Mandel, 2019), in which a two-stage approach is employed to ﬁrst enhance and then synthesize speech signals.
Revised: Some of these works adopt a generation or re-generation strategy (Maiti & Mandel, 2019), in which a two-stage approach is employed to ﬁrst enhance and then synthesize speech signals.

Original: Despite the relative success of this strategy, it is still an open question whether such approaches can perceptually outperform the purely supervised ones, especially in terms of realism and lack of voice artifacts.
Revised: Despite the relative success of this strategy, it is still an open question whether such approaches can perceptually outperform the purely supervised ones, especially in terms of realism and lack of voice artifacts.

Original: Besides noise and reverberation, a few works propose to go one step further by considering additional distortions.
Revised: Besides noise and reverberation, a few works propose to go one step further by considering additional distortions.

Original: Pascual et al.
Revised: Pascual et al.

Original: (2019) introduce a broader notion of speech enhancement by considering whispered speech, bandwidth reduction, silent gaps, and clipping.
Revised: (2019) introduce a broader notion of speech enhancement by considering whispered speech, bandwidth reduction, silent gaps, and clipping.

Original: More recently, Nair & Koishida (2021) consider silent gaps, clipping, and codec artifacts, and Zhang et al.
Revised: More recently, Nair & Koishida (2021) consider silent gaps, clipping, and codec artifacts, and Zhang et al.

Original: (2021a) consider clipping and codec artifacts.
Revised: (2021a) consider clipping and codec artifacts.

Original: In concurrent work, Liu et al.
Revised: In concurrent work, Liu et al.

Original: (2021) deal with bandwidth reduction and clipping in addition to noise and reverberation.
Revised: (2021) deal with bandwidth reduction and clipping in addition to noise and reverberation.

Original: Despite the recent efforts to go beyond pure denoising, we are not aware of any speech enhancement system that tackles more than 2–4 distortions at the same time.
Revised: Despite the recent efforts to go beyond pure denoising, we are not aware of any speech enhancement system that tackles more than 2–4 distortions at the same time.

Original: In this work, we take a holistic approach and regard the task of speech enhancement as a universal endeavor.
Revised: In this work, we take a holistic approach and regard the task of speech enhancement as a universal endeavor.

Original: We believe that, for realistic speech enhancement situations, algorithms need not only face and improve upon background noise and possibly reverberation, but also to correct a large number of typical but usually neglected distortions that are present in everyday recordings or amateur-produced audio, such as bandwidth reduction, clipping, codec artifacts, silent gaps, excessive dynamics compression/expansion, sub-optimal equalization, noise gating, and others (in total, we deal with 55 distortions, which can be grouped into 10 different families).
Revised: We believe that, for realistic speech enhancement situations, algorithms need not only face and improve upon background noise and possibly reverberation, but also to correct a large number of typical but usually neglected distortions that are present in everyday recordings or amateur-produced audio, such as bandwidth reduction, clipping, codec artifacts, silent gaps, excessive dynamics compression/expansion, sub-optimal equalization, noise gating, and others (in total, we deal with 55 distortions, which can be grouped into 10 different families).

Original: Our solution relies on an end-to-end approach, in which a generator network synthesizes clean speech and a conditioner network informs of what to generate.
Revised: Our solution relies on an end-to-end approach, in which a generator network synthesizes clean speech and a conditioner network informs of what to generate.

Original: The idea is that the generator learns from clean speech and both generator and conditioner have the capability of enhancing representations, with the latter undertaking the core part of this task.
Revised: The idea is that the generator learns from clean speech and both generator and conditioner have the capability of enhancing representations, with the latter undertaking the core part of this task.

Original: For the generator, we put together a number of known and less known advances in score-based diffusion models (Sohl-Dickstein et al., 2015;
Revised: For the generator, we put together a number of known and less known advances in score-based diffusion models (Sohl-Dickstein et al., 2015;

Original: Song & Ermon, 2019; Ho et al., 2020).
Revised: Song & Ermon, 2019; Ho et al., 2020).

Original: For the conditioner, we develop a number of improved architectural choices, and further propose the usage of auxiliary, out-of-path mixture density networks for enhancement in both the feature and the waveform domains.
Revised: For the conditioner, we develop a number of improved architectural choices, and further propose the usage of auxiliary, out-of-path mixture density networks for enhancement in both the feature and the waveform domains.

Original: We quantify the relative importance of these main development steps using objective metrics, and show how the ﬁnal solution outperforms the state of the art in all considered distortions using a subjective test with expert listeners (objective metrics for the denoising task are also reported in the Appendix).
Revised: We quantify the relative importance of these main development steps using objective metrics, and show how the ﬁnal solution outperforms the state of the art in all considered distortions using a subjective test with expert listeners (objective metrics for the denoising task are also reported in the Appendix).

Original: Finally, we also study the number of diffusion steps needed for performing high-quality universal speech enhancement, and ﬁnd it to be on par with the fastest diffusion-based neural vocoders without the need for any speciﬁc tuning.
Revised: Finally, we also study the number of diffusion steps needed for performing high-quality universal speech enhancement, and ﬁnd it to be on par with the fastest diffusion-based neural vocoders without the need for any speciﬁc tuning.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our approach is based on diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020).
Revised: Our approach is based on diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020).

Original: While diffusion models have been more extensively studied for unconditional or weaklyconditioned image generation, our work presents a number of techniques for strongly-conditioned speech re-generation or enhancement.
Revised: While diffusion models have been more extensively studied for unconditional or weaklyconditioned image generation, our work presents a number of techniques for strongly-conditioned speech re-generation or enhancement.

Original: Diffusion-based models achieve state-of-the-art quality on multiple generative tasks, in different domains.
Revised: Diffusion-based models achieve state-of-the-art quality on multiple generative tasks, in different domains.

Original: In the audio domain, they have been particularly successful in speech synthesis (Chen et al., 2021; Kong et al., 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), bandwidth extension (Lee & Han, 2021), or drum sound synthesis (Rouard & Hadjeres, 2021).
Revised: In the audio domain, they have been particularly successful in speech synthesis (Chen et al., 2021; Kong et al., 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), bandwidth extension (Lee & Han, 2021), or drum sound synthesis (Rouard & Hadjeres, 2021).

Original: An introduction to diffusion models is given in Appendix A
Revised: An introduction to diffusion models is given in Appendix A

Original: Diffusion-based models have also recently been used for speech denoising.
Revised: Diffusion-based models have also recently been used for speech denoising.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021a) expand the DiffWave vocoder (Kong et al., 2021) with a convolutional conditioner, and train that separately with an L1 loss for matching latent representations.
Revised: (2021a) expand the DiffWave vocoder (Kong et al., 2021) with a convolutional conditioner, and train that separately with an L1 loss for matching latent representations.

Original: Lu et al.
Revised: Lu et al.

Original: (2021) study the potential of DiffWave with noisy mel band inputs for speech denoising and, later, Lu et al.
Revised: (2021) study the potential of DiffWave with noisy mel band inputs for speech denoising and, later, Lu et al.

Original: and
Revised: and

Original: Welker et al.
Revised: Welker et al.

Original: (2022) propose formulations of the diffusion process that can adapt to (non-Gaussian) real audio noises.
Revised: (2022) propose formulations of the diffusion process that can adapt to (non-Gaussian) real audio noises.

Original: These studies with speech denoising show improvement over the considered baselines, but do not reach the objective scores achieved by state-of-the-art approaches (see also
Revised: These studies with speech denoising show improvement over the considered baselines, but do not reach the objective scores achieved by state-of-the-art approaches (see also

Original: Appendix E).
Revised: Appendix E).

Original: Our work stems from the WaveGrad architecture (Chen et al., 2021), introduces a number of crucial modiﬁcations and additional concepts, and pioneers universal enhancement by tackling an unprecedented amount of distortions.
Revised: Our work stems from the WaveGrad architecture (Chen et al., 2021), introduces a number of crucial modiﬁcations and additional concepts, and pioneers universal enhancement by tackling an unprecedented amount of distortions.

Original: The state of the art for speech denoising and dereverberation is dominated by regression and adversarial approaches (Défossez et al., 2020; Fu et al., 2021;
Revised: The state of the art for speech denoising and dereverberation is dominated by regression and adversarial approaches (Défossez et al., 2020; Fu et al., 2021;

Original: Su et al., 2021; Isik et al., 2020; Hao et al., 2021; Kim & Seo, 2021; Zheng et al., 2021; Kataria et al., 2021).
Revised: Su et al., 2021; Isik et al., 2020; Hao et al., 2021; Kim & Seo, 2021; Zheng et al., 2021; Kataria et al., 2021).

Original: However, if one considers further degradations of the signal like clipping, bandwidth removal, or silent gaps, it is intuitive to think that generative approaches have great potential (Polyak et al., 2021;
Revised: However, if one considers further degradations of the signal like clipping, bandwidth removal, or silent gaps, it is intuitive to think that generative approaches have great potential (Polyak et al., 2021;

Original: Pascual et al., 2019; Zhang et al., 2021b), as such degradations require generating signal where, simply, there is none.
Revised: Pascual et al., 2019; Zhang et al., 2021b), as such degradations require generating signal where, simply, there is none.

Original: Yet, to the best of our knowledge, this intuition has not been convincingly demonstrated through subjective tests involving human judgment.
Revised: Yet, to the best of our knowledge, this intuition has not been convincingly demonstrated through subjective tests involving human judgment.

Original: Our work sets a milestone in showing that a generative approach can outperform existing supervised and adversarial approaches when evaluated by expert listeners.
Revised: Our work sets a milestone in showing that a generative approach can outperform existing supervised and adversarial approaches when evaluated by expert listeners.

Original:  D IFFUSION - BASED U NIVERSAL S PEECH E NHANCEMENT 3. M ETHODOLOGY
Revised:  D IFFUSION - BASED U NIVERSAL S PEECH E NHANCEMENT 3. M ETHODOLOGY

Original: Data — To train our model, we use a data set of clean and programmatically-distorted pairs of speech recordings.
Revised: Data — To train our model, we use a data set of clean and programmatically-distorted pairs of speech recordings.

Original: To obtain the clean speech, we sample 1,500 h of audio from an internal pool of data sets and convert it to 16 kHz mono.
Revised: To obtain the clean speech, we sample 1,500 h of audio from an internal pool of data sets and convert it to 16 kHz mono.

Original: The speech sample consists of about 1.2 M utterances of between 3.5 and 5.5 s, from thousands of speakers, in more than 10 languages, and with over 50 different recording conditions (clean speech is chosen to be the most dry possible and of high quality).
Revised: The speech sample consists of about 1.2 M utterances of between 3.5 and 5.5 s, from thousands of speakers, in more than 10 languages, and with over 50 different recording conditions (clean speech is chosen to be the most dry possible and of high quality).

Original: To validate our model, we use 1.5 h of clean utterances sampled from VCTK (Yamagishi et al., 2019) and Harvard sentences (Henter et al., 2014), together with noises/backgrounds from
Revised: To validate our model, we use 1.5 h of clean utterances sampled from VCTK (Yamagishi et al., 2019) and Harvard sentences (Henter et al., 2014), together with noises/backgrounds from

Original: DEMAND (Thiemann et al., 2013) and FSDnoisy18k (Fonseca et al., 2019).
Revised: DEMAND (Thiemann et al., 2013) and FSDnoisy18k (Fonseca et al., 2019).

Original: Train data does not overlap with the validation partition nor with other data used for evaluation or subjective testing.
Revised: Train data does not overlap with the validation partition nor with other data used for evaluation or subjective testing.

Original: To programmatically generate distorted speech, we consider 10 distortion families: band limiting, codec artifacts, signal distortion, loudness dynamics, equalization, recorded noise, reverb, spectral manipulation, synthetic noise, and transmission.
Revised: To programmatically generate distorted speech, we consider 10 distortion families: band limiting, codec artifacts, signal distortion, loudness dynamics, equalization, recorded noise, reverb, spectral manipulation, synthetic noise, and transmission.

Original: Each family includes a variety of distortion algorithms, which we generically call ‘types’.
Revised: Each family includes a variety of distortion algorithms, which we generically call ‘types’.

Original: For instance, types of synthetic noise include colored noise, electricity tones, non-stationary noise bursts, etc., types of codecs include OPUS, Vorbis, MP3, EAC3, etc., types of reverb include algorithmic reverb, delays, and both real and simulated room impulse responses, and so on.
Revised: For instance, types of synthetic noise include colored noise, electricity tones, non-stationary noise bursts, etc., types of codecs include OPUS, Vorbis, MP3, EAC3, etc., types of reverb include algorithmic reverb, delays, and both real and simulated room impulse responses, and so on.

Original: In total, we leverage 55 distortion types.
Revised: In total, we leverage 55 distortion types.

Original: Distortion type parameters such as strength, frequency, or gain are set randomly within reasonable bounds.
Revised: Distortion type parameters such as strength, frequency, or gain are set randomly within reasonable bounds.

Original: A more comprehensive list of distortion families, types, and parameters can be found in Appendix C.
Revised: A more comprehensive list of distortion families, types, and parameters can be found in Appendix C.

Original: Evaluation — To measure relative improvement, we use objective speech quality metrics reﬂecting different criteria.
Revised: Evaluation — To measure relative improvement, we use objective speech quality metrics reﬂecting different criteria.

Original: On the one hand, we employ speech enhancement metrics COVL (Loizou, 2013) and STOI (Taal et al., 2010), which are widely used for denoising or dereverberation tasks.
Revised: On the one hand, we employ speech enhancement metrics COVL (Loizou, 2013) and STOI (Taal et al., 2010), which are widely used for denoising or dereverberation tasks.

Original: On the other hand, we employ the codec quality metric WARP-Q (Jassim et al., 2021) and an internal semi-supervised metric imitating SESQA (Serrà et al., 2021b), which should perform well with generative algorithms with perceptually-valid outputs that do not necessarily perfectly align with the target signal.
Revised: On the other hand, we employ the codec quality metric WARP-Q (Jassim et al., 2021) and an internal semi-supervised metric imitating SESQA (Serrà et al., 2021b), which should perform well with generative algorithms with perceptually-valid outputs that do not necessarily perfectly align with the target signal.

Original: We also consider the composite measure COMP that results from normalizing the previous four metrics between 0 and 10 and taking the average.
Revised: We also consider the composite measure COMP that results from normalizing the previous four metrics between 0 and 10 and taking the average.

Original: To compare with the state of the art, we perform a subjective preference test.
Revised: To compare with the state of the art, we perform a subjective preference test.

Original: In it, listeners are presented with a reference distorted signal plus two enhanced versions of it: one by an existing approach and one by the proposed approach.
Revised: In it, listeners are presented with a reference distorted signal plus two enhanced versions of it: one by an existing approach and one by the proposed approach.

Original: Then, they are asked to choose which of the two enhanced signals they prefer, based on the presence of the original nuisance, voice distortion, audible artifacts, etc.
Revised: Then, they are asked to choose which of the two enhanced signals they prefer, based on the presence of the original nuisance, voice distortion, audible artifacts, etc.

Original: The test was voluntarily performed by 22 expert listeners, each of them listening to randomly-chosen pairs of distorted and enhanced signals, taken from the online demo/example pages of 12 existing approaches, plus the corresponding version enhanced by our approach.
Revised: The test was voluntarily performed by 22 expert listeners, each of them listening to randomly-chosen pairs of distorted and enhanced signals, taken from the online demo/example pages of 12 existing approaches, plus the corresponding version enhanced by our approach.

Original: Further details of our evaluation methodology are provided in Appendix D.
Revised: Further details of our evaluation methodology are provided in Appendix D.

Original: 3. B ASE A PPROACH
Revised: 3. B ASE A PPROACH

Original: Score-based diffusion — In this work, we use a variance exploding (VE) diffusion approach (Song et al., 2021).
Revised: Score-based diffusion — In this work, we use a variance exploding (VE) diffusion approach (Song et al., 2021).

Original: We train our score network S following a denoising score matching paradigm (Song &
Revised: We train our score network S following a denoising score matching paradigm (Song &

Original: Ermon, 2019; 2020), using
Revised: Ermon, 2019; 2020), using

Original: [Equation]
Revised: [Equation]

Original: where t ∼ U (0 , 1) , z t ∼ N ( 0 , I ) ,
Revised: where t ∼ U (0 , 1) , z t ∼ N ( 0 , I ) ,

Original: x 0 ∼ p data , c is the conditioning signal, and values σ t follow a geometric noise schedule (Song & Ermon, 2019).
Revised: x 0 ∼ p data , c is the conditioning signal, and values σ t follow a geometric noise schedule (Song & Ermon, 2019).

Original: In all our experiments we use σ 0 = 5 · 10 − 4 and σ 1 = 5 , which we ﬁnd sufﬁcient for audio signals between − 1 and 1 (cf.
Revised: In all our experiments we use σ 0 = 5 · 10 − 4 and σ 1 = 5 , which we ﬁnd sufﬁcient for audio signals between − 1 and 1 (cf.

Original: Song & Ermon, 2020).
Revised: Song & Ermon, 2020).

Original: We consider different approaches to obtain c , but all of them have ˜ x 0 , the distorted version of x 0 , as the main and only input ( x 0 and ˜ x 0 are both in the waveform domain).
Revised: We consider different approaches to obtain c , but all of them have ˜ x 0 , the distorted version of x 0 , as the main and only input ( x 0 and ˜ x 0 are both in the waveform domain).

Original: To sample, we follow noise-consistent Langevin dynamics (Jolicoeur-Martineau et al., 2021), which corresponds to the recursion
Revised: To sample, we follow noise-consistent Langevin dynamics (Jolicoeur-Martineau et al., 2021), which corresponds to the recursion

Original: [Equation]
Revised: [Equation]

Original: over N uniformly discretized time steps t n ∈ [0 , 1] , where we set η and β with the help of a hyperparameter (cid:15) ∈ [1 , ∞ ) .
Revised: over N uniformly discretized time steps t n ∈ [0 , 1] , where we set η and β with the help of a hyperparameter (cid:15) ∈ [1 , ∞ ) .

Original: An extensive derivation of our approach to both training and sampling can be found in Appendix A, including an introduction to score-based diffusion.
Revised: An extensive derivation of our approach to both training and sampling can be found in Appendix A, including an introduction to score-based diffusion.

Original: General model description — We use convolutional blocks and a couple of bi-directional recurrent layers.
Revised: General model description — We use convolutional blocks and a couple of bi-directional recurrent layers.

Original: Convolutional blocks are formed by three 1D convolutional layers, each one preceded by a multi-parametric ReLU (PReLU) activation, and all of them under a residual connection.
Revised: Convolutional blocks are formed by three 1D convolutional layers, each one preceded by a multi-parametric ReLU (PReLU) activation, and all of them under a residual connection.

Original: If needed,
Revised: If needed,

Original: Conv
Revised: Conv

Original: ConvBlock
Revised: ConvBlock

Original: SigmaBlock
Revised: SigmaBlock

Original: ConvBlock ...
Revised: ConvBlock ...

Original: GRU(1)
Revised: GRU(1)

Original: ConvBlock
Revised: ConvBlock

Original: ConvBlock ...
Revised: ConvBlock ...

Original: PReLU Linear
Revised: PReLU Linear

Original: Conv
Revised: Conv

Original: ConvBlock
Revised: ConvBlock

Original: ConvBlock ...
Revised: ConvBlock ...

Original: ConvBlock
Revised: ConvBlock

Original: ConvBlock ...
Revised: ConvBlock ...

Original: ConvBlock
Revised: ConvBlock

Original: GRU(2)
Revised: GRU(2)

Original: ConvBlock
Revised: ConvBlock

Original: MDN
Revised: MDN

Original: ConvBlock
Revised: ConvBlock

Original: Mel bands
Revised: Mel bands

Original: ConvBlock PReLU StConv ... ...
Revised: ConvBlock PReLU StConv ... ...

Original: PReLU StConv
Revised: PReLU StConv

Original: A: Generator network
Revised: A: Generator network

Original: B: Conditioning network
Revised: B: Conditioning network

Original: Lin ...
Revised: Lin ...

Original: Lin
Revised: Lin

Original: Lin ...
Revised: Lin ...

Original: Lin
Revised: Lin

Original: Lin ...
Revised: Lin ...

Original: Lin
Revised: Lin

Original: MDN ... MDN ...
Revised: MDN ... MDN ...

Original: Conv up- or down-sampling is applied before or after the residual link, respectively (Appendix B).
Revised: Conv up- or down-sampling is applied before or after the residual link, respectively (Appendix B).

Original: We perform up-/down-sampling with transposed/strided convolutions, halving/doubling the number of channels at every step.
Revised: We perform up-/down-sampling with transposed/strided convolutions, halving/doubling the number of channels at every step.

Original: The down-sampling factors are {2,4,4,5}, which yield a 100 Hz latent representation for a 16 kHz input.
Revised: The down-sampling factors are {2,4,4,5}, which yield a 100 Hz latent representation for a 16 kHz input.

Original: The model consists of a generator and a conditioning network (Fig. 1).
Revised: The model consists of a generator and a conditioning network (Fig. 1).

Original: The generator network is formed by a UNet-like structure with skip connections v and a gated recurrent unit (GRU) in the middle (Fig. 1-A).
Revised: The generator network is formed by a UNet-like structure with skip connections v and a gated recurrent unit (GRU) in the middle (Fig. 1-A).

Original: Convolutional blocks in the generator receive adaptor signals g , which inform the network about the noise level σ , and conditioning signals c , which provide the necessary speech cues for synthesis.
Revised: Convolutional blocks in the generator receive adaptor signals g , which inform the network about the noise level σ , and conditioning signals c , which provide the necessary speech cues for synthesis.

Original: Signals g and c are mixed with the UNet activations using FiLM (Perez et al., 2018) and summation, respectively.
Revised: Signals g and c are mixed with the UNet activations using FiLM (Perez et al., 2018) and summation, respectively.

Original: To obtain g , we process the logarithm of σ with random Fourier feature embeddings and an MLP as in Rouard & Hadjeres (2021) (see also Appendix B).
Revised: To obtain g , we process the logarithm of σ with random Fourier feature embeddings and an MLP as in Rouard & Hadjeres (2021) (see also Appendix B).

Original: The conditioning network processes the distorted signal ˜ x with convolutional blocks featuring skip connections to a down-sampled latent that further exploits log-mel features extracted from ˜ x
Revised: The conditioning network processes the distorted signal ˜ x with convolutional blocks featuring skip connections to a down-sampled latent that further exploits log-mel features extracted from ˜ x

Original: (Fig.
Revised: (Fig.

Original: 1-B).
Revised: 1-B).

Original: The middle and decoding parts of the network are formed by a two-layer GRU and multiple convolutional blocks, with the decoder up-sampling the latent representation to provide multi-resolution conditionings c to the generator.
Revised: The middle and decoding parts of the network are formed by a two-layer GRU and multiple convolutional blocks, with the decoder up-sampling the latent representation to provide multi-resolution conditionings c to the generator.

Original: Multiple heads and target information are exploited to improve the latent representation and provide a better c (see below).
Revised: Multiple heads and target information are exploited to improve the latent representation and provide a better c (see below).

Original: We call our approach UNIVERSE, for universal speech enhancer.
Revised: We call our approach UNIVERSE, for universal speech enhancer.

Original: 3. D EVELOPING UNIVERSE
Revised: 3. D EVELOPING UNIVERSE

Original: In the following, we explain all the steps we took to arrive at the above-mentioned structure, motivating each decision and quantifying its impact with objective metrics (schematic diagrams are depicted in Fig. 2).
Revised: In the following, we explain all the steps we took to arrive at the above-mentioned structure, motivating each decision and quantifying its impact with objective metrics (schematic diagrams are depicted in Fig. 2).

Original: Unless stated otherwise, all models are trained with Adam and weight decay for 1 M iterations, with two-second long audio frames and a batch size of 32.
Revised: Unless stated otherwise, all models are trained with Adam and weight decay for 1 M iterations, with two-second long audio frames and a batch size of 32.

Original: We use a cosine learning rate schedule with a linear warm-up that spans the ﬁrst 5 % of the iterations.
Revised: We use a cosine learning rate schedule with a linear warm-up that spans the ﬁrst 5 % of the iterations.

Original: Under these speciﬁcations,
Revised: Under these speciﬁcations,

Original: Encoder
Revised: Encoder

Original: Decoder
Revised: Decoder

Original: Enc
Revised: Enc

Original: Dec
Revised: Dec

Original: Enc Encoder
Revised: Enc Encoder

Original: Decoder
Revised: Decoder

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: B B
Revised: B B

Original: B
Revised: B

Original: S1,S
Revised: S1,S

Original: S
Revised: S

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: CNN CNN
Revised: CNN CNN

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: MDN MDN
Revised: MDN MDN

Original: S
Revised: S

Original: S
Revised: S

Original: S6,S7,F,A1,A
Revised: S6,S7,F,A1,A

Original: A
Revised: A

Original: Enc +
Revised: Enc +

Original: Dec
Revised: Dec

Original: Enc +
Revised: Enc +

Original: Dec training a UNIVERSE model with 49 M parameters takes less than 5 days using two Tesla V
Revised: Dec training a UNIVERSE model with 49 M parameters takes less than 5 days using two Tesla V

Original: GPUs and PyTorch’s native automatic mixed precision (Paszke et al., 2019).
Revised: GPUs and PyTorch’s native automatic mixed precision (Paszke et al., 2019).

Original: Initial baselines (B1–B3) — We start with a structure inspired by WaveGrad (Chen et al., 2021), featuring a UNet-like architecture with the aforementioned convolutional blocks, each with 64, 128, 256, and 512 channels (Sec. 3.2 and Appendix B).
Revised: Initial baselines (B1–B3) — We start with a structure inspired by WaveGrad (Chen et al., 2021), featuring a UNet-like architecture with the aforementioned convolutional blocks, each with 64, 128, 256, and 512 channels (Sec. 3.2 and Appendix B).

Original: To condition we use the distorted speech ˜ x , from which we compute a 100 Hz log-mel representation with 80 bands that is summed to the UNet latent after a linear projection (Fig. 2-B1).
Revised: To condition we use the distorted speech ˜ x , from which we compute a 100 Hz log-mel representation with 80 bands that is summed to the UNet latent after a linear projection (Fig. 2-B1).

Original: This approach turns out to lack enhancement capabilities, probably hampered by the coarse mel representation of ˜ x .
Revised: This approach turns out to lack enhancement capabilities, probably hampered by the coarse mel representation of ˜ x .

Original: It also presents considerable speech distortion, probably due to the distortions in the mel representation (Table 1, B1).
Revised: It also presents considerable speech distortion, probably due to the distortions in the mel representation (Table 1, B1).

Original: We next consider an approach inspired by NU-Wave (Lee & Han, 2021).
Revised: We next consider an approach inspired by NU-Wave (Lee & Han, 2021).

Original: We employ the same convolutional UNet structure as before, but inject the conditioning signal ˜ x
Revised: We employ the same convolutional UNet structure as before, but inject the conditioning signal ˜ x

Original: at the input (Fig. 2-B2), concatenated with x + σ z (see Eq. 1).
Revised: at the input (Fig. 2-B2), concatenated with x + σ z (see Eq. 1).

Original: We ﬁnd that this approach yields less speech distortion and better scores than the previous one (Table 1, B2).
Revised: We ﬁnd that this approach yields less speech distortion and better scores than the previous one (Table 1, B2).

Original: However, after listening to some excepts, we notice that a lot of the audio is ‘bypassed’ from input to output.
Revised: However, after listening to some excepts, we notice that a lot of the audio is ‘bypassed’ from input to output.

Original: This leads to, for example, input noises present at the output, or silent gaps not being reconstructed.
Revised: This leads to, for example, input noises present at the output, or silent gaps not being reconstructed.

Original: We hypothesize that the UNet’s skip connections are the culprit for this behavior (and removing them affected training in a dramatic way).
Revised: We hypothesize that the UNet’s skip connections are the culprit for this behavior (and removing them affected training in a dramatic way).

Original: Another baseline approach we consider is inspired by ModDW (Zhang et al., 2021a).
Revised: Another baseline approach we consider is inspired by ModDW (Zhang et al., 2021a).

Original: In it, following
Revised: In it, following

Original: DiffWave (Kong et al., 2021), an auxiliary encoder and loss are used to learn a conditioning latent (Fig. 2-B3).
Revised: DiffWave (Kong et al., 2021), an auxiliary encoder and loss are used to learn a conditioning latent (Fig. 2-B3).

Original: This encoder uses the same blocks as the generative one, but with progressively downsampled skip connections.
Revised: This encoder uses the same blocks as the generative one, but with progressively downsampled skip connections.

Original: Also different from ModDW, we use mean squared error (MSE) instead of an L1 loss, and compare directly with the clean mel-band representation instead of a pre-learnt latent.
Revised: Also different from ModDW, we use mean squared error (MSE) instead of an L1 loss, and compare directly with the clean mel-band representation instead of a pre-learnt latent.

Original: With these two modiﬁcations, we ﬁnd no problems in training the model end-to-end, in contrast to the original ModDW, which had to be trained using separate stages (Zhang et al., 2021a).
Revised: With these two modiﬁcations, we ﬁnd no problems in training the model end-to-end, in contrast to the original ModDW, which had to be trained using separate stages (Zhang et al., 2021a).

Original: The scores for this approach are in the middle of the previous baselines, closer to the second one (Table 1, B3).
Revised: The scores for this approach are in the middle of the previous baselines, closer to the second one (Table 1, B3).

Original: Capacity and losses (S1–S7) — With the three baselines above, we decide to add processing capacity in order to obtain a better conditioning signal.
Revised: Capacity and losses (S1–S7) — With the three baselines above, we decide to add processing capacity in order to obtain a better conditioning signal.

Original: This decision was motivated by preliminary analysis showing that B1 and B3 were capable to synthesize high-quality speech in a vocoder setting (that is, when a clean mel-band conditioning was provided).
Revised: This decision was motivated by preliminary analysis showing that B1 and B3 were capable to synthesize high-quality speech in a vocoder setting (that is, when a clean mel-band conditioning was provided).

Original: Hence, with more capacity in the conditioning network, one should come closer to such clean mel-band scenario (an observation also made by
Revised: Hence, with more capacity in the conditioning network, one should come closer to such clean mel-band scenario (an observation also made by

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021a)).
Revised: (2021a)).

Original: Unfortunately, increasing the capacity for the best scoring model, B2, could not avoid the bypass problems outlined above.
Revised: Unfortunately, increasing the capacity for the best scoring model, B2, could not avoid the bypass problems outlined above.

Original: Therefore, we focus on improving B3, which had the best scores after B2 and a similar listening quality, with less noise but more speech distortion.
Revised: Therefore, we focus on improving B3, which had the best scores after B2 and a similar listening quality, with less noise but more speech distortion.

Original: We ﬁrst add an extra log-mel input to the conditioner’s encoder, with a convolutional block after it, and sum its output to the skip connections coming from the distorted waveform processing (mels are extracted from the same distorted signal ˜ x
Revised: We ﬁrst add an extra log-mel input to the conditioner’s encoder, with a convolutional block after it, and sum its output to the skip connections coming from the distorted waveform processing (mels are extracted from the same distorted signal ˜ x

Original: that is input to the waveform encoder, see Fig.
Revised: that is input to the waveform encoder, see Fig.

Original: 1-B).
Revised: 1-B).

Original: This
Revised: This

Original: [Table]
Revised: [Table]

Original: already results in an improvement with respect to B3, with objective scores that are very close to B (Table 1, S1).
Revised: already results in an improvement with respect to B3, with objective scores that are very close to B (Table 1, S1).

Original: Next, we add a two-layer GRU and two convolutional blocks after the skip summation.
Revised: Next, we add a two-layer GRU and two convolutional blocks after the skip summation.

Original: We also add a one-layer GRU to the generator latent.
Revised: We also add a one-layer GRU to the generator latent.

Original: Both GRUs should provide the model with a larger receptive ﬁeld and better sequential processing capabilities.
Revised: Both GRUs should provide the model with a larger receptive ﬁeld and better sequential processing capabilities.

Original: With this additions, the model increases the COMP score to 4.75 (Table 1, S2).
Revised: With this additions, the model increases the COMP score to 4.75 (Table 1, S2).

Original: After improving the encoders’ capacity, we decide to add a waveform loss to the existing MSE on log-mel latents.
Revised: After improving the encoders’ capacity, we decide to add a waveform loss to the existing MSE on log-mel latents.

Original: To do so, we also need to include additional decoder blocks for up-sampling the latent (Fig. 2-S3).
Revised: To do so, we also need to include additional decoder blocks for up-sampling the latent (Fig. 2-S3).

Original: For the waveform loss, we use MSE together with a multi-resolution STFT loss (Yamamoto et al., 2020).
Revised: For the waveform loss, we use MSE together with a multi-resolution STFT loss (Yamamoto et al., 2020).

Original: The result is a noticeable improvement (Table 1, S3).
Revised: The result is a noticeable improvement (Table 1, S3).

Original: Although this is the largest improvement in objective scores, we do not observe such a large difference throughout informal listening.
Revised: Although this is the largest improvement in objective scores, we do not observe such a large difference throughout informal listening.

Original: Better low-level details are present, but we suspect a large part of the score improvement is due to the objective measures paying too much attention to (sometimes irrelevant) low-level detail, which is now induced by the losses in the waveform domain.
Revised: Better low-level details are present, but we suspect a large part of the score improvement is due to the objective measures paying too much attention to (sometimes irrelevant) low-level detail, which is now induced by the losses in the waveform domain.

Original: Now that we have parallel up-sampling blocks in the generator and conditioning decoders, we can condition at multiple resolutions between 100 Hz and 16 kHz (Fig. 2-S4).
Revised: Now that we have parallel up-sampling blocks in the generator and conditioning decoders, we can condition at multiple resolutions between 100 Hz and 16 kHz (Fig. 2-S4).

Original: This provides an improvement (Table 1, S4).
Revised: This provides an improvement (Table 1, S4).

Original: However, after careful listening, we have the impression that loss errors have a strong effect on the conditioning, provoking alterations that difﬁcult the task of the generator (for instance, we hear a bit of mufﬂed speech, presumably resulting from using MSE in both mel-band and waveform domains).
Revised: However, after careful listening, we have the impression that loss errors have a strong effect on the conditioning, provoking alterations that difﬁcult the task of the generator (for instance, we hear a bit of mufﬂed speech, presumably resulting from using MSE in both mel-band and waveform domains).

Original: To try to alleviate these issues, we decouple the loss calculations from the main signal path of the conditioning network.
Revised: To try to alleviate these issues, we decouple the loss calculations from the main signal path of the conditioning network.

Original: To do so, we use two convolutional heads (Fig. 2-S5), which include layer normalization, PReLUs, and a convolution with a kernel width of 3.
Revised: To do so, we use two convolutional heads (Fig. 2-S5), which include layer normalization, PReLUs, and a convolution with a kernel width of 3.

Original: With that, we observe an improvement not only in objective metrics (Table 1, S5), but also in subjective quality.
Revised: With that, we observe an improvement not only in objective metrics (Table 1, S5), but also in subjective quality.

Original: Decoupled auxiliary heads allow us to further think of alternative loss functions that might work better for latent and waveform representations.
Revised: Decoupled auxiliary heads allow us to further think of alternative loss functions that might work better for latent and waveform representations.

Original: Interestingly, such loss functions do not need to be restricted to regression or adversarial approaches, as used in the literature, but can be probabilistic and model the representations’ distribution explicitly.
Revised: Interestingly, such loss functions do not need to be restricted to regression or adversarial approaches, as used in the literature, but can be probabilistic and model the representations’ distribution explicitly.

Original: Now that losses are out of the signal path, nothing prevents us from using approaches that otherwise would require to sample from a probabilistic prediction in order to proceed with the signal ﬂow.
Revised: Now that losses are out of the signal path, nothing prevents us from using approaches that otherwise would require to sample from a probabilistic prediction in order to proceed with the signal ﬂow.

Original: With the idea of better modeling the distribution of both log-mels and waveforms, we employ a mixture density network (MDN) approach (Bishop, 1994), with the same head architecture as before, but with 3 Gaussian components, each calculated from different convolutional layers (Fig. 2-S6).
Revised: With the idea of better modeling the distribution of both log-mels and waveforms, we employ a mixture density network (MDN) approach (Bishop, 1994), with the same head architecture as before, but with 3 Gaussian components, each calculated from different convolutional layers (Fig. 2-S6).

Original: Assuming independent time steps, the negative log-likelihood of each time step using k components of dimensionality d and diagonal covariance is
Revised: Assuming independent time steps, the negative log-likelihood of each time step using k components of dimensionality d and diagonal covariance is

Original: [Equation]
Revised: [Equation]

Original: where y is the target representation and α ( i ) , m ( i ) , and s ( i ) denote the output of the convolutional layer corresponding to the mixing probability, the mean, and the standard deviation of the i -th Gaussian, respectively.
Revised: where y is the target representation and α ( i ) , m ( i ) , and s ( i ) denote the output of the convolutional layer corresponding to the mixing probability, the mean, and the standard deviation of the i -th Gaussian, respectively.

Original: Replacing standard losses by MDNs has a positive effect on the objective scores (Table 1,
Revised: Replacing standard losses by MDNs has a positive effect on the objective scores (Table 1,

Original: S6).
Revised: S6).

Original: Together with moving the loss computation out of the signal path (S5), they provide a COMP increment of 0.41, the largest one besides using an auxiliary decoder and loss (S3).
Revised: Together with moving the loss computation out of the signal path (S5), they provide a COMP increment of 0.41, the largest one besides using an auxiliary decoder and loss (S3).

Original: After introducing the MDNs, our last step consists of including more targets in the latent predictions.
Revised: After introducing the MDNs, our last step consists of including more targets in the latent predictions.

Original: These additional targets are pitch and harmonicity, as provided by crepe (Kim et al., 2018), voice activity detection (VAD) and loudness, provided by simple internal algorithms, and the deltas of all of them.
Revised: These additional targets are pitch and harmonicity, as provided by crepe (Kim et al., 2018), voice activity detection (VAD) and loudness, provided by simple internal algorithms, and the deltas of all of them.

Original: We consider a separate MDN for each target type.
Revised: We consider a separate MDN for each target type.

Original: That is, one for mel bands and their deltas, one for pitch/harmonicity and their deltas, and one for VAD/loudness and their deltas.
Revised: That is, one for mel bands and their deltas, one for pitch/harmonicity and their deltas, and one for VAD/loudness and their deltas.

Original: The result only provides a marginal improvement in objective scores (Table 1, S7), but we ﬁnd such improvement to be consistent after listening to a number of enhanced signals.
Revised: The result only provides a marginal improvement in objective scores (Table 1, S7), but we ﬁnd such improvement to be consistent after listening to a number of enhanced signals.

Original: Scaling up (F) — Finally, after deﬁning our base model, we can train a larger model for a longer time.
Revised: Scaling up (F) — Finally, after deﬁning our base model, we can train a larger model for a longer time.

Original: We increase the model size by doubling the number of channels and we reduce the learning rate by two.
Revised: We increase the model size by doubling the number of channels and we reduce the learning rate by two.

Original: The resulting model has 189 M parameters, and is trained for 2.5 M iterations using a batch size of 64.
Revised: The resulting model has 189 M parameters, and is trained for 2.5 M iterations using a batch size of 64.

Original: This takes less than 14 days using 8 Tesla V100 GPUs.
Revised: This takes less than 14 days using 8 Tesla V100 GPUs.

Original: The result of scaling up parameters and training is a large improvement, both objectively and subjectively (Table 1, F).
Revised: The result of scaling up parameters and training is a large improvement, both objectively and subjectively (Table 1, F).

Original: This is the model we will use in our ﬁnal evaluation, with 50 diffusion steps (see Sec. 4.3).
Revised: This is the model we will use in our ﬁnal evaluation, with 50 diffusion steps (see Sec. 4.3).

Original: Further ablations (A1–A3) — Starting with S7, we can further assess the effect of some design choices.
Revised: Further ablations (A1–A3) — Starting with S7, we can further assess the effect of some design choices.

Original: For instance, it is common in speech enhancement to train multi-part models using multiple stages, or taking some pre-trained (frozen) parts (Polyak et al., 2021; Maiti & Mandel, 2019; Nair & Koishida, 2021;
Revised: For instance, it is common in speech enhancement to train multi-part models using multiple stages, or taking some pre-trained (frozen) parts (Polyak et al., 2021; Maiti & Mandel, 2019; Nair & Koishida, 2021;

Original: Liu et al., 2021).
Revised: Liu et al., 2021).

Original: The equivalent of this strategy for our two-part model would be to ﬁrst train the conditioner using all L MDN losses, freeze it, and then train the generator with
Revised: The equivalent of this strategy for our two-part model would be to ﬁrst train the conditioner using all L MDN losses, freeze it, and then train the generator with

Original: L SCORE .
Revised: L SCORE .

Original: An intuition for this could be that, this way, the generator network is ‘decoupled’ from the enhancement/conditioner one, and that therefore the two networks can fully concentrate in their commitment (generating and cleaning, respectively).
Revised: An intuition for this could be that, this way, the generator network is ‘decoupled’ from the enhancement/conditioner one, and that therefore the two networks can fully concentrate in their commitment (generating and cleaning, respectively).

Original: Nonetheless, this intuition seems to be at least partially wrong, as we obtain worse scores (Table 1, A1).
Revised: Nonetheless, this intuition seems to be at least partially wrong, as we obtain worse scores (Table 1, A1).

Original: In fact, this result points towards a ‘coupling’ situation, in which part of the generator performs some enhancement and part of the conditioner shapes the generator input.
Revised: In fact, this result points towards a ‘coupling’ situation, in which part of the generator performs some enhancement and part of the conditioner shapes the generator input.

Original: Another design choice we can question is the strategy to provide the diffusion model with the noise level σ (SigmaBlock, Fig. 1).
Revised: Another design choice we can question is the strategy to provide the diffusion model with the noise level σ (SigmaBlock, Fig. 1).

Original: This strategy is used by some audio generation models (like CRASH by Rouard & Hadjeres (2021), from which we borrow it), but other models employ other strategies or none.
Revised: This strategy is used by some audio generation models (like CRASH by Rouard & Hadjeres (2021), from which we borrow it), but other models employ other strategies or none.

Original: We observe that the use of this strategy is important for our generator network (Table 1, A2).
Revised: We observe that the use of this strategy is important for our generator network (Table 1, A2).

Original: Finally, we can also quantify the effect of the additional losses L MDN .
Revised: Finally, we can also quantify the effect of the additional losses L MDN .

Original: By removing them and training the whole model only with L SCORE , we observe a clear decrease in objective scores (Table 1, A3).
Revised: By removing them and training the whole model only with L SCORE , we observe a clear decrease in objective scores (Table 1, A3).

Original: Thus, we conclude that both auxiliary noise levels and losses are important.
Revised: Thus, we conclude that both auxiliary noise levels and losses are important.

Original:  R ESULTS 4. C OMPARISON WITH E XISTING A PPROACHES
Revised:  R ESULTS 4. C OMPARISON WITH E XISTING A PPROACHES

Original: To compare with the state of the art, a common approach is to use objective metrics and wellestablished test sets.
Revised: To compare with the state of the art, a common approach is to use objective metrics and wellestablished test sets.

Original: However, since the task of universal speech enhancement has not been formally addressed before, an established test set does not exist.
Revised: However, since the task of universal speech enhancement has not been formally addressed before, an established test set does not exist.

Original: Furthermore, it is not yet clear if common objective metrics provide a reasonable measurement for the enhancement of other distortions beyond additive noise and reverberation.
Revised: Furthermore, it is not yet clear if common objective metrics provide a reasonable measurement for the enhancement of other distortions beyond additive noise and reverberation.

Original: An alternative is to evaluate on separate, individual test sets, each of them focused on a particular task (for example denoising, declipping, codec artifact removal, and so on).
Revised: An alternative is to evaluate on separate, individual test sets, each of them focused on a particular task (for example denoising, declipping, codec artifact removal, and so on).

Original: However, to the best of our knowledge, there do not exist well-established test sets nor metrics for other tasks beyond denoising.
Revised: However, to the best of our knowledge, there do not exist well-established test sets nor metrics for other tasks beyond denoising.

Original: Therefore, to be the most fair possible to existing approaches, and in order to skip potentially ﬂawed objective metrics, we think the best approach is to conduct a subjective test with expert listeners (Sec. 3.1 and Appendix D).
Revised: Therefore, to be the most fair possible to existing approaches, and in order to skip potentially ﬂawed objective metrics, we think the best approach is to conduct a subjective test with expert listeners (Sec. 3.1 and Appendix D).

Original: Nonetheless, in Appendix E we also report objective scores for the two most common denoising test sets and show that the proposed approach achieves state of the art results despite being generative and not favored by objective metrics.
Revised: Nonetheless, in Appendix E we also report objective scores for the two most common denoising test sets and show that the proposed approach achieves competitive results despite being generative and not favored by objective metrics.

Original: In our subjective test, we compare against 12 existing approaches on different combinations of enhancement tasks (Table 2).
Revised: In our subjective test, we compare against 12 existing approaches on different combinations of enhancement tasks (Table 2).

Original: We ﬁnd that UNIVERSE outperforms all existing approaches by a
Revised: We ﬁnd that UNIVERSE outperforms all existing approaches by a

Original: [Table]
Revised: [Table]

Original: signiﬁcant margin (Table 2, top).
Revised: signiﬁcant margin (Table 2, top).

Original: In all considered distortions, UNIVERSE is preferred by expert listeners when compared to the corresponding competitor.
Revised: In all considered distortions, UNIVERSE is preferred by expert listeners when compared to the corresponding competitor.

Original: The closest competitors are HiFi-GAN-2, which considers denoising, dereverb
Revised: The closest competitors are HiFi-GAN-2, which considers denoising, dereverb

Original: [Table]
Revised: [Table]

Original: and equalization, SEANet, which only considers bandwidth extension
Revised: and equalization, SEANet, which only considers bandwidth extension

Original: [Table]
Revised: [Table]

Original: and VoiceFixer, which considers denoising, dereverb, bandwidth extension, and declipping.
Revised: and VoiceFixer, which considers denoising, dereverb, bandwidth extension, and declipping.

Original: We encourage the reader to listen to the UNIVERSE-enhanced examples in the companion website 1 .
Revised: We encourage the reader to listen to the UNIVERSE-enhanced examples in the companion website 1 .

Original: 4. V ARIATIONS AND I NSIGHTS
Revised: 4. V ARIATIONS AND I NSIGHTS

Original: Another interesting set of results stems from comparing against ablations or ground truth data (Table 2, bottom).
Revised: Another interesting set of results stems from comparing against ablations or ground truth data (Table 2, bottom).

Original: In particular, we try to answer the following questions:
Revised: In particular, we try to answer the following questions:

Original: Do we ﬁnd a clear gain from using a generative, diffusion-based approach compared to using classical regression losses?
Revised: Do we ﬁnd a clear gain from using a generative, diffusion-based approach compared to using classical regression losses?

Original: To answer this question, we performed several preliminary experiments with different regression-based alternatives, and concluded that the best candidate for the subjective test was
Revised: To answer this question, we performed several preliminary experiments with different regression-based alternatives, and concluded that the best candidate for the subjective test was

Original: a version of UNIVERSE with exactly the same conﬁguration and capacity which, instead of a score matching loss for the generator network and diffusion-based sampling, uses MSE and STFT losses for direct waveform prediction (the conditioner network was still found to be superior with the MDN losses).
Revised: a version of UNIVERSE with exactly the same conﬁguration and capacity which, instead of a score matching loss for the generator network and diffusion-based sampling, uses MSE and STFT losses for direct waveform prediction (the conditioner network was still found to be superior with the MDN losses).

Original: The result of this approach was not at the level of the generative version (Table 2, UNIVERSE-Regress), especially with regard to speech distortion and artifacts.
Revised: The result of this approach was not at the level of the generative version (Table 2, UNIVERSE-Regress), especially with regard to speech distortion and artifacts.

Original: Listeners preferred the generative version 88 % of the time.
Revised: Listeners preferred the generative version 88 % of the time.

Original: Can it be a problem for the model to consider more distortions beyond additive noise?
Revised: Can it be a problem for the model to consider more distortions beyond additive noise?

Original: To assess this, we trained exactly the same version of UNIVERSE with a 500 h train set that consisted of only additive noise mixtures, and evaluated only in the additive noise case.
Revised: To assess this, we trained exactly the same version of UNIVERSE with a 500 h train set that consisted of only additive noise mixtures, and evaluated only in the additive noise case.

Original: The reason for using 1/3 of the hours used for the universal enhancement case is that we estimate that this number is not far from the amount of real-world additive noise in the full multi-distortion train set.
Revised: The reason for using 1/3 of the hours used for the universal enhancement case is that we estimate that this number is not far from the amount of real-world additive noise in the full multi-distortion train set.

Original: In this case, the results indicate that adding extra data and extra distortions does not affect performance (Table 2, UNIVERSE-Denoise).
Revised: In this case, the results indicate that adding extra data and extra distortions does not affect performance (Table 2, UNIVERSE-Denoise).

Original: In fact, there seems to be a slight advantage in doing so (43 vs.
Revised: In fact, there seems to be a slight advantage in doing so (43 vs.

Original: 57 % preference), albeit not a signiﬁcant one.
Revised: 57 % preference), albeit not a signiﬁcant one.

Original: How far are we from the ideal targets?
Revised: How far are we from the ideal targets?

Original: To gain intuition about this question, we included recordings from the target speech to the subjective test (with input references featuring all considered distortions).
Revised: To gain intuition about this question, we included recordings from the target speech to the subjective test (with input references featuring all considered distortions).

Original: In this case, listeners preferred the ideal targets 86 % of the time (Table 2, Ground truth oracle).
Revised: In this case, listeners preferred the ideal targets 86 % of the time (Table 2, Ground truth oracle).

Original: This, beyond conﬁrming that listeners were able to spot the clean
Revised: This, beyond conﬁrming that listeners were able to spot the clean

Original: N
Revised: N

Original: C O
Revised: C O

Original: M P =1.3 =1.
Revised: M P =1.3 =1.

Original: =2.3 =3.
Revised: =2.3 =3.

Original: R T F
Revised: R T F

Original: Results are in Appendix E. references, indicates that there is still some room for improvement for UNIVERSE.
Revised: Results are in Appendix E. references, indicates that there is still some room for improvement for UNIVERSE.

Original: Informal listening by the authors indicates that two of the most problematic cases are with very loud noises and strong (and usually long) reverbs.
Revised: Informal listening by the authors indicates that two of the most problematic cases are with very loud noises and strong (and usually long) reverbs.

Original: The former tends to yield some babbling while the latter yields noticeable speech distortions and, in some cases, also babbling.
Revised: The former tends to yield some babbling while the latter yields noticeable speech distortions and, in some cases, also babbling.

Original: 4. S PEED -Q UALITY T RADEOFF
Revised: 4. S PEED -Q UALITY T RADEOFF

Original: Score-based diffusion models require performing multiple denoising steps for high-quality sampling, and efﬁcient strategies to tackle this issue have been the subject of recent research.
Revised: Score-based diffusion models require performing multiple denoising steps for high-quality sampling, and efﬁcient strategies to tackle this issue have been the subject of recent research.

Original: In the literature, we ﬁnd that high-quality sampling can be obtained with relatively few steps for tasks that have a rich conditioning signal such as speech vocoding (for example, less than 10 steps as in Chen et al.
Revised: In the literature, we ﬁnd that high-quality sampling can be obtained with relatively few steps for tasks that have a rich conditioning signal such as speech vocoding (for example, less than 10 steps as in Chen et al.

Original: (2021); Kong et al.
Revised: (2021); Kong et al.

Original: Speech enhancement, as formulated here, should also be considered a task with a rich conditioning signal (essentially, low-level speech details are contained at the input, except for the distorted parts).
Revised: Speech enhancement, as formulated here, should also be considered a task with a rich conditioning signal (essentially, low-level speech details are contained at the input, except for the distorted parts).

Original: Therefore, we hypothesize that high-quality synthesis can also be achieved with relatively few steps.
Revised: Therefore, we hypothesize that high-quality synthesis can also be achieved with relatively few steps.

Original: Our results conﬁrm this hypothesis and show that we can obtain good quality synthesis with as few as 4–8 denoising steps, with a speed above 10 times real-time on
Revised: Our results conﬁrm this hypothesis and show that we can obtain good quality synthesis with as few as 4–8 denoising steps, with a speed above 10 times real-time on

Original: GPU (Fig. 3 and Appendix E).
Revised: GPU (Fig. 3 and Appendix E).

Original: Importantly, this holds for a variety of values of (cid:15) , and without the use of any speciﬁc strategy nor any search for an appropriate schedule of σ t (we use plain geometric scheduling).
Revised: Importantly, this holds for a variety of values of (cid:15) , and without the use of any speciﬁc strategy nor any search for an appropriate schedule of σ t (we use plain geometric scheduling).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we consider the task of speech enhancement as a holistic endeavor, and propose a universal speech enhancer that makes use of score-based diffusion for generation and MDN auxiliary heads for conditioning.
Revised: In this work, we consider the task of speech enhancement as a holistic endeavor, and propose a universal speech enhancer that makes use of score-based diffusion for generation and MDN auxiliary heads for conditioning.

Original: We show that this approach, UNIVERSE, outperforms 12 state-of-the-art approaches as evaluated by expert listeners in a subjective test, and that it can achieve high-quality enhancement with only 4–8 diffusion steps.
Revised: We show that this approach, UNIVERSE, outperforms 12 state-of-the-art approaches as evaluated by expert listeners in a subjective test, and that it can achieve high-quality enhancement with only 4–8 diffusion steps.

Original: Regarding the potential societal impact, we do not foresee any serious implications at this initial research stage.
Revised: Regarding the potential societal impact, we do not foresee any serious implications at this initial research stage.

Original: Despite being a generative model, the nature of the task enforces to enhance what is being input to the system without changing major characteristics, keeping content and intent absolutely unaltered.
Revised: Despite being a generative model, the nature of the task enforces to enhance what is being input to the system without changing major characteristics, keeping content and intent absolutely unaltered.

Original: Thus, for example, the system should not change the speaker’s identity or words unless attacked by a third party (we explicitly ask expert listeners to consider identity/word changes in their judgment).
Revised: Thus, for example, the system should not change the speaker’s identity or words unless attacked by a third party (we explicitly ask expert listeners to consider identity/word changes in their judgment).

Original: Finally, it is also worth noting that data-driven models highly depend on the training data characteristics.
Revised: Finally, it is also worth noting that data-driven models highly depend on the training data characteristics.

Original: Accordingly, before deploying such models, one should ensure that target languages and use cases are sufﬁciently represented in the train set (we explicitly include multiple languages and recording conditions in our training set).
Revised: Accordingly, before deploying such models, one should ensure that target languages and use cases are sufﬁciently represented in the train set (we explicitly include multiple languages and recording conditions in our training set).

Original: A PPENDIX A S CORE - BASED D IFFUSION M ODELS A. T HEORY
Revised: A PPENDIX A S CORE - BASED D IFFUSION M ODELS A. T HEORY

Original: Diffusion-based models are deﬁned through a forward process where we progressively add noise to samples from the data distribution, x 0 ∼ p data , until we obtain a result that is indistinguishable from a prior tractable distribution, x 1 ∼ p known .
Revised: Diffusion-based models are deﬁned through a forward process where we progressively add noise to samples from the data distribution, x 0 ∼ p data , until we obtain a result that is indistinguishable from a prior tractable distribution, x 1 ∼ p known .

Original: In the case of Gaussian noise, p known also becomes approximately Gaussian, and this process can be modeled (Song et al., 2021) through a stochastic differential equation (SDE):
Revised: In the case of Gaussian noise, p known also becomes approximately Gaussian, and this process can be modeled (Song et al., 2021) through a stochastic differential equation (SDE):

Original: [Equation]
Revised: [Equation]

Original: where f, g : R → R and w is the standard Wiener process (Brownian motion) indexed by a continuous time variable t ∈ [0 , 1] .
Revised: where f, g : R → R and w is the standard Wiener process (Brownian motion) indexed by a continuous time variable t ∈ [0 , 1] .

Original: Different deﬁnitions of f and g yield to different but equivalent processes (Song et al., 2021).
Revised: Different deﬁnitions of f and g yield to different but equivalent processes (Song et al., 2021).

Original: To model the backward process where we go from p known to p data , we can then employ the reverse-time SDE (Anderson, 1982)
Revised: To model the backward process where we go from p known to p data , we can then employ the reverse-time SDE (Anderson, 1982)

Original: [Equation]
Revised: [Equation]

Original: where ¯ w is the standard Wiener process in which time ﬂows backward, and ∇ x log p t ( x ) corresponds to the score of p t , the marginal distribution at time t (Hyvärinen, 2005).
Revised: where ¯ w is the standard Wiener process in which time ﬂows backward, and ∇ x log p t ( x ) corresponds to the score of p t , the marginal distribution at time t (Hyvärinen, 2005).

Original: Thus, once f and g are deﬁned, in order to obtain x 0 ∼ p data , we only need to know the score function ∇ x log p t ( x ) to sample x 1 ∼ p known and simulate the process of Eq. 3.
Revised: Thus, once f and g are deﬁned, in order to obtain x 0 ∼ p data , we only need to know the score function ∇ x log p t ( x ) to sample x 1 ∼ p known and simulate the process of Eq. 3.

Original: Since one does not typically have direct access to neither p t nor ∇ x log p t ( x ) , the solution is to approximate the latter with a neural network S ( x , t ) .
Revised: Since one does not typically have direct access to neither p t nor ∇ x log p t ( x ) , the solution is to approximate the latter with a neural network S ( x , t ) .

Original: In order to train S , Vincent (2011) showed that, for a given t , minimizing the score matching objective
Revised: In order to train S , Vincent (2011) showed that, for a given t , minimizing the score matching objective

Original: [Equation]
Revised: [Equation]

Original: is equivalent to minimizing the denoising objective
Revised: is equivalent to minimizing the denoising objective

Original: [Equation]
Revised: [Equation]

Original: where p t ( x t | x 0 ) corresponds to a Gaussian kernel (Song et al., 2021; Särkkä & Solin, 2019), the transition kernel for the forward SDE (Eq. 2).
Revised: where p t ( x t | x 0 ) corresponds to a Gaussian kernel (Song et al., 2021; Särkkä & Solin, 2019), the transition kernel for the forward SDE (Eq. 2).

Original: For all t , one can use the continuous generalization (Song & Ermon, 2019)
Revised: For all t , one can use the continuous generalization (Song & Ermon, 2019)

Original: [Equation]
Revised: [Equation]

Original: with t ∼ U (0 , 1) , where λ t is an appropriately chosen weight that depends on t (Ho et al., 2020; Song & Ermon, 2020).
Revised: with t ∼ U (0 , 1) , where λ t is an appropriately chosen weight that depends on t (Ho et al., 2020; Song & Ermon, 2020).

Original: Sampling with score-based diffusion models is done by simulating or solving the reverse-time
Revised: Sampling with score-based diffusion models is done by simulating or solving the reverse-time

Original: SDE (Eq. 3) with a ﬁnite (discrete) time schedule.
Revised: SDE (Eq. 3) with a ﬁnite (discrete) time schedule.

Original: This can be done in many ways, for instance by using numerical SDE and ODE solvers, as introduced by Song et al.
Revised: This can be done in many ways, for instance by using numerical SDE and ODE solvers, as introduced by Song et al.

Original: Other schemes that have shown competitive performance include predictor-corrector schemes (Song et al., 2021),
Revised: Other schemes that have shown competitive performance include predictor-corrector schemes (Song et al., 2021),

Original: ancestral sampling
Revised: ancestral sampling

Original: (Ho et al., 2020), and variations of Langevin dynamics (Song & Ermon, 2020;
Revised: (Ho et al., 2020), and variations of Langevin dynamics (Song & Ermon, 2020;

Original: Jolicoeur-Martineau et al., 2021).
Revised: Jolicoeur-Martineau et al., 2021).

Original: A. I N P RACTICE
Revised: A. I N P RACTICE

Original: In our work, we employ the so-called variance exploding schedule (VE; Song et al., 2021), which corresponds to choosing
Revised: In our work, we employ the so-called variance exploding schedule (VE; Song et al., 2021), which corresponds to choosing

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (5) in Eq.
Revised: (5) in Eq.

Original: Then, the associated transition kernel for the forward process (Särkkä & Solin, 2019) is p t ( x t | x 0 ) =
Revised: Then, the associated transition kernel for the forward process (Särkkä & Solin, 2019) is p t ( x t | x 0 ) =

Original: N (cid:0) x t ; x 0 , [ σ 2 t − σ 20 ] I (cid:1) , which in practice is approximated by
Revised: N (cid:0) x t ; x 0 , [ σ 2 t − σ 20 ] I (cid:1) , which in practice is approximated by

Original: [Equation]
Revised: [Equation]

Original: since σ 0 → 0 (see also Song et al., 2021).
Revised: since σ 0 → 0 (see also Song et al., 2021).

Original: The intuition is that p t =0 becomes indistinguishable from p data , and that p t =1 becomes indistinguishable from p known (a Gaussian distribution).
Revised: The intuition is that p t =0 becomes indistinguishable from p data , and that p t =1 becomes indistinguishable from p known (a Gaussian distribution).

Original: In other words, perturbation and signal should become imperceptible at t = 0 and t = 1 , respectively.
Revised: In other words, perturbation and signal should become imperceptible at t = 0 and t = 1 , respectively.

Original: In the VE schedule, the scale of the signal x 0 is kept intact, and then it corresponds to g ( t ) to fulﬁll that notion through a variance schedule (see Eq. 5).
Revised: In the VE schedule, the scale of the signal x 0 is kept intact, and then it corresponds to g ( t ) to fulﬁll that notion through a variance schedule (see Eq. 5).

Original: Therefore, σ 0 should be negligible while σ 1 should be large, compared to the variability of x 0 .
Revised: Therefore, σ 0 should be negligible while σ 1 should be large, compared to the variability of x 0 .

Original: That is, σ 20 (cid:28) E
Revised: That is, σ 20 (cid:28) E

Original: [( x 0 − E [ x 0 ]) 2 ] (cid:28) σ 21 .
Revised: [( x 0 − E [ x 0 ]) 2 ] (cid:28) σ 21 .

Original: The use of the approximated transition kernel (Eq. 6) implies that x t = x 0 + σ t z t , z t ∼ N ( 0 , I ) , and that
Revised: The use of the approximated transition kernel (Eq. 6) implies that x t = x 0 + σ t z t , z t ∼ N ( 0 , I ) , and that

Original: [Equation]
Revised: [Equation]

Original: where C is a constant (see also Vincent, 2011)).
Revised: where C is a constant (see also Vincent, 2011)).

Original: Substituting these into Eq.
Revised: Substituting these into Eq.

Original: 4 and operating yields
Revised: 4 and operating yields

Original: [Equation]
Revised: [Equation]

Original: From Eq.
Revised: From Eq.

Original: 5, it now remains to set how σ evolves.
Revised: 5, it now remains to set how σ evolves.

Original: Song & Ermon (2019; 2020) choose a geometric progression for σ t , σ t = σ min (cid:18)
Revised: Song & Ermon (2019; 2020) choose a geometric progression for σ t , σ t = σ min (cid:18)

Original: [Equation]
Revised: [Equation]

Original: , and provide some generic guidance on how to select σ min and σ max .
Revised: , and provide some generic guidance on how to select σ min and σ max .

Original: They also justify setting λ proportional to the noise variance at time t : λ t = σ 2 t .
Revised: They also justify setting λ proportional to the noise variance at time t : λ t = σ 2 t .

Original: Using this weighting yields
Revised: Using this weighting yields

Original: [Equation]
Revised: [Equation]

Original: Finally, instead of using t directly as input for S , we use σ t and train S with a continuum of noise scales (Chen et al., 2021).
Revised: Finally, instead of using t directly as input for S , we use σ t and train S with a continuum of noise scales (Chen et al., 2021).

Original: In addition, we need to use some conditioning information c as an indication of what to generate (in our case, a signal derived from the distorted speech ˜ x ).
Revised: In addition, we need to use some conditioning information c as an indication of what to generate (in our case, a signal derived from the distorted speech ˜ x ).

Original: With that, our training loss becomes
Revised: With that, our training loss becomes

Original: [Equation]
Revised: [Equation]

Original: This is the loss denoted as L SCORE in the main paper.
Revised: This is the loss denoted as L SCORE in the main paper.

Original: For sampling with the VE schedule, we resort to consistent annealed sampling (Jolicoeur-Martineau et al., 2021).
Revised: For sampling with the VE schedule, we resort to consistent annealed sampling (Jolicoeur-Martineau et al., 2021).

Original: We use the discretization of t ∈ [0 , 1] into N uniform steps t n = ( n − 1) /
Revised: We use the discretization of t ∈ [0 , 1] into N uniform steps t n = ( n − 1) /

Original: ( N − 1) , n = { 1 , . . .
Revised: ( N − 1) , n = { 1 , . . .

Original: N } , which implies discretized progressions for x t and σ t .
Revised: N } , which implies discretized progressions for x t and σ t .

Original: Starting at n = N , we initialize x t N =
Revised: Starting at n = N , we initialize x t N =

Original: σ t N z t N , and then recursively compute
Revised: σ t N z t N , and then recursively compute

Original: [Equation]
Revised: [Equation]

Original: for n = N, N − 1 , . . .
Revised: for n = N, N − 1 , . . .

Original: Finally, at n = 1 ( t 1 = 0 ), we take
Revised: Finally, at n = 1 ( t 1 = 0 ), we take

Original: [Equation]
Revised: [Equation]

Original: which corresponds to the empirically denoised sample (Jolicoeur-Martineau et al., 2021)
Revised: which corresponds to the empirically denoised sample (Jolicoeur-Martineau et al., 2021)

Original: The values of η and β are determined through the parameterization (Serrà et al., 2021a)
Revised: The values of η and β are determined through the parameterization (Serrà et al., 2021a)

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ (0 , 1) is the ratio of the geometric progression of the noise, γ = σ t n /σ t n +1 , and the constant (cid:15) ∈ [1 , ∞ ) is left as a hyper-parameter.
Revised: where γ ∈ (0 , 1) is the ratio of the geometric progression of the noise, γ = σ t n /σ t n +1 , and the constant (cid:15) ∈ [1 , ∞ ) is left as a hyper-parameter.

Original: Note that the value of γ changes with the chosen number of discretization steps N , thus facilitating hyper-parameter search for continuous noise scales at different N (Serrà et al., 2021a).
Revised: Note that the value of γ changes with the chosen number of discretization steps N , thus facilitating hyper-parameter search for continuous noise scales at different N (Serrà et al., 2021a).

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: Architecture overview — The architecture contains two main parts that are jointly trained: the generator network, which is tasked with estimating the score of the perturbed speech distributions, and the conditioner network, which creates the conditioning signal required for synthesizing speech with preserved speaker characteristics.
Revised: Architecture overview — The architecture contains two main parts that are jointly trained: the generator network, which is tasked with estimating the score of the perturbed speech distributions, and the conditioner network, which creates the conditioning signal required for synthesizing speech with preserved speaker characteristics.

Original: Both the generator and the conditioner networks are essentially encoder-decoder architectures, enhanced with various conditioning signals and skip connections (see ﬁgure in main paper).
Revised: Both the generator and the conditioner networks are essentially encoder-decoder architectures, enhanced with various conditioning signals and skip connections (see ﬁgure in main paper).

Original: The encoding and decoding is done by down- and up-sampling, respectively, using convolutional blocks, which are the main building blocks of UNIVERSE.
Revised: The encoding and decoding is done by down- and up-sampling, respectively, using convolutional blocks, which are the main building blocks of UNIVERSE.

Original: Throughout the architecture, the multi-parametric rectiﬁed linear unit (PReLU) activation function is used prior to all the layers (except where we specify otherwise), and all convolutional layers are one dimensional.
Revised: Throughout the architecture, the multi-parametric rectiﬁed linear unit (PReLU) activation function is used prior to all the layers (except where we specify otherwise), and all convolutional layers are one dimensional.

Original: Unless stated otherwise, we use PyTorch’s defaults (Paszke et al., 2019, version 1.10.1).
Revised: Unless stated otherwise, we use PyTorch’s defaults (Paszke et al., 2019, version 1.10.1).

Original: Convolutional block — The convolutional block consists of a core part which is common between all blocks, and an optional prior or posterior part that can exist depending on the block’s functionality (Fig.
Revised: Convolutional block — The convolutional block consists of a core part which is common between all blocks, and an optional prior or posterior part that can exist depending on the block’s functionality (Fig.

Original: 4-B).
Revised: 4-B).

Original: The core part contains a convolutional layer of kernel width 5, followed by two convolutional layers of kernel width 3, and we use a residual sum between its input and output.
Revised: The core part contains a convolutional layer of kernel width 5, followed by two convolutional layers of kernel width 3, and we use a residual sum between its input and output.

Original: All residual sums and skip connections are weighted by 1 / √ r , where r is the number of elements in the sum.
Revised: All residual sums and skip connections are weighted by 1 / √ r , where r is the number of elements in the sum.

Original: If the convolutional block will be used for up-sampling, the core’s input is processed initially by a transposed convolutional layer of kernel width and stride equal to the up-sampling ratio, without padding (Pons et al., 2021).
Revised: If the convolutional block will be used for up-sampling, the core’s input is processed initially by a transposed convolutional layer of kernel width and stride equal to the up-sampling ratio, without padding (Pons et al., 2021).

Original: On the contrary, if the block will be used for down-sampling, the output of the core is processed by a strided convolutional layer of kernel width and stride equal to the down-sampling ratio, also without padding.
Revised: On the contrary, if the block will be used for down-sampling, the output of the core is processed by a strided convolutional layer of kernel width and stride equal to the down-sampling ratio, also without padding.

Original: Every down-sampling operation is accompanied by a channel expansion of factor 2, and up-sampling by a reduction of 2.
Revised: Every down-sampling operation is accompanied by a channel expansion of factor 2, and up-sampling by a reduction of 2.

Original: We start with 32 channels in the encoder and have 512 in the latent.
Revised: We start with 32 channels in the encoder and have 512 in the latent.

Original: Depending on a block’s position in the architecture we use skip connections and a maximum of two types of conditioning signals, all injected to a dedicated part in the convolutional block (Fig.
Revised: Depending on a block’s position in the architecture we use skip connections and a maximum of two types of conditioning signals, all injected to a dedicated part in the convolutional block (Fig.

Original: 4-B).
Revised: 4-B).

Original: If a signal will be extracted with a skip connection, we take it from the output of the residual sum, and insert it to the target block before the residual signal.
Revised: If a signal will be extracted with a skip connection, we take it from the output of the residual sum, and insert it to the target block before the residual signal.

Original: [Figure]
Revised: [Figure]

Original: In order to preserve speaker characteristics that are consistent throughout a frame, we use conditioning signals between the generator and the conditioner.
Revised: In order to preserve speaker characteristics that are consistent throughout a frame, we use conditioning signals between the generator and the conditioner.

Original: These conditioning signals are taken from the output of the ﬁrst convolutional layer of the core, and inserted to the same position in the target block’s core.
Revised: These conditioning signals are taken from the output of the ﬁrst convolutional layer of the core, and inserted to the same position in the target block’s core.

Original: Furthermore, a secondary conditioning signal containing information about the noise level is injected to all the convolutional blocks of the generator.
Revised: Furthermore, a secondary conditioning signal containing information about the noise level is injected to all the convolutional blocks of the generator.

Original: This conditioning is done with
Revised: This conditioning is done with

Original: FiLM (Perez et al., 2018).
Revised: FiLM (Perez et al., 2018).

Original: Conditioner network — The conditioner network contains a stem cell, a down-sampling encoder, a recursive middle part, and an up-sampling decoder.
Revised: Conditioner network — The conditioner network contains a stem cell, a down-sampling encoder, a recursive middle part, and an up-sampling decoder.

Original: Both the encoder and decoder are made up of convolutional blocks that perform only down-sampling and up-sampling, respectively.
Revised: Both the encoder and decoder are made up of convolutional blocks that perform only down-sampling and up-sampling, respectively.

Original: The distorted signal is down-sampled to a 100 Hz sampling rate, with the encoder following a {2,4,4,5} factor progression, recursively processed in the middle part, and up-sampled to the original sampling rate with the decoder, mirroring the encoder’s progression.
Revised: The distorted signal is down-sampled to a 100 Hz sampling rate, with the encoder following a {2,4,4,5} factor progression, recursively processed in the middle part, and up-sampled to the original sampling rate with the decoder, mirroring the encoder’s progression.

Original: At each encoder block, we use skip connections, process them with adaptor networks, and sum the adapted signals with the output of the last encoder block.
Revised: At each encoder block, we use skip connections, process them with adaptor networks, and sum the adapted signals with the output of the last encoder block.

Original: The adaptor networks consist of strided convolutions in order to match the sampling rate of the summation inputs.
Revised: The adaptor networks consist of strided convolutions in order to match the sampling rate of the summation inputs.

Original: Alongside this main path, we extract and normalize 80 mel band features from the distorted signal, process them with a convolutional layer followed by a convolutional block, and add the result to the summation of the down-sampled signals.
Revised: Alongside this main path, we extract and normalize 80 mel band features from the distorted signal, process them with a convolutional layer followed by a convolutional block, and add the result to the summation of the down-sampled signals.

Original: The hop size is equal to the total down-sampling rate, preserving the sampling rate of the features during processing.
Revised: The hop size is equal to the total down-sampling rate, preserving the sampling rate of the features during processing.

Original: We process the feature-enriched signal with the conditioner’s middle part, consisting of a convolutional block, a two-layer bi-directional
Revised: We process the feature-enriched signal with the conditioner’s middle part, consisting of a convolutional block, a two-layer bi-directional

Original: GRU with a residual sum, and a ﬁnal convolutional block, all of them preserving the sampling rate.
Revised: GRU with a residual sum, and a ﬁnal convolutional block, all of them preserving the sampling rate.

Original: We ﬁnally up-sample the latent representation to the original sampling rate with the decoder, using an initial convolutional block preserving the sampling rate and numerous blocks performing up-sampling.
Revised: We ﬁnally up-sample the latent representation to the original sampling rate with the decoder, using an initial convolutional block preserving the sampling rate and numerous blocks performing up-sampling.

Original: We obtain a hierarchical conditioning for the generator network by taking conditioning signals from each block of the decoder, corresponding to all the sampling rates.
Revised: We obtain a hierarchical conditioning for the generator network by taking conditioning signals from each block of the decoder, corresponding to all the sampling rates.

Original: In order to adapt these representations to the ones of the generator network, we process each conditioning signal with linear layers.
Revised: In order to adapt these representations to the ones of the generator network, we process each conditioning signal with linear layers.

Original: Generator network — The generator network is also an encoder-decoder architecture with UNetlike skip connections, where there is a direct connection between each encoder and decoder block pair sharing the same sampling rate.
Revised: Generator network — The generator network is also an encoder-decoder architecture with UNetlike skip connections, where there is a direct connection between each encoder and decoder block pair sharing the same sampling rate.

Original: We use the same down- and up-sampling progression as in the conditioner network, hence achieving parallel up-sampling blocks between the generator and conditioner decoders.
Revised: We use the same down- and up-sampling progression as in the conditioner network, hence achieving parallel up-sampling blocks between the generator and conditioner decoders.

Original: However, we limit the middle part to only a single bi-directional GRU layer without residual sums.
Revised: However, we limit the middle part to only a single bi-directional GRU layer without residual sums.

Original: The output of the ﬁnal block is processed by a convolutional layer to reduce its dimension back to the score’s dimension.
Revised: The output of the ﬁnal block is processed by a convolutional layer to reduce its dimension back to the score’s dimension.

Original: The generator is conditioned on two signals: Fourier features depending on the current noise level and the hierarchical conditioning extracted from the distorted speech.
Revised: The generator is conditioned on two signals: Fourier features depending on the current noise level and the hierarchical conditioning extracted from the distorted speech.

Original: For the former, we embed the noise level with logarithmic compression and use it to modulate frequencies of random Fourier features (RFFs) sampled from a standard normal distribution.
Revised: For the former, we embed the noise level with logarithmic compression and use it to modulate frequencies of random Fourier features (RFFs) sampled from a standard normal distribution.

Original: Following the extraction of RFFs, we process them with a fully connected network of 3 times repeated linear layers followed by PReLUs to obtain noise level embeddings.
Revised: Following the extraction of RFFs, we process them with a fully connected network of 3 times repeated linear layers followed by PReLUs to obtain noise level embeddings.

Original: This block, which we call SigmaBlock, expands 32 pairs of Fourier coefﬁcients into 256 channels.
Revised: This block, which we call SigmaBlock, expands 32 pairs of Fourier coefﬁcients into 256 channels.

Original: (Fig.
Revised: (Fig.

Original: 4-A).
Revised: 4-A).

Original: After the embeddings are extracted, they are adapted to each block’s frame rate and dimension with a linear projection layer.
Revised: After the embeddings are extracted, they are adapted to each block’s frame rate and dimension with a linear projection layer.

Original: The conditioning signals taken from the conditioner network’s decoder are injected to the generator network’s decoder at each block with the same sampling rate, using summation.
Revised: The conditioning signals taken from the conditioner network’s decoder are injected to the generator network’s decoder at each block with the same sampling rate, using summation.

Original: This way, the estimated score is conditioned with multiple sampling rates starting from the lowest (100 Hz), until the original (16 kHz).
Revised: This way, the estimated score is conditioned with multiple sampling rates starting from the lowest (100 Hz), until the original (16 kHz).

Original: Training objective — The main loss is the denoising score matching loss outlined in the main paper and derived in Appendix A.2.
Revised: Training objective — The main loss is the denoising score matching loss outlined in the main paper and derived in Appendix A.2.

Original: Additionally, we introduce auxiliary losses in order to structure the conditioner network’s latent representation and prime its ﬁnal block’s output to perform speech enhancement.
Revised: Additionally, we introduce auxiliary losses in order to structure the conditioner network’s latent representation and prime its ﬁnal block’s output to perform speech enhancement.

Original: We use separate mixture density networks (MDN; Bishop, 1994) to model the distributions of the clean signal waveform and features.
Revised: We use separate mixture density networks (MDN; Bishop, 1994) to model the distributions of the clean signal waveform and features.

Original: Each feature and its delta distribution is jointly estimated with an MDN that models the distribution using 3 multivariate Gaussian components with diagonal covariance.
Revised: Each feature and its delta distribution is jointly estimated with an MDN that models the distribution using 3 multivariate Gaussian components with diagonal covariance.

Original: The parameters of the mixture components are learned with convolutional layers of kernel width 3 and layer normalization (Fig.
Revised: The parameters of the mixture components are learned with convolutional layers of kernel width 3 and layer normalization (Fig.

Original: 4-C).
Revised: 4-C).

Original: We calculate the negative log-likelihood (NLL) of the feature MDNs and average them.
Revised: We calculate the negative log-likelihood (NLL) of the feature MDNs and average them.

Original: Then, the total objective is to minimize the sum of the denoising score matching loss, waveform NLL, and feature NLLs.
Revised: Then, the total objective is to minimize the sum of the denoising score matching loss, waveform NLL, and feature NLLs.

Original: Here, we would like to underline that auxiliary losses are generative, as they model continuous data distributions, making UNIVERSE a fully-generative model (that is, not using any classiﬁcation or regression losses).
Revised: Here, we would like to underline that auxiliary losses are generative, as they model continuous data distributions, making UNIVERSE a fully-generative model (that is, not using any classiﬁcation or regression losses).

Original: Moreover, as the auxiliary losses are taken out of the main signal path, we do not need any sampling procedure neither in training nor in validation stages.
Revised: Moreover, as the auxiliary losses are taken out of the main signal path, we do not need any sampling procedure neither in training nor in validation stages.

Original: Optimization — The resulting model contains about 49 M parameters.
Revised: Optimization — The resulting model contains about 49 M parameters.

Original: We train it with a batch size of 32, where each sample is an approximately two-seconds long frame with 16 kHz sampling rate, and using automatic mixed precision.
Revised: We train it with a batch size of 32, where each sample is an approximately two-seconds long frame with 16 kHz sampling rate, and using automatic mixed precision.

Original: We apply 1 M parameter updates, which takes approximately four and a half days using 2 Tesla V100 GPUs in parallel.
Revised: We apply 1 M parameter updates, which takes approximately four and a half days using 2 Tesla V100 GPUs in parallel.

Original: The large/ﬁnal model has 189 M parameters, is trained for 2.5 M iterations using a batch size of 64, and takes less than 14 days using 8 Tesla
Revised: The large/ﬁnal model has 189 M parameters, is trained for 2.5 M iterations using a batch size of 64, and takes less than 14 days using 8 Tesla

Original: V100 GPUs.
Revised: V100 GPUs.

Original: We use the Adam optimizer with a maximum learning rate of 2 · 10 − 4 and schedule the learning rate with a cosine scheduler.
Revised: We use the Adam optimizer with a maximum learning rate of 2 · 10 − 4 and schedule the learning rate with a cosine scheduler.

Original: The schedule includes a warm-up of 50 k iterations, increasing the learning rate from a starting point 1 . 6 · 10 − 6 .
Revised: The schedule includes a warm-up of 50 k iterations, increasing the learning rate from a starting point 1 . 6 · 10 − 6 .

Original: We additionally implement a manual weight decay, excluding biases and the PReLU weights, and set its coefﬁcient to 0.01.
Revised: We additionally implement a manual weight decay, excluding biases and the PReLU weights, and set its coefﬁcient to 0.01.

Original: C D ATA AND D ISTORTIONS
Revised: C D ATA AND D ISTORTIONS

Original: To create our training data, we sample chunks of 3.5–5.5 seconds from speech recordings of an internal pool of data sets, featuring multiple speakers, languages, prosody/emotions, and diverse recording conditions.
Revised: To create our training data, we sample chunks of 3.5–5.5 seconds from speech recordings of an internal pool of data sets, featuring multiple speakers, languages, prosody/emotions, and diverse recording conditions.

Original: Speech is pre-selected to be clean, but nonetheless can contain minimal background noise and reverb.
Revised: Speech is pre-selected to be clean, but nonetheless can contain minimal background noise and reverb.

Original: We down-sample to 16 kHz mono and follow two different signal paths to create input and target pairs.
Revised: We down-sample to 16 kHz mono and follow two different signal paths to create input and target pairs.

Original: With distortions that introduce delays we automatically time-align at the sample level using a custom correlation function.
Revised: With distortions that introduce delays we automatically time-align at the sample level using a custom correlation function.

Original: Input — To programmatically generate distorted versions, we randomly choose the number of distortions between {1, 2, 3, 4, 5}, with probabilities {0.35, 0.45, 0.15, 0.04, 0.01}, respectively.
Revised: Input — To programmatically generate distorted versions, we randomly choose the number of distortions between {1, 2, 3, 4, 5}, with probabilities {0.35, 0.45, 0.15, 0.04, 0.01}, respectively.

Original: Next, we cascade distortion types with random parameters (bounds for distortion parameters are selected such that distortions are perceptually noticeable by expert listeners; random selection is typically uniform, except for parameters that have an intuitive logarithmic behavior, such as frequency, in which case we sample uniformly in logarithmic space).
Revised: Next, we cascade distortion types with random parameters (bounds for distortion parameters are selected such that distortions are perceptually noticeable by expert listeners; random selection is typically uniform, except for parameters that have an intuitive logarithmic behavior, such as frequency, in which case we sample uniformly in logarithmic space).

Original: Table 3 summarizes the distortion families and types we consider, together with their weightsand some of their random parameters (distortion type weights will deﬁne the probability of selecting each distortion type).
Revised: Table 3 summarizes the distortion families and types we consider, together with their weights (distortion type weights will deﬁne the probability of selecting each distortion type).

Original: In total, we consider 55 distortions, which we group into 10 families.
Revised: In total, we consider 55 distortions, which we group into 10 families.

Revised: The ranges we consider for the distortion parameters include: SNRs between − 5 and 25 dB for all added noises, slopes between 0.01 and 0.7 for colored noise, DC components between 10 −and 10 − 1 , non-stationary noise lengths between 20 and 350 ms, frequency ranges between 100 and 7500 Hz for ﬁlters and tones (except low and high pass, with a minimum and maximum of 1 kHz, respectively),

Revised: Qs between 0.1 and 2 for ﬁlters, gains between − 12 and 6 dB for ﬁlters and random equalization, between 2 and 20 mel bands for random equalization, gap lengths between 20 and 80 ms, temporal probabilities between 0.5 and 3 times/sec for transmission distortions, percentiles between 0.5 and 50 for clipping, resample frequencies of 4, 6, 8, 11, 12, and 14 kHz, wet proportions between 0.2 and 1 for reverbs, augmentation of early and tail reﬂections for room impulse responses, room sizes between 3 and 1000 m 2 for algorithmic reverbs, bit rates between 2 and 96 kbps for codecs, compression ratios between 2 and 10, window lengths between 2 9 and 2 12 for spectral manipulations, amounts between 0.5 and 0.8 for spectral manipulation probability, wet proportions between 0.1 and 0.75 for other effects, and linear gains between 0 and 1 for plosive and sibilance addition.

Original: Target — Given that the design of UNIVERSE does not rely on assumptions regarding the nature of the distortions (for instance, a good example would be the assumption that noise is additive), nothing prevents us from using a different target than the original signal that was used as input to the distortion chain.
Revised: Target — Given that the design of UNIVERSE does not rely on assumptions regarding the nature of the distortions (for instance, a good example would be the assumption that noise is additive), nothing prevents us from using a different target than the original signal that was used as input to the distortion chain.

Original: In addition, we want to ensure that the generated signal is of the top/highest quality, which is a characteristic that is not shared by some of the original clean speech signals that we sample.
Revised: In addition, we want to ensure that the generated signal is of the top/highest quality, which is a characteristic that is not shared by some of the original clean speech signals that we sample.

Original: Therefore, we decide to apply a small enhancement chain to clean the signals to be used as target.
Revised: Therefore, we decide to apply a small enhancement chain to clean the signals to be used as target.

Original: In particular, we apply four consecutive steps 2 :
Revised: In particular, we apply four consecutive steps 2 :

Original: Denoising — We employ a denoiser to remove any (minimal) amount of noise that might be present in the recording.
Revised: Denoising — We employ a denoiser to remove any (minimal) amount of noise that might be present in the recording.

Original: Since the pre-selected speech is already of good quality, we expect thatany decent denoiser will have no problems and will not introduce any noticeable distortion. We carefully veriﬁed that by listening to several examples. Deplosive — We employ a deplosive algorithm, which is composed of detection and processing stages. The processing acts only on the level of the plosive bands.
Revised: Since the pre-selected speech is already of good quality, we expect that

Original: Deesser — We employ a basic deesser algorithm, which is composed of detection and processing stages.

Original: The processing acts only on the level of the sibilant bands.

Original: Dynamic EQ — We also employ a signal processing tool that aims at bringing the speech to a predeﬁned equalization target.

Original: The target is level-dependent, so that the algorithm can act as a compressor and/or expander depending on the characteristics of the input speech.

Original: Interestingly, passing the original clean recordings through this chain forces the model to not rely on bypassing input characteristics (especially low-level characteristics), which we ﬁnd particularly

Original: [Table]
Revised: [Table]

Original: characteristics. Chorus [Table] , chorus characteristics.
Revised: Chorus 1 Gain, chorus characteristics.

Original: Phaser 1 Gain, phaser characteristics.
Revised: Phaser 1 Gain, phaser characteristics.

Original: RIR convolution 120 Gain, room impulse response (RIR), augment.
Revised: RIR convolution 120 Gain, room impulse response (RIR), augment.

Original: Very short delay 3 Gain, delay time.
Revised: Very short delay 3 Gain, delay time.

Original: 35 Spectral manipulation Convolved spectrogram 1 Window, amount.
Revised: 35 Spectral manipulation Convolved spectrogram 1 Window, amount.

Original: Grifﬁn-Lim 3 Window, amount.
Revised: Grifﬁn-Lim 3 Window, amount.

Original: Phase randomization 1 Window, amount.
Revised: Phase randomization 1 Window, amount.

Original: Phase shufﬂe 1 Window, amount.
Revised: Phase shufﬂe 1 Window, amount.

Original: Spectral holes 1 Window, amount.
Revised: Spectral holes 1 Window, amount.

Original: Spectral noise 1 Window, amount.
Revised: Spectral noise 1 Window, amount.

Original: 41 Synthetic noise
Revised: 41 Synthetic noise

Original: Colored noise 15 SNR, slope.
Revised: Colored noise 15 SNR, slope.

Original: DC component 1 Amplitude.
Revised: DC component 1 Amplitude.

Original: Electricity tone 6 SNR, frequency, type of waveform.
Revised: Electricity tone 6 SNR, frequency, type of waveform.

Original: Non-stationary colored noise 5 SNR, slope, duration, temporal probability.
Revised: Non-stationary colored noise 5 SNR, slope, duration, temporal probability.

Original: Non-stationary DC component 1 Amplitude, duration, temporal probability.
Revised: Non-stationary DC component 1 Amplitude, duration, temporal probability.

Original: Non-stationary electricity tone 3 SNR, frequency, duration, temporal prob.
Revised: Non-stationary electricity tone 3 SNR, frequency, duration, temporal prob.

Original: Non-stationary random tone 1 SNR, frequency, duration, temporal prob.
Revised: Non-stationary random tone 1 SNR, frequency, duration, temporal prob.

Original: Random tone 2 SNR, frequency, type of waveform.
Revised: Random tone 2 SNR, frequency, type of waveform.

Original: 49 Transmission
Revised: 49 Transmission

Original: Frame shufﬂe 10 Length, temporal probability.
Revised: Frame shufﬂe 10 Length, temporal probability.

Original: Insert attenuation 3 Length, temporal probability, gain.
Revised: Insert attenuation 3 Length, temporal probability, gain.

Original: Insert noise 5 Length, temporal probability, SNR.
Revised: Insert noise 5 Length, temporal probability, SNR.

Original: Perturb amplitude 1 Length, temporal probability, gain.
Revised: Perturb amplitude 1 Length, temporal probability, gain.

Original: Sample duplicate 2 Length, temporal probability.
Revised: Sample duplicate 2 Length, temporal probability.

Original: Silent gap (packet loss) 15 Length, temporal probability.
Revised: Silent gap (packet loss) 15 Length, temporal probability.

Original: Telephonic speech 10 Frequencies, compression ratio, ﬁlter type.
Revised: Telephonic speech 10 Frequencies, compression ratio, ﬁlter type.

Revised: any decent denoiser will have no problems and will not introduce any noticeable distortion.

Revised: We carefully veriﬁed that by listening to several examples.

Revised: Deplosive — We employ a deplosive algorithm, which is composed of detection and processing stages.

Revised: The processing acts only on the level of the plosive bands.

Revised: Deesser — We employ a basic deesser algorithm, which is composed of detection and processing stages.

Original:  relevant to remove other non-desirable input characteristics.
Revised: The processing acts only on the level of the sibilant bands. Dynamic EQ — We also employ a signal processing tool that aims at bringing the speech to a predeﬁned equalization target. The target is level-dependent, so that the algorithm can act as a compressor and/or expander depending on the characteristics of the input speech. Interestingly, passing the original clean recordings through this chain forces the model to not rely on bypassing input characteristics (especially low-level characteristics), which we ﬁnd particularly relevant to remove other non-desirable input characteristics.

Original: In addition, the chain provides a more homogeneous character to the target speech, which should translate into some characteristic ‘imprint’ or ‘signature sound’ of UNIVERSE.
Revised: In addition, the chain provides a more homogeneous character to the target speech, which should translate into some characteristic ‘imprint’ or ‘signature sound’ of UNIVERSE.

Original: D E VALUATION D. O BJECTIVE M EASURES
Revised: D E VALUATION D. O BJECTIVE M EASURES

Original: We report results with COVL (Loizou, 2013) and STOI (Taal et al., 2010) as computed by the pysepmpackage using default parameters (version June 30, 2021).
Revised: We report results with COVL (Loizou, 2013) and STOI (Taal et al., 2010) as computed by the pysepmpackage using default parameters (version June 30, 2021).

Original: WARP-Q (Jassim et al., 2021) is used also with default parameters 4 (version March 14, 2021) and SESQA is a reference-based version of the reference-free speech quality measure presented in Serrà et al.
Revised: WARP-Q (Jassim et al., 2021) is used also with default parameters 4 (version March 14, 2021) and SESQA is a reference-based version of the reference-free speech quality measure presented in Serrà et al.

Original: (2021b), trained on comparable data and losses as explained by the authors.
Revised: (2021b), trained on comparable data and losses as explained by the authors.

Original: These four objective measures are only used to aid in the assessment of the development steps presented in the main paper.
Revised: These four objective measures are only used to aid in the assessment of the development steps presented in the main paper.

Original: PESQ (Rix et al., 2001), COVL, and STOI are also used to compare with the state of the art in the task of speech denoising in Tablebelow.
Revised: PESQ (Rix et al., 2001), COVL, and STOI are also used to compare with the state of the art in the task of speech denoising in Tablebelow.

Original: Calculation of PESQ is also done using pysepm 5 .
Revised: Calculation of PESQ is also done using pysepm 5 .

Original: D. S UBJECTIVE T EST
Revised: D. S UBJECTIVE T EST

Original: We perform a subjective test considering 15 competitor approaches (12 existing ones plus 3 ablations of the proposed approach, as reported in the main paper).
Revised: We perform a subjective test considering 15 competitor approaches (12 existing ones plus 3 ablations of the proposed approach, as reported in the main paper).

Original: For each existing approach, we download distorted and enhanced pairs of speech recordings from the corresponding demo websites, and use those as the main materials for the test.
Revised: For each existing approach, we download distorted and enhanced pairs of speech recordings from the corresponding demo websites, and use those as the main materials for the test.

Original: We randomly select at least 25 pairs per system, and remove a few ones for which we consider the distorted signal is too simple/easy to enhance (for instance, signals that have a very high SNR when evaluating a denoising system, almost no clipping when evaluating a de-cplipping system, and so forth).
Revised: We randomly select at least 25 pairs per system, and remove a few ones for which we consider the distorted signal is too simple/easy to enhance (for instance, signals that have a very high SNR when evaluating a denoising system, almost no clipping when evaluating a de-cplipping system, and so forth).

Original: This is done to reduce potentially ambivalent results in the test, as the two systems would presumably perform very well and the listener would be confused on which to choose (removed pairs were never more than 5 per approach).
Revised: This is done to reduce potentially ambivalent results in the test, as the two systems would presumably perform very well and the listener would be confused on which to choose (removed pairs were never more than 5 per approach).

Original: For a few systems that do not have enough material on their demo page, we download the code and enhance a random selection of distorted input signals from other systems.
Revised: For a few systems that do not have enough material on their demo page, we download the code and enhance a random selection of distorted input signals from other systems.

Original: That is the case of Demucs, MetricGAN+, and VoiceFixer.
Revised: That is the case of Demucs, MetricGAN+, and VoiceFixer.

Original: For the ablations of the proposed approach, we take the ﬁrst (random) 25 pairs of our validation set.
Revised: For the ablations of the proposed approach, we take the ﬁrst (random) 25 pairs of our validation set.

Original: In total, we have over (12 + 3) × 25 enhanced signals from which to judge preference.
Revised: In total, we have over (12 + 3) × 25 enhanced signals from which to judge preference.

Original: All distorted signals are enhanced by UNIVERSE using N = 64 diffusion steps and hyper-parameter (cid:15) = 2 . 3 .
Revised: All distorted signals are enhanced by UNIVERSE using N = 64 diffusion steps and hyper-parameter (cid:15) = 2 . 3 .

Original: A total of 22 expert listeners voluntarily participated in the test.
Revised: A total of 22 expert listeners voluntarily participated in the test.

Original: The test features a set of instructions and then shows a list of triplets from which to perform judgment (Fig. 5).
Revised: The test features a set of instructions and then shows a list of triplets from which to perform judgment (Fig. 5).

Original: Every time a listener lands on the test page, triplets of signals are uniformly randomly selected for each approach: input-distorted signal (Input), competitor-enhanced signal (A or B), and UNIVERSE-enhanced signal (A or B).
Revised: Every time a listener lands on the test page, triplets of signals are uniformly randomly selected for each approach: input-distorted signal (Input), competitor-enhanced signal (A or B), and UNIVERSE-enhanced signal (A or B).

Original: The order of A or B, as well as the order of the triplets is also uniformly randomly selected when landing to the web page.
Revised: The order of A or B, as well as the order of the triplets is also uniformly randomly selected when landing to the web page.

Original: Preference for an enhanced signal can only be A or B (forced-choice test).
Revised: Preference for an enhanced signal can only be A or B (forced-choice test).

Original: Every subject listens to two triplets per competitor system, performing a total of 30 preference choices for 15 approaches (two per approach, yielding a total of 22 × 2 preference judgments per system).
Revised: Every subject listens to two triplets per competitor system, performing a total of 30 preference choices for 15 approaches (two per approach, yielding a total of 22 × 2 preference judgments per system).

Original: The results of the test are based on counting preference choices per competitor approach (% of preference).
Revised: The results of the test are based on counting preference choices per competitor approach (% of preference).

Original: Statistical signiﬁcance is determined with a binomial test using p < 0 .
Revised: Statistical signiﬁcance is determined with a binomial test using p < 0 .

Original: 05 and the
Revised: 05 and the

Original: Holm-Bonferroni adjustment to compensate for multiple comparisons.
Revised: Holm-Bonferroni adjustment to compensate for multiple comparisons.

Original: In the results table, we also display which distortions were tackled by the competitor approach.
Revised: In the results table, we also display which distortions were tackled by the competitor approach.

Original: Note that, since testing materials have different distortions and contents per approach, one cannot infer a ranking of systems based on preference % (that is, one cannot compare preferences between rows of the table; actually they even correspond to different tasks: denoising, de-clipping, restoring codec artifacts, etc.).
Revised: Note that, since testing materials have different distortions and contents per approach, one cannot infer a ranking of systems based on preference % (that is, one cannot compare preferences between rows of the table; actually they even correspond to different tasks: denoising, de-clipping, restoring codec artifacts, etc.).

Original: Preference % only represents a pairwise comparison between UNIVERSE and the corresponding competitor approach listed on the left of the table.
Revised: Preference % only represents a pairwise comparison between UNIVERSE and the corresponding competitor approach listed on the left of the table.

Original: E A DDITIONAL R ESULTS E. D ENOISING T ASK W ITH O BJECTIVE M ETRICS
Revised: E A DDITIONAL R ESULTS E. D ENOISING T ASK W ITH O BJECTIVE M ETRICS

Original: To have a comparison on a well-established benchmark, we can consider the task of speech denoising, which has a long tradition in the speech community and, in the last years, has seen a consolidation of test data sets and objective metrics (this is something that, to the best of our knowledge, has not yet happened with other tasks like dereverberation, declipping, bandwidth extension, and so on).
Revised: To have a comparison on a well-established benchmark, we can consider the task of speech denoising, which has a long tradition in the speech community and, in the last years, has seen a consolidation of test data sets and objective metrics (this is something that, to the best of our knowledge, has not yet happened with other tasks like dereverberation, declipping, bandwidth extension, and so on).

Original: Two widely-used data sets with an established test partition are Voicebank-DEMAND (Valentini-Botinhao, 2017) and IS20 DNS-Challenge (Reddy et al., 2020).
Revised: Two widely-used data sets with an established test partition are Voicebank-DEMAND (Valentini-Botinhao, 2017) and IS20 DNS-Challenge (Reddy et al., 2020).

Original: Since the standard objective metrics are
Revised: Since the standard objective metrics are

Original: PESQ, COVL, and STOI, and given that UNIVERSE produces audio with further enhancements like leveling/dynamics or a speciﬁc equalization, we need to train a new version of UNIVERSE speciﬁcally for the denoising task.
Revised: PESQ, COVL, and STOI, and given that UNIVERSE produces audio with further enhancements like leveling/dynamics or a speciﬁc equalization, we need to train a new version of UNIVERSE speciﬁcally for the denoising task.

Original: This is important because, otherwise, UNIVERSE would be performing additional enhancements besides denoising and, more importantly, the standard objective metrics would not ﬁnd an agreement between the ground truth and the estimated clean speech, what would result in the whole evaluation being unfair to UNIVERSE.
Revised: This is important because, otherwise, UNIVERSE would be performing additional enhancements besides denoising and, more importantly, the standard objective metrics would not ﬁnd an agreement between the ground truth and the estimated clean speech, what would result in the whole evaluation being unfair to UNIVERSE.

Original:  In addition, because these standard metrics are known to disfavor generative approaches due to lack of waveform alignment or minor spectral nuances (see Jassim et al., 2021, for a discussion and further pointers), we need to devise a method to produce outputs that are ‘less generative’ and can therefore better coincide with what standard metrics measure.
Revised: [Table] In addition, because these standard metrics are known to disfavor generative approaches due to lack of waveform alignment or minor spectral nuances (see Jassim et al., 2021, for a discussion and further pointers) [Table] we need to devise a method to produce outputs that are ‘less generative’ and can therefore better coincide with what standard metrics measure.

Original: Inspired by Fejgin et al.
Revised: Inspired by Fejgin et al.

Original: (2020), we decide to use an expectation of the enhanced waveform.
Revised: (2020), we decide to use an expectation of the enhanced waveform.

Original: Intuitively, that expectation should minimize to a certain extent different nuances introduced by the generative model, especially regarding small misalignments and minor spectral nuances.
Revised: Intuitively, that expectation should minimize to a certain extent different nuances introduced by the generative model, especially regarding small misalignments and minor spectral nuances.

Original: To compute such expectation, we sample 10 times using the same conditioning (distorted) signal, and take the sample average of the resulting waveforms.
Revised: To compute such expectation, we sample 10 times using the same conditioning (distorted) signal [Table] and take the sample average of the resulting waveforms.

Original: While this has some audible effect such as lowering the presence of high frequencies, it clearly shows a boost in the standard objective metrics, probably because they focus more on small misalignments and low frequencies.
Revised: While this has some audible effect such as lowering the presence of high frequencies, it clearly shows a boost in the standard objective metrics, probably because they focus more on small misalignments and low frequencies.

Original: We use E to denote this version of the approach.
Revised: We use E to denote this version of the approach.

Original: We also want to stress that we solely use this expectation version for the result in the corresponding row of Table4.
Revised: We also want to stress that we solely use this expectation version for the result in the corresponding row of Table

Original: [Table]
Revised: [Table]

Original: Table 4 shows the results for the denoising task.
Revised: Table 4 shows the results for the denoising task.

Original: In it, we can observe the difference in standard metrics between generative and regression/adversarial approaches (ﬁrst two blocks of the table).
Revised: In it, we can observe the difference in standard metrics between generative and regression/adversarial approaches (ﬁrst two blocks of the table).

Original: The best-performing approaches in the generative block struggle to get the numbers of the worseperforming approaches in the regression/adversarial block (by listening to some examples from both blocks, we believe this is a metrics issue since we could not ﬁnd a clear perceptual difference).
Revised: The bestperforming approaches in the generative block struggle to get the numbers of the worse-performing approaches in the regression/adversarial block (by listening to some examples from both blocks, we believe this is a metrics issue since we could not ﬁnd a clear perceptual difference).

Original: Interestingly, we observe that even the generative version of the proposed approach (UNIVERSEDenoise) clearly surpasses all existing generative approaches in terms of standard metrics.
Revised: Interestingly, we observe that even the generative version of the proposed approach (UNIVERSE-Denoise) clearly surpasses all existing generative approaches in terms of standard metrics.

Original: In addition, the ‘less generative’ variant using an expectation over multiple realizations (UNIVERSE-Denoise- E ) shows a clear improvement for those metrics, with values that become state of the art for the two data sets and with all metrics.
Revised: In addition, the ‘less generative’ variant using an expectation over multiple realizations (UNIVERSE-Denoise- E ) shows a clear improvement for those metrics, with values that become competitive with the non-generative state-of-the-art.

Original: The only exception is with the COVL metric on the Voicebank-DEMAND data set, where nonetheless UNIVERSE is still competitive among the top-performing systems.

Original: The results of our subjective test, which include some of the competing approaches of Table 4, further conﬁrm the superiority of UNIVERSE  in a more realistic scenario.
Revised: The results of our subjective test, which include some of the competing approaches in Table 4, suggest that UNIVERSE outperforms those in such more realistic evaluation.

Original: E. S PEED - QUALITY T RADE - OFF
Revised: E. S PEED - QUALITY T RADE - OFF

Original: For completeness, we provide the speed-quality plots for every considered objective metric in Fig. 6.
Revised: For completeness, we provide the speed-quality plots for every considered objective metric in Fig. 6.

Original: As in the main paper, synthesis parameters are the number of denoising iterations N and the hyperparameter (cid:15) (Sec. A.2).
Revised: As in the main paper, synthesis parameters are the number of denoising iterations N and the hyperparameter (cid:15) (Sec. A.2).

Original: The real-time factor (RTF) is deﬁned as the time to process a recording usinga single Tesla V100 GPU divided by the duration of that recording (for instance, if UNIVERSE takes 2 seconds for enhancing a 20-second recording, RTF=0.1 and we say it is 10 times faster than real time).
Revised: The real-time factor (RTF) is deﬁned as the time to process a recording using

Original: [Figure]
Revised: [Figure]

Original: {+a single Tesla V100 GPU divided by the duration of that recording (for instance, if UNIVERSE takes 2 seconds for enhancing a 20-second recording, RTF=0.
+}
Revised: {+a single Tesla V100 GPU divided by the duration of that recording (for instance, if UNIVERSE takes 2 seconds for enhancing a 20-second recording, RTF=0.
+}

Revised: and we say it is 10 times faster than real time).