Multi-Domain Processing via Hybrid Denoising Networks for Speech Enhancement
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We present a hybrid framework that leverages the trade-off between temporal andfrequency precision in audio representations to improve the performance of speechenhancement task.
We ﬁrst show that conventional approaches using speciﬁcrepresentations such as raw-audio and spectrograms are each effective at targetingdifferent types of noise.
By integrating both approaches, our model can learn multiscale and multi-domain features, effectively removing noise existing on differentregions on the time-frequency space in a complementary way.
Experimental resultsshow that the proposed hybrid model yields better performance and robustnessthan using each model individually.
Introduction
The trade-off between temporal and frequency resolution is a well-known phenomenon in the signalprocessing community, e.g., the window size in discrete Fourier transformation (DFT) [1].
The largerthe time segment, the more frequencies are extracted, thus giving us higher frequency resolutionin the expense of temporal resolution.
Therefore, it is obvious that time-series and time-frequencyrepresentations can provide complementary views when investigating a given signal.
To the best ofour knowledge, however, existing deep learning-based approaches proposed for speech enhancementhave only taken either time-series (i.e., raw-audio) [6, 7] or time-frequency representation (i.e.,spectrogram) as an input [4, 10, 14].
In this work, we ﬁnd that models using different audiorepresentations each specialize at tackling speciﬁc types of noise, and are also complementary toeach other.
Grounding on this observation, we propose a hybrid framework which enables the modelto learn multi-scale and multi-domain features, dubbed multi-domain processing via hybrid denoisingnetworks (MDPhD).
We devise a sequential model integrating two modules of both representationsby employing auxiliary loss.
Experimental results and ablation studies show that the proposed modelcan effectively utilize complementary information of time and time-frequency domains.
Althoughour hybridizing strategy is rather straightforward, MDPhD shows better denoising performance thanother state of the art (SOTA) algorithms across a variety of noises under multiple measures.
Note thatthe hybrid framework is general and not restricted to the current speciﬁc model.
The performancecan be further improved by employing newly developed models from each domain, by equipping anew loss function, or by designing a better hybridizing strategy.
Our contributions are as follows: 1) We empirically show that the way a model performs denoisingdepends on its input representation.
We propose a hybrid framework that can exploit multi-scaleand multi-domain features.
To the best of our knowledge, this is the ﬁrst hybrid approach, effectivelyutilizing both time and time-frequency domain information.
The proposed hybrid model (MDPhD)outperforms SOTA algorithms in the speech enhancement task.
Model Description
We ﬁrst describe the objective function and the selected modules that have been reported to showcompetitive performance using either raw-audio [5] or spectrogram input [3].
Selected models areeach used later as components of our proposed hybrid model.
2.1 Objective function
We employ the energy-conserving loss function proposed in [7] which simultaneously considersspeech and noise signals.
Let the noisy input x consist of clean speech s and noise n .
The estimatedspeech by the model is referred to as ˆ s .
Then, our objective function is deﬁned as follows:
[Equation]
where ˆ n = x − ˆ s represents the estimated noise signal and (cid:107) · (cid:107) 1 denotes (cid:96) 1 norm.
2.2 Hybrid Model
We construct the time domain network based on TAS-Net [5] which employs one-dimensionaldilated convolution to handle long time sequences of raw-audio.
TAS-Net has shown competitivesample quality for speech source separation, which is a similar task to speech enhancement.
In ourexperiments, we used a reduced version of TAS-Net.
With a slight abuse of notation, we refer tothe network as "TAS-Net" for simplicity.
For the time-frequency (T-F) domain network, we employa U-net structure based on two-dimensional convolutions which has been widely used in varioussource separation tasks [3, 8].
The T-F domain network aims to learn an ideal binary mask (IBM) ofa noisy spectrogram input.
By multiplying the estimated mask to the noisy spectrogram, the modelcan remove the noise from the time-frequency space.
We hybridize both time and T-F domain networks in a cascaded way (Fig. 1).
To make bothnetworks contribute to the denoising task equally well, we devise our model with an auxiliary loss
L ( x, s, n, ˆ s i,mid ) at the intermediate conjunction, where ˆ s i,mid is the output of the former network.
Inaddition, to let both networks have access to the full data information that is not processed (denoised)by the other, we train the entire model by alternately switching the sequential order of each component.
For inference, we can either use a single path or average the results from both paths.
Here, we simplyaverage the output of the model, which showed the best performance.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
The ﬁnal objective of the hybrid model with auxiliary loss becomes
[Equation]
where θ denotes the network parameter.
Experiments
3.1 Data and Experimental Setup
Dataset We used the dataset [12] that has been used in the recent speech enhancement studies[6, 7].
The dataset was produced by synthesizing the clean speech of Voice Bank corpus [13] and the noise data of Diverse Environments Multichannel Acoustic Noise Database (DEMAND) [11].
Thetraining dataset consists of audio from 28 speakers, and the test dataset is composed of the recordingsfrom two speakers.
Each speaker’s data contains 400 sentences with four noise levels.
To deal withsignals without voices, we added noise-only data to the training dataset, which is a quarter of thetotal number.
In our experiments, all audio samples recorded at 48kHz were subsampled to 16kHz.
Experimental Setup During training and testing, we split speech waveforms with a sliding windowof approximately one second (16384 samples) every 500 ms (50% overlap).
To obtain the spectrograms, we used the short time Fourier transform (STFT) with 512 window size and 256 hop size.
The output spectrogram is converted back to the time domain using the inverse STFT.
For training,we used batch renormalization to cope with a small batch size of 16 and Adam optimizer with theinitial learning rate of 2e-4.
The learning rate was decayed by half every 100,000 iterations.
For moredetails, please refer to the supplementary material.
3.2 Experimental Results
Hybrid Framework Validation To show the complementary characteristic of the time and T-Fdomain networks, we additionally synthesized noisy signals consisting of speech signals from the testdataset and noises which are either babbles (DEMAND), high frequency sinusoidal noise of 1000 ∼5000 Hz, or both.
Note that the networks did not see any of these noises during the training phase.
As shown in ﬁgure 2, while the spectrogram approach (U-Net) successfully removes high frequencynoise that is prominent in the spectrogram, it suffers from dealing with babble noise which is hardlydistinguishable from the frequency components of speech signals.
On the other hand, the raw-audioapproach (TAS-Net) shows superior results on denoising babble noise, which were even better thanthat of U-Net with doubled parameter size (Table 1).
Note that, however, TAS-Net fails to removehigh frequency noise, which is supposedly hard to capture in the time domain (Fig. 2 red arrow).
Table 1 summarizes these observations along with ablation studies.
Our hybrid model (MDPhD)showed the best performance by combining the strength of each model.
While the other methods hada noticeable weak domain, MDPhD showed comparable performance across all noise types.
Notethat MDPhD showed the best performance when the noises are mixed, which is more practical in realworld applications.
When we only trained a single path of MDPhD, the model failed to fully utilizethe complementary information from both domains.
Interestingly, we found that the performance ofthe model tends to follow the characteristics of the network that comes ﬁrst in order.
For example,the U → D model shares the weakness of U-Net and vice versa.
We conjecture that this happens
[Figure]
[Table]
because the latter network cannot reconstruct the information that is already lost from the formernetwork.
In addition, we tested various objective functions and conﬁrmed that the complementarynature of the two approaches does not come from a speciﬁc choice of the objective function (see thesupplementary material).
3.3 Comparison with Other Methods
Using the test dataset, we compared our results to recent studies of speech enhancement ﬁeld.
Our model showed the best performance quantitatively and qualitatively among the others undervarious measures [2] (Table 2).
For the qualitative results, please refer to the web demo page(https://mdphdnet.github.io), where we have uploaded several denoised examples using the modelsintroduced in the table (except MMSE-GAN whose code is unavailable).
[Table]
Conclusion
We demonstrated that the conventional speech enhancement models have limitations due to usingspeciﬁc representations.
Based on this observation, we proposed a hybrid approach that exploitsmulti-domain features for speech enhancement, dubbed multi-domain processing via hybrid denoisingnetworks (MDPhD).
With respect to ﬁve metrics, MDPhD achieved the best performance comparedto the other concurrent models.
Because MDPhD is a general framework, future work may includedeveloping a more elegant way of hybridizing and extending this framework to other signal processingtasks, such as music and speech source separation.
A More Experimental Results
To show that the complementary nature of the time domain and T-F domain networks does notcome from a speciﬁc choice of the objective functions, we trained each network module withvarious objective functions.
Table 3 summarizes the results.
We found that the performance was notsigniﬁcantly different.
spectrogram
[Table]
B Model Architecture
In this section, we present the detailed conﬁguration of the models we used.
In the following ﬁgures,each block consists of a convolutional operation, normalization and an activation function.
Notethat, normalization is not used at the ﬁrst and the last layer of each model.
The operation (cid:12) meanselement-wise multiplication and the preceding layer of this operation uses sigmoid as an activationfunction.