Improved Speech Enhancement with the Wave-U-Net Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study the use of the Wave-U-Net architecture for speech enhancement, a modelintroduced by Stoller et al for the separation of music vocals and accompaniment.
This end-to-end learning method for audio source separation operates directly inthe time domain, permitting the integrated modelling of phase information andbeing able to take large temporal contexts into account.
Our experiments showthat the proposed method improves several metrics, namely PESQ, CSIG, CBAK,
COVL and SSNR, over the state-of-the-art with respect to the speech enhancementtask on the Voice Bank corpus (VCTK) dataset.
We ﬁnd that a reduced number ofhidden layers is sufﬁcient for speech enhancement in comparison to the originalsystem designed for singing voice separation in music.
We see this initial result asan encouraging signal to further explore speech enhancement in the time-domain,both as an end in itself and as a pre-processing step to speech recognition systems.
Introduction
Audio source separation refers to the problem of extracting one or more target sources while suppressing interfering sources and noise[18].
Two related tasks are those of speech enhancement andsinging voice separation, both of which involve extracting the human voice as a target source.
Theformer involves attempting to improve speech intelligibility and quality when obscured by additivenoise [7, 9, 18]; whilst the latter’s focus is on separating music vocals from accompaniment [12].
Most audio source separation methods operate not directly in the time-domain, but with timefrequency representations as input and output (front-end).
Since 2017, the U-Net architecture onmagnitude spectrograms has achieved new state of the art results in audio source separation formusic [6] and speech dereverbration [2].
Also, neural network architectures operating in the timedomain have recently been proposed for speech enhancement [9, 11].
These approaches have beencombined in the Wave-U-Net [12] and applied to singing voice separation.
In this paper we apply the
Wave-U-Net to speech enhancement and show that it produces results that are better than the currentstate of the art.
The remainder of this paper is structured as follows.
In section 2, we brieﬂy review related workfrom the literature.
In section 3, we introduce brieﬂy the Wave-U-Net architecture and its applicationto speech.
Section 4 presents the experiments we conducted and their results including comparisonto other methods.
Section 5 concludes this article with a ﬁnal summary and perspectives for futurework.
Related work
Source separation of audio has seen great improvement in recent years through deep learning models[8, 5].
These methods, as well as more traditional ones, mostly operate in the time-frequency
Clean speech
Noise
[Figure]
domain [6], from deep recurrent architectures predicting soft masks, such as [4], to convolutionalencoder-decoder architectures like that of [1],
More recently, models operating in the time domain have been developed.
The development of
Wavenet [17] inspired other developments, including [9, 11].
The SEGAN [9] architecture was developed for the purpose of speech enhancement and denoising.
Itemploys a neural network in the time-domain with an encoder and decoder pathway that successivelyhalves and doubles the resolution of feature maps in each layer, respectively, and features skipconnections between encoder and decoder layers.
It offers state-of-the-art results on the Voice Bank(VCTK) dataset ([14]).
The Wavenet for Speech Denoising [11], another architecture to operate directly in the time domain,takes its inspiration from [17].
It has a non-causal conditional input and a parallel output of samplesfor each prediction and is based on the repeated application of dilated convolutions with exponentiallyincreasing dilation factors to factor in context information.
The Wave-U-Net architecture we use here was combines elements of both of the abovementionedarchitectures with the U-Net of [6].
Wave-U-Net for Speech Enhancement
In applying the Wave-U-Net architecture of [12] to the application of speech enhancement, ourobjectives are similar - namely to separate a mixture waveform m ∈ [ − 1 , 1] L m × C into K sourcewaveforms S 1 , ..., S K with S k ∈ [1 , 1] L S × C for all k ∈ 1 , ..., K, C as the number of audio channelsand Lm and Ls as the respective numbers of audio samples.
As per the spectrogram-based U-Net architectures (e.g. [6], the network uses a series of downsamplingand upsampling blocks to make its predictions, whilst at each level of the network, the time resolutionis halved.
At the ﬁnal level, as described by [12], in estimating the two target sources, a 1Dconvolution prepares the features at each audio sample for source prediction of each sample.
To yieldan estimate of the target sources, a tanh nonlinearity follows, succeeded by a ﬁnal LeakyReLU.
[Table]
Experiments4.1 Datasets
We use the same VCTK dataset [14] as the SEGAN [9], which is availability as a public dataset,encouraging comparisons with future speech enhancement methods.
The dataset includes clean and noisy audio data at 48kHz sampling frequency.
The clean data arerecordings of sentences, sourced from various text passages, uttered by 30 English-speakers, maleand female, with various accents – 28 intended for training and 2 reserved for testing [16].
The noisydata were generated by mixing the clean data with various noise datasets, as per the instructionsprovided in [9, 14, 15].
With respect to the training set, 40 different noise conditions are considered [9, 16].
These arecomposed of 10 types of noise (2 of which are artiﬁcially-generated 1 and 8 sourced from the
DEMAND database [13], each mixed with clean speech at one of 4 signal-to-noise ratios (SNR)(15, 10, 5, and 0 dB).
In total, this yields 11 , 572 training samples, with approximately 10 differentsentences in each condition per training speaker.
The separate test set with 2 speakers consists of a total of 20 different noise conditions: 5 types ofnoise sourced from the DEMAND database at one of 4 SNRs each (17.5, 12.5, 7.5, and 2.5 dB)[14, 15].
This yields 824 test items, with approximately 20 different sentences in each condition pertest speaker [14, 15].
4.2 Experimental setup
As per [12], our baseline model trains on randomly-sampled audio excerpts, using the ADAMoptimization algorithm, a learning rate of 0.0001, decay rates β 1 = 0.9 and β 2 = 0.999 and a batchsize of 16.
We specify the number of time frames in the input and output waveforms per sample as16384, a network layer size of 12 with 16 extra ﬁlters per layer, downsampling block ﬁlters of size 15and upsampling block ﬁlters of size 5 like in [12].
We train for 2,000 iterations with mean squarederror (MSE) over all source output samples in a batch as loss and apply early stopping if there isno improvement on the validation set for 20 epochs.
We use a ﬁxed validation set of 10 randomlyselected tracks.
Then, the best model is ﬁne-tuned with the batch size doubled and the learning ratelowered to 0.00001, again until 20 epochs have passed without improved validation loss.
4.3 Results
To evaluate and compare the quality of the enhanced speech yielded by the Wave-U-Net, we mirrorthe objective measures provided in [9].
Each measurement compares the enhanced signal with theclean reference of each of the 824 test set items.
They have been calculated using the implementationprovided in [7] 2 .
The ﬁrst metric is that of the Perceptual Evaluation of Speech Quality (PESQ) -more
speciﬁcally the wide-band version recommended in ITU-T P.862.2 (from –0.5 to 4.5) [7, 9].
Secondly, composite measures of metrics that aim to computationally approximate the Mean Opinion
Score (MOS) that would be produced from human perceptual trials are computed [11].
These are:
CSIG, a prediction of the signal distortion attending only to the speech signal [3] (from 1 to 5); CBAK:
K
[Table]
a prediction of the intrusiveness of background noise [3] from 1 to 5); and , a prediction the is the
Signal-to-Noise Ratio
Table 1 shows the results of these metrics for comparison across different speech enhancementarchitectures.
As a comparative, reference, it also shows: the results of these metrics when applieddirectly to the noisy signals; to signals ﬁltered using the Wiener method, based on a priori SNRestimation; and to the SEGAN-enhanced signal, as provided in [9].
The results indicate that the
Wave-U-Net is the most effective model for speech enhancement. Table 2 shows the performance differences between different variations of the Wave-U-Net, withdifferent numbers of layers. In this experiment no ﬁne-tuning was performed, which explains thedifference between the 12-layer Wave-U-Nets in Table 1 and in Table 2.
The results suggest thatﬁne-tuning does not make a meaningful difference, except on the CBAK measurement, and thatsmaller models perform better.
This is likely due to the size of the receptive ﬁeld, where for speechthe optimal size is probably smaller than for music.
Conclusions
5.1 Summary
The Wave-U-Net combines the advantages of several of the most recent successful architectures formusic and speech source separation and our results show that it is particularly effective at speechenhancement.
The results improve over the state of the art by a good margin even without signiﬁcantadaptation or parameter tuning.
This indicates that there is great potential for this approach in speechenhancement.
5.2 Future work
In comparison to the SEGAN architecture it is possible that the advantage stems from the upsamplingthat avoids aliasing, which should be further investigated.
The results indicate that there is room forincreasing effectiveness and efﬁciency by further adapting the model size and parameeter to the task.