Improved Speech Enhancement with the Wave-U-Net Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study the use of the Wave-U-Net architecture for speech enhancement, a modelintroduced by Stoller et al for the separation of music vocals and accompaniment.
This end-to-end learning method for audio source separation operates directly inthe time domain, permitting the integrated modelling of phase information andbeing able to take large temporal contexts into account.
Our experiments showthat the proposed method improves several metrics, namely PESQ, CSIG, CBAK,
COVL and SSNR, over the state-of-the-art with respect to the speech enhancementtask on the Voice Bank corpus (VCTK) dataset.
We ﬁnd that a reduced number ofhidden layers is sufﬁcient for speech enhancement in comparison to the originalsystem designed for singing voice separation in music.
We see this initial result asan encouraging signal to further explore speech enhancement in the time-domain,both as an end in itself and as a pre-processing step to speech recognition systems.
Introduction
Audio source separation refers to the problem of extracting one or more target sources while suppressing interfering sources and noise[18].
Two related tasks are those of speech enhancement andsinging voice separation, both of which involve extracting the human voice as a target source.
Theformer involves attempting to improve speech intelligibility and quality when obscured by additivenoise [7, 9, 18]; whilst the latter’s focus is on separating music vocals from accompaniment [12].
Most audio source separation methods operate not directly in the time-domain but with timefrequency representations as input and output (front-end).
Since 2017, the U-Net architecture onmagnitude spectrograms has achieved new state of the art results in audio source separation formusic [6] and speech dereverbration [2].
Also, neural network architectures operating in the timedomain have recently been proposed for speech enhancement [9, 11].
These approaches have beencombined in the Wave-U-Net [12] and applied to singing voice separation.
In this paper we apply the
Wave-U-Net to speech enhancement and show that it produces results that are better than the currentstate of the art.
The remainder of this paper is structured as follows.
In section 2, we brieﬂy review related workfrom the literature.
In section 3, we introduce brieﬂy the Wave-U-Net architecture and its applicationto speech.
Section 4 presents the experiments we conducted and their results including comparisonto other methods.
Section 5 concludes this article with a ﬁnal summary and perspectives for futurework.
Related work
Source separation of audio has seen great improvement in recent years through deep learning models[5, 8].
These methods, as well as more traditional ones, mostly operate in the time-frequency domain,from deep recurrent architectures predicting soft masks, such as [4], to convolutional encoder-decoder speech
Clean speech
Noise
[Figure]
architectures like that of [1]. Recently, the U-Net architecture on magnitude spectrograms hasachieved new state of the art results in audio source separation for music [6] and speech dereverbration[2].
Also recently, models operating in the time domain have been developed.
The development of
Wavenet [17] inspired other developments, including [9, 11].
The SEGAN [9] architecture wasdeveloped for the purpose of speech enhancement and denoising.
It employs a neural network inthe time-domain with an encoder and decoder pathway that successively halves and doubles theresolution of feature maps in each layer, respectively, and features skip connections between encoderand decoder layers.
It offers state-of-the-art results on the Voice Bank (VCTK) dataset ([14]).
The Wavenet for Speech Denoising [11], another architecture to operate directly in the time domain,takes its inspiration from [17].
It has a non-causal conditional input and a parallel output of samplesfor each prediction and is based on the repeated application of dilated convolutions with exponentiallyincreasing dilation factors to factor in context information.
Wave-U-Net for Speech Enhancement
The Wave-U-Net architecture of [12] combines elements of both of the abovementioned architectureswith the U-Net. The overall architecture is a one-dimensional U-Net with down and upsamplingblocks.
As per the spectrogram-based U-Net architectures (e.g. [6]), the Wave-U-Net uses a series ofdownsampling and upsampling blocks to make its predictions, whilst at each level of the network, thetime resolution is halved.
At the ﬁnal level, as described by [12], in estimating the two target sources,a 1D convolution prepares the features at each audio sample for source prediction of each sample.
Toyield an estimate of the target sources, a tanh nonlinearity follows, succeeded by a ﬁnal LeakyReLU.
In applying the Wave-U-Net architecture of to the application of speech enhancement, our objectiveis to separate a mixture waveform m ∈ [ − 1 , 1] L m × C into K source waveforms S 1 , ..., S K with
S k ∈ [1 , 1] L S × C for all k ∈ 1 , ..., K, C as the number of audio channels and L m and L S as therespective numbers of audio samples.
In our case of monaural speech enhancement we have K = 2and C = 1 .
[Table]
[Table]
Experiments4.1 Datasets
We use the same VCTK dataset [14] as the SEGAN [9], which is available publicly, encouragingcomparisons with future speech enhancement methods.
The dataset includes clean and noisy audio data at 48kHz sampling frequency.
However, like the
SEGAN, we downsample to 16kHz for training and testing.
The clean data are recordings of sentences,sourced from various text passages, uttered by 30 English-speakers, male and female, with variousaccents – 28 intended for training and 2 reserved for testing [16].
The noisy data were generated bymixing the clean data with various noise datasets, as per the instructions provided in [9, 14, 15].
With respect to the training set, 40 different noise conditions are considered [9, 16].
These arecomposed of 10 types of noise (2 of which are artiﬁcially-generated 1 and 8 sourced from the
DEMAND database [13], each mixed with clean speech at one of 4 signal-to-noise ratios (SNR)(15, 10, 5, and 0 dB).
In total, this yields 11 , 572 training samples, with approximately 10 differentsentences in each condition per training speaker.
The separate test set with 2 speakers consists of a total of 20 different noise conditions: 5 types ofnoise sourced from the DEMAND database at one of 4 SNRs each (17.5, 12.5, 7.5, and 2.5 dB)[14, 15].
This yields 824 test items, with approximately 20 different sentences in each condition pertest speaker [14, 15].
4.2 Experimental setup
As per [12], our baseline model trains on randomly-sampled audio excerpts, using the ADAMoptimization algorithm, a learning rate of 0.0001, decay rates β 1 = 0.9 and β 2 = 0.999 and a batchsize of 16.
We specify a network layer size of 12 with 16 extra ﬁlters per layer, downsampling blockﬁlters of size 15 and upsampling block ﬁlters of size 5 like in [12].
We train for 2,000 iterations withmean squared error (MSE) over all source output samples in a batch as loss and apply early stoppingif there is no improvement on the validation set for 20 epochs.
We use a ﬁxed validation set of 10randomly selected tracks.
Then, the best model is ﬁne-tuned with the batch size doubled and thelearning rate lowered to 0.00001, again until 20 epochs have passed without improved validation loss.
4.3 Results
To evaluate and compare the quality of the enhanced speech yielded by the Wave-U-Net, we mirrorthe objective measures provided in [9].
Each measurement compares the enhanced signal with theclean reference of each of the 824 test set items.
They have been calculated using the implementationprovided in [7] 2 .
The ﬁrst metric is that of the Perceptual Evaluation of Speech Quality (PESQ) -more
speciﬁcally the wide-band version recommended in ITU-T P.862.2 (from –0.5 to 4.5) [7, 9].
Secondly, composite measures of metrics that aim to computationally approximate the Mean Opinion
Score (MOS) that would be produced from human perceptual trials are computed [11].
These are:
CSIG, a prediction of the signal distortion attending only to the speech signal [3] (from 1 to 5); CBAK:a prediction of the intrusiveness of background noise [3] (from 1 to 5); and COVL, a prediction ofthe overall effect [3] (from 1 to 5).
Last is the Segmental Signal-to-Noise Ratio (SSNR) [10] (from 0to ∞ ).
Table 1 shows the results of these metrics for comparison across different speech enhancementarchitectures.
As a comparative, reference, it also shows: the results of these metrics when applieddirectly to the noisy signals; to signals ﬁltered using the Wiener method, based on a priori SNRestimation; and to the SEGAN-enhanced signal, as provided in [9].
The results indicate that the
Wave-U-Net is the most effective model for speech enhancement.
Table 2 shows the performance differences between different variations of the Wave-U-Net, withdifferent numbers of layers.
In this experiment no ﬁne-tuning was performed, which explains thedifference between the 12-layer Wave-U-Nets in Table 1 and in Table 2.
The results suggest thatﬁne-tuning does not make a meaningful difference, except on the CBAK measurement, and thatsmaller models perform better.
This is likely due to the size of the receptive ﬁeld, where for speechthe optimal size is probably smaller than for music.
Conclusions
5.1 Summary
The Wave-U-Net combines the advantages of several of the most recent successful architectures formusic and speech source separation and our results show that it is particularly effective at speechenhancement.
The results improve over the state of the art by a good margin even without signiﬁcantadaptation or parameter tuning.
This indicates that there is great potential for this approach in speechenhancement.
5.2 Future work
In comparison to the SEGAN architecture, it is possible that the advantage stems from the upsamplingthat avoids aliasing, which should be further investigated.
The results indicate that there is room forincreasing effectiveness and efﬁciency by further adapting the model size and other parameters, e g.ﬁlter sizes, to the task and expanding to multi-channel audio and multi-source-separation.