Investigating the effect of residual and highway connections in speech enhancement models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Residual and skip connections play an important role in many current generativemodels.
Although their theoretical and numerical advantages are understood,their role in speech enhancement systems has not been investigated so far.
Whenperforming spectral speech enhancement, residual connections are very similar innature to spectral subtraction, which is the one of the most commonly employedspeech enhancement approaches.
Highway networks, on the other hand, can beseen as a combination of spectral masking and spectral subtraction.
However, whenusing deep neural networks, such operations would happen in a transformed spectraldomain, as opposed to traditional speech enhancement where all operations areoften done directly on the spectrum.
In this paper, we aim to investigate the role ofresidual and highway connections in deep neural networks for speech enhancement,and verify whether or not they operate similarly to their traditional, digital signalprocessing counterparts.
We visualize the outputs of such connections, projectedback to the spectral domain, in models trained for speech denoising, and show thatwhile skip connections do not necessarily improve performance with regards to thenumber of parameters, they make speech enhancement models more interpretable.
Introduction
Highway [Srivastava et al., 2015] and residual networks [He et al., 2015] have been proposed withthe objective of improving activation and gradient ﬂow in the training of deep neural networks.
On the other hand, in tasks like image reconstruction or speech enhancement, the use of such skipconnections serves a different purpose: if we model a corrupted signal x = y + n as the addition ofnoise n to a clean signal y and x is the input to a neural network, we know that the task at hand is topredict n .
In other words, to predict y , we have to alter the input x by subtracting n .
In speech enhancement, the two more commonly used approaches are spectral subtraction and spectralmasking.
In the ﬁrst, a statistical model of n is used to predict its magnitude spectrum N , whichis then subtracted from the input spectrum X to yield a clean magnitude spectrum estimate ˆ Y .
Inspectral masking, instead of performing subtraction, we ﬁnd a multiplicative mask M which aimsat either blocking time-frequency cells dominated by noise (in the case of binary masks) or scalingdown energies in such time-frequency cells to make them match that of the original clean signal.
Recent work in speech enhancement has explored skip connections as a way of performing masking[Mimilakis et al., 2018] and spectral estimation [Santos and Falk, 2018].
Time domain approaches,such as SEGAN [Pascual et al., 2017], use a UNet-style network which employs multiple skipconnections as well.
Other works, such as Williamson and Wang [2017], perform spectral maskingbut learn how to estimate an ideal mask instead of having the masking mechanism embedded in theneural network as a skip connection.
[Figure]
For better understanding of such models, we would like to understand whether there are any parallelsbetween such connections and two traditional DSP approaches to speech enhancement, namelyspectral subtraction and spectral masking.
We also want to understand whether models using skipconnections perform better for enhancement when such connections appear only once (resemblingtheir DSP counterparts) or repeated as multiple blocks (like in highway and residual networks).
1.1 Residual and highway networks
Highway networks and residual networks are related, in the sense that residual networks can be seenas a special case of highway networks.
A highway network block, as proposed in Srivastava et al.[2015], can be described by the following equation:
[Equation]
Residual blocks [He et al., 2015], on the other hand, can be described by the following equation:
[Equation]
which is equivalent (save for a multiplicative constant) to having a ﬁxed gate T that outputs 0.5 forall inputs.
In this paper, we additionally test another type of skip connection, which we call here amasking block.
In this type of connection, the computation in the block serves only to compute amultiplicative gating function which is then applied to the input of the block, as follows:
[Equation]
The reason for using this block is that we wanted to have a closer equivalent to spectral masking thanhighway networks.
In this paper, we used the above mentioned skip connections with stacked gated recurrent units(GRUs) followed by a feedforward output layer with linear activation for each block, as illustrated onﬁgure 1.
1.2 Skip connections in speech enhancement models
Spectral subtraction Boll [1979] speech enhancement models assume distortion is additive and try topredict the magnitude spectrum of the distortion and subtract it from the magnitude spectrum of theinput to yield an estimate of the clean signal:
[Equation]
N ( x, Θ N ) is a noise estimation model with parameters Θ N .
In speech enhancement, such modelsusually predict the noise magnitude spectrum based on several past input frames.
This is very similarto how the model based on residual blocks presented in this paper, and also the models proposed by Santos and Falk [2018] operate.
Spectral masking, on the other hand, is based on predicting time-frequency cells dominated by noiseand creating a multiplicative mask that ﬁlters them out:
[Equation]
Predicted masks can be either binary or ratio masks, although it has been shown that ratio maskspotentially lead to better speech quality [Wang et al., 2014].
The masking model presented in theprevious section, as well as the work done in [Mimilakis et al., 2018, Williamson et al., 2016] performenhancement using masking.
Highway networks, on the other hand, can be indirectly related to bothspectral masking and subtraction: the output of each highway block is the sum between a maskedinput signal and a masked predicted signal.
Experiments2.1 Datasets
For the experiments reported in this paper, we used a single speaker dataset which is publiclyavailable 1 .
It is a relatively small dataset that is comprised of the IEEE sentence list.
This list contains720 phonetically balanced sentences, separated into 72 lists with 10 sentences each, uttered by thesame male speaker.
We split the list into 680 sentences for training and validation (sentence lists01 to 67, and 50 sentences (sentence lists 68 to 72) for testing.
Both datasets can be reproduced byrunning the dataset generation scripts in our code repository 2 .
For the denoising dataset, we mixed the training and validation sentences with noises from the
DEMAND dataset [Thiemann et al., 2013] at SNRs of 12, 6, 3, 0, -3, and -6 dB. The testing sentenceswere mixed with four noises (babble, factory1, factory2, and volvo) from the NOISEX dataset[Varga and Steeneken, 1993], at SNRs of 13, 7, 4, 1, -2, and -5 dB. For each sentence, a randomnoise segment with the same length as the sentence was picked.
Signal energy for speech signalswere computed according to the P.56 standard (which aims at only considering energy from speechsegments and discarding silence segments), while the energy for noise signals was computed byits overall RMS value.
For each (noise type, SNR) pair, we mixed all sentences in the training +validation set or the test set, accordingly.
The training, validation, and test sets have 64923, 3417, and1200 samples each.
The dereverberation dataset was constructed in a similar way, but using simulated room impulseresponses (RIR) obtained using the fast Image-Source Method [Lehmann and Johansson, 2010].
We generated 20 RIRs for each reverberation time in the range 0.2 to 2.0, in 0.05 increments, andconvolved 50 sentences to each RIR (given the large number of RIRs, we did not use the entire datasetfor each).
The training, validation, and test sets have 35150, 1850, and 3700 samples each.
2.2 Model architectures, hyperparameters, and training
For training all models we used the same input representation: the log-magnitude spectrum of thesignal’s short-time Fourier transform with a window of 32 ms and 50% overlap, which correspondsto a dimensionality of 257 coefﬁcients per frame.
The inputs were standardized, but the outputs werenot changed.
All models used 3 layers of gated recurrent units (GRUs) with 256 hidden units each.
We tried bothhaving a single skip connection wrapping the 3 GRU layers as a single residual or highway block,and having 3 blocks with one GRU layer each.
All models were trained using Adam [Kingma and
Ba, 2014] with standard hyper-parameters for 100 epochs, with a mini-batch size of 32.
Analysis
3.1 Visualizing the role of skip connections
To understand how each type of skip connection and each block in a model works, we mainly usedvisualization of each output of a block.
Since skip connections perform operations between the output1 https://www.crcpress.com/downloads/K14513/K14513_CD_Files.ziphttps://anonymous.4open.science/repository/6487014f-35ad-4d1c-959e-5adc9316de95/ of each block and its input, we can assume all operations happen in the domain of the transformedinput, and can therefore be visualized by using the output layer of the model to bring them back tothe STFT domain.
Figure 2 shows the intermediate and ﬁnal outputs of the residual model with 3 blocks for a sentencecorrupted by babble noise at 1 dB. The ﬁrst row shows the input and respective target, the second rowthe residuals predicted in all blocks, and the last row the output of the residual block.
The x and yaxes represent the time and frequency bin respectively, with the color representing the log-magnitude(with blue representing low values, green intermediate values, and yellow high values).
We can seethat earlier blocks exhibit less frequency selectivity than later blocks, with the last block specializingstrongly in certain frequencies.
From block to block, we also see how the strongest components ofthe signal take shape, starting at the lower frequencies, followed by mid-bands and ﬁnally higherfrequencies in the last stage.
Figure 4 shows a similar plot for the masking model, with the second row representing the predictedmasks.
Since we chose to use linear masks instead of bounded masks (such as the outputs of a tanh orsigmoid activation), these masks can ﬂip the sign of time-frequency cells, which makes visualizationa bit more complicated to follow, therefore we used a different colormap for plotting the mask, wherevalues close to zero are shown in white, negative numbers in blue, and positive numbers in red.
Fromthe intermediate outputs, we can see in the ﬁrst two plots that the model is increasingly improvingthe predicted speech structure in all layers, but the signs of many frequency bands might be ﬂippedbefore
we reach the last layer.
Figure 3 shows a similar plot for highway models, with the second and third rows representing theoutputs of H ( x ) and T ( x ) , respectively.
The interpretation of this model is more complicated sincethe output is a linear combination of the elementwise products of T ( x ) and 1 − T ( x ) by H ( x ) andx , respectively.
We can interpret T ( x ) in these plots as showing us which elements of H ( x ) canwe trust more than the current input, which those being represented by green and yellow (stronger conﬁdence), while time-frequency cells marked in dark blue will be dominated by the input of thecurrent block.
We can see the ﬁrst block selects regions where the speech signal is stronger thanthe noise as having a low value for T ( x ) , and decides to reject the input and use its own internalestimate for a number of bands.
In the last layer, though, the model uses its internal estimate for alarge portion of the signal, as evidenced by the predominance of yellow in the last output for T ( x ) .
This seems counter-intuitive, especially as we notice that the output of the second blocks seems tohave a larger amount of noise than that of the ﬁrst block.
Although all results reported here take into account a single example for the denoising tests, thesecharacteristics are common to different sentences and distortion types.
The same observations are alsovalid for the reverberation models, although these use a different set of models so speciﬁc frequencyranges might not be the same but the observations in general still hold.
3.2 Objective quality and intelligibility assessment
We also evaluated the models using the objective speech quality and intelligibility metrics PESQand STOI
[Taal et al., 2010].
For both metrics, higher scores are better.
As can be seen in ﬁgures5-8, for most of the models there is not a large improvement over the baseline.
We can argue that apositive aspect of the use of residual or masking models is that they are more interpretable than thebaseline.
It should also be noted that direct comparison between these models is not completely fairsince the models tested have different numbers of parameters, with the baseline model having theleast parameters, followed by masking, residual, and highway, and models with 3 blocks have moreparameters than their single block counterparts.
We also note that these models are not competitive with state-of-the-art models presented morerecently in the literature.
In this preliminary study, we looked into these models as building blocks for more advanced models who achieve higher performance in these tasks, such as those previouslycited.
Conclusion
This paper shows early results of our investigation on the role of skip connections in speech enhancement models.
Our preliminary experiments show that, although they have no signiﬁcant impact inthe performance of the models, such connections might help making the models more interpretable,as we can identify the contribution of each individual layer to the task.
In the future, we intend toinvestigate more complex models, such as models based on the UNet architecture, as well as modelsthat employ a temporal context window at the input instead of a single frame (such as the work in
Santos and Falk [2018]), since those are more in line with state-of-the-art models in the literature.