V OICE F IXER : T OWARD G ENERAL S PEECH R ESTORATION W ITH N EURAL V OCODER
Anonymous authors
Paper under double-blind review
A BSTRACT
Speech restoration aims to remove distortions in speech signals.
Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping.
However, SSR systems do not address the general speech restoration problem and show limited performance in some speech restoration tasks such as speech super-resolution.
To overcome those limitations, we propose a general speech restoration (GSR) task that attempts to remove multiple distortions simultaneously.
Furthermore, we propose VoiceFixer 1 , a generative framework to address the GSR task.
VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system.
We employ a ResUNet to model the analysis stage and a neural vocoder to model the synthesis stage.
We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions.
Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech denoising SSR model.
VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256.
Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches.
I NTRODUCTION
Speech restoration is a process to restore degraded speech signals to high-quality speech signals.
Speech restoration is an important research topic since distortions are ubiquitous.
For example, speech is usually surrounded by background noise, blurred by room reverberations, or recorded by low-quality devices (Godsill et al., 2002).
Those distortions degrade the perceptual quality of speech for human listeners.
Speech restoration has a wide range of applications such as online meeting (Defossez et al., 2020), hearing aids (Van den Bogaert et al., 2009), and audio editting (Van Winkle, 2008).
Still, speech restoration remains a challenging problem due to the large variety of distortions in the world.
Previous works in speech restoration mainly focus on single task speech restoration (SSR), which deals with only one type of distortion at a time.
For example, speech denoising (Loizou, 2007), speech dereveberation (Naylor & Gaubitch, 2010), speech super-resolution (Kuleshov et al., 2017), or speech declipping (Z´aviˇska et al., 2020).
However, in the real world, speech signal can be degraded by several different distortions simultaneously, which means previous SSR systems oversimplify the speech distortion types (Kashani et al., 2019; Lin et al., 2021; Kuleshov et al., 2017;
Birnbaum et al., 2019).
The mismatch between the training data used in SSR and the testing data from the real world degrades the speech restoration performance.
To address the mismatch problem, we propose a new task called general speech restoration (GSR), which aims at restoring multiple distortions in a single model.
A numerous studies (Cutler et al., 2021; Cauchi et al., 2014; Han et al., 2015) have reported the beneﬁts of jointly training multiple speech restoration tasks.
Nevertheless, performing GSR using one-stage systems still suffer from the problems in each SSR task.
For example, on generative tasks such as speech superresolution (Kuleshov et al., 2017), one-stage models tend to overﬁtting the ﬁlter (Sulun & Davies, 2020) used during training.
Based on these observations, we propose a two-stage system called
VoiceFixer .
[Figure]
The design of VoiceFixer is motivated by the biological mechanisms of human hearing when restoring distorted speech.
Intuitively, if a person tries to identify a strongly distorted voice, his/her brain can do the recovery by utilizing both the distorted speech signal and the prior knowledge of the language.
As shown in Figure 1, the speech distortion perception is modeled by neuroscientists as a two-stage process, including an auditory scene analysis stage (Bregman, 1994), and a high level comprehension/synthesis stage (Grifﬁths & Warren, 2002).
In the analysis stage, the sound information is ﬁrst transformed into acoustic features by primary auditory cortex (PAC).
Then planum temporale (PT), the cortical area posterior to the auditory cortex, acts as a computational hub by segregating and matching the acoustic features to low level spectrotemporal representations.
In the synthesis stage, a high order cortical area is hypothesised to perform the high level perception tasks (Grifﬁths & Warren, 2002; Kennedy-Higgins, 2019).
Our proposed VoiceFixer systems model the analysis stage with spectral transformations and a deep residual UNet, and the synthesis stage with a convolutional vocoder trained using adversarial losses.
One advantage of the two-stage VoiceFixer is that the analysis and synthesis stages can be trained separately.
Two-stage methods have also been successfully applied to the speech synthesis task (Wang et al., 2016;
Ren et al., 2019) where acoustic models and vocoders are trained separately.
VoiceFixer is the ﬁrst GSR model that is able to restore a wide range of low-resolution speech sampled from 2 kHz to 44.1 kHz, which is different from previous studies working on constant sampling rates (Lim et al., 2018; Wang & Wang, 2021; Lee & Han, 2021).
To the best of our knowledge,
VoiceFixer is the ﬁrst model that jointly performs speech denoising, speech dereverberation, speech super-resolution, and speech declipping in a uniﬁed model.
The rest of this paper is organized as follows.
Section 2 introduces the formulations of speech distortions.
Section 3 describes the design of VoiceFixer .
Section 4 discusses the evaluation results.
Appendixes introduce related works and show speech restoration demos.
P ROBLEM F ORMULATIONS
We denote a segment of a speech signal as s ∈ R L , where L is the samples number in the segment.
We model the distortion process of the speech signal as a function d ( · ) .
The degraded speech x ∈ R L can be written as:
[Equation]
Speech restoration is a task to restore high-quality speech ˆ s from x :
[Equation]
where f ( · ) is the restoration function and can be viewed as a reverse process of d ( · ) .
The target is to estimate s by restoring ˆ s from the observed speech x .
Recently, several deep learning based one-stage methods have been proposed to model f ( · ) such as fully connected neural networks, recurrent neural networks, and convolutional neural networks.
Detailed introductions can be found in Appendix A.
Distortion modeling is an important step to simulate distorted speech when building speech restoration systems.
Several previous works model distortions in a sequential order (Vincent et al., 2017;
Tan et al., 2020; Zhao et al., 2019).
Similarly, we model the distortion d ( · ) as a composite function:
[Equation]
where ◦ stands for function composition and Q is the number of distortions to consist d ( · ) .
Set D
=
{ d v ( · ) }
Vv =1 is the set of distortion types where V is the total number of types.
Equationdescribes the procedure of compounding different distortions from D in a sequential order.
We introduce four speech distortions as follows.
Additive noise is one of the most common distortion and can be modeled by the addition between speech s and noise n ∈ R L :
[Equation]
Reverberation is caused by the reﬂections of signal in a room.
Reverberation makes speech signals sound distant and blurred.
It can be modeled by convolving speech signals with a room impulse response ﬁlter (RIR) r :
[Equation]
where ∗ stands for convolution operation.
Low-resolution distortions refer to audio recordings that are recorded in low sampling rates or with limited bandwidth.
There are many causes for low-resolution distortions.
For example, when microphones have low responses in high-frequency, or audio recordings are compressed to low sampling rates, the high frequencies information will be lost.
We follow the description in Wang &
Wang (2021) to produce low-resolution distortions but add more ﬁlter types (Sulun & Davies, 2020).
After designing a low pass ﬁlter h , we ﬁrst convolve it with s to avoid the aliasing phenomenon.
Then we perform resampling on the ﬁltered result from the original sampling rate o to a lower sampling rate u :
[Equation]
Clipping distortions refer to the clipped amplitude of audio signals, which are usually caused by low-quality microphones.
Clipping can be modeled by restricting signal amplitudes within [ − η, + η ] :
[Equation] [Equation]
In the frequency domain, the clipping effect produces harmonic components in the high-frequency part and degrades speech intelligibility accordingly.
M ETHODOLOGY 3. O NE - STAGE S PEECH R ESTORATION M ODELS
Previous deep learning based speech restoration models are usually in one stage.
That is, a model predicts restored speech ˆ s from input x directly:
[Equation]
The mapping function f ( · ) can be modeled by time domain speech restoration systems such as onedimensional convolutional neural networks (Luo & Mesgarani, 2019) or frequency domain systems such as mask-based (Narayanan & Wang, 2013) methods:
[Equation]
where X is the short-time fourier transform (STFT) of x and (cid:15) is a small positive constant.
X has a shape of T × F where T is the number of frame and F is the number of frequency bins.
The output of the mask estimation function F ( · ; θ ) is multiplied by the magnitude spectrogram | X | to produce the target spectrogram estimation ˆ S .
Then, inverse short-time fourier transform (iSTFT) is applied on ˆ S to obtain ˆ s .
The one-stage speech restoration models are typically optimized by minimizing the mean absolute error (MAE) loss between the estimated spectrogram ˆ S and the target spectrogram
S :
[Equation]
[Equation]
[Figure]
[Figure]
Previous one-stage models usually build on high-dimensional features such as time samples and the
STFT spectrograms.
However, Kuo & Sloan (2005) point out that the high-dimensional features will lead to exponential growth in search space.
The model can work on the high-dimensional features under the premise of enlarging the model capacity but may also fail in challenging tasks.
Therefore, it would be beneﬁcial if we could build a system on more delicate low-dimensional features.
3. V OICE F IXER
In this study, we propose VoiceFixer , a two-stage speech restoration framework.
Multi-stage methods have achieved state-of-the-art performance in many speech processing tasks (Jarrett et al., 2009;
Tan et al., 2020).
In speech restoration, our proposed VoiceFixer breaks the conventional one-stage system into a two-stage system:
[Equation]
Equation 11 denotes the analysis stage of VoiceFixer where a distorted speech x is mapped into a representation z .
Equation 12 denotes the synthesis stage of VoiceFixer , which synthesize z to the restored speech ˆ s .
Through the two-stage processing, VoiceFixer mimics the human perception of speech described in Section 1.
3.2. A NALYSIS S TAGE
The goal of the analysis stage is to predict the intermediate representation z , which can be used later to recover the speech signal.
In our study, we choose mel spectrogram as the intermediate representation.
Mel spectrogram has been widely used in many speech proceessing tasks (Shen et al., 2018; Kong et al., 2019; Narayanan & Wang, 2013).
The frequency dimension of mel spectrogram is usually much smaller than that of STFT thus can be regarded as a way of feature dimension [Figure] Loss functions and discriminators [Figure]
reduction.
So, the objective of the analysis stage becomes to restore mel spectrograms of the target signals.
The mel spectrogram restoration process can be written as the following equation,
[Equation]
where X mel is the mel spectrogram of x .
It is calculated by X mel = | X | W where W is a set of mel ﬁlter banks with shape of F × F (cid:48) .
The columns of W are not divided by the width of their mel bands, i.e., area normalization, because this will make the restoration model difﬁcult to recover the high-frequency part.
The mapping function f mel ( · ; α ) is the mel restoration mask estimation model parameterized by α .
The output of f mel is multiplied by X mel to predict the target mel spectrogram.
We use ResUNet (Kong et al., 2021a) to model the analysis stage as shown in Figure 3a, which is an improved UNet (Ronneberger et al., 2015).
The ResUNet consists of several encoder and decoder blocks.
There are skip connections between encoder and decoder blocks at the same level.
As is shown in Figure 3b, both encoder and decoder block share the same structure, which is a series of residual convolutions (ResConv).
Each convolutional layer in ResConv consists of a batch normalization (BN), a leakyReLU activation, and a linear convolutional operation.
The encoder blocks apply average pooling for downsampling.
The decoder blocks apply transpose convolution for upsampling.
In addition to ResUNet, we implement the analysis stage with fully connected deep neural network (DNN), and bidirectional gated recurrent units (BiGRU) (Chung et al., 2014) for comparison.
The DNN consists of six fully connected layers.
The BiGRU has similar structures with DNN except for replacing the last two layers of DNN into bi-directional GRU layers.
The details of these three models are discussed in Appendix B.1.
We will refer to ResUNet as UNet later for abbreviation.
We optimize the model in the analysis stage using the MAE loss between the estimated mel spectrogram ˆ S mel and the target mel spectrogram S mel :
[Equation]
3.2. S YNTHESIS S TAGE
The synthesis stage is realized by a neural vocoder that synthesizes the mel spectrogram into waveform as denoted in Equation 15:
[Equation]
where g ( · ; β ) stands for the vocoder model parameterized by β .
We employ a recently proposed nonautoregressive model, time and frequency domain based generative adversarial network (TFGAN), as the vocoder.
Figure 4 shows the detailed architecture of TFGAN , in which the input mel spectrogram X mel will ﬁrst pass through a condition network CondNet , which contains N 1 one-dimensional convolution layers with exponential linear unit activations (Clevert et al., 2015).
Then, in UpNet , the feature is upsampled N 2 times with ratios of s 0 , s 1 , ... , and s N 2 − 1 using UpsampleBlock and ResStacks .
Within the UpsampleBlock , the input is ﬁrst passed through a leakyReLU activation and then fed into a sinusoidal function, which output is added to its input to remove periodic artifacts in breathing part of speech.
Then, the output is bifurcated into two branches for upsampling.
One branch repeats the samples s n times followed by a one-dimensional convolution.
The other branch uses a stride s n transpose convolution.
The output of the repeat and transpose convolution branches are added together as the output of UpsampleBlock .
ResStacks module contains two dilated convolution layers with leakyReLU activations.
The exponentially growing dilation in ResStack enable the model to capture long range dependencies.
The TFGAN in our synthesis model applies N 2 = 4 .
After four
UpsampleBlock blocks with ratios [7 , 7 , 3 , 3] , each frame of the mel spectrogram is transformed into a sequence with 441 samples corresponding to 10 ms of audio sampled at 44.1 kHz.
The training criteria of the vocoder consist of frequency domain loss (cid:76) F , time domain loss (cid:76) T , and weighted discriminator loss
[Equation]
The frequency domain loss (cid:76) F is the combination of a mel loss (cid:76) mel and multi-resolution spectrogram losses:
[Equation]
) where (cid:76) sc and (cid:76) mag are the spectrogram losses calculated in the linear and log scale, respectively.
There are K F different window sizes ranging from 64 to 4096 to calculate (cid:76) sc and (cid:76) mag so that the trained vocoder is tolerant over phase mismatch (Yamamoto et al., 2020; Juvela et al., 2019).
Tablein Appendix B.2 shows the detailed conﬁgurations.
Time domain loss is complementary to frequency domain loss to address problems such as periodic artifacts.
Time domain loss combines segment loss (cid:76) ( k ) seg , energy loss (cid:76) ( k ) energy and phase loss (cid:76)
[Equation]
[Equation]
where segment loss (cid:76) ( k ) seg , energy loss (cid:76) ( k ) energy and phase loss (cid:76) ( k ) phase are described in Equation 24, 25, and 26 of Appendix B.2.
There are K T different window sizes ranging from 1 to 960 to calculate time domain loss at different resolutions.
The details of window sizes are shown in Table 3 of
Appendix B.2.
The energy loss and phase loss have the advantage of alleviating metallic sounds.
Discriminative training is an effective way to train neural vocoders (Kong et al., 2020; Kumar et al., 2019).
In our study, we utilize a group of discriminators, including a multi-resolution time discriminator D T , a subband discriminator D sub , and frequency discriminator D F :
[Equation]
[Equation]
The multi-resolution discriminators D T take signals from R T kinds of time resolutions after average pooling as input.
The subband discriminator D sub performs subband decomposition (Liu et al., 2020) on the waveform, producing four subband signals to feed into four T-discrminators , respectively.
Frequency discriminator D F takes the linear spectrogram as input and outputs real or fake labels.
The bottom part of Figure 4 shows the main idea of T-discriminator and F-discriminator .
Appendix B.2 describes the detailed discriminator architectures.
There are two advantages of using neural vocoder in the synthesis stage.
First, neural vocoder trained using a large amount of speech data contains prior knowledge on the structural distribution of speech signals, which is crucial to the restoration of distorted speech.
The amount of training data of vocoder is more than that used in conventional SSR methods with limited speaker numbers.
Second, the neural vocoder typically takes the mel spectrogram as input, resulting in fewer feature dimensions than the STFT features.
The reduction in dimension helps to lower computational costs and achieve better performance in the analysis stage.
E XPERIMENTS 4. D ATASETS AND E VALUATION M ETRICS
Training sets: The training speech datasets we use including VCTK (Yamagishi et al., 2019),
AISHELL-3 (Shi et al., 2020), and HQ-TTS.
We call the noise datasets used for training as VDNoise.
To simulate the reverberations, we employ a set of RIRs to create an RIR-44k dataset.
We use VCTK, VD-Noise, and the training part of RIR-44k to train the analysis stage.
AISHELL-3,
VCTK, and HQ-TTS datasets are used to train the vocoder in the synthesis stage.
The details of those datasets and the RIRs simulation conﬁgurations are discussed in Appendix C.1.
Test sets: We employ VCTK-Demand (Valentini-Botinhao et al., 2017) as the denoising test set and name it as DENOISE.
We call our speech super-resolution, declipping, and dereverberation evaluation test sets as SR, DECLI, and DEREV, respectively.
In addition, we create an ALL-GSR test set containing all distortions.
We introduce the details of how we build these test sets in Appendix C.3.
Evaluation metrics: The metrics we adopt include log-spectral distance (LSD) (Erell & Weintraub, 1990), wide band perceptual evaluation of speech quality (PESQ-wb) (Rix et al., 2001), structural similarity (SSIM)
(Wang et al., 2004), and scale-invariant signal to noise ratio (SiSNR) (Le Roux et al., 2019).
We use mean opinion scores (MOS) to subjevtively evaluate different systems.
The output of the vocoder is not strictly aligned on sample level with the target, as is often the case in generative model (Kumar et al., 2020).
This effect will degrade the metrics, especially for those calculated on time samples such as the SiSNR.
So, to compensate the SiSNR, we design similar metric, scale-invariant spectrogram to noise ratio (SiSPNR), to measure the discrepancies on the spectrograms.
Details of the metrics are described in Appendix C.4.
4. D ISTORTIONS S IMULATION
For the SSR task, we perform only one type of distortion for evaluation.
For the GSR task, we ﬁrst assume that D = { d noise , d rev , d low res , d clip } because those distortions are the most common distortions in daily environment (Ribas et al., 2016).
Second, we assume that Q ≤ 4 in Equation 3.
In other words, each distortion in D appears at most one time.
Then, we generate the distortions following a speciﬁc order d rev , d clip , d low res , and d noise .
These distortions are added randomly using random conﬁgurations.
4. B ASELINE S YSTEMS
Table 5 in Appendix D summarizes all the experiments in this study.
We implement several SSR and GSR systems using one-stage restoration models.
For the GSR, we train a ResUNet model called GSR-UNet with all distortions.
For the SSR models, we implement a Denoise-UNet for additive noise distortion, a Dereverb-UNet for reverberation distortion, a SR-UNet for low-resolution distortion, and a Declip-UNet for clipping distortion.
For the SR task, we also include two stateof-the-art models, NuWave (Lee & Han, 2021) and SEANet (Li et al., 2021) for comparison.
For declipping task, we compare with a state-of-the-art synthesis-based method SSPADE (Z´aviˇska et al.,
2019a).
To explore the impact of model size of the mel restoration model, we setup ResUNets with two sizes, UNet-S and UNet , which have one and four ResConv blocks in each encoder and decoder block, respectively.
4. E VALUATION R ESULTS
Neural vocoder To evaluate the performance of the neural vocoder, we compare two baselines.
The
Target system denotes using the perfect s for evaluation.
The Unprocessed system denotes using distorted speech x for evaluation.
The Oracle-Mel system denotes using the mel spectrogram of s as input to the vocoder, which marks the performance of the vocoder.
As shown in Table 1, the OracleMel system achieves a MOS score of 3.74, which is close to the Target MOS of 3.95, indicating that the vocoder performs well in the synthesis task.
[Table]
General speech restorations: Table 1 shows the evaluation results on ALL-GSR test set.
Figureshows the box plot of the MOS scores of these systems.
The GSR-UNet outperforms the two SSR models, Denoise-UNet and Dereverb-UNet by a large margin.
It surpasses Denoise-UNet model by 0.5 on MOS score, which suggests the GSR model is more powerful than the SSR model on this test set.
For convenience, we denote VoiceFixer as VF in tables and ﬁgures.
We observe that the
VF-UNet model achieves the highest MOS score and LSD score.
Speciﬁcally, VF-UNet obtains 0.256 higher MOS score than that of GSR-UNet .
This result indicates that VoiceFixer is better than
ResUNet based one-stage model on overall quality.
What’s more, we notice that the MOS score of VF-UNet is only 0.11 lower than the Oracle-Mel , demonstrating the good performance of the analysis stage.
Among the VoiceFixer analysis models, the UNet front-end achieves the best.
The
VF-BiGRU model achieves similar subjective metrics with the VF-UNet model but has much lower
MOS scores.
This phenomenon shows that the improvement in subjective metrics in VoiceFixer is not always consistent with objective evaluation results.
Super-resolution: Table 6 in Appendix D.1 shows the evaluation results on the super-resolution test set SR .
For the 2 kHz, 4 kHz, and 8 kHz to 44.1 kHz super-resolution tasks, VF-UNet achieves a signiﬁcantly higher LSD, SiSPNR and SSIM scores than other models.
The LSD value of VF-UNet in 2 kHz sampling rate is still higher than the 8 kHz sampling rate score of GSR-UNet , SR-UNet ,
NuWave , and SEANet .
This demonstrates the strong performance of VoiceFixer on dealing with low sampling rate cases.
The VF-BiGRU model outperforms VF-UNet-S model on average scores for its better performance on low upsample-ratio cases.
MOS box plot in Figure 6b shows that VF-UNet performs the best on 8 kHz to 44.1 kHz test set.
Figure 6e shows the MOS score of Unprocessed is close to Target on 24 kHz to 44.1 kHz test set, meaning limited perceptual difference between the two sampling rates.
On this test set, SEANet even achieves a higher MOS score than Target .
That’s due to the high-frequency part it generate contains more energy than that of Target , making the results sound clearer.
We evaluate the speech denoising performance on the DENOISE test set and show results in Table 7 in Appendix D.1.
We ﬁnd that GSR-UNet preserves more details in the highfrequency part and has better PESQ and SiSPNR values than the denoising only SSR model DenoiseUNet .
The reason might be that speech data augmentations and jointly performing super-resolution task can increase the generalization and inpainting ability of the model (Hao et al., 2020).
The PESQ
[Figure]
score of VF-UNet reaches 2.43, higher than SEGAN , WaveUNet , and the model trained with weakly labeled data in Kong et al.
(2021b).
The MOS evaluations in Figure 6f on speech denoising task also demonstrate that the result of VF-UNet sound comparable with one-stage speech denoising models.
Declipping and dereveberation: Table 9 and Table 8 in Appendix D.1 show similar performance trends on the speech declipping and speech dereveberation.
In both tasks, the SSR model DereverbUNet and Declip-UNet achieve the highest scores.
The performance of GSR-UNet is slightly worse, but it is acceptable considering that GSR-UNet does not need extra training for each task.
SSPADE performs better on SiSNR, but the PESQ and STOI scores are lower, especially in the 0.1 threshold case.
The MOS score in Figure 6d shows that the clipping effect in the 0.25 threshold case is not easy to perceive, leading to high MOS scores across all methods.
In Figure 6a, both Declip-UNet and VF-UNet achieve the highest objective scores on the 0.1 threshold clipping test set.
On the dereverberation test set DEREV, VF-UNet achieves the highest MOS score 3.52.
C ONCLUSIONS
In this work, we propose VoiceFixer , an effective approach for general speech restoration.
The evaluation results show that VoiceFixer achieves leading performance across the general speech restoration, speech super-resolution, speech denoising, speech dereverberation, and speech declipping tasks.
In the future, we will extend VoiceFixer to restore general audio signals, including music and general sounds.
R EPRODUCIBILITY
S TATEMENT
We make our code and datasets anonymously downloadable for painless reproducibility.
Our pre-trained VoiceFixer and inference code are presented in https://github.com/ anonymous20211004/iclr2022-vf .
The code for performing the experiments discussed in Section 4 is downloadable in https://github.com/anonymous20211004/ iclr2022-train-vf .
The code can be used to conduct evaluations and generate reports on the metrics mentioned in Section 4.1 automatically.
The NuWave is realized using the code open-sourced by Lee & Han (2021):
https://github.com/mindslab-ai/nuwave .
We reproduce SSPADE using the toolbox provided by Z´aviˇska et al.
(2020) at https://rajmic.github.io/ declipping2020 .
Besides, we upload the training and testing sets mentioned in Section 4.
to https://zenodo.org/record/5546723 .
The AISHELL-3 dataset is open-sourced at http://www.aishelltech.com/aishell_3 .
The HQ-TTS is a collection of datasets from openslr.org , as is described in Table 4 of Appendix C.1.
A A PPENDIX A A. S PEECH RESTORATION TASKS
Speech super-resolution
A lot of early studies (Nakatoh et al., 2002; Kontio et al., 2007) break super-resolution (SR) into spectral envelop estimation and excitation generation from the lowresolution part.
At that time, the direct mapping from the low-resolution part to the high-resolution feature is not widely explored since the dimension of the high-resolution part is relatively high.
Later, deep neural network (Li & Lee, 2015; Kuleshov et al., 2017) is introduced to perform SR.
These approaches show better subjective quality comparing with traditional methods.
To increase the modeling capacity, TFilm (Birnbaum et al., 2019) is proposed to model the afﬁne transformation among each time block.
Similarly, WaveNet also shows effectiveness in extending the bandwidth of a band-limited speech (Gupta et al., 2019).
To utilize the information both from the time and frequency domain, Wang & Wang (2021) propose a time-frequency loss that can yield a balanced performance on different metrics.
Recently, NU-GAN (Kumar et al., 2020) and NU-Wave (Lee &
Han, 2021) pushed the target sample rate in SR to high ﬁdelity, namely 48 kHz.
Although employing deep neural networks in BWE shows promising results, the generalization capability of these methods is still limited.
For example, previous approaches (Kuleshov et al., 2017) usually train and test models with a ﬁxed setup, i.e., ﬁxing the initial and target sampling rates.
However, in real-world applications, speech bandwidth is not usually constant.
In addition, since the real high-low quality speech pair is hard to collect, typically, we produces low-quality audio with lowpass or bandpass ﬁlters during training.
In this case, systems tend to overﬁt speciﬁc ﬁlters.
As discussed in Sulun & Davies (2020), when the kind of ﬁlter used during training and testing differ, the performance can fall considerably.
To alleviate ﬁlter overﬁtting, Sulun & Davies (2020) propose to train models with multiple kinds of lowpass ﬁlters, by which the unseen ﬁlters can be handled properly.
Speech declipping The methods for speech declipping can be categorized as supervised methods and unsupervised methods.
The unsupervised, or blind methods usually perform declipping based on some generic regularization and assumption of what natural audio should look like, such as
ASPADE (Kiti ´ c et al., 2015), dictionary learning (Rencker et al., 2019), and psychoacoustically motivated l1 minimization (Z ´ avi ˇ ska et al., 2019b).
The supervised models, mostly based on deep neural network (DNN) (Bie et al., 2015; Mack & Habets, 2019), are usually trained on clipped and unclipped data pairs.
For example, Kashani et al.
(2019) treat the declipping as an image-toimage translation problem and utilize the UNet to perform spectral mapping.
Currently, most of the state-of-the-art methods are unsupervised (Z´aviˇska et al., 2020) because they are usually designed to work on different kinds of audio, while the supervised model mainly specialized on data similar to its training data.
However, Z ´ avi ˇ ska et al.
(2020) believes supervised models still have the potential for better declipping performance.
Speech denoising Conventional methods are efﬁcient and effective on dealing with stationary noise, such as spectral substraction (Martin, 1994) and Wiener and Kalman ﬁltering (Kailath, 1981)
By comparison, deep learning based models such as Conv-TasNet (Luo & Mesgarani, 2019) show higher subjective score and robustness on complex cases.
Recently, new schemes have emerged for training speech denoising models.
SEGAN (Pascual et al., 2017) tried a generative way.
Kong et al.
(2021b) achieved a denoising model using only weakly labeled data.
And Polyak et al.
(2021) realized a denoising model using a regeneration approach.
Speech dereveberation Some of the early methods in speech dereverberation, such as inverse ﬁltering (Naylor & Gaubitch, 2010) and subband envelope estimation (Wang & Itakura, 1991), aiming at deconvolving the reverberant signal by estimating an inverse ﬁlter.
However, the inverse ﬁlter is hard and not robust to estimate accurately.
Other methods, like spectral substraction, is based on an overlap-masking (N´abˇelek et al., 1989) effect of reverberation.
Schwartz et al.
(2014) performed dereverberation using Kalman ﬁlter and expectation-maximization algorithm.
Recently, deep learning based dereverberation methods have emerged as the state-of-the-art.
Han et al.
(2015) used a fully connected DNN to learn a spectral mapping from reverberant speech to clean speech.
In
Williamson & Wang (2017), similar to the masking-based denoising methods, they proposed to perform dereverberation using a time-frequency mask.
A.1. J OINT RESTORATION AND SYNTHETIC RESTORATION
Joint restoration Many works have adopted the joint restoration approach to improving models.
To make the acoustic echo cancellation (AEC) result sound cleaner,
MC-TCN (Shu et al., 2021) proposed to jointly perform AEC and noise suppression at the same time.
MC-TCN achieved a mean opinion score of 4.41, outperforming the baseline of AEC Challenge (Cutler et al., 2021) by 0.54.
Moreover, in the REVERB challenge (Kinoshita et al., 2013), the test set has both reverberation and noise.
Therefore, the proposed methods in the challenge should perform both denoising and dereverberation.
In Han et al.
, the authors proposed to perform dereverberation and denoising within a single DNN and substantially outperform related methods regarding quality and intelligibility.
However, previous joint processing usually involved only two sub tasks.
In our study, we joint perform four or more tasks to achieve general restoration.
Synthetic restoration Directly estimate the source signal from the observed mixture is hard especially when the SNR is low.
Some studies adopted a regeneration approach.
In Polyak et al.
(2021), the authors utilized an ASR model, a pitch extraction model, and a loudness model to extract semantic level information from the speaker.
Then these features were fed to an encode-decoder network to regenerate the speech signal.
To maintain the consistency of speaker characteristics, it used an auxiliary identity network to compute the identity feature.
Similar to synthetic speech restoration,
TTS can be treated as the regeneration of speech from texts.
A.1. N EURAL VOCODER
Vocoder, which maps the encoded speech features to the waveforms, is an indispensable component in speech synthesis.
The most widely used input feature for vocoder is mel spectrogram.
In recent years, since the emergence of WaveNet (Oord et al., 2016), neural network based vocoders demonstrate clear advantages over traditional parametric methods (Morise et al., 2016).
Comparing with conventional methods, the quality of WaveNet is more closer to the human voice.
Later, WaveRNN (Yu et al., 2019) is proposed to model the waveform with a single GRU.
In this way, WaveRNN has much lower complexity comparing with WaveNet .
However, the autoregressive nature of these models and deep structure make their inference process hard to be paralleled.
To address this problem, non-autoregressive models like WaveGlow (Prenger et al., 2019) and WaveFlow (Ping et al., 2020) were proposed.
Afterward, non-autoregressive GAN-based models such as MelGAN (Kumar et al., 2019) push the synthesis quality to a comparable level with autoregressive models.
Recently,
TFGAN (Tian et al., 2020) demonstrated strong capability in vocoding.
Directed by multiple discriminators and loss functions, TFGAN was able to leverage information from both time domain and frequency domain.
As a result, the synthesis quality of TFGAN is more natural and less metallic comparing with other GAN-based non-autoregressive models.
In this work, we realize a universal vocoder based on TFGAN , which can reconstruct waveform from mel spectrogram of an arbitrary speaker with good perceptual quality.
The pre-trained vocoder is available online to facilitate future studies and reproduce our work.
B A PPENDIX B B. D ETAILS OF THE ANALYSIS STAGE
[Figure]
The DNN and BiGRU we use are shown in Figure 7.
DNN is a six layers fully connected network with BatchNorm and ReLU activations.
The DNN accept each time step of the low-quality spectrogram as the input feature and output the restoration mask.
Similarly, for the BiGRU model, we substitute some layers in DNN to a two-layer bidirectional GRU to capture the time dependency between time steps.
To increase the modeling capacity of BiGRU, we expanded the input dimension of GRU to twice the mel frequency dimension with full connected networks.
The detailed architecture of ResUNet is shown in Figure 3a.
In the down-path, the input low-quality mel spectrogram will go through 6 encoder blocks, which includes a stack of L 1 ResConv and a 2 × 2 average pooling.
In ResConv , the outputs of ConvBlock and the residual convolution are added together as the output.
ConvBlock is a typical two layers convolution with BatchNorm and leakyReLU activation functions.
The kernel size of residual convolution and the convolution in
ConvBlock is 1 × 1 and 3 × 3 .
Correspondingly, the decoder blocks have the symmetric structure of the encoder blocks.
It ﬁrst performs a transpose convolution with 2 × 2 stride and 3 × 3 kernels, which result is concatenated with the output of the encoder at the same level to form the input of the decoder.
The decoder also contain L 2 layers of ResConv .
The output of the ﬁnal decoder block is passed to a ﬁnal ConvBlock to ﬁt the output channel.
We use Adam optimizer with β 1 = 0 .
5 , β 2 = 0 .
999 and a 3e-4 learning rate to optimize the analysis stage of VoiceFixer .
We treat the ﬁrst 1000 steps as the warmup phase, during which the learning rate grows linearly from 0 to 3e-4.
We decay the learning rate by 0.9 every 400 hours of training data.
We perform an evaluation every 200 hours of training data.
If we observe three consecutive evaluations with no improvement, we will interrupt the experiment.
For all the STFT and iSTFT, we use the hanning window with a window length of 2048 and a hop length of 441.
As all the audio we use is at the 44.1 kHz sample rate, the corresponding spectrogram size in this setting will be T × 1025 , where T is the dimension of time frames.
For mel spectrogram, the dimensions of the linear spectrogram are transformed into T × 128 .
B. D ETAILS OF THE SYNTHESIS STAGE
As shown in Table 3, we use 7 kinds of STFT resolutions and 4 kinds of time resolution during the calculation of (cid:76) F and (cid:76) T .
So K F = 7 in Equation 17 and K T = 4 in Equation 18.
The mel loss (cid:76) mel , spectral convergence loss (cid:76) sc , STFT magnitude loss (cid:76) mag , segment loss (cid:76) seg , energy loss (cid:76) energy , and phase loss (cid:76) phase are deﬁned in Equation 21 26.
The function v ( · ) is the
[Table] [Table]
[Table]
windowing function that divide time sample into w windows and compute mean value within each window, v ( s ) 1 × w = (mean( s 0 ) , mean( s 1 ) , ..., mean( s w − 1 )) .
Each s w stand for windowed s .
∆ stand for ﬁrst difference.
[Equation]
[Equation]
Table 9 and Table 8 show the structure of frequency and time domain discriminators.
The subband discriminators D sub and multi-resolution time discriminators D ( r ) T ( s ) use the structure of Tdiscriminator , which is a stack of one dimensional convolution with grouping and large kernal size.
The frequency discriminator D F use the similar module ResConv similar to ResUNet shown in Figure 3b
[Figure]
[Figure]
For the training of vocoder, we setting up the λ D to λ seg value in Equation 16, Equation 17, and Equation 18 as λ D = 4 . 0 , λ mel = 50 , λ sc = 5 . 0 , λ mag = 5 . 0 , λ energy = 100 . 0 , λ phase = 100 . 0 , and λ seg = 200 .
C A PPENDIX C C. D ATASETS
Clean speech CSTR VCTK corpus (Yamagishi et al., 2019) is a multi-speaker English corpus containing 110 speakers with different accents.
We split it into a training part VCTK-Train and a testing part VCTK-Test.
The version of VCTK we used is 0.92.
To follow the data preparation strategy of Lee & Han (2021), only the mic1 microphone data is used for experiments, and p and p315 are omitted for the technical issues.
For the remaining 108 speakers, the last 8 speakers, p360,p361,p362,p363,p364,p374,p376,s5 are splitted as test set VCTK-Test.
Within the other 100 speakers, p232 and p257 are omitted because they are used later in the test set DENOISE, the remaining 98 speakers are deﬁned as VCTK-Train.
Except for the training of NuWave , all the utterances are resampled at the 44.1 kHz sample rate.
AISHELL-3 is an open-source Hi-Fi mandarin speech corpus, containing 88035 utterances with a total duration of 85 hours.
HQ-TTS dataset contains 191 hours of clean speech data collected from a serial of datasets on openslr.org .
In
Table 4, we include the details of HQ-TTS , including the URL and language types of each subset.
[Table] [Table]
openslr.
[Table] [Table]
.openslr.org/
[Table] [Table]
[Table]
openslr.
[Table] [Table]
openslr.
[Table] [Table]
[Table] [Table]
openslr.
[Table] [Table]
Noise data One of the noise dataset we use come from VCTK-Demand (VD) (Valentini-Botinhao et al., 2017), a widely used corpus for speech denoising and noise-robust TTS training.
This dataset contains a training part VD-Train and a testing part VD-Test, in which both contain two noisy set
VD-Train-Noisy, VD-Test-Noisy and two clean speech set VD-Train-Clean, VD-Test-Clean.
To obtain the noise data from this dataset, we minus each noisy data from VD-Train-Noisy with its corresponding clean part in VD-Train-Clean to get the ﬁnal training noise dataset VD-Noise.
The noise data are all resampled to 44.1 kHz.
Another noise dataset we adopt is the TUT urban acoustic scenes 2018 dataset (Mesaros et al., 2018), which is originally used for the acoustic scene classiﬁcation task of DCASE 2018 Challenge.
The dataset contains 89 hours of high-quality recording from 10 acoustic scenes such as airport and shopping mall.
The total amount of audio is divided into development DCASE-Dev and evaluation DCASE-Eval parts.
Both of them contain audio from all cities and all acoustic scenes.
Room impulse response We randomly simulated a collection of Room Impulse Response ﬁlters to simulate the 44.1 kHz speech room reverberation using an open-source tool 2 .
The meters of height, width, and length of the room are sampled randomly in a uniform distribution (cid:85) (1 , 12) .
The placement of the microphone is then randomly selected within the room space.
For the placement of the sound source, we ﬁrst determined the distance to the microphone, which is randomly sampled in a Gaussian distribution (cid:78) ( µ, σ 2 ) , µ = 2 , σ = 4 .
If the sampled value is negative or greater than ﬁve meters, we will sample the distance again until it meets the requirement.
After sampling the distance between the microphone and sound source, the placement of the sound source is randomly selected within the sphere centered at the microphone.
The RT60 value we choose come from the uniform distribution (cid:85) (0 . 05 , 1 . 0) .
For the pickup pattern of the microphone, we randomly choose from omnidirectional and cardioid types.
Finally, we simulated 43239 ﬁlters, in which we randomly split out 5000 ﬁlters as the test set RIR-Test and named other 38239 ﬁlters as RIR-Train.
C. T RAINING DATA
We describe the simulation of training data in Algorithm 1.
S =
{ s (0) , s (1) , ..., s ( i ) } , N = { n (0) , n (1) , ..., n ( i ) } , and R = { r (0) , r (1) , ..., r ( i ) } are the speech dataset, noise dataset, and
RIR dataset.
We use several helper function to describe this algorithm.
randomFilterType( · ) is a function that randomly select a type of ﬁlter within butterworth, chebyshev, bessel, and ellipic.
Resample( x , o 1 , u ) is a resampling function that resample the one dimensional signal x from a original samplerate o 1 to the target u samplerate.
buildFilter( t, c, o 2 ) is a ﬁlter design function that return a type t ﬁlter with cutoff frequency c and order o 2 .
max( · ) , min( · ) , and abs( · ) is the element wise maximum, minimum, and absolute value function.
mean( · ) calculate the mean value of the input.
We ﬁrst select a speech utterance s , a segment of noise n and a RIR ﬁlter r randomly from the dataset.
Then with p 1 probability, we add the reverberation effect using r .
And with p 2 probability, we add clipping effect with a clipping ratio η , which is sampled in a uniform distrubution (cid:85) ( η low , η high ) .
To produce low-resolution effect, after determining the ﬁlter type t , we randomly sample the cutoff frequency c and order o from the uniform distribution (cid:85) ( C low , C high ) and (cid:85) ( O low , O high ) .
Then we perform convolution between x and the type t order o lowpass ﬁlter with cutoff frequency c .
Finally the ﬁltered data will be resampled twice, one is resample to c ∗ 2 samplerate and another is resample back to 44.1 kHz.
We also perform the same lowpass ﬁltering to the noise signal randomly.
This operation is necessary because, if not, the model will overﬁt the pattern that the bandwidth of noise signal is always different from speech.
In this case, the model will fail to remove noise when the bandwidth of noise and speech are similar.
For the simulation of noisy environment, we randomly add the noise n into the speech signal x using a random SNR s ∼ (cid:85) ( S low , S high ) .
To ﬁt the model with all energy levels, we randomly conduct a q ∼ (cid:85) ( Q low , Q high ) scaling to the input and target data pair.
In our work, we choose the following parameters to perform this algorithm, p 1 = 0 .
25 , p 2 = 0 .
25 , p 3 = 0 .
5 , η low = 0 .
06 , η high = 0 .
9 , C low =
750 , C high = 22050 , O low = 2 ,
O high = 10 ,
S low = − 5 , S high = 40 , Q low = 0 .
3 , Q high = 1 . 0 .
Algorithm 1: Add random distortions to high-quality speech signal s
In: s ← S ; n ← N ; r ← R
Out: The high-quality speech s and its randomly distorted version x x = s ; with p 1 probability: x = x ∗ r ; /* Convolute with RIR filter */ with p 2 probability: θ ∼ (cid:85) (Θ low , Θ high ) ; /* Choose clipping ratio */ x = max(min
( x , θ ) , − θ ) ; /
* Hard clipping */ with p 3 probability: t = randomFilterType() ; c ∼ (cid:85) ( C low , C high ) ; o ∼ (cid:85) ( O low , O high ) ; /
* Random cutoff and order */ x = x ∗ buildFilter( t, c, o ) ; /
* Low pass filtering */ x = Resample(Resample( x , 44100 , c ∗ 2) , c ∗ 2 , 44100) ; /* Resample */ with p 4 probability: n = n ∗ buildFilter( t, c, o ) ; /
* Low pass filtering on noise */ n = Resample(Resample( n , c ∗ 2) , 44100) ; /* Resample */ with p 5 probability: s ∼
(cid:85) ( S low , S high
) ; q ∼ (cid:85)
( Q low
Q high
) ; /* Random SNR and scale */ n = n mean(abs( n )) /
mean(abs( x ))
; / * Normalize the energy of noise * / x = ( x + n 10 s/ 20 ) ; /*
Add noise */ s = q s ; /* Scaling */ x = q x ; /* Scaling */
C. T ESTING DATA
Testing data is crucial for the evaluation for each kind of distortion.
The testing data we use either come from open-sourced test set or simulated by ourselves.
Super-resolution The simulation of the SR test set follows the work of (Kuleshov et al., 2017;
Wang & Wang, 2021).
The low-resolution and target data pairs are obtained by transforming 44.
kHz sample rate utterances in target speech data VCTK-Test to a lower sample rate u .
To achieve that, we ﬁrst convolve the speech data with an order 8 Chebyshev type I lowpass ﬁlter with the ucutoff frequency.
Then we subsample the signal to u sample rate using polyphase ﬁltering.
In this work, to test the performance on different sampling rate settings, u are set at 2 kHz, 4 kHz, 8 kHz,kHz, and 24 kHz.
We denote the corresponding ﬁve testing set as VCTK-4k, VCTK-4k, VCTK-8k,
VCTK-16k, and VCTK-24k, respectively.
Denoising For the denoising task, we adopt the open-sourced testing set DENOISE described in
Appendix C.1.
This test set contains 824 utterances from a female speaker and a male speaker.
The type of noise data comprises a domestic noise, an ofﬁce noise, noise in the transport scene, and two street noises.
The test set is simulated at four SNR levels, which are 17.5 dB, 12.5 dB, 7.5 dB, and 2.5 dB.
The original data is sampled at 48 kHz.
We downsample it to 44.1 kHz to ﬁt our experiments.
Dereverberation The test set for dereverberation, DEREV, is simulated using VCTK-Test and RIRTest.
For each utterance in VCTK-Test, we ﬁrst randomly select an RIR from RIR-Test, then we calculate the convolution between the RIR and utterance to build the reverberant speech.
Finally, we build 2937 reverberant and target data pairs.
Declipping DECLI, the evaluation set for declipping, is also constructed based on VCTK-Test.
We perform clipping on VCTK-Test following the equation in Section 2 and choose 0.25, 0.1 as the two setups for the clipping ratio.
This result in two declipping test sets with different levels, each containing 2937 clipped speech and target audios.
General speech restoration To evaluate the performance on GSR, we simulate a test set ALLGSR comprising of speech with all kinds of distortion.
The clean speeches and noise data used to build ALL-GSR is VCTK-Test and DCASE-Eval.
The simulation procedure of ALL-GSR is almost the same to the training data simulation described in Section 4.2.
In total, 501 three seconds long utterances are produced in this test set.
MOS Evaluation
We select a small portion from the test sets to carry out MOS evaluation for each one.
In SR, DECLI, and DEREV, we select 38 utterances out for human ratings.
In DENOISE and
ALL-GSR, we randomly choose 42 and 51 utterances.
C. E VALUATION METRICS
Log-spectral distance LSD is a commonly used metrics on the evaluation of super-resolution performance (Kumar et al., 2020; Lee & Han, 2021; Wang & Wang, 2021).
For target signal s and output estimate ˆ s , LSD can be computed as Equation 27, where S and ˆ S stand for the magnitude spectrogram of s and ˆ s .
[Equation]
Perceptual evaluation of speech quality PESQ is widely used in speech restoration literature as their evaluation metrics (Pascual et al., 2017; Hu et al., 2020).
It was originally developed to model the subjective test commonly used in telecommunication.
PESQ provides a score ranging from -0.
to 4.5 and the higher the score, the better quality a speech has.
In our work, we used an open-sourced implementation of PESQ to compute these metrics.
Since PESQ only works on a 16 kHz sampling rate, we performed a 16 kHz downsampling to the output 44.1k audio before evaluation.
Structural similarity SSIM (Wang et al., 2004) is a metrics in image super-resolution.
It addresses the shortcoming of pixel-level metrics by taking the image texture into account.
We match the implementation of SSIM in (Wang et al., 2004) with ours and compute SSIM as Equation 28, where µ S and σ S is the mean and standard deviation of S .
Cov S
ˆ S is the Covariance of S and ˆ S .
(cid:15) 1 = 0 .
and (cid:15) 2 = 0 .
02 are two constant used to avoid zero division.
Similarity is measured within the K 7*7 blocks divided from S and ˆ S .
[Equation]
Scale-invariant signal to noise ratio SiSNR (Le Roux et al., 2019) is widely used in speech restoration literatures to compare the energy of a signal to its background noise.
SiSNR is calculated between target waveform s and waveform estimation ˆ s :
[Equation]
where ˆ s target = <
ˆ s , s > s (cid:107) ˆ s (cid:107) 2 .
A higher SiSNR indicates less discrepancy between the estimation and target.
Scale-invariant spectrogram to noise ratio SiSPNR is a spectral metric similar to SiSNR.
They have the similar idea except SiSPNR is computed on the magnitude spectrogram.
Given the target spectrogram S and estimation ˆ S , the computation of SiSPNR can be formulated as
[Equation]
[Equation] ) where ˆ S target = < ˆ S , S > S [Equation] ˆ S (cid:107) 2 .
The scale invariant is guranteed by mean normalization of estimated and target spectrogram.
[Equation]
A
[Equation]
D
[Table]
D. E VALUATION RESULTS
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
/ 24kHz
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
D
[Table]
A NALYSIS STAGE PERFORMANCE
In this section, we report the mel spectrogram restoration score on different test sets.
They are used to evaluate the performane of the analysis stage.
We calculate the LSD, SiSPNR, and SSIM values.
[Table]
The Unprocessed column is calculated using the target and unprocessed mel spectrogram.
And
the
Oracle-Mel column is calculated using the target spectrogram and itself
[Table] [Table]
Table 10 shows that on DENOISE, DEREV
[Table]
and ALL-GSR, all four VoiceFixer based models are effective on the restoration of mel spectrogram.
Among the four analysis stage models, UNet is consistently better than the other three models
[Table] [Table]
Table 11 lists the mel restoration performance on different sampling rates.
Although VF-BiGRU has fewer parameters than VF-UNet , it still achieved the highest score on LSD and SiSPNR averagely.
This result indicates that the recurrent structure might be more suitable for the mel spectrogram super-resolution task when the initial sampling rate is high.
D. D EMOS
In this section, we provide some restoration demos using our proposed VoiceFixer .
In Figure 12, we provide eight restoration demos using our VF-UNet model.
All the audios used in our demo are either collected from the internet or recorded by ourselves.
In each example, the left is the unprocessed spectrogram, and the right is the restored one.
After restoration, these seriously distorted speech signals can be revert to relatively high quality.
Figure 12b is the speech we recorded using Adobe Audition.
We set the sampling rate to 8 kHz and manually add the clipping effect.
It also contains some low-frequency noise and reverberation introduced by the recording device and environment.
Figure 12a is a speech delivered by Amelia
Earhart 3 , 1897-1937, appeared in the Library of Congress, United States.
The original version sounds like a mumble.
Figure 12f comes from an interview in a TV news program, which includes multiple distortions.
Figure 12e is collected from the audio uploaded by a Youtuber 4 .
Probably due to the recording device, her speech is deteriorated seriously by noise, and the energy of speech in the low-frequency part is also relatively low.
Figure 12c is the restoration of a Chinese famous old movie railroad guerrilla 5 , which speech has limited bandwidth.
The audio in Figure 12d is selected from a well-known TV series in China, romance of the three kindoms 6 , in which some parts of the spectrogram are masked off due to the previous audio compression.
Figure 12g is a recording 7 selected from a speech delivered by Sun Yat-sen , 1866-1925, which is in extremely lowresolution and includes multiple unknown distortions.
Figure 12h shows the result of a subway broadcasting we recorded in Shanghai.
The low-frequency part of speech almost lost completely, and the reverberation is serious.
To sum up, all these examples showcase the effectiveness of the VoiceFixer model on GSR.
And to our surprise, it can generate will on unseen distortions such as the spectrogram lost in Figure 12c, Figure 12f, and Figure 12d.
In addition, Figure 12e shows that VoiceFixer is effective for the compensation of low-frequency energy, making speech sound less machinery and distant.
Last but not least, despite the abnormal harmonic structure in the low-frequency part in Figure 12g, our proposed model can still restore its sound, which display the advantages of utilizing the prior knowledge of vocoder.
[Figure] [Figure]