A DVERSARIAL A UDIO S UPER -R ESOLUTION WITH U NSUPERVISED F EATURE L OSSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural network-based methods have recently demonstrated state-of-the-art results on image synthesis and super-resolution tasks, in particular by using variants of generative adversarial networks (GANs) with supervised feature losses.
Nevertheless, previous feature loss formulations rely on the availability of large auxiliary classiﬁer networks, and labeled datasets that enable such classiﬁers to be trained.
Furthermore, there has been comparatively little work to explore the applicability of GAN-based methods to domains other than images and video.
In this work we explore a GAN-based method for audio processing, and develop a convolutional neural network architecture to perform audio super-resolution.
In addition to several new architectural building blocks for audio processing, a key component of our approach is the use of an autoencoder-based loss that enables training in the
GAN framework, with feature losses derived from unlabeled data.
We explore the impact of our architectural choices, and demonstrate signiﬁcant improvements over previous works in terms of both objective and perceptual quality.
I NTRODUCTION
Deep convolutional neural networks (CNNs) have become a cornerstone in modern solutions for image and audio analysis.
Such networks have excelled at supervised descrimination tasks, for instance on ImageNet (Deng et al., 2009; Simonyan & Zisserman, 2014), where image classiﬁer networks are trained on a large corpus of labeled data.
More recently, CNNs have successfully been applied to data synthesis problems in the context of generative adversarial networks (GANs) (Goodfellow et al., 2014).
In the GAN framework, a neural network is used to synthesize new instances from a modeled distribution, or resolve missing details given lossy observations.
In the latter case, the GANs have been shown to greatly improve reconstruction of ﬁne texture details for images, compared to standalone sample-space losses that result in overly smoothed outputs (Dosovitskiy & Brox, 2016; Isola et al., 2017; Ledig et al., 2017).
However, GANs are notoriously hard to train, and the use of conventional sample-space objectives in conjunction with an adversarial loss either de-stabilizes training, or results in outputs with signiﬁcant artifacts (Figure 1).
To address the smoothness problem described above, previous works typically augment or replace conventional sample-space losses with a feature loss (also called a perceptual loss)
(Dosovitskiy & Brox, 2016; Ledig et al., 2017; Johnson et al., 2016).
Instead of distance in raw sample-space, such feature losses reﬂect distance in terms of the feature maps of an auxiliary neural network.
While classiﬁer-based feature losses are effective, they require either a pre-trained neural network that is applicable to the problem domain (e.g., synthesizing images of cats), or a labeled dataset that is amenable to training a relevant classiﬁer.
Training new classiﬁers for use in a feature loss can be non-trivial for numerous reasons.
Besides the difﬁculty of training large classiﬁers that are commonly used for feature losses, such as VGG (Simonyan & Zisserman, 2014), creating a labeled dataset that is sufﬁciently large and diverse is often infeasible.
In this work, we sidestep the difﬁculty of training auxiliary classiﬁers by developing a feature loss that is unsupervised.
In particular, we focus on an audio modeling task called super-resolution, where the goal is to generate high-quality audio given down-sampled, low-resolution input.
Inspired by previous work on audio and image super-resolution, we develop a neural network architecture for Audio samples available: https://sites.google.com/view/unsupervised-audiosr/home
[Figure] end-to-end super-resolution that operates on raw audio.
In addition to providing new algorithms to model audio, our work suggests new techniques to improve GAN-based methods in other domains such as images and video. Speciﬁcally, our contributions are as follows:
We formulate a new, general-purpose feature loss that is fully unsupervised and circumvents the need for problematic classiﬁer-based models.
We successfully adapt the adversarial framework for audio processing, and provide solutions to previously unsolved problems associated with the application of GANs to audio.
We demonstrate our methods in an end-to-end architecture for audio super-resolution, with stateof-the-art results on both speech and music tasks.
We analyze important architectural parameters of our model, and in particular discover previously-unobserved behavior with effective receptive ﬁeld sizes.
B ACKGROUND & R ELATED W ORK
Audio super-resolution Audio super-resolution is the task of constructing a high-resolution audio signal from a low-resolution signal that contains a fraction of the original samples.
Concretely, given a low-resolution sequence of audio samples x l = ( x 1 /R l , . . .
, x R l T/R l ) , we wish to synthesize a high-resolution audio signal x h
= ( x 1 /R h , . . .
, x R h T/R h ) , where R l and R h are the sampling rates of the low and high-resolution signals, respectively.
We denote R
= R h
/R l as the upsampling ratio , which ranges from 2 to 6 in this work.
Thus, the audio super-resolution problem is equivalent to reconstructing the missing frequency content between frequencies R l / 2 and R h / 2 .
There is a vast body of prior work on audio super-resolution in the signal and audio processing communities under the term artiﬁcial bandwidth extension (Larsen & Aarts, 2004).
Neural networkbased methods in this domain generally apply a DNN on top of hand-crafted features as part of complex bandwidth extension systems (Liu et al., 2015; Abel & Fingscheidt, 2018).
Gaussian mixture and hidden Markov models have also been used (Bachhav et al., 2017; Tokuda et al., 2013), but these methods generally perform worse compared to neural networks (Abel & Fingscheidt, 2018).
In contrast with the works above, our method does not rely on hand-crafted features (e.g., transformations or cepstrum coefﬁcients), and is not speciﬁc to problems in speech modeling.
Audio modeling with neural networks Learning-based approaches for audio have also been explored in the largely in the context of representation learning, generative modeling, and text-tospeech (TTS) systems.
Unsupervised methods such as convolutional deep belief networks (Lee et al., 2009) and bottleneck CNNs (Aytar et al., 2016) have been shown to learn useful representations from audio, such as phonemes and sound textures.
Stacked autoencoders (Vincent et al., 2010) and variational autoencoders (Kingma & Welling, 2014; Sonderby et al., 2016) have been used for denoising, image generation, and music synthesis (Sarroff & Casey, 2014).
Bottleneck-like CNNs have also demonstrated signiﬁcant improvements for audio super-resolution in supervised settings compared to previous DNN and spline-based methods (Kuleshov et al., 2017).
Donahue et al.
(2018) is among the ﬁrst to develop methods for raw audio synthesis with GANs.
Notably, Donahue et al.
(2018) show that non-trivial modiﬁcations of GAN architectures are required to generative diverse and plausible audio outputs.
We build on the works above by developing a GAN framework for audio super-resolution with an improved bottleneck-style generator, and show that leveraging representations learned from unsupervised training greatly aid the super-resolution task.
Autoregressive probabilistic models have recently demonstrated state-of-the-art results for generation of music (Engel et al., 2017), general audio (van den Oord et al., 2016; Mehri et al., 2017), and for parametric TTS systems (Sotelo et al., 2017). Several works have leveraged model distillation (van den Oord et al., 2018; ClariNet, 2019) 1 to mitigate the overhead of autoregressive methods, making them feasible for real-time audio generation. In general, our work can be used to augment existing speech synthesis systems, including those that employ autoregressive methods. For instance, the unsupervised feature loss proposed in our work could be used as a drop-in replacement for the classiﬁer-based feature loss used by van den Oord et al.
While we are not aware of any efforts that explore autoregressive modeling for audio super-resolution, we believe it may be a promising future direction.
Generative adversarial networks for images Generative methods have been extensively explored for image generation and super-resolution.
Building upon the original formulation of Goodfellow et al.
(2014), GANs have been continuously improved to generate plausible, high-ﬁdelity images (Radford et al., 2015; Denton et al., 2015; Berthelot et al., 2017; Karras et al., 2018).
GAN variants conditioned on class labels or object sketches have also demonstrated promising results on tasks such as in-painting and style transfer (Mirza & Osindero, 2014; Isola et al., 2017).
M ETHOD
GANs for Super-Resolution GANs developed for super-resolution tasks have several important differences compared to the original formulation of Goodfellow et al.
When used to generate new instances from a data distribution p data , the generator ( G ) parameterized by θ G learns the mapping to data space as G ( z ; θ G ) , where z is a latent noise prior.
The discriminator ( D ) parameterized by θ D then estimates the probability that G ( z ; θ G ) was drawn from p data rather than the generator distribution p g .
In contrast, for super-resolution, G is no longer conditioned on noise and learns the mapping to high-resolution data space p h as G ( x l ; θ G ) , where x l is drawn from the low-resolution data distribution p l .
The task of D is to discriminate between samples from the high-resolution and super-resolution (generator) distributions p h and p g , respectively.
Since low-resolution data x l corresponds directly to a downsampled version of x h during training, we expect G ( x l ; θ G )
≈ x h .
G and D are optimized according to the two-player minimax problem:
[Equation]
This framework enables the joint optimization of two neural networks - G generates super-resolution data with the goal of fooling D , and D is trained to distinguish between real and super-resolved data.
Thus, the GAN approach encourages G to learn solutions that are hard to distinguish from real, highresolution datum.
Architecture overview MU-GAN (Multiscale U-net GAN) is composed of three models - a generator ( G ), discriminator ( D ), and convolutional autoencoder ( A ) (Figure 2).
The generator’s task is to learn the mapping between the low and high-resolution data spaces, corresponding to signals x l and x h , respectively.
The discriminator’s task is then to classify whether presented data instances are real, or produced by the generator.
In addition to G and D , the autoencoder extracts perceptuallyrelevant features from both real and super-resolved data for use in feature-space loss functions.
The use of A is crucial in the GAN framework, as generators trained solely on L2 or other sample-space losses suffer from training instability or output artifacts (Ledig et al., 2017).
Multiscale convolutional layers In comparison to images, audio signals are inherently periodic with time-scales on the order of 10’s to 100’s of samples.
As a consequence, ﬁlters with very large receptive ﬁelds are required to create high quality, raw audio (Donahue et al., 2018; van den Oord et al., 2016). Previous work with classiﬁer models also suggests that varying the ﬁlter size within [Figure] Subpixel
Subpixel Inverse (Superpixel) ... [Figure] [Figure] a network helps capture information at multiple scales (Szegedy et al., 2015).
Leveraging these observations, we use a multiscale convolutional building block composed of concatenated 3x1, 9x1, 27x [Figure] and 81x1 ﬁlters. In practice
[Figure] and with a ﬁxed number of parameters for a given layer, we found that ﬁlters larger than 81x1 provided no additional beneﬁt [Figure] while omitting large ﬁlter sizes resulted in signiﬁcantly degraded audio quality.
We interpret the poor performance of small ﬁlters as being a byproduct of their frequency selectivity; it is well known from signal processing theory that the resolution of an FIR ﬁlter’s frequency response is proportional to the length of the ﬁlter.
Superpixel layers Recently, it has been shown that pooling and strided convolutions tend to induce periodic “checkerboard” artifacts (Odena et al., 2016; Donahue et al., 2018).
Shi et al.
(2016) developed a subpixel layer to increase spatial resolution, and showed that it is less prone to checkerboard artifacts.
While the subpixel layer was subsequently adopted by several works (Kuleshov et al., 2017), no efforts have evaluated the performance of the inverse operation for decreasing spatial resolution.
Concretely, the inverse subpixel operator interleaves samples from the time dimension into the channel dimension, and reduces the spatial resolution by an integer factor. We refer to this simple inverse operation as a superpixel layer (Figure 3), and use it as a drop-in replacement for strided convolution and pooling layers.
Generator network The high-level architecture for the generator network (Figure 4, top) is inspired by autoencoder-like U-net models (Ronneberger et al., 2015; Isola et al., 2017; Kuleshov et al., 2017).
In a U-net-style model, the ﬁrst half of the network consists of B downsampling blocks (D-blocks) that perform feature extraction at multiple scales and resolutions 2 .
The second half the model consists of B upsampling blocks (U-blocks), which successively increase the spatial resolution of the signal.
We use multi-scale convolutional layers throughput the generator network, and replace all strided convolutions with superpixel layers.
Descriminator network The Descriminator (Figure 4, bottom) is used during training to differentiate between real, high-resolution audio and super-resolved signals produced by the generator.
Our design is loosely based on the recommendations of Radford et al. (2015), and the image discriminator from Ledig et al.
All discriminator activations are LeakyReLU
(Maas et al., 2013) with α = 0 .
As with the generator, we use multi-scale convolutions, and the superpixel layer described above instead of strided convolutions to minimize artifacts in the loss gradients (Odena et al., 2016).
Autoencoder network
The autoencoder A is used to extract perceptually relevant features from the low and high-resolution signals.
The features extracted by A are incorporated in the generator’s feature loss L f , which is described in more detail in following sections.
For the speciﬁc implementation of A , we use a modiﬁed version of the generator model that excludes all additive and stacking skip connections.
Hence, the model for A is a convolutional autoencoder, augmented with multiscale convolutional layers, and super/subpixel layers for down/up-sampling.
Loss functions MU-GAN incorporates several loss terms for training the generator and discriminator.
The ﬁrst term in the generator loss is the sample-space L2 loss, given by
[Equation]
We found that using only the sample-space and adversarial losses either resulted in little to no improvement over the baseline non-GAN model, or introduced persistent audible artifacts (e.g., high-frequency tones, Figure 1).
These ﬁndings are in line with those of Ledig et al.
(2017), who experience similar issues with images 4 .
As described in Section 2, the use of a feature loss with GAN training encourages the generator to learn solutions that incorporate perceptually relevant texture details.
Given the autoencoder A , we denote the output feature tensor at the bottleneck of the autoencoder as φ .
The feature loss L f is then given by
[Equation]
where W f and C f denote the width and channel dimensions for the feature maps of autoencoder bottleneck.
The adversarial loss L adv is determined by discriminator’s ability to discern whether data produced by the generator is real or fake.
We use the gradient-friendly formulation originally posed in Goodfellow et al.
(2014), given by
[Equation]
The composite loss L G for the generator is then given by the sum of the losses above, and the discriminator loss L D derives directly from the GAN optimization objective in Equation 1, i.e.,
[Equation]
where λ f and λ adv are constant scaling factors.
E XPERIMENTS
Datasets We evaluate our methods on three super-resolution tasks derived from the VCTK Corpus (Yamagishi), and the non-vocal music dataset from Mehri et al.
For speech from VCTK, we compose a dataset with recordings from a single speaker (the Speaker1 task), and a dataset with recordings from multiple speakers (the Speaker99 task).
Speaker1 consists of the ﬁrstrecordings from VCTK speaker 225 for training, and the ﬁnal 8 recordings for testing.
Speaker uses all recordings from the ﬁrst 99 VCTK speakers for training, and recordings from the lastspeakers for testing.
Piano uses the standard 88%-6%-6% train/validation/test split.
For all tasks, the dataset is created by ﬁrst applying an anti-aliasing lowpass ﬁlter, and then sampling random patches of ﬁxed length from the resulting audio.
Note that for the sake of direct comparison, the datasets above are the same as those used in Kuleshov et al.
Training methodology For Speaker1 , we instantiate variants of MU-GAN and train forepochs.
For the larger datasets Speaker99 and Piano , models are trained for 150 epochs.
The epoch number is empirically selected based on observed convergence, and performance saturation on the validation set.
For all models, we use the ADAM optimizer with learning rate 1e-4, β 1 = 0 .
9 , β 2 = 0 .
999 , and a batch size of 32.
For the autoencoder feature losses, we instantiate a model with L = 4 , and train for 400 epochs on the same dataset as its associated GAN model.
The loss scaling factors λ f and λ adv are ﬁxed at 1.0 and 0.001, respectively.
Additional details on model hyperparameters can be found in Appendix A.1. Performance metrics We use three metrics to assess the quality of super-resolved audio: (1) signal-to-noise ratio (SNR), (2) log-spectral distance (LSD), and (3) mean opinion score (MOS).
The SNR is a standard metric in signal processing communities, deﬁned as
[Equation]
(7) where x is an approximation of reference signal x ref .
LSD (Gray & Markel, 1976) measures differences between signal frequencies, and has better correlation with perceptual quality compared to SNR (Jie et al., 2014; Kuleshov et al., 2017).
Given short-time discrete Fourier transforms X and X ref , the LSD is given by
[Equation]
where w and k are the window and frequency bin indices, respectively 5 .
Perceptual evaluation of speech quality (PESQ) (ITU-T, 2001) is an industry-standard methodology for the assessment of speech communication systems.
Given reference and degraded audio signals, PESQ models the mean opinion score (MOS) of a group of listeners.
Speciﬁcally, we use PESQ to produce MOS-LQO (listening quality objective) scores (ITU-T, 2003), which range from 1 to 5.
Impact of superpixel layers We ﬁnd that the use of superpixel layers results in ∼ 14% improvement in training time across model sizes, with insigniﬁcant differences in terms of objective quality metrics.
Differences in audio produced by the two methods were also imperceptible in [-Strided . We train both model types with the baseline Lloss for 400 epochs on the Speaker1 task. Table 1a shows the average wall time per minibatch with both L = 4 and L = 
-]informal selfblinded listening tests.
This indicates that superpixel layers may be a suitable replacement for conventional strided convolutions, while offering improvements in training time without performance loss.
Additional details are provided in Appendix A.2.
Objective performance evaluation Table 1 shows the quantitative performance of MU-GAN against other recent works.
We denote MU-GAN8 as an instance of MU-GAN with a depth parameter of L = 8 , i.e., with 8 downsampling and 8 upsampling blocks.
U-net4 is the model with L = 4 from Kuleshov et al.
To eliminate depth as a factor in the performance comparison, we reimplement the architecture from Kuleshov et al.
(2017) with L = 8 , denoted as U-net8 .
1 shows that MU-GAN8 often performs worse in terms of SNR compared to the baseline models, but has lower LSD and higher MOS-LQO.
This indicates that while MU-GAN8 produces reconstructions with lower SNR, deviating in terms of sample-wise distance results in synthesis of more perceptually-relevant frequency content.
The exception is with the Piano task, where MUGAN8 performs orders of magnitude better than the U-net baseline in terms of SNR. In general, we
[Table] [Table] [Table]
also ﬁnd that performance on the speech tasks generally saturates at R = 2 for both U-net8 and MUGAN8 . Informal listening tests conﬁrm that there are minimal differences at R = 2 , indicating that more difﬁcult up-sampling ratios (i.e., R = 4 , 6 ) are better suited for grounds of further comparison.
Subjective quality analysis To evaluate the performance of MU-GAN with real listeners, we perform a randomized, single-blinded user study with 22 participants (Table 2). The study presents pairs of audio clips produced by MU-GAN8 and the best baseline model U-net8 , and asks participants to select a preferred clip, or “No preference.” We present two clips from Piano , and four sonically diverse clips from Speaker99 . Table 2 shows that in all cases, listeners prefer audio produced by
MU-GAN8 over the baseline method. In general, we observe that audio produced by MU-GAN has greater clarity compared to audio produced by the baseline networks. The quality difference is most apparent during consonant sounds, which have more high-frequency content compared to typical vowel sounds.
For instance, in the phrase “Ask her to bring these things from the store,” (Figure 5, bottom row) the consonant sounds in ‘Ask,’ ‘things,’ and ‘store’ have noticeably better articulation.
In contrast, audio from the best baseline, U-net8 , sounds relatively dull and “mufﬂed” in comparison. Note that on some audio clips super-resolved at R = 4 , 6 , we observe intermittent, audible noise that is not present in the baseline reconstructions. To ensure a fair analysis, we include audio clips in the user study where this noise is apparent. Additional comments on spurious noise can be found in Appendix A.3.
Comparison with classiﬁer-based feature loss We compare the proposed unsupervised feature loss with the classiﬁer-based loss from Germain et al.
Germain et al.
(2018) use a VGGbased (Simonyan & Zisserman, 2014) network as a feature loss for speech denoising, and train the loss network on classiﬁcation and audio tagging tasks from DCASE 2016 (Mesaros et al., 2018).
Table 3 shows objective results obtained using the proposed loss L f and the classiﬁer-based loss L f,SV on the Speaker1 task.
Across all up-sampling ratios, the proposed unsupervised method performs on-par (and slightly better in some cases) compared to the classiﬁer-based loss.
Thus, our results suggest that using a domain-speciﬁc classiﬁer-based loss may not provide any advantage in terms of performance.
Given the issues related to training classiﬁer models on general audio (Section 1), our method may be an attractive solution that does not compromise audio quality.
[Figure]
[Table]
Ablation Analysis Table 4 shows the MOS-LQO metrics for the MU-GAN architecture with ablated model parameters.
While almost all variations perform similarly well at R = 2 , adding depth (i.e., from L = 4 to L = 8 ) and additional loss terms improves performance on harder up-sampling ratios.
Furthermore, adding the adversarial loss and unsupervised feature loss terms improve MOSLQO monotonically.
For MU-GAN8 , we see diminishing returns from adding the additional loss terms; much of the improvement over MU-GAN4 appears to come from the additional depth.
On the other hand, adding the L f and L adv losses to the MU-GAN4 variant yields signiﬁcant beneﬁts, such that its performance is comparable to that of MU-GAN8 .
This indicates that the feature and adversarial losses may be particularly useful to mitigate underﬁtting, or to decrease model size iso-performance.
Receptive Field Analysis
While previous works have stressed the importance of large receptive ﬁelds (van den Oord et al., 2016; Luo et al., 2016; Yu & Koltun, 2016), little work exists to quantify receptive ﬁeld sizes on practical problems.
We use the methods of Luo et al.
(2016) to measure the effective receptive ﬁeld (ERF) 7 of our model on different tasks with varying architectural hyperparameters.
Notably, we ﬁnd that while the theoretical receptive ﬁeld of our network is on the order of hundreds of milliseconds (thousands of samples), measured ERF’s are generally no wider than 100 samples.
Furthermore, we ﬁnd that ERF size is strongly correlated with problem difﬁculty, rather than architectural hyperparameters such as depth and speciﬁc loss terms.
While Luo et al.
(2016) found that ERF size always increased compared to the ERF at initialization, we ﬁnd that
ERF decreases in many cases (Figure 6).
Our ﬁndings imply that there may be subtle but important tradeoffs involved with receptive ﬁeld size (for instance, ERF size versus noise rejection), and suggest promising avenues for deeper investigation of architectures that rely on large receptive ﬁelds.
[Figure]  C ONCLUSION In this work we develop methods to enable the application of GANs to audio processing, in particular with classiﬁer-free feature losses.
In addition to several new model building blocks, we show that a convolutional autoencoder can be used to implement a high-performance feature loss in the context of audio super-resolution.
Demonstrated on several speech and music super-resolution tasks, we show that our architecture achieves state-of-the-art performance in both objective and subjective metrics. We perform a detailed analysis of our model and also show that effective receptive ﬁeld size may be an important property that is not well-explored. Finally, our work raises new possibilities for the design and analysis of neural network-based synthesis methods in important problem domains beyond audio processing A.
E VALUATION OF SUPERPIXEL LAYERS
We evaluate the impact of the superpixel layer proposed in Section 3 by comparing a baseline multiscale U-net with strided convolutions ( Strided ) against a multiscale U-net with superpixel layers ( Super ). We halve the number of convolutional kernels in each downsampling layer for Super such that the output feature map dimensions at each downsampling and upsampling layer are identical to those in Strided . Both model types are trained with the baseline L 2 loss for 400 epochs on the Speaker1 task.
[Table]
N OTE ON INTERMITTENT SPURIOUS NOISE
As described in Section 4, some audio clips super-resolved at R = 4 , 6 exhibit intermittent, audible noise that is not present in the baseline reconstructions.
We ﬁnd that this noise is not inherent to our speciﬁc feature loss
[Table]
and is present if we replace the unsupervised feature loss with a classiﬁerbased feature loss.
The spurious noise is also unrelated to the use of L adv , as it is present in audio generated by models trained only with L f .
While a deeper investigation is planned for future work, we hypothesize that such noise stems from phase ambiguity in missing high-frequency content.