A
DVERSARIAL A UDIO S YNTHESIS
Anonymous authors
Paper under double-blind review
A BSTRACT
While Generative Adversarial Networks (GANs) have seen wide success at the problem of synthesizing realistic images, they have seen little application to audio generation.
Unlike for images, a barrier to success is that the best discriminative representations for audio tend to be non-invertible, and thus cannot be used to synthesize listenable outputs.
In this paper we introduce WaveGAN, a ﬁrst attempt at applying GANs to unsupervised synthesis of raw-waveform audio.
WaveGAN is capable of synthesizing one second audio waveforms with temporal coherence, suitable for sound effect generation. Our experiments demonstrate that WaveGAN can produce intelligible words from a small vocabulary of speech, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano.
I NTRODUCTION
Synthesizing audio for speciﬁc domains has many practical applications in music and ﬁlm production.
Musicians and Foley artists scour large databases of sound effects to ﬁnd particular audio recordings suitable for speciﬁc scenarios.
This strategy is painstaking and may result in a negative outcome if the ideal sound effect does not exist in the library.
A better approach might allow a sound artist to explore a compact latent space of audio, taking broad steps to ﬁnd the types of sounds they are looking for (e.g. footsteps) and making small adjustments to latent variables to ﬁne-tune (e.g. a large boot lands on a gravel path).
However, audio signals have high temporal resolution, and strategies that learn such a latent representation must perform effectively in high dimensions.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are one such unsupervised strategy for mapping low-dimensional latent vectors to high-dimensional data.
The potential advantages of GAN-based approaches to audio synthesis are numerous.
Firstly, GANs could be useful for data augmentation (Shrivastava et al., 2017) in data-hungry speech recognition systems.
Secondly, GANs could enable rapid and straightforward sampling of large amounts of audio.
Furthermore, while the usefulness of generating static images with GANs is arguable, there are many applications (e.g. Foley) for which generating sound effects is immediately useful.
But despite their increasing ﬁdelity at synthesizing images (Radford et al., 2016; Berthelot et al., 2017; Karras et al., 2018), GANs have yet to be demonstrated capable of synthesizing audio in an unsupervised setting.
A na¨ıve solution for applying image-generating GANs to audio would be to operate them on imagelike spectrograms , i.e., time-frequency representations of audio.
This practice of bootstrapping image recognition algorithms for audio tasks is commonplace in the discriminative setting (Hershey et al., 2017).
In the generative setting however, this approach is problematic as the most perceptually-informed spectrograms are non-invertible, and hence cannot be listened to without lossy estimations (Grifﬁn & Lim, 1984) or learned inversion models (Shen et al., 2018).
Recent work (Oord et al., 2016; Mehri et al., 2017) has shown that neural networks can be trained with autoregression to operate on raw audio .
Such approaches are attractive as they dispense with engineered acoustic feature representations.
However, unlike with GANs, the autoregressive setting results in slow generation as output audio samples must be fed back into the model one at a time.
In this work, we investigate both waveform and spectrogram strategies for generating one-second slices of audio with GANs.
For our spectrogram approach (SpecGAN), we ﬁrst design an appropriate spectrogram representation that allows for approximate inversion, and bootstrap the two- dimensional deep convolutional GAN (DCGAN) method (Radford et al., 2016) to operate on these spectrograms.
In WaveGAN, our waveform approach, we ﬂatten the SpecGAN architecture to operate in one dimension, resulting in a model with the same number of parameters and numerical operations as its two-dimensional analog.
With WaveGAN, we provide both a starting point for practical audio synthesis with GANs and a recipe for modifying other image generation methods to operate on waveforms.
We primarily envisage our method being applied to the generation of short sound effects suitable for use in music and ﬁlm.
For example, we train a WaveGAN on drums, resulting in a procedural drum machine designed to assist electronic musicians (web demo https://bit.ly/2xINSu0 ).
While our objective is sound effect generation (e.g. generating drum sounds), human evaluation for these tasks would require expert listeners.
Therefore, we also consider a speech benchmark, facilitating straightforward assessment by human annotators.
Speciﬁcally, we explore a task where success can easily be judged by any English speaker: generating spoken digits “zero” through “nine”.
Though our evaluation focuses on a speech generation task, we note that it is not our goal to develop a text-to-speech synthesizer.
Instead, our investigation concerns whether unsupervised strategies can learn the semantic modes (e.g. words in speech data) implicit in high-dimensional audio signals, rather than being conditioned on them.
Our experiments on speech demonstrate that both WaveGAN and SpecGAN can generate spoken digits that are intelligible to humans.
On criteria of sound quality and speaker diversity, human judges indicate a preference for the audio generated by WaveGAN compared to that from SpecGAN.
GAN P RELIMINARIES
GANs learn mappings from low-dimensional latent vectors z ∈ Z , i.i.d.
samples from known prior
P Z , to points in the space of natural data X .
In their original formulation (Goodfellow et al., 2014), a generator G : Z (cid:55)→ X is pitted against a discriminator D : X (cid:55)→ [0 , 1] in a two-player minimax game.
G is trained to minimize the following value function, while D is trained to maximize it:
[Equation]
In other words, D is trained to determine if an example is real or fake, and G is trained to fool the discriminator into thinking its output is real.
Goodfellow et al.
(2014) demonstrate that their proposed training algorithm for Equation 1 equates to minimizing the Jensen-Shannon divergence between P X , the data distribution, and P G , the implicit distribution of the generator when z ∼ P Z .
In their original formulation, GANs are notoriously difﬁcult to train, and prone to catastrophic failure cases such as mode collapse, in which the generator outputs a single result for all z .
Arjovsky et al.
(2017) demonstrate that, under certain conditions, minimizing Jensen-Shannon divergence of P X and P G with gradient descent is ill-posed as it is discontinuous and provides no usable gradient.
As a smoother alternative, they suggest minimizing the Wasserstein-1 distance between distributions
[Equation]
[Equation] where (cid:107) f (cid:107) L ≤ 1 : X (cid:55)→ R is the family of functions that are 1 -Lipschitz.
To minimize Wasserstein distance, they suggest a GAN training algorithm (WGAN), similar to that of Goodfellow et al.
(2014), for the following value function:
[Equation]
With this formulation, D w : X (cid:55)→ R is not trained to identify examples as real or fake, but instead is trained as a function that assists in computing the Wasserstein distance.
Arjovsky et al.
(2017) suggest weight clipping as a means of enforcing that D w is 1-Lipschitz.
As an alternative strategy, Gulrajani et al.
(2017) replace weight clipping with a gradient penalty (WGAN-GP) that also enforces the constraint.
They demonstrate that their WGAN-GP strategy can successfully train a variety of model conﬁgurations where other GAN losses fail.
[Figure]
DCGAN uses small ( 5 x 5 ), twodimensional ﬁlters while WaveGAN uses longer (length- 25 ), one-dimensional ﬁlters and a larger upsampling factor.
Both strategies have the same number of parameters and numerical operations.
W AVE GAN
We motivate our design choices for WaveGAN by ﬁrst highlighting the different types of structure found in audio versus images.
3. I NTRINSIC DIFFERENCES BETWEEN AUDIO AND IMAGES
One way to illustrate the differences between audio and images is by examining the axes along which these types of data vary most substantially, i.e. by principal component analysis.
In Figure 1, we show the ﬁrst eight principal components for patches from natural images and slices from speech.
While the principal components of images generally capture intensity, gradient, and edge characteristics, those from audio form a periodic basis that decompose the audio into constituent frequency bands.
In general, natural audio signals are more likely to exhibit periodicity than natural images.
As a consequence, correlations across large windows are commonplace in audio.
For example, in a waveform sampled at 16 kHz , a 440 Hz sinusoid (the musical note A 4 ) takes over 36 samples to complete a single cycle.
This suggests that ﬁlters with larger receptive ﬁelds are needed to process raw audio.
This same intuition motivated Oord et al.
(2016) in their design of WaveNet, which uses dilated convolutions to exponentially increase the model’s effective receptive ﬁeld with layer depth.
3. W AVE GAN ARCHITECTURE
We base our WaveGAN architecture off of DCGAN (Radford et al., 2016) which popularized usage of GANs for image synthesis.
The DCGAN generator uses the transposed convolution operation (Figure 2) to iteratively upsample low-resolution feature maps into a high-resolution image.
Motivated by our above discussion, we modify this transposed convolution operation to widen its receptive ﬁeld.
Speciﬁcally, we use longer one-dimensional ﬁlters of length 25 instead of two-dimensional ﬁlters of size 5 x 5 , and we upsample by a factor of 4 instead of 2 at each layer (Figure 2).
We modify the discriminator in a similar way, using length- 25 ﬁlters in one dimension and increasing stride from 2 to 4 .
These changes result in WaveGAN having the same number of parameters, numerical operations, and output dimensionality as DCGAN.
Because DCGAN outputs 64 x 64 pixel images — equivalent to just 4096 audio samples — we add one additional layer to the model resulting in 16384 samples, slightly more than one second of audio at 16 kHz .
This length is already sufﬁcient for certain sound domains (e.g. sound effects, voice commands), and future work adapting megapixel image generation techniques (Karras et al., 2018) could expand the output length to more than a minute.
We requantize the real data from its 16bit integer representation (linear pulse code modulation) to 32 -bit ﬂoating point, and our generator similarly outputs ﬂoating point waveforms.
A complete description of our model is in Appendix D.
In summary, we outline our modiﬁcations to the DCGAN (Radford et al., 2016) method which result in WaveGAN.
This straightforward recipe already produces reasonable audio, and further contributions outlined below serve to reﬁne results.
Flatten 2 D convolutions into 1 D (e.g. 5 x 5 2 D convolution becomes length- 25 1 D).
Increase the stride factor for all convolutions (e.g. stride 2 x 2 becomes stride 4 ).
Train using the WGAN-GP (Gulrajani et al., 2017) strategy.
[Figure]
Generative image models that upsample by transposed convolution (such as DCGAN) are known to produce characteristic “checkerboard” artifacts in images (Odena et al., 2016).
Periodic patterns are less common in images (Section 3.1), and thus the discriminator can learn to reject images that contain them.
For audio, analogous artifacts are perceived as pitched noise which may overlap with frequencies commonplace in the real data, making the discriminator’s objective more challenging.
However, the artifact frequencies will always occur at a particular phase, allowing the discriminator to learn a trivial policy to reject generated examples.
This may inhibit the overall optimization problem.
To prevent the discriminator from learning such a solution, we propose the phase shufﬂe operation with hyperparameter n .
Phase shufﬂe randomly perturbs the phase of each layer’s activations by − n to n samples before input to the next layer (Figure 3).
We apply phase shufﬂe only to the discriminator, as the latent vector already provides the generator a mechanism to manipulate the phase of a resultant waveform.
Intuitively speaking, we want the discriminator to be invariant to the phase of the waveform it is classifying.
S PEC GAN: G ENERATING SEMI - INVERTIBLE SPECTROGRAMS
While a minority of recent research in discriminative audio classiﬁcation tasks has used raw audio input (Sainath et al., 2015; Lee et al., 2017), most of these approaches operate on spectrogram representations of audio.
A generative model may also beneﬁt from operating in such a time-frequency space.
However, commonly-used representations in the discriminative setting are uninvertible.
With SpecGAN, our frequency-domain audio generation model, we design a spectrogram representation that is both well-suited to GANs designed for image generation and can be approximately inverted.
Additionally, to facilitate direct comparison, our representation is designed to use the same dimensionality per unit of time as WaveGAN ( 16384 samples yield a 128 x 128 spectrogram).
To process audio into suitable spectrograms, we ﬁrst perform the short-time Fourier transform with 16 ms windows and 8 ms stride, resulting in 128 frequency bins linearly spaced from 0 to 8 kHz .
We take the magnitude of the resultant spectra and scale amplitude values logarithmically to better-align with human perception.
We then normalize each frequency bin to have zero mean and unit variance.
This type of preprocessing is commonplace in audio classiﬁcation, but produce spectrograms with unbounded values.
We therefore clip the spectra to 3 standard deviations (we observed through an informal listening test that this value did not produce an audible difference during inversion) and rescale to [ − 1 , 1 ].
SC
Piano Drums
R ea l
W a v e
GAN
S p ec
GAN
TIMIT
TIMIT (detail)
Birds
Once our dataset has been processed into this format, we operate the DCGAN (Radford et al., 2016) algorithm on the resultant spectra.
To render the resultant generated spectrograms as waveforms, we ﬁrst invert the steps of spectrogram preprocessing described above, resulting in linear-amplitude magnitude spectra.
We then employ the iterative Grifﬁn-Lim algorithm (Grifﬁn & Lim, 1984) with 16 iterations to estimate phase and produce 16384 audio samples.
E XPERIMENTAL PROTOCOL
To facilitate human evaluation, our experimentation focuses on the Speech Commands Dataset (Warden, 2018).
This dataset consists of many speakers recording individual words in uncontrolled recording conditions.
We explore a subset consisting of the spoken digits “zero” through “nine” and refer to this subset as the Speech Commands Zero Through Nine (SC09) dataset.
These ten words encompass many phonemes and two consist of multiple syllables.
Each recording is one second in length, and we do not attempt to align the words in time.
There are 1850 utterances of each word in the training set, resulting in 5 .
3 hours of speech.
The heterogeneity of alignments, speakers, and recording conditions make this a challenging dataset for generative modeling.
Our baseline conﬁguration for WaveGAN excludes phase shufﬂe.
We compare this to the performance of WaveGAN with phase shufﬂe ( n ∈ { 2 , 4 } ) and a variant of WaveGAN which uses nearest-neighbor upsampling rather than transposed convolution (Odena et al., 2016).
Hoping to reduce noisy artifacts, we also experiment with adding a wide (length- 512 ) post-processing ﬁlter to the output of the generator and learning its parameters with the rest of the generator variables (details in
Appendix B).
We use the WGAN-GP (Gulrajani et al., 2017) algorithm for all experiments, ﬁnding it to produce reasonable results where others (Radford et al., 2016; Mao et al., 2017; Arjovsky et al., 2017) failed.
We compare the performance of these conﬁgurations to that of SpecGAN.
We also perform experiments on four other datasets with different characteristics (Figure 4):
Drum sound effects ( 0 .
7 hours): Drum samples for kicks, snares, toms
, and cymbals
Bird vocalizations ( 12 . 2 hours): In-the-wild recordings of many species (Boesman, 2018) 3.
Piano ( 0 . 3 hours): Professional performer playing a variety of Bach compositions 4.
Large vocab speech (TIMIT) ( 2 . 4 hours): Multiple speakers, clean (Garofolo et al., 1993)
We train our networks using batches of size 64 on a single NVIDIA P100 GPU.
During our quantitative evaluation of SC09 (discussed below), our WaveGAN networks converge by their early stopping criteria (inception score) within four days ( 200 k iterations, equivalent to 700 epochs), and produce speech-like audio within the ﬁrst hour of training.
Our SpecGAN networks converge more quickly, within two days ( 175 epochs).
On the other four datasets, we train WaveGAN for 200 k iterations representing nearly 300 epochs for the largest dataset.
Unlike with autoregressive methods (Oord et al., 2016; Mehri et al., 2017), generation with WaveGAN is fully parallel and can produce an hour of audio in less than two seconds.
We list all hyperparameters in Appendix E.
E VALUATION METHODOLOGY
Evaluation of generative models is a fraught topic.
Theis et al.
(2016) demonstrate that quantitative measures of sample quality are poorly correlated with each other and human judgement.
Accordingly, we use several quantitative evaluation metrics for hyperparameter validation and discussion, and also evaluate our most promising models with human judges.
6. I NCEPTION SCORE
Salimans et al.
(2016) propose the inception score , which uses a pre-trained Inception classiﬁer (Szegedy et al., 2016) to measure both the diversity and semantic discriminability of generated images, ﬁnding that the measure correlates well with human judgement.
Given model scores P ( y | x ) with marginal P ( y ) , the inception score is deﬁned as exp( E x D KL ( P ( y | x ) || P ( y ))) , and is estimated over a large number of samples (e.g. 50 k).
For n labels, this measure ranges from 1 to n , and is maximized when the model is completely conﬁdent about each prediction but predicts each label equally often.
We will use this measure as our primary quantitative evaluation method and early stopping criteria.
To measure inception score, we train an audio classiﬁer on SC09.
Our classiﬁer ﬁrst computes a short-time Fourier transform of the input audio with 64 ms windows and 8 ms stride.
This representation is projected to 128 frequency bins equally spaced on the Mel scale (Stevens et al., 1937) from 40 Hz to 7800 Hz .
Amplitudes are scaled logarithmically and normalized so that each bin has zero mean and unit variance.
We process this perceptually-informed representation with four layers of convolution and pooling, projecting the result to a softmax layer with 10 classes.
We perform early stopping on the minimum negative log-likelihood of the validation set; the resultant model achieves 93% accuracy on the test set.
Because this classiﬁer observes spectrograms, our spectrogramgenerating models may have a representational advantage over our waveform-generating models.
6. N EAREST NEIGHBOR COMPARISONS
Inception score has two trivial failure cases in which a poor generative model can achieve a high score.
Firstly, a generative model that outputs a single example of each class with uniform probability will be assigned a high score.
Secondly, a generative model that overﬁts the training data will achieve a high score simply by outputting examples on which the classiﬁer was trained.
We use two indicators metrics to determine if a high inception score has been caused by either of these two undesirable cases.
Our ﬁrst indicator, | D | self , measures the average Euclidean distance of a set of 1 k examples to their nearest neighbor within the set (other than itself).
A higher | D | self indicates higher diversity amongst samples.
Because measuring Euclidean distance in time-domain audio poorly represents human perception, we evaluate distances in the same frequency-domain representation as our classiﬁer from Section 6.1 .
Our second indicator, | D | train , measures the average Euclidean distance of 1 k examples to their nearest neighbor in the real training data.
If the generative model simply produces examples from the training set, this measure will be 0 .
We report | D | train and | D | self relative to those of the test set.
6. Q UALITATIVE HUMAN JUDGEMENTS
While inception score is a useful metric for hyperparameter validation, our ultimate goal is to produce examples that are intelligible to humans.
To this end, we measure the ability of human an-
[Table]
notators on Amazon Mechanical Turk to label the generated audio.
Using our best WaveGAN and
SpecGAN models as measured by inception score, we generate random examples until we havefor each digit (as labeled by our classiﬁer from Section 6.1)— 3000 total.
In batches of ten random examples, we ask annotators to label which digit they perceive in each example, and compute their accuracy with respect to the classiﬁer’s labels (random accuracy would be 10% ).
After each batch, annotators assign subjective values of 1 – 5 for criteria of sound quality, ease of intelligibility, and speaker diversity.
We report accuracy ( n = 3000 ) and mean opinion scores ( n = 300 ) in Table 1.
R ESULTS AND DISCUSSION
Results for our evaluation appear in Table 1.
We also evaluate our metrics on the real training data, the real test data, and a version of SC09 generated by a parametric speech synthesizer (Buchner, 2017).
We also compare to SampleRNN (Mehri et al., 2017) and two public implementations of WaveNet (Oord et al., 2016), but neither method produced competitive results (details in Appendix C), and we excluded them from further evaluation.
These autoregressive models have not previously been examined on small-vocabulary speech data, and their success at generating full words has only been demonstrated when conditioning on rich linguistic features.
Sound examples for all experiments can be found at https://bit.ly/2DuMMZ5 .
While the maximum inception score for SC09 is 10 , any score higher than the test set score ofshould be seen as evidence that a generative model has overﬁt.
Our best WaveGAN model uses phase shufﬂe with n = 2 and achieves an inception score of 4 . 7 .
To determine if phase shufﬂe was improving the learning procedure simply by slowing down training, we also tried using 50% dropout in the discriminator’s activations with ﬁxed masks across timesteps.
Dropout resulted in a lower inception score compared to the baseline model.
Most experiments produced | D | self (diversity) values higher than that of the test data, and all experiments produced | D | train (distance from training data) values higher than that of the test data.
While these measures indicate that our generative models produce examples with statistics that deviate from those of the real data, neither metric indicates that the models achieve high inception scores by the trivial solutions outlined in Section 6.2.
While examples from SpecGAN achieve higher inception score ( 6 . 0 ) than those from our best WaveGAN model ( 4 . 7 ), human judges are able to label examples from the two models with similar accuracy ( 58% for WaveGAN vs 66% for SpecGAN).
However, on subjective criteria of sound quality and speaker diversity, humans indicate a preference for examples from WaveGAN.
This discrep- ancy can likely be attributed to the fact that inception scores are computed on spectrograms while subjective quality assessments are made by humans listening to waveforms. It is possible that the poor qualitative ratings for examples from SpecGAN are primarily caused by the lossy Grifﬁn-Lim inversion (Grifﬁn & Lim, 1984) and not the generative procedure.
We see promise in both waveform and spectrogram audio generation with GANs; our study does not suggest a decisive winner.
Finally, we train WaveGAN and SpecGAN models on the four other domains listed in Section 5.
Somewhat surprisingly, we ﬁnd that the frequency-domain spectra produced by WaveGAN (a timedomain method) are visually more consistent with the training data (e.g. in terms of sharpness) than those produced by SpecGAN (Figure 4).
For drum sound effects, WaveGAN captures semantic modes such as kick and snare drums.
On bird vocalizations, WaveGAN generates a variety of bird sounds but with more noise than the other domains.
On piano, WaveGAN produces musicallyconsonant motifs that, as with the training data, represent a variety of key signatures and rhythmic patterns.
For TIMIT, a large-vocabulary speech dataset with many speakers, WaveGAN produces speech-like babbling similar to results from unconditional autoregressive models (Oord et al., 2016).
R ELATED WORK
Much of the work within generative modeling of audio is within the context of text-to-speech.
Textto-speech systems are primarily either concatenative or parametric .
In concatenative systems, audio is generated by sequencing small, prerecorded portions of speech from a phonetically-indexed dictionary (Moulines & Charpentier, 1990; Hunt & Black, 1996).
Parametric systems map text to salient parameters of speech, which are then synthesized by a vocoder (Dudley, 1939); see (Zen et al., 2009) for a comprehensive review.
Some of these systems use learning-based approaches such as a hidden Markov models (Yoshimura, 2002; Tokuda et al., 2013), and separately-trained neural networks pipelines (Ling et al., 2015) to estimate speech parameters.
Recently, several researchers have investigated parametric speech synthesis with end-to-end neural network approaches that learn to produce vocoder features directly from text or phonetic embeddings (Arik et al., 2017;
Ping et al., 2018;
Sotelo et al., 2017; Wang et al., 2017; Shen et al., 2018).
These vocoder features are synthesized to raw audio using off-the-shelf methods such as WORLD (Morise et al., 2016) and Grifﬁn-Lim (Grifﬁn & Lim, 1984), or trained neural vocoders (Sotelo et al., 2017;
Shen et al., 2018; Ping et al., 2018).
All of these methods are supervised: they are trained to map linguistic features to audio outputs.
Several approaches have explored unsupervised generation of raw audio.
Oord et al.
(2016) propose WaveNet, a convolutional model which learns to predict raw audio samples by autoregressive modeling.
WaveNets conditioned on rich linguistic features have widely been deployed in text-tospeech systems, though they have not been demonstrated capable of generating cohesive words in the unconditional setting.
Engel et al.
(2017) pose WaveNet as an autoencoder to generate musical instrument sounds.
Chung et al.
; Mehri et al.
both train recurrent autoregressive models which learn to predict raw audio samples.
While autoregressive methods generally produce higher audio ﬁdelity than WaveGAN, synthesis with WaveGAN is orders of magnitude faster.
The application of GANs (Goodfellow et al., 2014) to audio has so far been limited to supervised learning problems in combination with traditional loss functions.
Pascual et al.
(2017) apply GANs to raw audio speech enhancement.
Their encoder-decoder approach combines the GAN objective with an L 2 loss.
Fan et al.
; Michelsanti & Tan (2017);
Donahue et al.
all use GANs in combination with unstructured losses to map spectrograms in one domain to spectrograms in another.
Chen et al.
(2017) use GANs to map musical performance images into spectrograms.
C ONCLUSION
We present WaveGAN, the ﬁrst application of GANs to unsupervised audio generation.
WaveGAN is fully parallelizable and can generate hours of audio in only a few seconds.
In its current form,
WaveGAN can be used to augment sound libraries for multimedia production.
In our future work we plan to extend WaveGAN to operate on variable-length audio and also explore a variety of label conditioning strategies.
By providing a template for modifying image generation models to operate on audio, we hope that this work catalyzes future investigation of GANs for audio synthesis.
A C HECKERBOARD ARTIFACTS IN AUDIO VERSUS IMAGES
Generative models that upsample by transposed convolution are known to produce characteristic “checkerboard” artifacts in images (Odena et al., 2016), artifacts with particular spatial periodicities.
The discriminator of image-generating GANs can learn to reject images with these artifacts because they are uncommon in real data (as discussed in Section 3.1).
However, in the audio domain, the discriminator might not have such luxury as these artifacts correspond to frequencies which might rightfully appear in the real data.
To characterize analogous artifacts in WaveGAN, we measure its impulse response by randomly initializing it 1000 times and passing unit impulses to its ﬁrst convolutional layer.
In Figure 5, we plot the average of these responses in the frequency domain.
The response has sharp peaks at linear multiples of the sample rates of each convolutional layer ( 250 Hz , 1 kHz , 4 kHz , etc.).
This is in agreement with our informal observation of results from WaveGAN, which often have a pitched noise close to the musical note B ( 247 × 2 n Hz ).
B L EARNED POST - PROCESSING FILTERS
We experiment with adding a post-processing ﬁlter to the generator, giving WaveGAN a simple mechanism to ﬁlter out undesirable frequencies created by the generative process.
This ﬁlter has a long window ( 512 samples) allowing it to represent intricate transfer functions, and the weights of the ﬁlter are learned as part of the generator’s parameters.
In Figure 5, we compare the postprocessing ﬁlters that WaveGAN learns for human speech and bird vocalizations.
The ﬁlters boost signal in regions of the frequency spectrum that are most prominent in the real data domain, and
Original samples
Zero insertion
Nearest neighbor
Linear interpolation
Cubic interpolation introduce notches at bands that are artifacts of the generative procedure as discussed in the previous section.
B. U PSAMPLING PROCEDURE
Transposed convolution upsamples signals by inserting zeros in between samples and applying a learned ﬁlterbank.
This operation introduces aliased frequencies, copies of pre-existing frequencies shifted by multiples of the new Nyquist rate, into the upsampled signal.
While aliased frequencies are usually seen as undesirable artifacts of a bad upsampling procedure, in the generative setting their existence may be crucial for producing ﬁne-grained details in the output.
We experiment with three other upsampling strategies in WaveGAN: nearest-neighbor, linear and cubic interpolation, all of which attenuate aliased frequencies.
In Figure 6, we compare these strategies visually.
While nearest neighbor upsampling resulted in similar audio output to transposed convolution, linear and cubic interpolation strategies resulted in qualitatively poor audio output (sound examples: https://bit.ly/2DuMMZ5 ).
We hypothesize that the aliased frequencies produced by upsampling convolutions may be more critical to audio generation than image generation.
C E XPERIMENTS WITH AUTOREGRESSIVE WAVEFORM MODELS
We developed our WaveGAN and SpecGAN models primarily to address the task of steerable sound effect generation.
This is an inherently different task than text to speech (TTS), however autoregressive waveform models (e.g. WaveNet (Oord et al., 2016) and SampleRNN (Mehri et al., 2017)) that were developed for TTS can also be used to model and generate waveforms unconditionally.
Hence, a comparison to these models for our task is reasonable.
One upside of autoregressive models for our task is that they have the potential to produce high-quality audio.
Potential downsides are 1) these models take several orders of magnitude longer to generate waveforms, and 2) they do not learn a compact latent space of waveforms, causing useful sound generation tasks like continuous exploration and interpolation to be impossible. We attempt to train two public implementations of WaveNet (ImplA 2 and ImplB 3 ) and SampleRNNon our SC09 digit generation task. We use default parameters for these libraries; the only modiﬁcations we make are to reduce the training example size to one second. To our ears, all three libraries failed to produce cohesive words (you can judge for yourself from our sound examples at the bottom https://bit.ly/2DuMMZ5 ).
This poor subjective performance is echoed by weak inception scores (weaker than any in Table 1): 1 . 07 ± 0 .
05 , 1 . 29 ± 0 .
28 ± 0 .
19 for WaveNet ImplA,
WaveNet ImplB, and SampleRNN respectively.
Note that these inception scores were calculated on far fewer examples ( < 1 k ) than all of the scores listed in Table 1 (which were computed on 50 k examples).
This is because it took over 24 hours to produce even a thousand one-second examples with these methods (whereas our methods produce 50 k examples in a few seconds).
Autoregressive methods have not been demonstrated capable of learning to synthesize coherent words without conditioning on rich linguistic features.
We are not claiming that these methods cannot learn to synthesize full words, merely that three open-source implementations were unable to do so with default parameters.
We want to be clear that our intent is not to disparage autoregressive waveform methods as these methods were developed for a different task , and hence we excluded these poor scores from our results table to avoid sending readers the wrong message.
We are merely highlighting that they were noncompetitive for our problem domain and less useful (due to slowness and lack of a latent space) for creative generation of sound effects.
[Table]
[Table]
D A RCHITECTURE DESCRIPTION
In Tables 2 and 3, we list the full architectures for our WaveGAN generator and discriminator respectively.
In Tables 4 and 5, we list the same for SpecGAN.
In these tables, n is the batch size, d modiﬁes model size, and c is the number of channels in the examples.
In all of our experiments in this paper, c = 1 .
All dense and convolutional layers include biases.
No batch normalization is used in WaveGAN or SpecGAN.
E T RAINING HYPERPARAMETERS
In Table 6, we list the values of these and all other hyperparameters for our experiments, which constitute our out-of-the-box recommendations for WaveGAN and SpecGAN.
[Table]
[Table] [Table]