D EEP A UDIO P RIORS E MERGE FROM H ARMONIC C ONVOLUTIONAL N ETWORKS
A BSTRACT
Convolutional neural networks CNNs) excel in image recognition and generation
Among many efforts to explain their effectiveness experiments show that CNNs carry strong inductive biases that capture natural image priors.
Do deep networks also have inductive biases for audio signals?
In this paper we empirically show that current network architectures for audio processing do not show strong evidence in capturing such priors.
We propose Harmonic Convolution , an operation that helps deep networks model priors in audio signals by explicitly utilizing the harmonic structure.
This is done by engineering the kernels to be supported by sets of harmonic series, instead of by local neighborhoods as convolutional kernels.
We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance on unsupervised audio restoration.
With
Harmonic Convolution, they also achieve better generalization performance for supervised musical source separation.
Code and examples are available at our project page: http://dap.csail.mit.edu .
I NTRODUCTION
Deep neural networks, in various forms and designs, have been proved extremely successful in both discriminative tasks such as image classiﬁcation (He et al., 2016), machine translation (Sutskever et al., 2014), speech recognition (Hinton et al., 2012), and in generative tasks such as image and audio generation (Goodfellow et al., 2014; Oord et al., 2017).
Recently, Lempitsky et al.
(2018) showed that convolutional neural networks (CNNs) come with strong inductive biases suitable for capturing natural image priors.
Speciﬁcally, a randomly initialized neural network is optimized by ﬁtting a degenerate signal, such as a noisy image, with a random Gaussian noise vector as input.
During the ﬁtting process, the network ﬁrst outputs a clean image, even though it has only seen its noisy version.
Lempitsky et al.
(2018) argued that this phenomenon shows that CNNs capture image priors by its deep convolutional structure; this is an intriguing perspective on the effectiveness of CNNs in generative modeling of images.
Due to the success of CNNs, audio processing networks usually adopt similar designs from their image processing counterparts, using spatial or temporal convolutions as building blocks.
Though such operations could capture image priors, are they capable of capturing audio priors?
Is it possible that the task of audio signal modeling needs unique components that cannot be found in image processing networks?
As an attempt to answer the questions above, we investigate whether various audio processing networks can capture audio priors, similar to what CNNs do for images (Lempitsky et al., 2018).
Recently, Michelashvili & Wolf (2019) reported that deep priors do exist for Wave-U-Net (Stoller et al., 2018), which can be exploited to perform unsupervised audio denoising.
However, the deﬁnition of deep priors is slightly different between works by Michelashvili & Wolf (2019) and Lempitsky et al.
Michelashvili & Wolf (2019) reported that the noisy signal causes more violent ﬂuctuations on the spectrogram, which is then utilized as a prior for estimating the noise signal.
In contrast, we focus on the deep priors deﬁned by Lempitsky et al.
(2018), and investigate whether there is evidence that current deep networks for audio related tasks carry inductive biases for audio priors; if not, what might be an alternative?
In this work, we ﬁrst empirically show that current architectures for audio signal modeling do not show strong evidence in capturing audio priors.
Speciﬁcally, we look at two general types of design: temporal CNNs (Michelashvili & Wolf, 2019; Stoller et al., 2018; Aytar et al., 2016) and spectrogram-based CNNs (Shen et al., 2018; Zhao et al., 2018).
To test their prior modeling ability, we use the setup identical to Lempitsky et al.
(2018): the networks are initialized randomly and tasked to ﬁt a single degenerated audio signal using random Gaussian noises as input.
If the network is capable of modeling the signal priors by its structure, it would ﬁt the signal faster than noise.
We show an illustrative example in Figure 1, where no substantial evidence is found for temporal or spectral-temporal CNNs, even with this simplest case.
What might be missing?
As shown in psychoacoustics experiments, the structure of the harmonic series is closely related to human perception (Moore et al., 1986; Popham et al., 2018).
We therefore propose the Harmonic Convolution , an operation that explicitly utilizes harmonic structures in audio signals.
Then, with multiple experiments, we show that Harmonic Convolution does enable neural networks to model audio signal priors better.
Finally, we show that Harmonic Convolution is useful in downstream applications, and prove its performance by comparing against various baselines.
The most natural application is unsupervised audio restoration, where we aim to recover a clean signal from its corrupted version, either by a high power Gaussian noise or aggressive quantizations.
In addition, we also demonstrate that networks with Harmonic Convolution achieve better generalization performances for supervised sources separation tasks.
In summary, our contributions are threefold.
First, we show that current audio processing architectures do not model audio signal priors naturally.
Second, we propose an operation called Harmonic
Convolution, serving as an effective inductive bias for neural networks to model audio signal priors.
Finally, we demonstrate that networks with Harmonic Convolution achieve state-of-the-art performances in unsupervised audio restoration tasks and improve the generalization ability on supervised musical source separation tasks.
M OTIVATION
In this section, we ﬁrst give a brief review of deep image priors.
We also provide a short survey on current popular network architecture designs for processing audio signals.
Then, we show a motivating toy example, where current architectures fail to model the signal priors, even when the signal is stationary.
Finally, we provide a heuristic analysis of why convolution-based approaches may not capture the audio priors, using local signal statistics.
2. D EEP P RIORS
Lempitsky et al.
ﬁrst proposed the notion of deep priors on images.
Speciﬁcally, they show that given a corrupted image x 0 , a deep neural network f θ , parameterized by θ , can serve as a natural regularization without any pretaining.
Formally, the deep prior method optimizes
[Equation]
where E ( · ; · ) is a task-speciﬁc data term, x 0 is the corrupted image, and z is a ﬁxed random noise vector.
Lempitsky et al.
(2018) showed that optimizing in the parameter space of the neural network f θ is surprisingly effective: after several iterations, the optimized network parameters θ ∗ gives the restored image x by forwarding the noise vector, i.e. x = f θ ∗ ( z ) .
Note that the network is randomly initialized and only trained to ﬁt the corrupted signal x 0 using a random vector.
The fact that it ﬁts the clean signal ﬁrst suggests that CNNs might be well suited for modeling images, where its structure and operation provide strong inductive biases.
2. C URRENT NETWORK DESIGNS FOR AUDIO PROCESSING
The network architectures for audio signal processing fall into two broad categories.
The ﬁrst one is to directly apply 1D convolutions on the raw audio signals (Michelashvili & Wolf, 2019; Stoller et al., 2018; Aytar et al., 2016).
For instance, Wave-U-Net (Stoller et al., 2018) is a 1D adaptation of the U-Net architecture, which utilizes skip connections and multiscale processings to model signal properties at different time scales.
The other category is characterized by performing 2D convolutions on spectrograms (Shen et al., 2018;
Zhao et al., 2018).
A common practice is ﬁrst to extract a spectral-temporal representation from the audio signal, then apply 2D convolutions.
[Figure] [Figure]
2. A MOTIVATING EXAMPLE
Inspired by the deep image prior experiments, we would like to see if the deep architectures above possess the proper inductive biases to model audio signals.
To this end, we ﬁrst test the above architectures to reconstruct a simple signal: a stationary signal composed of 1,000Hz, 2,000Hz, and 3,000Hz sinusoidal waves.
The corrupted version of this signal is generated by adding stationary
Gaussian noise with a standard deviation of 0.1.
As can be seen from Figure 1, all the methods start with very noisy ﬁttings at very early iterations (50 in the ﬁgure).
Then, they start to ﬁt the signal and the noise at a similar speed, rendering a noisy signal at 200 steps.
At 1,000 steps, the network would ﬁt a noisy signal, with slightly less noise than the input signal.
2. A HEURISTIC ANALYSIS
Here we provide a heuristic analysis on why plain convolution-based networks would fail for modeling audio signal priors by structure.
We show that the natural statistics (Torralba & Oliva, 2003) of images and audio signals are different.
This suggests that different designs might be needed to capture their priors.
In Figure 2, we show, side by side, the frequency statistics of clean natural images, clean speech spectrograms, clean speech signals, and their noisy versions by adding Gaussian noise.
All modalities share the same signal-to-noise ratio.
The natural images share a 1 /f 2 energy distribution in the frequency domain.
This can be seen in Figure 2 ( a ) , where the energy falloff is ap-
[Figure]
proximately linear in the log scale.
However, this does not hold for spectrograms or raw waveforms.
Details for generating the statistics in Figure 2 can be found in Appendix Section B.
A PPROACH
As shown in Figure 1, the architectures above, despite their success in discriminative tasks, do not show strong evidence for encoding deep audio priors.
In light of such facts, we aim to introduce new components for audio signal modeling.
We start with harmonic structures, the most common patterns for audio signals, which are also shown to be closely related to human perception (Popham et al., 2018; McPherson & McDermott, 2018).
Subsequently, we propose Harmonic Convolution, an operation that treats harmonic structures explicitly.
H ARMONIC C ONVOLUTION
Harmonic Convolution is designed to explicitly exploit harmonic structures as an inductive bias for auditory signal modeling.
Speciﬁcally, Harmonic Convolution operates on the discrete Short-Time
Fourier Transform (STFT) of a given audio signal, utilizing the spectral-temporal structure of this representation.
For audio signal x [ t ] , its discrete STFT spectrogram X [ ω, τ ] is given by
[Equation]
where w ( · ) is a locally supported windowing function and j denotes the imaginary unit.
Regular 2D convolutions on the spectrogram X [ ω, τ ] with a kernel function K is deﬁned as:
[Equation]
where the kernel K is supported on [ − Ω , Ω] × [ − T, T ] .
Note that regular convolution aggregates information in a 2Ω × 2 T window on local regions of X .
To utilize harmonic structures, we modify this information aggregation scheme to align with harmonics.
Speciﬁcally, Harmonic Convolution is deﬁned as an operation mapping X ( ω, τ ) to Y (ˆ ω, ˆ τ ) , where
[Equation]
Note Harmonic Convolution interprets the frequency dimension of the kernel as weights for K harmonic series at each target frequency location ˆ ω , where regular convolutions interpret the kernel as weights for a local neighborhood at target spectral temporal locations.
Figure 3 shows an illustration for both plain convolutions and Harmonic Convolution.
3. A NCHORS AND M IXING
As indicated by Equation 3, the output at target frequency ˆ ω is a weight sum of its K harmonic series, starting from ˆ ω to K ˆ ω .
Note that there also exist other possible harmonic series that include ˆ ω .
For example, { 0 . 25ˆ ω, 0 .
5ˆ ω, 0 .
75ˆ ω, ˆ ω, . . . } is also a valid harmonic series, but Equation 3 never aggregates the information from frequencies lower than ˆ ω .
To over come this problem, we add an extra parameter to Equation 3 called anchor, which indicates the order of harmonics at target frequency location ˆ ω .
Speciﬁcally, given the anchoring parameter n , we modify Equation 3 as:
[Equation]
An illustration for the effect of different anchoring parameters are shown in Figure 3(b)(c)(d).
In addition, we can make the output at the frequency location ˆ ω depend on multiple anchoring parameters.
To this end, we mix different Y n using a weighted sum: Y [ˆ ω, ˆ τ ] = (cid:80) Nn
=1 w n Y n [ˆ ω, ˆ τ ] , where N is the largest anchoring parameter, and w n can be seen as learnable parameters, similar to convolution kernels K .
Therefore, the ﬁnal Harmonic Convolution is deﬁned as:
[Equation]
where the learnable parameters are the convolution kernel K and the weights w n .
Implementation details.
We implement Equation 5 using the Deformable Convolution operation introduced by Dai et al.
For better efﬁciency, we factorize the 2D kernel K [ k, τ ] as the product of two 1D kernels, i.e. K [ k, τ ] =
K f [ k ] K t [ τ ] .
Anchoring is implemented using grouped
Deformable Convolution (Dai et al., 2017), and the weighted sum mixing is implemented as an extra 1 × 1 convolution.
Note that the notation is deﬁned on spectrograms, as all operations are implemented by real-valued operations.
In experiments, we treat complex STFT coefﬁcients as two separate real-valued channels.
E XPERIMENTS
In experiments, we test Harmonic Convolution under the deep prior modeling setup introduced by
Lempitsky et al.
(2018), where the networks are asked to ﬁt a corrupted signal.
We deﬁne a network’s ability to model audio priors as the quality of the audio produced during the ﬁtting process.
Under this deﬁnition, we show that networks equipped with Harmonic Convolution can model audio priors better than various baselines.
As a by-product, Harmonic Convolution performs comparably well with several state-of-art methods for unsupervised audio restoration.
Finally, we demonstrate that
Harmonic Convolution improves the generalization performance for supervised sound separation.
4. E XPERIMENT S ETUPS
We use the LJ-Speech (Ito, 2017) dataset and the MUSIC (Zhao et al., 2018) dataset.
LJ-Speech is a speech dataset consisting of short audio clips of a single speaker reading passages.
MUSIC is a video dataset of musical instrument solos crawled from Youtube.
We only use their audio tracks for all the experiments.
For fair comparisons, we use the same U-Net (Ronneberger et al., 2015) architecture with different operations, i.e., regular convolutions, dilated convolutions, and the Harmonic Convolution.
The details of our network architecture can be found in the appendix.
We train all the networks using the
Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.001 for all the experiments.
4. E XPERIMENTS ON D EEP A UDIO P RIORS
Following Lempitsky et al.
(2018), we test networks’ ability to model audio priors by ﬁtting a corrupted audio signal using a ﬁxed random input and randomly initialized weights.
If the network can produce a restored signal faster and with better quality, then we call this network having a stronger ability to model audio priors.
Setup.
The random input is drawn from a standard Gaussian distribution, and the weights are initialized by drawing from a zero-mean Gaussian distribution with a standard deviation of 0.02.
We test networks that rely on 2D spectral-temporal operations, i.e. regular convolutions, dilated convolutions, and Harmonic Convolution, to ﬁt the complex STFT coefﬁcients of the corrupted signal.
We also test networks that operate on the raw waveform, i.e., Wave-U-Net.
Again, all operations done by the network are on real numbers; we treat the real and imaginary part of the coefﬁcients as two separate channels.
The corrupted signal is generated by adding a clean signal with a zero-mean
Gaussian noise with a standard deviation of 0.1.
The complex coefﬁcients are generated by taking
PSNR between fitting result and clean signal
PSNR between fitting result and corrupted signal (target of fitting)
Wave-U-Net iteration
PS N R ( d B )
Convolution iteration
PS N R ( d B )
Harmonic convolution
Multiple Anchors + Mixing iteration
PS N R ( d B )
Harmonic convolution
Multiple Anchors, No Mixing iteration
PS N R ( d B )
Harmonic convolution
Single Anchor iteration
PS N R ( d B ) (a) (b) (d) (e) (f) iteration
PS N R ( d B ) (c)
Dilated the Short-Time-Fourier-Transform (STFT) of the corrupted signal, with a box ﬁlter of length 1,022.
We use a hop length of 64 to provide enough overlapping, reducing the noise introduced by taking the inverse STFT transformation of the ﬁtted result.
When comparing against Wave-U-Net, we use the publicly available implementations provided by the authors of Michelashvili & Wolf (2019).
Results.
We compare the ﬁtting progress of the networks with Harmonic Convolution against various baselines, where the quality of the output signal is measured by Peak Signal-to-Noise Ratio in the temporal domain at each step.
As demonstrated in Figure 4, Harmonic Convolution produces a cleaner signal than other methods (-3.5dB boost compared with networks using convolution and dilated convolution).
The example of the ﬁtting is randomly sampled from the LJ-Speech dataset.
More examples can be found at https://dap.csail.mit.edu .
In addition, we also conduct an ablation study using this ﬁtting process, showing that the design of anchoring and mixing helps with modeling audio priors.
As shown in Figure 4 (d) and (f), without the anchoring operation, the ﬁtting speed is much slower.
Using anchors without the mixing operation would lead to sub-optimal ﬁtting results.
4. A UDIO R ESTORATION
Similar to the experiments in Lempitsky et al.
(2018), networks that model signal priors can be used to perform unsupervised restoration tasks.
Here we examine the performance of the network with the Harmonic Convolution on restoring speech and music audios corrupted by random Gaussian noises and aggressive quantizations.
Setup.
We conducted experiments on both LJ-Speech and MUSIC datasets.
For each dataset, we randomly sample 200 audio clips and clip them to 3 seconds long for restoration tasks.
For restoring from Gaussian noise, we set the noise to be zero-mean, with a standard deviation of 0 . 1 .
[Table]
[Table]
For recovering from quantization noise, we use 1s clips of the 200 randomly sampled speech signals and quantize them into 16 bins, which uniformly covers the range from − 1 to 1 .
The input and output of the network share the same details described in Section 4.2.
Baselines.
We compare with the following baselines:
• Wiener:
Wiener ﬁltering (Scalart & Filho, 1996) is an optimization-based methods utilizing a Signal-to-Noise Ratio (SNR) prior.
We adopt the implementation where the SNR is estimated from the ﬁrst 1,024 samples of the signal.
• Wavelet: We use the MATLAB implementation for wavelet denoising.
Wavelet denoising is based on the sparse prior of audio signals, which assumes the wavelet coefﬁcients should be sparse for clean signals.
We use the 8-tap symlet wavelets for this task.
• DFL: Deep feature losses (Germain et al., 2018) is a state-of-the-art supervised speech denoising approach using perceptual loss tailored for speech signals.
When testing on LJ-Speech with Gaussian noise, we are testing its generalization performances under unseen settings.
• DNP:
Deep network priors (Michelashvili & Wolf, 2019) is an unsupervised method for audio denoising using deep priors.
Contrary to our method, the authors observed that during the ﬁtting process, the injected noise varies more violently than the signal itself.
Therefore, this property can be used to identify noise regions and provide an SNR estimate for traditional ﬁltering methods such as LSA (Ephraim & Malah or Wiener ﬁltering Scalart & Filho, 1996).
Metrics.
For the speech restoration task, we adopt multiple quality metrics to measure the audio restoration results, including the mean opinion score (MOS) predictor of signal distortion (CSIG), the MOS predictor of background-noise intrusiveness (CBAK), the MOS predictor of overall signal quality (COVL), the perceptual evaluation of speech quality (PESQ), and the segmental Signal-toNoise Ratio (SSNR).
For the music restoration task, we only report results measured by the SSNR, since other metrics do not apply to the music signals.
CSIG, CBAK, COVL, and PESQ are on a scale of 1 to 5, where 5 is the best quality.
Results.
Table 1 summarizes our results against the mentioned baselines over these metrics.
Our method consistently outperforms all the baselines according to all measures by a considerable margin.
This directly demonstrates that Harmonic Convolution can make neural networks more suited for modeling audio signals.
We also conduct an experiment to verify that Harmonic Convolution is not limited to the additive
Gaussian noise case.
In this experiment, we quantize 200 randomly sampled one-second speech signals into 16 bins, uniformly covering the range of [ − 1 , 1] .
The results are reported in Table 2.
Since the scores for each metric are rather close, we conduct a carefully designed perceptual experiment with human listeners.
For each model, we take 200 audio restorations from the same set for evaluation and each audio is evaluated by three independent Amazon Mechanical Turk (AMT) workers.
We present the model results in random order and ask the annotator to select the audio clip that has the best restoration results.
The results show nearly 80% of the subjects vote for the results produced by the network using Harmonic Convolution.
[Table]
4. G ENERALIZED S OUND S EPARATION
Here we examine whether the Harmonic Convolution can improve generalizations of the supervised sound separation task, compared with regular and dilated convolutions.
Setup.
To evaluate the generalization ability of sound separation networks, we select ﬁve musical instruments from the MUSIC dataset (Zhao et al., 2018): violin, cello, congas, erhu, and xylophone.
Each category consists of 50 6-seconds solo audio clips.
We aim to test the model’s ability to generalize to unseen musical instrument mixtures.
Speciﬁcally, the model is tasked with separating the sound of a target instrument from a clip that also contains sounds of another instrument.
During training, we avoid using clips that contain sounds of a selected holdout instrument class, so that the model has never ’heard’ of the sound of that type of instrument before.
We then test the separation performance of this model on mixtures made from the sounds of the model’s target instrument and the holdout instrument.
In particular, we train models to separate the sound of violins and use three different holdout instruments to test its generalizability (congas, violin, and xylophone).
The input to the network is the complex STFT coefﬁcients of the mixed audio, using a box ﬁlter of sizeand a hop length of 64.
The output of the network is a ratio mask, calculated as the ratio of the spectrogram between the sound of the target instrument and the input mixed sound.
To produce separated audios, we apply an inverse STFT transformation to the input complex STFT coefﬁcients multiplied by the predicted ratio mask.
Implementations.
We adopt the Mix-and-Separate framework (Zhao et al., 2018) for this task.
We ﬁrst generate a synthetic separation training set by mixing the audio signals from two different audio clips and then trains a neural network to separate the sound of the target instrument, e.g., violin.
During training, we take a 1-second mixed audio clip as input and transform it into complex coefﬁcients using the Short-Time-Fourier-Transform (STFT).
The spectrogram is then fed into a U-Net, whose architecture is described in Section 4.1.
The U-Net outputs a ratio mask, and the network is trained by minimizing the L1 loss between the predicted ratio masks and the ground-truth masks.
We use a 90:10 train-val split, and test the performance on the mixture between sounds of the target instrument and the sounds of the holdout instrument.
Results.
We compare the performance of the proposed Harmonic Convolution against the regular and dilated convolutions used in previous papers.
We use the Signal-to-Distortion Ratio (SDR),
Signal-to-Interference Ratio (SIR), and Signal-to-Artifact Ratio (SAR).
metrics from the opensource mir eval library (Raffel et al., 2014) to quantify performances.
Quantitative results are reported in Table 3.
We observe that while all networks suffer when tested on mixtures under novel recording conditions, the Harmonic Convolution exhibits better generalization performances.
This suggests that Harmonic Convolution not only can be used as a prior for unsupervised tasks but also has the potential to be helpful for supervised tasks.
R ELATED W ORK
Deep priors.
Our work is inspired by the recent paper on deep image priors (Lempitsky et al., 2018), which shows that the structure of CNNs imposes a strong prior to restore a single original image from the degraded image.
The idea of deep priors has also shown to be useful in many applications, including semantic photo manipulation (Bau et al., 2019), image super-resolution (Shocher et al., 2018), and image decomposition (Gandelsman et al., 2018).
While most prior papers focused on images, little work has explored deep priors on audio signals.
Deep learning for auditory signal modeling.
Deep networks have gained remarkable success on the audio signal modeling, such as speech recognition (Hinton et al., 2012;
Amodei et al., 2016), sound separation (Stoller et al., 2018; Zhao et al., 2018; 2019), audio denoising (Rethage et al., 2018; Germain et al., 2018), audio generation (Oord et al., 2017; Mehri et al., 2017), text to speech synthesis (Wang et al., 2017; Shen et al., 2018), and voice conversion (Hsu et al., 2017).
A detailed survey can be found at (Purwins et al., 2019; Qian et al., 2019).
However, it remains unclear if these architectures themselves capture the audio signal priors.
Most related to our work is Michelashvili & Wolf (2019), where they used deep networks as a prior to estimate the SNR prior on the spectrogram and then used classical post-processing algorithms to perform the speech denoising.
Note that the deep prior mentioned in works of Mehri et al.
(2017) is different from the ones mentioned in Lempitsky et al.
The former uses ﬁtting time variances as noise indicating priors.
We aim to ﬁnd designs that bias towards clean audio signal.
Structured operators for deep models.
Various algorithms have been proposed to include more dynamic structures beyond translational invariant kernels.
Dai et al.
(2017) proposed a convolution operator with dynamic offsets at each spatial location, which improves the object detection accuracy by distinguishing foreground and background features.
Shelhamer et al.
(2019) introduced a spatially adaptive Gaussian kernel to help scale and steer convolution kernels, without introducing a signiﬁcant amount of free parameters.
Ravanelli & Bengio (2018); Sainath et al.
(2013) proposed to learn parametrized ﬁlter banks that adapt to the convolution structure.
While most of these algorithms focus on the discriminative tasks, our research topic is more related to generative tasks.
Moreover, the learned ﬁlter banks in Ravanelli & Bengio (2018); Sainath et al.
(2013) are used only in the ﬁrst layer and not invertible in general, making them less suitable for the generative tasks where the output is an audio signal.
Psychoacoustics.
Harmonic structures are closely related to the human perception of audio signals.
The famous missing fundamental auditory illusion suggests that human can infer the missing fundamental frequency by only hearing its overtones (Todd & Loy, 1991).
Moore et al.
(1986) showed that shifts in harmonic components would be perceived as separate tones.
More recently,
Popham et al.
(2018) showed that the harmonic structure plays an important role for human to solve the cock-tail problem, where inharmonicity would cause difﬁculties for human to track speakers for the cocktail party problem.
McPherson & McDermott (2018) showed that pitch perception is closely related to the harmonicity of the sound.
C ONCLUSION
In this paper we examined various architectures on deep audio prior modeling.
We then proposed a novel operation called Harmonic Convolution which can help networks better capture priors in audio signals.
We showed that ﬁtting a randomly-initialized network equipped with Harmonic Convolution can achieve high performance for unsupervised audio restoration tasks.
We also showed that Harmonic Convolution improves the generalization ability in sound separation. Acknowledgment.
We would like to thank Josh H. McDermott for helpful discussions.
This work is supported by the NSF award #1447476 and IBM Research
A N ETWORK A RCHITECTURE
The U-Net used by all methods consists of 5 blocks, each block contains two operation layers, which can be instantiated by regular convolutions, dilated convolutions or the Harmonic Convolution.
The feature map is downsampled by average pooling with a stride of 2 and a window size of 2.
Downsampling is performed after the ﬁrst two operation layers.
As in common designs of U-Net, the upsampling is performed before the last two layers through bilinear upsampling with a scale of 2.
Finally, we attach a 1 × 1 convolution layer after the U-Net’s last layer to give the ﬁnal output.
Feature map sizes for all the layers are [input → input, input → 35 ], [ 35 → 35 , 35 → 70 ], [ 70 → 70 , 70 → 70 ], [ 140 → 140 , 140 → 35 ], [ 70 → 70 , 70 → 35 ], then followed by a 1 × 1 regular convolution layer mapping the ﬁnal 35 channels to the desired number of output channels.
In addition, we also keep the kernel sizes ﬁxed for all different operations.
For regular convolutions, we use 7 × 7 kernels for all the layers.
For dilated convolutions, we use the same kernel size ( 7 × 7 ) with dilation of 3 for all the layers.
For the Harmonic Convolution, we use a frequency kernel K f of length 7 and a temporal kernel K t also of length 7.
We use 7 anchors ( N =7 in Equation 5) for all the Harmonic Convolution operations.
We use instance normalization (Ulyanov et al., 2016) and
ReLu activations (Krizhevsky et al., 2012) for all the experiments.
B D ETAILS FOR NATURAL STATISTICS ANALYSIS
The image statistics are computed on 1,000 images randomly sampled from ImageNet (Russakovsky et al., 2015).
The audio spectrogram statistics are computed on 1,000 speech signals randomly sampled from the LJ-Speech dataset (Ito, 2017).
We compute their spectrograms using the ShortTime-Fourier-Transform and calculate the spatial frequency distribution of the spectrograms as if they are images.
The 1D frequency distribution of the audio signals is calculated on the same speech signals, where we interpret audio clips like 1D image patches.
C R ELATING H ARMONIC C ONVOLUTION WITH D ILATED C ONVOLUTION
The simplest form of Harmonic Convolution is deﬁned in Equation 3:
[Equation]
Contrasting dilated convolution, one can write its operation as
[Equation]
where n is the dilation parameter.