D ENOISING AND R EGULARIZATION VIA E XPLOITING THE S TRUCTURAL B IAS OF C ONVOLUTIONAL G ENERATORS
Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration.
A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images.
A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data, by simply ﬁtting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the corrupted image.
While this over-parameterized network can ﬁt the corrupted image perfectly, surprisingly after a few iterations of gradient descent it generates an almost uncorrupted image.
This intriguing phenomena enables state-of-the-art CNNbased denoising and regularization of other inverse problems.
In this paper we attribute this effect to a particular architectural choice of convolutional networks, namely convolutions with ﬁxed interpolating ﬁlters.
We then formally characterize the dynamics of ﬁtting a two layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes.
Our proof relies on showing that convolutional generators ﬁt the structured part of an image signiﬁcantly faster than the corrupted portion.
I NTRODUCTION
Convolutional neural networks are extremely popular for image generation.
The majority of image generating networks is convolutional, ranging from deep convolutional Generative Adversarial
Networks (DC-GANs) (Radford et al., 2015) to the U-Net (Ronneberger et al., 2015).
It is well known that convolutional neural networks incorporate implicit assumptions about the signals they generate, such as pixels that are close being related.
This makes them particularly well suited for modeling distributions of images.
It is less well known, however, that those prior assumptions build into the architecture are so strong that convolutional neural networks are useful even without ever being exposed to training data.
The latter was ﬁrst shown in the Deep Image Prior (DIP) paper (Ulyanov et al., 2018).
Ulyanov et al.
(2018) observed that when ‘training’ a standard convolutional auto-encoder such as the popular U-net (Ronneberger et al., 2015) on a single noisy image and regularizing by early stopping, the network denoises the image with excellent performance.
This is based on the empirical observation that un-trained convolutional auto-decoders ﬁt a single natural image faster when optimized with gradient descent than pure noise.
Many elements of an auto-encoder, however are irrelevant for this effect: A more recent paper (Heckel & Hand, 2019) proposed a much simpler image generating network, termed the deep decoder.
This network can be seen as the relevant part of a convolutional generator architecture to function as an image prior, and can be obtained from a standard convolutional autoencoder by removing the encoder, the skip connections, and perhaps most notably, the trainable convolutional ﬁlters of spatial extent larger than one.
Thus, the deep decoder does not use learned or trainable convolutional ﬁlters like conventional convolutional networks do, and instead only uses convolutions with ﬁxed convolutional kernels to generate an image.
[Figure]
[Figure]
In this paper, we study such a simple, untrained convolutional network theoretically.
We consider the over-parameterized regime where the network has sufﬁciently many parameters to represent an arbitrary image (including noise) perfectly and show that:
Fitting convolutional generators via early stopped gradient descent provably denoises “natural” images.
To prove this statement, we characterize how the network architecture governs the dynamics of ﬁtting over-parameterzed networks to a single (noisy) image.
In particular we prove:
Convolutional generators optimized with gradient descent ﬁt natural images faster than noise.
We depict this phenomena in Figure 1 where we ﬁt a randomly initialized over-parameterized convolutional generator to an image via running gradient descent on the objective L ( C )
= (cid:107) G ( C ) − y (cid:107) 22 .
Here, G ( C ) is the convolutional generator with weight parameters C , and y is either a noisy image, a clean image, or noise.
This experiment demonstrates that an over-parameterized convolutional network ﬁts a natural image (Figure 1b) much faster than noise (Figure 1c).
Thus, when ﬁtting the noisy image (Figure 1a), early stopping the optimization enables image denoising.
This effect is so strong that it gives state-of-the-art denoising performance (see Figure 2 and Appendix A).
Beyond denoising, this effect also enables signiﬁcant improvements in regularizing a variety of inverse problems such as compressive sensing (Veen et al., 2018; Heckel, 2019).
1. C ONTRIBUTIONS AND OVERVIEW OF RESULTS
In this paper we take a step towards understanding why ﬁtting a convolutional generator via early stopped gradient descent leads to such surprising denoising and regularization capabilities.
[Figure] • We ﬁrst show experimentally that this denoising phenomena is primarily attributed to convolutions with ﬁxed interpolation kernels, typically implicitly implemented by bi-linear upsampling operations in the convolutional network.
• We then show that ﬁtting an over-parameterized convolutional networks (with ﬁxed convolutional ﬁlters) via early stopped gradient descent to a signal provably denoises it.
Speciﬁcally, let x ∈ R n be a smooth signal that can be represented exactly as a linear combination of the p orthogonal trigonometric functions of lowest frequency (deﬁned in equation (5), see Figure 3 for a depiction).
A smooth signal is a good model for a natural image since natural images are well approximated by low-frequency components.
Specifically, Figure 4 in (Simoncelli & Olshausen, 2001) shows that the power spectrum of a natural image (i.e., the energy distribution by frequency) decays rapidly from low frequencies to high frequencies.
In contrast, Gaussian noise has a ﬂat power spectrum.
Our goal is to obtain an estimate of this signal from a noisy observation y =
x + z where z ∼ N (0 , ς 2 n I ) is Gaussian noise.
Let ˆ y
=
G ( C τ ) be the estimate obtained from early stopping the ﬁtting of a two-layer convolutional generator to y .
We prove this estimate achieves the denoising rate
[Equation]
with c a constant.
This rate is optimal up to a constant factor.
• Our denoising result follows from a detailed analysis of gradient descent applied to ﬁtting a convolutional generator G ( C ) to a noisy signal y =
x + z with x representing the signal and z the noise.
We show that there are weights σ 1 , . . . , σ n associated with the trigonometric basis function that only depend on the convolutional ﬁlter used, and which typically decay quickly from the low-frequency to the high-frequency trigonometric basis functions.
These weights in turn determine the speed at which the associated components of the signal are ﬁtted.
Speciﬁcally, we show that the dynamics of gradient descent are approximately given by
[Equation]
The convolutional ﬁlters commonly used are such that the weights σ 1 , . . . , σ n decays quickly, implying that low-frequency components in the trigonometric expansion are ﬁtted signiﬁcantly faster than high frequency ones.
So if the signal mostly consists of lowfrequency components, we can choose an early stopping time such that the error in ﬁtting the signal is very low, and thus the signal part is well described, whereas at the same time only a small part of the noise, speciﬁcally the part aligned with the low-frequency components has been ﬁtted.
C ONVOLUTIONAL GENERATORS
A convolutional generator maps an input tensor B 0 to an image only using upsampling and convolutional operations, followed by channel normalization (a special case of batch normalization) and applications of non-linearities, see Figure 4.
All previously mentioned convolutional generator networks (Radford et al., 2015; Ronneberger et al., 2015) including the networks considered in the
DIP paper (Ulyanov et al., 2018) primarily consist of those operations.
For motivating the architecture of the convolutional generators studied in this paper, we ﬁrst demonstrate in Section 2.1 that convolutions with ﬁxed interpolation ﬁlters are critical to the denoising [Figure] performance with early stopping.
Speciﬁcally, we empirically show that convolutions with ﬁxed convolutional kernels are critical for convolutional generators to ﬁt natural images faster than noise.
In Section 2.2 we formally introduce the class of convolutional generators studied in this paper, and ﬁnally, in Section 2.3, we introduce a minimal convolutional architecture which is the focus of our theoretical results.
2. T HE IMPORTANCE OF FIXED CONVOLUTIONAL FILTERS
Convolutions with ﬁxed convolutional kernels are critical for denoising with early stopping, because they are critical for the phenomena that natural images are ﬁtted signiﬁcantly faster than noise.
To see this, consider the experiment in Figure 4 in which we ﬁt an image and noise by minimizing the least-squares loss via gradient descent with i) a convolutional generator with only ﬁxed convolutional ﬁlters (see Section 2.2 below for a precise description) and ii) a conventional convolutional generator with trainable convolutional ﬁlters (essentially the architecture from the popular DC-GAN generators, see Appendix B for details and additional numerical evidence).
As illustrated in Figure 4, the convolutional network with ﬁxed ﬁlters ﬁts the natural image much faster than noise, whereas the network with learned convolutional ﬁlters, only ﬁts it slightly faster, and this effect vanishes as the network becomes highly overparameterized.
Thus, ﬁxed convolutional ﬁlters enable un-trained convolutional networks to function as highly effective image priors.
We note that the upsampling operation present in most architectures implicitly incorporates a convolution with a ﬁxed convolutional (interpolation) ﬁlter.
2. A RCHITECTURE OF CONVOLUTIONAL GENERATOR WITH FIXED CONVOLUTIONS
In this section, we describe the architecture of the deep decoder (Heckel & Hand, 2019), a convolutional network with ﬁxed convolutional operators only.
In this architecture, the channels in the ( i + 1) -th layer are given by
[Equation]
and ﬁnally, the output of the d -layer network is formed as
[Equation]
Here, the coefﬁcient matrices C i ∈
R k × k and C d +1 ∈ R
k × k out contain the weights of the network.
The number of channels, k , determines the number of weight parameters of the network, given by dk 2 + k out k .
Each column of the tensor B i C i ∈ R n i × k is formed by taking linear combinations of the channels of the tensor B i in a way that is consistent across all pixels, and the ReLU activation function is given by ReLU( t ) = max(0 , t ) .
Then, cn( · ) performs the channel normalization operation which normalizes each channel individually and can be viewed as a special case of the popular batch normalization operation (Ioffe & Szegedy, 2015).
The operator U i ∈
R
n i +1 × n i is a tensor implementing an upsampling and most importantly a convolution operation with a ﬁxed kernel.
This ﬁxed kernel was chosen in all experiments above as a triangular kernel so that U performs bi-linear 2x upsampling (this is the standard implementation in the popular software packages pytorch and tensorﬂow).
As mentioned earlier this convolution with a ﬁxed kernel is critical for ﬁtting natural images faster than complex ones.
2. T WO LAYER CONVOLUTIONAL GENERATOR STUDIED THEORETICALLY IN THIS PAPER
The simplest model to study the denoising capability of convolutional generators and the phenomena that a natural image is ﬁtted faster than a complex one theoretically is a network with only one hidden layers and one output channel i.e., G ( C ) ∈ R n .
Then, the generator becomes
[Equation]
where U ∈ R n × n is a circulant matrix that implements a convolution with a ﬁlter u .
In this paper we consider the over-parameterized regime where k ≥ 2 n .
Note that scaling the i -th column of C 0 with a non-negativ factor is equivalent to scaling the i -th entry of the output weights c 1 with the same factor.
We therefore ﬁx the output weights to c 1 = v , where v = [1 , . . .
, 1 , − 1 , . . . , − 1] / √ k . Next, note that both the generators with the ﬁxed output weights and non-ﬁxed output weights have the same range in the regime k ≥ 2 n . Next, because the matrix B 0 is Gaussian, with probability one, it has full rank and thus spans R n .
It follows that optimizing over the parameter C 0 in C = B 0 C 0 is equivalent to optimizing over the matrix C ∈ R n × k .
We therefore consider the generator
[Equation]
where v = [1 , . . .
, 1 , − 1 , . . .
, − 1] / √ k is ﬁxed and C ∈ R n × k is the new coefﬁcient matrix we optimize over.
Figure 11 in the appendix shows that even this simple two-layer convolutional network ﬁts a simple image faster than noise.
This is the simplest model in which the phenomena that a convolutional networks ﬁts structure faster than noise can reliably be observed.
As a consequence, the dynamics of training the model (1) are the focus of the remainder of this paper.
W ARMUP : D YNAMICS OF GRADIENT DESCENT ON LEAST SQUARES
As a prelude for studying the dynamics of ﬁtting convolutional generators via a non-linear least square problem, we study the dynamics of gradient descent applied to a linear least squares problem.
We demonstrate how early stopping can lead to denoising capabilities even with a simple linear model.
We consider a least-squares problem of the form
[Equation]
and study gradient descent with a constant step size η starting at c 0 = 0 .
The updates are given by
[Equation]
The following simple proposition characterizes the trajectory of gradient descent.
Proposition 1.
Let J ∈ R n × m be a matrix with left singular vectors w 1 , . .
w n ∈ R n and corresponding singular values σ 1 ≥ σ 2 ≥ . .
≥ σ n .
Then the residual after τ steps, r τ = y − Jc τ , of gradient descent starting at c 0 = 0 is
[Equation]
Suppose that the signal y lies in the column span of J , and that the stepsize is chosen sufﬁciently small (i.e., η ≤ 1 / (cid:107) J (cid:107) 2 ).
Then, by Proposition 1, gradient descent converges to a zero-loss solution and thus ﬁts the signal perfectly.
More importantly, gradient descent ﬁts the components of y corresponding to large singular values faster than it ﬁts the components corresponding to small singular values.
To explicitly show how this observation enables regularization via early stopped gradient descent, suppose our goal is to ﬁnd a good estimate of a signal x from a noisy observation y
= x
+ z ,
i s i
ngu l a r v a l u e σ i  1 ,0 .0 .
1 .1 .τ
(cid:107) y − J c τ (cid:107)  1 ,0 .0 .0 .τ
(cid:107) w− J c τ (cid:107) where the signal x lies in a signal subspace that is spanned by the p leading left-singular vectors of
J .
Then, by Proposition 1, the signal estimate after τ iterations, Jc τ , obeys
[Equation]
Thus, after a few iterations most of the signal has been ﬁtted (i.e., (1 − ησ p ) τ is small).
Furthermore, if we assume that the ratio σ p /σ
p
+1 is sufﬁciently large so that the spread between the two singular values separating the signal subspace from the rest is sufﬁciently large, most of the noise outside the signal subspace has not been ﬁtted (i.e., ((1 − ησ 2 i )
τ − 1) 2 ≈ 0 for i
= p + 1 , . .
. , n ).
In particular, suppose the noise vector has a Gaussian distribution given by z ∼ N (0 , ς 2 n I ) .
Then
E ( z ) ≈ ς (cid:112) pn so that after order τ = log( (cid:15) ) /
log(1 − ησ 2 p ) iterations, with high probability, (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:114) [Equation] n. This demonstrates, that provided the signal lies in a subspace spanned by the leading singular vectors, early stoped gradient descent reaches the optimal denoising rate of ς (cid:112) p/n after a few iterations.
See Figure 5 for a numerical example demonstrating this phenomena.
D YNAMICS OF GRADIENT DESCENT ON CONVOLUTIONAL GENERATORS
We are now ready to study the implicit bias of gradient descent towards natural/structured images theoretically.
Consider the two-layer network (introduced in Section 2.3) of the form
[Equation]
where v = [1 , . . .
, 1 , − 1 , . . .
, − 1] / √ k , and with weight parameter C ∈ R n × k , and recall that U is a circulant matrix implementing a convolution with a kernel u ∈ R n .
We ﬁt the generator to a signal y ∈ R n by minimizing the non-linear least squares objective [Equation]
[Equation]
) with (early-stopped gradient) descent with a constant stepsize η starting at a random initialization
C 0 of the weights.
The iterates are given by
[Equation]
In our warmup section on linear least squares we saw that the singular vectors and values of the matrix J determine the speed at which different components of the noisy signal y are ﬁtted by gradient descent.
The main insight that enables us to extend this intuition to the nonlinear case is that the role of the matrix J can be replaced with the Jacobian of the generator, deﬁned as J ( C ) :
= ∂ ∂ C G ( C ) .
[Figure] Contrary to the linear least squares problem, however, in the nonlinear case, the Jacobian is not constant and changes across iterations.
Nevertheless, we show that the eigen-values and vectors of the Jacobian at the random initialization govern the dynamics of ﬁtting the network throughout the iterative updates.
For the two-layer convolutional generator that we consider, the left eigenvectors of the Jacobian mapping can be well approximated by the trigonometric basis functions, deﬁned below, throughout the updates.
Interestingly, the form of these eigenvectors only depends on the network architecture and not the convolutional kernel used.
Deﬁnition 1.
The trigonometric basis functions w 1 , . . .
, w n are deﬁned as
[Equation]
Figure 3 depicts some of these eigenvectors.
In addition to the left eigenvectors we can also approximate the spectrum of the Jacobian throughout the updates by an associated ﬁlter/kernel that only depends on the original ﬁlter/kernel used in the network architecture.
Deﬁnition 2 (Dual kernel) .
Associated with a kernel u ∈ R n we deﬁne the dual kernel σ ∈ R n as σ = [Equation] u [Equation]
(cid:117)(cid:116) (cid:12)(cid:12) (cid:12) (cid:12)(cid:12)
[Equation]
(cid:32)
[Equation]
(cid:126)
[Equation]
(cid:107)
[Equation]
(cid:107)
{+[Equation] (cid:12)(cid:12) (cid:12) (cid:12)(cid:12) with g ( t ) = 
 [Equation]+} Here, for two vectors u , v ∈ R n , u (cid:126) v denotes their circular convolution, the scalar non-linearity g is applied entrywise, and F is the discrete Fourier transform matrix.
In Figure 6, we depict two commonly used interpolation kernels u , namely a triangular and a Gaussian kernel (recall that the standard upsampling operator is a convolution with a triangle), along with the induced dual kernel σ .
The ﬁgure shows that the dual kernel σ induced by these kernels has a few large values associated with the low frequency trigonometric functions, and the other values associated with high frequencies are very small.
With these deﬁnitions in place we are ready to state our main denoising result.
A denoising result requires a signal model—we assume a low-frequency signal x that can be represented as a linear combination of the ﬁrst p -trigonometric basis functions.
Note that this is a good model for a natural image since natural images are well approximated by low-frequency components.
Speciﬁcally,
Figure 4 in (Simoncelli & Olshausen, 2001) shows that the power spectrum of a natural image (i.e., the energy distribution by frequency) decays rapidly from low frequencies to high frequencies.
In contrast, Gaussian noise has a ﬂat power spectrum.
Theorem 1 (Denoising with early stopping) .
Let x ∈ R n be a signal in the span of the ﬁrst p trigonometric basis functions, and consider a noisy observation
[Equation]
where z is Gaussian noise with distribution N ( 0 , ς 2 n I ) , for some variance ς 2 ≥ 0 .
To denoise this signal, we ﬁt a two layer generator network G ( C ) =
ReLU( UC ) v with,
[Equation]
channels, for some (cid:15) > 0 , and with convolutional kernel u of the convolutional operator U and associated dual kernel σ , to the noisy signal y .
Here, C u a constant only depending on the convolutional kernel u .
Then, with probability at least 1 − e − k 2 − 1 n 2 , the reconstruction error obtained after τ = log(1 − (cid:112) p/n ) / log(1 − ησ 2 p +1 ) iterations of gradient descent (4) with step size η ≤
[Equation]
( Fu is the Fourier transform of u ) starting from C 0 with i.i.d.
N (0 , ω 2 ) , entries, ω ∝ (cid:107) y [Equation] 2 √ n , is bounded by
[Equation]
Note that for this choice of stopping time, provided that dual kernel decays sharply around the p -th singular value, the ﬁrst term in the bound (7) (i.e., (1 − ησ 2 p ) τ ≈ 0 ) essentially vanishes and the error bound becomes O ( ς (cid:112) pn ) .
The dual kernel decays sharply around the leading eigenvalues provided the kernel is for example a sufﬁciently wide triangular or Gaussian kernel (see Figure 6).
This result demonstrates that when the noiseless signal x is sufﬁciently structured (e.g. contains only the p lowest frequency components in the trigonometric basis) and the convolutional generator has sufﬁciently many channels, then early stopped gradient descent achieves a near optimal denoising performance proportional to ς (cid:112) pn .
This theorem is obtained from a more general result stated in the appendix which characterizes the evolution of the reconstruction error obtained by the convolutional generator.
Theorem 2 (Reconstruction dynamics of convolutional generators) .
Consider the setting and assumptions of Theorem 1 but now with a ﬁxed noise vector z , and without an explicit stopping time.
Then, for all iterates τ obeying τ ≤ 100 ησ 2 p and provided that k ≥ C u n/(cid:15) 8 , for some (cid:15) ∈ (0 , σ p σ 1 ] , with probability at least 1 − e − k 2 − 1 n 2 , the reconstruction error obeys
[Equation]
This theorem characterizes the reconstruction dynamics of convolutional generators throughout the updates.
In particular, it explains why convolutional generators ﬁt a natural signal signiﬁcantly faster than noise, and thus early stopping enables denoising and regularization.
To see this, note that as mentioned previously each of the basis functions w i has a (positive) weight σ i > 0 associated with it that only depends on the convolutional kernel used in the architecture (through the deﬁnition of the dual kernel).
These weights determine how fast the different components of the noisy signal are ﬁtted by gradient descent.
As we demonstrated earlier in Figure 6, for typical convolutional ﬁlters those weights decay very quickly from low to high frequency basis functions.
As a result, when the signal x is sufﬁciently structured (i.e. lies in the range of the p trigonometric functions with lowest frequencies), after a few iterations most of the signal is ﬁtted (i.e., (1 − ησ 2 p ) τ is small), while most of the noise has not been ﬁtted (i.e., ((1 − ησ 2 i )
τ − 1) 2 ≈ 0 for i
= p + 1 , . .
. , n ).
Thus, early stopping achieves denoising.
The proof, provided in the appendix, is based on associating the following linear least-squares problem with the non-linear least squares problem (3):
[Equation]
Here, vect( C 0 ) ∈ R kn is a vectorized version of the matrix C 0 , and J ∈ R n × nk approximates the Jacobian of G ( C ) around the initialization C 0 .
The linear least-squares problem is obtained by linearizing the non-linear problem around the initialization.
In the proof we show that if i) the network is sufﬁciently over-parameterized and if ii) the network is randomly initialized, then the linearized problem is a good approximation of the non-linear problem.
We then show that the singular values of the Jacobian J are the trigonometric basis functions, and the singular values are the values of the dual kernel.
The proof is then concluded by characterizing the trajectory of gradient descent applied to the linear least-squares problem above.
4. M ULTILAYER NETWORKS AND MODERATE OVER - PARAMETERIZATION
Our theoretical results rely on the ﬁnding that for over-parameterized single hidden-layer networks, the leading singular vectors of the Jacobian are the trigonometric functions throughout all iterations, and that the associated weights (i.e., the singular values) are concentrated towards the low frequency components.
In this regime, the dynamics are well approximated by a linear model.
This general strategy can be extended to multi-layer networks, however if the network is not highly overparameterized, an associated linear model might not be a good approximation.
In order to understand whether our ﬁnding of low-frequency components being being ﬁtted faster than high-frequency ones carries over to multi-layer networks and to the moderately overparameterized regime, in this section, we study muli-layer networks in the moderately overparameterized regime numerically.
We show that for a multilayer network, the spectrum of the Jacobian is concentrated towards singular vectors/functions that are similar to the low-frequency components.
We also show that throughout training those functions do vary, albeit the low frequency components do not change signiﬁcantly and the spectrum remains concentrated towards the low frequency components.
This shows that the implications of our theory continue to apply to muli-layer networks.
In more detail, we take a standard one dimensional deep decoder with d = 4 layers with output in Rand with k = 64 channels in each layer.
Recall that the standard one dimensional decoder obtains layer i +1 from layer i by linearly combining the channels of layer i with learnable coefﬁcients followed by linear upsampling (which involves convolution with the triangular kernel [1 / 2 , 1 , 1 / 2] ).
The number of parameters is d × k 2 = 32 · 512 , so the network is over-parameterized by a factor of 32 .
In Figure 7, we display the singular values as well as the leading singular vectors/function of the
Jacobian at initialization ( t = 1 ) and after t = 50 and t = 3500 iterations of gradient descent.
As can be seen the leading singular vectors ( s = 1 − 5 ) are close to the trigonometric basis functions and do not change dramatically throughout training.
The singular vectors corresponding to increasingly smaller singular values ( s = 20 , 50 , 100 , 150 ) contain increasingly higher frequency components but are far from the high-frequency trigonometric basis functions.
R ELATED LITERATURE
As mentioned before, the DIP paper (Ulyanov et al., 2018) was the ﬁrst to show that overparameterized convolutional networks enable solving denoising, inpainting, and super-resolution problems well even without any training data.
Subsequently, the paper (Heckel & Hand, 2019) proposed a much simpler image generating network, termed the deep decoder.
The papers (Veen et al., 2018; Heckel, 2019; Jagatap & Hegde, 2019; Bostan et al., 2020) have shown that the DIP and the deep decoder also enable solving or regularizing compressive sensing problems and other inverse problems.
Since the convolutional generators considered here are image-generating deep networks, our work is also related to methods that rely on trained deep image models.
Deep learning based methods are either trained end-to-end for tasks ranging from compression (Toderici et al., 2016;
Agustsson et al., 2017; Theis et al., 2017; Burger et al., 2012; Zhang et al., 2017) to denoising (Burger et al., 2012;
Zhang et al., 2017), or are based on learning a generative image model (by training an autoencoder or GAN (Hinton & Salakhutdinov, 2006; Goodfellow et al., 2014)) and then using the resulting model to solve inverse problems such as compressed sensing (Bora et al., 2017; Hand & Voroninski, 2018), denoising (Heckel et al., 2018), or phase retrieval (Hand et al., 2018), by minimizing an associated loss.
In contrast to the method studied here, where the optimization is over the weights
[Figure]
t =at t =
[Figure]
of the network
, in all the aforementioned methods, the weights are adjusted only during training and then are ﬁxed upon solving the inverse problem.
A large body of work focuses on understanding the optimization landscape of the simple nonlinearities or neural networks (Candes et al., 2015; Soltanolkotabi, 2017; Alon Brutzkus & Globerson, 2017; Zhong et al., 2017; Oymak, 2018; Fu et al., 2018; Tu et al., 2016) when the labels are created according to a planted model.
Our proofs rely on showing that the dynamics of gradient descent on an over-parameterized network can be related to that of a linear network or a kernel problem.
This proof technique has been utilized in a variety of recent publication (Soltanolkotabi et al., 2018;
Venturi et al., 2019; Du et al., 2018; Oymak & Soltanolkotabi, 2019a;b; Arora et al., 2019; Oymak et al., 2019).
Two recent publication have used this proof technique to show that functions are learned at different rates: Basri et al.
(2019) have shown that functions of different frequencies are learned at different speeds, and Arora et al.
(2019) has provided a theoretical explanation of the empirical observation that a simple 2-layer network ﬁts random labels slower than actual labels in the context of classiﬁcation.
A recent publication (Li et al., 2019) focuses on demonstrating how early stopping leads to robust classiﬁcation in the presence of label corruption under a cluster model for the input data.
Neither of the aforementioned publication however, does address denoising in a regression setting or ﬁtting convolutional generators of the form studied in this paper.
C ODE AND A CKNOWLEDGEMENTS
Code to reproduce the experiments is available at https://github.com/MLI-lab/ overparameterized_convolutional_generators . R. Heckel is partially supported by NSF award IIS-1816986, acknowledges support of the NVIDIA Corporation in form of a GPU, and would like to thank Tobit Klug for proofreading a previous version of this manuscript.
M. Soltanolkotabi is supported by the Packard Fellowship in Science and Engineering, a Sloan Research Fellowship in Mathematics, an NSF-CAREER under award #1846369, the Air Force Ofﬁce of Scientiﬁc Research Young Investigator Program (AFOSR-YIP) under award #FA9550-18-1-0078, an NSF-CIF award #1813877, DARPA under the Learning with
Less Labels (LwLL) program, and a Google faculty research award.
A D ENOISING PERFORMANCE OF UNTRAINED CONVOLUTIONAL GENERATORS
In this section, we provide further details on the denoising performance of deep neural networks.
We compare the denoising performance of four methods: i) the BM3D algorithm (Dabov et al., 2007) as a standard baseline for denoising, ii) the deep image prior (Ulyanov et al., 2018), which is a U-net like encoder-decoder convolutional architecture applied with exactly the parameters as proposed for denoising in the paper (Ulyanov et al., 2018), and early stopped after 1900 iterations, again with the same stopping time as proposed in the original paper, iii) an under-parameterized deep decoder with ﬁve layers and k = 128 channels in each layers, trained for 3000 iterations (which is close to convergence), iv) an over-parameterized deep decoder with ﬁve layers and k = 512 channels in each layer, early stopped at 1500 iterations.
We compared the performance of those four methods on denoising 100 randomly chosen images from the ImageNet validation set.
The code to reproduce the results contains a list of the images we considered.
Each image has 512 × 512 pixels and three color channels.
We added the same random Gaussian noise to each color channel, because we are interested in evaluating the performance by imposing structural assumptions on the image.
Tablebelow show that the overparameterized deep decoder with early stopping performs best for this task.
[Table]
.52 dB
B A NUMERICAL STUDY OF THE IMPLICIT BIAS OF CONVOLUTIONAL NETWORKS
In this section, we empirically demonstrate that convolutions with ﬁxed convolutional kernels are critical for convolutional generators to ﬁt natural images faster than noise.
Towards this goal, we study numerically the following four closely related architectural choices, which differ in the upsampling/no-upsampling and convolutional operations which generate the activations in the ( i + 1) -st layer, B i +1 , from the activations in the i -th layer, B i :
i)
Bilinear upsampling and linear combinations.
Layer i + 1 is obtained by linearly combining the channels of layer i with learnable coefﬁcients (i
e., performing one-times-one convolutions), followed by bi-linear upsampling.
This is the deep decoder architecture from (Heckel et al., 2018).
ii) Fixed interpolation kernels and linear combinations.
Layer i + 1 is obtained by linearly combining the channels of layer i with learnable coefﬁcients followed by convolving each channel with the same 4x4 interpolation kernel that is used in the linear upsampling operator.
iii)
Parameterized convolutions: Layer i + 1 is obtained from layer i though a convolutional layer.
iv) Deconvolutional network: Layer i + 1 is obtained from layer i though a deconvolution layer.
This is essentially the DC-GAN (Radford et al., 2015) generator architecture. [Figure]
To emphasize that architectures i)-iv) are structurally very similar operations, we recall that each operation consists only of upsampling and convolutional operations.
Let T ( c ): R n → R n be the convolutional operator with kernel c , let u the linear upsampling kernel (equal to u = [0 .
5 , 1 , 0 . 5] in the one-dimensional case), and let U : R n → R 2 n be an upsampling operator, that in the one dimensional case transforms [ x 1 , x 2 , . . .
, x n ] to [ x 1 , 0 , x 2 , 0 , . . .
x n , 0] .
In each of the architectures i)-iv), the (cid:96) -th channel of layer i + 1 is obtained from the channels in the i -th layer as: b i +1 ,(cid:96) =
ReLU
[Equation]
) b i (cid:17) , where the linear operator M is deﬁned as follows for the four architectures
[Equation]
, iv) M ( c ) =
T ( c ) U .
The coefﬁcients associated with the i -th layer are given by C i
= { c ij(cid:96) } , and all coefﬁcients of the networks are C
= { c ij(cid:96) } .
Note that here, the coefﬁcients or parameters of the networks are the weights and not the input to the network.
B. D EMONSTRATING IMPLICIT BIAS OF CONVOLUTIONAL GENERATORS
We next show that convolutional generators with ﬁxed convolutional operations ﬁt natural or simple images signiﬁcantly faster than complex images or noise.
Throughout this section, for each image or signal x ∗ we ﬁt weights by minimizing the loss
[Equation]
with respect to the network parameters C using plain gradient descent with a ﬁxed stepsize.
In order to exemplify the effect, we ﬁt the phantom MRI image as well as noise for each of the architectures above for a 5 -layer network.
We choose the number of channels, k , such that the over-parameterization factor (i.e., the ratio of number of parameters of the network over the output dimensionality) is 1 , 4 , and 16 , respectively.
The results in Figure 8 show that for architectures i) and ii) involving ﬁxed convolutional operations , gradient descent requires more than one order of magnitude fewer iterations to obtain a good ﬁt of the phantom MRI image relative to noise.
For architectures iii) and iv), with trainable convolutional ﬁlters, we see a smaller effect, but the effect essentially vanishes when the network is highly over-parameterized.
This effect continues to exist for natural images in general, as demonstrated by Figure 10 which depicts the average and standard deviation of the loss curves of 100 randomly chosen images from the imagenet dataset.
We also note that the effect continues to exist in the sense that highly structured images with a large number of discontinuities are difﬁcult to ﬁt.
An example is the checkerboard image in which each pixel alternates between 1 and 0; this image leads to the same loss curves as noise.
In our next experiment, we highlight that the distance between ﬁnal and initial network weights is a key feature that determines the difference of ﬁtting a natural image and noise. Towards this goal,
[Figure] noise [Figure] we again ﬁt the phantom MRI image and noise for the architecture i) and an over-parameterization factor of 4 and record, for each layer i the relative distance (cid:107) C ( t ) i − C (0)
i (cid:107) / (cid:107) C (0) i (cid:107) , where C (0) i are the weights at initialization (we initialize randomly) and C ( t ) i are the weights at the optimizer step t .
The results, plotted in Figure show that to ﬁt the noise, the weights have to change signiﬁcantly, while for ﬁtting a natural image they only change slightly.
C T HE SPECTRUM OF THE J ACOBIAN OF THE DEEP DECODER AND DEEP IMAGE PRIOR
Our theoretical results predict that for over-parameterized networks, the parts of the signal that is aligned with the leading singular vectors of the Jacobian at initialization is ﬁtted fastest.
In this section we brieﬂy show that natural images are much more aligned with the leading singular vectors than with Gaussian noise, which is equally aligned with each of the singular vectors.
[Figure]
[Figure]
Towards this goal, we compute the norm of the product of the Jacobian at a random initalization, C 0 , with a signal y as this measures the extent to which the signal is aligned with the leading singular vectors, due to
[Equation]
where J ( C 0 )
=
W Σ V T is the singular value decomposition of the Jacobian.
Figure 12 depicts the distribution of the norm of the product of the Jacobian at initialization, J ( C 0 ) , with an image y ∗ or noise z of equal norm ( (cid:107) y ∗ (cid:107)
= (cid:107) z (cid:107) ).
Since for both the deep decoder and the deep image prior, the norm of product of the Jacobian and the noise (i.e., (cid:13)(cid:13)
J T ( C 0 )
z (cid:13)(cid:13) ) is signiﬁcantly smaller than that with a natural image (i.e., (cid:13)(cid:13)
J T ( C 0 )
y ∗ (cid:13)(cid:13) ), it follows that a structured image is much better aligned with the leading singular vectors of the Jacobian than the Gaussian noise, which is approximately equally aligned with any of the singular vectors.
Thus, the Jacobian at random initialization has an approximate low-rank structure, with natural images lying in the space spanned by the leading singularvectors.
D P ROOFS AND FORMAL STATEMENT OF RESULTS
The results stated in the main text are obtained from a slightly more general result which applies beyond convolutional networks.
Consider neural network generators of the form
[Equation]
with C ∈ R n × k , and U ∈ R n × n an arbitrary ﬁxed matrix, and v ∈ R k , with half of the entries equal to +1 / √ k and the other half equal to − 1 / √ k .
The (transposed) Jacobian of ReLU( Uc ) is U T diag(ReLU (cid:48)
( Uc )) .
Thus the Jacobian of G ( C ) is given by
[Equation]
where ReLU (cid:48) is the derivative of the activation function.
Next we deﬁne a notion of expected Jacobian. Towards this goal, we ﬁrst deﬁne the matrix
[Equation]
associated with the generator G ( C ) =
ReLU( UC ) v .
Here, expectation is over C with iid N (0 , ω 2 ) entries.
Note that since the derivative of the ReLU non-linearity is invariant to the scaling of C , so is the Jacobian J ( C ) , but to be consistent throughout we take expectation over C with iid N (0 , ω 2 ) entries.
Consider the eigenvalue decomposition of Σ ( U ) given by
[Equation]
Our results depend on the largest and smallest eigenvalue of Σ ( U ) , deﬁned throughout as
[Equation]
With these deﬁnitions in place we are ready to state our result about neural generators.
Theorem 3.
Consider a noisy signal y ∈ R n given by
[Equation]
, where x ∈ R n is assumed to lie in the signal subspace spanned by the p leading singular vectors w 1 , . .
w n of Σ ( U ) , and z ∈ R n is an arbitrary noise vector.
Suppose that the number of channels obeys
[Equation]
where c is a ﬁxed numerical constant and ξ an error tolerance parameter obeying 0 < ξ ≤ 1 / (cid:113) 32 log (cid:0) 2 nδ (cid:1) . Moreover, T is the maximal number of gradient descent steps, and we assume it obeys 1 ≤ T ≤ 2 5 β 2 ηξ 2 α 2 .
We ﬁt the neural generator G ( C ) to the noisy signal y ∈ R n by minimizing the loss
[Equation]
via running gradient descent starting from C 0 with i.i.d.
N (0 , ω 2 ) entries, ω
= (cid:107) y (cid:107) 2 √ nβ ξ α 2 β 2 , and step size obeying η ≤ 1 /β 2 .
Then, with probability at least 1 − e − k 2 − δ , for all iterations τ ≤ T ,
[Equation] [Equation]
D. P ROOF OF T HEOREM
Theorem 2 stated in the main text follow directly from Theorem 3 above as follows.
We ﬁrst note that for a convolutional generator (where U implements a convolution and thus is circulant) the eigenvectors of the matrix Σ ( U ) are given by the trigonometric basis functions per Deﬁnition 1 and the eigenvalues are the square of the entries of the dual kernel (Deﬁnition 2).
To see this, we note that as detailed in Section H,
[Equation]
Because the matrix U implements a convolution with a kernel u that is equal to its ﬁrst column, the matrix Σ ( U ) is again a circulant matrix and is also Hermitian.
Thus, its spectrum is given by the Fourier transform of the ﬁrst column of the circulant matrix, and its left-singular vectors are given by the trigonometric basis functions deﬁned in equation (5) and depicted in Figure 3.
Furthermore, using the fact that the eigenvalues of a circulant matrix are given by its discrete Fourier transform we can substitute β = (cid:107) U (cid:107) =
(cid:107) Fu (cid:107) ∞ and α = σ n > 0 .
With the assumption τ ≤ T =ησ 2 p , and using that
[Equation]
where we used the assumption ξ ≤ σ p β , the condition (10) is implied by the assumption k ≥ C u n/ξ 8 , with C u ∝ β 18 α 18 (see equation (6), where this assumption is made).
This yields Theorem 2.
D. P ROOF OF T HEOREM
Finally, we note that to obtain the simpliﬁed ﬁnal expression in Theorem 1 from Theorem 2 we also used the fact that for a Gaussian vector z , the vector W T z is also Gaussian.
Furthermore, by the concentration of Lipschitz functions of Gaussians with high probability we have
[Equation]
Here, equation (i) follows from (cid:104) w i , z (cid:105) being zero mean Gaussian with variance
ς 2 /n (since z ∼
N (0 , ( ς 2 /n ) I ) , and (cid:107)
w i (cid:107) 2 = 1 ).
Finally, (ii) follows by choosing the early stopping time so that it obeys τ ≤ log(1 − (cid:112) p/n ) / log(1 − ησ 2 p +1 ) which in turn implies that (1 − ησ 2 i ) τ ≥ 1 − (cid:112) p/n , for all i > p , yielding that ((1 − ησ 2 i ) τ − 1) 2 ≤ p/n , for all i > p .
E T HE DYNAMICS OF LINEAR AND NONLINEAR LEAST - SQUARES
Theorem 3 builds on a more general result on the dynamics of a non-linear least squares problem which applies beyond convolutional networks, and that is stated and discussed in this section.
Consider a nonlinear least-squares ﬁtting problem of the form
[Equation]
Here, f : R N → R n is a non-linear model with parameters θ ∈ R N .
To solve this problem, we run gradient descent with a ﬁxed stepsize η , starting from an initial point θ 0 , with updates of the form θ τ +
[-τ +
-][Equation]
=
[Equation]
η ∇L ( θ τ ) where ∇L ( θ ) = J T ( θ )( f ( θ ) − y ) . Here, J ( θ ) ∈ R n × N is the Jacobian associated with the nonlinear map f with entries given by [ J ( θ )] i,j = ∂f
i ( θ ) ∂ θ j .
In order to study the properties of the gradient descent iterates (14), we relate the non-linear least squares problem to a linearized one in a ball around the initialization θ 0 .
We note that this general strategy has been utilized in a variety of recent publications (Du et al., 2018;
Arora et al., 2019; Oymak & Soltanolkotabi, 2019b; Oymak et al., 2019).
The associated linearized least-squares problem is deﬁned as
[Equation]
Here, J ∈ R n × p , refered to as the reference Jacobian, is a ﬁxed matrix independent of θ that approximates the Jacobian mapping at initialization, J ( θ 0 ) .
Starting from the same initial point θ 0 , the gradient descent updates of the linearized problem are
[Equation]
To show that the non-linear updates (14) are close to the linearized iterates (16), we make the following assumptions:
Assumption 1 (Bounded spectrum) .
We assume the singular values of the reference Jacobian obey
[Equation]
Furthermore, we assume that the Jacobian mapping associated with the nonlinear model f obeys
[Equation]
all θ ∈ R
[Equation]
Assumption 2 (Closeness of the reference and initialization Jacobians) .
We assume the reference
Jacobian and the Jacobian of the nonlinearity at initialization J ( θ 0 ) are (cid:15)
-close in the sense that
[Equation]
[Equation]
Assumption 3 (Bounded variation of Jacobian around initialization) .
We assume that within a radius
R around the initialization, the Jacobian varies by no more than (cid:15) in the sense that
[Equation]
for all
[Equation]
where B R ( θ 0 ) :
= { θ : (cid:107) θ − θ 0 (cid:107) ≤ R } is the ball with radius R around θ 0 .
Our ﬁrst result shows that under these assumptions the nonlinear iterative updates (14) are intimately related to the linear iterative updates (16).
Speciﬁcally, we show that the residuals associated with these two problems deﬁned below
[Equation]
are close in the proximity of the initialization.
Theorem 4 (Closeness of linear and nonlinear least-squares problems) .
Assume the Jacobian mapping J ( θ ) ∈ R n × N associated with the function f ( θ ) obeys Assumptions 1, 2, and 3 around an initial point θ 0 ∈ R N with respect to a reference Jacobian J ∈ R n × N and with parameters α, β, (cid:15) 0 , (cid:15) , and R .
Furthermore, assume the radius R is given by
[Equation]
(23) with T a constant obeying 1 ≤ T ≤ 1 2 η(cid:15) 2 , and J † the pseudo-inverse of J .
We run gradient descent with stepsize η ≤ 1 β 2 on the linear and non-linear least squares problem, starting from the same initialization θ 0 .
Then, for all τ ≤ T the iterates of the original and the linearized problems and their corresponding residuals obey [Equation] [Equation]
[Equation]
Moreover, for all iterates τ ≤ T ,
[Equation]
The above theorem formalizes that in a (small) radius around the initialization, the non-linear problem behaves similarly as its linearization.
Thus, to characterize the dynamics of the nonlinear problem, it sufﬁces to characterize the dynamics of the linearized problem.
This is the subject of our next theorem.
Theorem 5.
Consider a linear least squares problem (15) and let J = WΣV T ∈ R n × N = (cid:80) ni
{+=
+}
σ i w i v Ti be the singular value decomposition of the matrix J .
Then the residual (cid:101) r
τ after τ iterations of gradient descent with updates (16) is
[Equation]
Moreover, using a step size satisfying η ≤ 1 σ 21 , the linearized iterates (16) obey
[Equation]
In the next section we combine these two general theorems to provide guarantees for denoising using general neural networks.
E. P ROOF OF T HEOREM 4 ( CLOSENESS OF LINEAR AND NON - LINEAR LEAST - SQUARES )
The proof is by induction.
We suppose the statement, in particular the bounds (24), (25), and (26) hold for iterations t ≤ τ − 1 .
We then show that they continue to hold for iteration τ in four steps.
In Step I, we show that a weaker version of (26) holds, speciﬁcally that (cid:107) θ τ − θ 0 (cid:107) 2 ≤ R .
Using this result, in Steps II and III we show that the bounds (24) and (25) hold, respectively.
Finally, in
Step IV we utilize Steps I-III to complete the proof of equation (26).
Step I: Next iterate obeys θ τ ∈ B R ( θ 0 ) .
To prove θ τ ∈ B R ( θ 0 ) , ﬁrst note that by the triangle inequality and the induction assumption (26) we have
[Equation]
So to prove (cid:107) θ τ − θ 0 (cid:107) 2 ≤ R it sufﬁces to show that (cid:107) θ τ − θ τ − 1 (cid:107) 2 ≤ R/ 2 .
To this aim note that
[Equation]
[Table]
≤ R
In the above (i) follows from [-(cid:15) 
 + (cid:15) ≤ 12 κ (which is a consequence of (24))-]Assumptions 1, 2, and 3.
For (ii), we bounded the last term with the induction hypothesis (24), the middle term with (cid:107) (cid:101) r τ − 1 (cid:107) 2 ≤ (cid:107) r 0 (cid:107) 2 , and the ﬁrst term with the bound
[Equation]
Finally, the last inequality follows by deﬁnition of R in (23) together with the fact that T ≥ 1 .
Step II: Original and linearized residuals are close: In this step, we bound the deviation of the residuals of the original and linearized problem deﬁned as
[Equation]
This step relies on the following lemma, which is a variant of (Oymak et al., 2019, Lem. 6.7). Lemma 1 (Bound on growth of perturbations) .
Suppose that Assumptions 1, 2, and 3 hold and that θ τ , θ τ +1 ∈ B R ( θ 0 ) .
Then, provided the stepsize obeys η ≤ 1 /β 2 , the deviation of the residuals obeys
[Equation]
By the previous step, θ τ , θ τ +1 ∈ B R ( θ 0 ) .
We next bound the two terms on the right hand side of (29).
Regarding the ﬁrst term, we note that an immediate consequence of Theorem 5 is the following bound on the residual of the linearized iterates:
[Equation]
In order to bound the second term in (29), namely, (cid:107) e τ (cid:107) 2 , we used the following lemma, proven later in Section E.1.1.
Lemma 2.
Suppose that for positive scalars α, η, ρ, ξ > 0 , η ≤ 1 /α 2 , the sequences (cid:101) r τ and e τ obey [Equation]
[Equation]
Then, for all τ ≤ 1 2 η(cid:15) 2 , we have that
[Equation]
With these lemmas in place we now have all the tools to prove that the original and linear residuals are close.
In particular, from Step I, we know that θ τ ∈ B R ( θ 0 ) so that the assumptions of Lemmaare satisﬁed.
Lemma 1 implies that the assumption (32) of Lemma 2 is satisﬁed with ξ = β ( (cid:15) {+
+} + (cid:15) ) and the bound (30) implies that the assumption (31) of Lemma 2 is satisﬁed with ρ = (cid:107) r 0 (cid:107) 2 .
Thus,
Lemma 2 implies that for all τ ≤ 1 2 η(cid:15)
[Equation]
This concludes the proof of (24).
Step III: Original and linearized parameters are close: First note that by the triangle inequality and Assumptions 2 and 3 we have
[Equation]
The difference between the parameter of the original iterate θ and the linearized iterate (cid:101) θ obey
[Equation]
Here, inequality (i) follows from (35) and an application of Assumption 1, inequality (ii) from (30), inequality (iii) from η ≤ 1 /β 2 which implies (1 − ηα 2 ) ≥ 0 and (34).
This concludes the proof of the bound (25).
Step IV:
Completing the proof of (26) :
By the triangle inequality
[Equation]
where inequality (i) follows from the bound (25), which we just proved, and the fact that, from equation (36) in Theorem 5,
[Equation]
Finally, inequality (ii) follows from the deﬁnition of R in equation (23) along with τ ≤ T , by assumption.
E.1. P ROOF OF L EMMA
We prove the result by induction.
Assume equation (33) holds true for some τ .
We prove that then it also holds true for τ + 1 .
By the two assumptions in the lemma,
[Equation]
where (i) follows from the induction assumption (33).
Summing up the difference of the errors gives
[Equation]
where the last inequality follows from the assumption that τ ≤ 1 2 η(cid:15) 2 .
E. P ROOF OF T HEOREM
The proof of identity (27) is equivalent to the proof of Proposition (1).
Regarding inequality (28), note that
[Equation]
Thus
[Equation]
This in turn implies that
[Equation]
F P ROOFS FOR NEURAL NETWORK GENERATORS ( PROOF OF T HEOREM 3)
The proof of Theorem 3 relies on the fact that, in the overparameterized regime, the non-linear least squares problem is well approximated by an associated linearized least squares problem.
Studying the associated linear problem enables us to prove the result.
We apply Theorem 4, which ensures that the associated linear problem is a good approximation of the non-linear least squarest problem, with the nonlinearity G ( C ) =
ReLU( UC ) v and with the parameter given by θ = C .
Recall that v is a ﬁxed vector with half of the entries 1 / √ k , and the other half − 1 / √ k .
As the reference Jacobian in the associated linear problem, we choose a matrix J ∈ R n × nk , that is very close to the original Jacobian at initialization, J ( C 0 ) , and that obeys JJ T = E (cid:2)
J ( C ) J T ( C ) (cid:3) .
The concrete choice of J is discussed in the next paragraph.
We apply Theorem 4 with the following choices of parameters:
[Equation]
A few comments on the intuition behind those choices: As shown below, α and β are chosen so that
Assumption 1 is satisﬁed. Regarding the choices of (cid:15), (cid:15) 0 , and ω , recall that the difference between the linear and non-linear residual is bounded {+by 2 βα 2 ( (cid:15) 
 + (cid:15) ) (cid:107) r 0 (cid:107) 2 (by Theorem 4).+}
We {+want this error to be bounded by ξ (cid:107) y (cid:107) 2 . This bound is guaranteed by our choice of (cid:15) , by choosing (cid:15) 
 such that it obeys (cid:15) 0 ≤ (cid:15) , and by choosing ω so that+} the initial residual is bounded by (cid:107) r 0 (cid:107) 2 ≤ 2 (cid:107) y (cid:107) 2 . Note that (cid:15) 0 ≤ (cid:15) holds by assumption (10).
We next verify by applying a series of Lemmas proven in Appendix I that the conditions of Theorem 4 are satisﬁed (speciﬁcally, Assumptions 1, 2, 3 on the Jacobians of the non-linear map and the associated linearized map, to be bounded and sufﬁciently close to each other).
Choice of reference Jacobian and veriﬁcation of assumption 2: We chose the reference Jacobian J so that it is (cid:15) 0 close to the random initialization J ( C 0 ) (with high probability), and thus satisﬁes assumption 2.
Towards this goal, we require the following two lemmas.
Lemma 3 (Concentration lemma) .
Consider G ( C ) =
ReLU( UC ) v with v ∈ R k and U ∈ R n × k and associated Jacobian J ( C ) (9) . Let C ∈ R n × k be generated at random with i.i.d.
N (0 , ω 2 ) entries.
Then,
[Equation]
holds with probability at least 1 − δ .
To see that Lemma 3 implies the condition (19), we use the following lemma.
Lemma 4 ((Oymak et al., 2019, Lem. 6.4)) . Let X ∈ R n × N , N ≥ n and let Σ be n × n psd matrix obeying (cid:13)(cid:13) XX T − Σ (cid:13)(cid:13) ≤ (cid:15) 20 / 4 , for a scalar (cid:15) ≥ 0 .
Then there exists a matrix J ∈ R n × N obeying Σ
= JJ T such that [Equation]
In order to verify the condition (19), note that using the fact that (cid:80) k(cid:96) v 4 (cid:96) = 1 k by Lemma 3, with probability at least 1 − δ , we have
[Equation]
Combining inequality (37) with Lemma 4 (applied with X = J ( C 0 ) ), it follows that condition (19) holds for the chosen value of (cid:15) 0 , concluding the proof of Assumption 2 being satisﬁed. This proof speciﬁes the reference Jacobian J as the matrix that is (cid:15) 0 -close to J ( C 0 ) , and that exists by Lemma 4.
Verifying Assumption 1: To verify Assumption 1, note that by deﬁnition JJ T = Σ ( U ) thus trivially σ n ( Σ ( U )) ≥ α holds.
Furthermore, Lemma 5 below combined with the fact that (cid:107) v (cid:107) 2 =implies that (cid:107) J (cid:107) ≤ β
and (cid:107)J ( C ) (cid:107) ≤ β for all C .
This completes the veriﬁcation of Assumption 1.
It remains to show that the Jacobian has bounded spectrum:
Lemma 5 (Spectral norm of Jacobian) .
Consider G ( C ) =
ReLU( UC ) v with v ∈ R k and
U ∈ R n × k and associated Jacobian J ( C ) (9) , and let J be any matrix obeying JJ T =
E (cid:2)
J ( C ) J T ( C ) (cid:3) , where the expectation is over a matrix C with iid N (0 , ω 2 ) entries.
Then (cid:107)J ( C ) (cid:107) ≤ (cid:107) v (cid:107) 2 (cid:107) U (cid:107) and (cid:107) J (cid:107) ≤ (cid:107) v (cid:107) 2 (cid:107) U (cid:107) .
Bound on initial residual: For what follows, we require a bound on the initial residual. To prove this bound, we apply the following lemma.
Lemma 6 (Initial residual) .
Consider G ( C ) =
ReLU( UC ) v , and let C ∈ R n × k be generated at random with i.i.d.
N (0 , ω 2 ) entries.
Suppose half of the entries of v are ν/ √ k and the other half are − ν/ √ k , for some constant ν > 0 . Then, with probability at least 1 − δ ,
[Equation]
G
[Equation]
C )
[Equation]
Now, the initial residual can be upper bounded as [Equation]
) where we used that, by Lemma 6,
[Equation]
Here, the second inequality follows from (cid:107) U (cid:107) F ≤ √ n (cid:107) U (cid:107) and our choice of the parameter ω , and the last inequality follows from ξ ≤ 1 / (cid:112) 32 log(2 n/δ ) and α/β ≤ 1 .
Verifying Assumption 3: Veriﬁcation of the assumption requires us to control the perturbation of the Jacobian matrix around a random initialization.
Lemma 7 (Jacobian perturbation around initialization) .
Let C 0 be a matrix with i.i.d.
N (0 , ω 2 ) entries.
Then, for all C obeying
[Equation]
with
[Equation]
the Jacobian mapping (9) associated with the generator G ( C ) =
ReLU( UC ) v obeys
[Equation]
with probability at least 1 − ne − 12 (cid:101) R
[Equation]
In order to verify Assumption [Equation] ﬁrst note that the radius in the theorem
[Equation]
deﬁned in equation (23), obeys [Equation] Here, (i) follows from the fact that (cid:13)(cid:13)(cid:13) J †S r 0 (cid:13)(cid:13)(cid:13) 2 ≤ 1 α (cid:107) r 0 (cid:107) 2 , (ii) from the bound on the initial residual (38), using that (cid:15) 0 ≤ (cid:15) , as well as our choice of ω . Moreover (iii) follows from the deﬁnition of (cid:15) and (iv) from the lower bound on k in assumption (10).
Note that since ξ 2 16 α 2 β 2 ≤ 12 for this choice of radius ˜ R , by Lemma 7 we have, with (cid:107) v (cid:107) ∞ = 1 / √ k , that
[Equation]
holds with probability at least
[Equation]
where in (i) we used (10) together with ξ ≤ (cid:113) 8 log (cid:0) 2 nδ (cid:1) .
Therefore, Assumption 3 holds with high probability by our choice of (cid:15) =
ξ 8 α 2 β .
Verifying the bound on number of iterations: Finally we note that the constraint on the number of iterations, T , T ≤ 1 2 η(cid:15) 2 , from Theorem 4 is satisﬁed under the number of constraints of Theorem 3 by 1 2 η(cid:15) 2 = 2 5 β 2 ηξ 2 α 4 .
Concluding the proof of Theorem 3: Now that we have veriﬁed the conditions of Theorem 4 we can apply the theorem.
This allows us to conclude that
[Equation]
Here, (i) follows from the triangular inequality [Equation] (ii) from Theorem 4 equation (24) combined with our choice for (cid:15) and (cid:107) r 0 (cid:107) 2 ≤ 12 (cid:107) y (cid:107) 2 , shown above. Moreover, (iii) follows from Theorem 5, (iv) from r 0 =
G ( C 0 ) − y
=
G ( C 0 ) − x − z , (v) from the triangular inequality. Finally, for (vi) we used the fact that x ∈ span { w 1 , w 2 , . . .
, w p } combined with the fact that (cid:13)(cid:13) [Equation] ≤ 1 , as well as using the bound (cid:107) G ( C 0 ) (cid:107) 2 ≤ 12 (cid:107) y (cid:107) 2 proven in (39).
This proves the ﬁnal bound (12), as desired.
G P ROOF OF P ROPOSITION 1 AND EQUATION (2)
The residual of gradient descent at iteration τ is
[Equation]
where we used that c 0 = 0 and the SVD J = W Σ V T .
Expanding y in terms of the singular vectors w i (i.e., the columns of W ), as y = (cid:80) i w i (cid:104)
w i , y (cid:105) , and noting that ( I − η W Σ 2 W T )
τ = (cid:80) i (1 − ησ 2 i ) τ w i w Ti we get
[Equation]
, as desired.
Proof of equation (2) :
By proposition 1 and using that x and lies in the signal subspace
[Equation]
By the triangle inequality,
[Equation]
where the second inequality follows by using orthogonality of the w i and by using (1 − ησ 2 i ) ≤ 1 , from η ≤ 1 /σ 2max .
This concludes the proof of equation (2).
H T HE EXPECTED J ACOBIAN FOR CONVOLUTIONAL GENERATORS
We ﬁrst prove the closed form expression of the expected Jacobian, or more precisely of the matrix Σ ( U ) deﬁned in (13).
By the expression for the Jacobian in equation (9), we have that
[Equation]
where (cid:12) denotes the entrywise product of the two matrices.
Then,
[Equation]
Next [Equation] we have with (Daniely et al., 2016, Sec. 4.2) and using that the derivative of the ReLU function is the step function,
[Equation]
Using that (cid:107) v (cid:107) 2 = 1 , we get
[Equation]
where u i are the rows of U .
This concludes the proof of equation (13).
We next brieﬂy comment on the singular value decomposition of a circulant matrix and explain that the singular vectors are given by Deﬁnition 1.
Recall, that U ∈ R n × n is a circulant matrix, implementing the convolution with a ﬁlter.
Assume for simplicity that n is even.
It is well known that the discrete Fourier transform diagonalizes U , i.e.,
[Equation]
where F ∈ C n × n is the DFT matrix with entries
[Equation]
and ˆ U is a diagonal matrix with diagonal Fu , where u is the ﬁrst column of the circulant matrix
U .
From this, we can compute the singular value decomposition of U by using that ˆ u = Fu is conjugate symmetric (since u is real) so that
[Equation]
Let U
=
W Σ V T be the singular value decomposition of U .
The entries of the left singular vectors are given by the trigonometric basis functions deﬁned in (5), and the singular values are given by the absolute values of ˆ u .
I P ROOFS OF L EMMAS FOR NEURAL NETWORK DENOISERS (P ROOFS OF AUXILIARY LEMMAS IN S ECTION F) I. P ROOF OF L EMMA 7: J ACOBIAN PERTURBATION AROUND INITIALIZATION
The proof follows that of (Oymak & Soltanolkotabi, 2019b, Lem. 6.9)
[Equation] Step 1: We start by relating the perturbation of the Jacobian to a perturbation of the activation patterns.
For any C [Equation] C (cid:48) , we have that
[Equation]
To see this
[Equation]
ﬁrst note that [Equation] by (9),
[Equation]
This in turn implies that
[Equation]
where for (i) we used that for two positive semideﬁnite matrices A , B , λ max ( A (cid:12) B )
≤ λ max ( A ) max i B ii .
This concludes the proof of equation (41).
Step 2: Step one implies that we only need to control σ (cid:48)
( Uc j ) around a neighborhood of c (cid:48) j .
Since σ (cid:48) is the step function, we need to count the number of sign ﬂips between the matrices UC and UC (cid:48) .
Let | v | π ( q ) be the q -th smallest entry of v in absolute value.
Lemma 8.
Suppose that, for all i , and q ≤ k ,
[Equation]
Then
[Equation]
Proof.
Suppose that σ (cid:48) ( u T i C ) and σ (cid:48) ( u T i C (cid:48) ) have 2 q many different entries, then the conclusion of the statement would be violated.
We show that this implies the assumption is violated as well, proving the statement by contraction.
By the contradiction hypothesis,
[Equation]
where the last inequality follows by noting that at least 2 q many entries have different signs, thus their difference is larger than their individual magnitudes, and at least q many individual magnitudes are lower bounded by the q -th smallest one.
Step 3: Next, we note that, with probability at least 1 − ne − kq 2 / 2 , the q -th smallest entry of u Ti C [Equation]
∈ R k obeys
[Equation]
[Equation]
We note that it is sufﬁcient to prove this result for ν = 1 .
This follows from anti-concentration of Gaussian random variables.
Speciﬁcally, with the entries of C (cid:48) being iid N (0 , 1) distributed, the entries of g = u Ti C (cid:48) / (cid:107) u i (cid:107)
∈
R k are iid standard Gaussian random variables as well.
We show that with probability at least 1 − e − kq 2 / 2 , at most q entries are larger than q 2 k .
Let γ δ be the number for which P [ | g (cid:96) | ≤ γ δ ] ≤ δ , where g is a standard Gaussian random variable.
Note that γ δ ≥ (cid:112) π/ 2 δ ≥ δ .
Deﬁne the random variable
[Equation]
otherwise .
with δ = q 2 k .
With E [ δ (cid:96) ]
= δ , by Hoeffding’s inequality,
[Equation]
Thus, with probability at least 1 − ke − kq 2 / 2 no more than m entries are smaller than γ δ ≥ δ = q 2 k .
The results now follows by taking the union bound over all i = 1 , . . . , n .
We are now ready to conclude the proof of the lemma.
By equation (41),
[Equation]
provided that
[Equation]
with probability at least 1 − ne − kq 2 / 2 .
Setting q
=
(2 kR ) 2 / 3 concludes the proof (note that the assumption R ≤ 12 √ k ensures q ≤ k ).
I. P ROOF OF L EMMA 5: B OUNDED J ACOBIAN
By the expression of the Jacobian in equation (9),
[Equation]
where for (i) we used that for two positive semideﬁnite matrices
A
[Equation]
B [Equation] λ max ( A (cid:12) B ) ≤ λ max ( A ) max i B ii .
To prove the second inequality note that
[Equation]
[Table]
Here
[Table]
(i) follows from the fact that for two positive semideﬁnite matrices A [Table] B , λ max ( A (cid:12) B ) ≤ λ max ( A ) max i B ii .
I. P ROOF OF L EMMA 3: C ONCENTRATION LEMMA
We begin by deﬁning the zero-mean random matrices
[Equation]
With this notation,
[Equation]
To show concentration we use the matrix Hoeffding inequality.
To this aim note that the summands are centered in the sense that E (cid:2) S (cid:96) (cid:3) = 0 .
Next note that
[Equation]
[Table]
Similarly,
[Table]
Thus, −
[Table]
Therefore, using A (cid:96) := v 2 (cid:96) B 2 (cid:101) U (cid:101) U T we have
S
[Table]
(cid:22)
A
[Table]
, and σ 2 :
=
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k (cid:88)
[Table]
=
A 2 (cid:96) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ B(cid:32) k
(cid:88) (cid:96) =
v 4 (cid:96) (cid:33) (cid:13)(cid:13)(cid:13) (cid:101) U (cid:13)(cid:13)(cid:13)
To continue we will apply matrix Hoeffding inequality stated below.
Theorem 6 (Matrix Hoeffding inequality, Theorem 1.3 (Tropp, 2011)) .
Consider a ﬁnite sequence
S (cid:96) of independent, random, self-adjoint matrices with dimension n , and let { A (cid:96) } be a sequence of ﬁxed self-adjoint matrices.
Assume that each random matrix satisﬁes
E [ S (cid:96) ]
= 0 and S 2 (cid:96) (cid:22)
A 2 (cid:96) almost surely.
Then, for all t ≥ 0 ,
P (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k (cid:88) (cid:96) =
S (cid:96) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≥ t (cid:35) ≤ 2 ne − t 2 8 σwhere σ 2 :=
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k (cid:88) (cid:96) =
A 2 (cid:96) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Therefore, applying matrix Hoeffding inequality we get that
P (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k (cid:88) (cid:96) =
S (cid:96) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≥ t (cid:35) ≤ 2 ne − t 2 B 4 (cid:107) v (cid:107)
[Table]
which concludes the proof.
I. P ROOF OF L EMMA 6 ( BOUND ON INITIAL RESIDUAL )
Without loss of generality we prove the result for ν = 1 .
First note that by the triangle inequality
[Equation]
We next bound (cid:107) σ ( UC ) v (cid:107) 2 .
Consider the i -th entry of the vector σ ( UC ) v ∈ R n , given by σ ( u Ti C ) v , and note that q j = ( σ ( u Ti c j ) − σ ( u Ti c n − j )) / (cid:107) u
i (cid:107) 2 is sub-Gaussian with parameter 2 , i.e., P [ | q j | ≥ t ] ≤ 2 e − 2 t 2 .
It follows that
[Equation]
Thus,
[Equation]
where we used that | v j | = ξ/ √ k .
Taking a union bound over all n entries,
[Equation]
Choosing β
=
(cid:112) 8 log(2 n/δ ) concludes the proof.