C OMPRESSED S ENSING WITH D EEP I MAGE P RIOR AND L EARNED R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel method for compressed sensing recovery using untrained deep generative models.
Our method is based on the recently proposed Deep Image
Prior (DIP), wherein the convolutional weights of the network are optimized to match the observed measurements.
We show that this approach can be applied to solve any differentiable linear inverse problem, outperforming previous unlearned methods.
Unlike various learned approaches based on generative models, our method does not require pre-training over large datasets.
We further introduce a novel learned regularization technique, which incorporates prior information on the network weights.
This reduces reconstruction error, especially for noisy measurements.
Finally we prove that, using the DIP optimization approach, moderately overparameterized single-layer networks trained can perfectly ﬁt any signal despite the nonconvex nature of the ﬁtting problem.
This theoretical result provides justiﬁcation for early stopping
I NTRODUCTION
We consider the well-studied compressed sensing problem of recovering an unknown signal x ∗ ∈ R n by observing a set of noisy measurements y ∈ R m of the form
[Equation]
Here A ∈ R m × n is a known measurement matrix, typically generated with random independent
Gaussian entries.
Since the number of measurements m is smaller than the dimension n of the unknown vector x ∗ , this is an under-determined system of noisy linear equations and hence ill-posed.
There are many solutions, and some structure must be assumed on x ∗ to have any hope of recovery.
Pioneering research (Donoho, 2006; Cand`es et al., 2006; Candes & Tao, 2005) established that if x ∗ is assumed to be sparse in a known basis, a small number of measurements will be provably sufﬁcient to recover the unknown vector in polynomial time using methods such as Lasso (Tibshirani, 1996).
Sparsity approaches have proven successful, but more complex models with additional structure have been recently proposed such as model-based compressive sensing (Baraniuk et al., 2010) and manifold models (Hegde et al., 2008; Hegde & Baraniuk, 2012; Eftekhari & Wakin, 2015).
Bora et al.
(2017) showed that deep generative models can be used as excellent priors for images.
They also showed that backpropagation can be used to solve the signal recovery problem by performing gradient descent in the generative latent space.
This method enabled image generation with significantly fewer measurements compared to Lasso for a given reconstruction error.
Compressed sensing using deep generative models was further improved in very recent work (Tripathi et al., 2018;
Grover & Ermon, 2018a; Kabkab et al., 2018; Shah & Hegde, 2018; Fletcher & Rangan, 2017; Asim et al., 2018).
Additionally a theoretical analysis of the nonconvex gradient descent algorithm (Bora et al., 2017) was proposed by Hand & Voroninski (2017) under some assumptions on the generative model.
Inspired by these impressive beneﬁts of deep generative models, we chose to investigate the application of such methods for medical imaging, a canonical application of compressive sensing.
A signiﬁcant problem, however, is that all these previous methods require the existence of pre-trained models.
While this has been achieved for various types of images, e.g. human faces of CelebA (Liu et al., 2015) via DCGAN (Radford et al., 2015), it remains signiﬁcantly more challenging for med- ical images (Wolterink et al., 2017; Schlegl et al., 2017; Nie et al., 2017; Schlemper et al., 2017).
Instead of addressing this problem in generative models, we found an easier way to circumvent it.
Surprising recent work by Ulyanov et al.
(2017) proposed Deep Image Prior (DIP), which uses untrained convolutional neural networks.
In DIP-based schemes, a convolutional neural network generator (e.g. DCGAN) is initialized with random weights; these weights are subsequently optimized to make the network produce an output as close to the target image as possible.
This procedure is unlearned, using no prior information from other images.
The prior is enforced only by the ﬁxed convolutional structure of the generator network.
Generators used for DIP are typically over-parameterized, i.e. the number of network weights is much larger compared to the output dimension.
For this reason DIP has empirically been found to overﬁt to noise if run for too many iterations (Ulyanov et al., 2017).
In this paper we theoretically prove that this phenomenon occurs with gradient descent and justify the use of early stopping and other regularization methods.
Our Contributions:
• In Section 3 we propose DIP for compressed sensing (CS-DIP).
Our basic method is as follows.
Initialize a DCGAN generator with random weights; use gradient descent to optimize these weights such that the network produces an output which agrees with the observed measurements as much as possible.
This unlearned method can be improved with a novel learned regularization technique, which regularizes the DCGAN weights throughout the optimization process.
• In Section 4 we theoretically prove that DIP will ﬁt any signal to zero error with gradient descent.
Our result is established for a network with a single hidden layer and sufﬁcient constant fraction over-parametrization.
While it is expected that over-parametrized neural networks can ﬁt any signal, the fact that gradient descent can provably solve this nonconvex problem is interesting and provides theoretical justiﬁcation for early stopping.
• In Section 5 we empirically show that
CS-DIP outperforms previous unlearned methods in many cases.
While pre-trained or “learned” methods frequently perform better (Bora et al., 2017), we have the advantage of not requiring a generative model trained over large datasets.
As such, we can apply our method to various medical imaging datasets for which data acquisition is expensive and generative models are difﬁcult to train.
B ACKGROUND 2. C OMPRESSED S ENSING : C LASSICAL AND U NLEARNED A PPROACHES
A classical assumption made in compressed sensing is that the vector x ∗ is k -sparse in some basis such as wavelet or discrete cosine transform (DCT).
Finding the sparsest solution to an underdetermined linear system of equations is NP-hard in general; however, if the matrix A satisﬁes conditions such as the Restricted Eigenvalue Condition (REC) or Restricted Isometry Property (RIP) (Candes et al., 2006; Bickel et al., 2009; Donoho, 2006; Tibshirani, 1996), then x ∗ can be recovered in polynomial time via convex relaxations (Tropp, 2006) or iterative methods.
There is extensive compressed sensing literature regarding assumptions on A , numerous recovery algorithms, and variations of RIP and REC (Bickel et al., 2009; Negahban et al., 2009; Agarwal et al., 2010; Bach et al., 2012; Loh & Wainwright, 2011).
Compressed sensing methods have found many applications in imaging, for example the single-pixel camera (SPC) (Duarte et al., 2008).
Medical tomographic applications include x-ray radiography, microwave imaging, magnetic resonance imaging (MRI)
(Winters et al., 2010; Chen et al., 2008;
Lustig et al., 2007).
Obtaining measurements for medical imaging can be costly, time-consuming, and in some cases dangerous to the patient (Qaisar et al., 2013).
As such, an important goal is to reduce the number of measurements while maintaining good reconstruction quality.
Aside from the classical use of sparsity, recent work has used other priors to solve linear inverse problems.
Plug-and-play priors (Venkatakrishnan et al., 2013; Chan et al., 2017) and Regularization by Denoising (Romano et al., 2017) have shown how image denoisers can be used to solve general linear inverse problems.
A key example of this is BM3D-AMP, which applies a Block-Matching and 3D ﬁltering (BM3D) denoiser to an Approximate Message Passing (D-AMP) algorithm (Metzler et al., 2016; 2015).
AMP has also been applied to linear models in other contexts (Schniter et al., 2016).
Another related algorithm is TVAL3 (Zhang et al., 2013; Li et al., 2009) which leverages augmented Lagrangian multipliers to achieve impressive performance on compressed sensing problems.
In many different settings, we compare our algorithm to these prior methods: BM3D-AMP, TVAL3, and Lasso.
2. C OMPRESSED S ENSING : L EARNED A PPROACHES
While sparsity in some chosen basis is well-established, recent work has shown better empirical performance when neural networks are used (Bora et al., 2017).
This success is attributed to the fact that neural networks are capable of learning image priors from very large datasets (Goodfellow et al., 2014; Kingma & Welling, 2013).
There is signiﬁcant recent work on solving linear inverse problems using various learned techniques, e.g. recurrent generative models (Mardani et al., 2017b) and auto-regressive models (Dave et al., 2018).
Additionally approximate message passing (AMP) has been extended to a learned setting by Metzler et al.
Bora et al.
(2017) is the closest to our set-up.
In this work the authors assume that the unknown signal is in the range of a pre-trained generative model such as a generative adversarial network (GAN) (Goodfellow et al., 2014) or variational autoencoder (VAE) (Kingma & Welling, 2013).
The recovery of the unknown signal is obtained via gradient descent in the latent space by searching for a signal that satisﬁes the measurements.
This can be directly applied for linear inverse problems and more generally to any differentiable measurement process.
Recent work has built upon these methods using new optimization techniques (Chang et al., 2017), uncertainty autoencoders (Grover & Ermon, 2018b), and other approaches (Dhar et al., 2018;
Kabkab et al., 2018; Mixon & Villar, 2018; Pandit et al., 2019; Rusu et al., 2018; Hand et al., 2018).
The key point is that all this prior work requires pre-trained generative models, in contrast to CS-DIP.
Finally, there is signiﬁcant ongoing work to understand DIP and develop related approaches (Heckel et al., 2018; Dittmer et al., 2018).
P ROPOSED A LGORITHM
Let x ∗ ∈ R n be the signal that we are trying to reconstruct, A ∈ R m × n be the measurement matrix, and η ∈ R
m be independent noise.
Given the measurement matrix A and the observations y
=
Ax ∗ + η , we wish to reconstruct an ˆ x that is close to x ∗ .
A generative model is a deterministic function G ( z ; w ): R k
→ R n which takes as input a seed z ∈ R k and is parameterized by “weights” w ∈ R d , producing an output G ( z ; w ) ∈ R n .
These models have shown excellent performance generating real-life signals such as images (Goodfellow et al., 2014; Kingma & Welling, 2013) and audio (Van Den Oord et al., 2016).
We investigate deep convolutional generative models, a special case in which the model architecture has multiple cascaded layers of convolutional ﬁlters (Krizhevsky et al., 2012).
In this paper we apply a DCGAN (Radford et al., 2015) model and restrict the signals to be images.
3. C OMPRESSED S ENSING WITH D EEP I MAGE P RIOR (CS-DIP)
Our approach is to ﬁnd a set of weights for the convolutional network such that the measurement matrix applied to the network output, i.e. AG ( z ; w ) , matches the measurements y we are given.
Hence we initialize an untrained network G ( z ; w ) with some ﬁxed z and solve the following:
[Equation]
This is, of course, a non-convex problem because G ( z ; w ) is a complex feed-forward neural network.
Still we can use gradient-based optimizers for any generative model and measurement process that is differentiable.
Generator networks such as DCGAN are biased toward smooth, natural images due to their convolutional structure; thus the network structure alone provides a good prior for reconstructing images in problems such as inpainting and denoising (Ulyanov et al., 2017).
Our ﬁnding is that this applies to general linear measurement processes.
Furthermore, our method also directly applies to any differentiable forward operator A .
We restrict our solution to lie in the span of a convolutional neural network.
If a sufﬁcient number of measurements m is given, we obtain an output such that x ∗ ≈ G
( z ; w ∗ ) .
Note that this method uses an untrained generative model and optimizes over the network weights w .
In contrast previous methods, such as that of Bora et al.
(2017), use a trained model and optimize over the latent z -space, solving z ∗ =
arg min z (cid:107) y − AG ( z ; w ) (cid:107) 2 .
We instead initialize a random z with Gaussian i.i.d.
entries and keep this ﬁxed throughout the optimization process.
In our algorithm we leverage the well-established total variation regularization (Rudin et al., 1992;
Wang et al., 2008; Liu et al., 2018), denoted as TV ( G ( z ; w )) .
We also propose an additional learned regularization technique, LR ( w ) ; note that without this technique, i.e. when λ L = 0 , our method is completely unlearned.
Lastly we use early stopping, a phenomena that will be analyzed theoretically in Section 4.
Thus the ﬁnal optimization problem becomes
[Equation]
The regularization term contains hyperparameters λ T and λ L for total variation and learned regularization:
R ( w ; λ T , λ L ) =
λ T TV ( G ( z ; w ))
+ λ L LR ( w ) .
Next we discuss this LR ( w ) term.
3. L EARNED R EGULARIZATION
Without learned regularization CS-DIP relies only on linear measurements taken from one unknown image.
We now introduce a novel method which leverages a small amount of training data to optimize regularization.
In this case training data refers to measurements from additional ground truth of a similar type, e.g. other x-ray images.
To leverage this additional information, we pose Eqn.
3 as a Maximum a Posteriori (MAP) estimation problem and propose a novel prior on the weights of the generative model.
This prior then acts as a regularization term, penalizing the model toward an optimal set of weights w ∗ .
For a set of weights w ∈ R d , we model the likelihood of the measurements
y =
Ax, y ∈ R m , and the prior on the weights w as Gaussian distributions given by p
[Equation]
where µ ∈ R d and Σ ∈ R d × d .
In this setting we want to ﬁnd a set of weights w ∗ that maximizes the posterior on w given y , i.e.,
[Equation]
This gives us the learned regularization term
[Equation]
where the coefﬁcient λ L in Eqn.
4 controls the strength of the prior.
Our motivation for assuming a Gaussian distribution on the weights is to build upon the proven success of (cid:96) 2 regularization, which also makes this assumption.
Notice that when µ = 0 and Σ =
I d × d , this regularization term is equivalent to (cid:96) 2 -regularization.
Thus this method can be thought of as an adaptive version of standard weight decay.
Further, because the network weights are initialized
Gaussian i.i.d.
, we assumed the optimized weights would also be Gaussian.
Previous work has shown evidence that the convolutional weights in a trained network do indeed follow a Gaussian
[Table]
3.2. L EARNING THE P RIOR P ARAMETERS
In the previous section, we introduced the learned regularization term LR ( w ) deﬁned in Eqn. 5.
However we have not yet learned values for parameters ( µ, Σ) that incorporate prior knowledge of the network weights.
We now propose a way to estimate these parameters.
Assume we have a set of measurements S Y =
{ y 1 , y 2 , · · · , y Q } from Q different images S X
= { x 1 , x 2 , · · · , x Q } , each obtained with a different measurement matrix A .
For each measurement y q , q ∈ { 1 , 2 , ..., Q } , we run CS-DIP to solve the optimization problem in Eqn.
3 and obtain an optimal set of weights W ∗ =
{ w ∗ 1 , w ∗ 2 , · · · , w ∗ Q } .
Note that when optimizing for the weights W ∗ , we only have access to the measurements S Y , not the ground truth S X .
The number of weights d in deep networks tends to be very large.
As such, learning a distribution over each weight, i.e. estimating µ ∈ R d and Σ ∈ R d × d , becomes intractable.
We instead use a layer-wise approach: with L network layers, we have µ ∈ R L and Σ ∈
R L × L .
Thus each weight within layer l ∈ { 1 , 2 , ..., L } is modeled according to the same N ( µ l , Σ ll ) distribution.
For simplicity we assume Σ ij = 0 ∀ i (cid:54) =
j , i.e. that network weights are independent across layers.
The process of estimating statistics ( µ, Σ) from W ∗ is described in Algorithm 1 of the appendix.
We use this learned ( µ, Σ) in the regularization term LR ( w ) from Eqn.
5 for reconstructing measurements of images.
We refer to this technique as learned regularization .
While this may seem analogous to batch normalization (Ioffe & Szegedy, 2015), note that we only use ( µ, Σ) to penalize the (cid:96) 2 -norm of the weights and do not normalize the layer outputs themselves.
3.2. D ISCUSSION OF L EARNED R EGULARIZATION
The proposed CS-DIP does not require training if no learned regularization is used, i.e. if λ L =in Eqn. 3.
This means that CS-DIP can be applied only with measurements from a single image and no prior information of similar images in a dataset.
Our next idea, learned regularization, utilizes a small amount of prior information, requiring access to measurements from a small number of similar images (roughly 5 − 10 ).
In contrast, other pretrained models such as that of Bora et al.
(2017) require access to ground truth from a massive number of similar images (tens of thousands for CelebA).
If such a large dataset is available, and if a good generative model can be trained on that dataset, we expect that pre-trained models would outperform our method.
Our approach is instead more suitable for reconstructing problems where large amounts of data or good generative models are not readily available.
T HEORETICAL R ESULTS
In this section we provide theoretical evidence to highlight the importance of early stopping for DIPbased approaches.
Here we focus on denoising a noisy signal y ∈ R n by optimizing over network weights.
This problem takes the form:
[Equation]
This is a special instance of Eqn.
2 with the measurement matrix
A
= I corresponding to denoising.
We focus on generators consisting of a single hidden-layer ReLU network with k inputs, d hidden units, and n outputs.
Using w =
( W, V ) the generator model in this case is given by
[Equation]
(7) where z ∈ R k is the input,
W ∈ R d × k the input-to-hidden weights, and V ∈ R n × d the hiddento-output weights.
We assume V is ﬁxed at random and train over W using gradient descent.
With these formulations in place, we are now ready to state our theoretical result.
Theorem 4.1.
Consider ﬁtting a generator of the form W (cid:55)→ G ( z ; W, V )
= V · ReLU ( Wz ) to a signal y ∈ R n with z ∈ R k , W ∈ R
d × k , V ∈ R n × d , and ReLU ( z ) = max(0 , z ) .
Furthermore, assume V is a random matrix with i.i.d.
N (0 , ν 2 ) entries with ν = 1 √ dn (cid:107) y (cid:107) (cid:107) z (cid:107) .
Starting from an initial weight matrix W 0 selected at random with i.i.d.
N (0 , 1) entries, we run gradient descent updates of the form W τ +
=
W τ − η ∇L ( W τ ) on the loss
[Equation]
with step size η = ¯ η (cid:107) y
[Equation]
2 8 n 4 n + d where ¯ η ≤ 1 .
Assuming that d ≥ Cn, with C a ﬁxed numerical constant, then
[Equation]
holds for all τ with probability at least 1 − 5 e − n/ 2 − e − d/ 2 −
[Equation]
d 23 n 13 .
Our theoretical result shows that after many iterative updates, gradient descent will solve this nonconvex optimization problem and ﬁt any signal y , if the generator network is sufﬁciently wide.
This occurs as soon as the number of hidden units d exceeds the signal size n by a constant factor.
In our theorem we have focused on the case where the measurement matrix is the identity ( A = I ).
We note however that our theorem directly applies to many compressed sensing measurement matrices, in particular any matrix obtained by subsampling the rows of an orthonormal matrix (e.g. subsampling a Fourier matrix). This is possible as for any such matrix, AV has the same distribution as a Gaussian matrix with i.i.d.
entries.
Therefore, our theorem directly applies in this setting by replacing V with AV .
This result demonstrates that early stopping is necessary for DIP-based methods to be successful; otherwise the network can ﬁt any signal, including one that is noisy.
Our proof builds on theoretical ideas from Oymak & Soltanolkotabi (2019) which provide a general framework for establishing global convergence guarantees for overparameterized nonlinear learning problems based on various properties of the Jacobian mapping along the gradient descent trajectory.
Related literature can be found in Du et al.
; Oymak & Soltanolkotabi (2018).
We combine tools from empirical process theory, random matrix theory, and matrix algebra to show that, starting from a random initialization, the Jacobian mapping across all iterates has favorable properties with high probability, hence facilitating convergence to a global optima.
E XPERIMENTS 5. E XPERIMENTAL S ETUP
Measurements: We evaluate our algorithm using two different measurements processes, i.e. matrices A ∈ R m × n .
First we set the entries of A to be Gaussian i.i.d.
such that A i,j ∼ N (0 , 1 m ) .
Recall m is the number of measurements, and n is the number of pixels in the ground truth image.
This measurement process is standard practice in compressed sensing literature; hence we use it on each dataset.
Additionally we use a Fourier measurement process common in MRI applications (Mardani et al., 2018; 2017a; Hammernik et al., 2018;
Lehtinen et al., 2018; Lustig et al., 2008).
Datasets: We use our algorithm to reconstruct both grayscale and RGB images.
For grayscale we use the ﬁrst 100 images in the test set of MNIST (LeCun et al., 1998) and also 60 random images from the Shenzhen Chest X-Ray Dataset (Jaeger et al., 2014), downsampling a 512x512 crop to 256x256 pixels.
For RGB we use retinopathy images from the STARE dataset (Hoover et al., 2000) with 512x512 crops downsized to 128x128 pixels.
[Table]
[Figure]
Baselines: We compare our algorithm to state-of-the-art unlearned methods such as BM3DAMP (Metzler et al., 2016; 2015), TVAL3 (Li, 2011; Li et al., 2009; Zhang et al., 2013), and Lasso in a DCT basis (Ahmed et al., 1974).
We also evaluated the performance of Lasso in a Daubechies wavelet basis (Daubechies, 1988; Wasilewski, 2010) but found this performed worse than LassoDCT on all datasets.
Thus hereon we refer to Lasso - DCT as “Lasso” and do not include results of
Lasso - Wavelet.
Metrics: To quantitatively evaluate the performance of our algorithm, we use per-pixel meansquared error (MSE) between the reconstruction ˆ x and true image x ∗ , i.e. (cid:107) ˆ x − x ∗ (cid:107) 2 n .
Note that because these pixels are over the range [ − 1 , 1] , it’s possible for MSE to be greater than 1 .
Implementation:
To ﬁnd a set of weights w ∗ that minimize Eqn. 3, we use PyTorch (Paszke et al., 2017) with a DCGAN architecture.
For baselines BM3D-AMP and TVAL3, we use the repositories provided by the authors Metzler et al.
(2018) and Li et al., respectively.
For baseline reconstructions
Lasso, we use scikit-learn (Pedregosa et al., 2011).
Section A in the appendix provides further details on our experimental procedures, e.g. choosing hyperparameters.
The supplementary material also contains our code repository for these experiments.
5. E XPERIMENTAL R ESULTS 5.2. R ESULTS : L EARNED R EGULARIZATION
We ﬁrst evaluate the beneﬁts of learned regularization by comparing our algorithm with and without learned regularization, i.e. λ L = 100 and λ L = 0 , respectively.
The latter setting is an unlearned method, as we are not leveraging ( µ, Σ ) from a speciﬁc dataset.
In the former setting we ﬁrst learn ( µ, Σ ) from a particular set of x-ray images; we then evaluate on a different set of x-ray images.
We compare these two settings with varying noise and across different number of measurements.
Our results in Table 1 show that learned regularization does indeed provide beneﬁt.
This beneﬁt tends to increase with more noise or fewer measurements.
Thus we can infer that assuming a learned Gaussian distribution over weights is useful, especially when the original signal is noisy or signiﬁcantly compressed.
5.2. R ESULTS : U NLEARNED CS-DIP
For the remainder of this section, we evaluate our algorithm in the noiseless case without learned regularization, i.e. when η = 0 in Eqn.
1 and λ L = 0 in Eqn.
Hence CS-DIP is completely unlearned; as such, we compare it to other state-of-the-art unlearned algorithms on various datasets and with different measurement matrices.
O r i g i n a l
O u r s
B M 3 D - A
M P
T VA L
L a ss o (a
[Figure]
Reconstructions
[Figure]
ray
O r i g i n a l
O u r s
B M 3 D - A
M P
T VA L
L a ss o (b) Reconstructions - MNIST
MNIST:
In Figure 1b we plot reconstruction error with varying number of measurements m of n = 784.
This demonstrates that our algorithm outperforms baselines in almost all cases.
Figure 2b shows reconstructions for 75 measurements, while remaining reconstructions are in the appendix.
Chest X-Rays: In Figure 1a we plot reconstruction error with varying number of measurements m of n = 65536.
Figure 2a shows reconstructions for 2000 measurements; remaining reconstructions are in the appendix.
On this dataset we outperform all baselines except BM3D-AMP for higher m .
However for lower m , e.g. when the ratio mn ≤ 3% , BM3D-AMP often doesn’t converge.
This ﬁnding seems to support the work of Metzler et al.
(2015): BM3D-AMP performs well on higher m , e.g. mn ≥ 10% , but recovery at lower sampling rates is not demonstrated
Retinopathy: We plot reconstruction error with varying number of measurements m of n =in Figure 3a of the appendix.
On this RGB dataset we quantitatively outperform all baselines except
BM3D-AMP on higher m ; however, even at these higher m , patches of green and purple pixels corrupt the image reconstructions as seen in Figure 10.
Similar to x-ray for lower m , BM3D-AMP often fails to produce anything sensible.
All retinopathy reconstructions are located in the appendix.
Fourier Measurement Process: All previous experiments used a measurement matrix A containing
Gaussian i.i.d.
entries.
We now consider the case where the measurement matrix is a subsampled
Fourier matrix, as discussed in Section A of the appendix.
Our algorithm outperforms baselines on the x-ray dataset, as shown in the appendix Figure 3b.
Additional Experiments, Appendix A:
In Figure 4a we compare our algorithm against the learned approach of Bora et al.
In Figure 4b we compare our algorithm against baselines in the presence of noise, i.e. when η (cid:54) = 0 in Eqn.
We also demonstrate runtimes for all algorithms on the x-ray dataset in Table 2.
C ONCLUSION
We demonstrate how Deep Image Prior (DIP) can be generalized to solve any differentiable linear inverse problem.
We further propose a learned regularization method which, with a small amount of prior information, reduces reconstruction error for noisy or compressed measurements.
Lastly we prove that the DIP optimization technique can ﬁt any signal given a sufﬁciently wide single-layer network.
This provides theoretical justiﬁcation for regularization methods such as early stopping.
A A DDITIONAL E XPERIMENTS , D ETAILS , AND I NSIGHTS
Hyperparameters: Our algorithm CS-DIP is implemented in PyTorch using the RMSProp optimizer (Tieleman & Hinton, 2012) with learning rate 10 − 3 , momentum 0 .
9 , and 1000 update steps for every set of measurements.
These parameters are the same across all datasets.
We initialize one random measurement matrix A for each image in all experiments.
Further we set the TV hyperparameter λ T = 0 .
01 , and found that high frequency components of the image were not reconstructed as sharply with λ T = 0 .
Dataset-speciﬁc design choices: On larger images such as x-ray ( n = 65536 ) and retinopathy ( n = 49152 ), we found no difference using random restarts of the initial seed z .
However for smaller vectors such as MNIST ( n = 784 ), restarts did provide some beneﬁt.
As such our experiments utilize 5 random restarts for MNIST and one initial seed (no restarts) for x-ray and retinopathy images.
For choosing hyperparameter λ L in Eqn.
3, we used a standard grid search and selected the best one.
We used a similar grid search procedure for choosing dataset-speciﬁc hyperparameters in baseline algorithms BM3D-AMP, TVAL3, and Lasso.
Learned Regularization:
For learned regularization, 10 x-ray images are used to learn µ and Σ .
The results are then evaluated, i.e. averaged over x ∗ , on 50 additional x-ray images.
Network Architecture:
Our network has depth 7 and uses convolutional layers with ReLU activations.
The initial seed z in Eqn. 3 is initialized with random Gaussian i.i.d.
entries and then held ﬁxed as we optimize over network weights w .
We found negligible difference when varying the dimension of z (within reason), as this only affects the number of channels in the network’s ﬁrst layer.
Hence we set the dimension of z to be 128 , a standard choice for DCGAN architectures.
Stopping Criterion: We stop after 1000 iterations in all experiments.
We did not optimize this hyperparameter, so admittedly there could be room for improvement.
We further note that the “Error vs. Iterations” curve of CS-DIP with RMSProp did not monotonically decrease for some learning rates, even though error gradually decreased in all cases.
As such we implemented a stopping condition which chooses the reconstruction with least error over the last 20 iterations.
Note we choose this reconstruction based off measurement loss and do not look at the ground truth image.
Fourier Sampling:
The measurements are obtained by sampling Fourier coefﬁcients along radial lines as demonstrated in Figure 13.
For a 2D image x and a set of indices Ω , the measurements we receive are given by y ( i,j ) =
[ F ( x )]
( i,j ) , ( i, j ) ∈ Ω , where F is the 2D Fourier transform.
We choose Ω to be indices along radial lines, as shown in Figure 13 of the appendix; this choice of Ω is common in literature (Cand ` es et al., 2006) and MRI applications (Mardani et al., 2017a; Lustig et al., 2008; Eksioglu & Tanc, 2018).
While Fourier subsampling is common in MRI applications, we use it here on images of x-rays simply to demonstrate that our algorithm performs well with different measurement processes.
We compare our algorithm to baselines on the x-ray dataset for { 3 , 5 , 10 , 20 } radial lines in the
Fourier domain, which corresponds to { 381 , 634 , 1260 , 2500 } Fourier coefﬁcients, respectively.
We plot reconstruction error with varying number of Fourier coefﬁcients in Figure 3b, outperforming baselines.
Reconstructions be found in Figure 14.
[Table] [Table]
BM3D-AMP 51.
TVAL 13.
Lasso DCT 27.
[Figure]
[Figure] [Figure]
[Figure]
: Set of optimal weights W ∗ = { w ∗ 1 , w ∗ 2 , w ∗ obtained
L -
DCGAN over
Q images; number of samples S ; number of iterations T .
Output: Mean vector µ ∈ R L ; covariance matrix Σ ∈ R L × L .
1: for t = 1 to T do 2:
Sample q uniformly from { 1 , ..., Q } 3: for l = 1 to L { for each layer } do 4:
Get v ∈ R S , a vector of S uniformly sampled weights from the l th layer of w ∗ q 5:
M t [ l, :] ← v T where M t [ l, :] is the l th row of matrix M t ∈ R L × S 6: µ t [ l ] ← 1 S (cid:80)
Si
=1 v
i 7: end for 8: Σ t ← 1 S M t M Tt − µ t µ Tt 9: end for 10: µ ← 1 T (cid:80)
Tt =
1 µ t 11: Σ ← 1 T (cid:80)
Tt =
1 Σ t
B P ROOF OF S ECTION 4: T HEORETICAL J USTIFICATION FOR E ARLY S TOPPING
In this section we prove our theoretical result in Theorem 4.1.
We begin with a summary of some notations we use throughout in Section B.1.
Next, we state some preliminary calculations in Section
B.2.
Then, we state a few key lemmas in Section B.3 with the proofs deferred to Appendix C.
Finally, we complete the proof of Theorem 4.1 in Section B.4.
B. N OTATION
In this section we gather some notation used throughout the proofs.
We use φ ( z ) =
ReLU ( z ) = max(0 , z ) with φ (cid:48)
( z ) =
I { z ≥ 0 } .
For two matrices/vectors x and y of the same size we use x (cid:12) y to denote the entrywise Hadamard product of these two matrices/vectors.
We also use x ⊗ y to denote their Kronecker product.
For two matrices B ∈ R n × d 1 and C ∈ R n × d 2 , we use the Khatrio-Rao product as the matrix
A
= B ∗ C ∈ R n × d 1 d 2 with rows A
i given by A i =
B i ⊗ C i .
For a matrix
M ∈ R m × n we use vect ( M ) ∈
R mn to denote a vector obtained by aggregating the rows of the matrix M into a vector, i.e. vect ( M ) =
[ M
M. . .
M m ] T .
For a matrix X we use σ min ( X ) and (cid:107) X (cid:107) denotes the minimum singular value and spectral norm of X .
Similarly, for a symetric matrix M we use λ min ( M ) to denote its smallest eigenvalue.
B. P RELIMINARIES
In this section we carryout some simple calculations yielding simple formulas for the gradient and
Jacobian mappings.
We begin by noting we can rewrite the gradient descent iterations in the form
[Equation]
Here, vect
[Equation]
where
[Equation]
) f
[Equation]
) and is the Jacobian mapping associated to the network and
[Equation]
is the misﬁt or residual vector.
Note that
[Equation]
Thus
[Equation]
This in turn yields
[Equation]
B. L EMMAS FOR CONTROLLING THE SPECTRUM OF THE J ACOBIAN AND INITIAL MISFIT
In this section we state a few lemmas concerning the spectral properties of the Jacobian mapping, its perturbation and initial misﬁt of the model with the proofs deferred to Appendix C
[Equation]
Lemma B.1 (Minimum singular value of the Jacobian at initialization) .
Let V ∈ R n × d and W ∈ R
d
× k be random matrices with i.i.d.
N (0 , ν 2 ) and N (0 , 1) entries and deﬁne the Jacobian mapping
J ( W ) =
( V diag ( φ (cid:48)
( Wz )))
∗ (cid:0) 1 z T (cid:1) .
Then as long as d ≥ 3828 n ,
[Equation]
holds with probability at least 1 − 2 e − n .
Lemma B.2 (Perturbation lemma) .
Let V ∈ R n × d be a matrix with i.i.d.
N (0 , ν 2 ) entries, W ∈
R d × k , and deﬁne the Jacobian mapping J ( W )
=
( V diag ( φ (cid:48)
( Wz )))
∗ (cid:0) 1 z T (cid:1) .
Also let W 0 be a matrix with i.i.d.
N (0 , 1) entries.
Then,
[Equation]
holds for all W ∈ R d × k obeying (cid:107) W − W 0 (cid:107) ≤ R with probability at least 1 − e − n/ 2 − e − (2 dR ) .
Lemma B.3 (Spectral norm of the Jacobian) .
Let V ∈ R n × d be a matrix with i.i.d.
N (0 , ν 2 ) entries,
W ∈ R d × k , and deﬁne the Jacobian mapping J ( W ) =
( V diag ( φ (cid:48)
( Wz )))
∗ (cid:0) 1 z T (cid:1) .
Then, (cid:107)J ( W ) (cid:107) ≤ ν (cid:16) √ d + 2 √ n (cid:17) (cid:107) z (cid:107) , holds for all W ∈ R d × k with probability at least 1 − e − n/ 2 .
Lemma B.4 (Initial misﬁt) .
Let V ∈ R n × d be a matrix with i.i.d.
N (0 , ν 2 ) entries with ν = 1 √ dn (cid:107) y (cid:107) (cid:107) z (cid:107) .
Also let
W ∈ R
d
× k be a matrix with i.i.d.
N (0 , 1) entries.
Then (cid:107) V φ
( Wz ) − y (cid:107) ≤ 3 (cid:107)
y
(cid:107) , holds with probability at least 1 − e − n/ 2 − e − d/ 2 .
B. P ROOF OF T HEOREM 4.
Consider a nonlinear least-squares optimization problem of the form
[Equation]
with f : R p (cid:55)→ R n and y ∈ R n .
Suppose the Jacobian mapping associated with f obeys the following three assumptions.
Assumption 1.
Fix a point θ 0 .
We have that σ min ( J ( θ 0 )) ≥ 2 α .
Assumption 2.
Let (cid:107) · (cid:107) denote a norm that is dominated by the Euclidean norm i.e. (cid:107) θ (cid:107) ≤ (cid:107) θ (cid:107) holds for all θ ∈ R p .
Fix a point θ 0 and a number R > 0 .
For any θ satisfying (cid:107) θ − θ 0 (cid:107) ≤ R , we have that (cid:107)J ( θ 0 )
− J ( θ ) (cid:107) ≤ α/ 3 .
Assumption 3.
For all θ ∈ R p , we have that (cid:107)J ( θ ) (cid:107) ≤ β .
Under these assumptions we can state the following theorem from Oymak & Soltanolkotabi (2019).
Theorem B.5 (Non-smooth Overparameterized Optimization) .
Given θ 0 ∈ R p , suppose Assumptions 1, 2, and 3 hold with
[Equation]
Then, picking constant learning rate η ≤ 1 β 2 , all gradient iterations obey the followings
[Equation]
We shall apply this theorem to the case where the parameter is W and the nonlinear mapping is given by V φ ( Wz ) and φ = ReLU .
All that is needed to be able to apply this theorem is check that the assumptions hold.
Per the assumptions of the theorem we use
[Equation]
To this aim note that using Lemma B.1 Assumption 1 holds with
[Equation]
with probability at least 1 − 2 e − n .
Furthermore, by Lemma B.3 Assumption 3 holds with
[Equation]
with probability at least 1 − e − n/ 2 .
All that remains for applying the theorem above is to verify
Assumption 2 holds with high probability
[Equation]
In the above we have used Lemma B.4 to conclude that (cid:107) V φ ( Wz ) − y (cid:107) ≤ 3 (cid:107) y (cid:107) holds with probability at least 1 − e − n/ 2 − e − d/ 2 .
Thus, using Lemma B.2 all that remains is to show that
[Equation]
=
[Equation]
, holds with R = 60 √ n and with probability at least 1 − e − n/ 2 − e − (120)
[Equation]
n
[Equation]
e − n/
[Equation]
− e − 4 d 23 n 13 .
The latter is equivalent to
[Equation]
which can be rewritten in the form
[Equation]
which holds as long as d ≥ 4 .
3 × 10 15 n .
Thus with d ≥ Cn then Assumptions 1, 2, and 3 holds with probability at least 1 − 5 e − n/ 2 − e − d/ 2 − e − 4 d 23 n 13 .
Thus, Theorem B.5 holds with high probability.
Applying Theorem B.5 completes the proof.
C P ROOF OF L EMMAS FOR THE S PECTRAL P ROPERTIES OF THE J ACOBIAN C. P ROOF OF L EMMA B.
We prove the result for ν = 1 , the general result follows from a simple re-scaling.
Deﬁne the vectors a (cid:96)
[Equation]
V
[Equation] [Equation]
w
[Equation]
, z (cid:105) ) ∈ R n , with V (cid:96) the (cid:96) th column of V .
Using equation 8 we have
[Equation] [Equation]
To bound the minimum eigenvalue we state a result from Oliveira (2013).
Theorem C.1.
Assume A 1 , . . .
, A d ∈ R n × n are i.i.d.
random positive semideﬁnite matrices whose coordinates have bounded second moments.
Deﬁne Σ :=
E [ A 1 ] (this is an entry-wise expectation) and
[Equation]
Let h ∈ (1 , + ∞ ) be such that (cid:113) E (cid:2) ( u T A 1 u ) 2 (cid:3) ≤ hu T Σ u for all u ∈ R n .
Then for any δ ∈ (0 , 1) we have
[Equation]
We shall apply this theorem with A (cid:96) :
= a (cid:96) a T(cid:96) .
To do this we need to calculate the various parameters in the theorem.
We begin with Σ and note that for ReLU we have
[Equation]
To calculate h we have
[Equation]
Thus we can take h = √ 6 .
Therefore, using Theorem C.1 with δ = 2 e − n we can conclude that
[Equation]
holds with probability at least 1 − 2 e − n as long as d ≥ 3528 · n.
Plugging this into equation 11 we conclude that with probability at least 1 − 2 e − n
[Equation]
C. P ROOF OF L EMMA B.
We prove the result for ν = 1 , the general result follows from a simple rescaling.
Based on equation 8 we have
[Equation]
Thus
[Equation]
where S ( W ) ⊂ { 1 , 2 , . . .
, d } is the set of indices where Wz and W
z have different signs i.e. S ( W ) :
= { (cid:96) : sgn ( e T(cid:96)
Wz ) (cid:54) = sgn
( e T(cid:96)
W 0 z ) } and V S ( W ) is a submatrix V obtained by picking the columns corresponding to S ( W ) .
To continue further note that by Gordon’s lemma we have
[Equation]
with probability at least 1 − e − t 2 / 2 .
In particular using t = √ n we conclude that sup
[Equation]
with probability at least 1 − e − n/ 2 .
To continue further we state a lemma controlling the size of |S ( W ) | based on the size of the radius R .
Lemma C.2 (sign changes in local neighborhood) .
Let W 0 ∈ R
d
× k be a matrix with i.i.d.
N (0 , 1) entries.
Also for a matrix W ∈ R d × k deﬁne S ( W ) :
= { (cid:96) : sgn ( e T(cid:96)
Wz ) (cid:54) = sgn
( e T(cid:96)
W 0 z ) } .
Then for any W ∈ R d × k obeying (cid:107) W − W 0 (cid:107) ≤ R
[Equation]
holds with probability at least 1 − e − (2 dR ) .
Combining equation 12 together with equation 14 (using s = 3 (2 dR ) 23 ) and Lemma C.2 we conclude that
[Equation]
 holds with probability at least 1 − e − n/ 2 − e − (2 dR
[Equation]
C. P ROOF OF L EMMA C.
To prove this result we utilize two lemmas from Oymak & Soltanolkotabi (2019).
In these lemmas we use | v | m − to denote the m th smallest entry of v after sorting its entries in terms of absolute value.
Lemma C.3.
(Oymak & Soltanolkotabi, 2019, Lemma C.2)
Given an integer m , suppose
[Equation]
, then
[Equation]
Lemma C.4.
(Oymak & Soltanolkotabi, 2019, Lemma C.3) Let z ∈ R k .
Also let W
∈
R
d × k be a matrix with i.i.d.
N (0 , 1) entries.
Then, with probability at least 1 − e − m 6 ,
[Equation]
−
[Equation]
Combining the latter two lemmas with m = (cid:100) (2 dR ) 23 (cid:101) we conclude that when
[Equation]
then with probability at least 1 − e − (2 dR ) we have
[Equation]
C. P ROOF OF L EMMA B.
We prove the result for ν = 1 , the general result follows from a simple rescaling.
Using equationwe have
[Equation]
Thus
[Equation]
The proof is complete by using standard concentration results for the spectral norm of a Gaussian matrix that allow us to conclude that
[Equation]
holds with probability at least 1 − e − n/ 2 .
C. P ROOF OF L EMMA B.
By the triangular inequality we have
[Equation]
[Equation]
To continue further let us consider one entry of V φ ( Wz ) and note that it has the same distribution as
[Equation]
where g ∈ R d is random Gaussian vectors with distribution g ∼ N (0 , I d ) .
Thus
[Equation] [Equation]
with probability at least 1 − e − n/ 2 .
Furthermore, note that
[Equation]
where (cid:101)
g ∈ R
d is random Gaussian vectors with distribution (cid:101) g ∼ N (0 , I d ) .
Combining the latter with equation 16 we conclude that
[Equation]
holds with probability at least 1 − e − n/ 2 − e − d/ 2 .
Combining the latter with equation 15 we conclude that
[Equation]
holds with probability at least 1 − e − n/ 2 − e − d/ 2 .
D R ECONSTRUCTIONS
[Figure]
u
[Figure] [Figure]
[Figure]
i n a l
O u r s
[Figure]
M P
[Figure]
L
[Figure]
O r i g i n a l
O u r s
B M 3 D - A
M P
T VA L
L a
[Figure]
O u r
[Figure]
O r i g i n a l
O u r s
[Figure]
M 3 D - A
M P
L
[Figure]
o
[Figure]
[Figure] [Figure]
[Figure] [Figure]