Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
Many problems in image processing and computer vision rely, explicitly or implicitly, on prior probability models.
Describing the full density of natural images is a daunting problem, given the high dimensionality of the signal space.
Traditionally, models have been developed by combining assumed symmetry properties (e.g., translation-invariance, dilation-invariance), with simple parametric forms (e.g., Gaussian, exponential), often within pre-speciﬁed transformed coordinate systems (e.g., Fourier transform, wavelets).
While these models have led to steady advances in problems such as denoising (e.g., [1–7]), they are too simplistic to generate complex features that occur in our visual world, or to solve more demanding statistical inference problems.
Nearly all problems in image processing and computer vision have been revolutionalized by the use of deep Convolutional Neural Networks (CNNs).
These networks are generally trained to perform tasks in supervised fashion, and their embedding of prior information, arising from a combination of the distribution of the training data, the architecture of the network [8] and regularization terms included during optimization, is intertwined with the task for which they are optimized.
Generative Adversarial Networks (GANs) [9] have been shown capable of synthesizing novel high-quality images, that may be viewed as samples of an implicit prior density model and recent methods have been developed to use these priors in solving inverse problems [10, 11].
Although GANs can produce visually impressive synthetic samples, a number of results suggest that these samples are not representative of the full image density (a problem sometimes referred to as “mode collapse”)
Variational autoencoder (VAE) networks [13] have been used in conjunction with Score Matching methodologies to draw samples from an implicit prior [14–18].
Related work has developed algorithms for using MAP denoisers to regularize inverse problems, obtaining high quality results on deblurring and superresolution using conventional denoisers [19].
Here, we develop a simple but general algorithm for solving linear inverse problems using the implicit image prior from a CNN trained (supervised) for denoising.
We start with a little-known result from classical statistics [20] that states that a denoiser that aims to minimize squared error of images corrupted by additive Gaussian noise may be interpreted as computing the gradient of the log of the density of noisy images.
We develop a stochastic coarse-to-ﬁne gradient ascent procedure based on this denoiser-estimated gradient, and use it to draw high-probability samples from the implicit prior embedded within a CNN trained to perform blind (i.e., unknown noise level) least-squares denoising.
The gradient step sizes and the amplitude of injected noise are jointly and adaptively controlled by the denoiser, allowing rapid and reliable convergence.
More generally, we combine this procedure with constraints arising from any linear measurement of an image to draw samples from the prior conditioned on this measurement, thus providing a stochastic solution to the inverse problem.
We demonstrate that our method, using the prior implicit in a state-of-the-art CNN denoiser, produces high-quality results on image synthesis, inpainting, super-resolution, deblurring and recovery of missing pixels.
We also apply our method to recovering images from projections onto a random low-dimensional basis, demonstrating results that greatly improve on those obtained using sparse union-of-subspace priors typically assumed in the compressive sensing literature.
Image priors, manifolds, and noisy observations
Digital photographic images lie in a high-dimensional space ( R N , where N is the number of pixels), and simple thought experiments suggest that they are concentrated on or near low-dimensional manifolds.
For a given photograph, applying any of a variety of local continuous deformations (e.g., translations, rotations, dilations, intensity changes) yields a low-dimensional family of natural-appearing images.
These deformations follow complex curved trajectories in the space of pixels, and thus lie on a manifold.
In contrast, images generated with random pixels are almost always feature and content free, and thus not considered to be part of this manifold.
We can associate with this a prior probability model, p ( x ) , by assuming that images within the manifold have constant or slowly-varying probability, while unnatural or distorted images (which lie off the manifold) have low or zero probability.
Suppose we make a noisy observation of an image, y = x
+ z , where x ∈ R N is the original image drawn from p ( x ) , and z ∼ N (0 , σ 2 I N ) is a sample of Gaussian white noise.
The observation density p ( y ) (also known as the prior predictive density ) is related to the prior p ( x ) via marginalization:
[Equation]
where the noise distribution is
[Equation]
Equation (1) is in the form of a convolution, and thus p ( y ) is a Gaussian-blurred version of the signal prior, p ( x ) .
Moreover, the family of observation densities over different noise variances, p σ ( y ) , forms a Gaussian scale-space representation of the prior [21, 22], analogous to the temporal evolution of a diffusion process.
Least squares denoising and CNNs
Given a noisy observation, y , the least squares estimate (also called "minimum mean squared error", MMSE) of the true signal is well known to be the conditional mean of the posterior:
[Equation]
Traditionally, one obtains such estimators by choosing a prior probability model, p ( x ) (often with parameters ﬁt to sets of images), combining it with a likelihood function describing the noise, p ( y | x ) , and solving.
For example, the Wiener ﬁlter is derived by assuming a Gaussian prior in which variance falls inversely with spatial frequency [23].
Modern denoising solutions, on the other hand, are often based on discriminative training.
One expresses the estimation function (as opposed to the prior) in parametric form, and sets the parameters by minimizing the denoising MSE over a large training set of example signals and their noise-corrupted counterparts [24–27].
The size of the training set is virtually unlimited, since it can be constructed automatically from a set of photographic images, and does not rely on human labelling.
Current state-of-the-art denoising results using CNNs are far superior, both numerically and visually, to results of previous methods [28–30].
Recent work [31] demonstrates that these architectures can be simpliﬁed by removing all additive bias terms, with no loss of performance.
The resulting bias-free networks offer two important advantages.
First, they automatically generalize to all noise levels: a network trained on images with barely noticeable levels of noise can produce high quality results when applied to images corrupted by noise of any amplitude.
Second, they may be analyzed as adaptive linear systems, which reveals that they perform an approximate projection onto a low-dimensional subspace.
In our context, we interpret this subspace as a tangent hyperplane of the image manifold at a speciﬁc location.
Moreover, the dimensionality of these subspaces falls inversely with σ , and for a given noise sample, the subspaces associated with different noise amplitude are nested, with high-noise subspaces lying within their lower-noise counterparts.
In the limit as the noise variance goes to zero, the subspace dimensionality grows to match that of the manifold at that particular point.
Exposing the implicit prior through Empirical Bayes estimation
The trained CNN denoisers mentioned above embed detailed prior knowledge of image structure.
Given such a denoiser, how can we obtain access to this implicit prior?
Recent results have derived relationships between Score matching density estimates and denoising [14, 32, 17, 33], and have used these relationships to make use of implicit prior information.
Here, we exploit a much more direct but little-known result from the literature on Empirical Bayesian estimation.
The idea was introduced in [34], extended to the case of Gaussian additive noise in [20], and generalized to many other measurement models in [35].
For the Gaussian noise case, the least-squares estimate of Eq.
(2) may be rewritten as:
[Equation]
The proof of this result is relatively straightforward.
First note that the gradient of the observation density expressed in
Eq.
(1) is:
[Equation]
Multiplying both sides by σ 2 /p ( y ) and separating the right side into two terms gives:
[Equation]
Rearranging terms and using the chain rule to compute the gradient of the log gives Miyasawa’s result, as expressed in
Eq.
Intuitively, the Empirical Bayesian form in Eq.
(3) suggests that denoisers use a form of gradient ascent, removing noise from an observed signal by moving up a probability gradient.
But note that: 1) the relevant density is not the prior, p ( x ) , but the noisy observation density , p ( y ) ; 2) the gradient is computed on the log density (the associated “energy function”); and 3) the adjustment is not iterative - the optimal solution is achieved in a single step, and holds for any noise level, σ .
Drawing high-probability samples from the implicit prior
We wish to draw a sample from the prior implicit in a denoiser.
Equation (4) allows us to generate an image proportional to the gradient of log p ( y ) by computing the denoiser residual, f ( y )
=
ˆ x ( y ) − y .
Previous work [15, 17] developed a related computation in a Markov chain Monte Carlo (MCMC) scheme, combining gradient steps derived from
Score-matching and injected noise in a Langevin sampling algorithm to draw samples from a sequence of densities p σ ( y ) , while reducing σ in discrete steps, each associated with an appropriately trained denoiser.
In contrast, starting from a random initialization, y 0 , we aim to ﬁnd a high-probability image (i.e., an image from the manifold) using a simpler and more efﬁcient stochastic gradient ascent procedure.
We compute gradients using the residual of a bias-free universal CNN denoiser, which automatically adapts to each noise level.
On each iteration, we take a small step in the direction speciﬁed by the denoiser, which moves closer to the image manifold, thereby reducing the amplitude of the effective noise.
The reduction of noise is achieved by decreasing the amplitude in the directions orthogonal to the observable manifold while retaining the amplitude of image in the directions parallel to manifold which results in the emergence of image content.
As the effective noise decreases, the observable dimensionality of the image manifold increases [31], allowing the synthesis of detailed structures.
Since the family of observation densities, p σ ( y ) forms a scale-space representation of p ( x ) , the algorithm may be viewed as an adaptive form of coarse-to-ﬁne optimization [36–39].
Assuming the step sizes are adequately-controlled, the procedure converges to a point on the manifold.
Figure 2 illustrates this process in two dimensions.
Each iteration operates by taking a deterministic step in the direction of the gradient (as obtained from the denoising function) and injecting some additional noise:
[Equation]
where f ( y )
=
ˆ x ( y ) − y is the residual of the denoising function, which is proportional to the gradient of p ( y ) , from
Eq.
The parameter h t ∈ [0 , 1] controls the fraction of the denoising correction that is taken, and z t ∼ N (0 , I ) is a sample of white Gaussian noise, scaled by parameter γ t , whose purpose is to avoid getting stuck in local maxima.
The effective noise variance of image y t is:
[Equation]
where the ﬁrst term assumes that the denoiser successfully reduces the variance of the noise in y t − 1 by a factor of (1 − h t ) , and the second term is the variance arising from the injected noise.
To ensure convergence, we aim to reduce the effective noise variance on each time step, which we express in terms of a parameter β ∈ [0 , 1] as:
[Equation]
Combining this with Eq.
(6) allows us to solve for γ t :
[Equation]
where the second line assumes that the magnitude of the denoising residual provides a good estimate of the effective noise standard deviation, as was found in [31].
This allows the denoiser to adaptively control the gradient ascent step sizes, reducing them as the result approaches the manifold (see Figure 2 for a visualization in 2D).
This automatic adjustment of the gradient step size is an important feature of the procedure, which results in a fast an smooth convergence as illustrated in Figure 4.
We found that initial implementations with a small constant fractional step size h t produced high quality results, but required many iterations - a form of Zeno’s paradox.
To improve convergence speed, we introduced a schedule for increasing the step size according to h t = h 0 t 1+ h 0 ( t − 1) , starting from h 0 ∈ [0 , 1] .
The sampling algorithm is summarized in Algorithm 1 in section B, and is laid out in a block diagram in Figure 1 in the appendix.
Example convergence behavior is shown in Figure 3.
Sampling examples are shown in section F.
Solving linear inverse problems using the implicit prior
Many applications in signal processing can be expressed as linear inverse problems - deblurring, super-resolution, estimating missing pixels (e.g., inpainting), and compressive sensing are all examples.
Given a set of linear measurements of an image, x c = M T x , where M is a low-rank measurement matrix, one attempts to recover the original image.
In
Section 1, we developed a stochastic gradient-ascent algorithm for obtaining a high-probability sample from p ( x ) .
Here, we modify this algorithm to solve for a high-probability sample from the conditional density p ( x | M T x = x c ) .
Consider the distribution of a noisy image, y , conditioned on the linear determistic measurements, x c = M T x ,
[Equation]
where y c = M T y .
Without loss of generality 1 , we assume the measurement matrix has singular values that are equal to one (i.e., columns of M are orthogonal unit vectors, and thus M T M = I ).
It follows that M is the pseudo-inverse of
M T , and that matrix MM T can be used to project an image onto the measurement subspace.
As with the algorithm of
Section 1, we wish to obtain a local maximum of this function using stochastic coarse-to-ﬁne gradient ascent.
Applying the operator σ 2 ∇ log yields
[Equation]
The second term is the gradient of the observation noise distribution, projected into the measurement space.
If this is
Gaussian with variance σ 2 , it reduces to M ( y c − x c ) .
The ﬁrst term is the gradient of a function deﬁned only within the subspace orthogonal to the measurements, and thus can be computed by projecting the measurement subspace out of the full gradient.
Combining these gives:
[Equation]
[Equation]
Thus, we see that the gradient of the conditional density is partitioned into two orthogonal components, capturing the gradient of the (log) noisy density, and the deviation from the constraints, respectively.
To draw a high-probability sample from p ( x | x c ) , we use the same algorithm described in Section 1, substituting Eq.
(8) for the deterministic update vector, f ( y ) (see Algorithm 2, and Figure 1).
Examples of our algorithm solution to ﬁve linear inverse problems are shown in Section G.
Discussion and related work
We’ve described a general method of transferring the prior embedded in a denoiser to solve inverse problems.
Speciﬁcally, we developed a stochastic coarse-to-ﬁne gradient ascent algorithm that uses the denoiser to draw high-probability samples from its implicit prior, and a constrained variant that can be used to solve any linear inverse problem.
The derivation relies on the denoiser being optimized for mean squared error in removing additive Gaussian noise of unknown amplitude.
Denoisers can be trained using discriminative learning (nonlinear regression) on virtually unlimited amounts of unlabeled data, and thus, our method extends the power of supervised learning to a much broader set of problems, with no additional training.
Our method is similar to recent work that uses Score Matching to draw samples from an implicit prior [14–18], but differs in several important ways: (1) our derivation is direct and signiﬁcantly simpler, exploiting a little-known result from the classical statistical literature on Empirical Bayes estimation [20]; (2) our method assumes a single (universal) blind denoiser, rather than a family of denoisers trained for different noise levels; (3) our algorithm is efﬁcient - we use stochastic gradient ascent to maximize probability, rather than MCMC methods (such as Langevin dynamics) to draw samples from each of a sequence of densities, and our step sizes are automatically adapted by the denoiser to control convergence; and (4) we demonstrate the generality of our algorithm by applying it to ﬁve different linear inverse problems.
Another related line of research uses a denoiser to regularize an objective function for solving linear inverse problems [19, 40].
Our method differs in that (1) the term arising from our denoiser represents the exact gradient of the (noisy) prior (it is not derived as an approximation); (2) our method requires only that the denoiser must be trained on Gaussian-noise contaminated images to minimize squared error and must operate "blind" (without knowledge of noise level), whereas RED relies on a set of three additional assumed properties; (3) RED has been used to solve MAP estimation problems, whereas we have only applied our method to deterministic linear inverse problems (although we believe it can be generalized to MAP); and (4) our algorithm has two primary hyper-parameters ( h 0 and β ) to control step sizes, and is robust to choices of these (see Appendix), whereas RED includes multiple hyper-parameters, including the gradient step size whose adjustment is important for achieving convergence (the issue is resolved through use of an
ADMM method).
The performance of our method on linear inverse problems should ultimately be quantiﬁed by experiments with human observers, but we also plan to measure it using a no-reference perceptual quality metric (e.g., [41]).
Handling of nonlinear inverse problems with convex measurements (e g. recovery from quantized representation, such as JPEG) is a natural extension of the method, in which the algorithm must be modiﬁed to incorporate projection onto convex sets.
Finally, our method for image generation offers a means of visualizing and interpreting implicit prior of a denoiser, which arises from the combination of architecture, optimization, regularization and training set.
As such, it offers a means of experimentally isolating and elucidating the effects of these components.
A Description of BF-CNN denoiser
Architecture.
Throughout the paper, we use BF-CNN, described in [31], constructed from 20 bias-free convolutional layers, each consisting of 3 × 3 ﬁlters and 64 channels, bias-free batch normalization, and a ReLU nonlinearity.
Note that to construct a bias-free network, we remove all sources of additive bias, including the mean parameter of the batch-normalization in every layer.
Training Scheme.
We follow the training procedure described in [31].
The network is trained to denoise images corrupted by i.i.d.
Gaussian noise with standard deviations drawn from the range [0 , 0 .
4] (relative to image intensity range [0 , 1] ).
The training set consists of overlapping patches of size 40 × 40 cropped from the Berkeley Segmentation
Dataset [42].
Each original natural image is of size 180 × 180 .
Training is carried out on batches of size 128, forepochs.
B Algorithms
f ( y ) = ˆ x ( y ) − y . parameters: σ 0 , σ L , h 0 , β initialization: t = 1 , draw y 0 ∼ N (0 . 5 , σ 20 I ) while σ t − 1 ≤ σ L do h t
= h 0 t 1+ h 0 ( t − 1) ; d t = f
( y t − 1 ) ; σ 2 t = ||
d t || 2 N ; γ 2 t = (cid:0) (1 − βh t ) 2 − (1 − h t ) 2 (cid:1) σ 2 t ;
Draw z t ∼ N (0 , I ) ; y t ← y t − 1 + h t d t + γ t z t ; t ← t + 1 end parameters: σ 0 , σ L , h 0 , β , M , x c initialization: t=1; draw y 0 ∼ N (0 .
( I − MM T )
e + Mx c , σ 20 I ) while σ t − 1 ≤ σ L do h t = h 0 t 1+ h 0 ( t − 1) ; d t = ( I − MM T ) f
( y t − 1 )
+ ( Mx c − MM T y t − 1 ) ;
σ 2 t = ||
d t || 2 N ; γ 2 t = (cid:0) (1 − βh t ) 2 − (1 − h t ) 2 (cid:1) σ 2 t ;
Draw z t ∼ N (0 , I ) ; y t ← y t − 1 + h t d t + γ t z t ; t ← t + 1 end
[Equation]
[Figure]
E Convergence
Figure 3 illustrates the convergence of our iterative sampling algorithm, expressed in terms of the effective noise standard deviation σ = || d t || √ N averaged over synthesis of three images, for three different levels of the stochasticity parameter β .
Convergence is well-behaved and efﬁcient in all cases.
As expected, with smaller β (larger amounts of injected noise), effective standard deviation falls more slowly and convergence takes longer.
[Figure] [Figure]
When β = the effective noise the denoiser almost with the same as expected by the schedule.
As β decreases, i.e. for non-zero additive noise, convergence to L becomes faster than the scheduled
This accelerated convergence is probably because a portion of the additive noise is in line with the manifold, so does not contribute to
In addition to the total effective noise, we can compare the evolution of the removed noise versus injected noise.
Figure 5 shows the reduction in effective standard deviation, h t σ t = h t
||
d t || √ N in each iteration, along with the standard deviation of the added noise, γ t .
The amount of noise added relative to the amount removed is such that effective noise drops as σ t
= (1 − βh t ) σ t − 1 .
When β = 1 , the additive noise is zero, γ t = 0 , and the convergence of σ t is the fastest.
When β = 0 .
01 , a lot of noise is added in each iteration, and the convergence is the slowest.
[Figure]
= 0.
F Image synthesis examples
We examined the properties of images synthesized using this stochastic coarse-to-ﬁne ascent algorithm.
For a denoiser, we used BF-CNN [31], a bias-free variant of DnCNN [28], trained on 40 × 40 patches cropped from Berkeley segmentation training dataset (see Appendix A for further details).
We obtained similar results (not shown) using other bias-free CNN denoisers (see [31]).
For the sampling algorithm, we chose parameters σ 0 = 1 , σ L = 0 .
01 , and h 0 = 0 .
01 for all experiments.
Figure 6 illustrates the iterative generation of two images, starting from different random initializations, y 0 , with no additional noise injected (i.e, β = 1 ), demonstrating the way that the algorithm ampliﬁes and "hallucinates" structure found in the initial (noise) images.
Convergence is typically achieved in less than 40 iterations with stochasticity disabled ( β = 1 ).
The left panel in Figure 7 shows samples drawn with different initializations, y 0 , using a moderate level of injected noise ( β = 0 . 5 ).
Images contain natural-looking features, with sharp contours, junctions, shading, and in some cases, detailed texture regions.
The right panel in Figure 7 shows a set of samples drawn with more substantial injected noise ( β = 0 . 1 ).
The additional noise helps to avoid local maxima, and arrives at images that are smoother and higher probability, but still containing sharp boundaries.
As expected, the additional noise also lengthens convergence time (see Figure 3).
Figure 8 shows a set of samples drawn from the implicit prior of a denoiser trained on the MNIST dataset of handwritten digits.
G Examples of inverse problems
We demonstrate the results of applying our method to several linear inverse problems.
The same algorithm and parameters are used on all problems - only the measurement matrix M and measured values M T x are altered.
In particular, as in section F, we used BF-CNN [31], and chose parameters σ 0 = 1 , σ L = 0 .
01 , h 0 = 0 .
01 , β = 0 .
For each example, we show a row of original images ( x ), a row of direct least-squares reconstructions ( MM T x ), and a row of restored images generated by our algorithm.
For these applications, comparisons to ground truth are not particularly meaningful, at least when the measurement matrix is low-rank.
In these cases, the algorithm relies heavily on the prior to "hallucinate" the missing information, and the goal is not so much to reproduce the original image, but to create an image that looks natural while being consistent with the measurements.
Thus, the best measure of performance is a judgement of perceptual quality by a human observer.
Inpainting.
A simple example of a linear inverse problem involves restoring a block of missing pixels, conditioned on the surrounding content.
Here, the columns of the measurement matrix M are a subset of the identity matrix, corresponding to the measured (outer) pixel locations.
We choose a missing block of size 30 × 30 pixels, which is less than the size of the receptive ﬁeld of the BF-CNN network ( 40 × 40 ), the largest extent over which this denoiser can be expected to capture joint statistical relationships.
There is no single correct solution for this problem: Figure 9 shows multiple solutions, resulting from different initalizations.
Each appears plausible and consistent with the surrounding content.
Figure 10 shows additional examples.
Random missing pixels.
Consider a measurement process that discards a random subset of pixels.
M is a low rank matrix whose columns consist of a subset of the identity matrix corresponding to the randomly chosen set of preserved pixels.
Figure ??
shows examples with either 30% or 10% of pixels retained.
Despite the signiﬁcant number of missing pixels, the recovered images are remarkably similar to the originals.
Spatial super-resolution.
In this problem, the goal is to construct a high resolution image from a low resolution (i.e. downsampled) image.
Downsampling is typically performed after lowpass ﬁltering, which determines the measurement model, M .
Here, we use a 4 × 4 block-averaging ﬁlter.
Figure 11 shows examples.
Deblurring (spectral super-resolution).
The previous applications were based on partial measurements in the pixel domain.
Consider a blurring operator that operates by retaining a set of low-frequency coefﬁcient in the Fourier domain, discarding the rest.
In this case, M consists of the preserved low-frequency columns of the discrete Fourier transform, and MM T x is a blurred version of x .
Examples are shown in Figure 12.
Compressive sensing.
Compressive sensing [44, 45] provides a set of theoretical results regarding recovery of sparse signals from a small number of linear measurements.
Speciﬁcally, if one assumes that signals can be represented with at most k << N non-zero coefﬁcients in a known basis, they can be recovered from a measurements obtained by projecting onto a surprisingly small number of axes (approaching k log( N/k ) ), far fewer than expected from traditional
Shannon-Whitaker sampling theory.
The theory relies on the sparsity property, which corresponds to a “union of subspaces” prior [46], and on the measurement axes being incoherent (essentially, weakly correlated) with the sparse basis.
Typically, one chooses a sensing matrix containing a set of n << N random orthogonal axes.
Recovery is achieved by solving the sparse inverse problem, using any of a number of methods.
Photographic images are not truly sparse in any ﬁxed linear basis, but they can be reasonably approximated by lowdimensional subsets of Fourier or wavelet basis functions, and compressive sensing results are typically demonstrated using one of these.
The manifold prior embedded within our CNN denoiser corresponds to a nonlinear form of sparsity, and analogous to sparse inverse algorithms used in compressed sensing, our stochastic coarse-to-ﬁne ascent algorithm can be used to recover an image from a set of linear projections onto a set of random basis functions.
Figureshows four examples of images recovered from random projections using our denoiser-induced manifold prior, versus a sparse discrete cosine transform (DCT) prior.
In all cases, the denoiser-recovered images exhibit sharper edges, more detail, and fewer artifacts.
Numerical performance, in terms of both PSNR and SSIM, is consistent with the perceptual comparison.