I NVERTIBLE GENERATIVE MODELS FOR INVERSE PROBLEMS : MITIGATING REPRESENTATION ERROR AND DATASET BIAS
Anonymous authors
Paper under double-blind review
A BSTRACT
Trained generative models have shown remarkable performance as priors for inverse problems in imaging.
For example, Generative Adversarial Network priors permit recovery of test images from 5-10x fewer measurements than sparsity priors.
Unfortunately, these models may be unable to represent any particular image because of architectural choices, mode collapse, and bias in the training dataset.
In this paper, we demonstrate that invertible neural networks, which have zero representation error by design, can be effective natural signal priors at inverse problems such as denoising, compressive sensing, and inpainting.
Our formulation is an empirical risk minimization that does not directly optimize the likelihood of images, as one would expect.
Instead we optimize the likelihood of the latent representation of images as a proxy, as this is empirically easier.
For compressive sensing, our formulation can yield higher accuracy than sparsity priors across almost all undersampling ratios.
For the same accuracy on test images, they can use 10-20x fewer measurements.
We demonstrate that invertible priors can yield better reconstructions than sparsity priors for images that have rare features of variation within the biased training set, including out-of-distribution natural images.
I NTRODUCTION
[Figure]
Truth
Lasso
DCGAN
Ours particular natural signal classes, such as faces, ﬁngerprints, MRIs, and more (Karras et al., 2017;
Minaee and Abdolrashidi, 2018; Shin et al., 2018; Chen et al., 2018).
Some such models, including variational autoencoders (VAEs) and generative adversarial networks (GANs), learn an explicit low-dimensional manifold that approximates a natural signal class (Goodfellow et al., 2014; Kingma and Welling, 2013; Rezende et al., 2014).
We will refer to such models as GAN priors.
With an explicit parameterization of the natural signal manifold by a low dimensional latent representation, these generative models allow for direct optimization over a natural signal class.
Consequently, they can obtain signiﬁcant performance improvements over non-learning based methods.
For example,
GAN priors have been shown to outperform sparsity priors at compressed sensing with 5-10x fewer measurements.
Additionally, GAN priors have led to theory for signal recovery in the linear compressive sensing and nonlinear phase retrieval problems (Bora et al., 2017; Hand and Voroninski, 2017; Hand et al., 2018), and they have also shown promising results for the nonlinear blind image deblurring problem (Asim et al., 2018).
A signiﬁcant drawback of GAN priors for solving inverse problems is that they can have representation error or bias due to architecture and training.
This can happen for many reasons, including because the generator only approximates the natural signal manifold, because the natural signal manifold is of higher dimensionality than modeled, because of mode collapse, or because of bias in the training dataset itself.
As many aspects of generator architecture and training lack clear principles, representation error of GANs may continue to be a challenge even after substantial hand crafting and engineering.
Additionally, learning-based methods are particularly vulnerable to the biases of their training data, and training data, no matter how carefully collected, will always contain degrees of bias.
As an example, the CelebA dataset (Liu et al., 2015) is biased toward people who are young, who do not have facial hair or glasses, and who have a light skin tone.
As we will see, a GAN prior trained on this dataset learns these biases and exhibits image recovery failures because of them.
In contrast, invertible neural networks can be trained as generators with zero representation error.
These networks are invertible (one-to-one and onto) by architectural design (Dinh et al., 2016;
Gomez et al., 2017; Jacobsen et al., 2018; Kingma and Dhariwal, 2018).
Consequently, they are capable of recovering any image, including those signiﬁcantly out-of-distribution relative to a biased training set; see Figure 1.
We call the domain of an invertible generator the latent space, and we call the range of the generator the signal space.
These must have equal dimensionality.
Flow-based invertible generative models are composed of a sequence of learned invertible transformations.
Their strengths include: their architecture allows exact and efﬁcient latent-variable inference, direct loglikelihood evaluation, and efﬁcient image synthesis; they have the potential for signiﬁcant memory savings in gradient computations; and they can be trained by directly optimizing the likelihood of training images.
This paper emphasizes an additional strength: because they lack representation error, invertible models can mitigate dataset bias and improve performance on inverse problems with out-of-distribution data.
In this paper, we study generative invertible neural network priors for imaging inverse problems.
We will speciﬁcally use the Glow architecture, though our framework could be used with other architectures.
A Glow-based model is composed of a sequence of invertible afﬁne coupling layers, 1x1 convolutional layers, and normalization layers.
Glow models have been successfully trained to generate high resolution photorealistic images of human faces (Kingma and Dhariwal, 2018).
We present a method for using pretrained generative invertible neural networks as priors for imaging inverse problems.
The invertible generator, once trained, can be used for a wide variety of inverse problems, with no speciﬁc knowledge of those problems used during the training process.
Our method is an empirical risk formulation based on the following proxy: we penalize the likelihood of an image’s latent representation instead of the image’s likelihood itself.
While this may be couterintuitive, it admits optimization problems that are easier to solve empirically.
In the case of compressive sensing, our formulation succeeds even without direct penalization of this proxy likelihood, with regularization occuring through initialization of a gradient descent in latent space.
We train a generative invertible model using the CelebA dataset.
With this ﬁxed model as a signal prior, we study its performance at denoising, compressive sensing, and inpainting.
For denoising, it can outperform BM3D (Dabov et al., 2007).
For compressive sensing on test images, it can obtain higher quality reconstructions than Lasso across almost all subsampling ratios, and at similar reconstruction errors can succeed with 10-20x fewer measurements than Lasso.
It provides an improvement of about 2x fewer linear measurements when compared to Bora et al.
Despite being trained on the CelebA dataset, our generative invertible prior can give higher quality reconstructions than
Lasso on out-of-distribution images of faces, and, to a lesser extent, unrelated natural images.
Our invertible prior outperforms a pretrained DCGAN (Radford et al., 2015) at face inpainting and exhibits qualitatively reasonable results on out-of-distribution human faces.
We provide additional experiments in the appendix, including for training on other datasets.
M ETHOD AND M OTIVATION
We assume that we have access to a pretrained generative invertible neural network G : R n → R n .
We write x = G ( z ) and z = G − 1 ( x ) , where x ∈ R n is an image that corresponds to the latent representation z ∈ R n .
We will consider a G that has the Glow architecture introduced in Kingma and Dhariwal (2018).
It can be trained by direct optimization of the likelihood of a collection of training images of a natural signal class, under a standard Gaussian distribution over the latent space.
We consider recovering an image x from possibly-noisy linear measurements given by A ∈ R m × n , y =
Ax + η, where η ∈ R m models noise.
Given a pretrained invertible generator G , we have access to likelihood estimates for all images x ∈ R n .
Hence, it is natural to attempt to solve the above inverse problem by a maximum likelihood formulation given by
[Equation]
where p G is the likelihood function over x induced by G , and γ is a hyperparameter.
We have found this formulation to be empirically challenging to optimize; hence we study the following proxy:
[Equation]
Unless otherwise stated, we initialize (2) at z 0 = 0 .
The motivation for formulation (2) is as follows.
As a proxy for the likelihood of an image x ∈ R n , we will use the likelihood of its latent representation z
= G − 1 ( x ) .
Because the invertible network G was trained to map a standard normal in R n to a distribution over images, the log-likelihood of a point z is proportional to (cid:107) z (cid:107) 2 .
Instead of penalizing (cid:107) z (cid:107) 2 , we alternatively penalize the unsquared (cid:107) z (cid:107) .
In Appendix B, we show comparable performance for both the squared and unsquared formulations.
In principle, our formulation has an inherent ﬂaw: some high-likelihood latent representations z correspond to low-likelihood images x .
Mathematically, this comes from the Jacobian term that relates the likelihood in z to the likelihood in x upon application of the map G .
For multimodel distributions, such images must exist, which we will illustrate in the discussion.
This proxy formulation relies on the fact that the set of such images has low probability and that they are inconsistent with enough provided measurements.
Surprisingly, despite this potential weakness, we will observe image reconstructions that are superior to BM3D and GAN-based methods at denoising, and superior to GAN-based and
Lasso-based methods at compressive sensing.
In the case of compressive sensing and inpainting, we take γ = 0 in formulation (2).
The motivation for such a formulation initialized at z 0 = 0 is as follows.
There is a manifold of images that are consistent with the provided measurements.
We want to ﬁnd the image x of highest likelihood on this manifold.
Our proxy turns the likelihood maximization task over an afﬁne space in x into the geometric task of ﬁnding the point on a manifold in z -space that is closest to the origin with respect to the Euclidean norm.
In order to approximate that point, we run a gradient descent in z down the data misﬁt term starting at z 0 = 0 .
In the case of GAN priors for G : R k → R n , we will use the formulation from Bora et al.
(2017), which is the formulation above in the case where the optimization is performed over R k , γ = 0 , and initialization is selected randomly.
All the experiments that follow will be for an invertible model we trained on the CelebA dataset of celebrity faces, as in Kingma and Dhariwal (2018).
Similar results for models trained on birds and ﬂowers (Wah et al., 2011; Nilsback and Zisserman, 2008) can be found in the appendix.
Due to computational considerations, we run experiments on 64 × 64 color images with the pixel values scaled between [0 , 1] .
The train and test sets contain a total of 27,000 and 3,000 images, respectively.
We trained a Glow architecture (Kingma and Dhariwal, 2018); see Appendix A for details.
Once trained, the Glow prior is ﬁxed for use in each of the inverse problems below.
We also trained a
DCGAN for the same dataset.
We solve (2) using LBFGS, which was found to outperform Adam (Kingma and Ba, 2014).
DCGAN results are reported for an average of 3 runs because we observed some variance due to random initialization.
A PPLICATIONS 3. D ENOISING
We consider the denoising problem with A
=
I and η ∼ N (0 , σ 2 I ) , for images x in the CelebA test dataset.
We evaluate the performance of a Glow prior, a DCGAN prior, and BM3D for two different noise levels.
Figure 2 shows the recovered PSNR values as a function of γ for denoising by the
Glow and DCGAN priors, along with the PSNR by BM3D.
The ﬁgure shows that the performance of the regularized Glow prior increases with γ , and then decreases.
If γ is too low, then the network ﬁts to the noise in the image.
If γ is too high, then data ﬁt is not enforced strongly enough.
The left panel reveals that an appropriately regularized Glow prior can outperform BM3D by almostdB. The experiments also reveal that appropriately regularized Glow priors outperform the DCGAN prior, which suffers from representation error and is not aided by the regularization.
The right panel conﬁrms that with smaller noise levels, less regularization is needed for optimal performance.
A
[Figure]
All the results are averaged over 12 test set images.
For reference, we show the average PSNRs of the original noisy images, after applyig BM3D, and under the Glow prior in the noiseless case ( σ = 0 ).
visual comparison of the recoveries at the noise level σ = 0 .
1 using Glow, DCGAN priors, and
BM3D can be seen in Figure 3.
Note that the recoveries with Glow are sharper than BM3D. See
Appendix B for more quantitative and qualitative results.
[Figure]
u t
[Figure]
i s y
[Figure]
D C
GAN γ =
B M 3 D n .
a .
G L O
W γ =
3. C OMPRESSED S ENSING
In compressed sensing, one is given undersampled linear measurements of an image, and the goal is to recover the image from those measurements.
In our notation, A ∈ R m × n with m < n .
As the image x is undersampled, there is an afﬁne space of images consistent with the measurements, and an algorithm must select which is most ‘natural.’
A common proxy for naturalness in the literature has been sparsity with respect to the DCT or wavelet bases.
With a GAN prior, an image is considered natural if it lies in or near the range of the GAN.
For an invertible prior under our proxy for likelihood, we consider an image to be natural if it has a latent representation of small norm.
We study compressed sensing in the case that A is an m × n matrix of i.i.d.
N (0 , 1 /m )
entries, and x is an image from the CelebA test set.
Here, n = 64 × 64 × 3 = 12288 .
We consider the case where η is standard iid Gaussian random noise normalized such that (cid:112) E (cid:107) η (cid:107) 2 = 0 .
We compare Glow,
DCGAN, and Lasso 1 with respect to the DCT and wavelet bases.
Our main result is that the Glow prior with γ = 0 and initialization z
= 0 outperforms both DCGAN and Lasso in reconstruction quality over all undersampling ratios, as shown in the left panel of Figure 4.
Surprisingly, in the case of extreme undersampling, Glow substantially outperforms these methods even though it does not maintain a direct low-dimensional parameterization of the signal manifold.
The Glow prior (1) can result in 15 dB higher PSNRs than DCGAN, and (2) can give comparable recovery errors with 2-3x fewer measurements at high undersampling ratios.
This difference is explained by the representation error of DCGAN, which has been shown to be the dominant source of error in DCGAN by Bora et al.
Additional plots and visual comparisons, available in Appendix C, show notable improvements in quality of in- and out-of-distribution images using an invertible prior relative to DCGAN and Lasso.
[Figure]
We initialize with z 0 = 0 .
See Appendix C for a zoom-in of the case of small m .
The panel shows the resulting PSNR when m = 5000 with a Glow prior after different initialization strategies, as described in the text.
The highest PSNR was recovered with initialization z 0 = 0 and γ = 0 .
We conducted several additional experiments to understand the regularizing effects of γ and the initialization z 0 .
The right panel of Figure 4 shows the PSNRs under multiple initialization strategies:
z 0 = 0 , z 0 ∼ N (0 , 0 .
1 2 I ) , z 0 ∼ N (0 , 0 .
7 2 I ) , z
=
G − 1 ( x 0 ) with x 0 given by the solution to Lasso with respect to the wavelet basis, and z 0 =
G − 1 ( x 0 ) where x 0 is x perturbed by a random point in the null space of A .
The best performance was observed with initialization z 0 = 0 .
The hyperparameter γ can be taken to be zero, which is surprising because then there is no direct penalization of likelihood for this noisy compressive sensing problem.
In the case of γ = 0 , we observe that larger initializations result in recovered images of lower PSNR.
See Appendix C for additional experiments that show this effect.
We observe that initialization strategy can have a strong qualitative effect on the recovery formulation.
For example, if the optimization is initialized by the solution to the Lasso, then directly penalizing the likelihood of z can improve reconstruction PSNR, though those reconstruction are still worse than with initialization z 0 = 0 and γ = 0 .
Suboptimal initialization procedures apparently beneﬁt from direct penalization of likelihood, whereas the z 0 = 0 initialization apparently does not.
Finally, we observe that the Glow prior is much more robust to out-of-distribution examples than the GAN Prior.
Figure 5 shows recovered images using (2) for compressive sensing for images not belonging to the CelebA dataset.
DCGAN’s performance reveals biases of the underlying dataset and limitations of low-dimensional modeling.
For example, projecting onto the CelebA-trained DCGAN can cause incorrect skin tone, gender, and age.
It’s performance on out-of-distribution images is poor.
In contrast, the Glow prior mitigates this bias, even demonstrating image recovery for natural images that are not representative of the CelebA training set, including people who are older, have darker skin tones, wear glasses, have a beard, or have unusual makeup.
The Glow prior’s performance also extends to signiﬁcantly out-of-distribution images, such as animated characters and natural images unrelated to faces.
See Appendix C.2 for additional experiments.
[Figure]
r u t h
D C T
W V T
D C
[Figure] N
[Figure]
3. I NPAINTING
In inpainting, one is given a masked image of the form y =
M (cid:12) x , where M is a masking matrix with binary entries and x ∈ R n is an n-pixel image.
The goal is to ﬁnd x .
We could rewrite (2) with γ = 0 as
[Figure]
There is an afﬁne space of images consistent with the measurements, and an algorithm must select which is most natural.
As before, using the minimizer ˆ z , the estimated image is given by G (ˆ z ) .
Our experiments reveal the same story as for compressed sensing.
If initialized at z 0 = 0 , then the empirical risk formulation with γ = 0 exhibits high PSNRs on test images.
Algorithmic regularization is again occurring due to initialization.
In contrast, DCGAN is limited by its representation error.
See Figure 6, and Appendix D for more results, including visually reasonable face inpainting, even for out-of-distribution human faces.
D ISCUSSION
We have demonstrated that pretrained generative invertible models can be used as natural signal priors in imaging inverse problems.
Their strength is that every desired image is in the range of an invertible model , and the challenge that they overcome is that every undesired image is also in the range of the model and no explicit low-dimensional representation is kept .
We study a regularization for empirical loss minimization that promotes recovery of images that have a high value of a proxy for image likelihood under the generative model.
We demonstrate that this formulation can quantitatively and qualitatively outperform BM3D at denoising.
Additionally, it has lower recovery errors than Lasso across all levels of undersampling, and it can get comparable errors from 10-20x fewer measurements, which is a 2x reduction from Bora et al.
The superior recovery performance of the invertible prior at very extreme undersampling ratios is particularly surprising given that invertible nets do not maintain explicit low dimensional representations, as GANs do.
Additionally, our trained invertible model yields signiﬁcantly better reconstructions than Lasso even on out-of-distribution images, including images with rare features of variation, and on unrelated natural images.
The idea of analyzing inverse problems with invertible neural networks has appeared in Ardizzone et al.
The authors study estimation of the complete posterior parameter distribution under a forward process, conditioned on observed measurements.
Speciﬁcally, the authors approximate a particular forward process by training an invertible neural network.
The inverse map is then directly available.
In order to cope with information loss, the authors augment the measurements with additional variables.
This work differs from ours because it involves training a separate net for every particular inverse problem.
In contrast, our work studies how to use a pretrained invertible generator for a variety of inverse problems not known at training time.
Training invertible networks is challenging and computationally expensive; hence, it is desirable to separate the training of off-the-shelf invertible models from potential applications in a variety of scientiﬁc domains.
Why optimize a proxy for image likelihood instead of optimizing image likelihood directly?
As noted in Section 2, the immediate formulation one would write down for inverse problems under an invertible prior is to optimize a data misﬁt term together with an image log-likelihood term.
Unfortunately, we found it difﬁcult to get this optimization to converge in practice.
The likelihood term can exhibit rapid variation due to the Jacobian of the transformation z (cid:55)→ x = G ( z ) ; additionally the likelihood term may in principle even contain local minima or other geometric properties that make gradient descent difﬁcult.
Figure 7 compares the loss landscapes in x and z , illustrating that the learned likelihood function in x may lead to difﬁculty in choosing appropriate step sizes for gradient descent algorithms.
[Figure]
In contrast, there are nice geometric properties that appear in latent space from an invertible model.
As an illustration, consider the compressive sensing problem with noiseless measurements.
Here, the formulation corresponds to a gradient descent down the data misﬁt term (cid:107) AG ( z ) − y (cid:107) 2 starting at z 0 = 0 .
This data misﬁt term has a favorable geometry for optimization in that all local minima are global minima.
This is because the level sets in z of (cid:107) AG ( z ) − y (cid:107) 2 are given by G − 1 applied to the level sets in x of (cid:107) Ax − y (cid:107) 2 , which have a simple structure because of the linearity of the measurements in x .
There may be additional beneﬁts due to optimizing in z because the invertible net learns representations that permit interpolation between images and semantically meaningful arithmetic, as reported in Kingma and Dhariwal (2018).
Why is the likelihood of an image’s latent representation a reasonable proxy for the image’s likelihood?
The training process for an invertible generative model attempts to learn a target distribution in images space by directly maximizing the likelihood of provided samples from that distribution, given a standard Gaussian prior in latent space.
High probability regions in latent space map to regions in image space of equal probability.
Hence, broadly speaking, regions of small values of (cid:107) z (cid:107) are expected to map to regions of large likelihoods in image space.
There will be exceptions to this
[Figure]
property.
For example, natural image distributions have a multimodal character.
The preimage of high probability modes in image space will correspond to high likelihood regions in latent space.
Because the generator G is invertible and continuous, interpolation in latent space of these modes will provide images of high likelihood in z but low likelihood in the target distribution.
To illustrate this point, we trained a Real-NVP (Dinh et al., 2016) invertible neural network on the two dimensional set of points depicted in Figure 8 (left panel).
The middle and right panels show that high likelihood regions in latent space generally correspond to higher likelihood regions in image space, but that there are some regions of high likelihood in latent space that map to points of low likelihood in image space and in the target distribution.
We see that the spurious regions are of low total probability and would be unlikely to be the desired outcomes of an inverse problem arising from the target distribution.
How can solving compressive inverse problems be successful without direct penalization of the proxy image likelihood?
If there are fewer linear measurements than the dimensionality of the desired signal, an afﬁne space of images is consistent with the measurements.
In our formulation, regularization does not occur by direct penalization of our proxy for image likelihood; instead, it occurs implicitly by performing the optimization in z -space with an initialization of z 0 = 0 .
The set of latent representations z that are consistent with the compressive measurements deﬁne a m -dimensional nonlinear manifold.
As per the likelihood proxy mentioned above, the spirit of our formulation is to ﬁnd the point on this manifold that is closest to the origin with respect to the Euclidean norm.
Our speciﬁc way of estimating this point is to perform a gradient descent down a data misﬁt term in z -space, starting at the origin.
While a gradient ﬂow typically will not ﬁnd the closest point on the manifold, it empirically ﬁnds a reasonable approximation of that point.
In practice, one could further do a local search to reﬁne the output of this gradient ﬂow, but we elect not to do so for the sake of simplicity.
Why does the invertible prior do so well, especially on out-of-distribution images?
One reason that the invertible prior performs so well is because it has no representation error.
The lack of representation error of invertible nets presents a signiﬁcant opportunity for imaging with a learned prior.
Any image is potentially recoverable, even if the image is signiﬁcantly outside of the training distribution.
In contrast, methods based on projecting onto an explicit low-dimensional representation of a natural signal manifold will have representation error, perhaps due to modeling assumptions, mode collapse, or bias in a training set.
Such methods will see performance prematurely saturate as the number of measurements increases.
In contrast, an invertible prior would not see performance saturate.
In the extreme case of having a full set of exact measurements, an invertible prior could in principle recover any image exactly.
It is natural to wonder which images can be effectively recovered using an invertible prior trained on a particular signal class.
As expected, we see the best reconstruction errors on in-distribution images and performance degrades as images get further out-of-distribution.
Nonetheless, we observe that reconstruction errors of unrelated natural images are still of higher quality than with the Lasso.
It appears that the invertible generator learns some general attributes of natural images.
This leads to several questions: when a generative invertible net is trained, how far out-of-distribution can an image be while maintaining a high likelihood?
How do invertible nets learn useful statistics of natural images?
Is that due primarily to training, or is there architectural bias toward natural images, as with the Deep Image Prior and Deep Decoder (Ulyanov et al., 2018; Heckel and Hand, 2018)?
The results of this paper provide further evidence that reducing representational error of generators can signiﬁcantly enhance the performance of generative models for inverse problems in imaging.
This idea was also recently explored in Athar et al.
(2018), where the authors trained a GAN-like prior with a high- dimensional latent space.
The high dimensionality of this space lowers representational error, though it is not zero.
In their work, the high-dimensional latent space had a structure that was difﬁcult to directly optimize so the authors successfully modeled latent representations as the output of an untrained convolutional neural network whose parameters are estimated at test time.
Their paper and ours raises several questions: Which generator architectures provide a good balance between low representation error ease of training, and ease of inversion
Should a generative model be capable of producing all images in order to perform well on out-of-distribution images of interest?
Are there cheaper architectures that perform comparably?
These questions are quite important, as solving equation 2 in our 64 × 64 pixel color images experiments took
GPU
-minutes.
New developments are needed on architectures and frameworks in between low- dimensional generative priors and fully invertible generative priors
Such methods could leverage the strengths of invertible models while being much cheaper to train and use.
A E XPERIMENTAL S ETUP
Simulations were completed mainly on CelebA-HQ dataset, used in Kingma and Dhariwal (2018); it has 30,000 color images that were resized to 64 × 64 for computational reasons, and were split into 27,000 training and 3000 test images.
We also provide some additional experiments on the Flowers
Nilsback and Zisserman (2008), and Birds Wah et al.
(2011) datasets.
Flowers dataset containscolor images resized to 64 × 64 out of which 500 images are spared for testing.
Birds dataset contains a total of 11,788 images, which were center aligned and resized to 64 × 64 out of which 5794 images are set aside for testing.
We speciﬁcally model our invertible networks after the recently proposed Glow Kingma and Dhariwal (2018) architecture, which consists of a multiple ﬂow steps.
Each ﬂow step comprises of an activation normalization layer, a 1 × 1 convolutional layer, and an afﬁne coupling layer, each of which is invertible.
Let K be the number of steps of ﬂow before a splitting layer, and L be the number of times the splitting is performed.
To train over CelebA, we choose the network to have K = 48 , L = 4 and afﬁne coupling, and train it with a learning rate 0 . 0001 , and a batch size 6 at resolution 64 × 64 × 3 .
The model was trained over 5 − bit images with 10,000 warmup iterations as in Kingma and Dhariwal (2018), but when solving inverse problems using Glow original 8 − bit images were used.
We refer the reader to Kingma and Dhariwal (2018) for speciﬁc details on the operations performed in each of the network layer.
We use LBFGS to solve the inverse problem.
For best performance, we set the number of iterations and learning rate for denoising, compressed sensing, and inpainting to be 20 , 1 ; 30 , 0 . 1 ; and 20 , 1 ; respectively.
we use Pytorch to implement Glow network training and solve the inverse problem.
Glow training was conducted on a single Titan Xp GPU using a maximum allowable (under given computational constraints) batch size of 6.
In case of CS, recovering a single image on Titan Xp using LBFGS solver with 30 steps takes 889 .
125 seconds ( 14 . 82 minutes).
However, we can solveinverse problems in parallel on the given hardware platform.
Unless speciﬁed otherwise, inverse problem under Glow prior is always initialized with z 0 = 0 .
Whereas under DCGAN prior, we initialize with z 0 ∼ N (0 , 0 .
1 2 I ) and report average over three random restarts.
In all the quantitative experiments over, the reported quality metrics such as PSNR, and reconstruction errors are averaged over 12 randomly drawn test set images.
B D ENOISING : A DDITIONAL E XPERIMENTS
We present additional quantitative experiments on image denoising here.
Complete set of experiments on average PSNR over 12 CelebA (within distribution 2 ) test set images versus penalization parameter γ under noise levels σ = 0 . 01 , 0 . 05 , 0 . 1 , and 0 .
2 are presented in Figure 10 below.
The central message is that Glow prior outperforms DCGAN prior uniformly across all γ due to the representation limit of DCGAN.
In addition, striking the right balance between the misﬁt term and the penalization term by appropriately choosing γ improves the performance of Glow, and it also approaches stateof-the-art BM3D algorithm at low noise levels, and clearly visible in higher noise, for example, at a noise level of σ = 0 .
2 , the Glow prior improves upon BM3D by 2dB. Visually the results of Glow prior are clearly even superior to BM3D recoveries that are generally blurry and over smoothed as can be spotted in the qualitative results below.
To avoid ﬁtting the noisy image using the Glow model, we force the recoveries to be natural by choosing large enough γ .
γ
P S N
R σ = 0 .
GLOW ( σ = 0)
BM3D
Noise Level
GLOW
DCGAN      γ
P S N
R σ = 0 .
GLOW ( σ = 0)
BM3D
Noise Level
GLOW
DCGAN      γ
P S N
R σ = 0 .
GLOW ( σ = 0)
BM3D
Noise Level
GLOW
DCGAN      γ
P S N
R σ = 0 .
GLOW ( σ = 0)
BM3D
Noise Level
GLOW
DCGAN
Recall that we are solving a regularized empirical risk minimization program
[Equation]
In general, one can instead solve arg min z ∈ Domain ( G ) (cid:107) y − AG ( z ) (cid:107)
+ H ( (cid:107)
z (cid:107) ) , where H ( · ) is a monotonically increasing function.
Figure 11 shows the comparison of most common choices of linear (already used in the rest of the paper), and quadratic H in the context of densoing.
We ﬁnd that the highest achievable PSNR remains the same in both the cases, however, the penalization parameter γ has to be adjusted accordingly.
We train Glow and DCGAN on CelebA. Additional qualitative image denosing results under higher noise level σ = 0 .
1 and 0 .
2 comparing Glow prior against DCGAN prior, and BM3D are presented below in Figure 12, and 13.
We also trained Glow model on Flowers dataset.
Below we present its qualitative denoising performance against BM3D on the test set Flowers images.
We also show the effect of varying γ — smaller γ leads to overﬁtting and vice versa.
.3 .γ
P S N
R σ = 0 .
BM3D
GLOW ( k z k ) GLOW ( k z k 2 )
T r u t h n .
a .
N o i s y n .
a .
D C
GAN γ =
B M 3 D n .
a .
G L O
W γ =
G L O
W γ =
G L O
W γ =
[Figure] [Figure]
C C OMPRESSED S ENISNG : A DDITIONAL E XPERIMENTS
Some additional quantitative image recovery results on test set of CelebA dataset are presented in
Figure
; it depicts the comparison of Glow prior, DCGAN prior, LASSO-DCT, and LASSO-WVT at compressed sensing.
We plot the reconstruction error :
= 1 n (cid:107) x − ˆ x (cid:107) 22 , where ˆ x is the recovered image and n = 12288 is the number of pixels in the 64 × 64 × 3 CelebA images.
Glow uniformly outperforms DCGAN
[Figure]
and LASSO across entire range of the number of measuremnts.
LASSODCT and LASSO-WVT eventually catch up to Glow but only when observed measurements are a signiﬁcant fraction of the total number of pixels.
On the other hand, DCGAN is initially better than
LASSO but prematurely saturates due to limited representation capacity
[Figure] [Figure]
[Figure] [Figure]
Surprisingly, we observe that no explicit penalization of likelihood is necessary for compressive sensing with an invertible generative prior under formulation equation 2.
That is, we can take γ =when the optimization is initialized at z 0 = 0 .
This indicates that algorithmic regularization is occurring and that initialization plays a role.
We performed some additional experiments to study the role of initialization.
The left panel in Figure 19 shows that as the norm of the latent initialization increases, the norm of the recovered latent representation increases and the PSNR of the recovered image decreases.
Moreover, the right panel in Figure 19 shows the norm of the estimated latent representation at each iteration of the optimization.
In all our experiments, it monotonically grows versus iteration number.
These experiments provide further evidence that smaller latent initializations lead to outputs that are more natural and have smaller latent representations.
[Figure]
Recall that the natural face images correspond to smaller z 0 .
In Figure 20, we plot the norm of the latent codes of the iterates of each algorithm vs. the number of iterations.
The central message is that initializing with smaller norm z 0 tends to yield natural (smaller latent representations) recoveries.
This is one explanation as to why in compressed sensing, one is able to obtain the true solution out of the afﬁne space of solutions without penalizing the unnaturalness of the recoveries.
[Figure]
We now present visual recovery results on test images from the CelebA dataset under varying number of measurements in compressed sesing.
We compare recoveries under Glow prior, DCGAN prior,
LASSO-DCT, and LASSO-WVT.
[Figure]
[Figure]
T r u t h
D C T
W V T
D C
GAN
G L O
W
[Figure]
r
[Figure]
C
GAN
G
[Figure]
W
T r u t h
D C T
W V T
D C
GAN
G L O
W
[Figure]
r
[Figure]
C
GAN
G
[Figure]
W
T r u t h
D C T
W V T
D C
GAN
G L O
W
[Figure]
GAN
[Figure]
W
T r u t h
D C T
W V T
D C
GAN
G L O
W
[Figure]
W
C. C OMPRESSED S ENSING ON F LOWER AND B IRD D ATASET
We also performed compressed sensing experiments similar to those on CelebA dataset above on
Birds dataset, and Flowers dataset.
We trained a Glow invertible network for each dataset, and present below the quantitative and qualitative recoveries for each dataset.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
C. C OMPRESSED S ENSING ON O UT OF D ISTRIBUTION I MAGES
Lack of representation error in invertible nets leads us to an important and interesting question: does the trained network ﬁt related natural images that are underrepresented or even unrepresented in the training dataset?
Speciﬁcally, can a Glow network trained on CelebA faces be a good prior on other faces; for example, those with dark-skin tone, faces with glasses or facial hair, or even animated faces?
In general, our experiments show that Glow prior has an excellent performance on such out-of-distribution images that are semantically similar to celebrity faces but not representative of the
CelebA dataset.
In particular, we have been able to recover faces of darker skin tone, older people with beards, eastern women, men with hats, and animated characters such as Shrek, from compressed measurements under the Glow prior.
Recoveries under the Glow prior convincingly beat the DCGAN prior, which shows a deﬁnite bias due to training.
Not only that, the Glow prior also outperforms unbiased methods such as LASSO-DCT, and LASSO-WVT.
Can we expect the Glow prior to continue to be an effective proxy for arbitrarily out-of-distribution images?
To answer this question, we tested arbitrary natural images such as car, house door, and butterﬂy wings that are semantically unrelated to CelebA images.
In general, we found that Glow is an effective prior at compressed sensing of out-of-distribution natural images, which are assigned a high likelihood score (small normed latent representations).
On these images, Glow also outperforms
LASSO.
Recoveries of natural images that are assigned very low-likelihood scores by the Glow model generally run into instability issues.
During training, invertible nets learn to assign high likelihood scores to the training images.
All the network parameters such as scaling in the coupling layers of Glow network are learned to behave stably with such high likelihood representations.
However, on very low-likelihood representations, unseen during the training process, the networks becomes unstable and outputs of network begin to diverge to very large values; this may be due to several reasons, such as normalization (scaling) layers not being tuned to the unseen representations.
An LBFGS search for the solution of an inverse problem to recover a low-likelihood image leads the iterates into neighborhoods of low-likelihood representations that may lead the network to instability.
We ﬁnd that Glow network has the tendency to assign higher likelihood scores to arbitrarily outof-distribution natural images.
This means that invertible networks have at least partially learned something more general about natural images from CelebA dataset — may be some high level features that face images share with other natural images such as smooth regions followed by discontinuities, etc.
This allows Glow prior to extend its effectiveness as a prior to other natural images beyond just the training set.
Figure 42, 43 , 44, 45, and 46 compare the performance of LASSO-DCT, LASSO-WVT, DCGAN prior, and Glow prior on the compressed sensing of out-of-distribution images under varying number of measurements.
m = 1 ,
T r u t h
D C T
W V T
D C
GAN
G L O
W T r u t h
D C T
W V T D C
GAN G L O
W m = 2 , T r u t h
D C T
W V T
D C
GAN
G L O
W
T r u t h
D C T
W V T
D C
GAN
G L O
W m = 5 ,
T r u t h
D C T
W V T
D C
GAN
G L O
W
T r u t h
D C T
W V T
D C
GAN
G L O
W m = 7 ,
T r u t h
D C T
W V T
D C
GAN
G L O
W
T r u t h
D C T
W V T
D C
GAN
G L O
W m = 10 ,
T r u t h
D C T
W V T D C
GAN
G L O
W
T r u t h
D C T
W V T
D C
GAN
G L O
W
D I MAGE I NPAINITING
Our experiments with inpainting reveal a similar story as with compressed sensing.
Compared to
DCGAN, the recovered PSNRs using Glow prior are much higher under appropriate γ as depicted in the right panel in Figure 47.
If improperly initialized, then performance for γ = 0 could be poor.
Even if improperly initialized, sufﬁciently large γ leads to higher PSNRs.
As with compressive sensing, if the initialization is from a small latent variable, then the empirical risk formulation with γ = 0 exhibits high PSNRs.
Algorithmic regularization is again occurring due to the small latent variable initialization.
[Figure]
We present here qualitative results on image inpainting under the DCGAN prior, and the Glow prior on the CelebA test set.
to DCGAN, the reconstructions
[Figure]
a
[Figure]
s k e d n .
a .
D C
GAN γ =
G L O
W γ =
D. I MAGE I NPAINTING ON O UT OF D ISTRIBUTION I MAGES
We now perform image inpainiting under Glow prior, and DCGAN prior each trained on CelebA.
Figure 49 shows the visuals of out-of-distribution inpainiting.
As before, DCGAN continues to suffer due to representation limits and data bias while Glow achieves reasonable reconstructions on out-of-distribution images semantically similar to CelebA faces.
As one deviates to other natural images such as houses, doors, and butterﬂy wings, the inpainting performance deteriorates.
At compressed sensing, Glow performed much better on such arbitrarily out-of-distribution images as good recoveries there only require the network only to assign a higher likelihood score to the true image compared to the all the candidate static images given by the null space of the measurement operator.
T r u t h
M a s k e d
D C
GAN
G L O
W
T r u t h
M a s k e d
D C
GAN
G L O
W
E D ISCUSSION
Figure 50 conﬁrms the intuition brought up in the Discussion Section of the main paper that trained
Glow network assigns lower likelihoods (larger latent representations) to noisy images.
Histograms show that noisy images are generally occupy the less likelihood regimes or equivalently, the larger norm latent representations
[Figure]
Our experiments verify that natural images have smaller latent representations than unnatural images.
Here we also show that adding noise to natural images increases the norm of their latent representations, and that higher noise levels result in larger increases.
Additionally we provide evidence that random perturbations in image space induce larger changes in z than comparable natural perturbations in image space.
Figure 51 shows a plot of the norm of the change in image space, averaged overtest images, as a function of the size of a perturbation in latent space.
Natural directions are given by the interpolation between the latent representation of two test images.
For the denoising problem, this difference in sensitivity indicates that the optimization algorithm might obtain a larger decrease in (cid:107) z (cid:107) by an image modiﬁcation that reduces unnatural image components than by a correspondingly large modiﬁcation in a natural direction.
[Figure]
L OSS L ANDSCAPE : DCGAN VS . G LOW
In Figure 52, we plot (cid:107) y − AG ( z ∗ + αδ v + βδ w ) (cid:107) 2 versus ( α, β ) where δ v and δ w are scaled to have the same norm as z ∗ , the latent representation of a ﬁxed test image.
For DCGAN, we plot the loss landscape versus two pairs of random directions.
For Glow, we plot the loss landscape versus a pair of random directions and a pair of directions that linearly interpolate in latent space between z ∗ and another test image.
[Figure]
G I MAGE AND L ATENT S PACE F ORMULATIONS
As mentioned in the main paper, a natural formulation of the inverse problem is
[Equation]
where p G ( x ) is the target density.
We instead formulate the inverse problem as
[Equation]
space.
We will denote the target distribution by p G ( x ) and the latent Gaussian distribution by p ( z ) .
To illustrate the differences between equation 3 and equation 4, we train a Real-NVP model
Dinh et al.
(2016) on a synthetic two-dimensional dataset, visualize both the log p G ( x ) and log p ( z ) in latent and image space, and solve a simple compressive sensing recovery problem.
Our two dimensional data points are generated by sampling the ﬁrst coordinate x 1 from a bimodel Gaussian distribution and the second coordinate x 2 from a uniform distribution as shown in Figure 53.
[Equation]
[Figure] [Figure]
[Figure]
G. C OMPRESSIVE S ENSING IN 2D
To compare latent-space formulation equation 4 and data-space formulation equation
[Figure]
we construct a simple compressive sensing recovery problem for this two-dimensional data and illustrate the difference under both good and bad initializations.
Speciﬁcally, we want to recover a vector x = [ x 1 x 2 ] T from a single linear measurement y =
(cid:104)
a, x (cid:105) =
x 2 , where a = [0 1] T .
Figure 56 shows the gradient descent path, and ﬁnal solution, while solving equation 4 (left column), and equation 3 (right column) from a good and a bad initialization.
x -likelihood formulation seems more robust to a bad initialization in this case compared to z -likelihood as z -likelihood might not be a good proxy for x -likelihood for some points.
This bad case is carefully crafted to illustrate the difference between the two formulations, however, in practice, it seems unlikely that a low x -likelihood points that somehow achieves higher z -likelihood will also obey the measurement constraints.
[Figure]
optimization
[Figure]
latent space
[Figure]
G. C OMPRESSIVE S ENSING FOR C ELEB A
In case of CelebA images, we found that optimizing over direct likelihood of images proved very hard to tune.
To better understand why equation 4 is easier compared to equation 3, we draw the landscape of the loss surfaces of equation 4 versus z and equation 3 versus x under different γ in two random directions around an the ground truth in z , or x , as appropriate; see Figure 57.
In the x -formulation the loss surfaces (ﬁrst row) have a sharp dip at the ground truths, which comes from − log p G ( x ) term.
We believe that sharp dip in the loss surface makes it difﬁcult to tune the γ parameter, the learning rate, and makes the optimization using equation 3 numerically more challenging as observed in our experiments.
On the other hand, the loss surfaces for equation 4 (second row) appear smoother.
We now show a quantitative comparison of the x -likelihood formulation in equation 3, and zlikelihood formulation in equation 4 on compressive sensing for CelebA test images versus m for ﬁxed values of γ ; see Figure 58.
We initialize with z 0 = 0 , and x 0 = G ( z 0 ) , as appropriate.
We simply choose γ = 0 in equation 4.
However, we need to choose γ more carefully in equation 3, and different values of γ are appropriate across different undersampling ratios.
Even if one ignores the difﬁculty of choosing the hyperparameter γ , the formulation in equation 4 generally performs much better than equation 3 as evident from the plots.
[Figure]
[Figure]
To show the effect of noise on recovery in compressive sensing under different values of γ and noise levels, we plot PSNR of the iterates when solving equation 4 against iterations in Figure 59.
This plot shows, perhaps surprisingly, that even under noisy compressed measurements it is a good idea to solve the inverse compressed sensing problem equation 4 with γ = 0 .
G. D ENOISING FOR C ELEB A
For completeness, we also compare denoising using our latent space formulation equation 4, our image space forumation equation 3 under different noise levels σ = 0 .
05 and σ = 0 .
10 ; see Figure 60 and Figure 61 respectively.
For both noise levels, we observe equal performance (indicated by the highest PSNR) when optimizing in the latent or image space.
We do not report results over σ = 0 .as
it was hard to tune hyper paremeters for higher noise levels in equation 3.
[Figure] [Figure]
γ
[Figure] [Figure]
[Figure] [Figure]
γ
[Figure]
γ
[Figure] [Figure]