Paper:
LATENT CONVOLUTIONAL MODELS
We present a new latent model of natural images that can be learned on large-scale datasets. The learning process provides a latent embedding for every image in the training dataset, as well as a deep convolutional network that maps the latent space to the image space. After training, the new model provides a strong and universal image prior for a variety of image restoration tasks such as large-hole inpainting, superresolution, and colorization. To model high-resolution natural images, our approach uses latent spaces of very high dimensionality (one to two orders of magnitude higher than previous latent image models). To tackle this high dimensionality, we use latent spaces with a special manifold structure (convolutional manifolds) parameterized by a ConvNet of a certain architecture. In the experiments, we compare the learned latent models with latent models learned by autoencoders, advanced variants of generative adversarial networks, and a strong baseline system using simpler parameterization of the latent space. Our model outperforms the competing approaches over a range of restoration tasks.

Review:
# Summary
The paper proposes to embed natural images in a latent convolutional space of high dimensionality to obtain a universal image prior. Concretely, each image is embedded as a custom parameter vector of a CNN, which turns random noise into the input of a universal generator network to restore the image in pixel space.
Inference for image restoration is performed by minimizing the energy of a likelihood objective while constraining the latent representation of the restored image to be part of the learned latent space. Experiments for inpainting, super-resolution, and colorization are performed to evaluate the proposed method.
# Positive
As mentioned in the paper, I agree that the idea of learning a universal image prior is appealing, since it can be applied to (m)any image restoration tasks without adjustment.
I am not very familiar with the related work, but if I understood correctly, the paper seems to combine deep latent modeling (GLO, Bojanowski et al., 2018) and deep image priors (Ulyanov et al., 2018). The experiments show good results which qualitatively appear better than those of related methods. A user study also shows that people mostly prefer the results of the proposed method.
Did you try other standard restoration tasks, such as image denoising or deblurring? If not, do you think they would work equally well?
# Limitations
While I agree that a universal image prior is valuable, the paper should (briefly) mention what the disadvantages of the proposed approach are:
- A limitation (at least as presented) is that the corruption process has to be known analytically (as a likelihood objective) and must be differentiable for gradient-based inference.
- Furthermore, the disadvantage of the universal prior as presented in the paper is that restoring an image requires optimization (e.g. gradient descent). In contrast, corruption-specific neural nets typically just need a forward pass to restore the image and are thus easier and faster to use.
# Restoration inference
- How dependent is the restoration result with respect to the initialization? For example, when starting gradient descent with the degraded image vs. a random image.
- Roughly, how many iterations and runtime is needed for inference?
- Did you try different optimizers, such as L-BFGS?