Paper:
Generating Images with Perceptual Similarity Metrics based on Deep Networks
We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), allowing to generate sharp high resolution images from compressed abstract representations. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric reflects perceptual similarity of images much better and, thus, leads to better results. We demonstrate two examples of use cases of the proposed loss: (1) networks that invert the AlexNet convolutional network; (2) a modified version of a variational autoencoder that generates realistic high-resolution random images.

Review:
The manuscript 'Generating Images with Perceptual Similarity Metrics based on Deep Networks' proposes a distance measure between images that is meant to capture perceptual distances well. The distance measure consists of three terms:           1. A standard l2 distance in pixel space           2. A l2 distance in the feature space given by a late layer in a CNN (AlexNet here)           3. An adversarial loss that quantifies the probability with which a discriminator network thinks both images come from the same distribution.                      The manuscript presents two applications for that distance measure:           1. It shows that reconstructions of images from their compressed deep network representations look perceptually much closer to the original image when using the proposed distance measure as a loss function to train a generator network to invert the deep network representations.           2. It shows that images sampled from a VAE are of better perceptual quality when using the distance measure as the reconstruction loss term during training of the VAE.                      Furthermore it demonstrates how one can use the first application to smoothly interpolate between pairs of images by generating images from their linearly interpolated feature representations.
          I think the most important contribution of the manuscript is to describe a method that substantially improves image reconstruction from compressed deep network representations (e.g. 4k dimensions vs. 227^2 * 3 = 150k dimensions), hence showing exciting new ways of images compression using deep networks. In that regard I would have liked an analysis of the compression rate for the reconstructions from the different feature spaces. In particular because the difference in quality between layer conv5 and fc6 doesn't seem too large, whereas there is a 10-fold reduction in dimensionality ((13 x 13 x 256 = 43k vs. to 4k) in the feature representation.                      One factor that should definitely be discussed in the paper is that it appears as if the adversarial prior enforces to use image data from the training set in the reconstruction. This is not necessarily a problem in terms of image compression, but is an important factor: a careful choice of training data might be important depending on what type of images one wants to compress. And the ability to reconstruct arbitrary images might be limited by the limit on available training data.                      The manuscript argues that the described method has potential applications in terms of reconstructing images from measured brain activity. This is indeed an exciting possible application. However, as there probably won't be enough neural data in the foreseeable future to train a generator from, this argument could have been made a lot stronger by showing that the learned generators are useful to reconstruct images from deep representations of new networks. E.g. one could train a linear predictor from a deep VGG representation to a deep AlexNet representation and show that one can still use the generator trained with AlexNet to reconstruct the image from the VGG representation.                      Minor comments/questions:           - Was the comparator trained, it is mentioned that it could be trained in general but it doesn't say if it was in the described experiments. I assume it wasn't.           - To what extend is random comparator different to just using the img + adv loss? There is only one example to compare and there it could be quite similar, meaning that the random representations don't contribute much.           - The manuscript could mention that when training the generator for reconstruction that there is another prior enforced by the architecture of the generator in addition to the learned adversarial prior. This is important because the (convolutional) network architecture substantially limits the images that can be generated.                      In conclusion, I think that in general the manuscript is solid in its methods and arguments and demonstrates findings of importance to the field that will certainly influence future work.