Paper:
Single-image super-resolution (SISR) is a canonical problem with diverse applications. Leading methods like SRGAN (Ledig et al., 2017)  produce images that contain various artifacts, such as high-frequency noise, hallucinated colours and shape distortions, which adversely affect the realism of the result. In this paper, we propose an alternative approach based on an extension of the method of Implicit Maximum Likelihood Estimation (IMLE) (Li & Malik, 2018). We demonstrate greater effectiveness at noise reduction and preservation of the original colours and shapes, yielding more realistic super-resolved images.

Review:
Quality: The overall quality of this paper is good. It adopts a simple but novel idea on SISR and shows clear improvement against existing method (e.g., SRGAN). 
Clarify: This paper is well written and easy to follow. It shows a clear motivation for adopting the implicit probabilistic model.
Originality: To the best of my knowledge, this paper is the first work to learn multi-modal probabilistic model for SISR.
Significance: While the results can be further improved (still look a bit blurred), this paper shows an interesting and important direction to learn better mappings for SISR.
Pros:
+ The writing is clear.
+ The proposed method is well motivated and easy to understand.
+ The experimental results include both objective and subjective evaluations.
Cons:
- The two-stage architecture is similar to the following generative models and SR methods. It’s suggested to discuss them as well.
[a] Denton, E. L., Chintala, S., & Fergus, R. “Deep generative image models using a￼ laplacian pyramid of adversarial networks”. NIPS, 2015.
[b] Karras, T., Aila, T., Laine, S., & Lehtinen, J. “Progressive growing of gans for improved quality, stability, and variation”. ICLR 2018.
[c] Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. “Deep laplacian pyramid networks for fast and accurate super-resolution.” CVPR 2017.
[d] Wang, Y., Perazzi, F., McWilliams, B., Sorkine-Hornung, A., Sorkine-Hornung, O., & Schroers, C. “A Fully Progressive Approach to Single-Image Super-Resolution.”. CVPR Workshops 2018.
- In the hierarchical sampling (section 2.4), it’s not clear how to generate the upper noise vector “conditioned on the lower noise vector”. 
- The hierarchical sampling seems to improve the efficiency of training. I wonder does it affect the results of testing?
- In the implementation details (section 2.5), I don’t understand why you need to transfer the the feature activations from GPU to CPU? I think all the computation can be done on GPU for most common toolboxes. Projecting the activations to a lower dimension with a “random Gaussian matrix” sounds harmful to the results.
- How do you generate the low-resolution images? Are you using bicubic downsampling or other approaches? This detail should be clarified.
- While the evaluation with PSNR and SSIM is a reference to show the quality, many literatures already show that PSNR and SSIM are not correlated well with human perception. It is suggested to also evaluate with some perceptual metrics, e.g., LPIPS [e].
[e] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. “The unreasonable effectiveness of deep features as a perceptual metric.” CVPR 2018.
- In Figure 7, how do you generate different results from the same input image for SRGAN? From my understanding, SRGAN doesn’t take any noise vector as input and cannot generate multi-modal results.
- I feel that the comparison with only SRGAN is not enough. There are some GAN-based SR methods [f][g]. It’s also suggested to compare with MSE-based state-of-the-art SR algorithms [h][i].
[f] Sajjadi, M. S., Schölkopf, B., & Hirsch, M. “Enhancenet: Single image super-resolution through automated texture synthesis.“ ICCV 2017.
[g] Wang, X., Yu, K., Dong, C., & Loy, C. C. “Recovering realistic texture in image super-resolution by deep spatial feature transform.” CVPR 2018.
[h] Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. “Enhanced deep residual networks for single image super-resolution.” CVPR Workshops 2017.
[i] Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. “Residual dense network for image super-resolution.” CVPR 2018.