Paper:
Deep Learning (DL) algorithms based on Generative Adversarial Network (GAN) have demonstrated great potentials in computer vision tasks such as image restoration. Despite the rapid development of image restoration algorithms using DL and GANs, image restoration for specific scenarios, such as medical image enhancement and super-resolved identity recognition, are still facing challenges. How to ensure visually realistic restoration while avoiding hallucination or modecollapse? How to make sure the visually plausible results do not contain hallucinated features jeopardizing downstream tasks such as pathology identification and subject identification? Here we propose to resolve these challenges by coupling the GAN based image restoration framework with another task-specific network. With medical imaging restoration as an example, the proposed model conducts additional pathology recognition/classification task to ensure the preservation of detailed structures that are important to this task. Validated on multiple medical datasets, we demonstrate the proposed method leads to improved deep learning based image restoration while preserving the detailed structure and diagnostic features. Additionally, the trained task network show potentials to achieve superhuman level performance in identifying pathology and diagnosis. Further validation on super-resolved face identity recognition tasks also show that the proposed method can be generalized for diverse image restoration tasks.

Review:
This paper proposed a new method for image restoration based a task-discriminator in addition to the GAN network. It shows superior performance than the baseline methods without such task-discriminator on medical image restoration and image super-resolution. While the results are better, the idea seems straightforward and has limited novelty. Please see the following comments:
1. Adding an task-discriminator in a GAN network seems straightforward to improve the specific task. And this idea has already used in existing papers, e.g. Cycada.  
Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A.A. and Darrell, T., 2017. Cycada: Cycle-consistent adversarial domain adaptation. ICML, 2018
2. On the application side, the results are not very convincing because the baseline methods were not selected properly. For medical image reconstruction and image super-resolution, the proposed method was not compared with any of the state-of-the-art methods, but only with the same method without a task-discriminator as a baseline. For those tasks, there are many traditional methods and deep nets with different losses. For example, a simple L1/L2 or perceptual loss probably leads to better PSNR than the GAN loss, which is not compared at all. See the attached references. 
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z. and Shi, W., Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR 2017.
Johnson, J., Alahi, A. and Fei-Fei, L., Perceptual losses for real-time style transfer and super-resolution. In ECCV 2016.
Kim, J., Kwon Lee, J. and Mu Lee, K., Accurate image super-resolution using very deep convolutional networks. In CVPR 2016.
3. Some questions about medical image datasets. For the low-dose PET dataset, the input was randomly undersampled by a factor of 100. What is the random pattern? Is it uniform? In addition, why not acquire real low-dose data and show the quality results using the proposed model? For the multi-constast MRI data, how is the input generated and what is the ground-truth?