Paper:
Generative adversarial networks have achieved remarkable performance on various tasks but suffer from sensitivity to hyper-parameters, training instability, and mode collapse. We find that this is partly due to gradient given by non-robust discriminator containing uninformative adversarial noise, which can hinder generator from catching the pattern of real samples. Inspired by defense against adversarial samples, we introduce adversarial training of discriminator on real samples that does not exist in classic GANs framework to make adversarial training symmetric, which can balance min-max game and make discriminator more robust. Robust discriminator can give more informative gradient with less adversarial noise, which can stabilize training and accelerate convergence. We validate the proposed method on image generation tasks with varied network architectures quantitatively. Experiments show that FID score of generated samples is improved by 10% − 50% relative to the baseline with additional 25% computation cost. Training stability is improved and mode collapse is alleviated.

Review:
This paper presents an interesting idea based on introducing adversarial noise on real samples during GAN training. This novel approach may improve GAN training and have potentially large impact, but the paper in its current form is slightly below the standard of ICLR due to its lack of clarity.
While it is very interesting to apply adversarial noise in real data, this approach is not clearly motivated or explained. For example, at the beginning of page 2, why “As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise”? One may think that, on the contrary, the adversarial noise in generated samples would approximate that in real data when the generator distribution approximate the true data distribution. Similarly, after eq. 6, why “Consequently, gradient given by discriminator may vanish when discriminator becomes stronger than generator without capacity constrained.”? I would think the gradient would explode in this case given all the regularisation on gradients.
In addition, the term “non-robust discriminator” has been used several times in important places, but is not clearly defined. Properties used to justify the approach, such as “symmetric” and “balanced” need to be explained. For example, would it be possible to illustrate or measure the imbalance of discriminator?
The overhead of the algorithm needs to be stated more precisely. (after eq. 8) It is unclear to me that “backward propagation of Equation 5 with respect to x with negligible computation overhead”. Would this require back-prop through the entire discriminator, which can be very deep thus costly? It would be helpful to provide an estimation of runtime overhead supported by experiments.
In algorithm 1, why is it necessary to perform discriminator update twice? How about skipping step 4 (eq. 12)? I think this may better mirror the adversarial update for generated samples, which only involves updating generator parameters once.
In section 3.4, I am not sure it is similar to unrolled GAN when the perturbation is zero. Unrolled GAN required backprop through discriminator update, which I don’t think is the case here
Finally, the experimental results are confusing. In Table 1, why the reproduced results are already much better than those in the original papers? A solid baseline is necessary for any further assessment. Further more, since the DCGAN and ResNet baseline models did not use recent regularisation approaches such as spectral-norm, it is hard to assess whether the proposed method can work without those existing techniques. It would be helpful to use at least a SN-GAN baseline.
The analysis also need to be improved. It is hard to interpret the histograms and L1 norms in Figure 5 as related to “informative gradient” or “semantic information” as claimed in sectiion 4.4. Quantitative measurements such as correlation or mutual information may justify these claims.
Overall, I think the idea presented is certainly promising, but needs further development to be qualified for acceptance.
Nit:
The 1-line derivation of eq. 6 can be incorporated into the main text.
A transpose in eq.10 is missing.
Update:
I read the authors' rebuttal, which addressed many of my concerns and presented additional empirical results. The improvement over SN-GAN baseline is particularly convincing. I therefore believe the final camera-ready version can be a valuable contribution to the field, and would like to recommend accepting this paper.