Original: As KL is a rather strong metric between distributions, this is strong evidence that GANs are ﬁnding the true distribution and mode collapse is not happening.
Revised: As KL is a rather strong metric between distributions, this is strong evidence that GANs are ﬁnding the true distribution and mode collapse is not happening.

Original: (2) The W F (eval) and the KL divergence are highly correlated with each other, both along each training run and across different runs.
Revised: (2) The W F (eval) and the KL divergence are highly correlated with each other, both along each training run and across different runs.

Original: In particular, adding gradient penalty improves the optimization signiﬁcantly (which we see in the KL curve), and this improvement is also reﬂected by the W F curve.
Revised: In particular, adding gradient penalty improves the optimization signiﬁcantly (which we see in the KL curve), and this improvement is also reﬂected by the W F curve.

Original: Therefore the quantity W F can serve as a good metric for monitoring convergence and is at least much better than the training loss curve.
Revised: Therefore the quantity W F can serve as a good metric for monitoring convergence and is at least much better than the training loss curve.

Original: To test the necessity of the speciﬁc form of the discriminator we designed, we re-do the same experiment with vanilla fully-connected discriminator nets.
Revised: To test the necessity of the speciﬁc form of the discriminator we designed, we re-do the same experiment with vanilla fully-connected discriminator nets.

Original: Results (in Appendix G.4) show that IPM with vanilla discriminators also correlate well with the KL-divergence.
Revised: Results (in Appendix G.4) show that IPM with vanilla discriminators also correlate well with the KL-divergence.

Original: This is not surprising from a theoretical point of view because a standard fully-connected discriminator net (with some overparameterization) is likely to be able to approximate the log density of the generator distributions (which is essentially the only requirement of Lemma 4.3.)
Revised: This is not surprising from a theoretical point of view because a standard fully-connected discriminator net (with some overparameterization) is likely to be able to approximate the log density of the generator distributions (which is essentially the only requirement of Lemma 4.3.)

Original: For this synthetic case, we can see that the inferior performance in KL of the WGAN-Vanilla algorithm doesn’t come from the statistical properties of GANs, but rather the inferior training performance in terms of the convergence of the IPM.
Revised: For this synthetic case, we can see that the inferior performance in KL of the WGAN-Vanilla algorithm doesn’t come from the statistical properties of GANs, but rather the inferior training performance in terms of the convergence of the IPM.

Original: We conjecture similar phenomenon occurs in training
Revised: We conjecture similar phenomenon occurs in training

Original: GANs with real-life data as well.
Revised: GANs with real-life data as well.

Original: G. P ERTURBED GENERATORS
Revised: G. P ERTURBED GENERATORS

Original: In this section, we remove the effect of the optimization and directly test the correlation between p and its perturbations.
Revised: In this section, we remove the effect of the optimization and directly test the correlation between p and its perturbations.

Original: We compare the KL divergence and neural net IPM on pairs of perturbed generators .
Revised: We compare the KL divergence and neural net IPM on pairs of perturbed generators .

Original: In each instance, we generate a pair of generators ( G, G (cid:48) )
Revised: In each instance, we generate a pair of generators ( G, G (cid:48) )

Original: (with the same architecture as above), where G (cid:48) is a perturbation of G by adding small Gaussian noise.
Revised: (with the same architecture as above), where G (cid:48) is a perturbation of G by adding small Gaussian noise.

Original: We compute the KL divergence and the neural net IPM between G and G (cid:48) .
Revised: We compute the KL divergence and the neural net IPM between G and G (cid:48) .

Original: To denoise the unstable training process for computing the neural net IPM, we optimize the discriminator from 5 random initializations and pick the largest value as the output.
Revised: To denoise the unstable training process for computing the neural net IPM, we optimize the discriminator from 5 random initializations and pick the largest value as the output.

Original: As is shown in Figure 4, there is a clear positive correlation between the (symmetric) KL divergence and the neural net IPM.
Revised: As is shown in Figure 4, there is a clear positive correlation between the (symmetric) KL divergence and the neural net IPM.

Original: In particular, majority of the points fall around the line W F = 100 D kl , which is consistent with our theory that the neural net distance scales linearly in the KL divergence.
Revised: In particular, majority of the points fall around the line W F = 100 D kl , which is consistent with our theory that the neural net distance scales linearly in the KL divergence.

Original: Note that there are a few outliers with large KL.
Revised: Note that there are a few outliers with large KL.

Original: This happens mostly due to the perturbation being accidentally too large so that the weight matrices become poorly conditioned – in the context of our theory, they fall out of the good constraint set as deﬁned in Assumption 1.
Revised: This happens mostly due to the perturbation being accidentally too large so that the weight matrices become poorly conditioned – in the context of our theory, they fall out of the good constraint set as deﬁned in Assumption 1.

Original: G. E XPERIMENTS WITH VANILLA DISCRIMINATOR G.4. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE
Revised: G. E XPERIMENTS WITH VANILLA DISCRIMINATOR G.4. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE

Original: We re-do the experiments of Section G.2 with vanilla fully-connected discriminator nets.
Revised: We re-do the experiments of Section G.2 with vanilla fully-connected discriminator nets.

Original: We use a three-layer net with hidden dimensions 50-10, which has more parameters than the architecture with restricted approximability.
Revised: We use a three-layer net with hidden dimensions 50-10, which has more parameters than the architecture with restricted approximability.

Original: Results are plotted in Figure 5.
Revised: Results are plotted in Figure 5.

Original: We ﬁnd that the generators also converge well in the KL divergence, but the correlation is slightly weaker than the setting with restricted approximability (correlation still presents along each training run but weaker across different runs).
Revised: We ﬁnd that the generators also converge well in the KL divergence, but the correlation is slightly weaker than the setting with restricted approximability (correlation still presents along each training run but weaker across different runs)

Original: This suggests that vanilla discriminator structures might be practically quite satisfying for getting a good generator, though speciﬁc designs may help improve the quality of the distance W F. [Figure]
Revised: This suggests that vanilla discriminator structures might be practically quite satisfying for getting a good generator, though speciﬁc designs may help improve the quality of the distance W F

Original: P ERTURBED GENERATORS
Revised: P ERTURBED GENERATORS

Original: Correlation between KL and neural net IPM is computed with vanilla fully-connected discriminators and plotted in Figure 6.
Revised: Correlation between KL and neural net IPM is computed with vanilla fully-connected discriminators and plotted in Figure 6.

Original: The correlation (0.7489) is roughly the same as for discriminators with restricted approximability (0.7315).
Revised: The correlation (0.7489) is roughly the same as for discriminators with restricted approximability (0.7315).