Original: maximizes the contrast and we regard the f found by stochastic optimization an approximate maximizer, and the loss obtained an approximation of W F .
Revised: the contrast and we regard the f found by stochastic optimization an approximate maximizer, and the loss obtained an approximation of W F .

Original: Our theory shows that for our choice of G and F , WGAN is able to learn the true generator in
Revised: Our theory shows that for our choice of G and F , WGAN is able to learn the true generator in

Original: KL divergence, and the F -IPM (in evaluation instead of training) should be indicative of the KL divergence.
Revised: KL divergence, and the F -IPM (in evaluation instead of training) should be indicative of the KL divergence.

Original: We test this hypothesis in the following experiments.
Revised: We test this hypothesis in the following experiments.

Original: G. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE
Revised: G. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE

Original: In our ﬁrst experiment, G is a two-layer net in d = 10 dimensions.
Revised: In our ﬁrst experiment, G is a two-layer net in d = 10 dimensions.

Original: Though the generator is only a shallow neural net, the presence of the nonlinearity makes the estimation problem non-trivial.
Revised: Though the generator is only a shallow neural net, the presence of the nonlinearity makes the estimation problem non-trivial.

Original: We train a discriminator with the architecture speciﬁed in Lemma 4.1), using either Vanilla WGAN (clamping the weight into norm balls) or WGAN-GP (Gulrajani et al., 2017) (adding a gradient penalty).
Revised: We train a discriminator with the architecture speciﬁed in Lemma 4.1), using either Vanilla WGAN (clamping the weight into norm balls) or WGAN-GP (Gulrajani et al., 2017) (adding a gradient penalty).

Original: We ﬁx the same ground-truth generator and run each method from 6 different random initializations.
Revised: We ﬁx the same ground-truth generator and run each method from 6 different random initializations.

Original: Results are plotted in Figure 3.
Revised: Results are plotted in Figure 3.

Original: Our main ﬁndings are two-fold:
Revised: Our main ﬁndings are two-fold:

Original: (1) WGAN training with discriminator design of restricted approximability is able to learn the true distribution in KL divergence.
Revised: (1) WGAN training with discriminator design of restricted approximability is able to learn the true distribution in KL divergence.

Original: Indeed, the KL divergence starts at around 10 - 30 and the best run gets to KL lower than 1.
Revised: Indeed, the KL divergence starts at around 10 - 30 and the best run gets to KL lower than 1.

Original: As KL is a rather strong metric between distributions, this is strong evidence that GANs are ﬁnding the true distribution and mode collapse is not happening.
Revised: As KL is a rather strong metric between distributions, this is strong evidence that GANs are ﬁnding the true distribution and mode collapse is not happening.

Original: (2) The W F (eval) and the KL divergence are highly correlated with each other, both along each training run and across different runs.
Revised: (2) The W F (eval) and the KL divergence are highly correlated with each other, both along each training run and across different runs.

Original: In particular, adding gradient penalty improves the optimization signiﬁcantly (which we see in the KL curve), and this improvement is also reﬂected by the W F curve.
Revised: In particular, adding gradient penalty improves the optimization signiﬁcantly (which we see in the KL curve), and this improvement is also reﬂected by the W F curve.

Original: Therefore the quantity W F can serve as a good metric for monitoring convergence and is at least much better than the training loss curve.
Revised: Therefore the quantity W F can serve as a good metric for monitoring convergence and is at least much better than the training loss curve.

Original: To test the necessity of the speciﬁc form of the discriminator we designed, we re-do the same experiment with vanilla fully-connected discriminator nets.
Revised: To test the necessity of the speciﬁc form of the discriminator we designed, we re-do the same experiment with vanilla fully-connected discriminator nets.

Original: Results (in Appendix G.4) show that IPM with vanilla discriminators also correlate well with the KL-divergence.
Revised: Results (in Appendix G.4) show that IPM with vanilla discriminators also correlate well with the KL-divergence.

Original: This is not surprising from a theoretical point of view because a standard fully-connected discriminator net (with some overparameterization) is likely to be able to approximate the log density of the generator distributions (which is essentially the only requirement of Lemma 4.3.)
Revised: This is not surprising from a theoretical point of view because a standard fully-connected discriminator net (with some overparameterization) is likely to be able to approximate the log density of the generator distributions (which is essentially the only requirement of Lemma 4.3.)

Original: For this synthetic case, we can see that the inferior performance in KL of the WGAN-Vanilla algorithm doesn’t come from the statistical properties of GANs, but rather the inferior training performance in terms of the convergence of the IPM.
Revised: For this synthetic case, we can see that the inferior performance in KL of the WGAN-Vanilla algorithm doesn’t come from the statistical properties of GANs, but rather the inferior training performance in terms of the convergence of the IPM.

Original: We conjecture similar phenomenon occurs in training
Revised: We conjecture similar phenomenon occurs in training

Original: GANs with real-life data as well.
Revised: GANs with real-life data as well.

Original: G. P ERTURBED GENERATORS
Revised: G. P ERTURBED GENERATORS

Original: In this section, we remove the effect of the optimization and directly test the correlation between p and its perturbations.
Revised: In this section, we remove the effect of the optimization and directly test the correlation between p and its perturbations.

Original: We compare the KL divergence and neural net IPM on pairs of perturbed generators .
Revised: We compare the KL divergence and neural net IPM on pairs of perturbed generators .

Original: In each instance, we generate a pair of generators ( G, G (cid:48) )
Revised: In each instance, we generate a pair of generators ( G, G (cid:48) )

Original: (with the same architecture as above), where G (cid:48) is a perturbation of G by adding small Gaussian noise.
Revised: (with the same architecture as above), where G (cid:48) is a perturbation of G by adding small Gaussian noise.

Original: We compute the KL divergence and the neural net IPM between G and G (cid:48) .
Revised: We compute the KL divergence and the neural net IPM between G and G (cid:48) .

Original: To denoise the unstable training process for computing the neural net IPM, we optimize the discriminator from 5 random initializations and pick the largest value as the output.
Revised: To denoise the unstable training process for computing the neural net IPM, we optimize the discriminator from 5 random initializations and pick the largest value as the output.

Original: As is shown in Figure 4, there is a clear positive correlation between the (symmetric) KL divergence and the neural net IPM.
Revised: As is shown in Figure 4, there is a clear positive correlation between the (symmetric) KL divergence and the neural net IPM.

Original: In particular, majority of the points fall around the line W F = 100 D kl , which is consistent with our theory that the neural net distance scales linearly in the KL divergence.
Revised: In particular, majority of the points fall around the line W F = 100 D kl , which is consistent with our theory that the neural net distance scales linearly in the KL divergence.

Original: Note that there are a few outliers with large KL.
Revised: Note that there are a few outliers with large KL.

Original: This happens mostly due to the perturbation being accidentally too large so that the weight matrices become poorly conditioned – in the context of our theory, they fall out of the good constraint set as deﬁned in Assumption 1.
Revised: This happens mostly due to the perturbation being accidentally too large so that the weight matrices become poorly conditioned – in the context of our theory, they fall out of the good constraint set as deﬁned in Assumption 1.

Original: G. E XPERIMENTS WITH VANILLA DISCRIMINATOR G.4. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE
Revised: G. E XPERIMENTS WITH VANILLA DISCRIMINATOR G.4. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE

Original: We re-do the experiments of Section G.2 with vanilla fully-connected discriminator nets.
Revised: We re-do the experiments of Section G.2 with vanilla fully-connected discriminator nets.

Original: We use a three-layer net with hidden dimensions 50-10, which has more parameters than the architecture with restricted approximability.
Revised: We use a three-layer net with hidden dimensions 50-10, which has more parameters than the architecture with restricted approximability.

Original: Results are plotted in Figure 5.
Revised: Results are plotted in Figure 5.

Original: We ﬁnd that the generators also converge well in the KL divergence, but the correlation is slightly weaker than the setting with restricted approximability (correlation still presents along each training run but weaker across different runs)
Revised: We ﬁnd that the generators also converge well in the KL divergence, but the correlation is slightly weaker than the setting with restricted approximability (correlation still presents along each training run but weaker across different runs)

Original: This suggests that vanilla discriminator structures might be practically quite satisfying for getting a good generator, though speciﬁc designs may help improve the quality of the distance W F 
Revised: This suggests that vanilla discriminator structures might be practically quite satisfying for getting a good generator, though speciﬁc designs may help improve the quality of the distance W F G

Original:  P ERTURBED GENERATORS
Revised:  P ERTURBED GENERATORS

Original: Correlation between KL and neural net IPM is computed with vanilla fully-connected discriminators and plotted in Figure 6.
Revised: Correlation between KL and neural net IPM is computed with vanilla fully-connected discriminators and plotted in Figure 6.

Original: The correlation (0.7489) is roughly the same as for discriminators with restricted approximability (0.7315).
Revised: The correlation (0.7489) is roughly the same as for discriminators with restricted approximability (0.7315).