Paper:
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
Deep generative models provide powerful tools for distributions over complicated manifolds, such as those of natural images. But many of these methods, including generative adversarial networks (GANs), can be difficult to train, in part because they are prone to mode collapse, which means that they characterize only a few modes of the true distribution. To address this, we introduce VEEGAN, which features a reconstructor network, reversing the action of the generator by mapping from data to noise. Our training objective retains the original asymptotic consistency guarantee of GANs, and can be interpreted as a novel autoencoder loss over the noise. In sharp contrast to a traditional autoencoder over data points, VEEGAN does not require specifying a loss function over the data, but rather only over the representations, which are standard normal by assumption. On an extensive set of synthetic and real world image datasets, VEEGAN indeed resists mode collapsing to a far greater extent than other recent GAN variants, and produces more realistic samples.

Review:
This paper proposes a variation of the GAN algorithm aimed at improving the mode coverage of the generative distribution.
The main idea is to use three networks: a generator, an encoder and a discriminator where the discriminator takes pairs (x,z) as input.
The criterion to optimize is a GAN style criterion for the discriminator and a reconstruction in latent space for the encoder (unlike
auto-encoders where the reconstruction is in input space).
The proposed criterion is properly motivated and experimental evidence is provided to support the claim that
the new criterion allows to produce good quality samples (similarly to other GAN approaches) but with more diversity and coverage
of the target distribution (i.e. more modes are covered) than other approaches.
The results are fairly convincing and the algorithm is evaluated with several of the metrics used in previous literature on missing modes in generative models.
However, I believe that the paper could be improved on two aspects:
- the criterion is only partially justified by the proposed derivation starting from the cross entropy in Z space and there is an extra distance
term that is added without clear justification (it's ok to add terms, but making it sound like it falls out of the derivation is not great)
- the connection to other existing approaches, and in particular ALI could be further detailed and the differences could be better explained.
More specifically, regarding section 3.1 and the derivation of the objective, I think the exposition is somewhat obfuscating what's really going on:
Indeed, I would suggest to rewrite (3) as the following simple lemma:
H(Z,F_\theta(X)) <= KL(q_\gamma(x|z)p_0(z) \| p_\theta(z|x)p(x)) - E log p_0(z)
which directly follows from combining (1) and (2)
And then the authors could argue that they add this extra expected distance term (third term of (3)).
But really, this term is not justified by any of the derivation. However, it is important to notice that this term vanishes exactly
when the KL term is minimized since the minimum of the KL term is reached when q_\gamma inverses p_\theta i.e. when F_\theta
is the inverse of G_\gamma in which case z=F_\theta(G_\gamma(z)).
There is also the question of whether this term is necessary.
Of course, it helps training the encoder to invert the decoder but it seems possible to optimize only the first part of (5) with
respect to both theta and gamma
Regarding the connection to ALI: the discriminator is similar to the one in ALI and trained with the same objective. However what is different
is the way the discriminator is used when training the generator and "encoder" networks. Indeed, in ALI they are trained by optimizing
the JS divergence, while here the generator is trained by optimizing the (estimate of) KL divergence plus the additional reconstruction term and the encoder is purely trained using the reconstruction term.
It would be interesting to see what is the effect of varying the weight of the reconstruction term. The authors say that there is no need
to add a weight (indeed, the reconstruction term is zero when the KL term is zero as discussed above), but that doesn't mean it cannot have
an effect in practice.
Side note: there is inconsistency in the notations as d is a distance in latent space, so d(z^i, x_g^i) is not correct and should be
replaced by d(z^i, F_\theta(x_g^i)) everywhere.
Overall, I feel that this paper presents an interesting new algorithm for training a generative model and the experiments are showing that
it does have the claimed effect of improving the mode coverage of other comparable algorithms, but the derivation and justification of the
algorithm as well as the comparison to other algorithms could be improved in order to provide better intuition of why this would work.