Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Abstract
For Generative Adversarial Networks which map a latent distribution to the target distribution, in this paper, we study how the sampling in latent space can affect the generation performance, especially for images.
We observe that, as the neural generator is a continuous function, two close samples in latent space would be mapped into two nearby images, while their quality can differ much as the quality generally does not exhibit a continuous nature in pixel space.
From such a continuous mapping function perspective, it is also possible that two distant latent samples can be mapped into two close images (if not exactly the same).
In particular, if the latent samples are mapped in aggregation into a single mode, mode collapse occurs.
Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent z from its initial distribution, e.g., Gaussian.
This is achieved using well-developed adversarial sample mining techniques, e.g. iterative fast gradient sign method (I-FGSM).
We further propose new GAN training pipelines to obtain better generative mappings w.r.t quality and diversity by introducing targeted latent transforms into the bi-level optimization of
GAN.
Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity.
The implementation is publicly available at https://github.com/yangco-le/AdvLatGAN .
Introduction
Generative Adversarial Networks (GANs) [1] have shown effectiveness for generating high-fidelity data, especially for images under various settings [2; 3; 4; 5; 6].
Based on the zero-sum game, the model is trained by the adversarial process between the generator and the discriminator.
Many efforts have been made to achieve a more realistic generation from different perspectives.
For instance,
WGAN [7], SNGAN [8], LSGAN [9] aim to design better objective functions.
ImprovedGAN [10],
AC-GAN [11] propose practical training techniques.
While more complex network structures are studied in BigGAN [12], StyleGAN [13], SAGAN [14].
Despite the above progress, a relatively less studied question is how the latent space affects the generation quality and diversity, which can be orthogonal to the above frequently studied factors like model architectures and training techniques.
In this paper, we argue that the latent samples from a standard continuous distribution (e.g. Gaussian) can often be mapped to varying-quality samples for the generation.
One reason is that the generator is a continuous function (as neural nets)
[Figure]
generated distribution.
Due to the continuity of the networkbased generator, close samples in latent space will be mapped to close images in pixel space, while the quality can vary.
Cat (a) and Cat (b) are close in pixel space, but Cat (a) loses its right ear.
Our method transforms bad latent z towards the preimage of the nearest real manifold, thus avoiding terrible generation while the generation quality reflected by the matching degree to the natural image distribution in pixel space does not exhibit a continuous nature as two nearby images’ quality can differ much.
It has been shown in GAN literature [15] that image objects lie upon multiple disjoint manifolds.
A continuous mapping from a continuous latent distribution, e.g., Gaussian, may generate invalid samples in the between of the manifolds when the generator seeks to cover all the meaningful modes.
See Fig.
1 (a) for illustration.
This fact also implies a dilemma: for complex real-world data, there is a trade-off between quality and diversity when the latent vectors are sampled from a continuous uni-mode distribution.
On the other hand, to effectively capture the natural image distribution through a continuous generative mapping, it is necessary to maintain a disconnected latent space support.
The above contradiction between the discontinuity in generation data quality and the continuity of latent distribution as well as mapping function, can be one of the significant factors leading to the well-known training difficulty/instability of GANs, especially for generators, as one unreasonably hopes to enforce a discontinuous mapping via continuous neural nets.
On the other hand, the mapping is also critical to the performance of functions with continuous nature, as badly trained mapping can lead to poorly fitted generated distribution.
Meanwhile, by realizing that the mapping function allows for many-to-one mapping, mode collapse [16] can naturally happen when different latent samples are mapped into few or even a single mode in the target space.
See Fig.
2 for the schematic diagram.
To tackle these issues, we impose an extra (implicit) transform function z ∗ ( · ) on the raw sampling z , and then the generation can be written by G ( z ∗ ( z )) which requires us to make twofold efforts on both G ( · ) and z ∗ ( · ) .
From this perspective, most existing works employ z ∗ ( · ) as an identity function, while we try to find a more effective one.
We model the z transform process as making perturbations to the original sampling since z ∗ ( z ) shall not depart much from z as we hope the main content of the generated image remains the same.
The hope that small perturbations can achieve considerable positive quality variation leads us to the adversarial sample mining methods.
Specifically, the implicit function based transform z ∗ ( · ) can be achieved by updating the raw latent distribution samples z to minimize loss G i.e. log(1 − D ( G ( z ))) using adversarial sample mining techniques e.g. iterative fast gradient sign method (I-FGSM)
[17], and we name the method as AdvLatGAN-z. See Fig.
1 (b) for the schematic diagram.
We will show that our z transform (perturbations) can deliver a latent distribution better fitting the real distribution with the fixed G .
On top of our first effort, we propose a training strategy to improve the generative mapping G respectively for quality and diversity.
This is achieved by introducing z optimization (i.e., targeted implicit transforms on z ) into GAN training, specifically, updating raw latent distribution samples z using I-FGSM to find latent variables which benefit the optimization to calculate the loss during
GAN training.
Respectively for quality and diversity, we use two iterative updating strategies for z under different objectives, and we name the two training algorithms as AdvLatGAN-qua and
AdvLatGAN-div.
The highlights of this paper include: 1) We rethink the role of the generator as a continuous function, which may incur quality discontinuity in target space when the latent samples are directly sampled from a continuous distribution, together with poorly fitted generated distribution and mode aggregation (collapse) when the function suffers bad properties w.r.t.
quality or diversity.
We propose to formulate the mapping from G ( z ) to
G ( z ∗ ( z )) .
It provides new design guidance to GAN which requires efforts on both G ( · ) and z ∗ ( · ) .
2) By introducing adversarial techniques into GAN, we propose a sampling transform pipeline for z ∗ ( · ) and training pipelines for improving G ( · ) .
The generation results are obtained by first training generative mappings by our proposed training pipeline AdvLatGAN-qua/div.
Then the transform z ∗ ( · ) namely AdvLatGAN-z is enforced and computed implicitly to obtain an improved distribution for a more realistic generation.
The training algorithms AdvLatGAN-qua and -div involve latent sample transform under different objective functions for quality and diversity.
3) Experimental results on synthetic data and natural images including large-scale datasets e.g.
ImageNet show the notable improvements of our approach on both generation quality and diversity, whereby the performance gain is basically orthogonal to those from existing GAN techniques.
Related Work
Our latent vector transform pipeline mines effective latent samples and the resulting perturbations relate to adversarial methods.
The proposed GAN training pipelines include the aforementioned transform in network training, which can be viewed as mining adversarial samples in training, namely adversarial training.
Hence we briefly review existing works on latent space mining for GANs that modify latent space samples to achieve specific targets e.g. latent space sampling improvement for better generation and adversarial learning.
Also, techniques for avoiding mode collapse are studied.
Latent space mining in GANs.
Techniques have been devised for latent space mining for GANs.
By utilizing the steerability of GANs in latent space, StyleGAN [13] introduces a latent variable shift to better decouple the various styles by using a linear network (which is still continuous).
[19], [20] and [21] introduce the latent space mining to training to achieve clustering objectives or improve the dynamic of training.
[22] and [23] achieve conditional generation by performing gradient ascent in latent space to maximize classification neuron activations.
Some recent works focus on improving latent sampling quality to obtain high-quality images.
Interactive evolutionary computation is used to make the generation process more controllable in [24], which meanwhile also brings additional manual efforts.
To reduce manual intervention, [ 25; 26] use Koncept [27] as a criterion to improve the generation quality.
However, as discussed in [25], this strategy is somehow biased and can not work well for each dataset (e.g., Pokemons).
The reason is that the quality estimator is not even designated to access the real-data distribution information.
Instead, the generation is guided by a predefined score function.
Exploring the real distribution’s guidance from the discriminator, DOT [28] utilizes optimal transport in latent space to improve the initial sampled distribution, DRS [29] proposes a rejection sampling scheme to to improve sampling quality, and
DDLS [30] develops discriminator driven Langevin dynamics to obtain high-quality samples.
Mitigating mode collapse for GANs.
Efforts are made to mitigate mode collapse, which are based on the thinking that real-world samples lie in disjoint manifolds [15].
One way is to introduce a collection of generators such that each generator may cover a specific mode since a single generator can hardly fulfill a discontinuous function [15].
However, more works focus on improving mapping properties, explicitly obtaining a mapping covering more modes [31; 32; 33].
Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping.
This calls for the selective sampling of the continuous latent distribution or directly amending the latent distribution, which is rarely noticed in these works.
Adversarial samples and adversarial training.
Adversarial samples are manipulated samples crafted by adding indistinguishable perturbations to cause significant deep nets’ output variation e.g. wrong classification prediction.
Adversarial sample mining methods investigate how to manipulate such samples, which are known as adversarial attacks.
Since [34], various adversarial attack methods have been devised [35; 17; 36].
For defense, adversarial training is meanwhile developed [35; 37; 38].
In particular, recent studies have shown that adversarial training can bring many benefits to GANs, not just improving robustness.
RGAN [39] shows that adversarial training can enhance the generalization of the generator and discriminator.
Based on theoretical analysis and empirical results, [40; 41; 42] show that updating the discriminator’s parameters with the loss calculated by adversarial disturbed image samples can stabilize training and accelerate convergence.
In this paper, we use an off-the-shelf adversarial sample mining method iterative fast gradient sign method (I-FGSM)
[17], to perform latent transform and mine effective z samples that benefit
GAN training in its bi-level optimization process.
More adversarial methods e.g. PGD [36] and
MI-FGSM [43] are also tried and discussed in Appendix D. The pre-existing works mentioned above that interplay between GANs and adversarial learning make perturbations on raw pixel space (real samples or generated samples), while we instead work on mining the latent space and introducing perturbations on latent samples, which we show is a novel and effective way to improve the generation.
Proposed Method
This section presents our twofold effort to lift the final generative performance: AdvLatGAN-z to achieve the targeted transform z ∗ ( · ) ; training algorithms AdvLatGAN-qua and AdvLatGAN-div to train generation mapping G with better generation quality (i.e., -qua) and diversity (i.e., -div).
Both methods are derived by introducing the iterative updating in latent space.
3.1 Preliminaries
This section introduces adversarial attack/defense techniques and the regularization technique devised in MSGAN [32] for mode seeking.
Adversarial techniques include: i) adversarial samples which we introduce in the latent space sampling to achieve z ∗ ( · ) ; ii) adversarial training which we introduce in the bi-level optimization of GAN to develop new training algorithms for a more powerful G ( · ) .
Adversarial samples and adversarial training.
Adversarial training [36; 44] aims to defend against adversarial attacks.
It uses adversarial samples to calculate the loss.
In classification, for each image-label pair ( x i , y i ) from the labeled training set, the adversarial samples x ′ [34] is defined as:
[Equation]
Here ϵ is the radius of the closed ball to constrain perturbation.
While f θ and l denote the attack target (a neural net) and the corresponding loss (e.g. cross-entropy loss).
p denotes the norm dimension.
Given in total n adversarial samples, adversarial training is fulfilled by the optimization:
[Equation]
Mode coverage by regularizing distance of generated samples.
MSGAN [32] mitigates mode collapse for conditional generation by adding a regularizer over randomly sampled z 1 and z 2 :
[Equation]
where c is the condition vector for generation, and d
I and d z are the distance metrics in the target (image) space and latent space, respectively.
It encourages the distance to be maintained or even enlarged in the target space to avoid the generated samples being trapped into an aggregation.
By integrating the regularization term with the original objective of cGANs [45], the final minimiza- tion objective in MSGAN for training G and D is:
[Equation]
where L ori denotes the original objective, x denotes real data, and λ ms is the weight hyper-parameter.
Since Eq. 3 is to be maximized hence one can interpret λ ms as a negative coefficient.
3.2 Latent Adversarial Mining for Generation Quality 3.2.1 AdvLatGAN-z: Latent Space Transform
Note that as the real-data space is a union of disconnected manifolds [15], via a continuous generative mapping G , the optimal subset in the latent space w.r.t the fixed G is in no way connected.
We denote the real-data subset in the high-dimensional pixel space X as X r ⊂ X , and each disjoint submanifold as M i .
Then we obtain X r =
(cid:83) n r i =
1 M i where M 1 to M n r are disconnected to each other.
Here we require the splitting of X r to the extent that each submanifold is itself a connected set.
Definition 3.1.
Suppose G [ · ] : P ( Z ) → P ( X ) is the image-of-set function corresponding to the generative mapping function G ( · ) : Z → X , where Z and X denote subsets in latent space and pixel space, and P ( · ) denotes the power set.
Then Z op ( G ) is the optimal subset corresponding to G in latent space if G [ Z op ( G )] =
X r .
Proposition 3.2.
For any given continuous G [ · ] whose codomain includes multiple disconnected manifolds, Z op ( G ) is a union of disconnected subsets in the latent space.
The proof is given in Appendix B. Proposition 3.2 implies that with the fixed continuous G , fitting the real-data manifolds requires a disconnected latent space support.
Below we show how to implicitly transform the latent raw samples from Gaussian (or other forms e.g. uniform) to a new distribution closer to Z op ( G ) via the idea of mining adversarial samples as introduced above.
We utilize the discriminator’s output to quantify the generation quality, which is trained to capture the difference between the generated and real distribution.
Considering that small perturbations on z can cause considerable quality variation (technically D ’s output) as shown in Fig.
1 and Fig.
3, we naturally resort to adversarial perturbation-based methods to achieve the transform.
Guided by Eq.
1, we first deliver the constrained optimization below, where both G and D are post-trained.
[Equation]
(5) where z 0 denotes the original latent vector, z ∗ ( z 0 ) denotes the newly computed z , d ( · , · ) denotes the distance which is ℓ ∞ in this paper, in line with the popular protocol in literature [35].
The above target can be readily solved by the well-developed tools in adversarial sample mining, e.g. the classic iterative fast gradient sign method (I-FGSM)
[17] as formulated below in Eq. 6.
In Appendix D, we also present comparison of more adversarial techniques e.g. PGD [36], MIFGSM [43], that can be seamlessly incorporated by our framework, and we show the simple I-FGSM outperforms.
We conjecture the reason may be due to the simplicity of the latent space in our case.
[Equation]
From the distribution perspective
[Equation]
the transform is given by:
[Equation]
where p
z denotes the original latent distribution, p ∗ z denotes the newly searched one.
D ( p z , p 0 z ) ≤ ϵ means that there is a limit to the distribution variation.
A newly mined z can be considered as a sample from p ∗ z if the sampling is within the perturbation circle.
Fig.
4 shows the results by the pre-trained SNGAN [8] (i.e. the given G and D ) on STL-10 [46].
We then show that Eq.
7 guarantees the correct optimization direction that optimizes the source distribution p z towards the optimal probability distribution in latent space according to the current fixed G , i.e. p opz ( G ) defined in Definition.
The proof for Proposition 3.4 is given in Appendix B.
Definition 3.3.
Given G , p opz ( G ) is the optimal probability distribution in latent space corresponding to Z op ( G ) when p opz ( G ) satisfies that if z ∼ p opz ( G ) then G ( z ) ∼ p r .
Here p r denotes the real-data probability distribution in pixel space.
Proposition 3.4.
Optimizing GAN’s training criterion i.e. min G max D { E z ∼ p
z [log(1 − D ( G ( z )))]
+
E x ∼ p r [log( D ( x ))] } is to minimize JSD ( p opz ( G ) , p 0 z ) .
Prop. 3.4 implies that the GAN’s optimization is to minimize the difference between p opz ( G ) and p 0 z .
Note that for optimizing p z , the criterion merely involves E z ∼ p
z [log(1 − D ( G ( z )))] i.e. the objective in Eq.
Given optimized fixed G and D (and p opz is thereby fixed), leaving p z the only trainable parameter, optimizing the training criterion with p z in the neighborhood of p
z is driving p z closer to p opz .
Note that p z in the neighborhood of p
z is necessary otherwise the optimized criterion will depart much from JSD , and we achieve this condition by enforcing D ( p z , p 0 z ) ≤ ϵ in Eq.
3.2.2 AdvLatGAN-qua: Quality Targeted Training Pipeline
Considering the pairwise approach for adversarial sample mining, i.e. adversarial training fulfilled by
Eq. 2, we hope to mine samples that benefit the optimization in GAN training, which is achieved by utilizing the aforementioned latent space transform.
Conducting the transform in training to obtain the new (probably disconnected) latent distribution for each iteration can compensate for the difficulty for generators to align the pace of generators and discriminators, as it is widely recognized that it is much more challenging to train the generator than the discriminator, for its inherent defect that it cannot map a continuous distribution to a disconnected one with multiple modes.
Specifically, the transform is conducted to the raw Gaussian sampling in D updating iterations while the rest of the training algorithm remains consistent with vanilla GAN [1] and other variants with the orthogonal efforts to latent space mining.
We present the algorithm in Alg. 1 in the appendix which we call Adversarial Latent GAN with a post meaning quality (AdvLatGAN-qua).
The latent transform is first computed by I-FGSM to mine the targeted samples in latent space, then D and G are updated.
The iteration continues until enough steps or the convergence of G .
The above latent transform model does not consider how to avoid mode collapse, as each time only one latent sample is considered for optimization in isolation.
A similar constrained optimization formulation can be devised to improve generation diversity, as shown in the following subsection.
3.3 AdvLatGAN-div: Latent Adversarial Mining for Diversity
In this section, we tend to modify the regularizer for mode seeking in MSGAN [32] by substituting the randomly selected regularized z pairs with hard sample pairs that are more inclined to collapse, lifting the efficiency for avoiding mode collapse.
We first show how to search for such hard sample pairs.
Given a sample z 0 from the raw latent distribution, we aim to search its paired sample z ∗ to form a hard sample pair, in the sense that their generations are close in the target space given G .
We consider two aspects.
First, as hard samples, the paired samples shall still belong to the same category c which can be modeled with conditional GAN.
Second, their distance in the latent space shall also be not too close, otherwise they may be aggregated in the target space due to continuous mapping.
In other words, hard samples causing mode collapse shall be those close in target space while apart in latent space, when G is given.
Hence, guided by Eq.
1, we design our search procedure as the form:
[Equation]
[Table] [Table]
The symbols and terms are similar to Eq.
Akin to
AdvLatGAN-qua, we use I-FGSM to obtain z ∗ and again use ℓ ∞ to control deviation magnitude.
For each update step, we randomly select the first latent vector z 0 , and the second vector z ∗ is searched by n iterations of the I-FGSM method under the ℓ ∞ restriction starting from the Gaussian neighborhood of zwith a small standard deviation.
Table 1 shows the effectiveness of the above paired hard sample seeking method, where the results are calculated over 5k generated images with DCGAN [47] as backbone.
The visual results on Facades as shown in Fig.
6 suggest that the pair obtained by solving Eq.
8 tend to collapse (as they are hard samples), while the opposite leads to more different generation by using the inverse form of Eq. 8.
The above experiments verify the efficacy of our mining technique.
We further introduce it into the initial training process of MSGAN [32].
In MSGAN, the regularization term by Eq.
3 regularizes the optimization process, where the selection of the pair is entirely random, without using any heuristic information.
We instead use the z pair that is more inclined to collapse by randomly taking one z and iterating by the aforementioned method to get another z .
The other settings remain unchanged.
By using sample pairs mined by Eq. 8 in the optimization as inspired by Eq.
2, we finally develop our new diversity enhancing GAN training approach AdvLatGAN-div as shown in Alg. 2 in the appendix, which can be further enhanced by post-training as discussed in experiments.
Experiments
Experiments on ImageNet and CelebA are performed on GPUs of Tesla V100.
Other public benchmark results are performed on a single GPU of GeForce RTX 3090.
[Table]
4.1 Experimental Setup
We validate the effectiveness of the proposed methods for two parts,
i
[Table]
e., effective latent sampling improvement z ∗ and improved generation mapping G .
Five variant methods can be derived for different targets, among which AdvLatGAN-qua+ and AdvLatGAN-div+ are our final full version.
i)
AdvLatGAN-z: post-training latent sampling improvement fighting against quality discontinuity; ii) AdvLatGAN-qua: GAN training algorithm for better quality using in-training latent sampling transform; iii) AdvLatGAN-div: GAN training algorithm for
a more diverse generation by using in-training latent sampling transform; iv) AdvLatGAN-qua+: conducting -z over the generation results of -qua; v) AdvLatGAN-div+: conducting -div over the generation results of -div.
We compare these methods in Fig.
7 and Table 2.
We adopt the Inception Score, IS [48], Fréchet Inception Distance, FID [49] and density/coverage [50] for evaluation.
IS utilizes the classification results of the generated images by the Inception Network to evaluate the generation quality.
FID additionally considers the target distribution and evaluates the quality by fitting the distribution distance.
Density and coverage are more recently proposed evaluation metrics that analyze the target and generated manifolds and estimate the similarity.
In the following experiments, IS and FID are calculated over 50,000 images, while density and coverage are calculated over 10,000 images.
Overhead discussion is presented in Appendix C.
4.2 AdvLatGAN-z: Performance Boosting by Post-training Latent Sampling
We evaluate the boosting effect of post-training latent sampling improvement method AdvLatGAN-z (also previously denoted as z ∗ ), which is guided by Eq.
It aims to mitigate quality discontinuity.
Results on Synthetic Data.
We test AdvLatGAN-z on Grid and Ring datasets consisting of a mixture of 25 and 8 2-D Gaussians as done in [51].
The result is given in Fig.
By updating the latent variables, our method achieves high-quality generation (samples closer to target Gaussian centers).
Results on MNIST.
Low complexity of MNIST [52] makes it possible to generate recognizable results by two-dimensional latent variables which can be better visualized.
The iterative latent distribution change is illustrated in Fig.
We respectively conduct AdvLatGAN-z for the trained
DCGAN and WGAN-GP models, with single-step iteration constraint using ℓ ∞ and iterated step size ϵ is set as 0.03.
We present the first 30 iterations that can reflect the most significant changes.
Results on STL-10.
STL-10 [46] maintains a higher resolution and richer image representation.
We utilize AdvLatGAN-z for pre-trained DCGAN, WGAN-GP and SNGAN, with ℓ ∞ single-step constraint and the iteration step size ϵ is set as 0.05.
We conduct 20 steps each time.
IS and FID are adopted for evaluation, which are calculated over 10,000 generated images.
We compare other efforts enhancing GAN performance by latent space mining including DOT, DDLS and EvolGAN.
Tablepresent the results.
AdvLatGAN-z outperforms the baselines and delivers at most 24.3% advantage for FID on vanilla in the WGAN-GP setting.
[Figure]
Results on AFHQ and FFHQ.
To address the issues in
Fig.
3, we evaluate on AFHQ [54] and FFHQ [55] based on
StyleGAN2-ada [18].
In Fig.
3 setting, when the sampling encounters invalid examples, AdvLatGAN-z can transform the bad z to generation-friendly z ∗ ( z ) to avoid bad generation.
Here we verify whether invalid samples can be improved with AdvLatGAN-z. We select six bad generations with the first three corresponding to those in Fig.
3 and conduct AdvLatGAN-z to their latent vectors.
Fig.
10 shows that AdvLatGAN-z can effectively fix or avoid defects.
[Table]
with the defects as
[Table]
Hyper-parameter Study.
Considering the influence of the step size ϵ and the number of iterations
[Table]
we evaluate WGAN-GP’s performance on STL-10 adopting IS and FID metrics.
Fig.
8 shows the generative performance after the iteration number.
We observe that a larger ϵ (e.g. 0.05) can deliver fast improvement but can be unstable for more iterations, while a smaller ϵ can achieve stable improvement but is slightly less significant.
If time permits for sufficient parameter tuning, we recommend adopting a larger ϵ like 0.05 along with a particularly selected iteration number, otherwise a smaller ϵ like 0.01 with sufficient iterations will be enough for worthy performance gain.
4.3 AdvLatGAN-qua/div: Improving Generation Map
This section shows the effectiveness of our training pipelines, which aim to improve the generation mapping G respectively for quality and diversity.
The details are presented in Alg.1 and Alg.2.
AdvLatGAN-qua for Quality Improvement.
Table 4 shows the results on Cifar-10 and STL-10, using the mainstream architectures: DCGAN, WGAN, WGAN-GP, SNGAN, LSGAN, WGANdiv and ACGAN.
We do not include ACGAN in the unlabeled STL-10 setting because it requires labels [58].
IS and FID are adopted.
We adversarially train GAN using AdvLatGAN-qua.
The
[Table]
number of latent iterations in training is set as one per discriminator step, and other details is given in
Appendix J.1.
As they are orthogonal to each other, we integrate AdvLatGAN-z with AdvLatGANqua (i.e. AdvLatGAN-qua+) and include it in comparison.
The iteration step size ϵ of AdvLatGAN-z is set to 0.05 and we conduct 20 steps each time.
Our method achieves a notable improvement compared to that without the latent space transform strategy.
AdvLatGAN-qua+ achieves the best gain on IS in WGAN STL-10 setting from 6.51 to 8.57 and FID in WGAN-div from 37.7 to 23.0.
Table.
5 presents results on larger datasets: LSUN Church [59], CelebA [60] and ImageNet [61], using WGAN-GP and SNGAN as backbones, evaluated by FID, density and coverage.
We include
AdvLatGAN-qua and -qua+ in the comparison.
The number of latent iterations during training is set as 1 per discriminator step, and other details are given in Appendix J.1.
The iteration step size ϵ of AdvLatGAN-z is set to 0.01 for SNGAN and 0.002 for WGANGP
and we conduct 100 steps each time.
AdvLatGAN-qua+ has achieved the best performance gain on FID in SNGAN LSUNsetting from 11.961 to 7.285.
AdvLatGAN-div for Diversity Improvement.
We test on Cifar-10 and STL-10 for conditional generation and we adopt FID, density and coverage as the metrics, respectively for the entire set of generated images and for those in each label.
We test AdvLatGAN-div and AdvLatGAN-div+ compared to the baseline MSGAN.
The results are shown in Table 6.
Our method outperforms
MSGAN by all the three metrics.
Note that the models are trained on the overall set and for each label and the optimization may not guarantee targeted enough guidance.
AdvLatGAN-div+ achieves the best overall performance gain on FID from 30.225 to 26.169, density from 0.353 to 0.431 and coverage from 0.455 to 0.54.
Conclusion and Broader Impact
This work analyzes GANs from the continuous mapping perspective and notes that lifting the overall generative performance requires a twofold effort including latent distribution transform and mapping improvement.
Adversarial sample mining techniques are introduced to explore latent space and novel training pipelines are derived to improve the generative mapping.
The proposed AdvLatGAN has shown promising power in both quality and diversity.
Our technique is basically orthogonal to existing mainstream methods.
For its limitation, it remains open for how to effectively combine the proposed two techniques for diversity and quality.
For potential negative social impact, fake content by the generation models including ours shall always be carefully treated to avoid abuse.
Acknowledgements
This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Shanghai Municipal Science and Technology (Major) Project (2021SHZDZX0102,22511105100).