Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Anonymous Author(s)
AffiliationAddress email
Abstract
For Generative Adversarial Networks which map a latent distribution to the targetdistribution, in this paper, we study how the sampling in latent space can affectthe generation performance, especially for images.
We observe that, as the neural generator is a continuous function, two close samples in latent space wouldbe mapped into two nearby images, while their quality can differ much as thequality generally does not exhibit a continuous nature in pixel space.
From sucha continuous mapping function perspective, it is also possible that two distantlatent samples can be mapped into two close images (if not exactly the same).
In particular, if the latent samples are mapped in aggregation into a single mode,mode collapse occurs.
Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent z from its initial distribution,e.g., Gaussian.
This is achieved using well-developed adversarial sample miningtechniques, e.g. iterative fast gradient sign method (I-FGSM).
We further proposenew GAN training pipelines to obtain better generative mappings w.r.t quality anddiversity by introducing targeted latent transforms into the bi-level optimization of GAN.
Experimental results on visual data show that our method can effectivelyachieve improvement in both quality and diversity.
Introduction
Generative Adversarial Networks (GANs) [1] have shown effectiveness for generating high-fidelitydata, especially for images under various settings [2; 3; 4; 5; 6].
Based on the zero-sum game, themodel is trained by the adversarial process between the generator and the discriminator.
Many effortshave been made to achieve a more realistic generation from different perspectives.
For instance,
WGAN [7], SNGAN [8], LSGAN [9] aim to design better objective functions.
ImprovedGAN [10],
AC-GAN [11] propose practical training techniques.
While more complex network structures arestudied in BigGAN [12], StyleGAN [13], SAGAN [14].
Despite the above progress, a relatively less studied question is how the latent space affects thegeneration quality and diversity, which can be orthogonal to the above frequently studied factorslike model architectures and training techniques.
In this paper, we argue that the latent samplesfrom a standard continuous distribution (e.g. Gaussian) can often be mapped to varying-qualitysamples for the generation.
One reason is that the generator is a continuous function (as neural nets)while the generation quality reflected by the matching degree to the natural image distribution inpixel space does not exhibit a continuous nature as two nearby images’ quality can differ much.
It has been shown in GAN literature [15] that image objects lie upon multiple disjoint manifolds.
A continuous mapping from a continuous latent distribution, e.g., Gaussian, may generate invalidsamples in the between of the manifolds when the generator seeks to cover all the meaningful modes.
See Fig.
1 (a) for illustration.
This fact also implies a dilemma: for complex real-world data, thereis a trade-off between quality and diversity when the latent vectors are sampled from a continuous
[Figure]
uni-mode distribution.
On the other hand, to effectively capture the natural image distribution througha continuous generative mapping, it is necessary to maintain a disconnected latent space support.
The above contradiction between the discontinuity in generation data quality and the continuity oflatent distribution as well as mapping function, can be one of the significant factors leading to thewell-known training difficulty/instability of GANs, especially for generators, as one unreasonablyhopes to enforce a discontinuous mapping via continuous neural nets.
On the other hand, the mappingis also critical to the performance of functions with continuous nature, as badly trained mapping canlead to poorly fitted generated distribution.
Meanwhile, by realizing that the mapping function allowsfor many-to-one mapping, mode collapse [16] can naturally happen when different latent samples aremapped into few or even a single mode in the target space. See Fig.
2 for the schematic diagram. To tackle these issues we impose an extra (implicit) transform function z ∗ ( · ) on the raw sampling z ,and then the generation can be written by G ( z ∗ ( z )) which requires us to make twofold efforts onboth G ( · ) and z ∗ ( · ) .
From this perspective, most existing works employ z ∗ ( · ) as an identity function,while we try to find a more effective one.
We model the z transform process as making perturbationsto the original sampling since z ∗ ( z ) shall not depart much from z as we hope the main content ofthe generated image remains the same.
The hope that small perturbations can achieve considerablepositive quality variation leads us to the adversarial sample mining methods.
Specifically, the implicitfunction based transform z ∗ ( · ) can be achieved by updating the raw latent distribution samples z tominimize loss G i.e. log(1 − D ( G ( z ))) using adversarial sample mining techniques e.g. iterative fastgradient sign method (I-FGSM)
[17], and we name the method as AdvLatGAN-z. See Fig.
1 (b)for the schematic diagram.
We will show that our z transform (perturbations) can deliver a latentdistribution better fitting the real distribution with the fixed G .
On top of our first effort, we propose a training strategy to improve the generative mapping Grespectively for quality and diversity.
This is achieved by introducing z optimization (i.e., targetedimplicit transforms on z ) into GAN training, specifically, updating raw latent distribution samplesz using I-FGSM to find latent variables which benefit the optimization to calculate the loss during GAN training.
Respectively for quality and diversity, we use two iterative updating strategies forz under different objectives, and we name the two training algorithms as AdvLatGAN-qua and
AdvLatGAN-div.
The highlights of this paper include:
We rethink the role of the generator as a continuous function, which may incur quality discontinuityin target space when the latent samples are directly sampled from a continuous distribution, together with poorly fitted generated distribution and mode aggregation (collapse) when the function suffersbad properties w.r.t.
quality or diversity.
We propose to formulate the mapping from G ( z ) to G ( z ∗ ( z )) .
It provides new design guidance to GAN which requires efforts on both G ( · ) and z ∗ ( · ) .2)
By introducing adversarial techniques into GAN, we propose a sampling transform pipeline forz ∗ ( · ) and training pipelines for improving G ( · ) . The generation results are obtained by first traininggenerative mappings by our proposed training pipeline AdvLatGAN-qua/div.
Then the transform z ∗ ( · ) namely AdvLatGAN-z is enforced and computed implicitly to obtain an improved distributionfor a more realistic generation.
The training algorithms AdvLatGAN-qua and -div involve latentsample transform under different objective functions for quality and diversity.3)
Experimental results on synthetic data and natural images including large-scale datasets e.g.
ImageNet show the notable improvements of our approach on both generation quality and diversity,whereby the performance gain is basically orthogonal to those from existing GAN techniques.
Related Work
Our latent vector transform pipeline mines effective latent samples and the resulting perturbationsrelate to adversarial methods.
The proposed GAN training pipelines include the aforementionedtransform in network training, which can be viewed as mining adversarial samples in training, namelyadversarial training.
Hence we briefly review existing works on latent space mining for GANs thatmodify latent space samples to achieve specific targets e.g. latent space sampling improvement forbetter generation and adversarial learning.
Also, techniques for avoiding mode collapse are studied.
Latent space mining in GANs.
Techniques have been devised for latent space mining for GANs.
Byutilizing the steerability of GANs in latent space, StyleGAN [13] introduces a latent variable shift tobetter decouple the various styles by using a linear network (which is still continuous).
[19], [20]and [21] introduce the latent space mining to training to achieve clustering objectives or improve thedynamic of training.
[22] and [23] achieve conditional generation by performing gradient ascent inlatent space to maximize classification neuron activations.
Some recent works focus on improvinglatent sampling quality to obtain high-quality images.
Interactive evolutionary computation is usedto make the generation process more controllable in [24], which meanwhile also brings additionalmanual efforts.
To reduce manual intervention, [25; 26] use Koncept512 [27] as a criterion toimprove the generation quality.
However, as discussed in [25], this strategy is somehow biased andcan not work well for each dataset (e.g., Pokemons).
The reason is that the quality estimator is noteven designated to access the real-data distribution information.
Instead, the generation is guidedby a predefined score function.
Exploring the real distribution’s guidance from the discriminator, DOT [28] utilizes optimal transport in latent space to improve the initial sampled distribution, and DDLS [29] develops discriminator driven Langevin dynamics to obtain high-quality samples.
Mitigating mode collapse for GANs.
Efforts are made to mitigate mode collapse, which are basedon the thinking that real-world samples lie in disjoint manifolds [15].
One way is to introduce acollection of generators such that each generator may cover a specific mode since a single generatorcan hardly fulfill a discontinuous function [15].
However, more works focus on improving mappingproperties, explicitly obtaining a mapping covering more modes [30; 31; 32].
Note that the generateddistribution obtained by this method will contain invalid parts due to the continuity of the latentdistribution and the mapping.
This calls for the selective sampling of the continuous latent distributionor directly amending the latent distribution, which is rarely noticed in these works.
Adversarial samples and adversarial training.
Adversarial samples are manipulated samplescrafted by adding indistinguishable perturbations to cause significant deep nets’ output variation e.g.wrong classification prediction.
Adversarial sample mining methods investigate how to manipulate such samples, which are known as adversarial attacks.
Since [33], various adversarial attack methodshave been devised [34; 17; 35].
For defense, adversarial training is meanwhile developed [34; 36; 37].
In particular, recent studies have shown that adversarial training can bring many benefits to GANs, notjust improving robustness.
RGAN [38] shows that adversarial training can enhance the generalizationof the generator and discriminator.
Based on theoretical analysis and empirical results, [39; 40; 41]show that updating the discriminator’s parameters with the loss calculated by adversarial disturbedimage samples can stabilize training and accelerate convergence.
In this paper, we use an off-the-shelf adversarial sample mining method iterative fast gradient signmethod (I-FGSM)
[17], to perform latent transform and mine effective z samples that benefit GANtraining in its bi-level optimization process.
More adversarial methods e.g. PGD [35] and MIFGSM [42] are also tried and discussed in Appendix E.
The pre-existing works mentioned above thatinterplay between GANs and adversarial learning make perturbations on raw pixel space (real samplesor generated samples), while we instead work on mining the latent space and introducing perturbationson latent samples, which we show is a novel and effective way to improve the generation.
Proposed Method
This section presents our twofold effort to lift the final generative performance: AdvLatGAN-z toachieve the targeted transform z ∗ ( · ) ; training algorithms AdvLatGAN-qua and AdvLatGAN-div totrain generation mapping G with better generation quality (i.e., -qua) and diversity (i.e., -div).
Bothmethods are derived by introducing the iterative updating in latent space.
3.1 Preliminaries
This section introduces adversarial attack/defense techniques and the regularization technique devisedin MSGAN [31] for mode seeking. Adversarial techniques include: i) adversarial samples which weintroduce in the latent space sampling to achieve z ∗ ( · ) ; ii) adversarial training which we introduce inthe bi-level optimization of GAN to develop new training algorithms for a more powerful G ( · ) .
Adversarial samples and adversarial training.
Adversarial training aims to defend against adversarial attacks.
It uses adversarial samples to calculate the loss.
In classification, for each image-labelpair ( x i , y i ) from the labeled training set, the adversarial samples x ′ is defined as:
[Equation]
Here ϵ is the radius of the closed ball to constrain perturbation.
While f θ and l denote the attack target(a neural net) and the corresponding loss (e.g. cross-entropy loss).
p denotes the norm dimension.
Given in total n adversarial samples, adversarial training is fulfilled by the optimization:
[Equation]
Mode coverage by regularizing distance of generated samples.
MSGAN [31] mitigates modecollapse for conditional generation by adding a regularizer over randomly sampled z 1 and z 2 :
[Equation]
where c is the condition vector for generation, and d
I and d z are the distance metrics in the target(image) space and latent space, respectively.
It encourages the distance to be maintained or evenenlarged in the target space to avoid the generated samples being trapped into an aggregation.
By integrating the regularization term with the original objective of cGANs [43], the final minimiza- 149 tion objective in MSGAN for training G and D is:
[Equation]
( c, z )))] (4) where L ori denotes the original objective, x denotes real data, and λ ms is the weight hyper-parameter.
Since Eq. 3 is to be maximized hence one can interpret λ ms as a negative coefficient.
3.2 Latent Adversarial Mining for Generation Quality
3.2.1 AdvLatGAN-z: Latent Space Transform
Note that as the real-data space is a union of disconnected manifolds [15], via a continuous generativemapping G , the optimal subset in the latent space w.r.t the fixed G is in no way connected.
We denote the real-data subset in the high-dimensional pixel space X as X r ⊂ X , and each disjoint submanifoldas M i .
Then we obtain X r =
(cid:83) n r i =
1 M i where M 1 to M n r are disconnected to each other.
Here werequire the splitting of X r to the extent that each submanifold is itself a connected set.
Definition 3.1.
Suppose G [ · ] : P ( Z ) → P ( X ) is the image-of-set function corresponding to thegenerative mapping function G ( · ) : Z → X , where Z and X denote subsets in latent space and pixelspace, and P ( · ) denotes the power set.
Then Z op ( G ) is the optimal subset corresponding to G inlatent space if G [ Z op ( G )] =
X r .
Proposition 3.2.
For any given continuous G [ · ] whose codomain includes multiple disconnectedmanifolds, Z op ( G ) is a union of disconnected subsets in the latent space.
The proof is given in Appendix B. Proposition 3.2 implies that with the fixed continuous G , fitting thereal-data manifolds requires a disconnected latent space support.
Below we show how to implicitlytransform the latent raw samples from Gaussian (or other forms e.g. uniform) to a new distributioncloser to Z op ( G ) via the idea of mining adversarial samples as introduced above.
We utilize the discriminator’s output to quantify the generation quality, which is trained to capturethe difference between the generated and real distribution. Considering that small perturbations on zcan cause considerable quality variation (technically D ’s output) as shown in Fig.
1 and Fig. 3, wenaturally resort to adversarial perturbation-based methods to achieve the transform.
We first deliverthe constrained optimization below, where both G and D are post-trained.
[Equation]
where z 0 denotes the original latent vector, z ∗ ( z 0 ) denotes the newly computed z , d ( · , · ) denotes thedistance which is ℓ ∞ in this paper, in line with the popular protocol in literature [34].
The above target can be readily solved by the well-developed tools in adversarial sample mining,e.g. the classic iterative fast gradient sign method (I-FGSM)
[17] as formulated below in Eq. 6.
In Appendix E, we also present comparison of more adversarial techniques e.g. PGD [35], MIFGSM [42], that can be seamlessly incorporated by our framework, and we show the simple I-FGSMoutperforms.
We conjecture the reason may be due to the simplicity of the latent space in our case.
[Equation]
From the distribution perspective, the transform is given by:
[Equation]
(7) where p
z denotes the original latent distribution, p ∗ z denotes the newly searched one.
D ( p z , p 0 z ) ≤ ϵmeans that there is a limit to the distribution variation.
A newly mined z can be considered as asample from p ∗ z if the sampling is within the perturbation circle.
Fig.
4 shows the results by thepre-trained SNGAN [8] (i.e. the given G and D ) on STL-10 [44].
We then show that Eq.
7 guarantees the correct optimization direction that optimizes the sourcedistribution p z towards the optimal probability distribution in latent space according to the currentfixed G , i.e. p opz ( G ) defined in Definition.
The proof for Proposition 3.4 is given in Appendix B.
Definition 3.3.
Given G , p opz ( G ) is the optimal probability distribution in latent space correspondingto Z op ( G ) when p opz ( G ) satisfies that if z ∼ p opz ( G ) then G ( z ) ∼ p r .
Here p r denotes the real-dataprobability distribution in pixel space.
Proposition 3.4.
Optimizing GAN’s training criterion i.e. min G max D { E z ∼ p
z [log(1 − D ( G ( z )))]
+
E x ∼ p r [log( D ( x ))] } is to minimize JSD ( p opz ( G ) , p 0 z ) .
Prop. 3.4 implies that the GAN’s optimization is to minimize the difference between p opz ( G ) and p 0 z .
Note that for optimizing p z , the criterion merely involves E z ∼ p
z [log(1 − D ( G ( z )))] i.e. the objective in Eq.
Given optimized fixed G and D (and p opz is thereby fixed), leaving p z the only trainableparameter, optimizing the training criterion with p z in the neighborhood of p
z is driving p
z closer top opz .
Note that p z in the neighborhood of p
z is necessary otherwise the optimized criterion will departmuch from JSD , and we achieve this condition by enforcing D ( p z , p 0 z ) ≤ ϵ in Eq.
3.2.2 AdvLatGAN-qua: Quality Targeted Training Pipeline
Considering the pairwise approach for adversarial sample mining, i.e. adversarial training, we hopeto mine samples that benefit the optimization in GAN training, which is achieved by utilizing theaforementioned latent space transform.
Conducting the transform in training to obtain the new(probably disconnected) latent distribution for each iteration can compensate for the difficulty forgenerators to align the pace of generators and discriminators, as it is widely recognized that it is muchmore challenging to train the generator than the discriminator, for its inherent defect that it cannotmap a continuous distribution to a disconnected one with multiple modes.
Specifically, the transform is conducted to the raw Gaussian sampling in D updating iterations whilethe rest of the training algorithm remains consistent with vanilla GAN [1] and other variants withthe orthogonal efforts to latent space mining.
We present the algorithm in Alg.
1 in the appendixwhich we call Adversarial Latent GAN with a post meaning quality (AdvLatGAN-qua).
The latenttransform is first computed by I-FGSM to mine the targeted samples in latent space, then D and Gare updated.
The iteration continues until enough steps or the convergence of G .
The above latent transform model does not consider how to avoid mode collapse, as each time onlyone latent sample is considered for optimization in isolation.
A similar constrained optimizationformulation can be devised to improve generation diversity, as shown in the following subsection.
3.3 AdvLatGAN-div: Latent Adversarial Mining for Diversity
In this section, we tend to modify the regularizer for mode seeking in MSGAN [31] by substitutingthe randomly selected regularized z pairs with hard sample pairs that are more inclined to collapse,lifting the efficiency for avoiding mode collapse.
We first show how to search for such hard samplepairs.
Given a sample z 0 from the raw latent distribution, we aim to search its paired sample z ∗ toform a hard sample pair, in the sense that their generations are close in the target space given G .
Weconsider two aspects.
First, as hard samples, the paired samples shall still belong to the same categoryc which can be modeled with conditional GAN.
Second, their distance in the latent space shall alsobe not too close, otherwise they may be aggregated in the target space due to continuous mapping.
Inother words, hard samples causing mode collapse shall be those close in target space while apart inlatent space, when G is given.
Hence, we adopt the form of Eq.
8 to design our search procedure:
[Equation]
[Table]
The symbols and terms are similar to Eq.
Akin to
AdvLatGAN-qua, we use I-FGSM to obtain z ∗ and again useℓ ∞ to control deviation magnitude.
For each update step, werandomly select the first latent vector z 0 , and the second vectorz ∗ is searched by n iterations of the I-FGSM method under theℓ ∞ restriction starting from the Gaussian neighborhood of z 0with a small standard deviation.
Table 1 shows the effectiveness of the above paired hard sample seeking method, where the results arecalculated over 5k generated images with DCGAN [45] as backbone.
The visual results on Facadesas shown in Fig.
6 suggest that the pair obtained by solving Eq.
8 tend to collapse (as they are hardsamples), while the opposite leads to more different generation by using the inverse form of Eq. 8.
The above experiments verify the efficacy of our mining technique.
We further introduce it into theinitial training process of MSGAN [31].
In MSGAN, the regularization term by Eq.
3 regularizes theoptimization process, where the selection of the pair is entirely random, without using any heuristicinformation.
We instead use the z pair that is more inclined to collapse by randomly taking one z anditerating by the aforementioned method to get another z .
The other settings remain unchanged.
By using Eq.
8 to generate sample pairs to address mode collapse, we finally develop our new diversityenhancing GAN training approach AdvLatGAN-div as shown in Alg. 2 in the appendix, which canbe further enhanced by post-training as discussed in experiments. [Table]
Experiments
Experiments on ImageNet and CelebA are performed on GPUs of Tesla V100.
Other public benchmark results are performed on a single GPU of GeForce RTX 3090.
Code will be publicly available.
4.1 Experimental Setup
We validate the effectiveness of the proposed methods for two parts, i e., effective latent samplingimprovement z ∗ and improved generation mapping G .
Five variant methods can be derived fordifferent targets, among which AdvLatGAN-qua+ and AdvLatGAN-div+ are our final full version.
i)
AdvLatGAN-z: post-training latent sampling improvement fighting against quality discontinuity; ii)
AdvLatGAN-qua: GAN training algorithm for better quality using in-training latent samplingtransform; iii) AdvLatGAN-div: GAN training algorithm for a more diverse generation by usingin-training latent sampling transform; iv) AdvLatGAN-qua+: conducting -z over the generationresults of -qua; v) AdvLatGAN-div+: conducting -div over the generation results of -div.
Wecompare these methods in Fig.
7 and Table 2.
We adopt the Inception Score, IS [46], Fréchet Inception Distance, FID [47] and density/coverage [48]for evaluation.
IS utilizes the classification results of the generated images by the Inception Networkto evaluate the generation quality.
FID additionally considers the target distribution and evaluatesthe quality by fitting the distribution distance.
Density and coverage are more recently proposedevaluation metrics that analyze the target and generated manifolds and estimate the similarity.
Resultson hyper-parameter study and overhead discussion are presented in Appendix C and D.
4.2 AdvLatGAN-z: Performance Boosting by Post-training Latent Sampling
We evaluate the boosting effect of post-training latent sampling improvement method AdvLatGAN-z(also previously denoted as z ∗ ), which is guided by Eq.
It aims to mitigate quality discontinuity.
[Table] [Table]
Results on Synthetic Data.
We test AdvLatGAN-z on Grid and Ring datasets consisting of a mixtureof 25 and 8 2-D Gaussians as done in [49].
The result is given in Fig.
By updating the latentvariables, our method achieves high-quality generation (samples closer to target Gaussian centers).
Results on MNIST.
Low complexity of MNIST [50] makes it possible to generate recognizableresults by two-dimensional latent variables which can be better visualized.
The iterative latentdistribution change is illustrated in Fig.
We respectively conduct AdvLatGAN-z for the trained
DCGAN and WGAN-GP models, with single-step iteration constraint using ℓ ∞ and iterated step sizeϵ is set as 0.03.
We present the first 30 iterations that can reflect the most significant changes.
The bottom are
AdvLatGAN z can effectively mitigate defects
Results on STL-10.
STL-10 [44] maintains ahigher resolution and richer image representation.
We utilize AdvLatGAN-z for pre-trained
DCGAN, WGAN-GP and SNGAN, with ℓ ∞single-step constraint and the iteration step sizeϵ is set as 0.05.
We conduct 20 steps each time.
IS and FID are adopted for evaluation, which arecalculated over 10,000 generated images.
Wecompare other efforts enhancing GAN performance by latent space mining including DOT,
DDLS and EvolGAN.
Table 3 present the results.
AdvLatGAN-z outperforms the baselinesand delivers at most 24.3% advantage for FIDon vanilla in the WGAN-GP setting.
Results on AFHQ and FFHQ.
To address theissues in Fig.
3, we evaluate on AFHQ [55] and
FFHQ [56] based on StyleGAN2-ada [18].
Inthe setting of Fig.
3, when the sampling encounters invalid examples, AdvLatGAN-z is ableto transform the bad z to generation-friendlyz ∗ ( z ) to avoid bad generation.
Here we verifywhether invalid samples can be improved with
AdvLatGAN-z.
We select six bad generationswith the first three corresponding to those in Fig.
3 and conduct AdvLatGAN-z to their latent vectors.
Fig.
9 shows that AdvLatGAN-z can effectively fix or avoid image defects.
4.3 AdvLatGAN-qua/div: Improving Generation Map
This section shows the effectiveness of our training pipelines, which aim to improve the generationmapping G respectively for quality and diversity.
The details are presented in Alg.1 and Alg.2.
AdvLatGAN-qua for Quality Improvement.
Table 4 shows the results on Cifar-10 and STL-10,using the mainstream architectures: DCGAN, WGAN, WGAN-GP, SNGAN, LSGAN, WGAN- [Table]
div and ACGAN.
We do not include ACGAN in the unlabeled STL-10 setting because it requireslabels [54].
IS and FID are adopted.
We adversarially train GAN using AdvLatGAN-qua.
Thenumber of latent iterations in training is set as one per discriminator step, and other details is given in Appendix K.1.
As they are orthogonal to each other, we integrate AdvLatGAN-z with AdvLatGANqua (i.e. AdvLatGAN-qua+) and include it in comparison.
The iteration step size ϵ of AdvLatGAN-zis set to 0.05 and we conduct 20 steps each time.
Our method achieves a notable improvementcompared to that without the latent space transform strategy.
AdvLatGAN-qua+ achieves the bestgain on IS in WGAN STL-10 setting from 6.51 to 8.57 and FID in WGAN-div from 37.7 to 23.0.
[Table]
Table.
6 presents results on larger datasets:
LSUN Church [57], CelebA [58] and ImageNet [59], using WGAN-GP and SNGANas backbones, evaluated by FID, density andcoverage.
We include AdvLatGAN-qua and-qua+ in the comparison.
The number of latent iterations during training is set as 1 perdiscriminator step, and other details are givenin Appendix K.1.
The iteration step size ϵ of
AdvLatGAN-z is set to 0.01 for SNGAN and0.002 for WGANGP and we conduct 100 stepseach time.
AdvLatGAN-qua+ has achievedthe best performance gain on FID in SNGAN
LSUN-64 setting from 11.961 to 7.285.
AdvLatGAN-div for Diversity Improvement.
We test on Cifar-10 and STL-10 forconditional generation and we adopt FID, density and coverage as the metrics, respectivelyfor the entire set of generated images and forthose in each label.
We test AdvLatGAN-divand AdvLatGAN-div+ compared to the baseline MSGAN.
The results are shown in Table 5.
Our method outperforms MSGAN by all the three metrics.
Note that the models are trained on theoverall set and for each label and the optimization may not guarantee targeted enough guidance.
AdvLatGAN-div+ achieves the best overall performance gain on FID from 30.225 to 26.169, densityfrom 0.353 to 0.431 and coverage from 0.455 to 0.54.
Conclusion and Broader Impact
This work analyzes GANs from the continuous mapping perspective and notes that lifting the overallgenerative performance requires a twofold effort including latent distribution transform and mappingimprovement.
Adversarial sample mining techniques are introduced to explore latent space andnovel training pipelines are derived to improve the generative mapping.
The proposed AdvLatGANhas shown promising power in both quality and diversity.
Our technique is basically orthogonal toexisting mainstream methods.
For its limitation, it remains open for how to effectively combine theproposed two techniques for diversity and quality.
For potential negative social impact, fake contentby the generation models including ours shall always be carefully treated to avoid abuse.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4, Appendix I, Appendix J and Appendix K.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[No]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]