Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Anonymous Author(s)
AffiliationAddress email
Abstract
For Generative Adversarial Networks which map a latent distribution to the targetdistribution, in this paper, we study how the sampling in latent space can affectthe generation performance, especially for images.
We observe that, as the neuralgenerator is a continuous function, two close samples in latent space would bemapped into two nearby images, while their quality can differ much as the qualityis generally not a continuous function in pixel space.
From such a continuousmapping function perspective, it is also possible that two distant latent samplescan be mapped into two close images (if not exactly the same).
In particular, ifthe latent samples are mapped in aggregation into a single mode, mode collapseoccurs.
Accordingly, we propose adding an implicit latent transform before themapping function to improve latent z from its initial distribution, e.g., Gaussian.
This is achieved using well-developed adversarial sample mining techniques, e.g.iterative fast gradient sign method (I-FGSM).
We further propose new GAN training pipelines to obtain better generative mappings w.r.t quality and diversity byintroducing targeted latent transforms into the bi-level optimization of GAN.
Experimental results on visual data show that our method can effectively achieveimprovement in both quality and diversity.
Introduction
Generative Adversarial Networks (GANs) [1] have shown effectiveness for generating high-fidelitydata, especially for images under various settings [2; 3; 4; 5; 6].
Based on the zero-sum game, themodel is trained by the adversarial process between the generator and the discriminator.
Many effortshave been made to achieve a more realistic generation from different perspectives.
For instance,
WGAN [7], SNGAN [8], LSGAN [9] aim to design better objective functions.
ImprovedGAN [10],
AC-GAN [11] propose practical training techniques.
While more complex network structures arestudied in BigGAN [12], StyleGAN [13], SAGAN [14].
Despite the above progress, a relatively less studied question is how the latent space affects thegeneration quality and diversity, which can be orthogonal to the above frequently studied factors likemodel architectures and training techniques.
In this paper, we argue that the latent samples from astandard continuous distribution (e.g. Gaussian) can often be mapped to varying-quality samplesfor the generation.
One reason is that the generator is a continuous function (as neural nets) whilethe generation quality (e.g. natural images in pixel space) is not. Moreover, it has been shown in
GAN literature [15] that many image objects lie upon multiple disjoint manifolds.
A continuousmapping from a continuous latent distribution, e.g., Gaussian, may generate invalid samples in thebetween of the manifolds when the generator seeks to cover all the meaningful modes.
See Fig.
1 (a)for illustration.
This fact also implies a dilemma: for complex real-world data, there is a trade-offbetween quality and diversity when the latent vectors are sampled from a continuous uni-modedistribution, e.g., Gaussian, based on that the generator as a network is a continuous mapping.
[Figure]
Due close samples close pixel while the quality vary and
(b) are close in pixel space, but Cat (a) loses its right Our method transforms bad latent z towards the preimage of the nearest real manifold, thus avoiding terrible generation
The above contradiction between the discontinuity in generation data quality and the continuity oflatent distribution as well as mapping function, can be one of the significant factors to the wellknown difficulty in GANs training, especially for generators, as one unreasonably hopes to enforce adiscontinuous mapping via continuous neural nets.
On the other hand, the mapping is also critical tothe performance of functions with continuous nature, as badly trained mapping can lead to poorlyfitted generated distribution.
Meanwhile, by realizing that the mapping function allows for many-toone mapping, mode collapse [16] can naturally happen when different latent samples are mappedinto few or even a single mode in the target space.
To tackle these issues, we impose an extra (implicit) transform function z ∗ ( · ) on the raw sampling z ,and the generation can be written by G ( z ∗ ( z )) , which requires twofold efforts on both G ( · ) and z ∗ ( · ) .
From this perspective, most existing works employ z ∗ ( · ) as an identity function, while we try tofind a more effective one.
We model the z transform process as making perturbations to the originalsampling since z ∗ ( z ) shall not depart much from z otherwise the generative expression will differ.
This leads to the adversarial learning methods.
Specifically, the implicit function based transformz ∗ ( · ) can be achieved by updating the latent z ∼ p ( z ) to minimize loss G i.e. log(1 − D ( G ( z )))using adversarial sample mining techniques e.g. iterative fast gradient sign method (I-FGSM)
[17]with the parameters of G and D fixed, and we name the method as AdvLatGAN-z. See Fig.
1 (b)for the schematic diagram.
We will show that our z transform (perturbations) can deliver a latentdistribution better fitting the real distribution with the fixed G .
On top of our first effort, we propose a training strategy to further improve the generative mappingrespectively for quality and diversity.
See Fig. 2 for the schematic diagram. This is achieved byintroducing z optimization (i.e., targeted implicit transforms on z ) into GAN training, specifically,updating z ∼ p ( z ) using I-FGSM to find more effective latents in the bi-level optimization ofvanilla GAN.
Respectively for quality and diversity, we use two iterative updating strategies forz under different objectives, and we name the two training algorithms as AdvLatGAN-qua and
AdvLatGAN-div.
The highlights of this paper include:
We rethink the role of the generator as a continuous function, which may incur quality discontinuityin target space when the latent samples are directly sampled from a continuous distribution, togetherwith poorly fitted generated distribution and mode aggregation (collapse) when the function suffersfrom bad properties w.r.t.
quality or diversity.
Then, we propose a composite generation perspectiveby formulating the mapping from G ( z ) to G ( z ∗ ( z )) .
It provides new methodological design guidance to GAN which requires twofold efforts on both G ( · ) and z ∗ ( · ) .2)
We propose a pipeline for G ( · ) and z ∗ ( · ) , by first training generative mappings for better generationquality and diversity respectively.
Then the transform z ∗ ( · ) namely AdvLatGAN-z is enforced andcomputed implicitly to obtain an improved distribution for the effective generation.
The traininginvolves latent vector transform under different objective functions for quality and diversity, leadingto new GAN models: AdvLatGAN-qua/div and their variants.
notable improvement of our approach on both generation quality and diversity whereby the performance gain is basically orthogonal to those existing GAN techniques
Related Work
Our latent vector transform pipeline mines effective latent samples and the resulting perturbationsrelate to adversarial methods.
The proposed GAN training pipelines include the aforementionedtransform in network training, which can be viewed as mining adversarial samples in training, namelyadversarial training.
Hence we briefly review existing works on latent space mining for GANs andadversarial learning.
Also, techniques for avoiding mode collapse are studied.
Latent space mining in GANs.
Techniques have been devised for latent space mining for GANs.
Byutilizing the steerability of GANs in latent space, StyleGAN [13] introduces a latent variable shift tobetter decouple the various styles by using a linear network (which is still continuous).
[19], [20]and [21] introduce the latent space mining into training to achieve clustering objectives or improvethe dynamic of training.
In addition, some recent works focus on improving latent sampling qualityto obtain high-quality images.
For example, interactive evolutionary computation is used to makethe generation process more controllable in [22], which meanwhile also brings additional manualefforts.
To reduce manual intervention, [23; 24] use Koncept512 [25] as a criterion to improve thegeneration quality.
However, as discussed in [23], this strategy is somehow biased and can not workwell for each dataset (e.g., Pokemons).
The reason is that the quality estimator is not even designatedto access the real-data distribution information.
Instead, the generation is guided by a predefinedscore function.
Exploring the real distribution’s guidance from the discriminator, DOT [26] utilizesoptimal transport in latent space to improve the initial sampled distribution, and DDLS [27] developsdiscriminator driven Langevin dynamics to obtain high-quality samples.
Mitigating mode collapse for GANs.
Efforts are made to mitigate mode collapse, which are basedon the thinking that real-world samples lie in disjoint manifolds [15].
One way is to introduce acollection of generators such that each generator may cover a specific mode since a single generatorcan hardly fulfill a discontinuous function [15].
However, more works focus on improving mappingproperties, explicitly obtaining a mapping covering more modes [28; 29; 30].
Note that the generateddistribution obtained by this method will contain invalid parts due to the continuity of the latentdistribution and the mapping.
This calls for the selective sampling of the continuous latent distributionor directly amending the latent distribution, which is rarely noticed in these works.
Adversarial samples and adversarial training.
Adversarial samples are manipulated samplescrafted by adding indistinguishable perturbations to fool deep nets.
Since [31], various adversarialattack methods have been devised [32; 17; 33].
For defense, adversarial training is meanwhiledeveloped [32; 34; 35].
In particular, recent studies have shown that adversarial training can bringmany benefits to GANs, not just improving robustness.
RGAN [36] shows that adversarial trainingcan enhance the generalization of the generator and discriminator.
Based on theoretical analysisand empirical results, [37; 38; 39] show that updating the discriminator’s parameters with the losscalculated by adversarial disturbed image samples can stabilize training and accelerate convergence.
In this paper, we use an off-the-shelf adversarial sample mining method iterative fast gradient signmethod (I-FGSM)
[17], to perform latent transform and mine effective z samples that benefit GANtraining in its bi-level optimization process, which respectively relate to the procedure of adversarialsample mining and adversarial training.
More adversarial methods e.g. PGD [33] and MI-FGSM [40]are also tried and discussed in Appendix E.
The pre-existing works mentioned above that interplaybetween GANs and adversarial learning make perturbations on raw pixel space (real samples orgenerated samples), while we instead work on mining the latent space and introducing perturbationson latent samples, which we show is a novel and effective way to improve the generation.
Proposed Method
This section presents our twofold effort to lift the final generative performance: AdvLatGAN-z toachieve the targeted transform z ∗ ( · ) ; AdvLatGAN-qua and AdvLatGAN-div to obtain generationmapping G with better generation quality (i.e., -qua) and diversity (i.e., -div).
Both methods arederived from the iterative updating in latent space.
3.1 Preliminaries
We introduce adversarial samples and adversarial training which is used to facilitate: i) generatez ∗ ( z ) ; ii) develop training algorithms, including a new training pipeline based on vanilla adversarialprocess and regularization technique devised in MSGAN [29] for mode seeking.
Adversarial samples and adversarial training.
Adversarial training aims to defend against adversarial attacks.
It uses adversarial samples to calculate the loss.
In classification, for each image-labelpair ( x i , y i ) from the labeled training set, the adversarial samples x ′ is defined as:
[Equation]
Here ϵ is the radius of the closed ball to constrain perturbation.
While f θ and l denote the attack target(a neural net) and the corresponding loss (e.g. cross-entropy loss).
p denotes the norm dimension.
Given in total n adversarial samples, adversarial training is fulfilled by the optimization:
[Equation]
Mode coverage by regularizing distance of generated samples.
MSGAN [29] mitigates modecollapse for conditional generation by adding a regularizer over randomly sampled z 1 and z 2 :
[Equation]
where c is the condition vector for generation, and d
I and d z are the distance metrics in the target(image) space and latent space, respectively.
It encourages the distance to be maintained or evenenlarged in the target space to avoid the generated samples being trapped into an aggregation.
By integrating the regularization term with the original objective of cGANs [41], the final minimiza- 142 tion objective in MSGAN for training G and D is:
[Equation]
where L ori denotes the original objective, x denotes real data, and λ ms is the weight hyper-parameter.
Since Eq. 3 is to be maximized hence one can interpret λ ms as a negative coefficient.
3.2 Latent Adversarial Mining for Generation Quality
3.2.1 Latent Space Transform
Note that as the targeted real-data space is a union of disconnected manifolds [15], via a continuousgenerative mapping G , the optimal subset in the latent space w.r.t the fixed G is in no way connected.
We denote the targeted real-data subset in the high-dimensional pixel space X as X r ⊂ X , and eachdisjoint submanifold as M i .
Then we obtain X r =
[Equation] n r i =
1 M M 1 to M n r are disconnected to of r extent connected.
Definition 3.1.
Suppose G [ · ] : P ( Z ) → P ( X ) is the image-of-set function corresponding to thegenerative mapping function G ( · ) : Z → X , where Z and X denote subsets in latent space and pixelspace, and P ( · ) denotes the power set.
Then Z op ( G ) is the optimal subset corresponding to G inlatent space if G [ Z op ( G )] =
X r .
Proposition 3.2.
For any continuous G [ · ] whose codomain includes multiple disconnected manifolds, Z op ( G ) is a union of disconnected subsets in the latent space.
The proof is given in Appendix B. Proposition 3.2 implies that with the fixed continuous G , fitting thereal-data manifolds requires a disconnected latent space support.
Below we show how to implicitlytransform the latent raw samples from Gaussian (or other forms e.g. uniform) to a new distributioncloser to Z op ( G ) via the idea of mining adversarial samples as introduced above.
We design the following constrained optimization that requires the new z ∗ to be close enough 1 to z . Its prediction score by the discriminator D should be as high as possible to improve the generationquality.
Recall otherwise the given z 0 may correspond to a poor generation as discussed in Sec. 1. Here both G and D are fixed.
[Equation]
(5) where z 0 denotes the original latent vector, z ∗ ( z 0 ) denotes the newly computed z , d ( · , · ) denotes thedistance which is ℓ ∞ in this paper, in line with the popular protocol in literature [32].
The above target can be readily solved by the well-developed tools in adversarial sample mining,e.g. the classic iterative fast gradient sign method (I-FGSM)
[17] as formulated below in Eq. 6.
In Appendix E, we also present comparison of more adversarial techniques e.g. PGD [33], MIFGSM [40], that can be seamlessly incorporated by our framework, and we show the simple I-FGSMoutperforms.
We conjecture the reason may be due to the simplicity of the latent space in our case.
[Equation]
From the distribution perspective, the transform is given by:
[Equation]
where p
z denotes the original latent distribution, p ∗ z denotes the newly searched one.
D ( p z , p 0 z ) ≤ ϵmeans that there is a limit to the distribution variation.
A newly mined z can be considered as asample from p ∗ z if the sampling is within the perturbation circle.
Fig.
4 shows the results by thepre-trained SNGAN [8] (i.e. the given G and D ) on STL-10 [42].
We then show that Eq.
7 guarantees the correct optimization direction that optimizes the sourcedistribution p z towards the optimal probability distribution in latent space according to the currentfixed G , i.e. p opz ( G ) defined in Definition.
The proof for Proposition 3.4 is given in Appendix B.
Definition 3.3.
Given G , p opz ( G ) is the optimal probability distribution in latent space correspondingto Z op ( G ) when p opz ( G ) satisfies that if z ∼ p opz ( G ) then G ( z ) ∼ p r .
Here p r denotes the real-dataprobability distribution in pixel space.
Proposition 3.4.
Optimizing GAN’s training criterion i.e. min G max D { E z ∼ p
z [log(1 − D ( G ( z )))]
+
E x ∼ p r [log( D ( x ))] } is to minimize JSD ( p opz ( G ) , p 0 z ) .
Prop. 3.4 implies that the GAN’s optimization is to minimize the difference between p opz ( G ) and p 0 z .
Note that for optimizing p z , the criterion merely involves E z ∼ p
z [log(1 − D ( G ( z )))] i.e. the objectivein Eq.
Given optimized fixed G and D (and p opz is thereby fixed), leaving p z the only trainableparameter, optimizing the training criterion with p z in the neighborhood of p
z is driving p
z closer top opz .
Note that p z in the neighborhood of p
z is necessary otherwise the optimized criterion will departmuch from JSD , and we achieve this condition by enforcing D ( p z , p 0 z ) ≤ ϵ in Eq.
3.2.2 Quality Targeted Training Pipeline
Considering the pairwise approach for adversarial sample mining, i.e. adversarial training, we hopeto mine samples that benefit the optimization in GAN training, which is achieved by utilizing theaforementioned latent space transform.
Conducting the transform in training to obtain the new(probably disconnected) latent distribution for each iteration can compensate for the difficulty forgenerators to align the pace of generators and discriminators, as it is widely recognized that it is muchmore challenging to train the generator than the discriminator, for its inherent defect that it cannotmap a continuous distribution to a disconnected one with multiple modes.
Specifically, the transform is conducted to the raw Gaussian sampling in D updating iterations whilethe rest of the training algorithm remains consistent with vanilla GAN [1] and other variants withthe orthogonal efforts to latent space mining.
We present the algorithm in Alg.
1 in the appendixwhich we call Adversarial Latent GAN with a post meaning quality (AdvLatGAN-qua).
The latenttransform is first computed by I-FGSM to mine the adversarial samples in latent space, then D and Gare updated.
The iteration continues until enough steps or the convergence of G .
The above latent transform model does not consider how to avoid mode collapse, as each time onlyone latent sample is considered for optimization in isolation.
A similar constrained optimizationformulation can be devised to improve generation diversity, as shown in the following subsection.
3.3 Latent Adversarial Mining for Diversity
We take a hard sample mining perspective to improve mode coverage.
Given a sample z 0 from theraw latent distribution, we aim to search its paired sample z ∗ to form a hard sample pair, in the sensethat their generations are close in the target space given G .
We consider two aspects.
First, as hardsamples, the paired samples shall still belong to the same category c which can be modeled withconditional GAN.
Second, their distance in the latent space shall also be not too close, otherwisethey may be aggregated in the target space due to continuous mapping.
In other words, hard samplescausing mode collapse shall be those close in target space while apart in latent space, when G isgiven.
Hence, we adopt the form of Eq.
8 to design our search procedure:
[Equation]
[Table]
The symbols and terms are similar to Eq.
Akin to
AdvLatGAN-qua, we use I-FGSM to obtain z ∗ and again useℓ ∞ to control deviation magnitude.
For each update step, werandomly select the first latent vector z 0 , and the second vectorz ∗ is searched by n iterations of the I-FGSM method under theℓ ∞ restriction starting from the Gaussian neighborhood of z 0with a small standard deviation.
Table 1 shows the effectiveness of the above paired hard sample seeking method, with experimentson Cifar-10 [43] and Facades [44] with DCGAN [45] as backbone.
The visual results on Facadesas shown in Fig.
6 suggest that the pair obtained by solving Eq.
8 tend to collapse (as they are hardsamples), while the opposite leads to better diversity by using the inverse form of Eq. 8.
The above experiments verify the efficacy of our mining technique.
We further introduce it into theinitial training process of MSGAN [29].
In MSGAN, the regularization term by Eq.
3 regularizes theoptimization process, where the selection of the pair is entirely random, without using any heuristicinformation.
We instead use the z pair that is more inclined to collapse by randomly taking one z anditerating by the aforementioned method to get another z .
The other settings remain unchanged.
By using Eq. 8 to generate adversarial sample pairs to address mode collapse, we finally develop ournew diversity enhancing GAN training approach AdvLatGAN-div as shown in Alg. 2 in the appendix,which can be further enhanced by post-training as discussed in experiments.
Experiments
Experiments on ImageNet and CelebA are performed on GPUs of Tesla V100.
Other public benchmark results are performed on a single GPU of GeForce RTX 3090.
Code will be publicly available.
[Table]
4.1 Experimental Setup
We validate the effectiveness of the proposed methods for two parts, i e., effective latent samplingimprovement z ∗ and improved generation mapping G .
In particular, five variant methods can bederived for different targets, among which AdvLatGAN-qua+ and AdvLatGAN-div+ are the final fullversion of our contributions. We present the methods below:
i)
AdvLatGAN-z: post-training latent sampling improvement fighting against quality discontinuity;ii)
AdvLatGAN-qua: GAN training for better quality using in-training latent sampling transform; iii)AdvLatGAN-div: GAN training for a more diverse generation by using in-training latent sampling transform; iv) AdvLatGAN-qua+: integration of in and post-training latent sampling improvementfor generation quality; v) AdvLatGAN-div+: integration of in- and post-training latent samplingimprovement for diversity.
We compare these methods in Fig.
7 and Table 2.
We adopt the popular Inception Score [46], Fréchet Inception Distance [47] and density/coverage [48]for evaluation.
IS utilizes the classification results of the generated images by the Inception Networkto evaluate the generation quality.
FID additionally considers the target distribution and evaluates thequality by fitting the distribution distance.
Density and coverage are more recently proposed evaluationmetrics that analyze the target and generated manifolds and estimate the similarity.
Experimentalresults on hyper-parameter study and overhead discussion are presented in Appendix C and D.
4.2 AdvLatGAN-z: Performance Boosting by Post-training Latent Sampling
We evaluate the boosting effect by post-training latent sampling improvement termed as AdvLatGANz (previously denoted as z ∗ , an implicit transform on the original latent sampling from Gaussian).
Itaims to mitigate quality discontinuity.
The results are given as follows.
Results on Synthetic Data.
We test AdvLatGAN-z on Grid and Ring datasets consisting of a mixtureof 25 and 8 2-D Gaussians as done in [49].
The result is given in Fig.
By updating the latentvariables, our method achieves high-quality generation (samples closer to target Gaussian centers). [Table] [Table]
Results on MNIST.
Low complexity of MNIST [50] makes it possible to generate recognizableresults by two-dimensional latent variables which can be better visualized.
The iterative latentdistribution change is illustrated in Fig.
We respectively conduct AdvLatGAN-z for the trained
DCGAN and WGAN-GP models, with single-step iteration constraint using ℓ ∞ and iterated step sizeϵ is set as 0.03.
We present the first 30 iterations that can reflect the most significant changes.
Results on STL-10.
STL-10 [42] maintains ahigher resolution and richer image representation.
We utilize AdvLatGAN-z for pre-trained
DCGAN, WGAN-GP and SNGAN, with ℓ ∞single-step constraint and the iteration step sizeϵ is set as 0.05.
We conduct 20 steps each time.
IS and FID are adopted for evaluation, which arecalculated over 10,000 generated images.
Wecompare other efforts enhancing GAN performance by latent space mining including DOT,
DDLS and EvolGAN.
Table 3 present the results.
AdvLatGAN-z outperforms the baselinesand delivers at most 24.3% advantage for FIDon vanilla in the WGAN-GP setting.
Results on AFHQ and FFHQ.
To address theissues in Fig.
3, we evaluate on AFHQ [55] and
FFHQ [56] based on StyleGAN2-ada [18].
Inthe setting of Fig.
3, when the sampling encounters invalid examples, AdvLatGAN-z is ableto transform the bad z to generation-friendlyz ∗ ( z ) to avoid bad generation.
Here we verifywhether invalid samples can be improved with
AdvLatGAN-z.
We select six bad generationswith the first three corresponding to those in Fig.
3 and conduct AdvLatGAN-z to their latent vectors.
Fig.
9 shows that AdvLatGAN-z can effectively fix or avoid image defects.
4.3 AdvLatGAN-qua/div: Improving Generation Map
This section shows the effectiveness of the proposed GAN training pipelines, which aim to improvethe generation mapping G respectively for quality and diversity.
AdvLatGAN-qua for Quality Improvement.
Table 4 show the results on Cifar-10 and STL-10,using the mainstream architectures including DCGAN, WGAN, WGAN-GP, SNGAN, LSGAN,
WGAN-div and ACGAN.
We do not include ACGAN in the STL-10 setting because it requireslabels [54] and does not work on unlabeled STL-10.
IS and FID are adopted.
We adversariallytrain GAN using AdvLatGAN-qua.
The number of latent iterations during training is set as one per [Table] discriminator step, and other details will be given in Appendix K.1.
As they are orthogonal to eachother, we integrate AdvLatGAN-z with AdvLatGAN-qua (i.e. AdvLatGAN-qua+) and include itin comparison.
The iteration step size ϵ of AdvLatGAN-z is set to 0.05 and we conduct 20 steps [Table] each time.
Our method achieves a notable improvement compared to that without the latent spacetransform strategy.
AdvLatGAN-qua+ achieves the best gain on IS in WGAN STL-10 setting from6.51 to 8.57 and FID in WGAN-div setting from 37.7 to 23.0.
[Table]
[Table]
Table.
6 presents experimental results on largerscale datasets including LSUN Church [57],
CelebA [58] and ImageNet [59], using WGANGP and SNGAN as backbones. We adopt FID,density and coverage as the evaluation metrics.
We include AdvLatGAN-qua and AdvLatGANqua+ in the comparison.
The number of latentiterations during training is set as 1 per discriminator step, and other details will be givenin Appendix K.1.
The iteration step size ϵ of
AdvLatGAN-z is set to 0.01 for SNGAN and0.002 for WGANGP and we conduct 100 stepseach time.
AdvLatGAN-qua+ has achievedthe best performance gain on FID in SNGAN
LSUN-64 setting from 11.961 to 7.285.
AdvLatGAN-div for Diversity Improvement.
We test on Cifar-10 and STL-10 for conditionalgeneration and we adopt FID, density and coverage as the metrics, respectively for the entire setof generated images and for those in each label.
We test AdvLatGAN-div and AdvLatGAN-div+compared to the baseline MSGAN.
The results are shown in Table 5.
The proposed method outperforms MSGAN reflected by all three metrics.
Note that the models are trained on the overall setand for each label the optimization may not guarantee targeted enough guidance (there may existtrade-offs between optimization for different data), thus the improvement for each label may not beso significant but still shows superiority.
AdvLatGAN-div+ achieves the best overall performancegain on FID from 30.225 to 26.169, density from 0.353 to 0.431 and coverage from 0.455 to 0.54.
Conclusion and Broader Impact
This work analyzes GANs from the continuous mapping perspective and notes that lifting the overallgenerative performance requires a twofold effort including latent distribution transform and mappingimprovement.
Adversarial sample mining techniques are introduced to explore latent space andnovel training pipelines are derived to improve the generative mapping.
The proposed AdvLatGANhas shown promising power in both quality and diversity.
Our technique is basically orthogonal toexisting mainstream methods.
For its limitation, it remains open for how to effectively combine theproposed two techniques for diversity and quality.
For potential negative social impact, fake contentby the generation models including ours shall always be carefully treated to avoid abuse.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4, Appendix I, Appendix J and Appendix K.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c Did you include the estimated hourly wage paid
to
participants
and
the
total
amount spent on participant compensation
[N/A