I MPROVING G ENERATIVE A DVERSARIAL N ETWORKS VIA A DVERSARIAL L EARNING IN L ATENT S PACE
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks ( GANs) have been widely studied as generative models, which map a latent distribution to the target distribution.
Although many efforts have been made in terms of backbone architecture design loss function and training techniques
, few results have been obtained on how the sampling in latent space can affect the ﬁnal performance and existing works on latent space mainly focus on controllability
We observe that as the neural generator is a continuous function , two close samples in latent space would be mapped into two nearby images while their quality can differ much as the quality is not a continuous function in pixel space
From the above continuous mapping function perspective on the other hand two distant latent samples are also possible to be mapped into two close images
If the latent samples are mapped in aggregation into limited modes or even a single mode mode collapse occurs
Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent z from its initial distribution e.g.
Gaussian
This is achieved by using the iterative fast gradient sign method (I-FGSM).
We further propose new GAN training strategies to obtain better generation mappings w r
.t quality and diversity by introducing targeted latent transforms into the bi- level optimization of
GAN.
Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity
I NTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014a) have shown effectiveness for generating high-ﬁdelity data, especially for images under different settings (Ledig et al., 2017; Wang et al., 2018; Zhu et al., 2017; Zhan et al., 2019; Chakraborty et al., 2018).
Based on the zero-sum game, the model is trained by the adversarial process between the generator and the discriminator.
Many efforts have been made to achieve a more realistic generation from different perspectives.
WGAN (Arjovsky et al., 2017), SNGAN (Miyato et al., 2018), LSGAN (Mao et al., 2017) aim to design better objective functions.
ImprovedGAN (Salimans et al., 2016), AC-GAN (Odena et al., 2017) propose practical training techniques.
While more complex network structures are studied in
BigGAN (Brock et al., 2018), StyleGAN
(Karras et al., 2019), SAGAN (Zhang et al., 2019).
Despite the above progress, an intriguing and relatively less studied question is how the latent space affects the generation in terms of quality and diversity, which can be orthogonal to the above frequently studied factors like model architectures and training techniques.
Speciﬁcally, we argue that the latent samples from a standard continuous distribution (e.g., Gaussian) can often be mapped to varying-quality samples for the generation.
The reason is that the generator is a continuous function (by neural nets) while the generation quality (e.g., images in pixel space) is not.
Moreover, it has also been shown in GAN literature (Khayatkhoei et al., 2018) that many image objects lie upon multiple disjoint manifolds, and a continuous mapping from continuous latent distribution, e.g., Gaussian must generate many invalid samples in the between of the manifolds in order to cover all the meaningful modes.
See Fig.
1 (a) for the schematic diagram.
This fact also suggests a dilemma: for complex real-world data, there is a tradeoff between quality and diversity when the latent vectors are sampled from a simple uni-mode distribution, e.g., Gaussian, and the generator is a continuous mapping based on neural networks.
Moreover, partly due to the mapping function that allows for many-to-one mapping, mode collapse (Srivastava et al., 2017) can also happen when different latent samples are mapped into limited or even a single mode in target space.
[Figure]
Due to the continuity of the neural network-based generator, close samples in latent space will be mapped to close images in pixel space, while the images’ quality can vary.
Cat (a) and Cat (b) are close in pixel space, but Cat (a) loses its right ear.
Our method allows bad latent z to transform towards the inverse image of the nearest real manifold, thus avoiding terrible generation.
[Figure]
The above contradiction between the discontinuity in generation data quality (e.g., images) and the continuity of latent distribution as well as mapping function, can be one of the signiﬁcant factors to the well-known difﬁculty in GANs training, especially for generators, as one unreasonably hopes to enforce a discontinuous mapping via continuous neural nets.
Meanwhile, the mapping property can largely affect the generative performance for functions with continuous nature, which is associated with most previous works devoted to improving generative performance.
To tackle the above issues, essentially, we impose an extra (implicit) transform function z ∗ ( · ) on the raw sampling z , and the generation can be written by G ( z ∗ ( z )) .
From this perspective, most existing works, including the vanilla GAN, employ z ∗ ( · ) with an identical form.
Speciﬁcally, the implicit function z ∗ ( · ) is achieved by updating the latent z ∼ p ( z ) to minimize loss G (i.e. log(1 − D ( G ( z ))) ) using iterative fast gradient sign method (I-FGSM)
(Kurakin et al., 2016) with the parameters of G and D ﬁxed, and we name the method as AdvLatGAN-z. See Fig.
1 (b) for the schematic diagram.
On top of the ﬁrst effort, we propose a novel training strategy to improve the generative mapping G respectively for quality and diversity.
See Fig.
2 for the schematic diagram.
This is achieved by introducing z iterations (i.e., targeted implicit transforms on z ) into GAN training, speciﬁcally, updating z ∼ p ( z ) using I-FGSM to ﬁnd more optimization-friendly ones in the bi-level optimization of vanilla GAN.
Respectively for quality and diversity, we use two iterative updating strategies for z under different objective functions in training, and we name the two training algorithms respectively as AdvLatGAN-qua and AdvLatGAN-div.
The highlights of this paper include:
We rethink the role of the generator as a continuous function, which may incur quality discontinuity in target space and mode aggregation (collapse) when the latent samples are directly sampled from a trivial Gaussian.
Based on above theoretical considerations we propose a composite generation perspective by formulating the mapping from G ( z ) to G ( z ∗ ( z )) .
This formula provides a better methodological guidance to the progress of the generative model. We propose a new pipeline for both G ( · ) and z ∗ ( · ) , whereby proposed training strategies ﬁrst train generative mappings respectively for better generation quality and diversity, and then the implicit transform z ∗ ( · ) is enforced and computed implicitly to obtain an improved distribution for a more effective generation.
Our training involves adversarial learning under different objective functions for quality and diversity, leading to new GAN models: AdvLatGAN-qua/div and their further variants.
3) Experimental results on synthetic and visual data show the notable improvement of our approach on both generation quality and diversity and our technique is orthogonal to other techniques.
R ELATED W ORK
Implicitly adapting the raw latent vectors z from Gaussian to improve the generation relates to latent space learning for GAN.
While the speciﬁc training algorithm using latent vector adapting to mine optimization-friendly samples in training relates to adversarial training, which views the latent adapting in training as an attack to mine hard samples.
Latent space exploration in GANs.
Techniques have been proposed for latent space exploration for GANs.
By utilizing the steerability of GANs in latent space, Jahanian et al. (2019) shifts the distribution to ﬁt camera movement transformation.
InterFaceGAN (Shen et al., 2020) decouples entangled semantics with subspace projection to control facial attributes more precisely.
In addition to semantics manipulating, some recent works focus on improving latent sampling quality to generate high-quality images.
For example, interactive evolutionary computation is used to make the generation process more controllable in Bontrager et al. (2018), which meanwhile also brings additional manual efforts.
To reduce manual intervention, Roziere et al.
(2020; 2021) use Koncept512 (Hosu et al., 2020) as a criterion to improve the quality of fake images.
However, as discussed in Roziere et al. , this strategy is somehow biased, can not work well in each dataset (e.g., Pokemons).
The reason is that the quality estimator is not even designated to access the real-data distribution information.
Instead, the generation is guided by a predeﬁned score function.
Compared to the above works, we take a relevant idea to Roziere et al. (2020) that also adapt the latent vectors to an improved condition before input to the generator G .
Differently, our latent adaption approach incorporates the target distribution and utilize I-FGSM iterations to adapt latent vectors.
Mitigating mode collapse for GANs.
There are also efforts in mitigating mode collapse, which are based on the thinking that many real-world samples lie in disjoint manifolds (Khayatkhoei et al., 2018).
One way is to introduce a collection of generators such that each generator may cover a speciﬁc mode since a single generator can hardly fulﬁll a discontinuous function (Khayatkhoei et al., 2018).
However, more works focus on improving the properties of mapping, explicitly obtaining a mapping covering more modes (Elfeki et al., 2019; Meulemeester et al., 2020; Mao et al., 2019).
Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping.
This calls for selective sampling of the continuous latent distribution or directly amending the latent distribution, which is rarely noticed in these works.
Adversarial samples and adversarial training.
Adversarial samples are malicious samples that can easily fool deep models, which are indistinguishable to human eyes.
Since the seminal work (Szegedy et al., 2013), various adversarial attack methods have been devised (Goodfellow et al., 2014b; Akhtar & Mian, 2018; Kurakin et al., 2016; Madry et al., 2017).
To defend the models from attacks, adversarial training is meanwhile developed (Goodfellow et al., 2014b; Wang et al., 2019; Nøkland, 2015).
In particular, it is worth noting that several recent studies have shown that adversarial training can bring many beneﬁts to GANs, not just improving robustness.
RGAN (Zhang et al., 2020) shows that adversarial training can enhance the generalization of the generator and discriminator.
Based on both theoretical analysis and empirical results, Zhong et al.
; Liu et al.
(2021); Liu & Hsieh (2019) show that updating discriminator’s parameters with the loss calculated by adversarial samples generated by authentic images can stabilize training and accelerate convergence.
In this paper, we utilize the adversarial sample search method, the iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016) to conduct latent transform and mine optimization-friendly z samples in the GAN training process, which matches up with the procedure of adversarial training.
While most existing works make perturbations on pixel space (real samples or generated samples), instead we focus on mining the latent space and making perturbations on latent samples.
P ROPOSED M ETHOD
This section proposes AdvLatGAN-z to obtain the effective transformation z ∗ ( · ) and proposes training algorithms AdvLatGAN-qua and AdvLatGAN-div to obtain generation mapping G with better generation quality (i.e., -qua) or diversity (i.e., -div).
Both methods are derived from the iterative updating of the latent space vectors.
3. P RELIMINARIES
We introduce adversarial samples and adversarial training used to derive z ∗ ( z ) and devise training algorithms, and the regularization technique in MSGAN (Mao et al., 2019) for mode seeking.
Adversarial samples and adversarial training.
Adversarial training aims to defend against adversarial attacks.
It uses adversarial samples to calculate the optimized loss.
In classiﬁcation, for each image-label pair ( x i , y i ) ∈ S from the labeled training set, the adversarial samples x (cid:48) is deﬁned as:
[Equation]
Here (cid:15) is the radius of the closed ball to constrain perturbation.
While f θ and l denote the attack target (a neural net) and the corresponding loss (e.g. cross-entropy loss).
p denotes the norm dimension.
Given in total n adversarial samples, adversarial training is fulﬁlled by the optimization:
[Equation]
Mode coverage by regularizing distance between generated samples. MSGAN (mode seeking GAN) (Mao et al., 2019) mitigates mode collapse in conditional generation setting by introducing a regularization term for training G over two latent samples z 1 and z 2 as follows.
The hope is that the two generated targets shall be far away from each other as much as possible:
[Equation]
where c is the condition vector for generation, and d
I and d z are the distance metrics in the target (image) space and latent space, respectively.
It encourages the distance to be maintained or even enlarged in the target space to avoid the generated samples being trapped into an aggregation.
By integrating the regularization term with the original objective function of cGANs (Mirza &
Osindero, 2014), the ﬁnal objective for minimization in MSGAN for training G and D is given by:
[Equation]
where L ori denotes the original objective, x denotes real data, and λ ms is the weight hyper-parameter.
Note since Eq.
3 is aimed to be maximized hence one can interpret λ ms to be negative.
3. L ATENT S PACE A DVERSARIAL S AMPLE S EARCH FOR G ENERATION Q UALITY
Input: p z e.g. Gaussian, p r e.g. real images distribution; randomly initialized G and D ;
Output: trained generator G and discriminator D ; 1: while G has not converged do 2: // update D: 3: for i = 1 to t D do 4:
Sample { z ( k ) 0 } mk =1 ∼ p z ; 5: //
the proposed sampling shift ( t -step I-FGSM): 6: for i = 1 to t z do 7:
Obtain { z ( k ) i } mk =1 by Eq. 5; 8:
Sample { x ( k ) i } mk =1 ∼ p r ; 9:
Calculate loss D with { x ( k ) i } mk =1 and { z ( k ) t } mk =1 ; 10:
Update D with loss D ; 11: // update G: 12:
Sample { z ( k ) } mk =1 ∼ p z ; 13:
Calculate loss G with { z ( k ) } mk =1 ; 14:
Update G with loss G ;
We show how to implicitly transform the latent raw samples from Gaussian to a new distribution via the idea of searching adversarial samples as introduced above.
This transform breaks the dilemma in diversity and quality where the bottleneck is due to the continuous latent distribution and a continuous mapping function.
Moreover, conducting the implicit transform in training to obtain the new (probably disconnected) latent distribution for each iteration can compensate for the difﬁculty for generators to align the pace of generators and discriminators 1 , which cannot map a continuous distribution to a disconnected one with multiple modes.
Speciﬁcally, we design the following constrained optimization task that encourages the new z ∗ to be close enough 2 to z .
Its prediction score (as real data) by the discriminator D should be as high as possible to improve the generation quality.
Recall otherwise the given z 0 may correspond to a poor generation as discussed in Sec. 1.
Note here that both G and D are ﬁxed.
[Equation]
where z 0 denotes the original latent vectors, z ∗ ( z 0 ) denotes the newly computed z , d ( · ) denotes the distance of two vectors which is an (cid:96) ∞ distance in this paper, in line with the popular protocol in adversarial attack literature (Goodfellow et al., 2014b).
By borrowing the well-developed tools in adversarial attack, the solution of the above problem can be estimated by the so-called iterative fast gradient sign method (I-FGSM)
(Kurakin et al., 2016):
[Equation] [Equation]
From the distribution perspective, the transform can be characterized by:
[Equation]
(7) where p
z denotes the original latent distribution, p ∗ z denotes the newly searched distribution,
D ( p z , p 0 z ) ≤ (cid:15) means that there is a limit to the range of distribution variation.
A newly searched z can be considered as a sample from p ∗ z if the sampling distribution is within the perturbation circle.
Fig.
4 shows the results by the pre-trained SNGAN model (Miyato et al., 2018)
(i.e. the given G and
D ) on STL-10 (Coates et al., 2011).
We develop a novel GAN training algorithm by conducting the above latent distribution shift to the initial sampling of Gaussian in D updating iterations, while the rest of the training algorithm remains consistent with vanilla GAN (Goodfellow et al., 2014a).
We present the algorithm in Alg. 1 which we call Adversarial Latent GAN (AdvLatGAN-qua) with a post meaning quality. The latent tranform is ﬁrst computed using the I-FGSM to ﬁnd the adversarial samples in latent space, then D and G are updated.
The iteration continues until enough steps or convergence of G . The above latent transform model does not consider how to avoid mode collapse, as each time only one latent sample is considered for optimization in isolation. Similar constrained optimization formulation can be designed to improve the generation diversity, as shown in the following subsection.
Input: p z e.g. Gaussian, p r e.g. real images distribution; λ ms is a hyper-parameter; randomly initialized G and D ; Output: trained generator G and discriminator D ; 1: while G has not converged do 2: // update D: 3: for i = 1 to t D do 4:
Sample { z ( k ) i } mk =1 ∼ p z ; 5:
Sample { ( x ( k ) i , c ( k ) i ) } mk =1 ∼ p r ; 6:
Compute loss D with { ( x ( k ) i , c ( k ) i ) , z ( k ) i } mk =1 ; 7:
Update D with loss D ; 8: // update G: 9:
Randomly generate labels { c ( k ) } mk =1 ; 10:
Sample { z ( k ) 0 } mk =1 ∼ p z ; 11: //
the proposed sampling shift ( t -step I-FGSM): 12: for i = 1 to t z do 13:
Obtain { z ( k ) i } mk =1 by Eq. 8; 14:
Calculate loss G and loss ms (Eq. 3) with { c ( k ) } mk =1 , { z ( k ) 0 } m k =1 and { z ( k ) t } m k =1 ; 15: loss ← loss G + λ ms loss ms ; 16:
Update G with loss ;
We take a challenging sample mining perspective to improve mode coverage.
Given a sample z 0 from the raw latent distribution, we aim to search its paired sample z ∗ to form a hard sample pair, in the sense that their generations are close in the target space given G .
Speciﬁcally, we further consider two aspects.
First, as hard samples, this pair of samples shall still belong to the same category c which can be modeled with conditional GAN.
Second, their distance in the latent space shall also be not too close otherwise, they may be aggregated in target space due to continuous mapping.
In other words, hard samples causing mode collapse shall be those close-in target space while apart in the latent space, when G is given.
Based on the above considerations, we adopt the form of Eq.
3 to design our search procedure as follows:
[Equation] [Table]
The symbols and terms are similar to Eq.
Akin to
AdvLatGAN-qua, we use I-FGSM to obtain z ∗ and again use (cid:96) ∞ to control change magnitude.
For each update step, we randomly select the ﬁrst latent vector z 0 , and the second vector z ∗ is searched by n iterations of the I-FGSM method under the (cid:96) ∞ restriction starting from the neighborhood of z 0 , and the iterations are guided by Eq. 8.
Table 1 shows the effectiveness of the above paired complex sample seeking approach, with experiments on CIFAR-10 (Krizhevsky et al., 2009) and Facades (Tyleˇcek & Šára, 2013).
The visual results are shown in Fig.
We can see that the pair obtained by solving Eq.
8 tend to collapse (as they are hard samples), while the opposite leads to better diversity by using the inverse form of Eq. 8.
The above experiments demonstrate the effectiveness of our iteration method, and we further introduce it into the initial training process of MSGAN (Mao et al., 2019).
In MSGAN, the regularization term 3 is used to regularize the model optimization process, where the selection of the pair is entirely random, without using any heuristic information.
We instead use the z pair that are more inclined to collapse by randomly taking one z and iterating by the method mentioned above to get another z .
We keep the other details of the algorithm unchanged.
By using Eq.
8 to generate adversarial sample pairs to draw the model’s attention to address mode collapse, we ﬁnally develop our new diversity enhance GAN training approach AdvLatGAN-div as shown in Alg. 2, which are further enhance by post-training as discussed in experiments.
E XPERIMENTS
[Table] Experiments are performed on a single GPU of GeForce RTX 3090.
All datasets are publicly available.
4. E XPERIMENTAL S ETUP
We validate the effectiveness of the proposed methods for two parts, respectively for effective latent sampling improvement z ∗ and better generation mapping G .
To be explicit, ﬁve methods can be derived for different targets, among which AdvLatGANqua+ and AdvLatGAN-div+ are the ﬁnal full version of our contributions. We present the methods below:
i) AdvLatGAN-z: post-training latent sampling improvement ﬁghting against quality discontinuity
; ii) AdvLatGAN-qua: GAN training
algorithm for
better generative quality
using intraining latent sampling improvement; iii AdvLatGAN -div:
GAN training algorithm for a more diverse generation using in-training latent sampling improvement; iv) AdvLatGANqua+: both in and post-training latent sampling improvement for generation quality; v) AdvLatGAN-div+: both in and posttraining latent sampling improvement for generation diversity.
For illustrative comparison, we show the schematic overview of these methods in Fig. 6 and Table 2. We adopt Inception Score (Salimans et al., 2017) and Fréchet Inception Distance (Heusel et al., 2017) to evaluate the performance, which utilizes the classiﬁcation results of the generated images by the Inception Network to evaluate the performance of the model.
A higher Inception Score or a smaller FID means better performance.
Following (Richardson & Weiss, 2018), we also use JSD to measure the similarity between generated images and real images, which ﬁrst clusters real images and generated images into the same number of classes using the K-means algorithm, then calculate the similarity between real and fake clusters w.r.t. the clustering property.
[Table] [Table]
[Table]
. 4. A DV L AT GAN- Z : P OST - TRAINING L ATENT S AMPLING I MPROVEMENT
This section shows the effectiveness of the post-training latent sampling improvement AdvLatGAN-z (denoted as z ∗ in previous sections, which is an implicit transform on the original latent sampling from Gaussian), which aims to ﬁght against the quality discontinuity.
Results on Synthetic Data.
In line with (Metz et al., 2017), we apply the proposed iteration strategy guided by Eq.
5 in both Grid and Ring datasets to evaluate its performance.
By updating latent space variables, we achieve highquality generated samples.
The result in Fig.shows the effectiveness of our method.
Results on MNIST.
MNIST contains 70,000 images of handwritten digits (LeCun et al., 1998).
Due to the low complexity of the dataset, the images can be generated with two-dimensional vectors.
The two-dimensional latent space probability density distribution can be visualized, granting us better observe the changes in latent distribution during the iterations.
We respectively conduct the experiments for the trained DCGAN model and WGAN-GP model, with single-step iteration constraint using (cid:96) ∞ and iterated step size (cid:15) is set to 0.03.
To better reﬂect changes in the distribution, we only show the ﬁrst 30 iterations that can reﬂect larger changes.
The changes of latent distribution and the generated images corresponding to the probability densities at the peaks and valleys of new latent distribution in the WGAN experiment are shown in Fig.
Results on CIFAR-10 and STL-10.
We equally conduct experiments on CIFAR-10 (Krizhevsky et al., 2009) and STL-10 (Coates et al., 2011).
CIFAR-10 has been widely used whose complexity is not high, but more diverse and closer to the quality of high-resolution images than MNIST.
STLhas a higher resolution and richer image representation.
We respectively use implicit transform to [Table] [Table] the latent space vectors for pre-trained DCGAN and SNGAN, with single-step iteration constraint using (cid:96) ∞ and iteration step size (cid:15) set to 0.01.
We conduct 100 steps for each iteration process.
Inception Score and Fréchet Inception Distance are used to evaluate image quality, and the metrics are calculated over 1500 generated images in STL-10 experiment and 3000 in CIFAR-10 experiment.
In both cases, we use two different initializations, each initialized randomly and not explicitly chosen.
It can be observed that our method has a signiﬁcant impact on generation quality, with a maximum improvement of 12.03% by Inception Score.
The result is shown in Table 3.
4. A DV L AT GAN- QUA / DIV : I MPROVING THE G ENERATION M APPING
This section shows the effectiveness of the GAN training algorithms, which aims to improve the generation mapping G for better performance respectively on quality and diversity.
AdvLatGAN-qua for Quality Improvement.
We conduct experiments on CIFAR-10 and STL-10, using the mainstream architectures such as DCGAN, WGAN, WGAN-GP, SNGAN.
We adversarially train GAN using a quality-enhanced strategy previously mentioned as AdvLatGAN-qua, and the results show that the proposed method can greatly improve the quality of generated images.
IS and
FID are adopted to evaluate the performance of the model.
We conduct only one latent iteration to save computation cost per generator step, and more details will be given in the appendix.
We also try to use implicit transform after training to improve the generation quality further.
The result is shown in Table 4 and Table 5.
It can be seen that the proposed method has a signiﬁcant improvement in generative performance compared to that without the latent space transformation strategy.
In the
WGAN STL-10 setting, IS reaches from 6.51 to 8.16, and FID reaches from 73.0 to 49.1.
AdvLatGAN-div for Diversity Improvement.
We conduct experiments on both CIFAR-10 and STL-10 for conditional generation and adopt FID and JSD as the metrics, respectively for the entire set of generated images and each class.
We test the training results on the basis of implicit latent transform to ﬁght against quality discontinuity.
We can ﬁnd that the proposed method is prior over MSGAN reﬂected by JSD and FID metrics.
The results are shown in Table 6 and Table 7.
C ONCLUSION
This work begins with two basic observations in GANs from the perspective of continuous mapping. Based on the observations, we introduce the adversarial attack techniques in GAN to explore latent space and derive novel training algorithms to improve the mapping property. With the efforts of both latent transform and mapping improvement, the proposed AdvLatGAN achieves effectiveness both for generation quality and diversity veriﬁed by extensive experiments. A PPENDIX A R ELATIONSHIP WITH A DVERSARIAL A TTACKS AND D EFENSES
Our approach is closely related to adversarial attacks and defenses.
As for works in adversarial attacks, the fundamental standpoint is that deep neural networks’ performance can vary signiﬁcantly in the face of small perturbations.
This scenario can be very similar to our starting point as described in Sec 1 that a small perturbation of the latent vectors can lead to massive quality variation.
Thus it is quite natural to combine the adversarial attacks and defenses with generative adversarial networks, and speciﬁcally, we utilize the latent transform using I-FGSM iterative method to overcome the quality discontinuity. Our GAN training algorithms AdvLatGAN-qua and AdvLatGAN-div can be analogous to adversarial training techniques in the adversarial defense ﬁeld. Adversarial training is a proactive defense approach that strengthens the model against attacks or enhances its performance by modifying the inputs to use adversarial samples to train the model, making the model naturally defensive against attacks. This process involves conducting adversarial attacks during training and can be explained as mining targeted optimization-friendly samples for training.
While in AdvLatGAN-qua and AdvLatGAN-div, we conduct z iterations in the GAN training referencing attack method, which can be viewed as the same process as adversarial training.
AdvLatGAN introduces the latent vector z iterations into the bi-level optimization between the generator G and the discriminator D .
The training involves iterations of three components z , G ,and D , among which the introduced z iterations can mine the latent space during training, to some extent leading to the superiority of the method.
B D ESCRIPTION OF C ONSTRAINTS FOR I TERATION
There are two basic types of constraint used in the ﬁeld of adversarial attack: 1) (cid:96) 2 norm constraint; 2) (cid:96) ∞ norm constraint.
In this paper, we follow the well-known attack method, iterative fast gradient sign method (I-FGSM)
(Kurakin et al., 2016) to conduct the iterations of latent variables and we adopt the (cid:96) ∞ norm as our constraint. C D IFFERENCES FROM O THER A LGORITHMS We present the differences between our approach and other previous works, involving two topics i.e. latent exploration and GAN model with adversarial training. An overview is presented in Table 8.
C. L ATENT E XPLORATION
EvolGAN (Roziere et al., 2020), Tarsier (Roziere et al., 2021)
These two models both use the well-known quality evaluator Koncept512 (which is actually a classiﬁer) to guide the latent iterations.
Their drawback is that they do not exploit the information of the real distribution, which leads to that the optimization criterion of sampling quality is not for the proximity to the real distribution, but to match the data trained for the Koncept512 classiﬁer.
Our approach uses a discriminator to guide the latent variable updating, which makes full use of the information from the real distribution, while further achieving novel training algorithm to improve the generative mapping.
DDLS (Che et al., 2020)
DDLS analyzes the sample-improving approach from the perspective of the energy-based model, which differs from ours in terms of methodology.
Besides, it didn’t notice that when we enhance the quality, we hope that the image is changed as little as possible.
Combing the above two points, we explain the problem from the perspective of attack, which is quite different from the specialized sampling approach Langevin Dynamics used in (Che et al., 2020).
C. GAN M ODEL WITH A DVERSARIAL T RAINING
Our GAN training algorithm can be considered as a kind of adversarial training from the perspective of adversarial attack and defense, and we differ from several previous works that combine adversarial training and GAN.
[Table]
FastGAN (Zhong et al., 2020), Rob-GAN (Liu & Hsieh, 2019)
FastGAN (Zhong et al., 2020) and Rob-GAN add perturbation to the real images (the latter is an improvement in the loss function compared to the former), while we focus on the latent space, perturbing latent space vectors.
Robust GAN training (Zhou & Krähenbühl, 2018)
This algorithm makes perturbations on the generated images, again without focusing on the latent space compared to us
[Table]
D D ETAILS OF L ATENT S AMPLING I MPROVEMENT E XPERIMENT D
XPERIMENTS .
We simulate two synthetic datasets.
Ring dataset is a mixture of 8 2-D Gaussians p ( z ) with mean { (2 cos ( iπ/ 4) , 2 cos ( iπ/ 4)) } 8 i =1 and standard deviation 0 .
12.5K samples are simulated from each Gaussian distribution.
50K samples from p ( z ) are used to generate x for test.
Grid dataset is a mixture of 25 2-D isotropic Gaussians i.e. p ( z ) with mean { (2 i, 2 j ) } 2 i,j = − 2 and standard deviation 0 .
4K samples are simulated from each Gaussian.
20K samples from p ( z ) are used to generate target samples { ˜ x } for test.
In the synthetic experiments, we apply fully-connected networks for generation and the architectures are shown in Table 9 and Table 10.
[Table]
[Table]
[Figure]
[Figure]
[Figure]
D ETAILS FOR R ING S YNTHESIZING E XPERIMENTS .
To better demonstrate the practical effect and to show the characteristics of the network during the training process, we choose 2000 as the total number of iterations (1 epoch), and the result is shown in the left picture in Fig.
We apply the latent transform technique for the network in the left picture: frozen all parameters of the neural network and iterating 20,000 points sampled in a 2D-Gaussian distribution under the constraint of L-inﬁnity norm.
The step size is set as 0.003 and its optimizing target is upgrading discriminator judged score.
After iterating 8,000 times, We get the high-quality samples in Fig.
9 (shown in the middle) which outperform the results after training 100 epochs.
It is a powerful evidence to prove the problem of imbalance in the input latent space and that discriminator tends to converge faster.
D. D ETAILS FOR G RID S YNTHESIZING E XPERIMENTS .
We try to simulate the case that the inappropriate selection of network architecture to prove the application prospects of latent transform in this situation.
All hyperparameters are consistent with the previous ring experiment besides choosing training 50 epochs to make sure the network is fully trained.
Because of the inappropriate choosing model architecture, the training result is terrible.
However, by applying with latent transform policy, almost all points converge towards target points.
The result is shown in Fig.
D. D ETAILS FOR MNIST E XPERIMENT .
The DCGAN model and WGAN-GP model are used in this experiment.
We use DCGAN model for generating images of CIFAR-10 and ResNet model for STL-10 dataset.
The hyperparameters and model structures are consistent with that used in https://github.com/w86763777/ pytorch-gan-collections with a total of 30 iterations, each with a step size of 0.03 and a batch size of 8000.
The conﬁguration of the trained DCGAN model: batch size is 128, D-learning rate and G-learning rate are both 0.0002, the loss function is bce loss, the discriminator updates one step per generator iteration, the latent dimension is 2, and the total training step is 20,000.
The conﬁguration of training WGAN-GP model: the batch size is 128, the D-learning rate and G-learning
[Figure]
rate are both 0.0002, the parameters of discriminator updates one step per generator iteration, the latent dimension is 2, and the total training step is 50,000.
Details for CIFAR-10 and STL-10 Experiment.
The SNGAN and WGAN-GP are used in this experiment, referencing https://github.com/w86763777/ pytorch-gan-collections with a total of 100 iterations, each with step size of 0.
and the batch size is set as 1500 for STL-10 and 3000 for CIFAR-10.
The conﬁguration of SNGAN for CIFAR-10: the batch size is 128, the D-learning rate and G-learning rate are both 0.0002, the loss function is hinge loss, the discriminator updates one step per generator iteration, the latent dimension is 100, and the total training step is 100,000.
The training conﬁguration of the trained WGAN-GP model for CIFAR-10: the loss function is Wasserstein loss and other conﬁgurations are the same as SNGAN.
The training conﬁguration of SNGAN model for STL-10: the batch size is 64, the D-learning rate and
G-learning rate are both 0.0002, the loss function is hinge loss, the discriminator updates 5 steps per generator iteration, the latent dimension is 128, and the total training step is 100,000.
The training conﬁguration of the trained WGAN-GP model for CIFAR-10: the loss function is Wasserstein loss and other conﬁgurations are the same with SNGAN.
The results are shown in Fig.
11, Fig.
E E XPERIMENT FOR A DVERSARIAL S AMPLE S EARCH FOR D IVERSITY
We propose AdvLatGAN-div to improve the mapping property for diversity, which introduces the latent z iterations by Eq.
8 into the original bi-level optimization process.
The validity of the algorithm requires the effectiveness of the z iterations searching for samples with varying diversity performance guided by Eq. 8, which we verify by experiments.
Details for CIFAR-10 Experiment.
We ﬁrst ﬁnd a random latent vector of batch size 64, respectively do multiple-step gradient sign descent and multiple-step gradient sign ascent on the ratio of the distance between the original image and the target image to the distance of original input vector and target input vector (which denoted as Eq.
The results are presented in Fig.
The number of
[Figure]
iterative steps is set to 300 and the step length is 0.03.
The model is a pre-trained CIFAR-10 DCGAN model.
Details for FACADES Experiment.
Under the condition of the same image input, 20 latent space vectors are randomly generated to complete the generation, and each vector is updated in two directions to obtain 20 sets of comparison images.
F D ETAILS OF A DVERSARIAL L ATENT GAN E XPERIMENT F. E XPERIMENT FOR I TERATIVE Q UALITY I MPROVEMENT F.1. D ETAILS FOR CIFAR-10 E XPERIMENT
We regard latent space variables as iterable parameters and in case of divergence, we apply learning rate (cid:15) for latent vector updating process.
And the hyperparameters we use in this experiment are
[Table]
[Table]
shown in Table 13.
We use basic convolutional neural networks for test.
The architectures are presented in Table 11 and Table
[Table]
[Table]
[Table]
[Table]
The learning rates for updating latent variables in STL-10 experiment are presented in Table 14.
We use basic convolutional neural networks for test.
The architectures are presented in Table 15 and
Table 16,
F. E XPERIMENT FOR I TERATIVE D IVERSITY I MPROVEMENT F.3. D ETAILS FOR CIFAR-10 E XPRIMENT
The experiment is obtained based on the experimental extension of MSGAN (Mao et al., 2019), and the operation of latent variable search is added to the original design.
The latent vector is updated in the experiment guided by Eq.
8 with a gradient update strategy constrained by (cid:96) ∞ .
In our experimental setting, we adopt 3 updates per iteration and the step size is set as 0.01.
Other training parameters are referred to the original MSGAN experiment https://github.com/HelenMao/MSGAN without modiﬁcation.
The FID and JSD metrics for each class are calculated based on 5000 images generated for each class, and the total FID and JSD metrics are calculated by the collection of generated images for each class mentioned above.
F.3. D ETAILS FOR STL-10 E XPRIMENT
The basic setup of this experiment follows the CIFAR-10 experiment, and one difference is that: we use the supervised test part of STL10 as the real dataset for training and calculate the JSD and FID metrics for the generated images and the labeled training part of STL10.