U NDERSTANDING AND S TABILIZING GAN S ’ T RAINING D YNAMICS WITH C ONTROL T HEORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks (GANs) have made signiﬁcant progress on realistic image generation but often suffer from instability during the training process.
Most previous analyses mainly focus on the equilibrium that GANs achieve, whereas a gap exists between such theoretical analyses and practical implementations, where it is the training dynamics that play a vital role in the convergence and stability of GANs.
In this paper, we directly model the dynamics of GANs and adopt the control theory to understand and stabilize it.
Speciﬁcally, we interpret the training process of various GANs as certain types of dynamics in a uniﬁed perspective of control theory, which enables us to model the stability and convergence easily.
Borrowed from control theory, we adopt the widely-used negative feedback control to stabilize the training dynamics, which can be considered as an L 2 regularization on the output of the discriminator.
We empirically verify our method on both synthetic data and natural image datasets.
The results demonstrate that our method can efﬁciently stabilize the dynamics and converge to better results.
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have shown promise in generating realistic natural images (Brock et al., 2018) and facilitating unsupervised and semi-supervised learning (Chen et al., 2016;
Li et al., 2017; Donahue & Simonyan, 2019).
In GANs, an implicit generator ( G ) is deﬁned by mapping a noise distribution to the data space.
Since no density function is deﬁned for the implicit generator, a discriminator ( D ) is introduced to estimate the density ratio between the data distribution p D and the generating distribution p G by telling the real samples from fake ones.
G aims to recover the data distribution by maximizing this ratio.
This framework is formulated as a minimax optimization problem, which can be solved by optimizing G and D alternately.
In practice, however, GANs can be extremely sensitive to hyperparameters (Lucic et al., 2018;
Radford et al., 2015), and oscillations are often observed (Liang et al., 2018; Chavdarova &
Fleuret, 2018), thereby suffering from the instability of training (Goodfellow, 2016).
There are some theoretical analyses aiming to understand and stabilize the training process of GANs, such as designing different objective functions using other statistical divergences (Nowozin et al., 2016; Nguyen et al., 2017; Du et al., 2018; Mao et al., 2017), and introducing auxiliary regularization terms (Gulrajani et al., 2017; Arjovsky et al., 2017).
However, there is a gap between the theoretical analyses and practical implementations: most of previous work assumes that (1) D achieves optimal when training G (Goodfellow et al., 2014; Arjovsky et al., 2017; Gulrajani et al., 2017; Nowozin et al., 2016); and (2) the optimal D is a smooth function of G (Metz et al., 2016).
These two assumptions are violated in most practical cases.
First, GANs are optimized by alternating between k steps of optimizing D and one step of optimizing G , where k generally takes 1 or 5 .
It results in a sub-optimal
D , especially in the early stage of the training process.
Second, D is constant w.r.t.
G when G is optimized, which results in missing gradients and an unstable training process (Metz et al., 2016).
Instead, to ensure practical convergence, it is the training dynamics of both G and D , i.e., how G (or
D ) changes given current G and D during the training process, that should be considered, where the previous two assumptions are no more required.
There are some methods that focus on the dynamics of the parameters.
Mescheder et al.
(2017) and Nagarajan & Kolter (2017) propose to model the dynamics of the parameters using the vector ﬁeld deﬁned by its gradients.
In such cases, the local convergence and stability are fully determined by the eigenvalues of the Jacobian matrix.
Mescheder et al.
(2018) further propose a regularization term to stabilize GANs by adjusting the eigenvalues of the Jacobian matrix.
Gidel et al.
(2018) analyze the effect of momentum and propose to stabilize GANs using negative momentum.
In these methods, the Jacobian matrix is the key to understand and stabilize the dynamics, whose calculation is, however, computationally expensive.
It largely impedes us to generalize this analysis to complex models such as neural networks (LeCun et al., 2015).
In this paper, we argue that modeling the dynamics in the function space is more convenient for convergence analysis.
Speciﬁcally, we directly model G and D as two dynamics whose output is considered as a signal of time t , i.e., representing the output of G and D as G ( t, z ) and D ( t, x ) respectively.
The control theory (Kailath, 1980) provides a powerful tool to understand them and analyze their stability.
Speciﬁcally, differential equations can be used to denote the dynamics, e.g., the dynamics of G can be denoted as dG ( t,z ) dt = f
( G, D ) , which represents how G changes given current G and D .
By applying Laplacian transformation (Widder, 2015) to the differential equations, the dynamics can be represented as a transfer function, and then the stability and convergence can be easily modeled (Kailath, 1980) as introduced in Sec. 2.
Under the above perspective of control theory, we unify the dynamics of G and D as certain dynamics that are well-studied in control theory for various GANs, including Standard GAN (SGAN), WGAN and LSGAN (Mao et al., 2017), as illustrated in Sec. 3 and Appendix A. Speciﬁcally, the training process of GANs can be considered as a dynamic whose output is determined by its intrinsic properties (i.e., the objective function of G and D ) and its input (i.e., the data distribution).
Within the proposed framework, a variety of existing methods (Mao et al., 2017; Mescheder et al., 2018; Gidel et al., 2018) can be considered as certain controller which is widely used in control theory as discussed in Sec. 4.1 and Appendix C. It is worth noting that through control theory, the stability of
GANs can be easily inferred from the transfer function, instead of analyzing the complicated Jacobian matrix of the dynamics.
The proposed framework provides a promising direction that can further beneﬁt the training dynamics of GANs using advanced control methods.
To verify it, we propose to use the widely adopted controlling method, the negative feedback (NF) control (Åström & Hägglund, 1995), to stabilize GANs.
What’s more, when applying NF to stable models, the performance can still be improved.
The proposed NF acts as a regularization term which penalizes the L 2 norm of the output of D as we described in Sec. 4.
NF is veriﬁed on the toy data such as Dirac GAN as well as the natural images such as CIFAR10 (Krizhevsky et al., 2009).
The results demonstrate that our method can successfully stabilize the dynamics of GANs and outperform the baseline signiﬁcantly.
P RELIMINARY
In this section, we provide the background and preliminary about Laplacian transformation (LT) and control theory.
LT is a powerful tool to present an ordinary differential equation (ODE) as a rational fraction.
Since most dynamics can be represented as an ODE, LT can largely simplify the analysis of stability by merely analyzing the properties of the rational fraction.
2. L APLACIAN T RANSFORMATION
Laplacian transformation (LT) (Widder, 2015) can be considered as an extension of Fourier transformation (FT), which transforms a signal, i.e., a function of time, to a function of complex variables.
Formally, the transformation and its inversion are given by:
[Equation]
where f is a signal that is represented as a function over time t , c is a constant that ensures F ( s ) ≤ ∞ .
s ∈ C is a complex number, i.e., s =
σ + ωi with real numbers σ and ω and for each s ∈ C , F [Equation] ) denotes the components of certain frequency in the original signal f similar to FT.
With LT, the integration and derivation can be presented as a single operator, which is given by:
[Equation]
With this property, an ODE can be converted to an algebraic equation by substituting derivation as an operator s .
A simple example is illustrated as follows:
[Equation]
𝐻 𝑓 𝑠
= 𝑠 2 + 1 𝑈(𝑠) 𝑋(𝑠) 𝑈(𝑠) 𝑋(𝑠)
𝐻 𝑓 𝑠 𝐻 𝑐 𝑠 𝐻 𝑏 𝑠 + - where u and x are the input and output signal correspondingly
and U and X are their Laplacian transformation. This ODE can represent a spring dynamic, whose acceleration d 2 x dt 2 depends on the input u and the current position x according to Hooke’s law.
In this case, the ODE can be solved by simply applying the inverse Laplacian transformation to X =
[Equation] U ( s ) .
Here
H f ( s ) =
1 s 2 +1 is the transfer function of this dynamic which can also be represented as a diagram illustrated in Fig.
Since the input signal can be considered as summation of inﬁnity Dirac signal at different positions, i.e., f ( t ) = (cid:82) f ( u ) δ
( t − u ) du where δ denotes the Dirac delta function (Dirac, 1981), we only need to analyze the output of the dynamics for Dirac signal whose LT is constant, i.e.,
F ( δ ) = 1 .
Therefore, the transfer function represents the intrinsic properties of the dynamics.
2. C ONTROL T HEORY AND N EGATIVE F EEDBACK
Given a dynamic, or equivalently a transfer function, the stability can be analyzed according to the roots of the denominator of the transfer function, e.g., s 2 + 1 = 0 for Eqn.
The roots are called the poles of the dynamics.
Speciﬁcally, if all poles’ real parts are negative, the dynamic is stable; if there are poles which are pure imaginary numbers, the dynamic will oscillate; if there is at least one pole with positive real parts, the dynamic will diverge (Kailath, 1980).
In the above example, the poles of the dynamic are 0 ± i , which indicates that the object will oscillate around the original.
For those unstable dynamics, we need to stabilize it by adjusting the poles of the dynamics.
One of the most commonly adopted methods here is introducing negative feedback with an auxiliary controller (Kailath, 1980).
The diagram is illustrated in Fig.
1 (right).
The H b ( s ) denotes the transfer function of feedback and H c ( s ) denotes the transfer function of controller.
In this setting, the dynamic takes the error (i.e., E ( s ) =
U ( s ) − H b ( s ) X ( s ) ) rather than U ( s ) as input, resulting the fact that the transfer function of the whole controlled dynamic is given by:
[Equation]
which provides us an approach to adjust the poles of an unstable dynamic.
For most cases, simply adopting the negative feedback, i.e., let H b
= H c = 1 , is enough to stabilize a dynamic.
In the following, we provide the dynamics of GANs in the perspective of control theory and introduce the negative feedback control to stabilize the training process of GANs.
U NIFYING THE D YNAMICS OF GAN S IN C ONTROL T HEORY
In this section, we provide a novel perspective on the dynamic of GANs.
Unlike previous methods that mainly focus on the dynamics of the parameters, we directly model the output of two dynamics
G and D in the function space.
In this part, we mainly focus on the Dirac GAN as in Mescheder et al.
and unregularized WGAN.
This method can be extended to other GANs easily, as we illustrated in Appendix A.
To model the dynamics of GANs, we ﬁrst denote the output of G and D as a function with respect to time t , which is also known as a signal, i.e., D = D ( t, x ) and G = G ( t, z ) .
Then the training process can be represented as a dynamic using an ODE, and the equilibrium is equivalent to the ﬁnal value of the dynamics (Oppenheim et al., 1998), i.e., the output of the dynamics with t → ∞ .
3. T HE D YNAMICS OF D IRAC GAN
In Dirac GAN, the data distribution is deﬁned as p D ( x ) =
δ c ( x ) =
δ ( x − c ) which is the density of an idealized point mass at point c .
The generator distribution is deﬁned as p G ( x ) =
δ θ ( x ) , where θ is the learnable parameter of G .
In this case, we use G ( t ) =
G ( t, z ) for simplicity as its output does [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
not depend on z . The discriminator is a simple linear function D ( t, x ) = φ ( t ) × x . Therefore, the minimax optimization problem of Dirac GAN is formulated as:
[Equation]
For the discriminator, the objective function is given by: max φ U ( D ) = cφ − θφ , which indicates that the dynamics of D can be formulated as:
[Equation]
By considering c − θ as the error signal, i.e., e = c − θ , the dynamics of the discriminator can be modeled as an integral part D ( t, x ) =
x (cid:82) t 0 c − G
( t ) dt .
By applying LT to the dynamic, we have:
[Equation]
where we consider the dynamics of D at each x .
For the generator, the objective function is given by max G U ( G ) = θφ .
Therefore the dynamics of G can be formulated as:
[Equation]
Since D ( t, x ) = x (cid:82) t 0 c − G ( t ) dt , we have ∂D
[Equation]
=
[Equation] We therefore combine the x part in D ’s dynamic and the the partial derivation in G into an identity mapping.
Then D can be considered as an integral dynamic and its transfer function is given by H D ( s )
= 1 s .
Further, the generator also integrate the output of D , which is also 1 s .
Hence, the block diagram of the Dirac GAN is illustrated in the left panel of Fig.
As we can see,
D takes the difference between the real sample and fake sample, i.e., c − θ as input, and output the integration of the input signal.
The generator takes D ’s output as input and output the generator distribution which is the integration of D ’s output.
Remark 1. When treating G and D as dynamics, the both the inputs and outputs of the dynamics are functions over t .
For example, the input of D is p D ( t, x ) − p G ( t, x ) and the output is D ( t, x ) .
3. T HE D YNAMICS OF WGAN
In this subsection, we further model the dynamics of WGAN (Arjovsky et al., 2017) without Lipschitz continuity constraints in the function space.
In this case, we consider the nonparametric setting, where D is a piece-wise linear function, G is a mapping from the noise space to the data space, and both of them are of inﬁnite capacity.
This analysis can be extended to other variants of GANs easily.
In the general case, the objective function of D is given by:
[Equation]
According to the calculus of variations (Gelfand et al., 2000), the gradient of U ( D ) with respect to the function D , and the dynamic of D are given by: [Equation]
[Equation]
Since the input signal is p D − p G , the dynamics of WGAN is similar to that of the Dirac GAN, whose transfer function can be denoted as H D ( s )
= 1 s .
For the generator, the objective function and its dynamics are given by:
[Equation]
It is worth noting that compared to Dirac GAN, this integral is taken in the function space, i.e., G ( t + δ t , z ) =
G ( t, z )+ δ t p z ( z ) ∂D
( G ( z ))
∂G ( z ) .
Besides, the generator distribution p G is not a Dirac distribution, resulting in that another operation, which converts the generated samples to the distribution, should be added after G to make the model consistent.
The diagram blocks of WGAN is illustrated in
Fig.
2 (right).
Compared to Dirac-GAN, another two operations are introduced to the framework.
We further provide the dynamics of other GANs, including standard GAN proposed by Goodfellow et al.
and Least-Square GAN proposed by Mao et al.
(2017) in Appendix A. Remark 2.
We only consider the unregularized WGAN to simplify the analysis.
Note that the dynamics of D follows Eqn. (10) at least locally around the equilibrium where updating D according to Eqn. (10) will not violate the Lipschitz constraints. Formally proof is given in Appendix D.
Remark 3.
Here we model the dynamics of G and D in the functional space, which is more convenient for the following stability analysis and designing controllers.
In practice, we update the models in the parameter space to approximate the dynamics in Eqn.
(10) and Eqn.
U NDERSTANDING AND S TABILIZING GAN S WITH N EGATIVE F EEDBACK
In this section, we ﬁrst provide the stability analysis of GANs from the perspective of control theory.
Besides, many previous methods can be interpreted in this perspective, and we use the momentum as an example.
Other further examples are given in Appendix C. We further propose to stabilize the dynamics of GANs using the negative feedback controlling methods, and an example of another controller is given in Appendix B.
4. S TABILITY
In the dynamics of GANs, we argue that it is dominated by the two integral dynamics of G and D .
Therefore, we ignore the two non-linear operations in GANs in the following analysis.
Since both
G and D can be considered as integral parts, we can formulate the transfer function of G and D as
H G = H D = 1 s .
According to Eqn.
(4), the transfer function of WGAN is given by:
[Equation]
It is worth noting that the poles of the transfer function of the whole dynamic are 0 ± i , which indicates that the output will oscillate around the equilibrium rather than converge to 0.
This result is also consistent with the analysis in (Mescheder et al., 2018).
This perspective can also partially explain why larger learning rates or adopting multiple update steps for D , where the transfer function of D can be approximated by ks with k > 1 , cannot provide a principle solution for stabilizing GANs.
In this case, the poles of the dynamics are adjusted to 0 ± √ ki which is still an oscillation dynamics.
4.1. T HE A FFECT OF M OMENTUM
Our method can also provide stability analysis for other practical techniques used in training GANs.
In this part, we use the momentum as an example.
Gidel et al.
(2018) provide some theoretical analysis of momentum in training GANs, which is formulated in the parameter space.
In the following, we re-analyze the momentum in the function space under the perspective of control theory.
The momentum method (Qian, 1999) achieves great success in training deep neural networks.
Previous methods such as SGD with momentum can help neural networks to escape from local optima.
Its theoretical formulation is given by:
[Equation]
The β is the coefﬁcient for the exponential decay of the gradients, which generally takes 0 .
However, things changed when it comes to training GANs.
DCGAN (Radford et al., 2015) recommends a smaller momentum with β = 0 .
Furthermore, recent state-of-the-art models (Mescheder et al., 2018; Brock et al., 2018; Gulrajani et al., 2017) just remove the momentum. [Figure] 𝑥) +
[Figure]
𝑥)
[Figure] Estimate the objective of D: [Equation] Update φ to maximize U ( D ) with learning rate η .
Estimate the objective of G : U ( G ) = 1 N (cid:80) x ∈{ x f } D ( x ) .
Update θ to maximize U ( G
) with learning rate
η
11: until
Convergence
In control theory, the momentum is equivalent to adding an exponential decay to the input of the dynamics (An et al., 2018), and the LT of an exponential decay dynamic is 1 s + τ .
τ > 0 denotes the decay coefﬁcient which depends on β .
Therefore, the transfer function of the D with momentum is
H mD ( s ) =
1 s ( s + τ ) , and the transfer function of the WGAN with momentum is given by:
[Equation]
By letting ( s + a )( s + b )( s + c ) =
s 3 + τs 2 +1 , we have that a + b + c = τ > 0 , which indicates that there are at least one pole of this dynamic whose real part is larger than 0 , indicating the instability of the dynamics for GANs with momentum.
4. S TABILIZE GAN S
According to the above analysis, for both the unregularized WGAN and the GAN with momentum, the equilibrium point (i.e., p G = p D ) is not a stable point.
In the following, we propose to stabilize
GANs using negative feedback control, which is the most widely-used method in control theory.
In this setting, we set H NF =
λ which is the transfer function of the negative feedback for D .
After applying the negative feedback to D , The diagram of the controlled dynamics is given in Fig.
3 (a).
According to Eqn.
(4), the transfer function of D and the whole dynamic can be represented as:
[Equation]
Therefore, for a positive and small λ , the real part of the two poles are − λ 2 < 0 , which indicates that the dynamic is stable.
The dynamics of the Dirac GAN using different λ is illustrated in Fig.
3 (b).
As we can see, when no feedback is applied (i.e., λ = 0 ), the dynamic will oscillate around the equilibrium.
Applying negative feedback can signiﬁcant improve the stability of the dynamics.
4.2. P RACTICAL I MPLEMENTATION
The most direct way to add negative feed is to introduce a regularization term R D = λ 2 (cid:82) D 2 ( x ) dx to the objective function of D .
However, directly integrating over the input space is computationally infeasible.
Instead, we give an approximation to R D .
We assume that the area where | D ( x ) | is large mainly concentrates on its training samples, i.e., previous samples used to train D .
Therefore, we keep two buffers B r and B f of size N b to store the old real samples and fake samples, respectively.
Then R D is evaluated on these two buffers to regularize D , and the two buffers are updated with replacement at each step.
The training procedure is illustrated in Alg.
In the following, we use
NF-GAN ( · ) to denote our proposed method with the hyperparameters λ/ 2 denoted in the parentheses. [Figure]
R ELATED W ORK
GANs are known for its unstable training process.
Radford et al.
(2015) provide several practical techniques to stabilize the training, such as designing suitable architecture of neural networks and the hyperparameters selection.
The Unrolled GAN (Metz et al., 2016) proposes to make D as a function of G by unrolling the training process of the discriminator where the gap between the theoretical analysis and implementation can be reduced.
Arjovsky et al.
(2017) provide a theoretical analysis of the gradient vanishing problem and further propose to optimize GANs using Wasserstein distance.
Gulrajani et al.
(2017) use the gradient penalty to ensure the Lipschitz continuity in discriminator, which is required by WGAN.
However, GANs still suffer from instability in the training process.
Besides, there is some work that directly focuses on the stability of the training process.
Except for the works mentioned in Sec. 1, Peng et al.
(2019) also directly model the dynamics of the parameters and adjust the update rules to stabilize GANs.
Gidel et al.
(2016) propose to ﬁnd the saddle point of the optimization problem using Frank-Wolfe algorithms. Feizi et al.
(2017) also use the control theory to model the stability of GANs, while the dynamics of GANs are not explicitly captured.
Balduzzi et al. (2018) decompose dynamics into the oscillating and non-oscillating behaviour and model training dynamics as ﬁnding stable ﬁxed points. However, none of them only can generalize their methods to complex datasets such as natural images.
E XPERIMENTS
In this section, we empirically verify our proposed method on the widely-adopted CIFAR10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015) datasets.
We denote the regularization proposed in Mescheder et al.
as Reg-GAN, which is the direct baseline of our method.
Besides, we also apply NF-GAN to SN-GAN (Miyato et al., 2018) and provide a signiﬁcant improvement to the state-of-the-art method.
For fair comparisons, our implementation mainly follows the ofﬁcially released code for Reg-GAN and SN-GAN, respectively.
More details about the experimental setting and further results on a synthetic dataset can be found in Appendix B.
The code is provided here.
6. C OMPARISON TO R EG -GAN
[Table]
We ﬁrst compare our proposed method with RegGAN.
Since for large datasets, the non-linearity cannot be ignored, which results in the gap between the simulation of Dirac GAN and our methods.
A small λ/ 2 , such as 0 . 01 which is suitable for Dirac
GAN is not enough for stabilize the large models.
We therefore select the coefﬁcient λ/ 2 among { 1 , 2 , 5 , 10 , 15 , 20 } .
In this part, we only report representative results.
Though our analysis mainly focuses on the WGAN framework, NF can also easily generalize to SGAN whose stability can also be improved and NF-SGANs with different coefﬁcient λ/ 2 are evaluated.
We use the Inception Score (IS) (Salimans et al., 2016) to evaluate the image quality on CIFAR10 and FID score (Gulrajani et al., 2017) on both CIFAR10 and CelebA. The quantitative results are given in
Table 1 and the learning curve is reported in Fig.
Our method gives an improvement to the FID and IS for both WGAN and SGAN.
Speciﬁcally, for unregularized methods, the models demonstrate the unstable training process and ﬁnally diverge away from the data distribution.
Compared with Reg-GAN, our method maintains stability and provides better convergence results, as also shown in Table 1 (top).
Besides, our method is more computationally efﬁcient, since regularizing the output of D needs less computation compared to regularizing || ∂D ( x ) ∂x || and this gap is signiﬁcant for large dataset.
Speciﬁcally, on Geforce 1080Ti, our method can conduct approximate 12 / 8 iterations per second of training for CIFAR10/CelebA, whereas Reg-GAN can only conduct 10 / 4 iterations per second.
What’s more, our method is robust to the choice of the hyperparameter λ .
We present the generated images in Fig.
5 and Fig.
6. C OMPARISON TO SN-GAN
We further apply NF-GAN to the SN-GAN’s architecture. With stability guaranteed, our proposed method can further help SN-GAN to address the potentially unstable issue and converge to better ﬁnal results, as illustrated in Table 1 (bottom). We manually select λ/ 2 ∈ { 0 . 01 , 0 .
1 , 1 } and ﬁnally adopt λ/ 2 = 0 .
Though SN-GAN provides an elegant normalization method to ensure the Lipschitz continuity, SN-GAN with WGAN loss still cannot converge to the data distribution and provide unreasonable results, which veriﬁes with our stability analysis.
Instead, the hinge loss, as introduced
Eqn.
(16)&(17) in Miyato et al.
(2018), is used to stabilize SN-GAN but no theoretical guarantee is provided.
In contrast, NF-GAN can successfully regularize SN-GAN with WGAN loss and provide better results to previous state-of-the-art results.
What’s more, NF-GAN can also beneﬁt SN-GAN with Hinge loss and provide a signiﬁcant improvement the IS of CIFAR-10.
C ONCLUSION
In this paper, we propose a novel perspective to analyze the dynamics of GANs.
Instead of focusing on the equilibrium, we directly model the dynamics of the discriminator and the generator during the training process.
Using Laplacian transformation and control theory, the stability can be easily understood according to the poles of the dynamics.
Various methods can be motivated from this perspective, and we verify it with the widely used negative feedback control. The empirical results demonstrate that our method can successfully stabilize GANs and provide better convergence results.
Negative feedback shows promise results, but further analysis is required for better results.
For one thing, our analysis mainly focuses on the continuous cases, where the practical implementation optimizes both G and D in discrete time steps, where the Z -transformation provides better tools for discrete-time dynamics.
For another, two approximations are made: using the update in the parameter space to approximate the dynamics and ignore two non-linear operations in GANs.
Recent analyses of GANs on the functional spaces (Johnson & Zhang, 2018) provides a promising solution to solve the ﬁrst approximation and modern control theory and non-linear control methods (Khalil, 2002) can be adopted to solve the second one.
Indeed, our method does not provide a perfect solution to GANs’ dynamics.
It does provide a novel perspective to understand and stabilize GANs.
A GAN S AND THEIR CORRESPONDING DYNAMICS
In this section
[Equation]
we provide an illustration for different kinds of GANs and their corresponding dynamics.
A. V ANILLA GAN AND N ON - SATURATION GAN
The GANs proposed by (Goodfellow et al., 2014) are formulated as follows:
[Equation]
Therefore
[Equation]
the objective function and the corresponding dynamics of D are given by:
• The objective function:
[Equation]
• Given current G , the gradient of U ( D ) with respect to D at certain x is given by:
[Equation]
The objective function and the corresponding dynamics of G are given by:
• The objective function:
[Equation]
• Since G is optimized using reparameterization, we rewrite G ( z ) as x z for simplicity.
Given current D , the gradient of U ( G ) w.r.t.
G at a certain z is given by:
[Equation]
For a well-trained D , which can successfully tell real samples from generated samples with high conﬁdence, i.e., D ( x z ) ≈
for all z , the gradient for G will tend to zero almost everywhere, which will result in gradient vanishing.
This analysis is consistent with previous literature (Johnson &
Zhang, 2018; Arjovsky et al., 2017).
For non-saturation GAN, the objective function and dynamics of G are given by:
• The objective is given by:
[Equation]
• We rewrite G ( z ) as x z and provide the gradient of U ( G ) w.r.t.
x z as follows:
[Equation]
A. L EAST -S QUARE GAN
(Mao et al., 2017) propose to use the least square loss to train both the discriminator and the generator.
The objective for the D and G are selected differently according to its theoretical analysis.
Speciﬁcally, the objective function of D and G are given by:
[Equation]
Therefore
[Equation]
the dynamics of the D and G are given by:
• The dynamic of D :
[Equation] [Equation]
• The dynamic of G :
[Equation]
It is worth noting that if the G is ﬁxed, the dynamics of D can be considered as an exponential decay which is the same as the D with negative feedback control.
However, since G is changing during the training process, the dynamics of D is actually coupling with the dynamics of G , which generally results in the unstable dynamics (Kailath, 1980).
The experimental results on CIFAR also verify this argument.
It also explains why we use two buffers of old samples to approximate the regularization term instead of directly regularizing the output of D using current samples.
B F URTHER E XPERIMENTAL R ESULTS B. D ETAILED E XPERIMENTAL S ETTINGS
In this paper, we mainly focus on the stability of the training dynamics of GANs, and therefore directly compare our proposed method NF-GAN with the vanilla SGAN/WGAN and the most direct baseline Reg-GAN. We use the resnet (He et al., 2016) for both the generator and the discriminator following Mescheder et al. and adopt the ReLU activation (Glorot et al., 2011).
The batch size is 64, and the buffer size is set to 100 times of the batch size for all settings.
We use rmsprop (Hinton et al., 2012) optimizer with learning rate as 0 . 0001 and α = 0 .
99 for all models.
Note that we make the following two modiﬁcations to the original implementation of Reg-GAN, which results in a more challenging setting to the stability.
First, in the original Reg-GAN, a
[Figure]
coefﬁcient 0 .
1 is introduced to the residual connection of the ResNet for both G and D. Speciﬁcally, h L +1 =
h L + 0 .
1 ∗ f L ( h L ) where f L is the transformation deﬁned in the L -th layer and h L is the output of L − 1 -th layer as well as the input of the L -th layer.
We instead remove the coefﬁcient and just use the regular ResNet, i.e., h L +1 =
h L + f L ( h L ) .
Second, for both G and D, we use the ReLU activation, rather than leaky ReLU.
The above two modiﬁcations make the training of GANs more difﬁcult, which explains why the reported results are not consistent with the scores in the original paper.
In this more challenging setting, our method still outperforms the Reg-GAN.
Speciﬁcally, the inception score of Reg-GAN is below 6.0 in this difﬁcult setting and about 6.5 in the original setting.
In contrast, both NF-WGAN and NF-SGAN achieve more than 6.7 in this difﬁcult setting.
B. S YNTHETIC D ATA
In this section, we evaluate our proposed method on a mixture of Gaussian on the two dimensions.
The data distribution consists of 8 2D isotropic Gaussian distributions arranged in a ring, where the radius of the ring is 1 , and the deviation of each component Gaussian distribution is 0 .
For the coefﬁcient λ/ 2 , we follow the theoretical analysis and set it among λ/ 2 ∈ { 0 . 01 , 0 . 05 , 0 . 1 } .
We adopt two-layer MLPs for both the generator and the discriminator whose hidden units are 128 and 512 respectively.
The batch size is is 512 and other settings just follow the Reg-GAN.
The generated results are illustrated in Fig.
and we further provide the dynamics of the generator distribution in Fig. 8 in Fig.
As we can see, the unregularized WGAN and SGAN suffer from severe model collapse problem and cannot cover the whole data distribution.
Besides, the oscillation can be observed during the training process of WGAN: the generator distribution oscillates among the modes of data distribution.
Our method can successfully cover all modes compared to the WGAN and SGAN and is comparable to Reg-GAN.
Besides negative feedback, the most commonly used controller is the PID controller, which consists of three parts: the partial (P), integration (I) and derivation (D).
Speciﬁcally, the PID controller takes the error signal as input, and output the summation of the above three parts, whose formal deﬁnition is deﬁned as follows:
[Equation]
[Equation] K p , K i , K d denotes the corresponding coefﬁcients for PID controller.
e ( t ) denotes the error signal and c is the output of the controller, which is also the actual input of the dynamic.
Generally, the
I part can reduce the static error, and the D part can make the dynamic stable and accelerate the convergence.
The PID controller is introduced to adjust the poles of the dynamic.
Since the static error, i.e., the difference between the stable point and targeted point, is zero, the integral part is not necessary, and we only employ the PD controller in the following.
By letting H c ( s )
=
p + ds , where p and d is the corresponding coefﬁcient for the P and D part, the transfer function of the whole dynamic is given by:
[Equation]
As we can see, given positive p and d , the PD controller can successfully adjust the real parts poles of the dynamic to negative.
In the following analysis and implementation, we ﬁx p = 1 for simplicity. [Figure] [Equation] [Figure]
[Figure]
[Figure] [Figure] [Equation] [Figure]
[Equation]
[Figure]
The dynamics of Dirac GANs using PD controller is given in Fig.
9 (b).
With a PD controller, the dynamic can be successfully stabilized.
For general GAN, the objective of D at time t is given by: [Equation]
[Equation]
The regularization is actually minimizing the loss of D in the current step as well as maximizing the loss of D for previous step.
B.3. E XPERIMENTAL R ESULTS
The results of the synthetic data are illustrated in Fig.
WGAN with PD controller can successfully cover the whole data distribution and be stable at the equilibrium.
However, we fail to generalize PD controller to the natural image dataset such as CIFAR10.
We argue that the main reason is the overshooting for the dynamics of D , as illustrated in Fig.
9 (b), where the
D gives a large derivation from the equilibrium at the beginning.
Because of the highly-nonlinearity in neural networks, this overshooting will result in unexpected behavior of neural networks.
In contrast, for the negative feedback, the overshooting is much smaller, which enables it to generalize to the non-linear neural networks.
C U NIFYING P REVIOUS M ETHODS IN C ONTROL T HEORY
In this section, we also provide some analysis that interprets previous methods in the perspective of control theory.
C. R EG -GAN
Mescheder et al.
(2018) propose a regularization which is added on D ’s gradient of x , i.e., R d ( x ) =
|| ∂D ( x ) ∂x
|| 22 .
Therefore, it can be considered as an other form of negative feedback, which is illustrated in Fig.
Compare to our analysis, Reg-GAN provides more accurate modeling of the dynamics and directly regularizes on the input of the dynamic of G .
Therefore, Reg-GAN can also successfully stabilize the dynamic of GANs, which is consistent with our experimental results.
Compared to Reg-GAN, our method is more computationally efﬁcient.
C. W EIGHT D ECAY AND N EGATIVE M OMENTUM
Gidel et al.
(2018) analyze the effect of momentum in the training of GANs, and we already interpret this analysis in the context of control theory.
Besides, the authors also propose to use the negative momentum to stabilize GANs, which is illustrated as follows:
[Equation]
Together with weight decay, the controlled dynamics of W is given:
[Equation]
In this setting, Eqn.
(21) is equivalent to the PD controller with negative feedback in the parameter space whose diagram can be illustrated in Fig.
The λ 1 ( W t − W t − 1 ) denotes the D part and λ 2 W t denotes the P part.
D L OCAL APPROXIMATION TO THE REGULARIZED DYNAMICS
In this section, we prove that around the equilibrium, the dynamics of regularized D with Lipschitz constraint is equivalent to the unregularized D as in Eqn. Since the updating direction of D is the gradient of D in the functional space, we only need to prove that updating D according to Eqn.
(10) will not violate the Lipschitz constraints, at least locally around the equilibrium.
Here we make the following assumptions: Both p D ( x ) and p G ( t, x ) are C 1 -smooth: dp ( x ) dx exists and is continuous for p D and p G ( t, x ) , ∀ t . p ( x ) → 0 and dp ( x ) dx → 0 when x → 0 for p D and p G ( t, x ) , ∀ t . There exists an M such that | dp ( x ) dx | 2 < M for p D and p G ( t, x ) , ∀ t .
The above assumptions are satisﬁed for most probability density functions.
The distance in the function space is deﬁned as d ( p 1 , p 2 ) =
sup x ∈ R n | p 1 ( x ) − p 2 ( x ) | which always exists because of the 2-nd conditions above.
We deﬁne Ω L = { p ( x ) | p ( x ) ∈ C 1 , | dp ( x ) dx | 2 <
L ∀ x. } and B ( (cid:15) )
= { p ( x ) | p ( x ) ∈ C 1 , sup x | p ( x ) | < (cid:15) } .
Then we have the follow theorem:
Theorem 1.
There exists δ > 0 , such that ∀ D ( x ) ∈ Ω 0 .
5 , we have D ( x ) + δ ( p D ( x ) − p G ( x ))
∈ Ω 1 .
Proof.
By denoting D (cid:48) ( x ) = D ( x ) + δ ( p D ( x ) − p G ( x )) , We have: [Equation] Therefore, we have
[Equation]
[Equation]
By letting δ = 14 M , we have | d ( D (cid:48) )
dx | 2 ≤ 0 .
Therefore we have D (cid:48)
( x ) ∈ Ω 1 .
The above theorem indicates that when D ( x ) is sufﬁcient close to the equilibrium, then the dynamics of D still follows Eqn.
E C ONNECTION TO R EGULARIZATION ON J ACOBIAN M ATRIX
In this paper, we mainly analyze our proposed method in the functional space, including stability analysis and controller designing.
Instead, our proposed method can also be interpreted as certain regularization terms on the Jacobian matrix of the training dynamics.
Below we provide a formal demonstration.
First, we denote the equilibrium of G and D in the functional space as ( θ ∗ , φ ∗ ) , where p G ( x ; θ ∗ ) =
p D ( x ) and D ( x ; φ ∗ ) = 0 for all x .
Therefore
We denote U ( D, G ) as the objective function of the minimax optimization problem in WGAN without NF regularization.
Then the Jacobian matrix of the training dynamic can be denoted as:
[Equation]
Because of the linearity of the derivation, the training dynamics of the WGAN with NF regularization is denoted as:
[Equation]
where we abuse the 0 to denote the zero matrix with certain size to match the size of J .
Since ∂ 2 L ( D )
[Equation]
0 , we have − J L (cid:22) 0 .
Therefore, the NF regularization introduces a negative semi-deﬁnite matrix to the original Jacobian matrix, which is helpful to stabilize the training dynamics of GANs.