UniGAN: Reducing Mode Collapse in GANs using a Uniform Generator
Abstract
Despite the signiﬁcant progress that has been made in the training of Generative
Adversarial Networks (GANs), the mode collapse problem remains a major challenge in training GANs, which refers to a lack of diversity in generative samples.
In this paper, we propose a new type of generative diversity named uniform diversity , which relates to a newly proposed type of mode collapse named u -mode collapse where the generative samples distribute nonuniformly over the data manifold.
From a geometric perspective, we show that the uniform diversity is closely related with the generator uniformity property, and the maximum uniform diversity is achieved if the generator is uniform.
To learn a uniform generator, we propose UniGAN , a generative framework with a Normalizing Flow based generator and a simple yet sample efﬁcient generator uniformity regularization, which can be easily adapted to any other generative framework.
A new type of diversity metric named udiv is also proposed to estimate the uniform diversity given a set of generative samples in practice.
Experimental results verify the effectiveness of our UniGAN in learning a uniform generator and improving uniform diversity.
Introduction
Generative Adversarial Networks [1] (GANs) are a technique for unsupervised learning generative models to capture an unknown data distribution.
Given training samples, two modules, a generator and a discriminator , contest with each other in form of a zero-sum game, where the generator aims to produce realistic samples and the discriminator aims to distinguish generative samples from real data.
Though originally proposed as a generative model for unsupervised learning, GANs have also been widely used in a variety of learning scenarios [2, 3, 4] due to not only their ability to learning highly structured probability distributions but also their theoretical implications [1, 5, 6].
GANs training is notoriously difﬁcult and is very sensitive to almost every aspect of its setup due to the dynamic training paradigm [7, 8].
Despite signiﬁcant progress [9, 10, 11, 12, 13, 14] have been made in the training of GANs, the mode collapse [15] problem remains a major challenge in training
GANs, which refers to a lack of diversity in generative samples [15].
Existing methods reduce mode collapse by modifying model architectures [3, 16, 15, 17, 18, 19, 20], proposing novel optimization algorithms [21, 22, 23, 24, 25] and regularizations [26, 27, 28, 29], etc.
Several works also provide empirical explanations or formal analysis for mode collapse, including catastrophic forgetting [30] of the discriminator, sample weighting [31], and binary hypothesis testing [15].
In this paper, we adopt the manifold hypothesis [32, 33] to formalize our analysis and develop our framework named UniGAN for reducing mode collapse.
The manifold hypothesis assumes that high dimensional data is concentrated on a lower dimensional manifold.
First, this assumption provides us prior knowledge about the unknown ground-truth data distribution p .
Speciﬁcally, from the manifold hypothesis, it is natural to adopt the uniform distribution over the data manifold M where training samples come from as the ground-truth data distribution p , since samples on M are equally accepted as real samples for training GANs.
Given that p is the uniform distribution over M , we propose the uniform diversity of a generative distribution q with formalized deﬁnition that measures how close q is to p by computing the differential entropy of q , considering that the uniform distribution gives the minimum entropy.
Recent work [34, 29] also deem that a diverse generator should yield a generative distribution with lower entropy.
To quantify the extent of mode collapse with respect to the uniform diversity, we further propose the u -mode collapse , which is a new type of mode collapse compared to the existing ( ε, δ ) -mode collapse [15].
We also present theoretical analysis showing that the u -mode collapse is more generalized than the ( ε, δ ) -mode collapse and hence is more robust and efﬁcient in capturing mode collapse.
Second, to maximize uniform diversity, based on the manifold hypothesis, we derive the uniformity property of the generator, providing both formalized deﬁnition and intuitive geometric interpretation.
We also relate the generator uniformity to the uniform diversity, showing that the maximum uniform diversity is achieved if the generator is sufﬁciently uniform.
To maximize uniform diversity, we propose a framework named UniGAN , which consists of a normal discriminator, a Normalizing Flow [35, 36, 37] (NF) based generator, and a simple yet sample efﬁcient regularization for pursuing a uniform generator.
NFs are generative models with speciﬁcally designed architectures that are bijective and easy-to-invert.
Hence we use an NF-based generator since these favorable properties not only make the generator intrinsically satisfy the theoretical prerequisites for maximizing uniform diversity, but also facilitate us to develop a generator uniformity regularization in a sample efﬁcient manner.
Moreover, the regularization is simple and thus can be easily integrated into any models, which makes our UniGAN easy to adapt to any generative frameworks.
To realize the computation of uniform diversity in practice, we also propose a new metric named udiv estimating the uniform diversity of a given set of generative samples.
Experimental results show that UniGAN is effective on improving generator uniformity and uniform diversity.
Our contributions are threefold:
• We propose a new type of generative diversity named uniform diversity and the associated u -mode collapse that measures how close a generative distribution is to the data distribution which we chose to be the uniform distribution over the data manifold based on the manifold hypothesis.
Theoretical analysis shows that u -mode collapse is a generalization of existing ( ε, δ ) -mode collapse and is more robust and efﬁcient in capturing mode collapse.
• We derive the generator uniformity property that is closely related to uniform diversity, and provide analysis showing that the maximum uniform diversity is achieved if the generator is sufﬁciently uniform.
A geometric interpretation is also provided.
• We propose UniGAN , a generative framework with an NF-based generator and a simple yet sample efﬁcient regularization on generator uniformity, which can be easily adapted to any other generative framework.
We also propose a new diversity metric named udiv estimating the uniform diversity of a given set of generative samples.
Experimental results verify the effectiveness of our UniGAN in improving generator uniformity and uniform diversity.
Related Work
Reducing Mode Collapse A variety of methods have been proposed to mitigate the mode collapse problem in the training of GANs, including modiﬁed architectures [3, 4, 16, 38, 15], optimization algorithms [21, 22, 23, 39], and loss functions [34, 40, 26, 27, 28].
Several methods aim to maintain a bijection between the latent and data space by learning an encoder that projects data back into the latent space [3, 4, 16, 41, 42, 38].
For example, BiGANs/ALI [3, 4] and VEEGANs [41, 42] propose to learn an encoder implicitly by making the discriminator to distinguish not only in data space but jointly in data and latent space, and BicycleGAN [16] explicitly encourages reconstruction of both latent codes and training data by combining VAEGAN [41, 42] and LRGAN [16].
Several methods make hypothesis of weak generators [17, 18, 19] and weak discriminators [43, 44, 22, 9, 15, 20] for mode collapse problem.
For example, MAD-GAN [17] employs multiple generators to enforce that different generators capture diverse modes, and PacGAN [15] reduces mode collapse by modifying the discriminator to make decisions based on multiple rather than single samples from the same class, either real or artiﬁcially generated.
PacGAN also provide analysis on mode collapse by proposing a formal deﬁnition of mode collapse, namely ( ε, δ
) -mode collapse from the view of binary hypothesis testing.
Similarly, VirtualGAN [45] is proposed to merge multiple samples into one before training the discriminator.
Another line of research reduces mode collapse by proposing novel optimization algorithms.
Inspired by boosting algorithms, AdaGAN [21] is proposed as an iterative meta-algorithm where many potentially weak generators are greedily aggregated to form a strong composite generator.
The unrolled GAN [22] aims to reduce mode collapse and stabilize training of GANs by deﬁning the generator’s objective with respect to an unrolled optimization of the discriminator.
The progressive growing of GAN [23] aims to grow both the generator and discriminator progressively: starting from a low resolution, then add new layers that model increasingly ﬁne details as training progresses.
The
BourGAN [24] replaces Gaussian with a Gaussian mixture for the prior latent distribution, since the authors assume that the incompatibility between the prior latent distribution and the mode structure is a cause of the mode collapse problem, which is also supported by the analysis of [46] from the view of optimal transport.
There are also some works propose to reduce mode collapse by proposing novel regularizations.
For example, MSGAN [40, 26] maximizes the ratio of the distance between samples with respect to the distance between the corresponding latent codes to reduce mode collapse.
Isometry Learning For ease of practical implementation of our regularization on generator uniformity, we turn to a special case of uniform generators, namely isometric generators whose Jacobians are orthonormal (up to a constant) [47, 48, 49]
(see Section 3.3), which is related to isometry learning .
Several methods are proposed to learn isometric generators.
Following the deﬁnition of an isometry, namely J (cid:62) J
= I where J is the Jacobian of the generator and I is an identity matrix, the authors of
LGAN [50] propose to constrain v (cid:62) i v j =
δ ij where v i is the approximated i -th column vector of J and δ ij is the Kronecker delta.
In StyleGAN2 [51], a regularization on path lengths of the generator is proposed to encourage that a ﬁxed-size step in latent space results in a ﬁxed-magnitude change in the generative image, which implicitly encourages the generator to be an isometry.
For the task of manifold learning, an algorithm named Riemannian relaxation is proposed to construct an isometric mapping in [47].
However, this algorithm does not deal with high-dimensional input such as images, as mentioned by [49].
In RaDOGAGA [49], for an autoencoder, an isometric decoder is learned by injecting random noise to the latent code before reconstructing the input data.
There are also several methods [52] learn isometric mappings aided by traditional dimensionality reduction algorithms [53].
Methodology
We ﬁrstly propose the formal deﬁnitions of uniform diversity and the associated u -mode collapse in
Section 3.1, then relates the uniform diversity with the generator uniformity property in Section 3.2, which motivates us to develop the UniGAN framework for reducing mode collapse in Section 3.3.
3.1 The Uniform Diversity and u -Mode Collapse
Let P M be the set of all distributions over a ﬁnite domain M .
The uniform diversity of a distribution q ∈ P M measures how close q is to the uniform distribution over M , which is deﬁned as
Deﬁnition 1 (Uniform diversity) .
The uniform diversity of q ∈ P M is given as U q (cid:44) e H q m M ∈ [0 , 1] , where m M is the measure of M and H q is the differential entropy of q , namely
[Equation]
where supp ( q (cid:48) )
(cid:44) { x ∈ M| q (cid:48)
( x ) (cid:54) = 0 } is the support of q (cid:48) .
For H q =
−∞ , we deﬁne U q (cid:44) 0 .
Note that given a distribution p ∈ P M , we use p (cid:48) to denote its Probability Density Function (PDF).
We employ differential entropy H q in Deﬁnition 1 since it measures the uniformity of distribution q .
Speciﬁcally, from information theory [54] we know that H q could be −∞ which corresponds to the case where q is extremely nonuniform ( e.g. , q (cid:48) is a Dirac delta function where all probability mass is concentrated on a single point), and H q is upper bounded by log m M since M is ﬁnite, where
H q = log m M achieves only if q is the uniform distribution over M (see supplementary).
The more uniform the q , the higher the H q .
The uniform diversity U q is further deﬁned to normalize the range of H q to [0 , 1] .
Based on uniform diversity, the u -mode collapse is formally deﬁned as
Deﬁnition 2 ( u -mode collapse) .
We say that q ∈ P M exhibits u -mode collapse for 0 (cid:54) u < 1 if its uniform diversity U q is equal to u . [Figure] u changes as , δ )
We also select a point Q uε,δ to show that ε δ mode could be mild mode is severe represented diagram used
The proposed u -mode collapse measures how far a distribution q ∈ P M deviates from the uniform distribution over M .
Lower u indicates more severe mode collapse, and for U q = 1 , q is sufﬁciently uniform and does not exhibit u -mode collapse.
Connection to ( ε, δ ) -mode collapse Recently, another formal deﬁnition of mode collapse named ( ε, δ ) -mode collapse is proposed in [15], which leads to a two-dimensional representation of the mode collapse region of the pair of the data distribution p and a generative distribution q .
We discuss how it relates to our proposed u -mode collapse.
Recall that ( ε, δ ) -mode collapse is formally deﬁned as
Deﬁnition 3 ( ( ε, δ ) -mode collapse, [15]) .
Given data distribution p ∈ P M , a generative distribution q ∈ P M exhibits ( ε, δ ) -mode collapse for 0 (cid:54) ε < δ < 1 if there exists S ⊂ M such that p ( S ) (cid:62) δ and q ( S ) (cid:54) ε .
Given that the data distribution p is chosen as the uniform distribution over M , we show that u -mode collapse is more generalized than ( ε, δ ) -mode collapse, and is more robust and efﬁcient in capturing mode collapse.
We ﬁrst present the following Theorem 1 (see supplementary for proof):
Theorem 1.
For q ∈ P M that exhibits ( ε, δ ) -mode collapse, 0 (cid:54)
U q (cid:54) π ( ε, δ ) , where π ( ε, δ ) (cid:44) (cid:0) δε (cid:1) ε (cid:16) 1 − δ 1 − ε (cid:17) (1 − ε ) ∈
That is, a generative distribution that exhibits ( ε, δ ) -mode collapse at least exhibits π ( ε, δ ) -mode collapse.
Theorem 1 fundamentally speciﬁes the feasible region of three-dimensional representation ( u, ε, δ ) , which we illustrate in Fig.
1(a) using a two-dimensional diagram that shows the respective feasible regions of ( u, ε ) for different δ .
Based on Theorem 1, we then present the following corollaries:
Corollary 1.
The following claims can be deduced from Theorem 1:
(i) Let u ∗ ( ε, δ ) be the maximum uniform diversity that can be achieved for a generative distribution that exhibits ( ε, δ ) -mode collapse.
Then u ∗ ( ε (cid:48) , δ (cid:48) )
<
u ∗ ( ε, δ ) holds for δ (cid:48) >
δ and ε (cid:48) < ε .
Moreover, ∀ ξ > 0 , ∃ µ, ν > 0 , such that u ∗ ( ε, δ ) <
ξ if ε < µ and 1 − δ < ν .
(ii) Let q ∈ P M be a generative distribution that exhibits u -mode collapse for any u ∈ [0 , 1) , then for any δ ∈ (0 , 1) and ξ > 0 , q can exhibit ( ε, δ ) -mode collapse for δ − ε < ξ . [Figure]
See supplementary for proof.
We provide intuitive explanations to aforementioned claims.
In terms of ( ε, δ ) -mode collapse, larger δ and smaller ε indicate more severe mode collapse [15], therefore, we can learn from Claim (i) that as ( ε, δ ) -mode collapse becomes more severe, the maximum uniform diversity that can be achieved decreases, i.e. , u -mode collapse also becomes more severe.
Moreover,
Claim (i) states that u -mode collapse can be sufﬁciently severe ( i.e. , u → 0 ) as ( ε, δ ) -mode collapse becomes sufﬁciently severe ( i.e. , ε → 0 , δ → 1 ).
An illustrative example is given in Fig.
1(a), where
P uε,δ are points in feasible regions of ( u, ε ) given δ such that P uε,δ gives the maximum u that can be achieved for the given ( ε, δ ) .
We can intuitively see that as δ increases and ε decreases, u decreases for P uε,δ , and u can be sufﬁciently small as δ → 1 and ε → 0 .
Meanwhile, Claim (ii) indicates that distributions that exhibit severe u -mode collapse can exhibit mild ( ε, δ ) -mode collapse ( i.e. , ε ≈ δ ), as illustrated by the point Q uε,δ shown in Fig. 1(a).
Hence in summary, the u -mode collapse is severe as long as the ( ε, δ ) -mode collapse is severe, however, the ( ε, δ ) -mode collapse could be mild while the u -mode collapse is severe.
The relationship between ( ε, δ ) -mode collapse and u -mode collapse can be depicted using the Venn diagram as shown in Fig. 1(b).
Speciﬁcally, while severe ( ε, δ ) -mode collapse is covered by severe u -mode collapse, the ( ε, δ ) -mode collapse may not be able to identify severe u -mode collapse effectively.
Therefore, u -mode collapse can be regarded as a generalization of ( ε, δ ) -mode collapse, and is more robust and effective in capturing mode collapse.
We also provide distributions r, q in Fig.
2 as illustrative examples of different mode collapses.
Both r and q are deﬁned over domain [0 , 1] , i.e. , r, q ∈ P [0 , 1] .
We take r shown in Fig. 2(a) as an example that exhibits severe ( ε, δ ) -mode collapse and severe u -mode collapse.
By denoting S (cid:44) [0 . 1 , 1] , we can observe that r ( S ) = 0 .
01 meanwhile p ( S ) = 0 . 99 , namely r exhibits ( ε, δ ) -mode collapse for ε = 0 .
01 and δ = 0 .
99 , which corresponds to a severe ( ε, δ ) -mode collapse since ε ≈ 0 and δ ≈ 1 .
Regarding u -mode collapse, we can compute that u = 0 .
01 , which corresponds to a severe u -mode collapse since u ≈ 0 .
In Fig. 2(b), by denoting S (cid:44) [0 . 99 , 0 . 995) ∪ (0 .
995 , 1] , we can observe that q ( S ) =
meanwhile p ( S ) = 0 .
01 , hence q exhibits ( ε, δ ) -mode
collapse for ε =
and δ = 0 .
Regarding u -mode collapse, due to the inﬁnity of Dirac delta function ∆ 0 .
995 ( x ) at x = 0 .
995 , we can compute that u = 0 , since H q = −∞ .
Therefore, q exhibits severe u -mode collapse but mild ( ε, δ ) -mode collapse, since ε ≈ δ ≈ 0 and u = 0 .
3.2 The Generator Uniformity Property
We now relate the uniform diversity to the property of the generator which we refer to as uniformity .
Let Z ⊂ R K be the latent space and g : Z → R D be the generator with K < D .
From the manifold hypothesis [32, 33], the image of g , M (cid:44) g
( Z ) =
{ g ( z ) | z ∈ Z} , is assumed to be a K -dimensional manifold residing in the ambient space R D , and g pushes forward the prior distribution p z ∈ P Z to a generative distribution q ∈ P M that is denoted by q (cid:44) g
# p z .
According to the deﬁnition of PDF, for x = g ( z ) we have that
[Equation]
where d V Z | z and d V M | x are the volume elements [55] of Z at z and M at x , respectively.
Intuitively, the probability density is the ratio of the probability mass p (cid:48)
z ( z ) d V Z | z to the volume d V M | x of the region where that probability mass is concentrated.
Assume that g is a local diffeomorphism [55], the volume element d V M | x is given according to the Riemannian geometry [55] as
[Equation]
where G ( z ) (cid:44) J (cid:62) g ( z ) J g ( z ) ∈ R K × K is the matrix form of the Riemannian metric tensor [55], and
J g ( z ) ∈ R D
× K is the Jacobian of g at z .
Therefore, by plugging Eq.
(3) into Eq.
(2), we arrive at q
[Equation]
which is also known as the change of variables formula [36, 37, 35].
As aforementioned, the uniform diversity of a generative distribution q ∈ P M is maximized only if q is the uniform distribution over M , which is equivalent to q (cid:48) being constant over M .
For ease of analysis, we assume that p z is the uniform distribution over Z , i.e. , p (cid:48)
z is constant over Z .
In such a case, it is obvious that q (cid:48) is constant over M only if det J (cid:62) g J g is constant over Z .
On this account, we formalize the deﬁnition of the generator uniformity property as
Deﬁnition 4 (Generator uniformity) .
Given a generator g : Z → M that is a local diffeomorphism between Z ⊂ R K and M ⊂ R D with K < D , we say that g is uniform, if det J (cid:62) g J g is constant.
Obviously, the connection between the generator uniformity and a maximized uniform diversity can be formalized as the following Theorem 2 (see supplementary for proof)
Theorem 2.
Assume that the generator g : Z → M is a local diffeomorphism between Z ⊂ R K and M ⊂ R D with K < D , and the prior distribution p z ∈ P Z is the uniform distribution over Z .
The uniform diversity of the generative distribution q = g
# p z is maximized, if g is uniform.
We provide intuitive illustrations for a better understanding of the generator uniformity property.
In
Fig.
3, we set up two generators g, (cid:101) g with different uniformity.
The latent space Z is a 2 -dimensional
Euclidean subspace, and we divide Z into even grid blocks by drawing coordinate lines along axes zand z 2 , respectively.
The generator g ( resp. , (cid:101) g ) pushes forward coordinate lines of Z to those of M ( resp.
, (cid:102) M ).
Since the prior distribution p z ∈ P Z is the uniform distribution over Z and the area of each grid block is equal, we know that each grid block of Z contains the same probability mass.
So each grid bock of M ( resp. , (cid:102) M ) also contains the same probability mass, since the coordinate lines between Z and M ( resp.
, (cid:102) M ) correspond.
For (cid:102) M , because the area of each grid block is equal, the ratio of probability mass to the area, i.e. , the probability density, for each grid block is equal too, i.e. , the generative distribution is the uniform distribution over (cid:102) M .
However, the probability density for each grid blocks of M is not equal, since the area of each grid block is not equal.
Therefore, given that each grid block contains the same probability mass, the equality of areas of grid blocks matters regarding the uniformity of the generative distribution, and the “equality of areas of grid blocks” is essentially an intuitive interpretation of the generator uniformity.
We also explain how √ det J (cid:62) J relates to the area of a grid block.
As shown in Fig.
3, for a grid block of (cid:102) M , if we consider ρ 1 (cid:44) ∂ (cid:101) g ∂zand ρ 2 (cid:44) ∂ (cid:101) g ∂z [-
-]2 as two adjacent sides of that grid block, then the computation of det
[Equation]
implies that (cid:112) det ( J (cid:62) J ) actually gives the area of the parallelogram spanned by ρ 1 and ρ 2 , namely the area of the grid block, where θ is the angle between ρ 1 and ρ 2 .
Hence if det (cid:0) J (cid:62) J (cid:1) is constant, the area of each grid block is equal, and the generator is uniform since the “equality of areas of grid blocks” is an intuitive interpretation of the generator uniformity, which coincides Deﬁnition 4.
3.3 The UniGAN Framework
We introduce our UniGAN framework for reducing mode collapse based on aforementioned analysis from a manifold view.
Our goal is to maximize the uniform diversity of the generative distribution, which can be done by pursuing a uniform generator, as indicated by Theorem 2.
However, to apply
Theorem 2, we have to satisfy its prerequisite, i.e. , the generator g should be a local diffeomorphism.
Moreover, constraining det J (cid:62) g J g to be constant involves not only computing the Jacobian J g , but also computing the determinant, both of which are computationally expensive.
Hence it is nontrivial to implement the generator uniformity constraint in practice.
Regarding the above problems, we propose to employ an NF-based generative model.
On one hand,
NFs are intrinsically bijections, therefore, the diffeomorphism prerequisite of the generator g can be satisﬁed.
On the other hand, NFs are speciﬁcally designed architectures with tractable determinant of
Jacobian [35, 36, 37], which may ease the practical implementation of the constraint.
Unfortunately, the above advantages of NFs are at the cost of maintaining a latent space of as high a dimensionality as the data space, which leads to over-parameterized models and very slow training.
To overcome this problem, we propose to use an NF-based generator g : Z ⊂ R K
→ R D for arbitrary K < D with a hierarchical architecture as follows
[Equation]
where f i : R K i → R K i is a bijective and invertible NF module ( e.g. , coupling layers [36, 37, 56]) with tractable determinant of Jacobian, e i : R K i − 1 → R K i boosts the dimensionality of the input feature x from K i − 1 to K i by simply padding K
i − K
i − 1 zeros at the end of x [57], and the dimensionalities of hierarchical features satisfy K = K 0 < K 1 < · · · < K L = D .
See supplementary for a detailed introduction.
However, a notable issue is that det J (cid:62) g J g is intractable despite tractable determinants of Jacobian for f i .
Hence for ease of implementation, we turn to pursuing a special case of uniform generators where the K singular values of J g , { σ j } Kj =1 , are equal and constant over Z , considering the fact that det J (cid:62) g J g
=
σ 21 σ 22 · · · σ 2 K .
This is further realized by constraining the minimum and the maximum singular values of J g to be equal and constant over Z .
Therefore, the main concern is how to obtain the minimum and the maximum singular values of J g .
We propose the Linearized Transpose (LT) technique to estimate the spectral norm of the Jacobian of a given network in a sample efﬁcient manner.
Due to space limitation, we use LT g ( z ) to denote the estimated spectral norm of J g at z ∈ Z , and the detailed introduction of our proposed LT is provided in supplementary.
Since the spectral norm is essentially the maximum singular value, we know that
LT g ( z ) is the maximum singular value of J g at z .
To obtain the minimum singular value of J g , we propose to utilize the invertible property of f i , which makes it tractable to obtain the pseudo inverse of g , g † : R D
→ R K , deﬁned as
[Equation]
where f − 1 i : R K i → R K i is the inverse of f i , and e † i : R K i
→ R K i − 1 is the pseudo inverse of e i that reduces the dimensionality of the input feature x from K i to K i − 1 by simply dropping K
i − K
i −elements at the end of x .
Given g, g † , we can prove that (see supplementary)
Proposition 1.
∀ z ∈ Z , J g † ( g ( z ))
J g ( z ) =
I ∈ R K × K holds, where I is the identity matrix.
Given the above Prop. 1, by denoting σ ∗ ( J ) ( resp. , σ ∗ ( J ) ) as the maximum ( resp. , the minimum) singular value of a given matrix J , we intuitively illustrate that σ ∗ ( J g ( z ))
=
1 σ ∗ ( J g † ( g ( z )) ) by using
[Equation]
Fig. However, the illustration of Fig. 4 is only applicable in the case where the maximum singular value of J g † ( g ( z )) can be given by a vector u ∗ ∈ T g ( z ) M ⊂ R D that is on the tangent space of M at g ( z ) , see supplementary for illustration. To tackle this problem, we observe that by encouragingσ ∗ ( J g † ( g ( z )) )
=
σ ∗ ( J g ( z )) , the maximum singular value of J g † ( g ( z )) can be given by a vector u ∗ ∈ T g ( z ) M .
To see this, assume that (cid:101) u ∗ ∈ R D gives the maximum singular value of J g † ( g ( z )) , we have that
[Equation]
σ ∗ ( J g ) , where the ﬁrst equality holds if and only if u ∗ (cid:44) J g J g † (cid:101) u ∗ ∈ T g ( z ) M ⊂ R D gives the maximum singular value of J g † . So in the case of
[Equation]
z ) [Equation] , we have σ ∗ ( J g ( z )) =
[Equation]
† ( g ( z )) ) from Fig. 4, and hence σ ∗ ( J g ( z ))
=
σ ∗ ( J g ( z )) , namely the minimum and the maximum singular values of J g are equal.
Therefore, by letting a be the moving average of the maximum and the minimum singular values of
J g , the regularization for the generator uniformity implemented in practice is formalized as
[Equation] which is a simple yet sample efﬁcient regularization due to the superior sample efﬁciency of LT (see supplementary).
We then combine the above NF-based generator and a normal discriminator into a generative framework augmented using our generator uniformity regularization L gunif , obtaining our
UniGAN whose objective function is as follows,
[Equation]
where L gan is the original objective of GANs, and λ gunif is the balancing hyper-parameter.
Eq.
(9) also implies that L gunif can be easily integrated into any models by appending L gunif to the original objective function, and our UniGAN can be adapted to any other generative framework by replacing
L gan with the corresponding objective function.
Experiments
We show through experimental results that our proposed UniGAN framework is effective in learning a uniform generator and improving uniform diversity of the generative distribution.
To demonstrate the adaptability of our UniGAN framework, we adapt UniGAN for a variety of existing models that reduce mode collapse, including GAN [1], BiGAN [3]/ALI [4], MSGAN [40, 26], VEEGAN [38],
PacGAN [15], MDGAN [34] and RegGAN [34].
We also show that these models are inferior to ours regarding uniform diversity.
For intuitive demonstration, we ﬁrstly provide results on a toy synthetic data manifold, then provide results on simple datasets including MNIST [58], FashionMNIST [59] and their colored version [22], and CIFAR10 [60].
We also provide results on natural image datasets including CelebA [61], FFHQ [62], AFHQ [63] and LSUN [64].
To implement the computation of uniform diversity of a given set of generative samples in practice, we also propose a diversity metric named udiv .
Due to space limitation, the implementation details about our proposed udiv metric and experiments can be found in supplementary. [Figure]
We provide results in this section on synthetic datasets for the sake of intuitive understanding, while leaving more results on other datasets in supplementary due to space limitation.
We use a 2D synthetic dataset sector which samples points from a sector data manifold (cid:110) ( x, y ) | 1 (cid:54) (cid:112) x 2 + y 2 (cid:54) 2 (cid:111) in the ﬁrst quadrant.
We set up three major modes to simulate unbalanced distribution of modes in real data.
Speciﬁcally, we use µ r = 1 . 2 , σ r = 0 .
1 to obtain R 1 (cid:44) (cid:8)
r
( i ) ∼ N ( µ r , σ r ) | r ( i ) ∈
[1 , 2] (cid:9) N i
=1 , and use µ θ = π/ 4 , σ θ = π/ 2 to obtain Θ 1 (cid:44) (cid:8)
θ ( i ) ∼ N ( µ θ , σ θ ) | θ ( i ) ∈ [0 , π/ 2] (cid:9) N i =1 , and then perform polar transform, obtaining M 1 (cid:44) (cid:8)(cid:0) x
( i ) , y ( i ) (cid:1)(cid:9) N i
=1 with x ( i ) =
r ( i ) cos θ ( i ) and y ( i ) =
r ( i ) sin θ ( i ) as the ﬁrst mode.
Similarly, we use µ r = 1 . 8 , σ r = 0 .
1 and µ θ = 3 π/ 20 , σ θ = π/ 20 to obtain the second mode M 2 , and use µ r = 1 . 8 , σ r = 0 . 1 and µ θ = 7 π/ 20 , σ θ = π/ 20 to obtain the third mode
M 3 .
Finally, we obtain M sec (cid:44) ∪ 3 i =1 M i .
See Fig. 5(a).
We train baseline models and the corresponding UniGAN adaptations on the proposed 2D synthetic manifold sector .
The manifolds generated by different models are intuitively visualized in Fig. 5(b).
From Fig.
5(b), we learn that for almost every baseline model, their generative samples lack uniform diversity because we can intuitively observe that they distribute nonuniformly.
Meanwhile, samples generated by our UniGAN consistently distribute uniformly, hence our UniGAN outperforms baseline models in terms of uniform diversity, and can better interpolate between training samples on the data manifold compared to baseline models.
We also provide quantitative comparison between baselines and our UniGAN in terms of udiv and generator uniformity as shown in Tbl.
We see that compared to baseline models, our UniGAN achieves a much higher udiv score and maintains lower variance of log det J (cid:62) J , therefore, our UniGAN consistently outperforms baseline models in terms of generator uniformity and uniform diversity of generative samples. [Table] [Table] We further experimentally demonstrate that the uniform diversity is closely related with the generator uniformity, which justiﬁes our theoretical analysis in Sec. 3.2.
Speciﬁcally, in Fig.
6, we visualize the correspondences between the values of log det J (cid:62) J and the probability densities on the generated manifold for different models trained on the 2D synthetic dataset sector .
From Fig. 6, we learn that the probability densities of generated samples and the corresponding values of log det J (cid:62) J are inversely correlated, i.e. , the higher the probability density, the lower the values of log det J (cid:62) J . Quantitative results provided in Tbl. 2 show that the correlation coefﬁcients corr between the probability densities and the values of log det J (cid:62) J are lower than − 0 .
8 across all baselines, which indicates that they are strongly negatively correlated and coincides with the intuitive illustration of the generator uniformity in Fig.
3 and the analysis on the relationship between the generator uniformity and uniform diversity.  Conclusion In this paper, we propose the uniform diversity , which is associated with a new type of mode collapse named u -mode collapse where generative samples distribute nonuniformly over the data manifold, and is closely related with the generator uniformity property.
To maximize the uniform diversity, we propose a simple yet effective generative framework named UniGAN with an NF-based generator and a sample efﬁcient regularization on the generator uniformity, which can be adapted to any other framework.
A new type of diversity metric named udiv is also proposed to estimate uniform diversity in practice.
Experimental results verify the effectiveness of our UniGAN framework. Acknowledgements
The work was supported by the National Science Foundation of China (62076162), and the Shanghai
Municipal Science and Technology Major/
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See supplementary.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See supplementary.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include code in the supplementary material, including the implementation of our NF-based generator, the LT technique, the generator uniformity regularization, and the udiv metric.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report p -value of signiﬁcant test.
See Tbl.
1 and supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]