S TEIN L ATENT O PTIMIZATION FOR G ENERATIVE A DVERSARIAL N ETWORKS
A BSTRACT
Generative adversarial networks (GANs) with clustered latent spaces can perform conditional generation in a completely unsupervised manner.
In the real world, the salient attributes of unlabeled data can be imbalanced.
However, most of existing unsupervised conditional GANs cannot cluster attributes of these data in their latent spaces properly because they assume uniform distributions of the attributes.
To address this problem, we theoretically derive Stein latent optimization that provides reparameterizable gradient estimations of the latent distribution parameters assuming a Gaussian mixture prior in a continuous latent space.
Structurally, we introduce an encoder network and novel unsupervised conditional contrastive loss to ensure that data generated from a single mixture component represent a single attribute.
We confirm that the proposed method, named Stein Latent Optimization for GANs (SLOGAN), successfully learns balanced or imbalanced attributes and achieves state-of-the-art unsupervised conditional generation performance even in the absence of attribute information (e.g., the imbalance ratio).
Moreover, we demonstrate that the attributes to be learned can be manipulated using a small amount of probe data.
I NTRODUCTION
GANs have shown remarkable results in the synthesis of realistic data conditioned on a specific class (Odena et al., 2017; Miyato & Koyama, 2018; Kang & Park, 2020).
Training conditional GANs requires a massive amount of labeled data; however, data are often unlabeled or possess only a few labels.
For unsupervised conditional generation, the salient attributes of the data are first identified by unsupervised learning and used for conditional generation of data.
Recently, several unsupervised conditional GANs have been proposed (Chen et al., 2016; Mukherjee et al., 2019; Pan et al., 2021;
Armandpour et al., 2021).
By maximizing a lower bound of mutual information between latent codes and generated data, they cluster the attributes of the underlying data distribution in their latent spaces.
These GANs achieve satisfactory performance when the salient attributes of data are balanced.
However, the attributes of real-world data can be imbalanced .
For example, in the CelebA dataset (Liu et al., 2015), examples with one attribute (not wearing eyeglasses) outnumber the other attribute (wearing eyeglasses).
Similarly, the number of examples with disease-related attributes in a biomedical dataset might be miniscule (Hwang et al., 2019).
Thus, the imbalanced nature of real-world attributes must be considered for unsupervised conditional generation.
Most of existing unsupervised conditional GANs are not suitable for real-world attributes, because they assume balanced attributes if the imbalance ratio is unknown (Chen et al., 2016; Mukherjee et al., 2019; Pan et al., 2021).
Examples where existing methods fail to learn imbalanced attributes are shown in Figure 1 (a), (b) and (c).
In this paper, we propose unsupervised conditional GANs, referred to as Stein Latent Optimization for GANs (SLOGAN).
We define the latent distribution of the GAN models as Gaussian mixtures to enable the imbalanced attributes to be naturally clustered in a continuous latent space.
We derive reparameterizable gradient identities for the mean vectors, full covariance matrices, and mixing coefficients of the latent distribution using Stein’s lemma.
This enables stable learning and makes [Figure] latent distribution parameters, including the mixing coefficient, learnable.
We then devise a GAN framework with an encoder network and an unsupervised conditional contrastive loss (U2C loss), which can interact well with the learnable Gaussian mixture prior (Figure 2).
This framework facilitates the association of data generated from a Gaussian component with a single attribute.
For the synthetic dataset, our method (Figure 1 (d)) shows superior performance on unsupervised conditional generation, with the accurately learned mixing coefficients.
We performed experiments on various real-world datasets including MNIST (LeCun et al., 1998),
Fashion-MNIST (Xiao et al., 2017),
CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), CelebA-HQ (Karras et al., 2017), and AFHQ (Choi et al., 2020) using architectures such as DCGAN (Radford et al., 2016), ResGAN (Gulrajani et al., 2017), and StyleGAN2 (Karras et al., 2020).
Through experiments, we verified that the proposed method outperforms existing unsupervised conditional GANs in unsupervised conditional generation on datasets with balanced or imbalanced attributes.
Furthermore, we confirmed that we could control the attributes to be learned when a small set of probe data is provided.
The contributions of this work are summarized as follows:
• We propose novel Stein Latent Optimization for GANs (SLOGAN).
To the best of our knowledge, this is one of the first methods that can perform unsupervised conditional generation by considering the imbalanced attributes of real-world data.
• To enable this, we derive the implicit reparameterization for Gaussian mixture prior using Stein’s lemma.
Then, we devise a GAN framework with an encoder and an unsupervised conditional contrastive loss (U2C loss) suitable for implicit reparameterization.
• SLOGAN significantly outperforms the existing methods on unsupervised learning tasks, such as cluster assignment, unconditional data generation, and unsupervised conditional generation, on datasets that include balanced or imbalanced attributes.
B ACKGROUND 2. G ENERATIVE A DVERSARIAL N ETWORKS
Unsupervised conditional generation Several models including InfoGAN (Chen et al., 2016),
ClusterGAN (Mukherjee et al., 2019), Self-conditioned GAN (Liu et al., 2020), CD-GAN (Pan et al., 2021), and PGMGAN (Armandpour et al., 2021) have been proposed to perform conditional generation in a completely unsupervised manner.
However, these models primarily have two drawbacks: (1)
Most of these methods embed the attributes in discrete variables, which induces discontinuity among the embedded attributes.
(2) Most of them assume uniform distributions of the attributes, and thus fail to learn the imbalance in attributes when the imbalance ratio is not provided.
In Appendix B.2, we discuss the above models in detail.
By contrast, our work addresses the aforementioned limitations by combining GANs with the gradient estimation of the Gaussian mixture prior via Stein’s lemma and representation learning on the latent space. [Figure]
[Figure] GANs with Gaussian mixture prior DeLiGAN (Gurumurthy et al., 2017) is analogous to the proposed method, as it assumes a Gaussian mixture prior and learns the mean vectors and covariance matrices via the reparameterization trick.
However, DeLiGAN assumes uniform mixing coefficients without updating them.
As a result, it fails to perform unsupervised conditional generation on datasets with imbalanced attributes.
In addition, it uses the explicit reparameterization trick, which inevitably suffers from high variance in the estimated gradients.
This will be discussed further in Section 2.3.
2. C ONTRASTIVE L EARNING
Contrastive learning aims to learn representations by contrasting neighboring with non-neighboring instances (Hadsell et al., 2006).
In general, contrastive loss is defined as a critic function that approximates the log density ratio log p ( y | x ) /p
( y ) of two random variables X and Y .
By minimizing the loss, the lower bound of the mutual information I ( X ; Y ) is approximately maximized (Poole et al., 2019).
Several studies have shown that contrastive losses are advantageous for the representation learning of imbalanced data (Kang et al., 2021; 2020; Wanyan et al., 2021).
Motivated by these observations, we propose a contrastive loss that cooperates with a learnable latent distribution.
2. G RADIENT E STIMATION FOR G AUSSIAN M IXTURE
Stein’s lemma Stein’s lemma provides a first-order gradient identity for a multivariate Gaussian distribution.
The univariate case of Stein’s lemma can be described as follows:
Lemma 1.
Let function h ( · ) :
R (cid:55)→ R be continuously differentiable.
q ( z ) is a univariate Gaussian distribution parameterized by the mean µ and variance σ .
Then, the following identity holds:
[Equation]
Lin et al. (2019b) generalized Stein’s lemma to exponential family mixtures and linked it to the implicit reparameterization trick.
Stein’s lemma has been applied to various fields of deep learning, including Bayesian deep learning (Lin et al., 2019a) and adversarial robustness (Wang et al., 2020).
To the best of our knowledge, our work is the first to apply Stein’s lemma to GANs.
Reparameterization trick A simple method to estimate gradients of the parameters of Gaussian mixtures is explicit reparameterization, used in DeLiGAN.
When the c -th component is selected according to the mixing coefficient p ( c ) , the latent variable is calculated as follows:
z = µ c + ϵ · Σ 1 / 2 c , where ϵ ∼ N ( 0 , I ) .
Derivatives of a loss function ∂ L ( z ) ∂ µ and ∂ L ( z ) ∂ Σ only update the mean and covariance matrices of the selected ( c -th) component µ c and Σ c , respectively.
Gradient estimation using explicit reparameterization is unbiased; however, it has a distinctly high variance.
For a single latent vector z , the implicit reparameterization trick (Figurnov et al., 2018) updates the parameters of all the latent components.
Gradient estimation using implicit reparameterization is unbiased and has a lower variance, which enables a more stable and faster convergence of the model.
The gradients for the parameters of the Gaussian mixture prior in our method are implicitly reparameterizable.
P ROPOSED M ETHOD
In the following sections, we propose Stein Latent Optimization for GANs (SLOGAN).
We assume a Gaussian mixture prior (Section 3.1), derive implicit reparameterization of the parameters of the mixture prior (Section 3.2), and construct a GAN framework with U2C loss (Section 3.3).
Additionally, we devise a method to manipulate attributes to be learned if necessary (Section 3.4).
An overview of SLOGAN is shown in Figure 2.
3. G AUSSIAN MIXTURE P RIOR
We consider a GAN with a generator G : R d z (cid:55)→ R d x and a discriminator D : R d x (cid:55)→ R , where d z and d x are the dimensions of latent and data spaces, respectively.
In the latent space Z ∈ R d z , we consider a conditional latent distribution q ( z | c ) =
N ( z ; µ c , Σ c ) , c = 1 , ..., K , where K is the number of components we initially set and µ c , Σ c are the mean vector and covariance matrix of the cth component, respectively.
Subsequently, we consider a Gaussian mixture q ( z )
=
(cid:80)
Kc =1 p ( c ) q
( z | c ) parameterized by µ = { µ c } Kc =1 , Σ = { Σ c } Kc =1 and π = { π c } Kc =1 = { p ( c ) }
Kc =1 as the prior.
We hypothesize that a mixture prior in a continuous space could model some continuous attributes of real-world data (e.g., hair color) more naturally than categorical priors which could introduce discontinuity (Mukherjee et al., 2019).
Because we use implicit reparameterization of a mixture of
Gaussian priors (derived in Section 3.2), SLOGAN can fully benefit from implicit reparameterization and U2C loss.
By contrast, the implicit reparameterization of prior distributions that do not belong to the exponential family (e.g., categorical priors) remains an open question.
In the experiments, the elements of µ c were sampled from N (0 , 0 . 1) , and we selected Σ c
= I and π c = 1 /K as the initial values.
For the convenience of notation, we define the latent distribution q = q ( z ) , the mixing coefficient π c = p ( c ) , and δ ( z ) =
{ δ ( z ) c } Kc =1 , where δ ( z ) c
= q ( z | c ) /q
( z ) .
q ( c | z ) , the responsibility of component c for a latent vector z , can be expressed as follows:
[Equation]
3. G RADIENT I DENTITIES
We present gradient identities for the latent distribution parameters.
To derive the identities, we use the generalized Stein’s lemma for Gaussian mixtures with full covariance matrices (Lin et al., 2019b).
First, we derive a gradient identity for the mean vector using Bonnet’s theorem (Bonnet, 1964).
Theorem 1.
Given an expected loss of the generator L and a loss function for a sample ℓ ( · ) : R d z (cid:55)→ R , we assume ℓ to be continuously differentiable.
Then, the following identity holds:
[Equation]
Proof of Theorem 1 is given in Appendix C.1.
We derive a gradient identity for the covariance matrix via Price’s theorem (Price, 1958).
Among the two versions of the Price’s theorem, we use the first-order identity to minimize computational cost.
Theorem 2.
With the same assumptions as in Theorem 1, the following gradient identity holds:
[Equation]
[Equation]
Proof of Theorem 2 is given in Appendix C.2.
In the implementation, we replaced the expectation of the right-hand side of Equation 4 with the average for a batch of latent vectors; hence, the updated Σ c may not be symmetric or positive-definite.
To force a valid covariance matrix, we modify the updates of the covariance matrix as follows:
[Equation]
Initialize µ , Σ , ρ , parameters of D , G , and E while training loss is not converged do
Sample a batch of data { x i } Bi =1 ∼ p ( x )
Sample a batch of latent vectors { z i } Bi =1 ∼ q ( z ) for i = 1 , ..., B do
Calculate ℓ adv ( z i ) and ℓ U2C ( z i ) for a latent vector z
i
S z i ← δ ( z i ) c
π c Σ − 1 c ( z i − µ c ) ∇ T z i ( ℓ adv ( z i )+ λℓ U2C ( z i ) ) end for for c = 1 , ...,
K do
Update µ c , Σ c and ρ c via stochastic gradient estimation
[Table]
end for
Update G , E and D using SGD
[Table]
end while input : {{ x iy } Ni =1 } Ky =1 - Data sampled from p ( x | y ) for y = 1 , ..., K ; {{ z ic } Ni =1 } Kc =1 - Latent vectors sam- pled from q ( z | c ) for c
= 1 , ..., K output: ICFID - Intra-cluster FID; Y c - Class-cluster assignments
Y ← { 1 , ..., K } C ← { 1 , ..., K } for each class y in Y do X r ← { x iy } Ni =1 for each cluster c in C do
X g ← { x ic } Ni =1 d ( y, c ) ← FID ( X r , X g ) end for c ∗
← argmin c ∈ C d ( y, c )
ICFID ( y ) ← d
( y, c ∗ )
Y c ( y ) ← c ∗
Remove c ∗ from C end for
ICFID ← 1 K (cid:80) Ky
=1 ICFID ( y ) where S z = δ ( z )
c π c Σ − 1 c ( z − µ c ) ∇ T z ℓ ( z ) , and γ denotes the learning rate for Σ c .
Equationholds as ∆Σ c = 12 E q [ S z ] = 12 E q (cid:2)
S T z (cid:3) .
Motivated by Lin et al.
(2020), Equation 6 ensures the positive-definiteness of the covariance matrix, which is proved by Theorem 3.
Theorem 3.
The updated covariance matrix Σ ′ c = Σ c + γ ∆Σ ′ c with the modified update rule specified in Equation 6 is positive-definite if Σ c is positive-definite.
Proof of Theorem 3 is provided in Appendix C.3.
We introduce a mixing coefficient parameter ρ c , which is updated instead of the mixing coefficient π c , to guarantee that the updated mixing coefficients are non-negative and summed to one.
π c can be calculated using the softmax function (i.e., π c = exp( ρ c ) / (cid:80) Ki =1 exp( ρ i ) ).
We can then derive the gradient identity for the mixing coefficient parameter as follows:
Theorem 4.
Let ρ c be a mixing coefficient parameter.
Then, the following gradient identity holds:
[Equation]
Proof of Theorem 4 is given in Appendix C.4.
Because the gradients of the latent vector with respect to the latent parameters are computed by implicit differentiation via Stein’s lemma, we obtain the implicit reparameterization gradients introduced by Figurnov et al.
3. C ONTRASTIVE L EARNING
We introduce new unsupervised conditional contrastive loss (U2C loss) to learn salient attributes from data and to facilitate unsupervised conditional generation.
We consider a batch of latent vectors { z i } Bi =1 ∼ q ( z ) , where B is the batch size.
Generator G receives the i -th
latent vector z i and generates data x ig = G
( z i ) .
The adversarial loss for G with respect to the sample z i is as follows:
[Equation]
We also introduce an encoder network E to implement U2C loss.
The synthesized data x ig enters
E , and E generates an encoded vector e i x
= E ( x ig ) .
Then, we find the mean vector µ iC , where C is the component ID with the highest responsibility q
( c | z i ) .
We calculate C first because a generated sample should have the attribute of the most responsible component among multiple components in the continuous space.
Second, to update the parameters of the prior using implicit reparameterization, the loss should be a function of a latent vector z i , as proved in Theorems 1, 2, and 4.
The component
ID for each sample is calculated as follows:
[Equation]
where q ( c | z i )
=
δ ( z i ) c
π c is derived from Equation 2.
To satisfy the assumption of the continuously differentiable loss function in Theorems 1 and 2, we adopt the Gumbel-Softmax relaxation (Jang et al., 2017), instead of the argmax function.
We use µ i C
= (cid:80)
Kc =1 C ic µ c to calculate U2C loss to ensure that the loss function is continuously differentiable with respect to z i , where C i =
Gumbel-Softmax τ ( δ ( z i ) π ) and τ = 0 .
We derive U2C loss as follows:
[Equation]
where we select the cosine similarity between e i x and µ j C , cos θ ij = e i x · µ
j C / ∥ e i x ∥∥ µ j C ∥ as the critic function that approximates the log density ratio log p ( C j | x ig ) /p
( C j ) for contrastive learning.
Given a test data, the probability for each cluster can be calculated using the assumption of the critic function, which enables us to assign a cluster for the data.
Cluster assignment is described in Appendix D.2.
Intuitively, a mean vector µ i C of a latent mixture component is regarded as a prototype of each attribute.
U2C loss encourages the encoded vector e i x of the generated sample to be similar to their assigned low-dimensional prototypes µ i C in the latent space.
This allows each salient attribute clusters in the latent space, and each component of the learned latent distribution is responsible for a certain attribute of the data.
If cos θ ii is proportional to the log density ratio log p ( C i | x i g ) /p
( C i ) , minimizing U2C loss in Equation 10 is equivalent to maximizing the lower bound of the mutual information
I ( C i ; x ig ) , as discussed by Poole et al.
and
Zhong et al.
G and E are trained to minimize 1 B (cid:80) Bi
=1 (cid:0) ℓ adv ( z i ) + λℓ U2C ( z i ) (cid:1) , where λ denotes the coefficient of
U2C loss.
Both µ and Σ are learned by substituting ℓ adv ( z i )+ λℓ U2C ( z i ) into ℓ of Equations 3 and 6, respectively.
When U2C loss is used to update π , U2C loss hinders π from estimating the imbalance ratio of attributes in the data well, which is discussed in Appendix A.3 with a detailed explanation and an empirical result.
Therefore, ρ , from which π is calculated, uses only the adversarial loss, and ℓ of Equation 7 is substituted by ℓ adv ( z i ) .
µ , Σ and ρ are learned using a batch average of estimated gradients, which is referred to as stochastic gradient estimation, instead of expectation over the latent distribution q .
The entire training procedure of SLOGAN is presented in Algorithm 1.
To help that the latent space does not learn low-level attributes, such as background color, we additionally used the SimCLR
(Chen et al., 2020) loss on the generated data with DiffAugment (Zhao et al., 2020) to train the encoder on colored image datasets.
Methodological details and discussion on
SimCLR are presented in Appendix D.4 and A.3, respectively.
3. A TTRIBUTE M ANIPULATION
For datasets such as face attributes, a data point can have multiple attributes simultaneously.
To learn a desired attribute from such data, a probe dataset { x ic } Mi =1 for the c -th latent component, which consists of M data points with the desired attribute, can be utilized.
We propose the following loss:
[Equation]
[Equation] where cos θ ik = E ( x ic ) · µ k / ∥ E ( x ic ) ∥∥ µ k ∥ is the cosine similarity between E ( x ic ) and µ k .
Our model manipulates attributes by minimizing L m for µ , Σ , G , and E .
In addition, mixup (Zhang et al., 2018) can be used to better learn attributes from a small probe dataset.
The advantage of SLOGAN in attribute manipulation is that it can learn imbalanced attributes even if the attributes in the probe dataset are balanced, and perform better conditional generation.
The detailed procedure of attribute manipulation is described in Appendix D.3.
E XPERIMENTS 4. D ATASETS
We used the MNIST (LeCun et al., 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), CIFAR-
(Krizhevsky et al., 2009), CelebA (Liu et al., 2015), CelebA-HQ (Karras et al., 2017), and AFHQ (Choi et al., 2020) datasets to evaluate the proposed method.
We also constructed some datasets with
[Table] [Table]
[Figure]
[Table]
[Figure]
imbalanced attributes.
For example, we used two classes of the MNIST dataset (0 vs. 4, referred to as MNIST-2), two classes of the CIFAR-10 dataset (frogs vs. planes, referred to as CIFAR-2), and five clusters of the FMNIST dataset ( { Trouser } , { Bag } , { T-shirt/top, Dress } , { Pullover, Coat, Shirt } , { Sneaker, Sandal, Ankle Boot } , referred to as FMNIST-5 with an imbalance ratio of 1:1:2:3:3).
Details of the datasets are provided in Appendix E.
Although SLOGAN and other methods do not utilize labels for training, the data in experimental settings have labels predefined by humans.
We consider that each class of dataset contains a distinct attribute.
Thus, the model performance was measured using classes of datasets.
The number of latent components or the dimension of the discrete latent code ( K ) was set as the number of classes of data.
4. E VALUATION M ETRICS
The performance of our method was evaluated quantitatively in three aspects: (1) whether the model could learn distinct attributes and cluster real data (i.e., cluster assignment), which is evaluated using normalized mutual information (NMI) (Mukherjee et al., 2019), (2) whether the overall data distribution p ( x r ) could be estimated (i.e., unconditional data generation), which is measured using the Fr´echet inception distance (FID)
(Heusel et al., 2017), and, most importantly, (3) whether the data distribution for each attribute p ( x r | c ) could be estimated (i.e., unsupervised conditional generation).
For unsupervised conditional generation, it is important to account for intra-cluster diversity as well as the quality of the generated samples.
We introduce a modified version of FID named intra-cluster
Fr´echet inception distance (ICFID) described in Algorithm 2.
We calculate FIDs between the real data of each class and generated data from each latent code (a mixture component for DeLiGAN and
SLOGAN, and a category for other methods).
We then greedily match a latent code with a class of real data with the smallest FID.
We define ICFID as the average FID between the matched pairs and use it as an evaluation metric for unsupervised conditional generation.
ICFID additionally provides class-cluster assignment (i.e., which cluster is the closest to the class). [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
4. E VALUATION R ESULTS
We compared SLOGAN with InfoGAN (Chen et al., 2016), DeLiGAN
(Gurumurthy et al., 2017),
ClusterGAN (Mukherjee et al., 2019), Self-conditioned GAN (SCGAN) (Liu et al., 2020), CD-GAN (Pan et al., 2021)
[Table]
and PGMGAN (Armandpour et al., 2021).
Following Mukherjee et al.
(2019), we used k-means clustering on the encoder outputs of the test data to calculate NMI.
DeLiGAN has no encoder network; hence the pre-activation of the penultimate layer of D was used for the clustering metrics.
For a fair comparison, we also compared DeLiGAN with an encoder network (referred to as DeLiGAN+).
The same network architecture and hyperparameters (e.g., learning rate) were used across all methods for comparison.
Details of the experiments and DeLiGAN+ are presented in
Appendices E and D.5, respectively.
Balanced attributes We compare SLOGAN with existing unsupervised conditional GANs on datasets with balanced attributes.
As shown in Table 1 (The complete version is given in Appendix
A.1.)
, SLOGAN outperformed other GANs, and comparisons with methods with categorical priors (ClusterGAN and CD-GAN) verified the advantages of the mixture priors.
Imbalanced attributes In Table 2 (The complete version is presented in Appendix A.2), we compare SLOGAN with existing methods on datasets with imbalanced attributes.
ICFIDs of our method are much better than those of other methods, which indicates that SLOGAN was able to robustly capture the minority attributes in datasets and can generate data conditioned on the learned attributes.
In CIFAR-2 (7:3), the ratio of frog and plane is 7 to 3 and the estimated π is (0.69 ± 0.02, 0.31 ± 0.02), which are very close to the ground-truth (0.7, 0.3).
Figure 3 (a) shows the images generated from each latent component of SLOGAN on AFHQ (Cat:Dog=1:2).
More qualitative results are presented in Appendix A.7.
Performance with respect to imbalance ratio We compared the performance of SLOGAN with competitive benchmarks (ClusterGAN and CD-GAN) by changing the imbalance ratios of
CIFAR-2 from 9:1 to 1:9.
SLOGAN showed higher performance than the benchmarks on cluster assignment (Figure 4 (a)) and unsupervised conditional generation (Figure 4 (b)) for all imbalance ratios.
Furthermore, our method shows a larger gap in ICFID with the benchmarks when the ratio of planes is low.
This implies that SLOGAN works robustly in situations in which the attributes of data are highly imbalanced.
We conducted additional experiments including interpolation in the latent space, benefits of ICFID.
The results of the additional experiments are shown in Appendix A.
[Figure]
[Table] [Figure]
A BLATION S TUDY
U2C loss Table 3 (The complete version is given in Appendix A.3) shows the benefit of U2C loss on several datasets.
Low-level features (e.g., color) of the CIFAR dataset differ depending on the class, which enables SLOGAN to function to some extent without U2C loss on CIFAR-10.
In the
MNIST dataset, the colors of the background (black) and object (white) are the same, and only the shape of objects differs depending on the class.
U2C loss played an essential role on MNIST (7:3).
The modes of the Synthetic dataset (Figure 1) are placed adjacent to each other, and SLOGAN cannot function on this dataset without U2C loss.
From the results, we observed that the effectiveness of
U2C loss depends on the properties of the datasets.
Implicit reparameterization To show the advantage of implicit over explicit reparameterization, we implemented DeLiGAN with U2C loss by applying explicit reparameterization on µ and Σ .
Because the mixing coefficient cannot be updated with explicit reparameterization to the best of our knowledge, we also implemented DeLiGAN with U2C loss and implicit reparameterization on ρ using Equation 7.
In Table 4, SLOGAN using implicit reparameterization outperformed explicit reparameterization.
When implicit ρ update was added, the prior collapsed into a single component ( π y =
= 1 ) and ICFID increased.
The lower variance of implicit reparameterized gradients prevents the prior from collapsing into a single component and improves the performance.
Additional ablation studies and discussions are presented in Appendix A.3.
4. E FFECTS OF P ROBE D ATA
CelebA + ResGAN
We demonstrate that SLOGAN can learn the desired attributes using a small amount of probe data.
Among multiple attributes which co-exist in the CelebA dataset, we chose
Male (1:1) and Eyeglasses (14:1).
We randomly selected 30 probe images for each latent component.
π y =0 represents the learned mixing coefficient that correspond to latent components associated with faces without the attribute.
As shown in Figure 5 and Table 5, we observed that SLOGAN learned the desired attributes.
Additional experiments on attribute manipulation are presented in Appendix A.3.
CelebA-HQ + StyleGAN2 StyleGAN2 (Karras et al., 2020) differs from other GANs in that the latent vectors are used for style .
Despite this difference, the implicit reparameterization and U2C loss can be applied to the input space of the mapping network.
On the CelebA-HQ dataset, we usedmale and 30 female faces as probe data.
As shown in Figure 3 (b), SLOGAN successfully performed on high-resolution images and a recent architecture, even simultaneously with imbalanced attributes.
C ONCLUSION
We have proposed a method called SLOGAN to generate data conditioned on learned attributes on real-world datasets with balanced or imbalanced attributes.
We derive implicit reparameterization for the parameters of the latent distribution.
We then proposed a GAN framework and unsupervised conditional contrastive loss (U2C loss).
We verified that SLOGAN achieved state-of-the-art unsupervised conditional generation performance.
In addition, a small amount of probe data helps SLOGAN control attributes.
In future work, we will consider a principled method to learn the number and hierarchy of attributes in real-world data.
In addition, improving the quality of samples with minority attributes is an important avenue for future research on unsupervised conditional GANs. A CKNOWLEDGEMENT
This work was supported by the BK21 FOUR program of the Education and Research Program for
Future ICT Pioneers, Seoul National University in 2021, AI Institute at Seoul National University (AIIS) in 2021, and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
[NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)], AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund A A DDITIONAL R ESULTS AND D ISCUSSION
We also compared the proposed method with WGAN (Arjovsky et al., 2017) with the proposed method.
We used the pre-activation of the penultimate layer of D for the clustering metrics because
WGAN has no encoder network.
We could not measure ICFID of WGAN because it cannot perform unsupervised conditional generation.
In addition to various datasets, we also used the 10x 73k dataset (Zheng et al., 2017) [Table] which consists of RNA transcript counts.
From the results of the clustering performances on the 10x 73k dataset, we show that SLOGAN learns useful imbalanced attributes and can be helpful in the use of unlabeled biomedical data
[Table]
A. P ERFORMANCE C OMPARISON ON B ALANCED A TTRIBUTES
[Table]
A. P ERFORMANCE C OMPARISON ON I MALANCED A TTRIBUTES
[Table]
A. A BLATION S TUDY
Table 8 shows the ablation study on SLOGAN trained with CIFAR-2 (7:3).
π y =0 and π y =1 represent the mixing coefficients of the latent components that correspond to the frogs and planes, respectively, and the ground-truth of π y =0 is 0.7.
Factor analysis Rows 1-6 of Table 8 compare the performance depending on the factors affecting the performance of SLOGAN ( µ , Σ ,
ρ updates, and ℓ U2C ).
We confirmed that SLOGAN with all the factors demonstrated the highest performance.
Among the parameters of the latent distribution, the µ update leads to the highest performance improvement.
The intra-cluster Fr´echet inception distance (ICFID) of SLOGAN without ρ update (the 5th row of Table 8) indicates that SLOGAN outperformed existing unsupervised conditional GANs even when assuming a uniform distribution of the attributes.
Loss for ρ update We do not use U2C loss ℓ U2C to learn the mixing coefficient parameters ρ . We construct U2C loss to approximate the negative mutual information − I ( C ; x g ) that can be
[Table] [Table]
U2C 0.
SLOGAN 0.
decomposed into entropy and conditional entropy as follows:
[Equation]
The conditional entropy term reduces the uncertainty of the component from which the generated data are obtained.
The entropy term promotes that component IDs are uniformly distributed.
In terms of ρ update, the entropy term H ( C ) drives p ( C ) toward a discrete uniform distribution.
Therefore, using ℓ U2C for learning ρ pulls π to a discrete uniform distribution and can hinder the learned π from accurately estimating the imbalance ratio inherent in the data.
In the 9th row of Table 8, we observed that the unsupervised conditional generation performance was undermined and the estimated imbalance ratio ( π y =0 ) was learned closer to a discrete uniform distribution ( 0 . 5 ) when ℓ U2C was used for ρ update.
SimCLR analysis For the colored image datasets, SLOGAN uses the SimCLR loss for the encoder with only fake (generated) data to further enhance the unsupervised conditional generation performance.
The 10th to 12th rows of Table 8 show several ablation studies that analyzed the effect of SimCLR loss on SLOGAN.
SLOGAN without SimCLR still showed at least approximately 35% performance improvement compared to the existing unsupervised conditional GANs (ICFID of
ClusterGAN: 75.52, CD-GAN: 76.91 in Table 2), even considering the fair computational cost and memory consumption.
The SimCLR loss shows the highest performance improvement especially when applied only to fake data.
SimCLR improved the performance by 7%.
Attribute manipulation As shown in the 13th and 14th rows of Table 8, probe data significantly improved the performance of SLOGAN on CIFAR-2 (7:3) with 10 probe data for each latent component.
We also confirmed that the mixup applied to the probe data further enhanced the overall performance of our model.
Figure 6 (a) shows the data generated from SLOGAN trained on CIFAR- (9:1) without the probe data.
With extremely imbalanced attributes, SLOGAN mapped frog images with a white background onto the same component as airplanes in its latent space.
When we useprobe data for each latent component, as shown in Figure 6 (b), frogs with a white background were generated from the same latent component as the other frog images.
[Figure]
Feature scale We introduced the feature scale s described in Appendix D.1 to reinforce the discriminative power of U2C loss.
For the MNIST-2 (7:3) dataset, s was set to 4.
Such a parameter configuration is justified by a greedy search in [0 . 5 , 1 , 2 , 4 , 8] .
The performances of SLOGAN on
MNIST-2 (7:3) with different feature scales are shown in Table 10.
[Table]
Intuitively, increasing the feature scale s makes the samples generated from the same component closer to each other in the embedding space.
From these results, we observed that the optimal choice of the temperature factor enhances the discriminative power of U2C loss.
A
. S TATISTICAL S IGNIFICANCE
We completed the statistical tests between SLOGAN and other methods in Tables 1 and 2.
Since the results of the experiment could not satisfy normality and homogeneity of variance, we used the
Wilcoxon rank sum test (Wilcoxon, 1992).
When p-value > 0 .
05 , we measured the effect size using
Cohen’s d (Cohen, 2013).
We validated that all the experiments are statistically significant or showed large or medium effect sizes, with the exception of FID of InfoGAN vs. SLOGAN for FMNIST in
Table 1, NMI of DeLiGAN+ vs. SLOGAN for FMNIST-5 in Table 2.
A. I NTERPOLATION IN L ATENT S PACE
We also qualitatively show that the continuous nature of the prior distribution of SLOGAN makes superbly smooth interpolation possible in the latent space.
In Figure 7, we visualize images generated from latent vectors obtained via interpolation among the mean vectors of the trained latent components.
The generated images gradually changed to 3, 5, and 8 for the MNIST dataset, and t-shirt/top, pullover, and dress for the Fashion-MNIST dataset.
In particular, we confirmed that the face images generated from the model trained with the CelebA data changed smoothly.
The continuous attributes of realworld data are well mapped to the continuous latent space assumed by us, unlike most other methods using separated latent spaces induced via discrete latent codes.
A. B ENEFITS OF ICFID
DeLiGAN and ClusterGAN trained on the MNIST-2 (7:3) exhibited comparable FIDs to SLOGAN (DeLiGAN: 4.21, ClusterGAN: 4.25, and SLOGAN: 4.02); however they showed ICFIDs approximately four times higher (DeLiGAN: 25.34, ClusterGAN: 25.61, and SLOGAN: 5.91).
From the data generated from each latent component of DeLiGAN and ClusterGAN in Figure 8 (b) and (c), we confirm that the attributes were not learned well in the latent space of DeLiGAN.
By contrast, from the data generated from each latent component of SLOGAN presented in Figure 8 (a), SLOGAN successfully learned the attributes in its latent space.
This shows that ICFID is useful for evaluating the performance of unsupervised conditional generation.
In addition, ICFID can evaluate the diversity of images generated from a discrete latent code or mode because ICFID is based on FID.
As shown in
Figure 9, when a mode collapse occurs, the diversity of samples decreases drastically, and DeLiGAN trained on the CIFAR-2 (7:3) shows approximately twice the ICFID than those of InfoGAN and
ClusterGAN (DeLiGAN: 186.31, InfoGAN: 88.49, and ClusterGAN: 75:52).
[Figure] [Figure]
A. Q UALITATIVE RESULTS
Synthetic data Figures 10 and 11 shows the synthetic data and data generated by SLOGAN.
Each color denotes generated data from each mixture component of the latent distribution.
As the training progresses, each mixture component in the latent space is more strongly associated with each
Gaussian distribution in the data space.
Generated images and latent spaces Figures 12, 13, 17, 18, 19, the left plot of Figure 15, and the upper plot of Figure 16 show the images generated from each latent component of SLOGAN trained on various datasets.
Figure 14, the right plot of Figure 15, and the lower plot of Figurevisualize 1,000 latent vectors of SLOGAN trained on the MNIST, Fashion-MNIST, MNIST-2 (7:3), and FMNIST-5 datasets using 3D principal component analysis (PCA).
Each color represents the component with the highest responsibility, and each image shows the generated image from the latent vector.
As shown in Figure 14, similar attributes (e.g., 4, 7, and 9) are mapped to nearby components in the latent space.
Comparisons with the most recent methods We compare our method with the most recent methods such as CD-GAN (Pan et al., 2021) and PGMGAN (Armandpour et al., 2021) on the
CIFAR-2 (7:3) dataset.
From the results shown in Figure 20, we qualitatively confirm that SLOGAN learns imbalanced attributes of the dataset most robustly.
Highly imbalanced multi-class data We trained our method on highly imbalanced multi-class datasets by setting class 8 of the MNIST dataset to very low proportions of the other nine classes (e.g., 10:10:10:10:10:10:10:10:1:10 and 10:10:10:10:10:10:10:10:2:10).
When class 8 is 0.1 fraction of the other nine classes, images of class 7 with a horizontal line outnumber images of class 8, and
SLOGAN identifies 7 with a horizontal line as a more salient attribute than 8 as shown in the red boxes in Figure 21 (a).
On the other hand, when class 8 is 0.2 fraction of the other nine classes, images of class 8 outnumber images of class 7 with a horizontal line.
Therefore, SLOGAN successfully identifies 8 as a salient attribute as shown in the red box in Figure 21 (b).
Qualitative analysis with various imbalance ratios Figure 22 shows generated images from each latent component of SLOGAN trained on the AFHQ dataset.
For various imbalance ratios of cats and dogs, we qualitatively analyze SLOGAN without using probe data.
When the imbalance ratios are 1: and 1:2, SLOGAN identifies cat/dog as the most salient attribute and learned the attribute successfully as presented in Figure 22 (a) and (b).
When the imbalance ratio is 1:5, SLOGAN discovers folded ears as the most salient attribute as shown in Figure 22 (c).
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
B A DDITIONAL B ACKGROUND
Our work is closely related to Stein’s lemma and the reparameterization trick for Gaussian mixture and also related to several topics of GAN studies such as representation learning, supervised/unsupervised conditional generation, and Gaussian mixture prior.
B. S UPERVISED CONDITIONAL GENERATION
Conditional GANs including ACGAN (Odena et al., 2017), projection discriminator (Miyato &
Koyama, 2018), and ContraGAN (Kang & Park, 2020) have led to state-of-the-art performances in conditional image generation.
However, these conditional GANs are trained with supervision and require a large amount of labeled data.
B. U NSUPERVISED CONDITIONAL GENERATION
InfoGAN (Chen et al., 2016) introduces latent codes composed of categorical and continuous variables and derives the lower bound of mutual information between the latent codes and representations.
By maximizing the lower bound, InfoGAN learns disentangled latent variables.
ClusterGAN (Mukherjee et al., 2019) assumes a discrete-continuous Gaussian prior wherein discrete variables are defined as a one-hot vector and continuous variables are sampled from a Gaussian distribution.
The latent distribution is clustered in an unsupervised manner through reconstruction loses for discrete and continuous variables.
CD-GAN (Pan et al., 2021) is similar to ClusterGAN, but it uses contrastive loss to disentangle attributes.
These methods assume uniform distribution of the attributes, and if the imbalance ratio is unknown, these models cannot thoroughly learn imbalanced attributes.
Recently, unsupervised conditional GANs which do not assume uniform distribution of the attributes have been proposed.
Self-conditioned GAN (Liu et al., 2020) performs unsupervised conditional generation by using clustering of discriminator features as labels.
However, it has no loss to facilitate clustering of discriminator features, and the performance seems to be largely influenced by the architecture of the discriminator.
PGMGAN (Armandpour et al., 2021) applies a contrastive clustering method named SCAN (Van Gansbeke et al., 2020) to perform unsupervised conditional generation.
The pretrained space partitioner and a penalization loss function are used to encourage the generator to generate images with salient attributes.
To avoid degenerated clusters where one partition contains most of the data, PGMGAN uses an entropy term that regularizes the average cluster probability vector to a uniform distribution.
However, when learning PGMGAN on datasets with imbalanced attributes, it is difficult to adjust the coefficient of the regularizer because the performance seems to be sensitive to the strength of the regularizer in practice (e.g., the results on CIFAR-2 (7:3) and (9:1) in Table 7).
In addition, clustering results does not seem to be reliable when the space partitioner is trained on datasets where transformations are not available (e.g., the result on 10x 73k in Table 7).
NEMGAN (Mishra et al., 2020) argues that it considers the imbalance problem of attributes.
However, it assumes that some labeled samples are provided, which is unrealistic in real-world scenarios.
In addition, labeled samples should have the same imbalance ratio as the training data to estimate the imbalance ratio.
If these samples are not given, NEMGAN is the same method as ClusterGAN.
On the other hand, SLOGAN does not require labeled samples to learn imbalanced attributes.
Even when a small amount of probe data is used to learn specific attributes, imbalanced attributes can be learned with balanced probe data (e.g., 30 male and 30 female faces on the CelebA-HQ dataset).
B. R EPRESENTATION LEARNING
Representation learning refers to the discovery of meaningful semantics in datasets (Bengio et al., 2013).
Perceptually salient and semantically meaningful representations can induce better performance in downstream tasks.
In particular, Representation learning in generative models enables the learned latent representations to be semantically meaningful without labels in the training data.
BiGAN (Donahue et al., 2017), ALI (Dumoulin et al., 2017), and their variants (Jaiswal et al., 2018;
Belghazi et al., 2018) are similar to our study in that they add an encoder network to the original
GAN framework.
The additional encoder may serve as a inversion mapping of the generator, and learned feature representation is useful for supervised tasks.
However, the learned generator cannot perform conditional generation without supervision.
C P ROOFS C. G RADIENT I DENTITY FOR µ c
Theorem 1.
Given an expected loss of the generator L and a loss function for a sample ℓ ( · ) : R d z (cid:55)→ R , we assume ℓ is continuously differentiable.
Then, the following identity holds:
[Equation]
Proof.
To prove Theorem
[Equation]
the following lemma (Bonnet’s theorem) is introduced.
Lemma 2.
Let h ( z ) :
R d (cid:55)→ R be continuously differentiable.
q ( z ) is a multivariate Gaussian distribution N ( z ; µ , Σ ) .
Then, the following identity holds:
[Equation]
The proof of Lemma 2 is described by Theorem 3 of Lin et al.
Using Lemma
[Equation]
we show that
[Equation]
C. F IRST - ORDER G RADIENT I DENTITY FOR Σ c
Theorem 2.
With the same assumptions from Theorem
[Equation]
we have the following gradient identity
[Equation]
Proof.
In order to prove Theorem
[Equation]
we introduce the following lemma (Price’s theorem)
[Equation]
Lemma 3.
Let h ( z ) :
R d (cid:55)→ R and its derivative ∇ h ( z ) be continuously differentiable.
We further assume that E [ h ( z )] is well-defined.
Then, the following identity holds:
[Equation]
The proof of Lemma 3 is presented in Theorem 4 of Lin et al.
The rest of the proof is similar with the proof of Theorem 1.
Using the first-order gradient identity of Lemma 3, we get
[Equation]
[Equation]
C. E NSURING P OSITIVE - DEFINITENESS OF Σ c
Theorem 3.
The updated covariance matrix Σ ′ c = Σ c + γ ∆Σ ′ c with the modified update rule specified in Equation 6 is positive-definite if Σ c is positive-definite.
Proof.
Because Σ c is symmetric and positive-definite, we can decompose Σ c = LL T using Cholesky decomposition, where L is the lower triangular matrix.
Then, we can prove the positive-definiteness of the updated covariance matrix as follows:
[Equation]
Let us define U :=
L T + γ L − 1 ∆Σ c .
Then, we have the following:
[Equation]
34) where both Σ c and U T U are positive-definite, concluding the proof.
C. G RADIENT I DENTITY FOR ρ c
Theorem 4.
Let ρ c be a mixing coefficient parameter, and the following gradient identity holds:
[Equation]
Proof.
The gradient of the latent distribution with respect to the mixing coefficient parameter is derived as follows:
[Equation]
where softmax( · ) denotes the softmax function (e
[Equation]
g., p ( i ) =
π i = softmax( ρ i ) ).
Using the above equation
[Equation]
we have
[Equation]
Here, [Equation] Ki = [Equation]
π i
[Equation]
z
[Equation]
i =
[Equation]
= 1 .
Plugging this back into Equation 40, we obtain
[Equation]
C. S ECOND - ORDER G RADIENT I DENTITY FOR Σ c
We consider a generator σ ( G ( z ))
=
σ ( G ) =
σ G , a discriminator D ( σ G ) =
D with the piecewiselinear activation functions (e.g. ReLU or LeakyReLU) except the activation function of the output layer of the generator which is the hyperbolic tangent function σ ( x ) =
( e x − e − x ) / ( e x + e − x ) , and a
Wasserstein adversarial loss function for each latent vector ℓ ( z ) = − D ( σ ( G ( z ))) =
ℓ .
We note that the following holds except on a set of zero Lebesgue measure because the piecewise-linear activation functions are linear except where they switch:
[Equation]
We provide the second-order gradient identity of the Wasserstein GAN loss for the covariance matrix Σ c that does not compute second-order derivatives.
However, it is impractical and not included in our method because of the excessive computational cost of Jacobian matrices.
Theorem 5.
Given the Wasserstein GAN loss for the generator L and a loss function for a sample ℓ ( z ) : R d (cid:55)→ R , we assume ℓ and its derivative ∇ ℓ ( z ) are continuously differentiable.
We further assume that E [ ℓ ( z )] is well-defined.
Then, the following identity holds:
[Equation]
where ⊙ denotes element-wise multiplication of vectors and x ◦ i denotes the i-th Hadamard (elementwise) power of a vector x .
Proof.
From the second-order identity of Lemma
[Equation]
we have the following:
[Equation]
The Hessian of the sample loss with respect to z is given as follows:
[Equation]
where σ ′ G = ∇ G σ G .
To simplify an element ( ∇ 2 z ℓ ( z ))
[Equation]
= − ∂∂z j (cid:16)(cid:80)
[Equation]
x
[Equation]
G )
[Equation]
σ
[Equation]
i ∂z k
[Equation]
for arbitrary j [Equation] k ∈ { 1 , ..., d z } , we have
[Equation]
Therefore
[Equation]
the simplified element is
[Equation]
We now vectorize the expression of ∇ 2 z ℓ ( z ) as follows:
[Equation]
Plugging this to Equation 47, the following is obtained:
[Equation]
D M ETHODOLOGICAL D ETAILS D. A DDITIVE A NGULAR M ARGIN
To enhance the discriminative power of U2C loss, we adopted the additive angular margin (Deng et al., 2019) as follows:
[Equation]
where s denotes the feature scale, and m is the angular margin.
The feature scale m and the coefficient of U2C loss λ are linearly decayed to 0 during training, so that SLOGAN can focus more on the adversarial loss as training progresses.
D. C LUSTER A SSIGNMENT
In Section 3.3, we chose cos θ ic as the critic function assuming that it is proportional to log p ( c | x g ) .
If the real data distribution p ( x r ) and the generator distribution p ( x g ) are sufficiently similar via adversarial learning, the cosine similarity between E ( x r ) and µ c can also be considered proportional to log p ( c | x r ) .
Therefore, for real data, we obtain the probability for each cluster as follows:
[Equation]
where cos θ k = E ( x r ) · µ k / ∥ E ( x r ) ∥∥ µ k ∥ is the cosine similarity between E ( x r ) and µ k .
The data can then be assigned to the cluster with the highest probability (i.e., argmax c ˆ p ( c | x r ) ).
D. A TTRIBUTE M ANIPULATION
We utilized mixup (Zhang et al., 2018) to make the best use of a small amount of probe data when manipulating attributes.
Algorithm 3 describes the procedure for using mixup for attribute manipulation when K = 2 .
We applied the same feature scale and angular margin to L m , as shown in Equation 66.
The number of iterations for the mixup ( T ) was set to five.
Initialize probe data with the desired attribute X c =1 ← { x ic =1 } Mi =1 and ¯ X c =1 ← { x ic =1 } Mi =
Initialize probe data without the desired attribute X c =
← { x ic =0 } Mi =1 and ¯ X c =
← { x ic =0 } Mi =1 for each mixup iteration t in { 1 , ..., T } do ¯ X c
=1 ← ¯ X c
=1 ∪ M IXUP ( X c =1 , P ERMUTE ( X c =1 ))
¯ X c =
← ¯ X c =0 ∪ M IXUP ( X c =0 , P ERMUTE ( X c =0 )) end for for each augmented data index j in { 1 , ..., M ( T + 1) } do cos θ
[Equation]
←
[Equation]
cos θ
[Equation]
01 ←
[Equation]
cos θ j 10 ← E (¯ x jc =1 ) · µ
[Equation]
/ ∥ E (¯ x jc =1 ) ∥∥ µ 0 ∥ cos θ j 11 ← E (¯ x jc =1 ) · µ 1 / ∥ E (¯ x jc =1 ) ∥∥ µ 1 ∥ end for
L m = −
[Equation]
log
[Equation]
j
[Equation]
exp
[Equation]
exp
[Equation]
cos
[Equation]
θ j
[Equation]
Minimize L p with respect to E , G , and µ
D. S IM CLR
For the CIFAR and CelebA datasets, we used the SimCLR loss (Chen et al., 2020) for the encoder.
We applied color, translation, and cutout transformations to the generated data using DiffAugment
(Zhao et al., 2020).
The SimCLR loss is calculated using the generated data x ig and augmentation A as follows:
[Equation]
where x ig = G ( z i ) .
The encoder E is trained to minimize 1 B (cid:80) Bi
=1 (cid:0) ℓ adv ( z i ) + ℓ SimCLR
( z i ) +
[Equation]
[Equation] (cid:1)
[Equation]
D. D E L I GAN+
Among the existing unsupervised conditional GANs, DeLiGAN lacks an encoder network.
Therefore, for a fair comparison, we added an encoder network and named it DeLiGAN+.
We set the output dimension of the encoder to equal the number of mixture components of the latent distribution.
For the i -th example in the batch, when the c i -th mixture component of the latent distribution is selected,
DeLiGAN+ is learned through the following objective:
[Equation]
(69) where c i is the one-hot vector corresponding to c
i and λ CE is the coefficient of the cross entropy loss.
We set λ CE to 10 in the experiments.
D. E VALUATION M ETRIC
Cluster assignment We do not use clustering purity which is an evaluation metric for cluster assignment.
To compute the clustering purity, the most frequent class in the cluster is obtained, and the ratio of the data points belonging to the class is calculated.
However, if the attributes in the data are imbalanced, multiple clusters can be assigned to a single class in duplicate, and this high clustering purity misleads the results.
Therefore, we utilized the normalized mutual information (NMI) implemented in scikit-learn 2 .
Unconditional generation FID has the advantage of considering not only sample quality but also diversity, whereas Inception score (IS) cannot assess the diversity properly because IS does not compare generated samples with real samples (Shmelkov et al., 2018).
Therefore, we used FID as the evaluation metric for unsupervised generation.
Unsupervised conditional generation If attributes in data are severely imbalanced, FID does not increase (deteriorate) considerably even if the model does not generate data containing the minority attributes.
Therefore, the FID cannot accurately measure the unsupervised conditional generation performance for data with severely imbalanced attributes.
We introduce ICFID to evaluate the performance of unsupervised conditional generation.
When calculating ICFID, multiple clusters cannot be assigned to a single class in duplicate.
Therefore, if data of a single class are generated from multiple discrete latent variables or modes, the model shows high (bad) ICFID.
E I MPLEMENTATION D ETAILS E. G ENERAL S ETTINGS AND E NVIRONMENTS
For simplicity, we denote the learning rate of G as η and the learning rate of Σ as γ .
Throughout the experiments, we set the learning rate of E to η , and D to 4 η using the two-timescale update rule (TTUR)
(Heusel et al., 2017).
We set the learning rate of µ to 10 γ , and the learning rate of ρ to γ .
We set B to 64 and the number of training steps to 100k.
To stabilize discriminator learning, we used Lipschitz penalty (Petzka et al., 2018) for the synthetic, MNIST, FMNIST, and 10x
73k datasets, and adversarial Lipschitz regularization (Terj´ek, 2020) for the CIFAR-10 and CelebA datasets.
We repeated each experiment 3 times and reported the means and standard deviations of model performances.
Hyperparameters are determined by a grid search.
We used the Adam optimizer (Kingma & Ba, 2014) for training D , G , and E , and a gradient descent optimizer for training µ , Σ , and ρ .
The experiments herein were conducted on a machine equipped with an Intel Xeon Gold 6242 CPU and an NVIDIA Quadro RTX 8000 GPU.
The code is implemented in Python 3.7 and
Tensorflow 1.14 (Abadi et al., 2016).
E. S YNTHETIC D ATASET
For the synthetic dataset, we first set the mean of eight 2-dimensional Gaussian distributions as (0 , 2) , ( √ 2 , √ 2) , (2 , 0) , ( √ 2 , −√ 2) , (0 , − 2) , ( −√ 2 , −√ 2) , ( − 2 , 0) , and ( −√ 2 , √ 2) , and the variance as 0 .
01 I .
In Figure 11, we also set the variances as 0 .
I and 0 .
1 I .
The number of data sampled from the Gaussian distributions was set to 5,000, 5,000, 5,000, 5,000, 15,000, 15,000, 15,000, and 15,000.
We scaled a total of 80,000 data points to a range between -1 and 1.
Table 11 shows the network architectures of SLOGAN used for the synthetic dataset.
Linear n denotes a fully-connected layer with n output units.
BN and SN denote batch normalization and spectral normalization, respectively.
LReLU denotes the leaky ReLU.
We set λ = 4 , η = 0 .
001 , γ = 0 .
01 , s = 2 , and m = 0 .
[Table]
E. MNIST AND F ASHION -MNIST D ATASETS
The MNIST dataset (LeCun et al., 1998) consists of handwritten digits, and the Fashion-MNIST (FMNIST) dataset (Xiao et al., 2017) is comprised of fashion products.
Both the MNIST and FashionMNIST datasets have 60,000 training and 10,000 test 28 × 28 grayscale images.
Each pixel was scaled to a range of 0 − 1.
The datasets consist of 10 classes, and the number of data points per class is balanced.
Table 12 shows the network architectures of SLOGAN used for the MNIST and FMNIST datasets.
Conv k × k , s , n denotes a convolutional network with n feature maps, filter size k × k , and stride s .
Deconv k × k , s , n denotes a deconvolutional network with n feature maps, filter size k × k , and stride s .
For the MNIST dataset, we set λ
= 10 , η = 0 . 0001 , γ = 0 .
002 , s = 8 , and m = 0 .
For MNIST-2, we set λ = 4 , η = 0 . 0001 , γ = 0 .
002 , s = 4 , and m = 0 .
For the
FMNIST dataset, we set λ
= 10 , η = 0 . 0001 , γ = 0 .
001 , s = 1 , and m = 0 .
For FMNIST-5, we set λ = 1 , η = 0 . 0002 , γ = 0 .
004 , s = 4 , and m = 0 .
E. 10 X 73 K D ATASET
The 10x 73k dataset (Zheng et al., 2017) consists of 73,233 720-dimensional vectors, which are obtained from RNA transcript counts, and has eight cell types (classes).
The number of data points per cell type is 10,085, 2,612, 9,232, 8,385, 10,224, 11,953, 10,479, and 10,263.
We converted each element to logscale (i.e., log 2 ( x + 1) ) and scaled each element to a range between 0 and 1.
Table 13 shows the network architectures of SLOGAN used for the 10x 73k dataset.
We set λ = 10 , η = 0 . 0001 , γ = 0 .
004 , s = 4 , and m = 0 .
[Table] [Table]
E. CIFAR-10 D ATASET
The CIFAR-10 (Krizhevsky et al., 2009) dataset comprises 50,000 training and 10,000 test 32 ×color images.
Each pixel was scaled to a range of -1 to 1.
The number of data points per class is balanced.
Figure 23 and Table 14 show the network architectures of residual blocks and SLOGAN used for the CIFAR datasets.
AvgPool and GlobalAvgPool denote the average pooling and global average pooling layers, respectively.
For the CIFAR-10, CIFAR-2, CIFAR-2 (7:3)
[Table]
and CIFAR- (9:1) datasets, we set λ
= 1 , η = 0 . 0001 , γ = 0 .
002 , s = 4 , and m = 0 .
[Figure]
× 3,
[Figure]
Conv 3 × 3, 1,
[Figure]
ReLU
Conv 3 × 3, 1,
BN + ReLU Upsample 2 × BN + ReLU
Upsample 2 × Conv 3 × 3, 1,
Conv 3 × 3, 1, Conv 3 × 3, 1,
ReLU
ReLU
AvgPool 2 ×
AvgPool 2 ×
[Figure]
[Figure]
[Table] [Figure]
E. C ELEB A D ATASET
The CelebA dataset (Liu et al., 2015) consists of 202,599 face attributes.
We cropped the face part of each image to 140 × 140 pixels, resized it to 64 × 64 pixels
[Figure]
and scaled it to a range between -1 and
[Figure]
The imbalanced ratio is different for each attribute, and the imbalanced ratios of male and eyeglasses used in the experiment are 1:1 and 14:1, respectively.
Table 15 lists the network architectures of
SLOGAN used for the CelebA dataset.
We set λ = 1 , η = 0 . 0002 , γ = 0 .
0006 , s = 4 , and m = 0 .
[Table]
E. C ELEB A-HQ D ATASET
The CelebA-HQ dataset (Karras et al., 2017) consists of 30,000 face attributes.
We resized each image to 128 × 128 and 256 × 256 pixels, and scaled it to a range between -1 and 1.
The imbalance ratio of male used in the experiment was 1.7:1.
We used StyleGAN2 (Karras et al., 2020) architecture with DiffAugment 3 and applied implicit reparameterization to the input space of the mapping network.
We set λ = 1 , η = 0 .
002 , γ = 0 .
0006 , s = 1 , and m = 0 .
E. AFHQ D ATASET
The AFHQ dataset (Choi et al., 2020) consists of 15,000 high-quality animal faces.
We used cats and dogs, and there are about 5,000 images each in the dataset.
We resized each image to 256 ×pixels, and scaled it to a range between -1 and 1.
We set the imbalance ratios of cats and dogs to 1:1, 1:2, and 1:5 by reducing the number of cats in the training dataset.
We used the same model architecture and hyperparameters as for the CelebA-HQ dataset.
E. C ODE A VAILABILITY
Code is available at https://github.com/shinyflight/SLOGAN