C C GAN: C ONTINUOUS C ONDITIONAL G ENERATIVE A DVERSARIAL N ETWORKS FOR I MAGE G ENERATION
A BSTRACT
This work proposes the continuous conditional generative adversarial network (CcGAN), the ﬁrst generative model for image generation conditional on continuous , scalar conditions (termed regression labels ).
Existing conditional GANs (cGANs) are mainly designed for categorical conditions (e.g., class labels); conditioning on regression labels is mathematically distinct and raises two fundamental problems: (P1) Since there may be very few (even zero) real images for some regression labels, minimizing existing empirical versions of cGAN losses (a.k.a. empirical cGAN losses) often fails in practice; (P2)
Since regression labels are scalar and inﬁnitely many, conventional label input methods (e.g., combining a hidden map of the generator/discriminator with a one-hot encoded label) are not applicable.
The proposed CcGAN solves the above problems, respectively, by (S1) reformulating existing empirical cGAN losses to be appropriate for the continuous scenario; and (S2) proposing a novel method to incorporate regression labels into the generator and the discriminator.
The reformulation in (S1) leads to two novel empirical discriminator losses, termed the hard vicinal discriminator loss (HVDL) and the soft vicinal discriminator loss (SVDL) respectively, and a novel empirical generator loss.
The error bounds of a discriminator trained with HVDL and SVDL are derived under mild assumptions in this work.
A new benchmark dataset, RC-49, is also proposed for generative image modeling conditional on regression labels.
Our experiments on the Circular 2-D Gaussians, RC-49, and UTKFace datasets show that CcGAN is able to generate diverse, high-quality samples from the image distribution conditional on a given regression
label.
Moreover, in these experiments,
CcGAN substantially outperforms cGAN both visually and quantitatively
I NTRODUCTION
Conditional generative adversarial networks (cGANs), ﬁrst proposed in (Mirza & Osindero, 2014), aim to estimate the distribution of images conditioning on some auxiliary information, especially class labels.
Subsequent studies (Odena et al., 2017; Miyato & Koyama, 2018; Brock et al., 2019;
Zhang et al., 2019) conﬁrm the feasibility of generating diverse, high-quality (even photo-realistic), and class-label consistent fake images from class-conditional GANs.
Unfortunately, these cGANs do not work well for image generation with continuous, scalar conditions, termed regression labels , due to two problems:
(P1) cGANs are often trained to minimize the empirical versions of their losses (a.k.a.
the empirical cGAN losses) on some training data, a principle also known as the empirical risk minimization (ERM) (Vapnik, 2000).
The success of ERM relies on a large sample size for each distinct condition.
Unfortunately, we usually have only a few real images for some regression labels.
Moreover, since regression labels are continuous, some values may not even appear in the training set.
Consequently, a cGAN cannot accurately estimate the image distribution conditional on such missing labels
(P2)
In class-conditional image generation, class labels are often encoded by one-hot vectors or label embedding and then fed into the generator and discriminator by hidden concatenation (Mirza & Osindero, 2014), an auxiliary classiﬁer (Odena et al., 2017) or label projection (Miyato & Koyama, 2018).
A precondition for such label encoding is that the number of distinct labels (e.g., the number of classes) is ﬁnite and known.
Unfortunately, in the continuous scenario, we may have inﬁnite distinct regression labels.
A naive approach to solve (P1)-(P2) is to “bin” the regression labels into a series of disjoint intervals and still train a cGAN in the class-conditional manner (these interval are treated as independent classes)
(Olmschenk, 2019).
However, this approach has four shortcomings: (1) our experiments in Section 4 show that this approach often makes cGANs collapse; (2) we can only estimate the image distribution conditional on membership in an interval and not on the target label; (3) a large interval width leads to high label inconsistency; (4) inter-class correlation is not considered (images in successive intervals have similar distributions).
In machine learning, vicinal risk minimization (VRM) (Vapnik, 2000; Chapelle et al., 2001) is an alternative rule to ERM.
VRM assumes that a sample point shares the same label with other samples in its vicinity.
Motivated by VRM, in generative modeling conditional on regression labels where we estimate a conditional distribution p ( x | y ) ( x is an image and y is a regression label), it is natural to assume that a small perturbation to y results in a negligible change to p ( x | y ) .
This assumption is consistent with our perception of the world.
For example, the image distribution of facial features for a population of 15-year-old teenagers should be close to that of 16-year olds.
We therefore introduce the continuous conditional GAN (CcGAN) to tackle (P1) and (P2) .
To our best knowledge, this is the ﬁrst generative model for image generation conditional on regression labels.
It is noted that Rezagholizadeh et al.
and Rezagholiradeh & Haidar (2018) train
GANs in an unsupervised manner and synthesize unlabeled fake images for a subsequent image regression task.
Olmschenk et al.
(2019) proposes a semi-supervised GAN for dense crowd counting.
CcGAN is fundamentally different from these works since they do not estimate the conditional image distribution.
Our contributions can be summarized as follows:
• We propose in Section 2 the CcGAN to address (P1) and (P2) , which consists of two novel empirical discriminator losses, termed the hard vicinal discriminator loss (HVDL) and the soft vicinal discriminator loss (SVDL), a novel empirical generator loss, and a novel label input method.
We take the vanilla cGAN loss as an example to show how to derive HVDL, SVDL, and the novel empirical generator loss by reformulating existing empirical cGAN losses.
• We derive in Section 3 the error bounds of a discriminator trained with HVDL and SVDL.
• In Section 4, we propose a new benchmark dataset, RC-49, for the generative image modeling conditional on regression labels, since very few benchmark datasets are suitable for the studied continuous scenario.
We conduct experiments on several datasets and our experiments show that CcGAN not only generates diverse, high-quality, and label consistent images, but also substantially outperforms cGAN both visually and quantitatively.
F ROM C GAN TO C C GAN
In this section, we provide the solutions (S1) - (S2) to (P1) - (P2) in a one-to-one manner by introducing the continuous conditional GAN (CcGAN).
Please note that theoretically cGAN losses (e.g., the vanilla cGAN loss (Mirza & Osindero, 2014), the Wasserstein loss (Arjovsky et al., 2017), and the hinge loss (Miyato et al., 2018)) are suitable for both class labels and regression labels; however, their empirical versions fail in the continuous scenario (i.e., (P1) ).
Our ﬁrst solution (S1) focuses on reformulating these empirical cGAN losses to ﬁt into the continuous scenario.
Without loss of generality, we only take the vanilla cGAN loss as an example to show such reformulation (the empirical versions of the Wasserstein loss and the hinge loss can be reformulated similarly) [Equation]
The vanilla discriminator loss and generator loss (Mirza & Osindero, 2014) are deﬁned as:
[Equation]
[Equation] where x ∈ X is an image of size d × d , y ∈ Y is a label, p r ( y ) and p g ( y ) are respectively the true and fake label marginal distributions, p r ( x | y ) and p g ( x | y ) are respectively the true and fake image distributions conditional on y , p r ( x , y ) and p g ( x , y ) are respectively the true and fake joint distributions of x and y , and q ( z ) is the probability density function of N ( 0 , I ) .
Since the distributions in the losses of Eqs.
(1) and (2) are unknown, for class-conditional image generation, Mirza & Osindero (2014) follows ERM and minimizes the empirical losses:
[Equation]
where C is the number of classes, N r and N g are respectively the number of real and fake images,
N rc and N gc are respectively the number of real and fake images with label c , x rc,j and x gc,j are respectively the j -th real image and the j -th fake image with label c , and the z c,j are independently and identically sampled from q ( z ) .
Eq.
(3) implies we estimate p r ( x , y ) and p g ( x , y ) by their empirical probability density functions as follows:
[Equation]
where δ ( · ) is a Dirac delta mass centered at 0.
However, ˆ p δr ( x , y ) and ˆ p δg ( x , y ) in Eq.
(5) are not good estimates in the continuous scenario because of (P1) .
To overcome (P1) , we propose a novel estimate for each of p r ( x , y ) and p g ( x , y ) , termed the hard vicinal estimate (HVE).
We also provide an intuitive alternative to HVE, named the soft vicinal estimate (SVE).
The HVEs of p r ( x , y ) and p g ( x , y ) are:
[Equation]
where x r i and x gi are respectively real image i and fake image i ,
y r i and y gi are respectively the labels of x ri and x gi , κ and σ are two positive hyper-parameters, C 1 and C 2 are two constants making these two estimates valid probability density functions, N ry,κ is the number of the y ri satisfying | y − y ri | ≤ κ ,
N gy,κ is the number of the y gi satisfying | y − y gi | ≤ κ , and 1 is an indicator function with support in the subscript.
The terms in the ﬁrst square brackets of ˆ p HVE r and ˆ p HVE
g imply we estimate the marginal label distributions p r ( y ) and p g ( y ) by kernel density estimates (KDEs) (Silverman, 1986).
The terms in the second square brackets are designed based on the assumption that a small perturbation to y results in negligible changes to p r ( x | y ) and p g ( x | y ) .
If this assumption holds, we can use images with labels in a small vicinity of y to estimate p r ( x | y ) and p g ( x | y ) .
The SVEs of p r ( x , y ) and p g ( x , y ) are:
[Equation]
where C 3 and C 4 are two constants making these two estimates valid probability density functions, [Equation] and [Equation]
, y
[Equation]
and the hyper-parameter ν > 0 .
In Eq.
(7), similar to the HVEs, we estimate p r ( y ) and p g ( y ) by
KDEs.
Instead of using samples in a hard vicinity, the SVEs use all respective samples to estimate p r ( x | y ) and p g ( x | y )
but each sample is assigned with a weight based on the distance of its label from y .
Two diagrams in Fig.
1 visualize the process of using hard/soft vicinal samples to estimate p ( x | y ) , i.e., a univariate Gaussian distribution conditional on its mean y . [Figure]
By plugging Eq.
(6) and (7) into Eq.
(1), we derive the hard vicinal discriminator loss (HVDL) and the soft vicinal discriminator loss (SVDL) as follows:
[Equation]
y − y gj , and C 5 , C 6 , C 7 , and C 8 are some constants.
Generator training: The generator of CcGAN is trained by minimizing Eq.
[Equation]
[Equation]
How do HVDL, SVDL, and Eq.
(11) overcome (P1)?
The solution (S1) includes
: (i) Given a label y
as the condition, we use images in a hard/soft vicinity of y to train the discriminator instead of just using images with label y .
It enables us to estimate p r ( x | y ) when there are not enough real images with label y .
(ii) From Eqs.
(9) and (10), we can see that the KDEs in Eqs.
(6) and (7) are adjusted by adding Gaussian noise to the labels.
Moreover, in Eq.
(11), we add Gaussian noise to seen labels (assume
y gi ’s are seen) to train the generator to generate images at unseen labels.
This enables estimation of p r ( x | y (cid:48) ) when y (cid:48) is not in the training set.
How is (P2) solved?
We propose a novel label input method.
For G , we add the label y elementwisely to the output of its ﬁrst linear layer.
For D , an extra linear layer is trained together with D to embed y in a latent space.
We then incorporate the embedded label into D by the label projection (Miyato & Koyama, 2018).
Please refer to Supp.
S.3 for more details.
Remark 1.
An algorithm is proposed in Supp.
S.2 for training CcGAN in practice.
Moreover, CcGAN does not require any speciﬁc network architecture, therefore it can also use the state-of-art architectures in practice such as SNGAN (Miyato et al., 2018) and BigGAN (Brock et al., 2019).
E RROR BOUNDS
In this section, we derive the error bounds of a discriminator trained with (cid:98) L HVDL and (cid:98) L SVDL under the theoretical loss L .
First, without loss of generality, we assume y ∈ [0 , 1] .
Then, we introduce some notations.
Let D stand for the Hypothesis Space of D .
Let ˆ p KDE r ( y ) and ˆ p KDE g ( y ) stand for the KDEs of p r ( y ) and p g ( y ) respectively.
Let p r w ( y (cid:48) | y ) (cid:44) w r
( y (cid:48) ,y ) p r ( y (cid:48) )
W r ( y ) , p g w ( y (cid:48) | y ) (cid:44) w g ( y (cid:48) ,y )
p g ( y (cid:48) )
W g ( y ) , W r ( y ) (cid:44) (cid:82) w r
( y (cid:48) , y ) p r ( y (cid:48) ) dy (cid:48) and W g ( y ) (cid:44) (cid:82) w g ( y (cid:48) , y )
p g ( y (cid:48) ) dy (cid:48) .
Denote by D ∗ the optimal discriminator (Goodfellow et al., 2014) which minimizes L but may not be in D .
Let (cid:101) D
(cid:44) arg min D ∈D
L ( D ) .
Let (cid:98) D HVDL (cid:44) arg min D ∈D (cid:98) L HVDL ( D ) ; similarly, we deﬁne (cid:98) D SVDL .
Deﬁnition 1.
(H ¨ older Class) Deﬁne the H ¨ older class of functions
[Equation]
Please see Supp.
S.5.1 for more details of these notations.
Moreover, we will also work with the following assumptions: (A1)
All D ’s in D are measurable and uniformly bounded by U .
Let U (cid:44) max { sup D ∈D [ − log D ] , sup D ∈D [ − log(1 − D )] } and U < ∞ ; (A2)
For ∀ x ∈ X and y, y (cid:48) ∈ Y , ∃ g r ( x ) > 0 and M r > 0 , s.t.
| p r ( x | y (cid:48) )
− p r
( x | y ) | ≤ g r ( x ) | y (cid:48) − y | with (cid:82) g r ( x
) d x = M r ; (A3)
For ∀ x ∈ X and y, y (cid:48) ∈ Y , ∃ g g ( x ) > 0 and M g > 0 , s.t.
| p g ( x | y (cid:48) ) − p g
( x | y ) | ≤ g g ( x ) | y (cid:48) − y | with (cid:82) g g ( x ) d x = M g ; (A4) p r ( y ) ∈ Σ( L r ) and
p g ( y ) ∈ Σ( L g ) .
Theorem 1.
Assume that (A1)-(A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
L ( (cid:98) D HVDL
[Equation]
− L
[Equation]
D ∗ )
[Equation]
[Equation]
for some constants C KDE 1 ,δ , C KDE 2 ,δ depending on δ .
Theorem 2.
Assume that (A1)-(A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
L ( (cid:98) D SVDL
[Equation]
− L [Equation] D ∗ [Equation]
[Equation]
Remark 2.
The error bounds in both theorems reﬂect the distance of (cid:98) D HVDL and (cid:98) D SVDL from D ∗ .
Enlightened by the two upper bounds, when implementing CcGAN, we should (1) avoid letting D output extreme values (close to 0 or 1) so that U is kept at a moderate level; (2) avoid using a too small or a too large κ or ν to keep the third and fourth terms moderate in Eqs.
(13) and (14).
Please see Supp.
S.5.2.5 for a more detailed interpretation and Supp.
S.5.2 for the proofs.
E XPERIMENT
In this section, we study the effectiveness of CcGAN on three datasets where cGAN (Mirza &
Osindero, 2014) cannot generate realistic samples.
For a fair comparison, cGAN and CcGAN use the same network architecture (a customized architecture for Circular 2-D Gaussians and the SNGAN (Miyato et al., 2018) architecture for RC-49 and UTKFace) except for the label input modules.
For stability, image labels are normalized to [0 , 1] in the RC-49 and UTKFace datasets during training.
4. C IRCULAR 2-D G AUSSIANS
We ﬁrst test on the synthetic data generated from 120 2-D Gaussians with different means.
Experimental setup: The means of the 120 Gaussians are evenly arranged on a unit circle centered at the origin O of a 2-D space.
The Gaussians share a common covariance matrix ˜ σ 2 I 2 × 2 , where ˜ σ = 0 .
We generate 10 samples from each Gaussian for training.
Fig.
2a shows 1,200 training samples (blue dots) from these Gaussians with their means (red dots) on a unit circle.
The unit circle can be seen as a clock where we take the mean at 12 o’clock (point A ) as the baseline point.
Given another mean on the circle (point B ), the label y for samples generated from the Gaussian with mean
B is deﬁned as the clockwise angle (in radians) between line segments OA and OB .
E.g., the label for samples from the Gaussian at A is 0.
Both cGAN and our proposed CcGAN are trained on this training set.
When implementing cGAN, angles are treated as class labels (each Gaussian is treated as a class); while when implementing CcGAN, angles are treated as real numbers.
The network architectures of cGAN and CcGAN are shown in Supp.
S.6.1.
Both cGAN and CcGAN are trained for 6,000 iterations.
We use the rule of thumb formulae in Supp.
S.4 to select the hyper-parameters of HVDL and SVDL, i.e., σ ≈ 0 .
074 , κ ≈ 0 .
017 and ν = 3600 (see Supp.
S.6.2 for details).
For testing, we choose 360 points evenly distributed on the unit circle as the means of 360 Gaussians.
For each Gaussian, we generate 100 samples, yielding a test set with 36,000 samples.
It should be noted that, among these 360 Gaussians, at least 240 are not used at the training.
In other words, there are at least 240 labels in the testing set which do not appear in the training set.
For each test angle, we generate 100 fake samples from each trained GAN, yielding 36,000 fake samples from each GAN in total.
The quality of these fake samples is evaluated.
We repeat the whole experiment three times and report in Table 1 the average quality over three repetitions.
Evaluation metrics and quantitative results: In the label-conditional scenario, each fake sample x with label y is compared with the mean (sin( y ) , cos( y )) of a Gaussian on the unit circle with label y .
A fake sample is deﬁned as ”high-quality” if its Euclidean distance from x to (sin( y ) , cos( y )) is smaller than 4˜ σ = 0 .
A mode (i.e., a Gaussian) is said to be recovered if at least one high-quality sample is assigned to it.
We also measure the quality of fake samples with label y by computing the 2-Wasserstein Distance ( W 2 ) (Peyr´e et al., 2019) between p r ( x | y )
=
N ([sin( y ) , cos( y )] (cid:124) , ˜ σ I ) and p g ( x | y )
=
N ( µ gy , Σ gy ) , where we assume p g ( x | y ) is Gaussian and its mean and covariance are estimated by the sample mean and sample covariance of 100 fake samples with label y .
In Table 1, we report the average percentage of high-quality fake samples and the average percentage of recovered modes over 3 repetitions.
We also report the average W 2 over 360 testing angles.
We can see CcGAN substantially outperforms cGAN.
Visual results: We select 12 angles which do not appear in the training set.
We then use cGAN and
CcGAN to generate 100 samples for each unobserved angle.
Fig.
2 visually conﬁrms the obervation from the numerical metrics: the fake samples from the two CcGAN methods are more realistic. 4. RC-
Since most benchmark datasets in the GAN literature do not have continuous, scalar regression labels, we propose a new benchmark dataset—RC-49, a synthetic dataset created by rendering 49 3-D chair
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
models at different yaw angles.
Each of 49 chair models is rendered at 899 yaw angles ranging from 0.1 to 89.9 with step size 0.1.
Therefore, RC-49 consists of 44,051 64 × 64 rendered RGB images and 899 distinct angles.
Please see Supp.
S.7 for more details of the data generation.
Some example images are shown in Fig.
Experimental setup: Not all images are used for the cGAN and CcGAN training.
A yaw angle is selected for training if its last digit is odd.
Moreover, at each selected angle, only 25 images are randomly chosen for training.
Thus, the training set includes 11250 images and 450 distinct angles.
The remaining images are held out for evaluation.
When training cGAN, we divide [0 . 1 , 89 . 9] into 150 equal intervals where each interval is treated as a class.
When training CcGAN, we use the rule of thumb formulae in Supp.
S.4 to select the three hyper-parameters of HVDL and SVDL, i.e., σ ≈ 0 .
047 , κ ≈ 0 .
004 and ν = 50625 .
Both cGAN and
CcGAN are trained for 30,000 iterations with batch size 256.
Afterwards, we evaluate the trained
GANs on all 899 angles by generating 200 fake images for each angle.
Please see Supp.
S.7 for the network architectures and more details about the training/testing setup.
Quantitative and visual results: To evaluate (1) the visual quality, (2) the intra-label diversity, and (3) the label consistency (whether assigned labels of fake images are consistent with their true labels) of fake images, we study an overall metric and three separate metrics here.
(i) Intra-FID (Miyato &
Koyama, 2018) is utilized as the overall metric.
It computes the Fr´echet inception distance (FID) (Heusel et al., 2017) separately at each of the 899 evaluation angles and reports the average FID score.
(ii) Naturalness Image Quality Evaluator (NIQE) (Mittal et al., 2012) measures the visual quality only.
(iii) Diversity is the average entropy of predicted chair types of fake images over evaluation angles.
(iv) Label Score is the average absolute error between assigned labels and predicted labels.
Please see Supp.
S.7.5 for details of these metrics.
We report in Table 2 the performances of each GAN.
The example fake images in Fig.
3 and line graphs in Fig.
5 support the quantitative results.
cGAN often generates unrealistic, identical images for a target angle (i.e., low visual quality and low intra-label diversity).
“Binning” [0 . 1 , 89 . 9] into other number of classes (e.g., 90 classes and 210 classes) is also tried but does not improve cGAN’s performance.
In contrast, strikingly better visual quality and higher intra-label diversity of both
CcGAN methods are visually evident.
Please note that CcGAN is designed to sacriﬁce some (not too much) label consistency for better visual quality and high diversity, and this explains why CcGAN does not outperform cGAN in terms of the label score in Table 2.
[Table]
In this section, we compare CcGAN and cGAN on UTKFace (Zhang et al., 2017), a dataset consisting of RGB images of human faces which are labeled by age.
Experimental setup: In this experiment, we only use images with age in [1 , 60] .
Some images with bad visual quality and watermarks are also discarded.
After the preprocessing, 14,760 images are left.
The number of images for each age ranges from 50 to 1051.
We resize all selected images to 64 × 64 .
Some example UTKFace images are shown in the ﬁrst image array in Fig.4.
When implementing cGAN, each age is treated as a class.
For CcGAN we use the rule of thumb formulae in Supp.
S.4 to select the three hyper-parameters of HVDL and SVDL, i.e., σ ≈ 0 .
041 , κ ≈ 0 .
017 and ν = 3600 .
Both cGAN and CcGAN are trained for 40,000 iterations with batch size 512.
In testing, we generate 1,000 fake images from each trained GAN for each age.
Please see Supp.
S.8 for more details of data preprocessing, network architectures and training/testing setup.
Quantitative and visual results: Similar to the RC-49 experiment, we evaluate the quality of fake images by Intra-FID, NIQE, Diversity (entropy of predicted races), and Label Score.
We report in Table 3 the average quality of 60,000 fake images.
We also show in Fig.
4 some example fake images from cGAN and CcGAN and line graphs of FID/NIQE versus ages in Fig.
Analogous to the quantitative comparisons, we can see that CcGAN performs much better than cGAN.
[Table]
[Figure] [Figure]
80 Angle of yaw rotation 0.
F I D cGANCcGAN (HVDL) CcGAN (SVDL) (a) RC-49: FID vs Angle    80 Angle of yaw rotation 1.
N I Q E cGANvanilla CcGAN (HVDL) vanilla CcGAN (SVDL) ( [Figure] RC [Figure]
: NIQE [Figure] Angle      60 Age F I D cGANCcGAN (HVDL) CcGAN (SVDL) (c) UTKFace: FID
[Figure] Age 10 20 30 40
60 Age 1.
N I Q E cGANvanilla CcGAN (HVDL) vanilla CcGAN (SVDL) (
[Figure]  C ONCLUSION As the ﬁrst generative model, we propose the CcGAN in this paper for image generation conditional on regression labels. In CcGAN, two novel empirical discriminator losses (HVDL and SVDL), a novel empirical generator loss and a novel label input method are proposed to overcome the two problems of existing cGANs.
The error bounds of a discriminator trained under HVDL and SVDL are studied in this work.
A new benchmark dataset, RC-49, is also proposed for the continuous scenario.
Finally we demonstrate the superiority of the proposed CcGAN to cGAN on the Circular 2-D Gaussians, RC- and UTKFace datasets.
A CKNOWLEDGMENTS
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grants CRDPJ 476594-14, RGPIN-2019-05019, and RGPAS2017-507965.
S UPPLEMENTARY M ATERIAL S. G IT H UB REPOSITORY
Please ﬁnd the codes for this paper at Github: https://github.com/UBCDingXin/improved_CcGAN
S. A LGORITHMS FOR C C GAN TRAINING
Data: N r real image-label pairs Ω r = { ( x r 1 , y r 1 ) , . . .
, ( x rN r , y rN r ) } , N r uy ordered distinct labels Υ
= { y r [1] , . . .
, y r
[ N r uy ] } in the dataset, preset σ and κ , number of iterations K , the discriminator batch size m d and generator batch size m g .
Result:
Trained generator G .
do fd
Ω fd
Remark S.3.
It should be noted that, for computational efﬁciency, the normalizing constants
N r y r j + (cid:15) r ,κ , N g y g j + (cid:15) g ,κ , (cid:80) N r i =1 w r
( y ri , y rj + (cid:15) r ) , and (cid:80) N
g
i
=
1 w g
y g i , y g j + (cid:15) g ) in Eq.
(9) and (10) are excluded from the training and only used for theoretical analysis.
S. M ORE DETAILS OF THE PROPOSED LABEL INPUT METHOD IN S ECTION
We propose a novel way to input labels to the conditional generative adversarial networks.
For the generator, we add a regression label element-wise to the feature map of the ﬁrst linear layer.
For the discriminator, labels are ﬁrst projected to a latent space learned by an extra linear layer.
Then, we incorporate the embedded labels into the discriminator by the label projection (Miyato & Koyama, 2018).
Figs.
S.3.6 and S.3.7 visualizes our proposed label input method.
Please refer to our codes for more details.
S. A RULE OF THUMB FOR HYPER - PARAMETER SELECTION
In our experiments, we normalize labels to real numbers in [0 , 1] and the hyper-parameter selection is conducted based on the normalized labels.
To be more speciﬁc, the hyper-parameter σ is computed based on a rule-of-thumb formula for the bandwidth selection of KDE (Silverman, 1986), i.e., log10 − evaluating G where z ∼ N I
Let Ω fd =
Ω fd
( x
Compute w gi y y
= e −
; end
Update D with samples in set Ω rd and Ω fd via gradient-based optimizers based on Eq.(7);
Train G ;
Draw m g labels with
Create another set of target labels training is on
Generate m g fake conditional on Y g, and put these image-label pairs in Ω fg ;
Update G with samples in Ω fg via gradient-based optimizers based on Eq. ; 18 end
[Figure] [Figure]
𝑥
Image 𝑦 label
Add
Inner product
Linear layerreal/fake σ = (cid:0) 4ˆ σ 5 y r / 3 N r (cid:1) 1 / 5 , where ˆ σ y r is the sample standard deviation of normalized labels in the training set.
Let κ base = max (cid:16) y r
[2] − y r
[1] , y r
[3] − y r
, y r
[ N r uy ] − y r
[ N r uy − 1] (cid:17) , where y r [ l ] is the l -th smallest normalized distinct real label and N r uy is the number of normalized distinct labels in the training set.
The κ is set as a multiple of κ base (i.e., κ = m κ κ base ) where the multiplier m κ stands for 50% of the minimum number of neighboring labels used for estimating p r ( x | y ) given a label y .
For example, m κ = 1 implies using 2 neighboring labels (one on the left while the other one on the right).
In our experiments, m κ is generally set as 1 or 2.
In some extreme case when many distinct labels have too few real samples, we may consider increasing m κ .
We also found ν = 1 /κ 2 works well in our experiments.
S. M ORE DETAILS OF T HEOREMS S.4 AND S. S.5. S OME NECESSARY DEFINITIONS AND NOTATIONS
• The hypothesis space D is a set of functions that can be represented by D (a neural network with determined architecture).
• In the HVDL case, denote by p y,κr ( x ) (cid:44) (cid:82) y
+ κ y − κ p r ( x | y (cid:48) )
p r ( y (cid:48) ) dy (cid:48)
the marginal distribution of real images with labels in [ y − κ, y + κ ] and similarly to p y,κg ( x ) of fake images.
• In the SVDL case, given y and weight functions (E.q. (8)), if the number of real and fake images are inﬁnite, the empirical density converges to p y,w r r ( x ) (cid:44) (cid:82) p r ( x | y (cid:48) )
w r ( y (cid:48) ,y ) p r ( y (cid:48) )
W r ( y ) dy (cid:48) and p y,w g g ( x ) (cid:44) (cid:82) p g ( x | y (cid:48) )
w g ( y (cid:48) ,y )
p g ( y ) W g ( y ) dy (cid:48) respectively, where W r ( y ) (cid:44) (cid:82) w r ( y (cid:48) , y )
p r ( y (cid:48) ) dy (cid:48) and W g ( y ) (cid:44) (cid:82) w g ( y (cid:48) , y )
p g ( y (cid:48) ) dy (cid:48) .
• Let p rw ( y (cid:48)
| y ) (cid:44) w r
( y (cid:48) ,y ) p r ( y (cid:48) )
W r ( y ) and p gw ( y (cid:48) | y )
(cid:44) w g
y (cid:48) ,y )
p g ( y (cid:48) )
W g ( y ) .
• The H¨older Class deﬁned in Deﬁnition 1 is a set of functions with bounded second derivatives, which controls the variation of the function when parameter changes.
(A4) implies the two probability density functions p r ( y ) and p g ( y ) are assumed in the H ¨ older Class.
• Given a G , the optimal discriminator which minimizes L is in the form of
[Equation]
However, D ∗ may not be covered by the hypothesis space D .
The (cid:101) D is the minimizer of L in the hypothesis space D .
Thus, L ( (cid:101) D ) − L ( D ∗ ) should be a non-negative constant.
In CcGAN, we minimize (cid:98) L HVDL ( D ) or (cid:98) L HVDL ( D ) with respect to D ∈ D , so we are more interested in the distance of (cid:98) D HVDL and (cid:98) D SVDL from D ∗ , i.e., L ( (cid:98) D HVDL ) − L
( D ∗ ) and L ( (cid:98) D SVDL ) − L ( D ∗ ) .
S
S.5.2. T ECHNICAL LEMMAS
Before we move to the proofs of Theorems 1 and 2, we provide several technical lemmas used in the later proof.
Recall notations and assumptions in Sections 3 and S.5.1, then we derive the following lemmas.
Lemma S.1.
Suppose that (A1)-(A2) and (A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ , sup D ∈D
[Equation]
for a given y .
Proof.
Triangle inequality yields
[Equation]
We then bound the two terms of the RHS separately as follows:
Real images with labels in [ y − κ, k + κ ] can be seen as independent samples from p y,κr ( x ) .
Then the ﬁrst term can be bounded by applying Hoeffding’s inequality as follows: ∀ δ ∈ ( with at least probability 1 − δ ,
[Equation]
For the second term, by the deﬁnition of p y,κr ( x ) and deﬁning p
κ
[Equation]
y
[Equation] [Equation]
we have
[Equation]
by the deﬁnition of total variation and the boundness
of
[Equation]
[Equation]
Then, focusing on | p y
[Equation]
κr ( x ) − p r ( x |
y ) | ,
[Equation]
Thus, Eq.
(S.18) is upper bounded as follows,
[Equation]
By combining Eq.
(S.17) and (S.19), we can get Eq.
(S.16), which ﬁnishes the proof.
Similarly, we apply identical proof strategy to the fake images x g and generator distribution p g ( x | y ) .
Lemma S.2.
Suppose that (A1), (A3) and (A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
for a given y .
Proof.
This proof is omitted because it is almost identical to the one for Lemma S.1.
The following two lemmas provide the bounds for SVDL.
Lemma S.3.
Suppose that (A1), (A2) and (A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
for a given y .
Proof.
For brevity, denote by f ( x, y )
= − log D ( x, y ) and F = − log D .
Then,
[Equation]
where the inequality is by triangular inequality.
We then derive bounds for both two terms of the last line.
For the ﬁrst term, we can further split it into two parts,
[Equation]
Focusing on the ﬁrst part of RHS of Eq.(S.23).
By (A1),
[Equation]
Note that ∀ y, y (cid:48) , w r ( y (cid:48) , y )
=
e − ν | y − y (cid:48) | 2 ≤ 1 and hence given y , w r ( y (cid:48) , y ) is a random variable bounded by 1 .
Apply Hoeffding’s inequality to the numerator of above, yielding that with probability at least 1 − δ (cid:48) ,
[Equation]
U
[Equation] [Equation]
[Equation] (cid:19) .
[Equation]
Then, consider the second part of RHS of Eq.(S.23).
Recall that p y,w r r
[Equation]
x )
[Equation]
w r ( y (cid:48)
[Equation]
dy [Equation]
Thus,
[Equation]
where p r ( x , y (cid:48) )
=
p r ( x | y (cid:48) )
p r ( y (cid:48) ) denotes the joint distribution of real image and its label.
Again, since w r ( y (cid:48) , y ) f
( x ri , y ) is uniformly bounded by U under (A1)
[Equation]
we can apply
Hoeffding’s inequality.
This implies that with probability at least 1 − δ (cid:48) , the above can be upper bounded by
[Equation]
Combining Eq.
(S.24) and (S.25) and by setting δ (cid:48)
=
δ 2 , we have with probability at least 1 − δ ,
[Equation]
Since this holds for ∀ f ∈ F , taking supremum over f , we have
[Equation]
[Table]
[Table]
on
[Table]
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
By (A.2) and y ∈ [0 , 1] , the above is upper bounded by g r ( x ) E y (cid:48) ∼ p
rw ( y (cid:48) | y )
[ | y − y (cid:48) | ] .
Thus,
[Equation]
Therefore, combining both Eq.(S.26) and (S.27), with probability at least 1 − δ ,
[Equation]
This ﬁnishes the proof
[Equation]
Lemma S
[Equation]
Suppose that (A1), (A3) and (A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
for a given y .
Proof.
This proof is omitted because it is almost identical to the one for Lemma S.21.
As introduced in Section 2, we use KDE for the marginal label distribution with Gaussian kernel.
The next theorem characterizes the difference between a p r ( y ) , p g ( y ) and their KDE using n i.i.d.
samples.
Theorem S.3.
Let ˆ p KDE r ( y ) and ˆ p KDE g ( y ) stand for the KDE of p r ( y ) and p g ( y ) respectively.
Under condition (A4), if the KDEs are based on n i.i.d.
samples from p r /p g and a bandwidth σ , for all δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
for some constants C KDE 1 ,δ , C KDE 2 ,δ depending on δ .
Proof.
By ((Wasserman); P.12), for any p ( t ) ∈ Σ( L ) (the H ¨ older Class, see Deﬁnition 1), with probability at least 1 − δ ,
[Equation]
for some constants C KDE δ and c , where C depends on δ and c = L (cid:82) K ( s ) | s | 2 ds .
Since in this work,
K is chosen as Gaussian kernel, c = L (cid:82) K ( s ) | s | 2 ds = L .
S.5.2. E RROR BOUNDS FOR HVDL AND SVDL
Based on the lemmas and theorems in Supp.
S.5.2.1, we derive the error bounds of HVDL and SVDL, which will be used in the proofs of Theorems 1 and 2.
Theorem S.4.
Assume that (A1)-(A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
∈D
[Equation]
for some constants C KDE 1 ,δ , C KDE 2 ,δ depending on δ .
Proof.
We ﬁrst decompose sup D ∈D (cid:12)(cid:12)(cid:12) (cid:98) L HVDL ( D ) − L ( D ) (cid:12)(cid:12)(cid:12) as follows
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] + sup D ∈D (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:90) (cid:34)
N gy
[Equation]
κ
N
[Equation]
i =
[Equation]
] (cid:35)
[Equation]
g (
[Equation]
(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) .
These four terms in the RHS can be bounded separately as follows
The ﬁrst term can be bounded by using Theorem S.3 and the boundness of D and y ∈ [0 , 1] .
For the ﬁrst term, ∀ δ 1 ∈ (0 , 1) , with at least probability 1 − δ 1 ,
[Equation]
for some constants C KDE 1 ,
[Equation]
1 depending on δ
The second term can be bounded by using Theorem S.3 and the boundness of D and y ∈ [0 , 1] .
For the ﬁrst term, ∀ δ 2 ∈ (0 , 1) , with at least probability 1 − δ 2 ,
[Equation]
for some constants C KDE 2 ,δ 2 depending on δ 2 . 3.
The third term can be bounded by using Lemma S.1 and S.2.
For the third term, ∀ δ 3 ∈ (0 , 1) , with at least probability 1 − δ 3 ,
[Equation] [Equation]
r ( y ) dy
Note that N ry,κ = (cid:80) N r i =
[Equation]
1 {| y − y ri |} , which is a random variable of y (cid:48)
i s .
The above can be expressed as
[Equation]
Similarly, for the fourth term, ∀ δ 4 ∈ (0 ,
[Equation]
, with at least probability [Equation] δ
[Equation]
With δ 1 =
δ 2 =
δ 3 =
δ 4 = δ [Equation] combining Eq.
(S.33) - (S.36) leads to the upper bound in Theorem
S.4.
Theorem S.5.
Assume that (A1)-(A4) hold, then ∀ δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
for some constant C KDE 1 ,δ , C KDE 2 ,δ depending on δ .
Proof.
Similar to the decomposition for
Theorem
S.4, we can decompose sup D ∈D (cid:12)(cid:12)(cid:12) (cid:98) L SVDL ( D ) − L ( D ) (cid:12)(cid:12)(cid:12) into four terms which can be bounded by using Theorem S.3, the boundness of D , Lemma S.3, and Lemma S.4.
The detail is omitted because it is almost identical to the one of Theorem S.4.
S.5.2. P ROOF OF T HEOREM
Based on Theorem S.4, we derive Theorem 1.
Proof.
We ﬁrst decompose L ( (cid:98) D HVDL )
− L ( D ∗ ) as follows
[Equation]
by
[Equation]
S
[Equation]
[Equation]
P ROOF OF
T HEOREM
Based on Theorem S.5, we derive Theorem 2.
Proof.
The detail is omitted because it is almost identical to the one of Theorem 1 in Supp.
S.5.2.3.
S.5.2.
I NTERPRETATION OF T HEOREMS 1 AND
Both theorems imply HVDL and SVDL perform well if the output of D is not too close to 0 or 1 (i.e., favor small U ).
The ﬁrst two terms in both upper bounds control the quality of KDE, which implies
KDE works better if we have larger N r and N g and a smaller σ .
The rest terms of the two bounds are different.
In the HVDL case, we favor smaller κ , M r , and M g .
However, we should avoid setting κ for a too small value because we prefer larger N ry,κ and N gy,κ .
In the SVDL case, we prefer small
M r and M g but large W r ( y ) and W g ( y ) .
Large W r ( y ) and W g ( y ) imply that the weight function decays slowly (i.e., small ν ; similar to large N ry,κ and N gy,κ in Eq.(S.32)).
However, we should avoid setting ν too small because a small ν leads to large E y (cid:48) ∼ ˆ p
rw ( y (cid:48) | y )
| y (cid:48) − y | and E y (cid:48) ∼ ˆ p gw ( y (cid:48) | y )
| y (cid:48) − y | (i.e., y (cid:48) ’s which are far away from y have large weights).
In our experiments, we use some rule-ofthumb formulae to select κ and ν .
As a future work, a reﬁned hyper-parameter selection method should be proposed.
S. M ORE DETAILS OF THE SIMULATION IN S ECTION 4. S.6. N ETWORK ARCHITECTURES
Please refer to Table S.6.1 and Table S.6.2 for the network architectures we adopted for cGAN and
CcGAN in our Simulation experiments.
S.6. T RAINING SETUPS
The cGAN and CcGAN are trained for 6000 iterations on the training set with the Adam (Kingma &
Ba, 2015) optimizer (with β 1 = 0 .
5 and β 2 = 0 .
999 ), a constant learning rate 5 × 10 − 5 and batch size 128.
The rule of thumb formulae in Section S.4 are used to select the hyper-parameters for
HVDL and SVDL, where we let m κ = 2 .
Thus, the three hyper-parameters in this experiments are set as follows:
σ = 0 .
074 , κ = 0 .
017 , ν = 3600 .
[Table]
concat(output of previous layer, y ) ∈ R 220 , where y ∈ R 120 is the label of x .
[Table]
[Table]
[Table] [Table]
, sin( y ) , cos( y ) )
∈
Rfc → 100 ; ReLU fc → 100 ; ReLU fc → 100 ; ReLU fc → 100 ; ReLU fc → 100 ; ReLU fc → 1 ; Sigmoid
S.6. T ESTING SETUPS
When evaluating the trained cGAN, if a test label y (cid:48) is unseen in the training set, we ﬁrst ﬁnd its closest, seen label y .
Then, we generate samples from the trained cGAN at y instead of at y (cid:48) .
On the contrary, generating samples from CcGAN at unseen labels is well-deﬁned.
S.6. E XTRA E XPERIMENTS S.6.4. V ARYING N UMBER OF G AUSSIANS FOR T RAINING D ATA G ENERATION
In this section, we study the inﬂuence of the number of Gaussians used for training data generation on the performance of cGAN and CcGAN.
We vary the number of Gaussians from 120 towith step size 10 but keep other settings in Section 4.1 unchanged and plot the line graphs of 2-Wasserstein Distance (log scale) versus the number of Gaussians in Fig.
S.6.1.
Reducing the number of Gaussians for training implies a larger gap between any two consecutive distinct angles in the training set.
As the number of Gaussians decreases, the continuous scenario gradually degenerates to the categorical scenario, therefore the assumption that a small perturbation to y results in a negligible change to p ( x | y ) is no longer satisﬁed.
Consequently, the 2-Wasserstein distances of the proposed two CcGAN methods gradually increase and eventually surpass the 2Wasserstein distance of cGAN when the number of Gaussians is small (e.g., less than 40).
Note that reducing the number of Gaussians in the training data generation will not improve the performance of cGAN in the testing because many angles seen in the testing stage (we evaluate each method on 360 angles) do not appear in the training set.
S. M ORE DETAILS OF THE EXPERIMENT ON RC-49 IN S ECTION 4. S.7. D ESCRIPTION OF RC-
To generate RC-49, ﬁrstly we randomly select 49 3-D chair object models from the “Chair” category provided by ShapeNet (Chang et al., 2015).
Then we use Blender v2.79 1 to render these 3-D models.
Speciﬁcally, during the rendering, we rotate each chair model along with the yaw axis for a degree between 0 .
1 ◦ and 89 .
9 ◦ (angle resolution as 0 .
1 ◦ ) where we use the scene image mode to compose our dataset.
The rendered images are converted from the RGBA to RGB color model.
In total, the
RC-49 dataset consists of 44051 images of image size 64 × 64 in the PNG format.
S.7. N ETWORK ARCHITECTURES
The RC-49 dataset is a more sophisticated dataset compared with the simulation, thus it requires networks with deeper layers.
We employ the SNGAN architecture (Miyato et al., 2018) in both cGAN and CcGAN consisting of residual blocks for the generator and the discriminator.
Moreover, for the generator in cGAN, the regression labels are input into the network by the label embedding (Akata et al., 2015) and the conditional batch normalization (De Vries et al., 2017).
For the discriminator in cGAN, the regression labels are fed into the network by the label embedding and the label projection (Miyato & Koyama, 2018).
For CcGAN, the regression labels are fed into networks by our proposed label input method in Section 2.
Please refer to our codes for more details about the network speciﬁcations of cGAN and CcGAN.
S.7. T RAINING SETUPS
The cGAN and CcGAN are trained for 30,000 iterations on the training set with the Adam (Kingma & Ba, 2015) optimizer (with β 1 = 0 .
5 and β 2 = 0 .
999 ), a constant learning rate 10 − 4 and batch size 256.
The rule of thumb formulae in Section S.4 are used to select the hyper-parameters for HVDL and SVDL, where we let m κ = 2 .
Thus, the three hyper-parameters in this experiments are set as follows:
σ = 0 .
0473 , κ = 0 .
004 , ν = 50625 .
S.7. T ESTING SETUPS
The RC-49 dataset consists of 899 distinct yaw angles and at each angle there are 49 images (corresponding to 49 types of chairs).
At the test stage, we ask the trained cGAN or CcGAN to generate 200 fake images at each of these 899 yaw angles.
Please note that, among these 899 yaw angles, only 450 of them are seen at the training stage so real images at the rest 449 angles are not used in the training.
We evaluate the quality of the fake images from three perspectives, i.e., visual quality, intra-label diversity, and label consistency.
One overall metric (Intra-FID) and three separate metrics (NIQE,
Diversity, and Label Score) are used.
Their details are shown in Supp.
S.7.5.
S.7. P ERFORMANCE MEASURES
Before we conduct the evaluation in terms of the four metrics, we ﬁrst train an autoencoder (AE) , a regression CNN and a classiﬁcation CNN on all real images in RC-49.
The bottleneck dimension of the AE is 512 and the AE is trained to reconstruct the real images in RC-49 with MSE as the loss function.
The regression CNN is trained to predict the yaw angle of a given image.
The classiﬁcation
CNN is trained to predict the chair type of a given image.
The autoencoder and both two CNNs are trained for 200 epochs with a batch size 256.
• Intra-FID (Miyato & Koyama, 2018):
We take Intra-FID as the overall score to evaluate the quality of fake images and we prefer the small Intra-FID score.
At each evaluation angle, we compute the FID (Heusel et al., 2017) between 49 real images and 200 fake images in terms of the bottleneck feature of the pre-trained AE.
The Intra-FID score is the average FID over all 899 evaluation angles.
Please note that we also try to use the classiﬁcation CNN to compute the Intra-FID but the Intra-FID scores vary in a very wide range and sometimes obviously contradict with the three separate metrics.
• NIQE (Mittal et al., 2012): NIQE is used to evaluate the visual quality of fake images with the real images as the reference and we prefer the small NIQE score.
We train one NIQE model with the 49 real images at each of the 899 angles so we have 899 NIQE models.
During evaluation, a NIQE score is computed for each evaluation angle based on the NIQE model at that angle.
Finally, we report the average and standard deviations of the 899 NIQE scores over the 899 yaw angels.
Note that the NIQE is implemented by the NIQE module in MATLAB .
• Diversity : Diversity is used to evaluate the intra-label diversity and the larger the better.
In RC-49, there are 49 chair types.
At each evaluation angle, we ask the pre-trained classiﬁcation to predict the chair types of the 200 fake images and an entropy is computed based on these predicted chair types.
The diversity reported in Table 2 is the average of the 899 entropies over all evaluation angles.
• Label Score : Label Score is used to evaluate the label consistency and the smaller the better.
We ask the pre-trained regression CNN to predict the yaw angles of all fake images and the predicted angles are then compared with the assigned angles.
The Label Score is deﬁned as the average absolute distance between the predicted angles and assigned angles over all fake images, which is equivalent to the Mean Absolute Error (MAE).
S
. E XTRA EXPERIMENTS S.7.6. M ORE LINE GRAPHS S.7.6. I NTERPOLATION
In Fig.
S.7.3, we present some interpolation results of the two CcGAN methods (i.e., HVDL and
SVDL).
For an input pair ( z, y ) , we ﬁx the noise z but perform label-wise interpolations, i.e., varying label y from 4.5 to 85.5.
Clearly, all generated images are visually realistic and we can see the chair distribution smoothly changes over continuous angles.
Please note that, Fig.
S.7.3 is meant to show the smooth change of the chair distribution instead of one single chair so the chair type may change over angles.
This conﬁrms CcGAN is capable of capturing the underlying conditional image distribution rather than simply memorizing training data.
80 Angle of yaw rotation 0.
F I D cGANCcGAN (HVDL) CcGAN (SVDL) (a) FID vs Angle    80 Angle of yaw rotation 1.
N I Q E cGANvanilla CcGAN (HVDL) vanilla CcGAN (SVDL) (b) NIQE vs Angle    80 Angle of yaw rotation 0.
D i v e r
s
i t y cGANCcGAN (HVDL) CcGAN (SVDL) (
[Figure]
Angle
[Figure]
S.7.6. D EGENERATED C C GAN
In this experiment we consider the extreme case of the proposed CcGAN (degenerated CcGAN), i e., σ → 0 and κ → 0 or ν → + ∞ .
Then we train the degenerated CcGAN with the same experimental setting as for CcGANs.
Some examples from degenerated CcGANs are shown in Fig.
S.7.4.
Since, at each angle, the degenerated CcGAN only uses the images at this angle, it leads to the mode collapse problem (e
Note that the degenerated CcGAN is still different from cGAN, since we still treat y as a continuous scalar instead of a class label here and we use the proposed label input method to incorporate y into the generator and the discriminator.
S.7.6. C GAN: DIFFERENT NUMBER OF CLASSES
In this experiment, we show that cGAN still fails even though we bin [0 . 1 , 89 . 9] into other number of classes.
We experimented with three different bin setting – grouping labels into 90 classes,classes and 210 classes, respectively.
Experimental results are shown in Fig.
S.7.5 and we observe that all three cGANs fail.
S.7.6. V ARYING S AMPLE S IZE FOR E ACH D ISTINCT A NGLE
To test cGAN and CcGAN under more challenging scenarios, we vary the sample size for each distinct angle in the training set from 45 to 5.
We visualize the line graphs of Intra-FID versus the sample size for each distinct training angle in Fig.
S.7.6.
From this ﬁgure, we can see the two CcGAN methods substantially outperform cGAN no matter what is the sample size for each distinct angle in the training set.
The overall trend in this ﬁgure also shows that smaller sample size reduces the performance of both cGAN and CcGAN.
S.7.6. V ARYING THE STRENGTH OF THE CORRELATION BETWEEN THE IMAGE AND ITS LABEL
To study how the strength of the correlation between the image x and its label y (i.e., the label power) inﬂuences the performance of cGAN and CcGAN, in this study, we randomly add Gaussian noises with a preset standard deviation to the raw regression labels in the training set.
The strength of the correlation is controlled by the standard deviation of the Gaussian noises which varies from
Angle 4. degenerated CcGAN
90 classes Angle 4.
150 classes 210 classes
0 (no Gaussian noise) to 25 (the unit is degree).
A large standard deviation corresponds to a weak correlation.
The training setup is consistent with the main study in Section 4.2 except that we replicate the training set ﬁve times and randomly add Gaussian noises to the raw regression labels in the replicated training set.
Therefore, each training sample has ﬁve noisy labels.
We plot the line graphs of Intra-FID versus the standard deviation of Gaussian noise in Fig.
S.7.7.
From Fig.
S.7.7, we can see that the performance of two CcGAN methods deteriorates as the standard deviation increases; however, the line graph of the performance of cGAN does not have a clear increasing or decreasing trend and the Intra-FID of cGAN always stays at a high level.
S. M ORE DETAILS OF THE EXPERIMENT ON THE UTKF ACE DATASET IN S ECTION 4. S.8. D ESCRIPTION OF THE UTKF ACE DATASET
The UTKFace dataset is an age regression dataset (Zhang et al., 2017), with human face images collected in the wild.
We use the preprocessed version (cropped and aligned), with ages spanning from 1 to 60.
After data cleaning (i.e., removing images of very low quality or with clearly wrong labels), the overall number of images is 14760.
Images are resized to 64 × 64 .
The histogram of
UTKFace dataset w.r.t.
ages 1-60 is shown in S.8.8.
From Fig.
S.8.8, we can see UTKFace dataset is very imbalanced so the samples from the minority age groups are unlikely to be chosen at each iteration during the GAN training.
Consequently, cGAN and CcGAN may not be well-trained at these minority age groups.
To increase the chance of drawing these minority samples during training, we randomly replicate samples in the minority age groups to ensure that the sample size of each age is more than 200.
S.8. N ETWORK ARCHITECTURES
The network architectures used in this experiment is similar to those in the RC-49 experiment.
Please refer to our codes for more details about the network speciﬁcations.
S.8. T RAINING SETUPS
The cGAN and CcGAN are trained for 40,000 iterations on the training set with the Adam (Kingma & Ba, 2015) optimizer (with β 1 = 0 .
5 and β 2 = 0 .
999 ), a constant learning rate 10 − 4 and batch size 512.
The rule of thumb formulae in Section S.4 are used to select the hyper-parameters for HVDL and SVDL, where we let m κ = 1 .
S. . P ERFORMANCE MEASURES
Similar to the RC-49 experiment, we evaluate the quality of fake images by Intra-FID, NIQE,
Diversity, and Label Score.
We also train an AE (bottleneck dimension is 512), a classiﬁcation CNN, and a regression CNN on all images.
Please note that, the UTKFace dataset consists of face images from 5 races based on which we train the classiﬁcation CNN.
The AE and both two CNNs are trained for 200 epochs with a batch size 256.
S
E XTRA EXPERIMENTS
S.8.5.
M ORE LINE
[Figure]
CcGANs outperform cGAN
S.8.5. I NTERPOLATION
To perform label interpolation experiments we keep the noise vector z ﬁxed and vary label from age 3 to age 57 for CcGANs with HVDL and SVDL losses.
The interpolation results are illustrated in
S.8.10.
As age y increases, we observe the synthetic face gradually becomes older in appearance.
This observation convincingly shows that both HVDL and SVDL based CcGANs do not simply memorize or overﬁt to the training set.
Indeed, our CcGANs demonstrate continuous control over synthetic images with respect to ages.
S.8.5. D EGENERATED C C GAN
We consider the extreme cases of the proposed CcGANs on the UTKFace dataset.
As shown in Fig.
S.8.11, the degenerated CcGANs fails to generate facial images at some ages (e.g., 51 and 57) because of too small sample sizes.
S.8.5. C GAN: DIFFERENT NUMBER OF CLASSES
In the last experiment, we bin samples into different number of classes based on ground-truth labels, in order to increase the number of training samples at each class.
Then we train cGAN using samples from the binned classes.
We experimented with two different bin setting, i.e., binning image samples into 60 classes and 40 classes, respectively.
The results are reported in Fig.
S.8.12.
The results demonstrate cGANs consistently fail to generate diverse synthetic images with labels aligned with their conditional information.
Moreover, the image quality is much worse than those from the proposed CcGANs.
In conclusion, compared with existing cGANs, our proposed CcGANs have substantially better performance in terms of the image quality and diversity.
[Figure]
[Figure] [Figure]
Age S.8.5. In this section, we train cGAN and CcGAN on images for odd ages only and test them on evennumbered ages.
Since the training set in this experiment is half of the one in the main study in Section 4.3, we reduce the number of iterations for the GAN training from 40,000 to 20,000 while other settings are unchanged.
The quantitative results for cGAN and two CcGAN methods are summarized in Table S.8.3.
From Table S.8.3, we can see two CcGAN methods are still much better than cGAN in terms of all metrics except Label Score, since CcGAN is designed to sacriﬁce some (not too much) label consistency for much better visual quality and diversity. [Table]
[Table]
T RAINING WITH SMALLER SAMPLE SIZES
The histogram in Fig.
S.8.8 shows that the UTKFace dataset is highly imbalanced.
To balance the training data and also test the performance of cGAN and CcGAN under smaller sample sizes, we vary the maximum sample size for each distinct age in the training from 200 to 50.
Note that, in the main study in Section 4.3, we do not restrict the maximum sample size.
Since we have a much smaller sample size, we reduce the number of iterations for the GAN training from 40,000 to 20, and slightly increase m κ in Supp.
S.4 from 1 to 2 (we therefore use a wider hard/soft vicinity).
We visualize the line graphs of Intra-FID versus the maximum sample size for each age of cGAN and
CcGAN in Fig.
S.8.13.
From the ﬁgure, we can clearly see that a smaller sample size worsens the performance of both cGAN and CcGAN.
Moreover, the Intra-FID scores of cGAN always stay at a high level and are much larger than those of two CcGAN methods.
S.8.5.
D IFF A UGMENT CANNOT SAVE C GAN IN THE CONTINUOUS SCENARIO
DiffAugment (Zhao et al., 2020) is a concurrent work which proposes differentiable transformations (i.e., translation, random brightness, contrast, saturation, and cutout) on both real and fake images during the GAN training.
This method shows promising results in improving the performance of unconditional GANs (e.g., StyleGAN2 (Karras et al., 2020)) and class-conditional GANs
(e.g., BigGAN (Brock et al., 2019)) when training samples are limited.
However, DiffAugment is fundamentally different from CcGAN since it is designed for the unconditional and class-conditional scenarios rather than the continuous scenario.
Even though incorporating DiffAugment into the cGAN training in the continuous scenario, the two problems (P1) and (P2) discussed in Section 1 are still unsolved.
First, (P1) is still unsolved because DiffAugment does not provide a solution better than binning the regression labels into a series of disjoint intervals to tackle the problem that some regression labels do not exist in the training set.
Second, since DiffAugment is designed for the unconditional and class-conditional scenarios where the number of distinct conditions is always ﬁnite and known,
DiffAugment doesn’t provide a solution to (P2) .
Besides these two unsolved problems, another concern of DiffAugment in the continuous scenario is that the ordinal information in the regression labels is not utilized while our CcGAN implicitly uses this ordinal information to construct the soft/hard vicinity.
To support our arguments, we incorporate DiffAugment into the cGAN training in the UTKFace experiment while other settings are kept constant.
When implementing DiffAugment, we use the ofﬁcial codes from the GitHub repository of DiffAugment 2 .
The strongest transformation combination (Color + Translation + Cutout) is used in the cGAN training.
Quantitative results from cGAN+DiffAugment are summarized in Table S.8.4 and some example images from cGAN+DiffAugment are shown in Fig.
S.8.14.
The quantitative results show that DiffAugment substantially improves the visual quality and diversity of the baseline cGAN; however, the performance of cGAN+DiffAugment is still much worse than that of the proposed two CcGAN methods.
The visual results also support the quantitative evaluations.
Therefore, cGAN+DiffAugment still does not solve the two fundamental problems in the continuous scenario, since it is not designed for this purpose.
[Table]
[Figure]
[Table]
[Figure]
[Table]
quality in the continuous scenario.
S. P OTENTIAL A PPLICATIONS AND I MPACTS OF C C GAN S
Generally, there are three label scenarios where we can apply CcGANs: Scenario I, mathematically continuous labels (e.g., angles); Scenario II, discrete but ordinal labels (e.g., ages); and Scenario III, discrete, categorical labels but sharing close relationships among different label categories (e.g., ﬁnegrained bird image generation).
CcGANs can have potential applications in all three scenarios.
For example, in Scenario I, CcGANs could have potential impacts on autonomous driving which involves predicting the steering angle ( a continuous scalar) to have better controllability over autonomous cars.
In Scenario II, the proposed methods are potentially meaningful in some medical applications.
E.g., in medical experiments, an important task is cell counting, where the cell counting regression needs to predict the number of cells (i.e., ordinal integers) from a microscopic image.
Even with limited microscopic cell images, the proposed CcGAN can generate visually synthetic and diverse microscopic images for the regression model training.
In this way, CcGAN may help save tedious efforts of medical reseachers in gathering microscopic images.
In Scenario III, as suggested by
AnonReviewer 5 (Q3), CcGAN could be used on some ﬁne-grained image classiﬁcation datasets, e.g., on the bird dataset where birds of different categories may share close similarities.
The generated bird images can be used to enhance the ﬁne-grained bird image classiﬁers, and potentially help us better recognize birds and protect the environment.
More generally, CcGANs can be potentially used for image generation in regression datasets (associated with scalar labels y).
In summary, CcGANs can cover a wide range of tasks and applications which could potentially beneﬁt the society.