Mining Multi-Label Samples from Single Positive Labels
Abstract
Conditional generative adversarial networks (cGANs) have shown superior results in class-conditional generation tasks.
To simultaneously control multiple conditions, cGANs require multi-label training datasets, where multiple labels can be assigned to each data instance.
Nevertheless, the tremendous annotation cost limits the accessibility of multi-label datasets in real-world scenarios.
Therefore, in this study we explore the practical setting called the single positive setting , where each data instance is annotated by only one positive label with no explicit negative labels.
To generate multi-label data in the single positive setting, we propose a novel sampling approach called single-to-multi-label (S2M) sampling, based on the
Markov chain Monte Carlo method.
As a widely applicable “add-on” method, our proposed S2M sampling method enables existing unconditional and conditional
GANs to draw high-quality multi-label data with a minimal annotation cost.
Extensive experiments on real image datasets verify the effectiveness and correctness of our method, even when compared to a model trained with fully annotated datasets.
Introduction
Since being proposed by Goodfellow et al.
[1], generative adversarial networks (GANs) have gained much attention due to their realistic output in a wide range of applications, e g. , image synthesis [2, 3, 4], image translation [5, 6, 7, 8] and data augmentation [9, 10].
As an advanced task, generating images from a given condition has been achieved by conditional GANs (cGANs) and their variants [11, 12].
To reﬂect the nature of real data where each data instance can belong to multiple classes, multi-label datasets have been introduced in the applications of cGANs [8, 13].
In a multi-label dataset, each data instance can be speciﬁed with multiple attributes.
For example, in a multi-label facial image dataset, each image is labeled for entire classes such as Black-hair , Smile , and Male .
The label for each class is given as a binary value and indicates the presence or absence of the corresponding attribute.
Despite its usefulness, as claimed in earlier studies [14, 15, 16], access to multi-label datasets is severely limited in practice due to the difﬁculty of annotating all attributes.
Under these circumstances, a weakly annotated dataset is used as an alternative for cGANs to reduce the annotation cost.
In this paper, we introduce the single positive setting [16], originally proposed for classiﬁcation tasks, to conditional generation.
Each data instance has a label indicating only the presence of one attribute ( i.e. , a single positive label) in this setting, and the presence of the rest of the attributes remains unknown.
For instance, in a facial image dataset, all attributes are fully speciﬁed in a multi-label setting ( e.g. , Smiling black-haired man ) whereas only one attribute is speciﬁed in the single positive setting ( e.g. , Black-hair ).
The single positive setting allows us not only to reduce the annotation cost, but also to model the intrinsic relationships among classes.
[Figure] To generate multi-label data using only single positive labels, we consider two types of combinatorial classes from two classes, A and B , each corresponding to a single positive label: (1) overlapping class , where the data instances belong to both classes ( A \ B ), and (2) non-overlapping class , where the data instances belong to only one of the classes ( A \ B or B \ A ).
Figure 1 shows examples of overlapping class and non-overlapping classes when two types of single positive labels are given.
By extending this concept, we can consider combinatorial classes, where data instances belong to certain classes but not to the rest.
We denote these classes as joint classes .
By accessing a joint class, we can generate multi-label samples using only single positive labels.
Ideally, we can represent all possible label combinations with at least one positive label.
Several attempts have been made to consider such a joint class in generation tasks.
Specially designed generative models such as GenPU [17] and RumiGAN [18] were proposed to generate samples from one class while excluding another class.
However, these studies deal with only two classes without considering the overlapping class.
Recently, Kaneko et al.
[19] proposed CP-GAN to capture between-class relationships in conditional generation.
To generate images belonging to n classes, they equally assign 1 /n as the class posterior for each selected class.
This indicates that CP-GAN generates images that have an equal probability of being classiﬁed as each class.
However, because an image in the real world that belongs to multiple classes does not have equal class posteriors, sample diversity is lost.
Consequently, we focus on a sampling approach to precisely draw samples from complex distributions that are difﬁcult to directly sample.
In recent GANs studies, sampling approaches [20, 21] employed the rejection sampling or Markov chain Monte Carlo method to obtain realistic data.
These approaches adopt the sampling process as the post-processing method of GANs and ﬁlter generated images using the pretrained classiﬁers.
In line with these studies, we propose a novel sampling framework called single-to-multi-label (S2M) sampling to correctly generate data from both overlapping and non-overlapping classes.
We newly propose a tractable formulation to estimate the conditional density of joint classes.
Concretely, we employ several classiﬁcation networks to estimate the target conditional density and apply the Markov chain Monte Carlo method to pretrained unconditional and conditional GANs.
In Figure 2, we depict the conceptual difference against the existing approaches ( 1 st row) and provide the empirical results on a 1D Gaussian example ( 2 nd row).
As S2M sampling performs at the inference time of generation, it fully preserves the image quality and eliminates the need for changing the objective functions and architectures.
We validate the effectiveness of our method compared to the existing models on diverse datasets such as MNIST, CIFAR-10 and CelebA. To the best of our knowledge, our approach is the ﬁrst sampling framework that generates multi-label data from single positive labels.
Our contributions can be summarized as follows:
• We introduce the single positive setting in the conditional generation task and provide the theoretical framework for estimating conditional densities of joint classes.
• We propose a novel sampling framework based on the Markov chain Monte Carlo method for generating multi-label data from single positive labels.
• Through extensive experiments, we show that our proposed S2M sampling method correctly draws multi-label data while fully preserving the quality of generated images. [Figure]
Related Work
Conditional GANs.
The aim of conditional GANs (cGANs)
[11] is to model complex distributions and control data generation by reﬂecting the label input.
Various studies of cGANs have made signiﬁcant advances in class-conditional image generation by introducing an auxiliary classiﬁer [12, 22], modifying the architecture [23, 2], and applying metric learning [24].
In a weakly-supervised setting, GenPU [17] and RumiLSGAN [18] specify only two classes and draw samples that belong to one class but not the other.
CP-GAN [19] learns to draw samples conditioned on the probability output of the classiﬁer.
Given that this model tends to draw samples on a limited region of the data space, it is challenging to ensure a variety of samples as shown in Figure 2.
In contrast, we propose a sampling method that draws multi-label samples without sacriﬁcing diversity.
Sampling in GANs.
Sampling methods are used to improve the sample quality in GANs.
Discriminator rejection sampling [20] uses the scheme of rejection sampling and takes samples close to real data by estimating the density ratio with the discriminator.
In addition, Metropolis-Hastings GAN [21] adopts the Markov chain Monte Carlo (MCMC) method and calibrates the discriminator to improve the sample quality in a high-dimensional data space.
Discriminator driven latent sampling [25] uses the MCMC method in the latent space of GANs to draw realistic samples efﬁciently.
GOLD estimator [26] uses a sampling algorithm to improve the quality of images for class-conditional generation.
While previous studies focus on improving the sample quality, our S2M sampling method aims to draw multi-label samples while also improving sample quality.
Methods 3.1 Problem Setting
Let x 2 X be a data point as a random variable and let y 1: n 2 { 0 , 1 } n denote its corresponding multi-labels as binary random variables.
Here, for every k , y k
= 1 indicates that x is contained in the k -th class while
y k
= 0 indicates that x is not.
We consider two index sets, an intersection index set
I and a difference index set J , so that the pair ( I, J ) can be used as an index to indicate the class, where data points contained in all classes indicated by I but excluded from all classes indicated by J .
Let I be a collection of all possible pairs of I and J , deﬁned as
[Equation]
where N = { 1 , 2 , ..., n } is a ﬁnite index set of all classes and P ( N ) is the power set of N .
That is, the intersection index set indicates at least one class and is distinct from the difference index set.
Especially for I [ J = N , the class indicated by ( I, J ) is called the joint class .
Let p ( x, y 1 , y 2 , ..., y n ) be the joint probability density function, and let p data ( x, c ) be the joint density of an observed data point x 2 X and a class label c
2 N such that p data ( x | c ) =
p ( x | y c = 1) .
Given the class priors p data ( c ) , ⇡ c := p ( y c = 1) and samples drawn from the class-conditional density p data ( x | c ) for each c = 1 , 2 , ..., n , our goal is to draw samples from the conditional density [Equation]
[Equation] [Equation] [Equation] for ( I, J ) 2 I and ⇡ ( I,J ) :
= p ( 8 i 2 I, 8 j 2 J, y i = 1 , y j = 0) > 0 . In this work, we propose adding a mild constraint which will allow our sampling algorithm to derive the target density. (b) Outputs S2M sampling ① 1
I = {1} J = {2, 3} 1
I = {1, 2} J = {3} 1
I = {1, 2, 3} J = ∅ ③ (a) Single positive setting ③ ② ① A
B
C ?
A
B
C ? ??
(1) (2) (3) ②
Assumption 1.
For every i, j 2 N such that i {+
+}
= j , if p ( y i = 1 , y j = 0) >
0 and p ( y j = 1 , y i = 0) > 0 , then supp p ( x | y i = 1 , y j = 0) \ supp p ( x | y j = 1 , y i = 0)
= ; .
Assumption 1 states that no data points are likely to be assigned two mutually exclusive labels, which can be naturally assumed in many practical situations.
Figure 3 illustrates our problem setting.
3.2 Mining Multi-Label Data with S2M Sampling
Naturally, a question may arise as to whether the supervision given to us is sufﬁcient to obtain multilabel samples.
To gain insight into this, we derive a useful theorem which provides an alternative formulation for the target density (2).
Theorem 1.
Let { f ( I,J ) :
X ! ( I,J ) {+
 I+} be an indexed family of non-negative measurable functions on X , and let f k :=
f ( { k } , ; ) .
Then, the following conditions hold:
(a) 8 ( I, J ) 2 I , f
( I,J ) = [Equation]
S [Equation] I ✓ S,J ✓ N \ S
[Equation] S,N \ S ) (b) 8 i, j 2 N s.t.
i
[Equation] = j, supp f ( { i } , { j } ) \ supp f ( { [Equation]
[Equation] i } ) = [Equation]
if and only if, for every ( I, J ) 2 I ,
[Equation]
where ( · ) + represents the positive part.
Proof.
Please refer to Appendix A.
Let f ( I,J ) ( x ) =
p ( x, 8 i 2 I, 8 j 2 J, y i = 1 , y j = 0) for every ( I, J ) 2 I .
Then, both ( a ) and ( b ) in
Theorem 1 hold by the Assumption 1.
According to the Theorem 1, if ⇡ ( I,J ) > 0 ,
[Equation]
The alternative formula ( 4 ) shows that the density of the joint class can be derived from the classconditional densities of the single positive labels.
Despite their clear relationship, training a generator that can model the density p ( I,J ) is a non-trivial problem since the formula consists of several implicitly deﬁned conditional densities, class priors, and variable sets.
To address this issue, we propose the application of sampling approaches upon existing GANs.
Interestingly, our sampling framework called S2M sampling allows not only for samples to be drawn from the target density, but also the modiﬁcation of I , J , and class priors at inference time.
The rest of this section introduces the main approaches of our S2M sampling method.
Density Ratio Estimation.
We utilize several classiﬁcation networks to compute implicitly deﬁned density ratios.
For simplicity, we denote G as a pretrained generator for both unconditional and conditional GANs.
G produces data x by taking a latent z and a class label c for class-conditional generation and only z for unconditional generation.
We consider three classiﬁers D v , D r , and D f which are obtained by minimizing L v , L r , and L f , respectively, i.e. ,
[Equation]
where p G is the distribution of the generated samples by G .
The optimal classiﬁers trained by these losses D ⇤ v , D ⇤ r , and D ⇤ f satisfy the following equations: D ⇤ v ( x ) =
p data ( x ) / ( p data ( x ) + p G ( x )) , D ⇤ r ( c | x ) = p ( x | y c = 1) p data ( c ) /p data ( x ) , D ⇤ f
( c | x ) =
p G ( x | c ) p G ( c ) /p
G ( x ) .
From D ⇤ v ,
D ⇤ r , and D ⇤ f , we can access the density ratios p data ( x ) /p
G ( x ) , p ( x | y c = 1) /p data ( x ) , and p G ( x | c ) /p
G ( x ) which will be used to compute the acceptance probability of the MCMC method.
S2M Sampling for Unconditional GANs.
We apply Metropolis-Hastings (MH) independence sampling [27, 21] to draw samples from the complex target distribution p t .
The MH algorithm uses a
Markov process where each transition from a current state x k to the next state x k +1 is made by an accept-reject step.
At each step of MH independent sampling, a new proposal x 0 is sampled from a proposal distribution q ( x 0 | x ) =
q ( x 0 ) and is then accepted with probability ↵ ( x 0 , x ) which is given by ↵ ( x 0 , x ) = min ⇣ [Equation]
[Equation] x [Equation] ⌘ .
We set x k +
= x 0 if x 0 is accepted, and x k {++
+} = x
k otherwise.
Under mild conditions, the chain x 1: K converges to the unique stationary distribution p t as K {+! 
+}
Let G be a generator where the support of p G contains that of p ( I,J ) .
To draw samples from p ( I,J ) , we let p t =
p
I,J )
and take multiple samples from G as independent proposals, i.e. q ( x 0 | x )
= p G ( x 0 ) .
Then, the desired acceptance probability ↵ ( x 0 , x ) can be calculated using D ⇤ v and D ⇤ r : r ( I,J ) ( x ) :=
✓ min ⇢
[Equation]
↵ ( x 0 , x ) = min
[Equation]
Different from Turner et al.
[21], the term r ( I,J ) ( x 0 ) /r
( I,J ) ( x ) is added to the acceptance probability formula to draw multi-label samples.
To obtain uncorrelated samples, a sample is taken after a ﬁxed number of iterations for each chain.
The sampling approach allows one to control the parameters
I , J , and  k := ⇡ k /p data ( k ) without any additional training of the model.
A summary of our S2M sampling algorithm is provided in Appendix B.1.
S2M Sampling for Conditional GANs.
cGANs can provide a proposal distribution close to the target distribution p ( I,J ) , which greatly increases the sample efﬁciency of the MCMC method.
Let c be a class label such that the support of class-conditional density p G ( ·| c ) contains that of p ( I,J ) .
At each step of the MH algorithm, the proposal x 0 ⇠ q ( x 0 | x ) = p G ( x 0 | c ) is accepted with a probability ↵ c ( x 0 , x ) .
The desired ↵ c ( x 0 , x ) can be calculated as ↵ c ( x 0 , x ) =
[Equation]
That is, the sampling method can be adopted to cGANs by additionally computing D ⇤ f
( c | x ) /D ⇤ f
( c | x 0 ) .
Latent Adaptation in S2M Sampling.
While our S2M sampling method allows us to draw multilabel samples, the algorithm rejects certain samples through the sampling procedure if the target distribution p ( I,J ) is signiﬁcantly different from the generator distribution.
Speciﬁcally, an independent
MH algorithm takes time inversely proportional to the probability of a generated sample belonging to the target class.
To alleviate this issue, we propose a technique to improve the sample efﬁciency from past sampling attempts.
Initially, a certain number of pilot target class samples x 1: m are drawn using
S2M sampling, and then the corresponding target latent samples t 1: m ; x k = G ( t k ) are obtained.
Since the latent samples are nearly restricted to the latent prior of GANs ( e.g. , Gaussian distribution), the distribution ˜ p
z of the target latent can be roughly estimated by ﬁtting a simple probabilistic model using t 1: m .
Let ˜ p G be the newly obtained generator distribution induced by G , i.e. , G ( z ) =
x ⇠ ˜ p G ( x ) for z ⇠ ˜ p
z ( z ) .
The sample efﬁciency can be further improved by using ˜ p G as the proposal distribution in the MH algorithm since ˜ p G is much close to the target distribution.
To run the MH algorithm without retraining the classiﬁers, we approximate p G ( G ( z )) /
˜ p G ( G ( z ))
⇡ C · p z ( z ) /
˜ p
z ( z ) 2 for some constant C .
Then, at each step of the MH algorithm, the proposal x 0 ⇠ q ( x 0 | x ) =
˜ p G ( x 0 ) is accepted
[Figure]
Even 3/
[Figure]
[Figure]
with a probability ˜ ↵ ( x 0 , x ) which can be calculated as
[Equation]
, x
[Equation]
min
[Equation]
[Figure]
[Equation] [Figure] [Equation] [Figure]
[Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
(8) where x 0 = G ( z 0 ) and x = G ( z ) .
Here, we can explicitly compute the density ratio between p z and ˜ p
z by letting ˜ p
z be a known probability density function such as a Gaussian mixture.
In practice, if the target class samples rarely appear in the generated samples, one can consider applying latent adaptation repeatedly.
For instance, to draw Black-haired man samples, we can ﬁrst search the latent space of Man by using latent adaptation, and then perform it in that space again to search the space of
Black-haired man .
In terms of time complexity, this is performed more efﬁciently than searching the space of Black-haired man at once.
A detailed description of latent adaptation and a discussion about the time complexity are provided in Appendix B.2 and Appendix C, respectively.
Practical Considerations.
We employ three classiﬁers D v , D r , and D f , to compute the acceptance probability used in the MCMC method.
For better training efﬁciency of the classiﬁers, we use shared layers, except for the last linear layer.
Since the classiﬁers are not optimal in practice, we scale the temperature of classiﬁers [30] and adjust  k to calibrate the sampling algorithm.
Detailed settings and the ablation study are provided in Appendix D and Appendix E.4, respectively.
Experiments
In this section, we validate that our S2M sampling method properly draws samples within the joint classes.
Speciﬁcally, we mainly consider two cases in the single positive setting: (i) one class is entirely contained in another class, (ii) multiple classes are partially overlapping with each other.
In the former setting, we verify that S2M sampling can completely ﬁlter out samples of a smaller class, and compare our method to the existing models including GenPU [17] and RumiLSGAN [18] on
MNIST and Fashion-MNIST (FMNIST).
In the latter setting, all possible joint classes are assessed with our S2M sampling method.
Subsequently, we evaluate how accurately S2M sampling draws samples in these classes compared to CP-GAN [19] and the models trained with a fully annotated multi-label dataset.
To verify whether a method can accurately generate samples for each joint class, we evaluate accuracy which indicates how many generated images are assigned to the target joint classes by a classiﬁer trained with fully annotated data ( i.e. , multi-label data).
Since accuracy itself cannot fully evaluate the distance between the distribution of generated samples and that of real samples, we additionally introduce various metrics to evaluate ﬁdelity ( i.e. , how realistic the generated samples are) and diversity of generated images: (i) Inception Score (IS) [31], (ii) Fréchet Inception Distance (FID) [32], (iii) improved precision and recall [33], and (iv) density and coverage [34].
FID and IS are metrics to evaluate the ﬁdelity and the diversity of generated samples using the features of the pretrained Inception-V3 network [35].
A lower FID and a higher IS indicate higher ﬁdelity and diversity of generated samples.
Precision and density measure the ratio of generated samples that falls within the manifold of real samples, so that they evaluate the ﬁdelity of generated samples.
Contraily, recall and coverage measure the ratio of real samples that falls within the manifold of the generated samples, and thus evaluating their diversity.
That is, a higher value of these metrics indicates that the generated samples have a distribution similar to the real one.
Since several metrics for ﬁdelity and diversity are not applicable for non-ImageNet-like images ( e.g. ,
MNIST and FMNIST), we instead use the FID score computed from activations of LeNet5 [36] and denote this as FID † .
The test set is used as a reference dataset for evaluating FID.
All quantitative results are averaged over three independent trials, and the standard deviation is denoted by subscripts.
[Figure]
The qualitative results are randomly sampled in all experiments.
Detailed experimental settings, such as architectures and hyperparameters, are provided in Appendix D.
4.1 Sampling for Classes with Inclusion
Experimental Settings.
We consider a special case of our problem setting, where one class is entirely contained in another class ( B ⇢ A ) as shown in Figure 4 (a).
This setting is similar to the positive-unlabeled setting [37, 38] where the positive data is included in the unlabeled data.
Under this constraint, GenPU [17], RumiLSGAN [18], and CP-GAN [19] can be used to generate samples from the non-overlapping class ( A \ B ).
Along with these models, we attempt to draw samples from the nonoverlapping class by adopting S2M sampling to unconditional WGAN-GP [39].
These experiments were conducted in three settings (See Figure 4): (i) MNIST-3/5 ( A ={3, 5}, B ={3}), (ii) MNIST-Even ( A ={0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, B ={1, 3, 5, 7, 9}), and (iii) FMNIST-Even ( A = { 0 T-shirt/Top , 1 Trouser , 2 Pullover , 3 Dress , 4 Coat , 5 Sandal , 6 Shirt , 7 Sneaker , 8 Bag , 9 Ankle boot } , B = { 1 , 3 , 5 , 7 , 9 } ).
For S2M sampling, samples are obtained at 100 MCMC iterations.
Quantitative Results.
Table 1 shows the results of our S2M sampling method and the baselines for the non-overlapping classes.
The performance of GenPU is reported only for MNIST-3/5 due to its mode collapse issue [40, 41].
S2M sampling adopted to unconditional GAN shows promising results in terms of both accuracy and FID † .
In fact, our results are superior to the existing methods specially designed for generating non-overlapping data such as GenPU and RumiLSGAN.
Notably,
S2M sampling improves accuracy by 8 . 5% and 6 . 8% while decreasing FID † compared to the secondbest models on MNIST-Even and FMNIST-Even, respectively.
This indicates that S2M sampling accurately samples the images of the non-overlapping classes without being biased to a speciﬁc mode.
[Table]
Qualitative Results.
Figure 5 shows the qualitative results of RumiLSGAN, CPGAN and our method adopted to WGAN-GP.
As indicated by the red boxes, RumiLSGAN and CP-GAN fail to completely eliminate the samples of the smaller class ( e.g. , odd digits for MNIST-Even or Sneaker for FMNIST-Even).
Conversely, S2M sampling correctly draws samples for the target classes.
4.2 Sampling for Multiple Classes
Experimental Settings.
Here, we consider a general case of single positive setting where multiple classes can be overlapping as shown in Figure 4 (b-d).
Given n classes, at most 2 n  1 joint classes can be obtained.
In this setting, we attempt to obtain samples of these joint classes using only single positive labels.
We consider cGANs with a projection discriminator (cGAN-PD) [23], AC-GAN [12], and CP-GAN [19] as the baseline generative models.
Since traditional conditional GANs such as cGAN-PD and AC-GAN are not originally designed for generating joint classes, we naively introduce these models in our setting with slight modiﬁcations.
Concretely, as a method to generate images belonging to n classes, we provide 1 /n as the conditional value for each class, following Kaneko et al.
In our experiments, we expect these models to have a lower bound performance of our results and indicate their results with asterisks ( ⇤ ).
In contrast, conditional GANs trained on a fully annotated dataset where all joint classes are speciﬁed can be considered as strong baselines in our settings.
cGAN-PD and AC-GAN are trained in this setting,
Airplane
Cat Deer
Dog
Frog
[Figure]
Bird
[Figure]
Automobile
[Figure]
C
I F
AR - 7 t o
C e l e b
A ( B
M S )
[Table]
and are denoted as oracle models .
In the experiments, S2M sampling is adopted on unconditional
GAN and cGAN-PD.
When adopting S2M sampling to cGAN-PD, a class containing the target joint class is used as the conditional value of cGAN-PD.
BigGAN
[2] is used as the backbone architecture for all generative models, and only single positive labels are used except for the oracle models.
To demonstrate the effectiveness of our S2M sampling method, we use three real-world image datasets: (i)
CIFAR-7to3 that has three classes ( A , B , C ), each of which contains four original classes of CIFAR-10, i.e. A ={ Airplance , Automobile , Dog , Frog },
B ={ Automobile , Bird , Cat , Frog },
C =
{ Cat , Deer , Dog , Frog }, (ii) CelebA-BMS that consists of human face images, each of which is labeled for one of three attributes ( Black-hair , Male , and Smiling ), and (iii) CelebA-ABMNSW that consists of human face images, each of which is labeled for one of six attributes ( bAngs , Black-hair ,
Male , No-beard , Smiling , and Wearing-lipstick ).
As depicted in Figure 4 (b-d), these datasets can have up to 2 3  1 or 2 6  1 joint classes.
For S2M sampling, samples are obtained at 200 MCMC iterations for CIFAR-7to3 and CelebA-BMS, and obtained at 1000 MCMC iterations for CelebA-ABMNSW.
Quantitative Results.
Table 2 summarizes the quantitative results of joint class generation on
CIFAR-7to3, CelebA-BMS, and CelebA-ABMNSW.
As expected, we can observe that cGAN-PD ⇤ and AC-GAN ⇤ cannot accurately generate samples from joint classes, which is consistent with the ﬁndings of Kaneko et al.
Although CP-GAN shows reasonable accuracy for all experiments, it suffers from generating diverse samples.
As discussed in Section 1, the high FID and low coverage support that CP-GAN tends to generate samples in a narrow scope of real data space.
In all experiments, our S2M sampling method adopted to cGAN-PD consistently outperformes the non-oracle baselines in terms of accuracy and diversity, e.g. , in CIFAR-7to3, our method improves accuracy and FID of CP-GAN by 38% and 13.74, respectively.
Such improvements verify that our
S2M sampling method generates correct samples for joint classes, and also preserves diversity within them.
Despite a large sample space of GAN, S2M sampling adopted to unconditional GAN also surpasses non-oracle baselines in terms of both ﬁdelity and diversity while achieving reasonable accuracy.
Surprisingly, despite the fact that the oracle models are trained with fully annotated data,
S2M sampling shows a comparable performance against those models using only single positive labels.
Moreover, as an post-processing method, S2M sampling fully preserves the image quality of
GANs and shows the highest ﬁdelity in terms of precision and density.
Qualitative Results.
Figure 6 depicts the results of S2M sampling for every possible joint classes on
CIFAR-7to3 and CelebA-BMS.
The results show that S2M sampling can draw diverse high-quality images from the distributions of all joint classes.
Furthermore, S2M sampling is model-agnostic and does not require the retraining of GANs; thus, it can be easily integrated with various GANs including state-of-the-art GANs.
We depict more qualitative results of adopting our S2M sampling method to unconditioanl GAN and pretrained StyleGANv2 in Appendix E.5 and
Appendix F.
Analysis on Latent Adaptation.
To demonstrate the effect of latent adaptation, we tested our latent adaptation technique in a overlapping class (B+M+S) and a non-overlapping class (B-M-S) on the
CelebA-BMS dataset.
We ﬁrst perform S2M sampling on unconditional BigGAN in these classes and draw 10 k pilot latent samples obtained at 100 MCMC iterations.
Using these latent samples, a Gaussian mixture of eight components is ﬁt as a new latent of GAN (See Appendix B.2).
Due to the discrepancy between the generator distribution and the target distribution, S2M sampling originally shows a low acceptance probability, as shown in Figure 7.
In this case, applying latent adaptation increases the acceptance probability and signiﬁcantly improves both accuracy and FID at the early stage of iterations.
This indicates that the generator distribution becomes closer to the target distribution by adopting latent adaptation, resulting in an accelerated convergence speed.
Limitations
[Figure]
In general, as the number of attributes increases, the number of each joint class samples in the training dataset are drastically reduced.
This may cause two primary problems: (i) training GANs to be able to generate each joint class samples becomes challenging, and S2M sampling cannot draw samples in which GANs do not generate, and (ii) the computation time of the sampling procedure increases.
In this case, it may be necessary to apply the latent adaptation technique to shorten MCMC chains.
In principle, we can reduce the sampling time complexity to be proportional to the number of attributes (See Appendix C).
As shown in Figure 8, the number of training samples in joint classes rapidly decreases.
Because of this, six attributes at most are used on the CelebA dataset for experiments.
Almost half of the joint classes have fewer than 100 training samples when using six attributes ( i.e. , CelebA-ABMNSW).
Nevertheless, our method handles a larger number of attributes compared to the existing methods.
We believe that our work can be further extended to larger attributes with the sufﬁcient amount of data.
Conclusion
In this study, we investigate the single positive setting in the class-conditional generation task and propose a novel sampling framework called S2M sampling.
We demonstrate that our proposed S2M sampling method can accurately draw high quality multi-label samples in the single positive setting.
To improve sampling efﬁciency, we also introduce the latent adaptation technique.
We believe that our theoretical framework provides a better understanding for weakly-supervised multi-label data generation.
Our S2M sampling method has the potential to be applied to a wide variety of generative models as well, including clustering based GANs [42] and diffusion models [43, 44], among others.
Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korean government (MSIT)
(No. 2019-0-00075, Artiﬁcial
Intelligence Graduate School Program (KAIST) and No. 2021-0-01778, Development of human image synthesis and discrimination technology below the perceptual threshold).
This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. NRF-2022R1A2B5B02001913).