Mining Multi-Label Samples from Single Positive Labels
Anonymous Author(s)
AffiliationAddress email
Abstract
Conditional generative adversarial networks (cGANs) have shown superior resultsin class-conditional generation tasks.
In order to simultaneously control multipleconditions, cGANs require multi-label training datasets, where multiple labelscan be assigned to each data instance.
Nevertheless, the tremendous annotationcost limits the accessibility of multi-label datasets in the real-world scenarios.
Hence, we explore the practical setting called single positive setting , where eachdata instance is annotated by only one positive label with no explicit negativelabels.
To generate multi-label data in the single positive setting, we propose anovel sampling approach called single-to-multi-label (S2M) sampling, based onthe Markov chain Monte Carlo method.
As a widely applicable “add-on” method,our proposed S2M sampling enables existing unconditional and conditional GANsto draw high-quality multi-label data with a minimal annotation cost.
Extensiveexperiments on real image datasets verify the effectiveness and correctness of ourmethod, even when compared to a model trained with fully annotated datasets.
Introduction
Since proposed by Goodfellow et al.
[1], generative adversarial networks (GANs) gained muchattention due to its realistic output in a wide range of applications, e g. , image synthesis [2, 3, 4],image translation [5, 6, 7, 8] and data augmentation [9, 10].
As an advanced task, generating imagesfrom a given condition has been achieved by conditional GANs (cGANs) and its variants [11, 12].
Toreflect the nature of real data where each data instance can belong to multiple classes, multi-labeldatasets have been introduced in the applications of cGANs [8, 13].
In a multi-label dataset, each datainstance can be specified with multiple attributes.
For example, in a multi-label facial image dataset,each image is labeled for entire classes such as Black-hair , Smile and Male .
The label for each classis given as a binary value and indicates the presence or absence of the corresponding attribute.
Despiteits usefulness, as claimed in earlier studies [14, 15, 16], access to multi-label datasets is severelylimited in practice due to the difficulty of annotating all attributes.
Under these circumstances, weaklyannotated dataset is used as an alternative for cGANs to reduce the annotation cost.
In this paper, we introduce the single positive setting [16], originally proposed for classification tasks,to conditional generation.
Each data instance has a label indicating only the presence of one attribute( i.e. , a single positive label) in this setting, and the presence of the rest of the attributes remainsunknown.
For instance, in a facial image dataset, all attributes are fully specified in a multi-labelsetting ( e.g. , Smiling black-haired man ) whereas only one attribute is specified in the single positivesetting ( e.g. , Black-hair ).
The single positive setting allows us not only to reduce the annotation cost,but also to model the intrinsic relationships among classes.
To generate multi-label data using only single positive labels, we consider two types of combinatorialclasses from two classes A and B that each corresponds to a single positive label: (1) overlapping
[Figure]
class , where the data instances belong to both classes ( A ∩ B ), and (2) non-overlapping class , wherethe data instances belong to only one of the classes ( A \ B or B \ A ).
Figure 1 shows an example ofoverlapping class and non-overlapping classes when two types of single positive label are given.
Byextending this concept, we can consider combinatorial classes, where data instances belong to certainclasses but not to the rest.
We denote these classes as joint classes .
By accessing a joint class, we cangenerate multi-label samples using only single positive labels.
Ideally, we can represent all possiblelabel combinations with at least one positive label.
Several attempts have been made to consider such a joint class in generation tasks.
Specially designedgenerative models such as GenPU [17] and RumiGAN [18] were proposed to generate samples fromone class while excluding another class.
However, these studies dealt with only two classes anddid not consider the overlapping class.
Recently, Kaneko et al.
[19] proposed CP-GAN to capturebetween-class relationships in conditional generation.
To generate images belonging to n classes,they equally assign 1 /n as the class posterior for each selected class.
This means that CP-GANgenerates images that have an equal probability of being classified as each class.
However, due tothe fact that an image in the real world that belongs to multiple classes does not have equal classposteriors, leading to the loss of sample diversity.
In consequence, we focus on a sampling approachto precisely draw samples from complex distributions that are difficult to directly sample.
In recentstudies of GANs, sampling approaches [20, 21] employed the rejection sampling or Markov chain
Monte Carlo method to obtain realistic data.
They adopt the sampling process as the post-processingmethod of GANs and filter generated images using the pretrained classifiers.
In line with these studies, we propose a novel sampling framework called single-to-multi-label (S2M)sampling to correctly generate data from both overlapping and non-overlapping classes.
We newlypropose a tractable formulation to estimate the conditional density of joint classes.
Concretely, weemploy several classification networks to estimate the target conditional density and apply Markovchain Monte Carlo method upon pretrained unconditional and conditional GANs.
In Figure 2, wedepict the conceptual difference against the existing approaches ( 1 st row) and provide the empiricalresults on 1D Gaussian example ( 2 nd row).
Due to the fact that S2M sampling performs at theinference time of generation, it fully preserves the image quality and eliminates the need for changingthe objective functions and architectures.
We validate the effectiveness of our method compared tothe existing models on diverse datasets such as MNIST, CIFAR-10 and CelebA. To the best of ourknowledge, our approach is the first sampling framework that generates multi-label data from singlepositive labels.
Our contributions can be summarized as follows:
• We introduce the single positive setting in conditional generation task and provide the theoretical
framework for estimating conditional densities of joint classes.•
We propose a novel sampling framework based on Markov chain Monte Carlo method forgenerating multi-label data from single positive labels.• Through vast experiments, we show that our S2M sampling correctly draws multi-label datawhile fully preserving the quality of generated images.
Related Work
Conditional GANs.
The aim of conditional GANs (cGANs)
[11] is to model complex distributionsand to control data generation by reflecting the label input.
Various studies of cGANs have madesignificant advances in class-conditional image generation by introducing the auxiliary classifier [12,
[Figure]
22], modifying the architecture [23, 2], and applying metric learning [24].
In a weakly-supervisedsetting, GenPU [17] and RumiLSGAN [18] specify only two classes and draw samples that belongto one class but not the other.
CP-GAN [19] learns to draw samples conditioned on the probabilityoutput of the classifier.
Given that this model tends to draw samples on a limited region of the dataspace, it is challenging to ensure the variety of samples as shown in Figure 2.
In contrast, we proposea sampling method that draws multi-label samples without sacrificing diversity.
Sampling in GANs.
Sampling methods are used to improve the sample quality in GANs.
Discriminator rejection sampling [20] uses the scheme of rejection sampling and takes samples close to real databy estimating the density ratio with the discriminator.
In addition, Metropolis-Hastings GAN [21]adopts Markov chain Monte Carlo (MCMC) method and calibrates the discriminator to improvethe sample quality in a high-dimensional data space.
Discriminator driven latent sampling [25]uses the MCMC method in the latent space of GANs to draw realistic samples efficiently.
GOLDestimator [26] uses a sampling algorithm to improve the quality of images for class-conditionalgeneration.
While previous studies focus on improving the sample quality, our S2M sampling aims todraw multi-label samples while also improving the quality.
Methods
3.1 Problem Setting
Let x ∈
X be a data point as a random variable and let y 1: n ∈ { 0 , 1 } n denote its correspondingmulti-labels as binary random variables.
Here, for every k , y k
= 1 indicates that x is contained in thek -th class while
y k
= 0 indicates that x is not.
We consider two index sets, an intersection index set
I and a difference index set J , so that the pair ( I, J ) can be used as an index to indicate the class,where data points contained in all classes indicated by I but excluded from all classes indicated by J .
Let I be a collection of all possible pairs of I and J , defined as
[Equation]
where N = { 1 , 2 , ..., n } is a finite index set of all classes and P ( N ) is the power set of N .
Thatis, the intersection index set indicates at least one class and is distinct from the difference index set.
Especially for I ∪ J = N , the class indicated by ( I, J ) is called the joint class .
Let p ( x, y 1 , y 2 , ..., y n ) be the joint probability density function, and let p data ( x, c ) be the joint densityof an observed data point x ∈ X and a class label c ∈ N such that p data ( x | c )
=
p ( x | y c = 1) .
Giventhe class priors p data ( c ) , π c :=
p ( y c = 1) and samples drawn from the class-conditional densityp data ( x | c ) for each c = 1 , 2 , ..., n , our goal is to draw samples from the conditional density
[Equation]
for ( I, J ) ∈ I , and π ( I,J ) :
= p ( ∀ i ∈ I, ∀ j ∈ J, y i = 1 , y j = 0) >
In this work, we propose toimpose a mild constraint which will allow our sampling algorithm to derive the target density.
Assumption 1.
For every i, j ∈ N such that i ̸
= j , if p ( y i = 1 , y j = 0) >
and p ( y j = 1 , y i =0) > 0 , then supp p ( x | y i = 1 , y j = 0)
∩ supp p ( x | y j = 1 , y i = 0)
= ∅ .
The Assumption 1 states that no data points are likely to be assigned two mutually exclusive labels,which can be naturally assumed in many practical situations.
Figure 3 illustrates our problem setting.
(b) Outputs of S2M sampling ① 1
I = {1} J = {2, 3} 1
I = {1, 2} J = {3} 1
I = {1, 2, 3} J = ∅ ③ (a) Single positive setting ③ ② ① A
B
C ?
A
B
C ? ??
(1) (2) (3) ②
3.2 Mining Multi-Label Data with S2M Sampling
A natural question may arise as to whether the supervision given to us is sufficient to obtain multilabel samples.
To gain insight into this, we derive a useful theorem which provides an alternativeformulation for the target density (2).
Theorem 1.
Let { f ( I,J ) :
X → [0 , ∞ ) } ( I,J ) ∈I be an indexed family of non-negative measurablefunctions on X , and let f k :=
f ( { k } , ∅ ) .
Then, the following conditions hold:
[Equation] [Equation]
if and only if, for every ( I, J ) ∈ I ,
[Equation] [Equation]
) where ( · ) + represents the positive part.
Proof.
Please see Appendix.
Let f ( I,J ) ( x ) =
p ( x, ∀ i ∈ I, ∀ j ∈ J, y i = 1 ,
y j
= 0) for every ( I, J ) ∈
I .
Then, both ( a ) and ( b ) in
Theorem 1 hold by the Assumption 1.
According to the Theorem 1, if π ( I,J ) > 0 ,
[Equation]
The alternative formula ( 4 ) shows that the density of joint class can be derived from the classconditional densities of single positive labels.
Despite their clear relationship, training a generatorthat can model the density p ( I,J ) is non-trivial problem since the formula consists of several implicitlydefined conditional densities, class priors, and variable sets.
To address this issue, we propose toapply sampling approaches upon existing GANs.
Interestingly, our sampling framework called S2Msampling allows not only to draw samples from the target density, but also to modify I , J , and classpriors at inference time.
The rest of this section introduces the main approaches of our S2M sampling.
Density Ratio Estimation.
We utilize several classification networks to compute implicitly defineddensity ratios.
For simplicity, we denote G as a pretrained generator for both unconditional andconditional GANs.
G produces data x by taking a latent z and a class label c for class-conditionalgeneration and only z for unconditional generation.
We consider three classifiers D v , D r , and D fwhich are obtained by minimizing L v , L r , and L f , respectively, i.e. ,
[Equation]
where p G is the distribution of generated samples by G .
The optimal classifiers trained by these losses
D ∗ v , D ∗ r , and D ∗ f satisfy the following equations: D ∗ v ( x ) =
p data ( x ) / ( p data ( x )+ p G ( x )) , D ∗ r ( c | x ) =p ( x | y c = 1) p data ( c ) /p data ( x ) , D ∗ f
( c | x ) =
p G ( x | c ) p G ( c ) /p
G ( x ) .
From D ∗ v , D ∗ r , and D ∗ f , we canaccess the density ratios p data ( x )
/p
G ( x ) , p ( x | y c = 1) /p data ( x ) , and p G ( x | c ) /p
G ( x ) which will beused to compute the acceptance probability of the MCMC method.
S2M Sampling for Unconditional GANs.
We apply Metropolis-Hastings (MH) independencesampling [27, 21] to draw samples from the complex target distribution p t .
MH algorithm usesa
Markov process where each transition from a current state x k to next state x k +1 is made byaccept-reject step.
At each step of MH independent sampling, we sample a new proposal x ′ froma proposal distribution q ( x ′ | x ) =
q ( x ′ )
and then accept it with probability α ( x ′ , x ) which is given by α ( x ′ , x ) =
min (cid:16) 1 , p t ( x ′ )
q ( x ) p t ( x ) q ( x ′ ) (cid:17) .
We set x k +
= x ′ if x ′ is accepted, x k +
=
x
k otherwise.
Then,the chain of samples x 1: K converges to the unique stationary distribution p t as K → ∞ .
Given a generator G where the support of p G contains that of p ( I,J ) .
To draw samples from p ( I,J ) ,we let p t =
p
I,J )
and take multiple samples from G as independent proposals, i.e. q ( x ′ | x )
= p G ( x ′ ) .
Then, the desired acceptance probability α ( x ′ , x ) can be calculated using D ∗ v and D ∗ r :r ( I,J ) ( x ) :=
[Equation]
Different from Turner et al.
[21], the term r ( I,J ) ( x ′ ) /r
( I,J ) ( x ) is added to the acceptance probabilityformula to draw multi-label samples.
To obtain uncorrelated samples, we take a sample after a fixednumber of iterations for each chain.
We note that the sampling approach allows one to control theparameters I , J , and γ k :=
π k /p data ( k ) without any additional training of the model.
A summary ofour S2M sampling algorithm is provided in Appendix.
S2M Sampling for Conditional GANs.
cGANs can provide a proposal distribution close to thetarget distribution p ( I,J ) , which greatly increases the sample efficiency of the MCMC method.
Let cbe a class label such that the support of class-conditional density p G ( ·| c ) contains that of p ( I,J ) .
Ateach step of the MH algorithm, the proposal x ′ ∼ q ( x ′ | x ) =
p G ( x ′ | c ) is accepted with a probabilityα c ( x ′ , x ) .
The desired α c ( x ′ , x ) can be calculated asα c ( x ′ , x ) =
min (cid:18) 1 , p ( I
[Equation]
( x
[Equation]
) /p G (
[Equation]
That is, the sampling method can be adopted to cGANs by additionally computing D ∗ f
( c | x ) /D ∗ f
( c | x ′ ) .
Latent Adaptation in S2M Sampling.
While our S2M sampling allows us to draw multi-labelsamples, the algorithm rejects certain samples through the sampling procedure if the target distributionp ( I,J ) is significantly different from the generator distribution.
Specifically, independent MH algorithm takes time inversely proportional to the probability of a generated sample belonging to the targetclass (See Appendix).
To alleviate this issue, we propose a technique to improve the sample efficiencyfrom past sampling attempts.
We initially draw a certain number of pilot target class samples x 1: m using S2M sampling, and then obtain the corresponding target latent samples t 1: m ; x k = G ( t k ) .
Sincethe latent samples are nearly restricted to the latent prior of GANs ( e.g. , Gaussian distribution), we canroughly estimate the distribution ˜ p
z of the target latent by fitting a simple probabilistic model usingt 1: m .
Let ˜ p G be the newly obtained generator distribution induced by G , i.e. , G ( z )
= x ∼ ˜ p G ( x ) forz ∼ ˜ p
z ( z ) .
We can further improve the sample efficiency by using ˜ p G as the proposal distributionin the MH algorithm since ˜ p G is much close to the target distribution.
To run the MH algorithmwithout retraining the classifiers, we approximate p G ( G ( z )) /
˜ p G ( G ( z ))
≈ C · p z ( z ) /
˜ p
z ( z ) 1 for someconstant C .
Then, at each step of the MH algorithm, the proposal x ′ ∼ q ( x ′ | x ) =
˜
p G ( x ′ ) is acceptedwith a probability ˜ α
( x ′ , x ) which can be calculated as
[Equation]
where x ′ =
G ( z ′ ) and x = G ( z ) .
Here, we can explicitly compute the density ratio between p z and˜ p z by letting ˜ p
z be a known probability density function such as Gaussian mixture.
In practice, if thetarget class samples rarely appear in generated samples, one can consider applying latent adaptationrepeatedly.
For instance, to draw Black-haired man samples, we can first search the latent spaceof Man by using latent adaptation, and then perform it in that space again to search the space of
Black-haired man .
In terms of time complexity, this is performed more efficiently than searching thespace of Black-haired man at once.
Detailed discussion about the time complexity and the descriptionof latent adaptation are provided in Appendix.
0 : Airplane 1 : Automobile 2 : Bird 3 : Cat 4 : Deer 5 : Dog 6 : Frog
A : {3, 5} B : {3}
A
B
A : All B : Odd digits
Even 3/
A
B 63 1 54 C
A : bAngs B : Black-hair M : Male
N : No-beard S : Smiling
W : Wearing- lipstick
A+B+M+N+S+W B
M
[Figure]
A
B
M
N
[Figure]
Practical Considerations.
We employ three classifiers D v , D r , and D f , to compute the acceptanceprobability used in MCMC method.
For better training efficiency of the classifiers, we use sharedlayers, except for the last linear layer.
Since the classifiers are not optimal in practice, we scale thetemperature of classifiers [29] and adjust γ k to calibrate the sampling algorithm.
We provide thedetailed settings and the ablation study in Appendix.
Experiments
In this section, we validate that our S2M sampling properly draw samples within the joint classes.
Specifically, we mainly consider two cases in the single positive setting: (i) one class is entirelycontained in another class, (ii) multiple classes are partially overlapped with each other.
In the formersetting, we verify that our S2M sampling can completely filter out samples of a smaller class, andcompare our method to the existing models including GenPU [17] and RumiLSGAN [18] on MNISTand Fashion-MNIST (FMNIST).
In the latter setting, we assess all possible joint classes with our
S2M sampling.
Subsequently, we evaluate how accurately our S2M sampling draws samples in theseclasses compared to CP-GAN [19] and the models trained with fully annotated multi-label dataset.
To verify whether a method can accurately generate samples for each joint class, we evaluateaccuracy which indicates how many generated images are assigned to the target joint classes by aclassifier trained with fully annotated data ( i.e. , multi-label data).
Since accuracy itself cannot fullyevaluate the distance between the distribution of generated samples and that of real samples, weadditionally introduce various metrics to evaluate fidelity ( i.e. , how realistic the generated samplesare) and diversity of generated images: (i) Inception Score (IS) [30], (ii) Fréchet Inception Distance(FID)
[31], (iii) improved precision and recall [32], and (iv) density and coverage [33].
FID and IS are metrics to evaluate the fidelity and diversity of generated samples using the featuresof the pretrained Inception-V3 network [34].
A lower FID and a higher IS indicate higher fidelity anddiversity of generated samples.
Precision and density measure the ratio of generated samples thatfalls within the manifold of real samples, so that they evaluate the fidelity of generated samples.
Onthe contrary, recall and coverage measure the ratio of real samples that falls within the manifold ofgenerated samples, and thus it evaluates the diversity of generated samples.
That is, a higher value ofthese metrics indicates that the generated samples have a distribution similar to the real one.
Since several metrics for fidelity and diversity are not applicable for non-ImageNet-like images( e.g. , MNIST and FMNIST), we instead use FID score computed from activations of LeNet5 [35]and denote this FID † .
We use the test set as a reference dataset for evaluating FID.
All results areaveraged over three independent trials, and the standard deviation is denoted by subscripts.
Detailedexperimental settings, such as architectures and hyperparameters, are provided in Appendix.
4.1 Sampling for Classes with Inclusion
Experimental Settings.
We consider a special case of our problem setting, where one class isentirely contained in another class ( B ⊂ A ) as shown in Figure 4 (a).
This setting is similar tothe positive-unlabeled setting [36, 37] where the positive data is included in the unlabeled data.
Under this constraint, GenPU [17], RumiLSGAN [18], and CP-GAN [19] can be used to generatesamples from the non-overlapping class ( A \ B ).
Along with these models, we attempt to drawsamples from the non-overlapping class by adopting our S2M sampling to unconditional WGANGP [38].
These experiments are conducted in three settings (See Figure 4): (i) MNIST-3/5 ( A ={3,5}, B ={3})
, (ii) MNIST-Even ( A ={0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, B ={1, 3, 5, 7, 9}), and (iii)
FMNIST-Even ( A = { 0 T-shirt/Top , 1 Trouser , 2 Pullover , 3 Dress , 4 Coat , 5 Sandal , 6 Shirt , 7 Sneaker , 8 Bag , 9 Ankle boot } , 231 B = { 1 , 3 , 5 , 7 , 9 } ).
For S2M sampling, we obtain samples at 100 MCMC iterations.
[Figure]
Quantitative Results.
Table 1 shows the results of our S2M sampling and the baselines for thenon-overlapping classes.
The performance of GenPU is reported only for MNIST-3/5 due to itsmode collapse issue [39, 40].
In the results, our S2M sampling adopted to unconditional GANshows promising results in terms of both accuracy and FID † .
In fact, our results prove superior tothe existing methods specially designed for generating non-overlapping data such as GenPU and
RumiLSGAN.
Notably, our S2M sampling improves the accuracy by 8 . 5% and 6 . 8% while decreasing
FID † compared to the second-best models on MNIST-Even and FMNIST-Even, respectively.
Thisindicates that our S2M sampling accurately samples the images of the non-overlapping classeswithout being biased to a specific mode.
[Table]
Qualitative Results.
Figure 5 shows the qualitative results of RumiLSGAN, CPGAN and ourmethod adopted to WGAN-GP.
As indicated by red boxes, RumiLSGAN and CP-GAN fail tocompletely eliminate the samples of the smaller class ( e.g. , odd digits for MNIST-Even or Sneaker for
FMNIST-Even).
On the other hand, our S2M sampling correctly draw samples for the target classes.
4.2 Sampling for Multiple Classes
Experimental Settings.
We consider a general case of single positive setting where multiple classescan be overlapping as shown in Figure 4 (b-d).
Given n classes, at most 2 n − 1 joint classes can beobtained.
In this setting, we attempt to get samples of these joint classes using only single positivelabels.
We consider cGANs with projection discriminator (cGAN-PD) [23], AC-GAN [12], and
CP-GAN [19] as the baseline generative models.
Since the traditional conditional GANs such as cGAN-PD and AC-GAN do not originally designed forgenerating joint classes, we naively introduce those models in our setting with the slight modification.
Concretely, as a method to generate images belonging to n classes, we provide 1 /n as the conditionalvalue for each class, following Kaneko et al.
In our experiments, we expect these models toserve a lower bound performance of our results and indicate their results by asterisks ( ∗ ).
In contrast,conditional GANs trained on a fully annotated dataset where all joint classes are specified as strongbaselines in our settings.
We train cGAN-PD and AC-GAN in this setting, and denote them as oraclemodels .
In the experiments, our S2M sampling is adopted on unconditional GAN and cGAN-PD.
When adopting our S2M sampling to cGAN-PD, we use a class containing the target joint class as theconditional value of cGAN-PD.
We use BigGAN [2] as the backbone architecture for all generativemodels, and only single positive labels are used except for the oracle models.
To demonstrate the effectiveness of our S2M sampling, we use three real-world image datasets: (i)
CIFAR-7to3 that has three classes, each of which contains four original classes of CIFAR-10, (ii)
CelebA-BMS that consists of human face images, each of which is labeled for one of three attributes( Black-hair , Male , and Smiling ), and (iii) CelebA-ABMNSW that consists of human face images,each of which is labeled for one of six attributes ( bAngs , Black-hair , Male , No-beard , Smiling , and
Wearing-lipstick ).
As depicted in Figure 4 (b-d), these datasets can have up to 2 3 − 1 or 2 6 − 1 jointclasses.
For S2M sampling, samples are obtained at 200 MCMC iterations for CIFAR-7to3 and
CelebA-BMS, and obtained at 1000 MCMC iterations for CelebA-ABMNSW.
Airplane
Cat Deer
Dog
Frog
[Figure]
Bird
[Figure]
I
[Figure]
AR
[Figure]
7 t o
[Figure]
l
[Figure]
A
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Quantitative Results.
Table 2 summarizes the quantitative results of joint class generation on
CIFAR-7to3, CelebA-BMS, and CelebA-ABMNSW.
As expected, we can observe that cGAN-PD ∗and AC-GAN ∗ cannot accurately generate samples from joint classes, which is consistent with thefindings of Kaneko et al.
Although CP-GAN shows reasonable accuracy for all experiments, itsuffers from generating diverse samples.
As discussed in Section 1, the high FID and low coveragesupport that CP-GAN tends to generate samples in a narrow scope of real data space.
In all experiments, our S2M sampling adopted to cGAN-PD consistently outperforms the non-oraclebaselines in terms of accuracy and diversity, e.g. , in CIFAR-7to3, our method improves accuracy and
FID of CP-GAN by 38% and 13.74, respectively.
Such improvements verify that our S2M samplinggenerate correct samples for joint classes, and also preserves diversity within them.
Despite a largesample space of GAN, S2M sampling adopted to unconditional GAN also surpasses non-oraclebaselines in terms of both fidelity and diversity while achieving reasonable accuracy.
Surprisingly,despite the fact that the oracle models are trained with fully annotated data, our S2M sampling showsa comparable performance against those models using only single positive labels.
Moreover, as anpost-processing method, our S2M sampling fully preserves the image quality of GANs and shows thehighest fidelity in terms of precision and density.
[Figure]
Qualitative Results.
Figure 6 depicts the results of our S2M sampling for every possible jointclasses on CIFAR-7to3 and CelebA-BMS.
The results show that our S2M sampling can draw diversehigh-quality images from the distributions of all joint classes.
Furthermore, our S2M sampling ismodel-agnostic and does not require the retraining of GANs.
Thus, it can be easily integrated withvarious GANs including the state-of-the-art GANs.
We depict more qualitative results of adoptingour S2M sampling to unconditioanl GAN and pretrained StyleGANv2 in Appendix.
Analysis on Latent Adaptation.
To demonstrate the effect of latent adaptation, we test our latentadaptation technique in a overlapping class (B+M+S) and a non-overlapping class (B-M-S) on
CelebA-BMS dataset.
We first perform our S2M sampling on unconditional BigGAN in these classesand draw 10 k pilot latent samples obtained at 100 MCMC iterations.
Using these latent samples,we fit Gaussian mixture of eight components as a new latent of GAN (See Appendix).
Due to thediscrepancy between the generator distribution and the target distribution, S2M sampling originallyshows a low acceptance probability as shown in Figure 7.
In this case, applying latent adaptationincreases a acceptance probability and significantly improves both accuracy and FID at the early stageof iterations.
This indicates that the generator distribution becomes closer to the target distribution byadopting latent adaptation, resulting in accelerating convergence speed.
Limitations
[Figure]
In general, as the number of attributes increases,the number of each joint class samples in thetraining dataset is drastically reduced.
This maycause two main problems: (i) Training GANsto be able to generate each joint class samplesbecomes challenging, and our S2M samplingcannot draw samples which GANs do not generate.
(ii)
The computation time of the samplingprocedure increases.
In this case, we may needto apply latent adaptation technique to shorten MCMC chains.
We provide the detailed timecomplexity analysis in Appendix.
As shown in Figure 8, the number of training samples in joint classes rapidly decreases.
Due to thisnature, we use at most six attributes on CelebA dataset for experiments.
Almost half of the jointclasses have fewer than 100 training samples when using six attributes ( i.e. , CelebA-ABMNSW).
Nevertheless, we deal with a larger number of attributes compared to the existing methods.
Webelieve that our work can be further extended to larger attributes with the sufficient amounts of data.
Conclusion
We investigate the single positive setting in the class-conditional generation task and propose anovel sampling framework called S2M sampling.
We demonstrate that our proposed S2M samplingcan accurately draw high quality multi-label samples in the single positive setting.
To improve thesampling efficiency, we also introduce the latent adaptation technique.
We believe that our theoreticalframework provides a better understanding for weakly-supervised multi-label data generation.
Our
S2M sampling has the potential to be applied to a wide variety of generative models as well, includingclustering based GANs [41] and diffusion models [42, 43], among others.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide the
codes in the supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes] We reported the standard deviation in the experimental results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[No] We used publicly available benchmark
[Table]
datasets: MNIST, FMNIST, CIFAR10, CelebA, and CelebA-HQ.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
We provide the codes in the supplemental material.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[Yes] The benchmark datasets used in our paper are widely used for [Table] research purpose.
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?
[Yes] The benchmark datasets used in our paper are [Table] widely used by others and do not contain any offensive contents. [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
[Table]