D IFFERENCE -S EEKING G ENERATIVE A DVERSARIAL N ETWORK –U NSEEN S AMPLE G ENERATION
A BSTRACT
Unseen data, which are not samples from the distribution of training data and are difﬁcult to collect, have exhibited importance in numerous applications, ( e.g., novelty detection, semi-supervised learning, and adversarial training).
In this paper, we introduce a general framework called d ifference- s eeking g enerative a dversarial n etwork (DSGAN), to generate various types of unseen data.
Its novelty is the consideration of the probability density of the unseen data distribution as the difference between two distributions p ¯ d and p d whose samples are relatively easy to collect.
The DSGAN can learn the target distribution, p t , (or the unseen data distribution) from only the samples from the two distributions, p d and p ¯ d .
In our scenario, p d is the distribution of the seen data, and p ¯ d can be obtained from p d via simple operations, so that we only need the samples of p d during the training.
Two key applications, semi-supervised learning and novelty detection, are taken as case studies to illustrate that the DSGAN enables the production of various unseen data.
We also provide theoretical analyses about the convergence of the DSGAN.
I NTRODUCTION
Unseen data 1 are not samples from the distribution of the training data and are difﬁcult to collect.
It has been demonstrated that unseen samples can be applied to several applications.
Dai et al.
(2017) proposed how to create complement data, and theoretically showed that complement data, considered as unseen data, could improve semi-supervised learning.
In novelty detection, Yu et al.
(2017) proposed a method to generate unseen data and used them to train an anomaly detector.
Another related area is adversarial training Goodfellow et al.
(2015), where classiﬁers are trained to resist adversarial examples, which are unseen during the training phase.
However, the aforementioned methods only focus on producing speciﬁc types of unseen data, instead of enabling the generation of general types of unseen data.
In this paper, we propose a general framework called d ifference- s eeking g enerative a dversarial n etwork (DSGAN), to generate a variety of unseen data.
The DSGAN is a generative approach.
Traditionally, generative approaches, which are usually conducted in an unsupervised learning manner, are developed for learning the data distribution from its samples, from which subsequently, they produce novel and high-dimensional samples, such as the synthesized image Saito et al.
A state-of-the-art approach is the so-called generative adversarial network (GAN) Goodfellow et al.
GAN produces sharp images based on a game-theoretic framework, but it can be difﬁcult and unstable to train owing to multiple interaction losses.
Speciﬁcally, GAN consists of two functions: generator and discriminator.
Both functions are represented as parameterized neural networks.
The discriminator network is trained to determine whether the inputs belong to the real dataset or fake dataset created by the generator.
The generator learns to map a sample from a latent space to some distribution to increase the classiﬁcation errors of the discriminator.
Nevertheless, if a generator can learn to create unseen data, then a traditional GAN requires numerous training samples of unseen classes for training, leading to a contradiction with the deﬁnition of the unseen data.
This fact motivates us to present the DSGAN, which can generate unseen data by adopting seen data as training samples (see Fig.
9, which illustrates the difference between GAN and the DSGAN, in Appendix A).
The key concept is to consider the distribution of the unseen data as the difference between two distributions that are relatively easy to obtain.
For example, the out-of-distribution examples in the MNIST dataset, from another perspective, are found to belong to the differences between the sets of examples in MNIST and the universal set.
It should be noted that in traditional GAN, the target distribution is identical to the training data distribution; however, in the
DSGAN these two distributions, are considered to be different.
This paper makes the following contributions:
(1) We propose the DSGAN to generate any unseen data only if the density of the target (unseen data) distribution is the difference between those of any two distributions, p ¯ d and p d .
(2) We show that the DSGAN possesses the ﬂexibility to learn different target (unseen data) distributions in two key applications, semi-supervised learning and novelty detection.
Specifically, for novelty detection, the DSGAN can produce boundary points around the seen data because this type of unseen data is easily misclassiﬁed.
For semi-supervised learning, the unseen data are linear combinations of any labeled data and unlabeled data, excluding the labeled and unlabeled data themselves 2 .
The DSGAN yields results comparable to a semi-supervised learning but with a short training time and low memory consumption.
In novelty detection, combining both the DSGAN and variational auto-encoder (VAE, Kingma & Welling (2014b)) methods achieve the state-of-the-art results.
P ROPOSED M ETHOD -DSGAN 2. F ORMULATION
We denote the generator distribution as p g and training data distribution as p d , both in an Ndimensional space.
Let p ¯ d be the distribution decided by the user.
For example, p ¯ d can be the convolution of p d and normal distribution.
Let p t be the target distribution that the user is interested in, and it can be expressed as
[Equation]
where α ∈ [0 , 1] .
Our method, the DSGAN, aims to learn p g such that p g =
p t .
Note that if the support set of p d belongs to that of p ¯ d , then there exists at least an α such that the equality in (1) holds.
However, even if the equality does not hold, intuitively, the DSGAN attempts to learn p g such that p g ( x ) ∼ p
¯ d ( x ) − αp d ( x ) 1 − α with the constraint, p g ( x ) ≥ 0 .
Speciﬁcally, the generator will output samples located in the high-density areas of p ¯ d − αp d .
Furthermore, we show that the DSGAN can learn p g , whose support set is the difference between those of p ¯ d and p d in Theorem 1.
First, we formulate the generator and discriminator in GANs.
The inputs, z , of the generator are drawn from p z ( z ) in an M -dimensional space.
The generator function, G ( z ; θ g ) :
R M → R N , represents a mapping to the data space, where G is a differentiable function with parameter θ g .
The discriminator is deﬁned as D ( x ; θ d ) : R N → [0 , 1] , which outputs a single scalar.
D ( x ) can be considered as the probability that x belongs to a class of the real data.
Similar to traditional GAN, we train D to distinguish the real data from the fake data sampled from G .
Concurrently, G is trained to produce realistic data that can mislead D .
However, in the
DSGAN, the deﬁnitions of “real data” and “fake data” are different from those in traditional GAN.
The samples from p ¯ d are considered as real, but those from the mixture distribution between p d and p g are considered as fake.
The objective function is deﬁned as follows:
from the "seen" data, according to applications.
[Equation]
We optimize (2) by a min–max game between G and D , i.e.,
[Equation]
During the training procedure, an iterative approach, like traditional GAN, is to alternate between k steps of training D and one step of training G .
In practice, minibatch stochastic gradient descent via backpropagation is used to update θ d and θ g .
Thus, for each p g , p d , and p ¯ d , m samples are required for computing the gradients, where m is the number of samples in a minibatch.
The training procedure is illustrated in Algorithm 1 in Appendix A.
The DSGAN suffers from the same drawbacks as traditional GAN, ( e.g. , mode collapse, overﬁtting, and strong discriminator) so that the generator gradient vanishes.
There are literature Salimans et al.
; Arjovsky & Bottou (2017);
Miyato et al.
(2018) focusing on dealing with the above problems, and such concepts can be readily combined with the DSGAN.
Li et al.
(2017) and Reed et al.
(2016) proposed an objective function similar to (2).
Their goal was to learn the conditional distribution of training data.
However, we aim to learn the target distribution, p t , in Eq.
(1), and not the training data distribution.
2. C ASE S TUDY ON V ARIOUS U NSEEN D ATA G ENERATION
To achieve a more intuitive understanding about the DSGAN, we conduct several case studies on two-dimensional ( 2 D) synthetic datasets and MNIST.
In Eq.
(1), α = 0 .
8 is used.
First, the convolution of p d and normal distribution ensure the density on the boundary is no longer zero.
Second, we seek p g such that Eq.
(1) holds, where the support set of p g is approximated by the difference of those between p ¯ d and p d .
Complement samples generation Fig.
1 illustrates that the DSGAN can generate complement samples between 2 circles.
Denoting the density function of the two circles as p d , we assign the samples drawn from p ¯ d as linear combinations of the two circles.
Then, by applying the DSGAN, we achieve our goal of generating complement samples.
In fact, this type of unseen data is used in semi-supervised learning.
Boundary samples generation Fig.
2 illustrates that the DSGAN generates boundary points between four circles.
This type of unseen data is used in novelty detection.
In this case, we assign p d and p ¯ d as “the density function of four circles” and “the convolution of p d and normal distribution,” respectively.
The basis of our concept is also illustrated by a one-dimensional ( 1 D) example in Fig.
Difference-set generation We also validate the DSGAN on a high-dimensional dataset such as
MNIST.
In this example, we deﬁne p d as the distribution of digit “1” and p ¯ d as the distribution containing two digits “1” and “7”.
Because the density, p d ( x ) , is high when x is digit “1,” the generator is prone to output digit “7” with a high probability. More sample qualities of DSGAN on
CelebA can be refer to Appendix G. From the above results, we can observe two properties of the generator distribution, p g : i) the higher the density of p d ( x ) , the lower the density of p g ( x ) ; ii) p g prefers to output samples from the high-density areas of p ¯ d ( x ) − αp d ( x ) .
2. D ESIGNING p ¯ d
Thus far, we have demonstrated how the DSGAN can produce various types of unseen data by choosing a speciﬁc p ¯ d .
In this section, we introduce a standard procedure to design p ¯ d , and illustrate each step with pictures.
Step 1.
First, the training data, p d , are collected (Fig. 6 (a)).
Step 2.
Second, based on the applications, the desired unseen data distribution is deﬁned ( e.g. , complement samples for semi-supervised learning) (Fig.
6 (b)).
Step 3.
Third, p ¯ d is deﬁned as a mixed distribution of (1 − α ) p g + ( α )
p d (Fig.
(c)).
Step 4.
Finally, a suitable mapping function that can transform p d to p ¯ d is designed ( e.g , linear combination of any two samples of p d ) (a
[Figure]
In the above procedure, the most important step is to determine which types of unseen data are suitable for a speciﬁc problem (Step 2).
In this paper, we show two types of unseen data, which are useful in semi-supervised learning and novelty detection.
However, determining all types of unseen data for all applications is beyond the scope of this study, and we leave this for future work.
Furthermore, we provide a method (see Appendix B in supplementary materials) by reformulating the objective function (2), so that it is more stable to train the DSGAN.
T HEORETICAL R ESULTS
In this section, we show that by choosing an appropriate α , the support set of p g belongs to the difference set between p ¯ d and p d , so that the samples from p g are unseen from the p d perspective.
We start our proofs from two assumptions.
First, in a non-parametric setting, we assume that both the generator and discriminator have inﬁnite capacities.
Second, p g is deﬁned as the distribution of the samples drawn from G ( z ) under z ∼ p z .
In the following, we show that the support set of p g is contained within the differences in the support sets of p ¯ d and p d while achieving the global minimum such that we can generate the desired p g by designing an appropriate p ¯ d .
Theorem 1.
Suppose αp d ( x ) ≥
p ¯ d ( x ) for all x ∈ Supp( p d ) and all density functions p d ( x ) , and p ¯ d ( x ) and p g ( x ) are continuous.
If the global minimum of C ( G ) is achieved, then Supp
[Equation]
p g )
[Equation]
Supp
[Equation]
p [Equation] d [Equation]
Supp
[Equation]
p d ) , where
[Equation]
Proof.
See Appendix C for the details.
Summarizing, the generator is prone to output samples that are located in the high-density areas of p ¯ d − αp d .
A PPLICATIONS
The DSGAN was applied to two problems: semi-supervised learning and novelty detection.
In the semi-supervised learning, the DSGAN acts as a “bad generator,” which creates complement samples (unseen data) in the feature space of the training data.
For the novelty detection, the DSGAN generates the samples (unseen data) as boundary points around the training data.
4. S EMI -S UPERVISED L EARNING
Semi-supervised learning (SSL) is a type of learning model that uses a few labeled data and numerous unlabeled data.
The existing SSL methods based on a generative model, ( e.g., VAE Kingma et al.
(2014) and GAN Salimans et al.
(2016)), yield good empirical results.
Dai et al.
(2017) theoretically showed that a good semi-supervised learning required a bad GAN with the following objective function:
[Equation]
where ( x, y ) denotes a pair of data, and its corresponding label, { 1 , 2 , . .
, K } denotes the label space for the classiﬁcation, and L = { ( x, y ) } is the label dataset.
Moreover, under the semi-supervised settings, p d in (3) is the distribution of the unlabeled data.
Note that the discriminator, D , in GAN also plays the role of a classiﬁer.
If the generator distribution exactly matches the real data distribution ( i.e. , p g = p d ), then the classiﬁer trained by the objective function (3) with the unlabeled data cannot have a better performance than that trained by the supervised learning with the objective function.
Speciﬁcally,
[Equation]
Contrastingly, the generator is preferred to generate complement samples, which lie on the lowdensity area of p d .
Under some mild assumptions, these complement samples help D to learn the correct decision boundaries in the low-density area because the probabilities of the true classes are forced to be low in the out-of-distribution areas.
The complement samples in Dai et al.
(2017) are complex to produce.
In Sec. 5.2, we will demonstrate that with the DSGAN, complement samples can be easily generated.
4. N OVELTY D ETECTION
Novelty detection determines if a query example belongs to a seen class.
If the samples of one seen class are considered as positive data, then this difﬁculty is the absence of negative data in the training phase, so that the supervised learning cannot function.
Recently, novelty detection has made signiﬁcant progress with the advent of deep leaning.
Pidhorskyi et al.
(2018)Sakurada & Yairi (2014) focused on learning a representative latent space for a seen class.
When testing, the query image was projected onto the learned latent space.
Then, the difference between the query image and its inverse image (reconstruction) was measured.
Thus, only an encoder was needed to be trained for the projection and a decoder for the reconstruction.
Under the circumstance, an autoencoder (AE) is generally is adopted to learn both the encoder and decoder
Pidhorskyi et al.
(2018)Perera et al.
Let Enc( · ) be the encoder and Dec( · ) be the decoder.
The loss function of the AE is deﬁned as
[Equation]
where p pos is the distribution of a seen class.
After the training, a query example, x test , is classiﬁed as the seen class if
[Equation]
where τ ∈ R + plays the trade-off between the true positive rate and false positive rate.
However, (6) is based on two assumptions: (1) the positive samples from one seen class should have a small reconstruction error; (2) the AE (or latent space) cannot well describe the negative examples from the unseen classes, leading to a relatively large reconstruction error.
In general, the ﬁrst assumption inherently holds when both the testing and training data originate from the same seen class.
However,
Pidhorskyi et al.
(2018)Perera et al.
(2019) observed that assumption (2) does not hold at all times because the loss function in (5) does not include a loss term to enforce the negative data to have a large reconstruction error.
For assumption (2) to hold, given positive data as the training inputs, we propose using the DSGAN to generate negative examples in the latent space, as discussed in Sec.
Then, the loss function of the AE is modiﬁed to enforce the negative data to have a large reconstruction error.
E XPERIMENTS
Our experiments are divided into three parts.
The ﬁrst one examines how the hyperparameter, α , inﬂuences the learned generator distribution, p g .
In the second and third experiments, we obtain empirical results about semi-supervised learning and novelty detection, which are presented in Sec.
5.2 and Sec. 5.3, respectively.
Note that the training procedure of the DSGAN can be improved by other extensions of GANs such as WGAN Arjovsky et al.
WGAN-GP Gulrajani et al.
EBGAN Zhao et al.
(2017), and LSGAN Mao et al.
In our method, the WGAN-GP was adopted for the stability of the DSGAN in training and reduction in the mode collapse.
5. DSGAN WITH DIFFERENT α
The impacts of different α values on the DSGAN are illustrated in Fig.
In this example, the support of p d is the area bounded by a red dotted line, and the orange points are the samples from p d .
Concurrently, we shift p d to the right by 1 unit and create the distribution, p ¯ d , whose support is bounded by blue dotted lines.
The overlapping area between p ¯ d and p d is 0 . 5 unit (assuming the area of p d is 1 unit).
Based on our theoretical results, α = 0 .
5 is the smallest selected value allowing p g to be disjoint to p d .
Therefore, we can see that some generated samples, as presented in Fig. 7(a), still belong to the support set of p d .
Fig. 7(b) shows that there is a perfect agreement between our theoretical and experiment results with α = 0 .
When α = 0 .
8 , there is a remarkable gap between the generated (green) points and yellow points, as shown in Fig. 7(c).
In theory, the result obtained at α = 0 .
8 should be the same as that obtained at α = 0 .
This is because the discriminator should assign the entire area, which is the intersection of the complement of support set of p d and support set of p ¯ d , to the same score, under the assumption that the discriminator has an inﬁnite capacity.
However, in practice, the capacity of the discriminator is limited.
Therefore, the score of the area near p d is lower than that far from it, when α is large.
Therefore, p g tends to repel p d to achieve a high score (to deceive the discriminator).
5. DSGAN IN S EMI -S UPERVISED L EARNING
We ﬁrst introduce how the DSGAN generates the complement samples in the feature space.
Dai et al.
(2017) proved that if the complement samples generated by G could satisfy the following two assumptions in (7) and (8), i.e.,
[Equation]
where f is the feature extractor and w i is the linear classiﬁer for the i th class, and
[Equation]
then all the unlabeled data would be correctly classiﬁed by the objective function (3).
Speciﬁcally, (7) ensures that the classiﬁers can discriminate the generated data from the unlabeled data, and (8) causes the decision boundary to be located in the low-density areas of p d . [Figure]
move farther away from p d as α increases; however, they are still bounded by the support of p ¯ d .
When α is 0 .
5 , the support set of p g is disjoint to that of p d , satisying the theoretical results.
When α is 0 . 8 , p g generates the rightmost points of p ¯ d . The level curves from the discriminator show that the generator is more prone to producing samples in a region with higher score than in that with a lower score. Note that the outputs of the discriminator are not restricted in [0 , 1] , because we use the formulation of the WGAN in this experiment. The assumption in (8) implies that the complement samples must be in the space created by the linear combination of the labeled and unlabeled data.
In addition, they cannot fall into the real data distribution, p d , owing to the assumption (7).
To allow the DSGAN to generate such samples, we let the samples of p ¯ d be linear combinations of those from L and p d .
Since p g ( x ) ≈
p ¯
d ( x ) − αp d ( x ) 1 − α , p g will tend to match p ¯ d , whereas the term, − αp d , ensures that the samples from p g do not belong to p d .
Thus, p g satisﬁes the assumption in (8).
Moreover, (7) is also satisﬁed by training the classiﬁer with (3) based on substituting the generator distribution in (3) into the learned p g .
Following the previous works, we apply the proposed DSGAN to semi-supervised learning on three benchmark datasets: MNIST LeCun et al.
SVHN Netzer et al.
(2011), and CIFARKrizhevsky (2009).
The details of the experiments can be found in Appendix D.
5.2. S IMULATION R ESULTS
First, the selected hyperparameters are listed in Table 5 in Appendix D.1.
Second, the results obtained from the DSGAN and state-of-the-art methods on the three benchmark datasets are summarized in Table 1.
It can be observed that our method can compete with the state-of-the-art methods on the three datasets.
Note that we report the results of badGAN not only from the original papers in the literature but also by reproducing them using the released codes of the authors.
The reason of presenting both the results is that we cannot reproduce parts of the results.
The experiments in Li et al.
(2019) also showed a similar problem.
In comparison with Dai et al. (2017), our methods do not need to rely on an additional density estimation network, PixelCNN++ Salimans et al.
Although
PixelCNN++ is one of the best density estimation networks, learning such a deep architecture requires large computation and high memory consumption.
In Table 2, we list the training time and memory consumption for our method and badGAN.
Compared to badGAN, our method consumes 15 .
8% less training time and saves about 9000 MB during the training.
Moreover, it can also be observed from Table 1 that our results are comparable to the best record of badGAN and CAGAN.
are better than those of other approaches on the MNIST and SVHN datasets.
On CIFAR-10, our method is only inferior to the CT-GAN.
However, this might not be a reasonable comparison because the CT-GAN uses extra techniques, including temporal ensembling and data augmentation, which the other methods do not use.
5. DSGAN IN N OVELTY D ETECTION
In this section, we study how to use the DSGAN for assisting novelty detection.
As mentioned in
Sec.
4.2, we need to train the auto-encoder (AE) such that (i) the positive samples from one seen class have a small reconstruction error; (ii) negative samples from the unseen classes incur relatively higher reconstruction errors. [Table] [Table]
The fundamental concept is to use the DSGAN to generate negative samples, which originally do not exist under the scenario of novelty detection.
Next, we add a new loss term to penalize the small reconstruction errors of the negative samples (see the third stage below).
Three stages are required to train our model (AE):
The encoder, Enc( · ) , and decoder, Dec( · ) , are trained using the loss function (5).
Given x ∼ p pos , Enc( x ) are collected as the samples drawn from p d .
p ¯ d is the convolution of p d having a normal distribution with a zero mean and variance σ .
Then, we train the DSGAN to generate negative samples, which are drawn from p ¯ d ( x ) − p d ( x ) and are the boundary points around the positive samples in the latent space.
Note that there are some variations in the DSGAN: the input of the generator, G , is Enc( x ) , instead of a random vector z in the latent space.
We also add (cid:107) Enc( x ) − G (Enc( x ))
(cid:107) 22 , which will be explained in the next step, to train the generator. 3.
Fixing the encoder, we retrain the decoder by the modiﬁed loss function,
[Equation]
where w is the trade-off between the reconstruction errors of positive samples
Enc( x ) and negative samples G (Enc( x )) .
Note that in the previous step, we add (cid:107) Enc( x ) − G
(Enc( x )) (cid:107) 22 to ensure that the outputs of the generator are around the input.
Thus, the second term charges even though the negative samples are close to the corresponding positive sample, and they still exhibit a high reconstruction error, which is bounded by m (Zhao et al.
The above algorithm, called VAE+DSGAN, can be used to strengthen the existing AE-based methods by using them in the ﬁrst stage.
In the simulation, we used a variational autoencoder (VAE) Kingma & Welling (2014a) because it performs better than the AE in the novelty detection.
5.3. S IMULATION R ESULTS
In this section, following Perera et al.
, the performance was evaluated using the area under the curve (AUC) of the receiver operating characteristics (ROC) curve.
Given a dataset, a class was chosen as the seen class for training, and all the classes were used for testing.
There exist several testing benchmarks for novelty detection, such as MNIST, COIL100 Nene et al.
and
CIFAR-10.
The state-of-the-art method Perera et al.
(2019) achieves high performance in AUC on
MNIST and COIL100 (AUC is larger than 0.97).
However, for CIFAR-10, Perera et al.
(2019) only [Figure] [Table] achieves 0.656.
Thus, we chose the challenging dataset, CIFAR-10, as the benchmark to evaluate our method.
The detailed network architecture can be found in Appendix E.
Because VAE+DSGAN can be considered as a ﬁne tuning VAE Kingma & Welling (2014a), we ﬁrst illustrate the key difference between the VAE and VAE+DSGAN, as shown in Fig. 8.
The seen class, which is at the bottom of the images, is a car.
Other rows are the images from the unseen classes.
One can see that the reconstructed images are reasonably good even for the unseen class in the VAE.
By contrast, our method enforces the reconstructed images of the unseen classes to be blurred while still preserving the reconstruction quality of the seen class.
Thus, our method achieves a relatively larger gap, in terms of the reconstruction error between the seen data and unseen data, than the VAE.
In Table 3, we compare the proposed method with several methods, including the VAE Kingma &
Welling (2014a), AND Abati et al.
, DSVDD Ruff et al.
(2018), and OCGAN Perera et al.
(2019), in terms of the AUC value.
One can see that in most cases, our method almost outperforms the VAE.
Furthermore, the mean of the AUC values of our method also is larger than those of the state-of-the-art methods.
It is worth mentioning that in addition to the VAE, the DSGAN has potential of being combined with other AE-based methods.
R ELATED W ORKS ABOUT U NSEEN D ATA G ENERATION
Yu et al.
(2017) proposed a method to generate samples of unseen classes in a unsupervised manner via an adversarial learning strategy.
However, it requires solving an optimization problem for each sample, which certainly leads to a high computation cost.
By contrast, the DSGAN has the capability to create inﬁnite diverse unseen samples.
Hou et al.
(2018) presented a new GAN architecture that could learn two distributions of unseen data from a part of seen data and the unlabeled data.
However, the unlabeled data must be a mixture of seen and unseen samples; the DSGAN does not require any unseen data.
Kliger & Fleishman (2018) also applied GAN in novelty detection.
Their objective was to learn a generator whose distribution is a mixture of novelty data distribution and training data distribution.
To this end, they used feature matching (FM) to train the generator and expected p g to learn the mixture of distributions.
However, the ultimate goal of FM is still to learn p g = p d ; therefore, their method might fail when GAN learns well.
Dai et al.
(2017) aimed to generate complementary samples (or out-of-distribution samples), but assumed that the in-distribution could be estimated by a pre-trained model, such as PixelCNN++, which might be difﬁcult and expensive to train.
Lee et al.
(2018) used a simple classiﬁer to replace the role of PixelCNN++ in Dai et al.
(2017) so that the training was comparatively much easier and more suitable.
Nevertheless, their method only focused on generating unseen data surrounding the low-density area of seen data.
In comparison, the DSGAN has more ﬂexibility to generate different types of unseen data ( e.g., a linear combination of seen data, as described in Sec. 5.2).
In addition, their method needs the label information of the data, whereas our method is fully unsupervised.
C ONCLUSIONS
We propose the DSGAN, which can produce any unseen data based on the assumption that the density of the unseen data distribution is the difference between the densities of any two distributions.
The
DSGAN is useful in an environment when the samples from the unseen data distribution are more difﬁcult to collect than those from the two known distributions.
Empirical and theoretical results are provided to validate the effectiveness of the DSGAN.
Finally, because the DSGAN is developed based on GAN, it is easy to apply any improved versions of GAN to the DSGAN.  A CKNOWLEDGEMENT
This work was partially supported by grants MOST 107-2221-E-001-015-MY2 and MOST 108-2634F-007-010 from Ministry of Science and Technology, Taiwan, ROC.
A PPENDIX A F LOWCHART AND A LGORITHM OF DSGAN
for number of training iterations do 02.
for k steps do 03.
Sample minibatch of m noise samples z (1) , ..., z ( m ) from p g ( z ) .
Sample minibatch of m samples x (1) d , ..., x ( m )
[Equation]
from p d ( x ) .
Sample minibatch of m samples x (1)¯
[Equation]
[Equation] [Equation]
( m )
[Equation]
d from p ¯ d ( x ) .
Update the discriminator by ascending its stochastic gradient:
[Equation]
(cid:17)(cid:17)(cid:35) 07.
end for 08.
Sample minibatch of m noise samples z (1) , ..., z ( m ) from p g ( z ) .
Update the generator by descending its stochastic gradient:
[Equation]
end for
B T RICKS FOR S TABLE T RAINING
We provide a trick to stabilize the training procedure by reformulating the objective function.
Speciﬁcally, V ( G, D ) in (2) is reformulated as:
[Equation]
Instead of sampling a mini-batch of m samples from p z and p d in Algorithm 1, (1 − α ) m and αm samples from both distributions are required, respectively.
The computation cost in training can be reduced due to fewer samples.
Furthermore, although (9) is equivalent to (2) in theory, we ﬁnd that the training using (9) achieves better performance than using (2) via empirical validation in Table 4.
We conjecture that the equivalence between (9) and (2) is based on the linearity of expectation, but mini-batch stochastic gradient descent in practical training may lead to the different outcomes.
[Table]
C P ROOF OF T HEOREM
In this section, we show Theorem 1.
This proof includes two parts: the ﬁrst part shows that the objective function is equivalent to minimizing the Jensen–Shannon divergence in the mixture distribution ( p d and p g ) and p ¯ d if G and
D are assigned sufﬁcient capacity; the second part shows that by choosing an appropriate α , the support set of p g belongs to the difference set between p ¯ d and p d , so that the samples from p g are unseen from the p d perspective.
For the ﬁrst part, we show the optimal discriminator given G , and then show that minimizing V ( G, D ) via G , given the optimal discriminator, is equivalent to minimizing the Jensen–Shannon divergence between (1 − α ) p g
+ αp d and p ¯ d .
Proposition 1.
If G is ﬁxed, the optimal discriminator, D , is
[Equation]
Proof.
Given any generator G , the training criterion for the discriminator D is to maximize the quantity V ( G, D ) :
[Equation]
For any ( a, b ) ∈ R 2 \{ 0 , 0 } , the function a log ( y )
+ b log (1 − y ) achieves its maximum in [0 , 1] at y
= aa + b .
The discriminator only needs to be deﬁned within Supp( p ¯ d ) (cid:83)
Supp( p d ) (cid:83) Supp( p g ) .
We complete this proof.
Moreover,
D can be considered to discriminate between samples from p ¯ d and ((1 − α ) p g ( x ) + αp d ( x )) .
By replacing the optimal discriminator in V ( G, D ) , we trivially obtain
[Equation]
Actually, the results thus far yield the optimal solution of D given G is ﬁxed in (1).
Now, the next step is to determine the optimal G with D ∗ G as ﬁxed.
Theorem 2.
The global minimum of C ( G ) is achieved if and only if (1 − α )
p g ( x ) + αp d ( x ) =
p ¯
d ( x ) for all x .
Then, C ( G ) achieves the value, − log
[Equation]
Proof.
We start from
[Equation]
where p ∗ ( x ) =
(1 − α ) p g ( x ) + αp d ( x ) , KL is the Kullback-Leibler divergence and JSD is the
Jensen-Shannon divergence.
The JSD returns the minimal value, which is 0 , iff both distributions are the same, namely p ¯ d = (1 − α )
p g + αp d .
Because p g ( x ) ’s are always non-negative, it should be noted both distributions are the only d ( p ¯ d ( all
’s We complete this proof. Note that (1 − α )
p g ( x ) + αp d ( x ) =
p ¯ d ( x ) may not hold if αp d ( x ) > p ¯ d ( x ) .
However, the DSGAN still works based on two facts: i) given D , V ( G, D ) is a convex function in p g and ii) because (cid:90) x
p g ( x ) dx = 1 , the set collecting all the feasible solutions of p g is convex.
Thus, there always exists a global minimum of V ( G, D ) given D , but it may not be − log(4) .
Now, we go back to prove Theorem 1.
We show that the support set of p g is contained within the differences in the support sets of p ¯ d and p d while achieving the global minimum such that we can generate the desired p g by designing an appropriate p ¯ d .
Proof.
Recall that
[Equation]
S ( p g ; x ) is used to simplify the notations inside the integral.
For any x , S ( p g ; x ) in p g ( x ) is nonincreasing and S ( p g ; x ) ≤ 0 always holds.
Speciﬁcally, S ( p g ; x ) is decreasing along the increase of p g ( x ) if p ¯ d ( x ) > 0 ; S ( p g ; x ) attains the maximum value, zero, for any p g ( x ) if p ¯ d ( x ) = 0 .
Since
DSGAN aims to minimize C ( G ) with the constraint (cid:90) x
p g ( d ) dx = 1 , the solution attaining the global minima must satisfy p g ( x ) =
if p ¯ d ( x ) = 0 ; otherwise, there exists another solution with smaller value of C ( G ) .
Thus, Supp ( p g ) ⊆
Supp ( p ¯ d ) .
Furthermore, T ( p g ; x ) =
[Equation]
p
[Equation]
+
[Equation] [Equation] [Equation] ) (cid:19) , which is expected to be as small as possible to minimize C ( G ) , is increasing on p g ( x ) and converges to 0.
Then, we show that T ( p g ; x ) for x ∈ Supp( p ¯ d ) (cid:84) Supp( p d ) is always larger than that for x ∈ Supp( p ¯ d ) − Supp( p d ) for all p g .
Speciﬁcally,
When x ∈ Supp( p ¯ d ) (cid:84) Supp( p d ) , T ( p g ; x ) ≥ log 12 always holds due to the assumption of αp d ( x ) ≥ p ¯ d ( x ) .
When x ∈ Supp( p ¯ d ) − Supp( p d ) , T ( p g ; x ) < log 12 for all p g ( x ) ’s satisfying (1 − α ) p g ( x ) ≤ p ¯ d ( x ) .
Thus, the minimizer prefers p g ( x ) > 0 for x ∈ Supp( p ¯ d ) − Supp( p d ) and (1 − α ) p g ( x ) ≤ p ¯ d ( x ) .
We check whether there exists a solution p g such that (1 − α ) p g ( x ) ≤ p ¯ d ( x ) and (cid:90) x ∈ Supp( p ¯ d ) − Supp( p d ) p g ( d ) dx = 1 , implying p g ( x ) = 0 for x ∈ Supp( p ¯ d ) (cid:84)
Supp( p d ) .
Based on the following expression,
[Equation]
the last inequality implies that there must exist a feasible solution.
We complete this proof.
Another concern is the convergence of Algorithm 1.
Proposition 2.
The discriminator reaches its optimal value given G in Algorithm 1, and p g is updated by minimizing
[Equation]
If G and D have sufﬁcient capacities, then p g converges to argmin p
[Equation]
Proof.
Consider V ( G, D )
=
U ( p g , D ) as a function of p g .
By the proof idea of Theorem 2 in
Goodfellow et al.
(2014), if f ( x ) =
sup α ∈A f α ( x ) and f α ( x ) is convex in x for every α , then ∂f β ( x ) ∈ ∂f if β = argsup α ∈A f α ( x ) .
In other words, if sup D V ( G, D ) is convex in p g , the subderivatives of sup D V ( G, D ) includes the derivative of the function at the point, where the maximum is attained, implying the convergence with sufﬁciently small updates of p g .
We complete this proof.
D E XPERIMENTAL D ETAILS FOR S EMI -S UPERVISED L EARNING D.0. D ATASETS : MNIST, SVHN, AND CIFAR-
For evaluating the semi-supervised learning task, we used 60000 / 73257 / 50000 samples and 10000 / 26032 / 10000 samples from the MNIST/
SVHN/ CIFAR-10 datasets for the training and testing, respectively.
Under the semi-supervised setting, we randomly chose 100 / 1000 / 4000 samples from the training samples, which are the MNIST/
SVHN/ CIFAR-10 labeled datasets, and the amounts of the labeled data for all the classes are equal.
Furthermore, our criterion to determine the hyperparameters is introduced in Appendix D.1, and the network architectures are described in
Appendix D.2.
We performed testing with 10 / 5 / 5 runs on
MNIST/ SVHN/ CIFAR-10 based on the selected hyperparameters, and randomly selected the labeled dataset.
The results were recorded as the mean and standard deviation of the number of errors from each run.
D. H YPERPARAMETERS
The hyperparameters were chosen to make our generated samples consistent with the assumptions in (7) and (8).
However, in practice, if we make all the samples produced by the generator following the assumption in (8), then the generated distribution is not close to the true distribution, even a large margin between them exists, which is not what we desire.
So, in our experiments, we make a concession that the percentage of generated samples, which accords with the assumption, is around 90% .
To meet this objective, we tune the hyperparameters.
Table 5 shows our setting of hyperparameters, where β is deﬁned in (8).
[Table]
D. A RCHITECTURE
In order to fairly compare with other methods, our generators and classiﬁers for MNIST, SVHN, and CIFAR-10 are same as in Salimans et al.
and Dai et al.
However, different from previous works that have only a generator and a discriminator, we design an additional discriminator in the feature space, and its architecture is similar across all datasets with only the difference in the input dimensions.
Following Dai et al.
(2017), we also deﬁne the feature space as the input space of the output layer of discriminators.
Compared to SVHN and CIFAR-10, MNIST is a simple dataset as it is only composed of fully connected layers.
Batch normalization (BN) or weight normalization (WN) is used in every layer to stable training.
Moreover, Gaussian noise is added before each layer in the classiﬁer, as proposed in Rasmus et al.
We ﬁnd that the added Gaussian noise exhibits a positive effect for semisupervised learning.
The architecture is shown in Table 6.
Table 7 and Table 8 are models for SVHN and CIFAR-10, respectively, and these models are almost the same except for some implicit differences, e.g. , the number of convolutional ﬁlters and types of dropout.
In these tables, given a dropping rate, “Dropout” denotes a normal dropout in that the elements of input tensor are randomly set to zero while Dropout2d is a dropout only applied on the channels to randomly zero all the elements.
[Table]
Furthermore, the training procedure alternates between k steps of optimizing D and one step of optimizing G .
We ﬁnd that k in Algorithm 1 is a key role in the problem of mode collapse for different applications.
For semi-supervised learning, we set k = 1 for all datasets
[Table]
E E XPERIMENTAL D ETAILS FOR N OVELTY D ETECTION
The architecture of GAN and VAE are depicted in Table 9 and 10, respectively.
In the experiment
[Table]
we ﬁrst trained the VAE for 500 epochs and then we trained DSGAN forepochs with m = 1 . 5 and w = 0 . 5 . Third
[Table] we ﬁxed the encoder and tuned the decoder with both positive and negative samples (generated by DSGAN) for 600 epochs.
[Table] [Table]
[Table]
F A BLATION S TUDY ON D IFFERENT α V ALUES FOR S EMI - SUPERVISED L EARNING
Fig.
7 shows how different α values inﬂuence DSGAN.
The optimal α for DSGAN to generate “unseen” data depends on p ¯ d and p d .
According to Fig.
7, we can ﬁgure out that DSGAN is prone to generating unseen data under a larger α .
Recall that Theorem 1 illustrates α should be expected to be as large as possible if both network G and D have inﬁnite capacity.
Though the networks never have the inﬁnite capacity in real applications, a general rule is to pick a large α and force the complement data to be far from p d , which is similar to the results in Sec. 5.1.
[Table]
Here, we conduct the experiments on different α under semi-supervised learning settings.
From
Sec. 4.1 and 5.2, badGAN already shows that, if the desired unseen data can be generated, then the classiﬁer will put the correct decision boundary in the low-density area
[Table]
In Table
[Table]
we demonstrate the classiﬁcation results on α = 0 .
5 and α = 0 .
[Table]
respectively.
We can observe that the results generated at α = 0 .
8 is better than those generated at α = 0 . 5 , meeting the above discussion.
From our empirical observations, DSGAN is prone to generating unseen data at α = 0 . 8 , leading a better classiﬁer.
[Table]
G S AMPLE Q UALITY OF DSGAN ON C ELEB A
We show one more experiment on CelebA (Liu et al.
to demonstrate DSGAN can work well even for complicated images.
In this experiment, we generate the color images of size 64 × 64 .
Similar to our 1/7 experiments on the MNIST dataset, we let p ¯ d be the distribution of face images with glasses and without glasses and let p d be the distribution of images without glasses.
We validate
DSGAN with α = 0 .
5 and α = 0 .
8 , respectively.
For α = 0 .
5 , we sample 10000 images with glasses and 10000 images without glasses from CelebA. When α is 0.8, we sample 40000 instead ofimages without glasses.
We also train GAN to verify the generated image quality of DSGAN.
For fair comparison, GAN is trained under two kinds of settings.
The ﬁrst one is that GAN is only trained with the images with glasses.
Second, it is pretrained with all images, and is ﬁnetuned with the images with glasses, namely transferring GANs in Wang et al.
It should be noted that transferring GAN uses the same amount of training data as DSGAN and serves as a stronger baseline than GAN under the ﬁrst setting.
Fréchet Inception Distance (FID) (Heusel et al.
(2017)) is used to evaluate the quality of generated images.
FID calculates the Wasserstein-2 distance between generated images and real images (images with glasses) in the feature space of Inception-v3 network (Szegedy et al.
We train both networks for 600 epochs, and use WGAN-GP as the backbone for both GAN and DSGAN.
In addition, transferring GANs are pretrained for 500 epochs, then being ﬁnetuned for 600 epochs.
Fig.
10 and Table 12 show generated images and FID for all methods, respectively.
We can see that our DSGAN can generate images with glasses from the given p d and p ¯ d , and the FID of DSGAN are comparable to that of GAN.
The experiment validates that DSGAN still works well to create complement data for complicate images.
[Figure] [Figure]
[Table]
FID 22.
[Table]