Improving Model Compatibility of Generative Adversarial Networks by Boundary Calibration
Abstract
Generative Adversarial Networks (GANs) is a powerful family of models that learn an underlying distribution to generate synthetic data.
Many existing studies of GANs focus on improving the realness of the generated image data for visual applications, and few of them concern about improving the quality of the generated data for training other classiﬁers—a task known as the model compatibility problem.
Literature also show that some GANs often prefer generating ‘easier’ synthetic data that are far from the boundaries of the classiﬁers, and refrain from generating near-boundary data, which are known to play an important roles in training the classiﬁers.
To improve GAN in terms of model compatibility, we propose Boundary-Calibration GANs (BCGANs), which leverage the boundary information from a set of pre-trained classiﬁers using the original data.
In particular, we introduce an auxiliary Boundary-Calibration loss (BC-loss) into the generator of GAN to match the statistics between the posterior distributions of original data and generated data with respect to the boundaries of the pre-trained classiﬁers.
The
BC-loss is provably unbiased and can be easily coupled with different GAN variants to improve their model compatibility.
Experimental results demonstrate that
BCGANs not only generate realistic images like original GANs but also achieves superior model compatibility than the original GANs.
Introduction
The success of machine learning relies on not only the advances of different models (e.g. deep learning) but also data with sufﬁcient quality and quantity.
Nowadays, companies spend tremendous efforts and expense collecting data to build their products.
To better solve complicated real-world problems with public or third-party machine learning experts, many companies now needs release some data sets for competitions (e g. Kaggle) or proof-of-concept purposes.
However, considering the costs of collecting data, companies may not be willing to release the dataset if possible.
As a result, a technique which can generate synthetic data with properties similar to the original data is in demand.
To be speciﬁc, we are looking for generating a dataset with the property that machine learning models trained on the generated dataset can exhibit similar performance to ones trained on the original data.
This property is called model compatibility [24] or machine learning efﬁcacy [32].
The organizations can share the generated data with high model compatibility to the public and enjoy the solution derived from it without leaking the real dataset.
When it comes to data generation, generative adversarial networks (GANs; 5) is a popular family of generative algorithms because of its impressive performance on generating realistic images [12].
In
GANs, the generator is trained via minimizing a neural network (discriminator) deﬁned probability divergence [5, 1, 22].
In addition to image generation, GANs are also widely used in other applications, such as style transfer [11, 34, 13] and image processing [25, 16, 2], and generating different types of data, including time series [20, 3], text [33, 26], point clouds [18], voxels [31] and tabular data [24, 32].
Although GANs are versatile as mentioned above, most of their development focus on the metrics such as quality and diversity of the data [29, 9, 19].
Generating high model compatibility data via GANs is still under explored.
The pioneering work [32] ﬁrst shows that data generated from conditional GANs [21] enjoys better model compatibility than VAEs [14].
It shows a promising potential of generating tabular data by GANs and has been used in generating privacy-sensitive data such as clinical data [27] and insurance records [15].
In this work we wonder that whether we can further improve the model compatibility by considering the boundary between classes, which can be approximated by the models trained on the original data.
For example, Wasserstein GAN (WGAN; 1) performs a mean-matching between the distribution of real data and generated data.
However, only mean-matching is sometimes not enough to learn the whole distribution especially for those boundary cases.
Apparently, if a GAN knows the boundary between different classes, it may be able to generate instances which are close to the boundary with correct labels.
These boundary points will guide a classiﬁer to learn the correct decision boundary.
In this work, we try to improve GANs with regards to model compatibility in classiﬁcation problems.
We use a set of pre-trained classiﬁers to obtain multiple decision boundaries.
Then use an auxiliary loss function called Boundary-Calibration loss (BC-loss) to calibrate the generating distribution according to the decision boundaries of these pre-trained classiﬁers.
The main contributions of this work are:
• In Section 2, we propose a way to evaluate model compatibility in classiﬁcation problems.
We consider a variety of machine learning algorithms and average the performance to obtain a comprehensive metric.
In Section 4, we propose a loss function called Boundary-Calibration loss (BC-loss) which helps typical GANs to learn a distribution with better model compatibility.
The loss considers the decision boundaries of pre-trained classiﬁers and minimizes the maximum mean discrepancy (MMD; 6) between the original dataset and the generated dataset.
In addition, we show that optimizing the BC-loss would not change the optimal solution of the original GAN, but it reduces the feasible set to ensure the model compatibility.
• In Section 5, we demonstrate how BC-loss affects the boundary of the generated data with a two-dimensional toy dataset.
We also show that the BC-loss improves model compatibility of the generated data with different types of classiﬁers and a variety of datasets.
Finally, we inspect the feature selection results to examine how the interpretation of machine learning models may be effected.
Last, in Section 3, we discuss some works which are similar to our work and describe how does our work differ from them.
Model compatibility in classiﬁcation
In this work, we focus on generating data for fully-supervised classiﬁcation learning.
Given a dataset D = { ( x i , y i ) } ni =1 , where x i ∈ X represents features of an instance, y i =
f
( x i ) ∈ Y represents the corresponding label of x i according f : X → Y , and ( x i , y i ) ∼ P D , a learning algorithm
A : ( X , Y ) n
→ H learns a hypothesis h ∈ H to approximate the mapping function, i.e. A ( D )
=
h
≈ f .
Our goal is to obtain a generator G which generates a synthetic dataset
D (cid:48)
= { ( x (cid:48)
j , y (cid:48)
j ) } mj =1 such that A ( D (cid:48) )
=
h (cid:48)
≈ h .
We call this property model compatibility as proposed in Park et al.
To measure the model compatibility of a generated dataset quantitatively, we consider the performance of a classiﬁer trained on the generated dataset comparing to the one trained on the real dataset.
We evaluate the accuracy on a separate test dataset to indicate the performance of a given classiﬁer.
In addition, we calculate relative accuracy by scaling the test accuracy of the classiﬁer trained on the generated dataset by the accuracy of the classiﬁer trained on the real dataset.
The relative accuracy allows us to average the results from different machine learning algorithms more fairly.
The ﬁnal evaluation is :
[Equation]
where A is a set of learning algorithms , D ( t ) =
{ ( x ( t ) i , y ( t ) i ) }
Ni =1 is the test dataset, and acc ( h, D ( t ) ) is the accuracy of hypothesis h on test data D ( t ) .
We can determine A as a wide variety of learning algorithms to make the metric provide a more comprehensive measurement of model compatibility.
Related work
Research about generating data for classiﬁcation can be divided into two categories: formulation and architecture.
For formulation, Conditional GAN (CGAN; 21 ) is an intuitive way to generate instances with corresponding labels.
We can learn the distribution of labels by counting and sample the instances from CGAN conditionally.
Auxiliary Classiﬁer GAN (ACGAN; 23) is considered as a better way for conditional generation.
It uses an auxiliary classiﬁer to provide information about the boundary between each classes.
However, ACGAN has been proved that the objective is biased so it tends to generate data with lower entropy for the auxiliary classiﬁer [30].
Thus, the lose of instances near the decision boundary may worsen the model compatibility.
In this work, we use CGAN along with the proposed BC-loss to generate data with model compatibility.
On the other hand, the other line of research focuses on generating data with different network architecture or data processing procedure.
Recent works that also consider model compatibility are
Table GAN [24] and Tabular GAN [32].
Table GAN focuses on the privacy of generated data and thus their is a trade off between privacy and model compatibility.
To achieve privacy preserving, they do not improve the model compatibility compared to the original GAN.
On the other hand, Tabular
GAN puts emphasis on increasing model compatibility of generated data.
They propose a framework with a more complicated data processing procedure and use LSTM to better parameterize the target distribution.
In contrast to these works, our work focus on the formulation of GANs and can be applied to most variants of GANs, including Table-GAN and Tabular GAN.
Moreover, while these former works only focus on tabular data, our BC-loss is applicable to generate image datasets as well.
Some GAN variants are named similarly to our work but they pay attention to different problems.
For example, the boundary described in boundary-seeking GAN [10] means the decision boundary of the discriminator rather than the decision boundary for the supervised labels.
To the best of our knowledge, we are the ﬁrst work trying to improve model compatibility by modifying the formulation of GANs.
Boundary-Calibration GAN
To achieve better model compatibility of GAN, we propose an auxiliary GAN loss which we call boundary-calibration loss (or BC-loss).
We assume that we have a set of pre-trained classiﬁers which are well-trained on the original dataset.
The BC-loss helps GANs to calibrate the distribution with respect to the distribution of decision values predicted by pre-trained classiﬁers.
The calibration leads to more accurate data generation near the decision boundary and thus enabling a machine learning algorithm to learn a similar hypothesis to one that learns from the original dataset.
To ease the complexity of learning to generate ( x , y ) jointly, we infer P ( y ) by counting the proportion of each class in the original dataset and train a conditional generator G such that G ( z , y ) ∼ P X| y , where
P X| y is the conditional data distribution and z ∼ P Z is the initial randomness such as Gaussian distribution.
Therefore we can generate ( x , y ) by sampling y ∼ P ( y ) and G ( z , y ) .
4.1 Boundary Calibration
Given a pre-trained classiﬁer C , we hope the generated dataset will adopt the same statistics as the original dataset while considering the decision boundary of C .
To include the information about the boundary, we calculate posterior P C
( y | x i ) from the decision values predicted by the classiﬁer.
In practice, we can apply a softmax function to the outputs of a classiﬁer to obtain the posterior.
The posterior provides information of an instance from the classiﬁer’s aspect.
Therefore, given the real dataset X = { x 1 , x 2 , ..., x n } , we can obtain a set of posterior vector C ( X ) =
( P C ( y | x 1 ) , P C ( y | x 2 ) , ..., P C ( y | x n )) .
To generate data X (cid:48) with the same distribution of posteriors to the boundary, we match the statistics of C ( X ) and C ( X (cid:48) ) by optimizing a distance M :
[Equation]
Here M can be any distance metric which measures the distance between two sets of samples.
In statistics, distinguishing whether two sets of samples are from the same distribution is called TwoSample Test .
A classical solution to two-sample test is kernel maximum mean discrepancy (MMD; 6).
The idea is to compare the statistics between the two sets of samples.
If the statistics are similar then these two sets might be sampled from the same distribution.
Given two sets of samples X = { x i } ni =
and Y =
{ y j } nj =1 , an unbiased estimator of MMD with kernel k is deﬁned as:
[Equation]
In practice, we use Gaussian kernel
[Equation]
) = exp( (cid:107) x − x (cid:48) (cid:107) 2 ) in MMD since Gaussian kernel is a characteristic kernel which ensures that the distance is zero if and only if the two distributions are the same [6].
The BC-loss can be applied in generator of any GAN variants to improve the model compatibility.
In addition, to better ﬁt the real unknown boundary, we can use multiple classiﬁers to calibrate the distributions from different aspects.
As a result, for a loss function of generator L G , we can modify the loss to be:
[Equation]
where Z is a set of noises, C is a set of pre-trained classiﬁers and λ is a hyper-parameter to control the weight of BC-loss.
4.2 Analysis of optimal solution
Next we prove that adding our proposed BC-loss would not change the optimal solution of the original objective.
Here we assume the loss of the generator L G achieves its optimal value in the its GAN objectives L G if and only if the distribution of G ( z , y ) recovers P X| y for all y ∈ Y , which holds for the vanilla GAN [5] and most of other GAN variants.
Theorem 1 (Gretton et al. 6) .
Given a kernel k , if k is a characteristic kernel, then M k ( P, Q ) =
⇐⇒ P = Q .
Theorem 2 (Equivalence of optimal solution) .
G is an optimal solution of L G ⇐⇒ G is an optimal solution of ˆ L G
Proof.
( ⇒ )
According to the assumption, G is an optimal solution of L G implies G ( Z , y ) recovers
P X| y for all y ∈ Y .
Therefore, P C ( X ) =
P C ( G ( Z , Y )) and M k ( P C ( X ) , P C ( G ( Z , Y )) )
= 0 by Theorem 1.
Now ˆ L G =
L G + 0 =
L G and G is an optimal solution of L G , so G is also an optimal solution of ˆ
L G .
( ⇐ ) Since L BC ≥ 0 , we have ˆ L G =
L G + L BC ≥ L G .
From above, we know ˆ L G ( G ) =
if G = P X .
Thus, for an optimal solution G ∗ , 0 ≥ ˆ L G ( G ∗ ) ≥ L G ( G ∗ ) ≥ 0 , which implies ˆ L G ( G ∗ ) =
L G ( G ∗ ) = 0 .
Therefore, G ∗ is also an optimal solution of L G .
The proof shows that the proposed BC-loss does not change the optimal solution of the original optimization problem.
However, we can consider BC-loss as a Lagrangian constraint which restricts the solution to a subspace where the generator owns higher model compatibility .
[Figure]
4.3 Comparison to MMD GAN
MMD GAN [17] is a variant of GAN where the generator tries to minimize the MMD between generated data and original data and the discriminator learns a kernel which maximizes the MMD.
Though the formulation of MMD GAN and BC-loss are similar, they still do not conﬂict because
MMD GAN do not known the information about the classiﬁer and the objective of MMD GAN would not lead the discriminator to a classiﬁer.
Therefore, BC-loss may still improve MMD GAN by guiding the generator to not generate points across the boundary.
To understand the improvement in
MMD GAN from BC-loss, we use MMD GAN as one of the baselines in our experiments.
Experiments
In this section, we use a toy dataset to illustrate how the proposed method improves the model compatibility.
To be more realistic, we provide more comprehensive results for four different realworld dataset from UCI dataset repository [4]: Adult, Connect-4, Covertype and Sensorless.
We then show our method is also applicable in image dataset: MNIST and Cifar10 without losing the image quality.
In addition, we investigate the results of feature selections on the generated dataset to see whether the generated data can preserve the interpretation of machine learning models.
5.1 Experimental settings Evaluation
In this work, we focus on model compatibility of generated datasets.
We use a wide variety of machine learning algorithms including linear SVM, decision tree (DT), random forest (RF), and multi-layer perception (MLP) to evaluate the model compatibility.
As described in Section 2, we evaluate the relative accuracy for each type of machine learning model, where the relative accuracy is calculated by dividing the accuracy of classiﬁer trained on generated data to the accuracy of classiﬁer trained on original data.
Compared methods
We take Wasserstein GAN (WGAN) and MMD GAN as our baselines to evaluate the effectiveness of the proposed boundary-calibration technique.
We denotes their counterparts with BC-loss as
BWGAN and BMMDGAN respectively.
All of the methods use gradient penalty to enforce the
Lipschitz constraint on the discriminator [7, 17].
To achieve conditional data generation as described in Section 4, we add an embedding layer to learn the embedding vector for each class.
The embedding vector is concatenated as additional input features for both generators and discriminators on UCI datasets.
For image datasets, the embedding vectors are used as described in [21].
5.2 2D Toy Dataset
We use a 2D toy dataset with two classes to illustrate the results generated by different GAN methods in Figure 1.
Figure 1a shows the distribution of the original training data.
We use these generated
[Table]
data to train a random forest and depict the decision boundary by different background color.
From
Figure 1b, we can see that although ACGAN can make use of the auxiliary classiﬁer during training, it learns a biased distribution that push the generated data away from the boundary.
The large margin between the two clusters brings more uncertainty to the decision boundary and thus leads to worse test accuracy.
In Figure 1c, WGAN approximates the original distribution well in the center part of the two cluster, but do not get a clear boundary between the two classes.
It generates some ambiguous points near the boundary that would confuse the classiﬁer.
Finally, our BWGAN generates points near the boundary more precisely, as shown in Figure 1c.
5.3 UCI dataset
We evaluate our proposed BC-GAN on four datasets from UCI repository.
The attributes of the datasets can be found in Appendix A. Discrete features are processed to one-hot encoding and continuous features are scaled to [0 , 1] .
For each dataset, we train six multi-layer perceptrons with a random split of half of training data as pre-trained classiﬁers.
In these experiments, generators and discriminators are consist of 3 fully-connected hidden layer with 128 units.
A logistic function is applied to the output layer of generators to generate features within 0 and 1.
The weight of BC-loss is set to be λ = 100 for all datasets.
Table 1 summarize the comparison between different methods.
We calculate the relative accuracy of different machine learning models mentioned in Section 5.1 and average the relative accuracy to indicate the model compatibility of generated data for each dataset.
The table shows that the proposed BC-loss improves the accuracy of classiﬁers generally compared to original WGAN and
MMD GAN.
Moreover, ACGAN performs worst on three out of four datasets and exhibit a signiﬁcant deﬁciency though it is proved to have the state-of-the-art generation quality.
This again prove that the biased objective of ACGAN worsen the model compatibility seriously.
The breakdown results and real accuracy are provided in Appendix B.
To further investigate the advantage of boundary-calibration, we visualize the generated results of
Sensorless in Figure 2.
We train a fully-connected neural network with a 2-units hidden layer before the output layer to project the generated samples to a 2-dimensional embedding space.
The projection classiﬁer is well-trained and achieves over 99% testing accuracy so we can use it to determine whether a sample is generated with incorrect label.
The ﬁgure shows that there are less mislabeled data generated by BWGAN, especially at the center and the bottom-left region.
The fact indicates that boundary-calibration helps GANs generate labeled data more accurately, which may lead to the improvement of classiﬁcation accuracy.
5.4 Interpretability
In addition to accuracy, it is also important that the model trained on generated data should give us the same interpretation of a model trained on the original data.
We investigate the interpretability by two common feature selection techniques.
First, we train two random forests on the generated and original dataset respectively.
Each random forest can provide the importances of the features.
We evaluate the consistency of interpretation by calculating precision at Kth, which means how many features ranked top-k in random forest trained on original data are in the top-k importance feature of the random forest trained on generated data.
The results are shown in Table 2.
We provide the results of training a classiﬁer on the same original data with a different random seed as REAL for comparison.
The effect of BC-loss is not signiﬁcant in this aspect.
However, the scores of ACGAN drop seriously,
[Figure]
[Table]
which means training a classiﬁer on data generated by ACGAN is somehow dangerous because the meaning of model may be totally different.
Another way to select feature is training a linear model with (cid:96) 1 regularization.
In Table 3 we use linear SVM with (cid:96) 1 regularization to select features.
Then we calculate the F1 score of features selected by classiﬁers trained on generated data to known how similar between the two sets of features selected by classiﬁers trained on original and generated dataset.
The results again shows that using boundary-calibration does not has signiﬁcant effect to feature selection and ACGAN is not proper to generated data for training.
5.5 Image dataset
We further use MNIST and CIFAR-10 dataset to investigate the effectiveness of boundary-calibration on image datasets.
For MNIST, we train six 4-layer convolution neural networks (CNN) with random sampling half of training data as pre-trained classiﬁers, and use the same classiﬁer set in Section 5. to evaluate model compatibility.
For CIFAR-10, we use ResNet56v2 [8] to obtain three pre-trained classiﬁer and evaluate on CNN and ResNet56v2.
In both task, we use DCGAN [28] as network structure in all GANs.
The weight of BC-loss is set to be λ = 1 for these two datasets.
Table 4 and Table 5 show the relative accuracy of classiﬁers trained on generated data.
The proposed
BWGAN still outperforms WGAN with better accuracy in general.
The results generated by WGAN and BWGAN are pictured in Figure 3.
The Inception score and Frechet Inception Distance (FID) for
CIFAR-10 are also provided in the caption of Figure 3.
Though the difference of quality between the images generated from WGAN and BWGAN is not signiﬁcant in visual, the quantitative scores for quality of generated samples of CIFAR-10 are slightly improved.
The results indicate that even though our method seems not improve the image quality, it is still able to improve the model compatibility without losing image quality.
[Table]
[Figure]
Discussion
We introduce an auxiliary loss in GANs which improves the model compatibility of generated dataset.
We prove the new loss is unbiased and is applicable to all variants of GAN to improve model compatibility.
We further demonstrate that our method has clear advantages with a variety of machine learning models trained on generated dataset.
In addition, we investigate the results of feature selection and found that the BC-loss doesn’t effect the interpretation of machine learning models.
While this work only focus on classiﬁcation problem, generating data for regression problem is also worth studying.
We hope our work will open the path for GANs with better model compatibility so that synthetic data can be more useful in practice.
[Table] [Table]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Dataset Information
[Table]
B Detail Result
[Table] [Table]