ST A S Y : S CORE - BASED T ABULAR DATA S YNTHESIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Tabular data synthesis is a long-standing research topic in machine learning.
Many different methods have been proposed over the past decades, ranging from statistical methods to deep generative methods.
However, it has not always been successful due to the complicated nature of real-world tabular data.
In this paper, we present a new model named S core-based Ta bular data Sy nthesis ( STaSy ) and its training strategy based on the paradigm of score-based generative modeling.
Despite the fact that score-based generative models have resolved many issues in generative models, there still exists room for improvement in tabular data synthesis.
Our proposed training strategy includes a self-paced learning technique and a fine-tuning strategy, which further increases the sampling quality and diversity by stabilizing the denoising score matching training.
Furthermore, we also conduct rigorous experimental studies in terms of the generative task trilemma: sampling quality, diversity, and time.
In our experiments with 15 benchmark tabular datasets andbaselines, our method outperforms existing methods in terms of task-dependant evaluations and diversity.
I NTRODUCTION
[Table]
Tabular data synthesis is of non-trivial importance in real-world applications for various reasons: protecting the privacy of original tabular data by releasing fake tabular data (Park et al., 2018; Lee et al., 2021), augmenting the original tabular data with fake data for better training machine learning models (Chawla et al., 2002; Han et al., 2005;
He et al., 2008; Kim et al., 2022), and so on.
However, it is well-known that tabular data frequently has such peculiar characteristics that deep generative models are not able to synthesize all possible details of the original tabular data (Park et al., 2018; Xu et al., 2019) — given a set of columns in tabular data, columns typically follow unpredictable (multi-modal) distributions and therefore, it is hard to model their joint probability.
A couple of recent methods, however, showed remarkable successes (with some failure cases) in synthesizing fake tabular data, such as CTGAN (Xu et al., 2019), TVAE (Xu et al., 2019),
IT-GAN (Lee et al., 2021), and OCT-GAN (Kim et al., 2021).
In addition, a recent generative model paradigm, called score-based generative modeling (SGMs), successfully resolves the two problems of the generative learning trilemma (Xiao et al., 2021), i.e., score-based generative models provide high sampling quality and diversity, although their training/sampling time is relatively longer than other deep generative models.
In this paper, we adopt a score-based generative modeling paradigm and design a S core-based Ta bular data Sy nthesis ( STaSy ) method.
Our model designs significantly outperform all existing baselines in terms of the sampling quality and diversity (cf.
Naïve-STaSy and STaSy in Table 1) — Naïve-STaSy is a naive conversion of SGMs toward tabular data, and STaSy additionally uses our proposed self-paced learning and fine-tuning methods.
Figure 1 shows the uneven and long-tailed loss distribution of Naïve-STaSy at the end of its training process. The figure implies the training of Naïve-STaSy by the denoising score matching failed to learn the score values of some records. This may let the model be (partially) underfitted to training data.
In contrast, STaSy with our two proposed training methods yields many loss values around the left corner (i.e., close to 0).
[Figure]
score matching loss in Shoppers
In order to alleviate the training difficulty of Naïve-STaSy , we design i) a self-paced learning method, and ii) a fine-tuning approach.
Our proposed self-paced learning technique trains our model from easy to hard records based on their loss values by modifying the objective function. The technique makes the model learn records selectively and eventually.
During this process, the model can be better trained.
In addition, our proposed fine-tuning method, which modestly adjusts the model parameters, can further improve the sampling quality and diversity.
In Table 1, we summarize our experimental results, where we compare our STaSy with other existing tabular data synthesis methods in terms of the sampling quality, diversity, and time.
As shown, our basic model even without our proposed selfpaced learning and fine-tuning, denoted Naïve-STaSy , significantly outperforms all baselines except for runtime.
In summary, our contributions are as follows: i)
We design a score-based generative model for tabular data synthesis.
ii)
We alleviate the training difficulty of the denoising score matching loss by designing a self-paced learning strategy and further enhance the sampling quality and diversity using a proposed fine-tuning method.
STaSy , thus, clearly balances among the generative learning trilemma : sampling quality, diversity, and time.
iii)
Our proposed method outperforms other deep learning methods in all cases by large margins, which we consider a significant advance in the field of tabular data synthesis.
iv)
We evaluate various methods in terms of the generative learning trilemma in a rigorous manner.
R ELATED WORK 2. S CORE - BASED GENERATIVE MODELS
Score-based generative models (SGMs) use a diffusion process defined by the following Itô stochastic differential equation (SDE):
[Equation]
where f ( x , t )
=
f ( t ) x , f and g are drift and diffusion coefficients of x ( t ) , and w is the standard
Wiener process.
Depending on the types of f and g , SGMs can be divided into variance exploding (VE), variance preserving (VP), and sub-variance preserving (sub-VP) models (Song et al., 2021).
The definitions of f and g are in Appendix
A.
The reverse of the diffusion process is a denoising process as follows:
[Equation]
where this reverse SDE is a process of generating samples.
The score function ∇ x log p t ( x ) is approximated by a time-dependent score-based model S θ ( x , t ) , called score network .
In general, following the diffusion process in Equation 1, we can derive x ( t ) at time t ∈ [0 , T ] , where x (0) and x ( T ) means a real and noisy sample, respectively.
The transition probability p ( x ( t ) | x (0)) at time t is easily approximated by this process, and it always follows a Gaussian distribution.
It allows us to collect the gradient of the log transition probability, ∇ x ( t ) log p ( x ( t ) | x (0)) , during the diffusion process.
Therefore, we can train a score network S θ ( x , t ) as follows:
[Equation]
where λ ( t ) is to control the trade-off between the sampling quality and likelihood.
This is called denoising score matching , and θ ∗ solving Equation 3 can accurately solve the reverse SDE in Equation 2 (Vincent, 2011).
After the training process, we can synthesize fake data records with i) the predictor-corrector framework or ii) the probability flow method, a deterministic method based on the ordinary differential equation (ODE) whose marginal distribution is equal to that of Equation 1 (Song et al., 2021).
In particular, the latter enables fast sampling and exact log-probability computation.
2. T ABULAR DATA SYNTHESIS
Many distinct methods exist for tabular data synthesis, which creates realistic synthetic tables depending on the data types.
For example, a recursive table modeling utilizing a Gaussian copula is used to synthesize continuous variables (Patki et al., 2016).
Discrete variables can be generated by Bayesian networks (Zhang et al., 2017; Aviñó et al., 2018) and decision trees (Reiter, 2005).
Several data synthesis methods based on GANs have been presented to generate tabular data in recent years.
RGAN (Esteban et al., 2017) creates continuous time-series healthcare records, whereas
MedGAN (Choi et al., 2017) and corrGAN (Patel et al., 2018) generate discrete records.
EhrGAN (Che et al., 2017) utilizes semi-supervised learning to generate plausible labeled records to supplement limited training data.
PATE-GAN (Jordon et al., 2019) generates synthetic data without jeopardizing the privacy of real data.
TableGAN (Park et al., 2018) employs convolutional neural networks to enhance tabular data synthesis and maximize label column prediction accuracy.
CTGAN and
TVAE (Xu et al., 2019) adopt column-type-specific preprocessing steps to deal with multi-modality in the original dataset distribution.
OCT-GAN (Kim et al., 2021) is a generative model design based on neural ODEs.
SOS (Kim et al., 2022) proposed a style-transfer-based oversampling method for imbalanced tabular data using SGMs, whose main strategy is converting a major sample to a minor sample.
Since its task is not compatible to our task to generate from the scratch, direct comparisons are not possible.
However, we convert our method to an oversampling method following their design guidance and compare with SOS in Appendix B.
2. S ELF - PACED LEARNING
Self-paced learning (SPL) is a training strategy related to curriculum learning to select training records in a meaningful order, inspired by the learning process of humans (Kumar et al., 2010b; Jiang et al., 2014).
It refers to training a model only with a subset of data that has low training losses and gradually expanding to the entire training data.
We denote the training set as D =
{ x i } Ni =1 , where x i is the i -th record.
The model M with parameters θ has a loss l i =
L ( M ( x i , θ )) , where L is the loss function.
A vector v =
[ v i ] Ni =1 , v i ∈ { 0 , 1 } indicates whether x i is easy or not for all i .
SPL aims to learn the model parameter θ
and the selection importance v by minimizing:
[Equation]
where K is a parameter to control the learning pace.
In general, the second term in Equation 4, called a self-paced regularizer, can be customized for a downstream task.
The alternative convex search (ACS)
(Bazaraa et al., 1993) is typically used to solve Equation 4 (Kumar et al., 2010a; Tang et al., 2012).
By alternately optimizing variables while fixing others, we can optimize Equation 4, i.e., update v after fixing θ , and vice versa.
With fixed θ , the global optimum v ∗ =
[ v ∗ i ]
Ni =1 is defined as follows:
[Equation]
When updating v with fixed θ , a record x i with l i < 1 K is regarded as an easy record and will be chosen for training.
Only easy records are used to train the model.
Otherwise, x i is regarded as a hard record and will be unselected.
To involve more records in the training process, K is gradually decreased.
P ROPOSED METHOD
STaSy is an SGM-based method for tabular data synthesis.
STaSy uses SPL to ensure its training stability.
The suggested fine-tuning method takes advantage of a favorable property of SGMs, which is that we can measure the log-probabilities of records.
3. S CORE NETWORK ARCHITECTURE & MISCELLANEOUS DESIGNS
It is known that each column in tabular data typically has complicated distributions, whereas pixel values in image datasets typically follow Gaussian distributions (Xu et al., 2019).
Moreover, tabular synthesis models should learn the joint probability of multiple columns to generate a record, which is one main reason why tabular data synthesis is difficult.
However, one good design point is that the dimensionality of tabular data is typically far less than that of image data, e.g., 784 pixels even in
MNIST, one of the simplest image datasets, vs. 30 columns in Credit .
We found through our preliminary experiments that the SDE in Equations 1 and 2 can well model the joint probability iff its score network, which approximates ∇ x log p t ( x ) , is well trained.
We carefully design our score network for tabular data synthesis considering these points.
Our proposed score network architecture is in Appendix C.
The network consists of residual blocks of FC layers.
Since SGMs were theoretically designed from the idea of perturbing data with an infinite number of noise scales, SGMs typically require large-scale computation, e.g., T = 1 , 000 for images in (Song et al., 2021), as an approximation to the infinite number.
With a large number of steps, the denoising process requires a long time to complete, which is one part of the generative learning trilemma .
However, we found that T = 50 steps in Equation 1 are enough to train a network to approximate the gradient of the log-likelihood, which means that our STaSy naturally has less sampling time than
SGMs for images with T = 1 , 000 steps.
Pre/post-processing of tabular data To handle mixed types of data, which is a challenge in tabular data generation, we pre/post-process columns.
We use the min-max scaler to pre-process numerical columns, and its reverse scaler is used for post-processing after generation.
We also apply one-hot encoding to pre-process categorical columns, and use the softmax function, followed by the rounding function, when generating.
How to generate After sampling a noisy vector z ∼ N ( µ , σ 2 I ) , the reverse SDE can convert z into a fake record.
The prior distribution z ∼ N ( µ , σ 2 I ) varies depending on the type of SDEs:
N ( 0 , σ 2 max I ) for VE, and N ( 0 , I ) for VP and sub-VP.
σ max is a hyperparameter.
In particular, we adopt the probability flow method to solve the reverse SDE, which will be shortly described in Equation 10.
3. S ELF - PACED LEARNING APPROACH
In order to alleviate the training difficulty, we apply a curriculum learning technique for STaSy , more specifically, self-paced learning.
Instead of letting v i ∈ { 0 , 1 } , we use a “soft” record sampling method, i.e., v i ∈ [0 , 1] .
If l i , which is the denoising score matching objective on i -th record, is less than a threshold, we set v i to 1 to ensure that the record is fully involved in training.
At the end of the training, v i must be set to 1 for all i to train the model with the entire data.
The denoising score matching loss for i -th training record x i is defined as follows:
[Equation]
Then, we have the following STaSy objective:
[Equation]
where 0 ≤ v i ≤ 1 for all i , r ( · ) is a self-paced regularizer.
α ∈ [0 , 1] and β ∈ [0 , 1] are variables to define thresholds, which are monotonically increasing as training goes on (see Appendix D for their exact controlling mechanism).
Definition 1.
Let Q ( p ) be a quantile function defined as inf { l ∈ R : p ≤ F ( l ) } , where F is a cumulative distribution function of the denoising score matching objective.
That is, Q ( p ) is the minimum value for which the CDF is greater than or equal to the given probability p .
Theorem 1.
Let the self-paced regularizer r ( v ; α, β ) be defined as follows:
[Equation]
1 Initialize θ /* Train SGM based on our SPL training strategy */ 2 for each mini-batch of records do
Update v with Equation Update θ after fixing v with Equation
Update α and β with the control method in Appendix D /* Fine-tune the trained model using log-probability */ 6 τ i ← log p ( x i ) 7 F ← { x i | log p ( x i ) , where x i ∈ D , is smaller than the average (or median) log-probability . } 8 for each fine-tune epoch do for each x i ∈ F do
Update θ with Equation
F ← { x i | log p ( x i ) <
τ i } 12 return θ where the closed-form optimal solution for v ∗ =
[ v ∗ 1 , v ∗ 2 , . . .
, v ∗ N ] , given fixed θ , is defined as follows — its proof is in Appendix E:
[Equation]
Specifically, records with l i ≤ Q ( α ) are considered easy records and will be selected for training, whereas records with l i ≥ Q ( β ) are considered complicated (or potentially noisy) and will not be selected.
If not both cases, records will be partially selected during training, i.e., v i ∈ [0 , 1] .
α and β are gradually increased to 1 from the initial values α 0 and β 0 , proportionally to training progress to ensure that all data records are involved in training.
As α and β increase, the difficult records are gradually involved in training, and the model also becomes more robust to those difficult cases.
We set α 0 and β 0 in such a way that more than 80% of the training records are included in the learning process from the beginning.
3. F INE - TUNING APPROACH
For solving the reverse SDE process, score-based generative models rely on various numerical approaches.
One of the techniques is the probability flow method in Equation 10 (Song et al., 2021), which uses a deterministic process whose marginal probability is the same as the SDE.
With the approximated score function S θ ( · ) , the probability flow method uses the following neural ordinary differential equation (NODE) based model (Chen et al., 2018):
[Equation]
In our experiments, the probability flow shows better quality than other methods to solve the original reverse SDE, and our default solver is the probability flow (see Section 4.3).
In addition, NODEs facilitate computing the log-probability defined in Equation 10 through the instantaneous change of variables theorem.
Consequently, we can calculate the exact log-probability efficiently with the unbiased Hutchinson’s estimator (Hutchinson, 1989; Grathwohl et al., 2018).
Thus, we propose to fine-tune based on the exact log-probability.
After learning the model parameter θ as described in Section 3.2, we set the sample-wise threshold τ i to log p ( x i ) (cf.
Line 6 of Algorithm 1).
We then prepare the fine-tuning candidate set F (cf.
Lineof Algorithm 1).
After fine-tuning for the samples in F , we update the candidate set (cf.
Lineof Algorithm 1).
Our goal is for achieving a better log-probability than the initial one τ i before the fine-tuning process.
[Table]
TableGAN
[Table]
3. T RAINING ALGORITHM
Algorithm 1 shows the overall training process for our STaSy .
Firstly, we initialize the parameters of the score network θ .
Utilizing ACS, we then train STaSy with the SPL training strategy.
At this step, we iteratively optimize θ and v .
We can obtain an optimal θ by optimizing Equation 7 with fixed v , and the global optimum v is calculated by Equation 9.
We also update α and β in proportion to training progress.
After finishing the main SPL training step, we can generate fake records from the model.
To further improve the trained score network, we retrain our model for every record x i whose log-probabilities are less than its threshold τ i .
At Line 10 of Algorithm 1, one can use the log-probability instead of Equation 6 as a fine-tuning objective.
However, we found that Equationis more effective (see Appendix F).
E XPERIMENTS
We analyze methods in terms of the generative learning trilemma .
We list only aggregated results over datasets in the main paper — details are in Appendix G.
We repeat experiments 5 times.
4. E XPERIMENTAL ENVIRONMENTS
A brief description of our experimental environments is as follows: i)
We use 15 real-world tabular datasets for classification and regression and 7 baseline methods.
To be specific about our models,
Naïve-STaSy is a naive conversion of SGMs without our proposed training strategies, and STaSy is trained with self-paced learning and the fine-tuning method.
ii)
In general, we follow the “train on synthetic, test on real (TSTR)” framework (Esteban et al., 2017; Jordon et al., 2019), which is a widely used evaluation method for tabular data (Xu et al., 2019; Kim et al., 2021; Lee et al., 2021), to evaluate the quality of sampling — in other words, we train various models, including DecisionTree,
AdaBoost, Logistic/Linear Regression, MLP classifier/regressor, RandomForest, and XGBoost, with fake data, validate with original training data, and test them with real test data.
For Identity , we train with real training data, choose the best performing model using the cross-validation, and test with real test data, whose score can be a criterion to evaluate the sampling quality of various generative methods. iii)
We use various metrics to evaluate in various aspects.
For the sampling quality, we mainly use average F1 for classification, and also report AUROC and Weighted-F1.
We use R 2 and RMSE for regression.
For the sampling diversity, we use coverage (Naeem et al., 2020), which was proposed to measure the diversity of generated records.
Full results are in Appendix G.
Detailed environments and hyperparameter settings are in Appendix H and I, respectively.
4. E XPERIMENTAL RESULTS 4.2. S AMPLING QUALITY
Table 2 summarizes the key results on the sampling quality.
We use task-oriented metrics, such as F1, R 2 , and so on, under the TSTR evaluation framework.
MedGAN and VEEGAN , two early
GAN-based methods, show relatively lower test scores than other GAN-based methods, i.e., CTGAN ,
TableGAN , and OCT-GAN .
In general, CTGAN and OCT-GAN show reliable quality among the baselines.
However, our two score-based models always mark the best and the second best quality.
Our methods, Naïve-STaSy and STaSy , significantly outperform all the baselines by large margins.
In particular, our methods perform well in small datasets, e.g., Crowdsource , Obesity , and Robot , while other methods show poor quality, as shown in Table 14 of Appendix G.1.
These multi-class classification datasets have a small number of per class record, e.g., the smallest class in
Crowdsource has 79 records in the training set, which means that our methods are able to capture fine-grained modes from the original data.
Moreover, in Credit , which has a severe class imbalance ratio of 99.7% for class 0 and 0.3% for class 1, more than half of the baselines failed to generate the minority class, showing an F1 score close to 0 in Table 11 of Appendix G.1.
CTGAN and TableGAN also achieve a good F1 score close to that of Identity , but STaSy takes the first place again.
[Table]
As flow-based generative models and SGMs with the probability flow method can calculate the exact logprobability of records, we present the log-probability as another metric for the sampling quality.
Table 3 shows the median of the log-probabilities of testing records, averaged over all datasets.
Since the log-probability is not bounded, we take a median of them to handle the case of outliers.
Our methods, even without the fine-tuning, show a much better log-probability than RNODE , which optimizes the log-probability as its objective.
Moreover, the median log-probability of testing records even improves after the proposed fine-tuning method.
Putting it all together, our proposed score-based generative models, i.e., Naïve-STaSy and STaSy , show reasonable performance in all cases regarding the machine learning efficacy and log-probability.
Furthermore, as shown in Tables 2 and 3, the sampling quality always improves with the proposed training strategies, i.e., the self-paced learning and the fine-tuning, which justifies their efficacy.
4.2. S AMPLING DIVERSITY
[Figure]
STaSy
[Table]
For the quantitative evaluation of the sampling diversity between existing methods and our proposed method, we use the coverage score (Naeem et al., 2020), which is bounded between 0 and 1.
Coverage is the ratio of real records that have at least one fake record in its manifold.
A manifold is a sphere around the sample with radius r , where r is the distance between the sample and the k -th nearest neighborhood.
Table 4 summarizes the averaged coverage of each method.
MedGAN and
VEEGAN show poor coverage scores, close to 0.
This trend is also shown in the t-SNE visualizations in Appendix J.2.
Among the baseline methods, CTGAN performs the best in terms of the sampling quality, whereas it shows relatively inferior coverage performance than others.
In specific, in Robot , STaSy shows a coverage of 0.94, while other three top-performing baselines, CTGAN , TableGAN , and OCT-GAN , show coverage scores less than 0.26 in Table 20 of Appendix G.2.
Figurealso presents the diversity of each fake data by each method qualitatively, which reflects the results of
[Figure]
coverage.
In general, STaSy shows stable performance across the sampling quality and the sampling diversity, outperforming others by large margins.
In Figure 3 (Left and Middle), the fake data by STaSy shows an almost identical distribution to that of real data.
In contrast, OCT-GAN , which was proposed to address the multi-modality issue of tabular data, fails to do it.
This means that STaSy is able to capture every mode in the columns, while
OCT-GAN is not.
In Figure 3 (Right), CTGAN generates some out-of-distribution records, highlighted in red.
4.2. S AMPLING TIME
[Table]
We summarize runtime in Table 5.
In order to compare the runtime of all methods, we measure the wall-clock time taken to sample N records, where N is training size, 5 times, and average them.
In general, simple GAN-based methods, especially TableGAN and TVAE , show faster runtime.
On the other hand, SGMs, OCT-GAN , and RNODE take a relatively long time for sampling.
Our proposed methods, Naïve-STaSy and STaSy , take a long sampling time compared to simple GAN-based methods but are faster than OCT-GAN and RNODE , which means a well-balanced trade-off between the sampling quality, diversity, and time.
4. A BLATION & SENSITIVITY STUDIES
[Table]
We define three ablation models: ‘ Naïve-STaSy ’ without SPL and fine-tuning, ‘w/o fine-tuning’ without fine-tuning but with SPL, and ‘w/o SPL’ without SPL but with fine-tuning.
All ablation models are inferior to STaSy in Table 6, showing the effectiveness of SPL and fine-tuning.
In particular, SPL improves the sampling diversity as in Figure 4.
Naïve-STaSy suffers from mild mode collapses, as highlighted in red.
[Table]
Table 7 shows sensitivity analyses w.r.t.
some important hyperparameters.
In general, all settings show reasonable results, outperforming the baselines.
We recommend 0.2 and 0.25 for α 0 and 0.
and 0.95 for β 0 .
We can adopt a variety of methods to solve the reverse SDE process in Equation 2.
Our method can generate fake records with the predictor-corrector framework (Pred.
Corr.) or the probability flow (PF) method (Song et al., 2021).
The former uses the ancestral sampling (AS), reverse diffusion (RD), or Euler-Maruyama (EM) method for solving the reverse SDE, and for the correction process, the
Langevin corrector.
In Table 8, the probability flow method in Equation 10 mostly leads to successful results, and other datasets also show similar results. [Table] [Table] predictor
[Table]
only method
[Table]
C ONCLUSIONS AND DISCUSSIONS
Synthesizing tabular data is an important yet non-trivial task, as it requires modeling a joint probability of multi-modal columns.
To this end, we presented our detailed designs and experimental results with thorough analyses.
Our proposed method, STaSy , is a score-based model equipped with our proposed self-paced learning and fine-tuning methods.
In our experiments with 15 benchmark datasets and 7 baselines, STaSy outperforms other deep learning methods in terms of the sampling quality and diversity (and with an acceptable sampling time).
Based on these considerations, we believe that
STaSy shows significant advancements in tabular data synthesis.
We expect much follow-up work in utilizing SGMs for tabular data synthesis.
Limitations.
Although our model shows the best balance for the deep generative task trilemma, we think that there exists room to improve runtime further — existing simple GAN-based methods are faster than our method for sampling fake records.
In addition, SGMs are known to be sometimes unstable for high-dimensional data, e.g., high-resolution images, but in general, stable for lowdimensional data, e.g., tabular data.
Therefore, we think that SGMs have much potential for tabular data synthesis in the future.
E THICS STATEMENT
Indeed, people do not always use artificial intelligence technology for righteous purposes.
One can use our method to achieve his/her wrongful goals, e.g., selling high-quality fake data generated by our method, and retrieving private original data records from synthetic data.
However, we believe that our research has much more beneficial points.
One can use our method to generate fake data and share (after hiding the original data) to prevent potential privacy leakages.
We, of course, need more studies to achieve the privacy protection goal based on our model.
However, a research trend exists where researchers try to use a deep generative model to protect privacy (Park et al., 2018; Lee et al., 2021).
R EPRODUCIBILITY STATEMENT
To reproduce the experimental results, we have made the following efforts: 1) Source codes used in the experiments are available in the supplementary material.
By following the README guidance, the main results are easily reproducible.
All the experiments are repeated five times, and their mean and standard deviation values are reported in Appendix.
We provide extensive experimental details in Appendix H
A VE, VP, AND SUB -VP SDE S
We introduce the definitions of f and g as follows:
[Equation]
12) where σ ( t ) and γ ( t ) are noise functions w.r.t.
time t .
σ ( t ) =
σ min (cid:16) σ max σ min (cid:17) t for t ∈ [0 , 1] , where σ min and σ max are hyperparameters, and we use σ min = { 0 . 01 , 0 . 1 } and σ max = { 5 . 0 , 10 . 0 } .
γ ( t ) =
γ min + t ( γ max − γ min ) for t ∈ [0 , 1] , where γ min and γ max are hyperparameters, and we use γ min = { 0 . 01 , 0 . 1 } and γ max = { 5 . 0 , 10 . 0 } .
B C OMPARISON BETWEEN SOS AND ST A S Y FOR THE OVERSAMPLING TASK
[Table]
In this section, we discuss the difference between SOS and STaSy .
They are both based on SGMs, but they are optimized towards different goals by using different objective functions and training strategies.
SOS has many design points specialized to augment minor classes only (rather than synthesizing entire tabular data) — for instance, SOS adopts a style-transfer-based idea to convert a major class sample to a minor one via their own SGM model without any consideration on the training difficulty of the denoising score matching.
However, our STaSy more focuses on synthesizing entire tabular data by proposing special self-paced training and fine-tuning methods.
We use five imbalanced datasets to oversample — many datasets in our main experiments are not imbalanced.
We conduct the oversampling experiments with STaSy to compare with SOS (Kim et al., 2022). In this experiment, STaSy is converted to an oversampling method following the design guidance of SOS , i.e., each minority class has its own score network and is separately trained.
We train, for fair comparison, STaSy w/o fine-tuning for each minor class and generate minority samples to be the same size of the majority class.
We compare the two models in terms of the sampling quality using Weighted-F1 which is specialized in evaluating imbalanced data.
We note that Identity means that we do not use any oversampling methods, which is, therefore, a minimum quality requirement upon oversampling.
As shown in Table 9, STaSy w/o fine-tuning outperforms SOS .
The result shows that our proposed training strategy, i.e., the self-paced learning, improves the model training regardless of tasks.
C N ETWORK ARCHITECTURE
We propose the following score network S θ ( x ( t ) , t ) :
[Equation]
FC ( h d N ) , where x ( t ) is a record (or a row) at time t in tabular data, h i is the i -th hidden vector, and ω is an activation function.
d N is the number of hidden layers.
For various layer types of H i
( h i − 1 , t ) , we provide the following options:
[Equation]
where we can choose one of the three possible layer types as a hyperparameter, ⊙ means the elementwise multiplication, ⊕ means the concatenation operator, ψ is the Sigmoid function, and FC is a fully connected layer.
We modify the architecture of (Song et al., 2021) by using the layer types, being inspired by (Grathwohl et al., 2018).
D T HRESHOLD CONTROLLING MECHANISM
Our threshold controlling mechanism is designed to meet the following 3 requirements: i) it can control when the entire dataset is used for training, starting from a subset, ii) it should gradually increase the size of used training records while logarithmically decreasing the number of hard records (that are not involved in training), and iii) it should be a monotonically increasing/decreasing function to guarantee that the training difficulty gets more challenging as the training process goes on.
The threshold controlling variables α and β , where 0 ≤ α ≤ β ≤ 1 , are gradually increased to 1 to involve the entire data records for training.
We increase them proportionally to training steps, where α = α 0 + log (cid:16)
+ c (cid:16) e − 1 S (cid:17) (1 − α 0 ) (cid:17) and β = β 0 + log (cid:16)
+ c (cid:16) e − 1 S (cid:17) (1 − β 0 ) (cid:17) .
e is the base of the natural logarithm, α 0 and β 0 are initial values of α and β , c is the current training step, and S determines when to utilize the entire data records.
We use 10,000 for S .
We set β 0 at least 0.8 to ensure 80% of the data records are involved in training at the start of the training. We set α = αand β =
β 0 at the beginning of the training, since c is 0.
We note that a training sample x i whose quantile of loss value is greater than β is regarded as a hard sample and v i is set to be 0. If we set βto be 0.8, the top 20% of difficult samples will not be used at the first training step. In this way, one can control the proportion of training samples involved in the training from the beginning. E P ROOF OF T HEOREM
As defined in Section 3.2, the STaSy objective is as follows:
[Equation]
where l i is the score matching loss for i -th training record as in Equation 6.
We can rewrite the optimal solution for each training record v i with respect to fixed θ in the vertex form.
Let L ( v i ) be the objective with fixed θ , which is a quadratic function with respect to v i .
Then,
L
[Equation]
Because Q ( α ) − Q ( β )is less than or equal to 0 and Q ( α ) − Q ( β )(cid:16) l
i − Q ( β ) Q ( α ) − Q
[Equation]
β ) (cid:17) 2 is a constant, the solution v i which minimizes Equation 14 is v i
=
l i − Q ( β ) Q ( α ) − Q ( β ) .
Considering v i ∈ [0 , 1] , we can get the optimal v i as follows:
[Equation]
F T HE H UTCHINSON ’ S ESTIMATION AS A FINE - TUNING OBJECTIVE
We use the denoising score matching loss in Line 10 of Algorithm 1.
In this section, we describe the results of an additional experiment in which the Hutchinson’s log-probability estimation is used for the fine-tuning objective.
Table 10 summarizes the F1 score and the median of log-probabilities when we use the denoising score matching loss and the Hutchinson’s estimation as the tine-tuning objective.
In Contraceptive , there does not exist a clear winner between the two fine-tuning objectives, and similar results are also shown in other datasets.
However, in some datasets, e.g., Shoppers and Crowdsource , the former shows better F1 scores and better medians of the log-probabilities than the latter by large margins.
In addition, when we update θ using the Hutchinson’s estimation, in Default , the sampling quality is lower than before fine-tuning.
Considering these results, we use the denoising score matching loss, which shows the generalizability, as our default fine-tuning objective.
[Table]
G A DDITIONAL EXPERIMENTAL RESULTS G. S AMPLING QUALITY
We mainly use F1 (resp.
R 2 ) for the classfication (resp.
regression) TSTR evaluation, and also report
AUROC and Weighted-F1 (resp.
RMSE) results.
Full results for all datasets are in Tables 11, 12, and 13 for binary classification, Tables 14, 15, and 16 for multi-class classification, and Table 17 for regression.
We train and test various base classifiers/regressors and report their mean and standard deviation.
Moreover, we use the log-probability as another metric for the sampling quality.
Full results are in Table 18.
The best results are highlighted in bold face and the second best results with underline.
As shown, Naïve-STaSy and STaSy show the best and the second best performances in almost all cases.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
TableGAN
[Table]
RNODE
[Table]
STaSy
B i n a r y
Credit 71.
[Table]
117.404 Default 55.
131.122 HTRU 27.
5.326 Magic 17.
30.216 Phishing 53.
228.914 Shoppers 62.
220.589 Spambase 140.
M u lti .
Bean 46.
92.635 Contraceptive 26.
225.638 Crowdsource 23.
48.905 Obesity 130.
333.017 Robot 14.
79.355 Shuttle 48.
R e g .
Beijing 42.
18.192 News 128.
G. S AMPLING DIVERSITY
We use the coverage score as a metric for the sampling diversity.
Full results for all datasets are in
Tables 19, 20, and 21.
We measure the coverage score 5 times with different fake records and report their mean and standard deviation.
Coverage is bounded between 0 and 1, and higher coverage means more diverse samples.
This k-NN-based measurement is expected to achieve 100% performance when the real and fake records are identical, but in practice, this is not always the case.
For the dataset whose coverage score does not show 1 for two same data records, we choose the hyperparameter k to achieve at least greater than 0.95.
In our experiments, k for Phishing is 7, and for others, k is 5.
As shown in Tables 19, 20, and 21, in 12 out of 15 datasets, our methods outperform others by large margins
[Table] [Table]
[Table] [Table]
G. S AMPLING TIME
Tables 22, 23, and 24 show runtime evaluation results of each method.
We measure the wall-clock time taken to sample fake records 5 times, and report their mean and standard deviation.
In almost all datasets, Naïve-STaSy and STaSy show faster runtime than OCT-GAN and RNODE .
TableGAN and TVAE take a short sampling time, but considering their inferior sampling quality and diversity, only our proposed model resolves the problems of the generative learning trilemma .
[Table] [Table]
[Table]
H E XPERIMENTAL ENVIRONMENTS
Our software and hardware environments are as follows: U BUNTU 18.04 LTS, P YTHON 3.8.2, P YTORCH 1.8.1, CUDA 11.4, and NVIDIA Driver 470.42.01, i9 CPU, and NVIDIA RTX 3090.
Our code for the experiments is mainly based on https://github.com/yang-song/score_ sde_pytorch (Apache License 2.0).
H. B ASELINES
We utilize a set of baselines that includes various generative models.
• Identity is a case where we do not synthesize but use original data.
• MedGAN 1 (Choi et al., 2017) is a GAN that incorporates non-adversarial losses to generate discrete medical records.
• VEEGAN 1 (Srivastava et al., 2017) is a GAN for tabular data that avoids mode collapse by adding a reconstructor network.
• CTGAN 1 (Xu et al., 2019) and TVAE
(Xu et al., 2019) are a conditional GAN and a VAE for tabular data with mixed types of variables.
• TableGAN
(Park et al., 2018) is a GAN for tabular data using convolutional neural networks.
• OCT-GAN 2 (Kim et al., 2021) is a GAN that has a generator and discriminator based on neural ordinary differential equations.
• RNODE 3 (Finlay et al., 2020) is an advanced flow-based model with two regularization terms added to the training objective of FFJORD (Grathwohl et al., 2018).
H. D ATASETS
In this section, we describe 15 real-world tabular datasets for our experiments.
We select the datasets for experiments with two metrics: 1) how many times a dataset has been cited/used in previous papers, and 2) how many times a dataset has been viewed/downloaded in famous repositories, such as UCI
Machine Learning Repository and Kaggle.
Among them, we choose the datasets that can be used for classification and regression tasks, with more than 5 columns and 1,000 rows.
• Credit is a binary classification dataset collected from European cardholders for credit card fraud detection.
• Default (Lichman, 2013) is a binary classification dataset describing the information on credit card clients in Taiwan regarding default payments.
• HTRU (Lyon, 2017) is a binary classification dataset that describes a sample of pulsar candidates collected during the High Time Resolution Universe Survey.
• Magic (Bock, 2007) is a binary classification dataset that simulates the registration of high-energy gamma particles in the atmospheric telescope.
• Phishing (Mohammad, 2015) is a binary classification dataset used to distinguish between phishing and legitimate web pages.
• Shoppers (C Okan Sakar, 2019) is a binary classification dataset about online shoppers’ intention.
• Spambase (Hopkins, 1999) is a binary classification dataset that indicates whether an email is spam or non-spam.
• Bean (Koklu & Ozkan, 2020) is a multi-class classification dataset that includes types of beans with their characteristics.
• Contraceptive (Lim, 1997) is a multi-class classification dataset about Indonesia contraceptive prevalence.
• Crowdsource (Johnson & Iizuka, 2016) is a multi-class classification dataset used to classify satellite images into different land cover classes.
• Obesity (Palechor & de la Hoz Manotas, 2019) is a multi-class classification dataset describing obesity levels based on eating habits and physical condition.
• Robot (Freire, 2010) is a multi-class classification dataset collected as the robot moves around the room, following the wall using ultrasound sensors.
[Table]
Magic
[Table]
Phishing
[Table]
Shoppers
[Table]
Spambase
[Table]
Bean
[Table]
Crowdsource
[Table]
Obesity
[Table] [Table]
Robot
[Table]
K
[Table]
.1K
[Table]
Multi
[Table]
class classification (4) Shuttle 46.4K 11.6K
Multi-class classification (7) Beijing
[Table]
News
[Table]
• Shuttle (shu) is a multi-class classification dataset for extracting conditions in which automatic landing is preferred over manual control of the spacecraft.
• Beijing (Liang et al., 2015) is a regression dataset about PM2.5 air quality in the city of Beijing.
• News (Fernandes, 2015) is a regression dataset about online news articles to predict the number of shares in social networks.
The statistical information of datasets used in our experiments is in Table 25.
#train, #test, #continuous, #categorical, and #class mean the number of training data, testing data, continuous columns, categorical columns, and class, respectively.
The raw data of 15 datasets are available online:
• Credit : https://www.kaggle.com/mlg-ulb/creditcardfraud (DbCL 1.0) • Default : https://archive.ics.uci.edu/ml/datasets/default+of+ credit+card+clients (CC BY 4.0) • HTRU :
https://archive.ics.uci.edu/ml/datasets/HTRU2 (CC BY 4.0) • Magic : https://archive.ics.uci.edu/ml/datasets/magic+gamma+ telescope (CC BY 4.0) •
Phishing : https://archive.ics.uci.edu/ml/datasets/phishing+ websites (CC BY 4.0) • Shoppers :
https://archive.ics.uci.edu/ml/datasets/Online+ Shoppers+Purchasing+Intention+Dataset (CC BY 4.0) •
Spambase : https://archive.ics.uci.edu/ml/datasets/spambase (CC BY 4.0) •
Bean : https://archive.ics.uci.edu/ml/datasets/Dry+Bean+ Dataset (CC BY 4.0) • Contraceptive : https://archive.ics.uci.edu/ml/datasets/ Contraceptive+Method+Choice (CC BY 4.0) • Crowdsource : https://archive.ics.uci.edu/ml/datasets/ Crowdsourced+Mapping#
(CC BY 4.0) • Obesity : https://archive.ics.uci.edu/ml/datasets/Estimation+ of+obesity+levels+based+on+eating+habits+and+physical+ condition+ (CC BY 4.0)
• Robot : https://archive.ics.uci.edu/ml/datasets/ Wall-Following+Robot+Navigation
Data
• Shuttle : https://archive.ics.uci.edu/ml/datasets/Statlog+ (Shuttle) (CC BY 4.0) • Beijing : https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.
5+Data (CC BY 4.0) • News : https://archive.ics.uci.edu/ml/datasets/online+news+ popularity
H.
E VALUATION METHODS
The reported scores for TSTR results in the paper are calculated as follows:
We download a dataset.
If used previously, we use their train-test split.
If not used before, we perform a new train-test split.
The train-test split ratio is 80% and 20%, respectively.
Generate fake records which has the same number of records as the original training set for other fake data generation methods. 3.
Using the training records from Step2, we train base classifiers/regressors to predict.
We search the best hyperparameter set for each classifier/regressor.
For validation, we use the original training data’s task scores, e.g., F1 or R 2 .
In Table 26, considered hyperparameters and their candidate settings are summarized.
We use DecisionTree, AdaBoost, Logistic Regression, MLP classifiers, RamdomForest, and XGBoost for binary classification tasks; DecisionTree, MLP classifiers, RandomForest, and XGBoost for multi-class classification tasks; MLP regressor, RandomForest, XGBoost, and Linear Regression for regression tasks.
Test the best performing classifiers/regressors with testing data.
We use various evaluation metrics for rigorous evaluations as reported earlier.
We repeat the 4th step five times for all datasets.
We then calculate the average score for each method and for each evaluation metric.
Detailed metrics for our experiment are as follows:
Binary F1 for binary classification datasets: f1_score from sklearn.metrics after setting the ‘average’ option to ‘binary’.
Macro F1 for multi-class classification datasets: f1_score from sklearn.metrics after setting the ‘average’ option to ‘macro’.
Weighted-F1 for classification datasets: Weighted-F1 =
(cid:80) Ni =
w i s i , where N is the number of classes, the weight of i -th class w i is 1 − p
i N − 1 , p i is the proportion of i -th class’s cardinality in a total dataset, and score s
i is a per-class F1 of i -th class (in a One-vs-Rest manner).
This formula allows us to evaluate
synthesized tables
with more focus on mode collapse by giving a higher weight to a smaller class, which is more likely to be forgotten by the model.
AUROC: roc_auc_score from sklearn.metrics.
Coverage:
compute_prdc from https://github.com/clovaai/ generative-evaluation-prdc .
I H YPERPARAMETERS
Hyperparameter settings for the best models are in Table 27.
We have three SDE types, which are VE,
VP, and sub-VP, and three layer types as shown in Appendix C: Concat, Squash, and Concatsquash.
We use a learning rate in { 2 e − 03 , 2 e − 04 } .
We search for α 0 and β 0 , in total, with 9 combinations using α 0 = { 0 . 20 , 0 .
25 , 0 . 30 } and β
= { 0 . 80 , 0 . 90 , 0 .
We also consider the hyperparameters for the fine-tuning process.
To compute the exact log-probability with the Hutchinson’s estimation (Hutchinson, 1989; Grathwohl et al., 2018), we use a Gaussian or a
Rademacher distribution for p ( ϵ ) , where ϵ is a noise vector.
The fine-tuning epoch is { 1 , . . .
, 20 } , and the fine-tuning learning rate is { 2 × 10 − i | i = { 4 , 5 , 6 , 7 }} .
[Table]
Linear Regression
[Table]
[Table]
J A DDITIONAL VISUALIZATIONS
We show several visualizations that are missing in the main paper.
In each subsection, we show column-wise histograms and t-SNE visualizations on HTRU , Robot , and News , respectively.
J. A DDITIONAL VISUALIZATIONS IN HTRU
As shown in Figure 5, the fake data distributions of TVAE , TableGAN , and RNODE are dissimilar to the real data distributions, and these baselines fail to sample high-quality fake records.
In Figure 6,
CTGAN , TableGAN , OCT-GAN , and RNODE suffer from mode collapses as highlighted in red.
In
STaSy , however, the mode collapse problem is clearly alleviated, which means that STaSy is effective in enhancing the diversity.
[Figure]
[Figure]
J. A DDITIONAL VISUALIZATIONS IN R OBOT
In Robot , TVAE , OCT-GAN , and STaSy generate relatively similar distribution to that of real data as shown in Figure 7.
In Figure 8, CTGAN , TableGAN , and RNODE generate out-of-distribution records as highlighted in red, and TVAE and OCT-GAN suffer from mode collapses.
Our proposed methods generate the most diverse records among various methods.
[Figure]
[Figure]
J. A DDITIONAL VISUALIZATIONS IN N EWS
In News , the fake data by STaSy shows more reliable column-wise histogram than others in Figure 9.
As shown in Figure
[Figure]
all methods except for TableGAN and RNODE well generate fake records
[Figure] [Figure]