Paper: Diffusion Models for Tabular Data Imputation and Synthetic Data Generation (ICLR-2024)

Reviewer comment (Evaluation):
For ML efficiency, the original TabDDPM paper demonstrates multiple examples where the generated data is able to achieve better performance over the baseline. However, this behavior is not seen here?

Author response:
In our study, we defined the baseline as an oracle that either generates or imputates the real data. We have clarified this in the updated caption of Table 2 in the results section. Our baseline is the performance of XGBoost trained and tested on real data, representing the optimal scenario. In the case of TabDDPM they also compare training with the real data and their model outperforms that baselines for 4 datasets, while in our case we outperform it for the Insurance dataset only. 

One key difference between TabDDPM and ours is in the method used for measuring the ML efficiency. While we use the exact same number of samples as the original training data, TabDDPM employs a “proportion of samples” hyper-parameter that can lead to generating up to eight times the volume of real samples for certain datasets. This difference in the volume of generated samples could potentially account for the observed variations in performance relative to the baseline.

Reviewer comment (Experiments):
Is it possible to conditionally generate diverse synthetic data by conditioning on an outcome feature? E.g. death event or housing price?

Author response:
Yes, it is possible to generate synthetic data conditioned on any feature.

Reviewer comment (Evaluation):
Does this model faithfully generate data that captures low-domain clusters / phenotypes in the original data space?

Author response:
Our model is designed to capture and replicate the complex distributions and relationships present in the original data. However, the exact fidelity in representing low-domain clusters or specific phenotypes would require further detailed analysis, which could be a direction for future research.

Reviewer comment (Evaluation):
Why does the best performance switch for TabGenDDPM I and II in the cardio dataset?

Author response:
The observed performance variation between the models is interesting and leads us to consider several hypotheses. First, we hypothesize that in the case of the Cardio dataset, the discriminative power of the XGBoost model is somewhat limited, as indicated by its relatively lower performance even when trained with real data (baseline). This limitation introduces a level of noise to the results, potentially influencing the observed switch in performance ranking between TabGenDDPM I and II.

A second hypothesis considers the characteristics of the cardio dataset, which contains features that are highly correlated. The dynamic conditioning mechanism in our model, which varies the masked features during training, could potentially disrupt these correlations and may contribute to training a more robust model. 
 Although these hypotheses may provide possible explanations, definitive understanding of the performance switch would need a deeper analysis.

Reviewer comment (Reproducibility):
The code seems not to be open source.

Author response:
Thank you for pointing this out. To maintain the integrity of the double-blind review process, we've included the code as supplementary material in a zip file. We plan to publish it in github after the review process is over.

Reviewer comment (Experiments):
The hyper-parameter space of TabDDPM seems modified in your experiment (e.g. no batch size 4096 and no learning rate). Why?

Author response:
You're correct in observing the differences in hyper-parameter space. Our decision was driven by the results of preliminary hyper-parameter optimization using Optuna, which indicated superior performance with smaller batch sizes, hence the absence of batch sizes above 1024 in our experiments. Additionally, given the constraints of our computational resources, we had to limit the scope of our search space.

Reviewer comment (Theory):
With the masking system it is possible to condition on any feature. Why keep a specific treatment for the target value?

Author response:
This is an insightful observation. Our decision to treat the target variable distinctly and keep it always unmasked was primarily motivated by our focus on generating synthetic samples conditioned on the target.  This approach aligns with the class-conditional model used by TabDDPM,

Reviewer comment (Evaluation):
The statistical similarity metric is not usual and does not permit an easy comparison with other papers, why not use "sdmetrics" library to provide other metrics (notably C2ST detection metrics)?

Author response:
Thank you for this suggestion. Our choice of statistical similarity metrics was influenced by precedents in related works, such as TabDDPM and CTAB-GAN [1], which utilize correlation similarity, Wasserstein, and Jensen-Shannon distances. However, we appreciate the relevance of the "sdmetrics" library and we certainly plan to incorporate these in future revisions of our work or subsequent studies to provide a more comprehensive and comparable analysis.  

[1] Zhao, Zilong, et al. "Ctab-gan: Effective table data synthesizing." Asian Conference on Machine Learning. PMLR, 2021.

Reviewer comment (Novelty):
The overall contribution of this paper is limited. All of the content except the transformer conditioning architecture is already known. The architecture design is heuristic, which has no theoretical guarantees of the performance. Moreover, they build upon Variance Preserving (VP) SDE (e.g., DDPM or TabDDPM in tabular data). The author does not mention whether their method works for Variance Exploding (VE) SDE (e.g., Score-based generative model, StaSy [1] in tabular data).  
[1]: Kim, J., Lee, C.E., & Park, N. STaSy: Score-based Tabular data Synthesis. ICLR 2023.

Author response:
Dear Reviewer uxeP,
 
 
Thank you for your insightful comments and for bringing the works involving Variance Exploding SDE like StaSy. We acknowledge that including a comparison with StaSy would have enriched our study. At the time of our research and manuscript preparation, we were not aware of this work and the application of VE SDE to tabular data. We plan to adapt the model to VE SDE and compare with other Score-based tabular generative models in future revisions of this work or subsequent studies.

Reviewer comment (Novelty):
Overclaiming the contribution of transformer conditioning architecture. Diffusion models can work on imputation together with generation (conditional generation) without the proposed transformer conditioning architecture. These approaches are well studied in the literature [1,2].  
[1]: Tashiro, Y., Song, J., Song, Y., & Ermon, S. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. NIPS 2021.  
[2]: Ouyang, Y., Xie, L., Li, C., & Cheng, G. (2023). MissDiff: Training Diffusion Models on Tabular Data with Missing Values. ArXiv, abs/2307.00467.

Author response:
Regarding other works that perform data imputation together with generation, the first paper CSDI only works with continuous variables in temporal series, and it is not straightforward of how to adapt it for tabular data. The second proposed paper, MissDiff, rises a broader discussion whether considering preprint papers under review for same conference. MissDiff is also under review for ICLR 2024 https://openreview.net/forum?id=vULHgaoASR, which we will consider for future research.

Reviewer comment (Evaluation):
The evaluation of imputation performance is not aligned with standard practices. The standard evaluation of imputation performance is the mean squared error of imputed value against oracle value, instead of the efficiency criterion used in the paragraph "Machine Learning efficiency - Data imputation." Using the efficiency criterion faces the problem mentioned in the paper of "when the generative model needs to fill in the most significant feature or a feature that has a minimal impact on XGBoost output." If the authors adopt the traditional evaluation for this task, many designs in that paragraph would not be needed.

Author response:
* We have used ML efficiency for evaluating the data imputation task, because the imputed data doesn't need to be exactly the same as the original data, but just follow the same distribution conditioned on the available data. In our opinion, ML efficiency measures the imputation performance better than MSE of imputed data with real data. In addition to that, there are also categorical variables in the datasets, where the MSE is not as descriptive as in continuous variables.

Reviewer comment (Experiments):
The evaluation of TabGenDDPM on the imputation task should include comparisons with other imputation methods, such as GAIN [3] and MIWAE [4], rather than only with TabDDPM.  
[3]: Yoon, J., Jordon, J., & Schaar, M.V. GAIN: Missing Data Imputation using Generative Adversarial Nets. ICML 2018.  
[4]: Mattei, P., & Frellsen, J. MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. ICML 2019.

Author response:
* We have added two baselines for the data imputation task and updated the graphs and the text under  “Data imputation” in the section “Results”. The baselines are GAIN and missForest. We didn't include MIWAE, because after carefully analyzing the code available we think it wouldn't be easy to adapt to categorical variables.

Reviewer comment (Experiments):
The authors should compare their method with other diffusion-based models for tabular data, such as StaSy [1], and provide discussion and experimental results on whether transformer conditioning can be developed on Variance Exploding (VE) SDE.  
[1]: Kim, J., Lee, C.E., & Park, N. STaSy: Score-based Tabular data Synthesis. ICLR 2023.

Author response:
Dear Reviewer uxeP,
 
 
Thank you for your insightful comments and for bringing the works involving Variance Exploding SDE like StaSy. We acknowledge that including a comparison with StaSy would have enriched our study. At the time of our research and manuscript preparation, we were not aware of this work and the application of VE SDE to tabular data. We plan to adapt the model to VE SDE and compare with other Score-based tabular generative models in future revisions of this work or subsequent studies.

Reviewer comment (Experiments):
The experimental setup is unclear and needs clearer explanation. There is confusion regarding:  
- Which method the baseline in Figure 3 represents, as it does not seem to match the methods mentioned in section 5.2.  
- The presentation in Table 4, where three different evaluation criteria have different properties (e.g., smaller correlation is better, unlike privacy risk), making the use of up/down arrows beside dataset names unclear.  
- The reason why experimental results are reported on only six datasets in Table 4 instead of the eight datasets listed in Table 2.  
- The absence of performance results on each individual dataset for Table 3 in the appendix.

Author response:
* We have also updated the caption in Table 2 and Figure 3. Regarding the baseline here, it is the same XGBoost trained on original real data and tested on real data, and the scores of the baseline stand as what would be the best score in optimal conditions. Nevertheless, for the “Insurance” dataset all tested methods surpass this oracle baseline.
* In Figure 3. The baseline is an oracle data imputation that imputes real data and that its performance is the same as in Table 2. We have updated the table to make it clear.
* In table 4, we have changed the caption to make clear that the “up arrow” stands for a maximization optimization measuring the ML efficiency in F1 score and a “down arrow” for a minimization optimization of the MSE.

Reviewer comment (Evaluation):
What is the running time of the proposed approach and how does it compare with the other state-of-the-art algorithms?

Author response:
**A**: We have not conducted rigorous timing measurements, but the approximate order of runtime, from fastest to slowest, is as follows: TVAE < CTGAN < TabDDPM < TabGenDDPM. The difference in running time between TVAE and CTGAN is minimal, similar to that between TabDDPM and TabGenDDPM.

Reviewer comment (Experiments):
How does it perform when the feature set is large and/or the number of samples is small?

Author response:
**A**: The dataset with the smallest sample size and the largest feature set that we have tested is HELOC. For this dataset our proposed method outperforms the baselines by a greater margin than for the other datasets.

Reviewer comment (Experiments):
How does it work in augmentation tasks where the training is a mix of real + synthetic and testing on real?

Author response:
**A**: We conducted some preliminary tests on smaller datasets, where combining real and synthetic data generally improved test scores. However, these were not included in the paper to maintain consistency with other methods and the oracle baseline. While we haven't explored this in depth, it's an area we're considering for future research. In Table 2 of our results, we demonstrate that for the “Insurance” dataset, an XGBoost model trained on synthetic data outperforms one trained on real data. This suggests that synthetic data could indeed be beneficial for augmentation purposes.