Paper: Programmable Synthetic Data Generation (ICLR-2024)

Reviewer comment (Novelty):
Emphasizing the programmability of ProgSyn sounds a bit exaggerated. For example, Figure 3 looks like a conventional config file instead of say a Python program. When using DP, a user always specifies epsilon and delta, but this is not called programming.

Author response:
**Q4: Does the accompanying domain specific language (DSL) provide further programmability than that achievable by simple configuration files?**

Yes, which we believe is also clearly demonstrated by the examples provided both in the main paper and in the list of commands used in the appendix. Note that we support composite expressions both for logical and statistical constraints that naturally require recursive parsing, best facilitated in a DSL. Note also that programmability here does not refer to a general purpose programming language, but rather to an easy-to-use and intuitive interface in which the complex specifications can be declared in a fashion that closely resembles those expressions' mathematical definition.

To exemplify the usefulness of ProgSyn’s programmability, consider the following example on how the user has to input the constraint I2 from Table 2 to ProgSyn, SDV models, and to AIM, respectively:

ProgSyn:
```
ENFORCE: IMPLICATION:
	marital_status in {Divorced, Never_married} IMPLIES relationship not in {Husband, Wife}
```

SDV:
```
def is_valid_I2(column_names: list, data: pd.DataFrame, extra_parameter) -> pd.Series:
	data = data.reset_index(drop=True)
	validity_filter = np.ones(len(data)).astype(bool)
	antecedent_mask = (data['marital-status'] == 'Divorced') | (data['marital-status'] == 'Never-married')
	antecedent_indices = data[antecedent_mask].index.to_numpy()
	consequent_mask = np.logical_and((data[antecedent_mask]['relationship'] != 'Husband').to_numpy(), (data[antecedent_mask]['relationship'] != 'Wife').to_numpy())
	validity_filter[antecedent_indices] = consequent_mask
	return pd.Series(validity_filter)
```

AIM:
```
szeros = {('marital-status', 'relationship'): [(1, 2), (1, 0), (2, 2), (2, 0)]}
```

As we can see, ProgSyn provides a compact, intuitive, and practical interface for declaring constraints.

Reviewer comment (Experiments):
It is not clear why DP should be optimized together with fairness holistically. Why not generate a fair dataset using an existing technique or a fairness-only version of ProgSyn and then add random noise to satisfy DP? This two-step approach should be a baseline and compared with ProgSyn empirically.

Author response:
**Q2: Is the current approach to fairness under differential privacy constraints necessary for strong performance?**

Yes, we believe so. Note also that, strictly speaking, we do not optimize fairness and differential privacy together in the current approach. Rather, we first conduct differentially private pre-training and then fine-tune for fairness, where the differential privacy guarantee is maintained due to the pos-processing property. As established by [1], DP can have a detrimental impact on the fairness of a given dataset, therefore it is necessary that the DP model/data is adapted for fairness achieving (or reinstating) a state of low bias.

To demonstrate the weakness of a pipeline with an alternative ordering, we conduct the following experiment on Adult. We generate non-private fair synthetic data using ProgSyn (as it is the best current method for this purpose), then use the marginals obtained from these fair samples to train a differentially private ProgSyn. We compare the accuracy and the demographic parity distance w.r.t. to sex to the original pipeline in the table below:

| |Accuracy [%]|Dem. Par.|  
|:--|:---:|:---:|  
|First DP then Fairness (as in the paper)| **82.1**|**0.01**|  
|First Fairness then DP|81.6|0.05|

As we can observe, the currently employed pipeline prevails both in terms of accuracy and bias score.

Reviewer comment (Experiments):
It would be more interesting to see the limitations of this method where the accuracy actually has to drop in order to satisfy various constraints. The current experiments only show success cases, but fairness and privacy are not necessarily aligning, so there has to be a point where the accuracy cannot be maintained. In Table 3, there is almost no reduction of accuracy after applying DP, which suggests that the proposed method may have not been stress tested enough. Hence, there should be more extensive experiments showing what happens in truly challenging scenarios.

Author response:
**Q3: Can you please discuss the performance of ProgSyn under several simultaneous constraints in more detail? At which point does it break down?**

Yes, we appreciate the reviewer's question, and we also believe that it can be informative to explore the extremes of the method. Our original consideration with the experiment was to include a reasonable amount of constraints, i.e., a number that could commonly come up in usual use-cases, and see if the method can still perform at a satisfactory level.

Note that we still observe an accuracy trade-off for added constraints, however, it is rather low, with the method qualitatively maintaining roughly the same performance level. Smoother decreases, but still within reasonable minimal performance, can be observed on the same experiment on other datasets, as shown in Tables 10-12 in the appendix.

To complete the picture, we added more constraints, even contradicting constraints to the experiment on Adult, and reordered it such that we can observe a smoother decrease of the accuracy:

|Acc. [%]|Mean Age|M-F Age Diff|I3|I2|Cov(sex, y)|R2|I1|Dem. Par.|  
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|  
|85.1|37.29|2.33|0.60|0.99|0.22|0.40|0.93|0.19|  
|84.5|30.20|1.46|0.59|0.99|0.22|0.34|0.93|0.20|  
|84.5|30.18|0.02|0.59|0.99|0.22|0.33|0.93|0.20|  
|84.7|30.18|0.03|0.99|0.98|0.22|0.34|0.93|0.18|  
|84.6|30.19|0.04|0.98|1.00|0.22|0.34|0.92|0.19|  
|83.0|30.19|0.00|0.93|1.00|0.00|0.33|0.92|0.11|  
|81.3|35.28|0.02|0.98|1.00|0.00|1.00|0.93|0.04|  
|80.3|35.28|0.01|0.92|1.00|0.00|1.00|1.00|0.04|  
|77.5|35.28|0.01|0.96|1.00|0.00|1.00|1.00|0.11|

For the sake of illustration, we did not employ rejection sampling over the logical constraints here, such that we have a better insight in the influence of the constraints on eachother. As we can see, with this many complex constraints the performance smoothly decreases as more and more constraints are added, to the point where the satisfaction scores of certain specifications are starting to worsen as well. The breaking point happens, when the sixth constraint R2 is introduced, which constrains the age feature to be between 35 and 55, which contradicts the first constraint that is setting the mean age to 30 and also interferes with the second constraint which is trying to equalize the mean age of males and females. This destabilizes the optimization, and a larger than expected accuracy loss is incurred. From this point on, additional constraints put further strain on the model, with the accuracy decreasing until 77.5%. Nevertheless, even at this complexity level, the model still produces data that is compliant with most of the given specifications.

Note that the above breakdown is lemon-picked, and we believe that most use-cases will not include logically contradicting and several interfering specifications.

Reviewer comment (Evaluation):
Theoretical Framework: Although the research area explored in this paper holds promise for further investigation, the technical aspects in the paper appear somewhat dated, primarily covering foundational theories and methods. Considering the goal of ProgSyn is to generate sufficiently realistic simulated data with privacy protection properties, the authenticity aspect is addressed mainly in terms of experimental accuracy (given that XGBoost is a robust classifier), without validating its reliability from a statistical hypothesis testing perspective. This arrangement may lead to a somewhat one-sided argument in the paper.

Author response:
**Q1: Does the empirical evaluation in the paper provide a compelling argument?**

Yes, we believe so. The evaluation method used in the paper to obtain our results follows the practices established in the literature on synthetic tabular data generation (e.g., [1, 2, 3, 4]). Although we also believe that providing a robust and principled statistical hypothesis test for validating the synthetic data would be ideal, this is currently not the method of evaluation in this field, due to it being fundamentally hard. Due to the high complexity and dimensionality of the involved distributions, there is currently no gold-standard method that is tractable and reliable to a degree that would allow it to be adopted by the field of synthetic tabular data research for evaluation. The closest to this comes the comparison of low dimensional marginals, which is the basic building block of our method already. All in all, we believe that strong contributions can be made here that would benefit the community, but addressing this problem is beyond the scope of our work.

In more detail, we evaluate mainly using the downstream accuracy of a state-of-the-art classifier, and, as shown in the experiments in Appendix C, we also measure the TV-distance between the learned and the true k-way marginals, effectively calculating a low-dimensional approximation of the divergence between the true and the modeled distribution. Note that we measured both metrics for all conducted experiments, merely, for presentational brevity, we opted to include the accuracy metric in the main paper, due to its easy understanding, and higher interest for practitioners. Furthermore, a strong classifier is also a meaningful proxy summarizing how well basic to very high order correlations have been preserved in the synthetic sample with respect to the original data.

As for assessing the alignment of the synthetic data with the custom specifications, we directly measure their satisfaction rates/degrees on the generated synthetic samples, providing the closest metric of interest.

Reviewer comment (Presentation):
Writing Clarity: The content in the paper's introduction appears somewhat disorganized, as it combines background introduction with an overview of the framework's methodology. It is recommended to organize the sections logically as "introduction," "related work," and "formulation" to help readers better understand the core content of the paper.

Author response:
**Q6: Please clarify the organization of the introduction section, and address other comments regarding the formatting of certain elements in the paper.**

We believe that a good presentation in a paper is extremely important, therefore, we appreciate any feedback and comments on it, and are eager to incorporate them, further improving the paper.

**Introduction:** We believe that the introduction section of our paper follows the common pattern of how introductions are and have to be structured, at least what is commonly adopted in the machine learning literature. We first start by setting a general scene to our problem (first paragraph), then we introduce the background and most related work, highlighting the research gap our work will target (the two paragraphs connected to “Synthetic data”), finally, we briefly introduce our method, structured in a short technical introduction, a working example, and a summary of our results. For some examples on this structure, see [4], [7], [12], [13], and [14], all published papers at prestigious venues. Nevertheless, we are always looking to improve our paper, therefore, if the reviewer has any concrete suggestions for improving the introduction, we would be highly appreciative of that.

Reviewer comment (Presentation):
Writing Clarity: There are instances of non-standard writing in the paper, such as lengthy formulas (page 6), pseudocode (page 5), page breaks (page 15, 20, 25), and inconsistencies in paper formatting (page 10-12). Also, it's important to address the improper use of color in tables.

Author response:
**Lengthy formulas:** Although due to the space restrictions of the paper format it will be unavoidable to sometimes have inline math formulas, we thank the reviewer for raising our awareness on readability concerns, and have uploaded a revised version of the paper.

**Pseudocode:** In general, we believe that including excerpts of code in the paper, especially if it is core to its message, is not non-standard writing. Additionally, the referenced code snippet is not just mere pseudocode, but an actual example of how specifications can be defined and passed to ProgSyn, written in a domain specific language constructed by us for this task. If this has been unclear from the presentation in the paper, we are eager to improve it, and would be very grateful if the reviewer could share their concrete suggestions with us.  

**Page breaks:** As the appendix contains a plethora of tables and other floating objects, to ensure that they are organized within the context and chapter they belong to and therefore enhancing readability and overview, we have included page breaks after certain sections. In our updated version of the paper we have tried to reduce their number while aiming to maintain structural coherence.

**Citation formatting:** We believe that the reviewer commenting on the formatting on pages 10-12 means the formatting of the citations in the references. Note that we use bibliography and other style files provided by the conference, and extract the bibtex citations for the referenced papers from google scholar, conference pages, and arxiv. As these bibtex entries may contain varying levels of information, the resulting references also may differ, yet *still conform with citation guidelines*. Nevertheless, we have uploaded a revised version of the paper, where we have post-processed the citations as much as possible using a citation cleaning tool *rebiber*, to achieve more homogeneity in the included information.

**“improper use of color in tables”:** We used color in the tables to help readability and for ease of understanding. If the reviewer has other suggestions, we welcome them.

Reviewer comment (Reproducibility):
Code: To facilitate the research framework's wider adoption, it's advisable to update the code version requirements. Personally, I encountered issues with running the code on an NVIDIA GeForce RTX 4090 with CUDA capability sm_89, and it would be beneficial to address compatibility concerns to ensure broader accessibility and usability.

Author response:
**Q7: Although the code provides an excellent resource for further understanding, some compatibility issues arose running on an NVIDIA GeForce RTX 4090.**

We appreciate the reviewer's efforts in looking at and attempting to run our code. We believe that good code writing practices and reproducibility is core to good research, and are pleased that the reviewer has recognized our efforts in this regard, listing our code as a strength of our paper. Although we thoroughly tested our code, we acknowledge that errors can occasionally occur, particularly when porting to new systems. Would it be possible for the reviewer to share the specific errors encountered, ideally paste the error message and reproduction details? Given the reviewer's description of their system, we suspect the problem arose from a driver-CUDA-pyTorch installation compatibility issue, but we would appreciate more details, as such error diagnoses ultimately can contribute also to us improving our code and descriptions. Thank you.

Reviewer comment (Experiments):
Typically, table data is more widespread and common than image data. Does the controllable generation method proposed in this paper aim to fill the missing distributions in real data, as opposed to directly applying conditional filtering within the table? Because it is cheaper to filter tabular data through simple rules than to generate data compared to other data structures.

Author response:
**Q4: “Does the controllable generation method proposed in this paper aim to fill the missing distributions in real data, as opposed to directly applying conditional filtering within the table? Because it is cheaper to filter tabular data through simple rules than to generate data compared to other data structures.”**

No, the goal of our method is to learn to generate new synthetic samples based on a source dataset. The distribution of these synthetic samples has to adhere to two often conflicting objectives: (i) has to resemble the real data distribution as closely as possible to preserve utility in replacement of the real data, and (ii) has to follow the customization set by the user, such as providing differential privacy guarantees, exhibiting decreased bias, and other custom manipulations facilitated by ProgSyn. Note also that post-hoc filtering of the data is a subideal approach, as (i) often, the motivation to create synthetic data is data scarcity, therefore we cannot afford to throw away data points, (ii) it is unclear how non-row-wise constraints, such as fairness, or statistical could be facilitated, (iii) and privacy aspects would remain unaddressed.

In general, filling in missing values in a table is referred to as data imputation in the literature, and extending existing tabular datasets with additional, varied samples is data augmentation. Although both of these problems are important and exhibit their own specialized line of work, the problem considered in this paper, synthetic data generation, follows the more general goal of learning the distribution (or a manipulated distribution as in our case) of a source tabular dataset. Note that synthetic data can be used for both of the above tasks, but it is mainly motivated by other data limitations (also highlighted in section 1 of our paper):

- lack of (high quality) data: only a few real samples are available, e.g., because the real experiments are very expensive
- regulatory considerations: storing, processing, passing on, or deriving products from the real data is forbidden by regulations (e.g., GDPR)
- privacy: the data has to be decoupled from identifiable individuals, providing formal guarantees (for instance by using differential privacy)
- bias: the real data may exhibit biases that have to be mitigated before a data release is made
- proprietary information: the real data may contain higher order proprietary information that is difficult to filter on the real data, so an altered distribution has to be learned before a data release can be made

At this point we would like to again highlight that ProgSyn is the first framework supporting **all** of these objectives.

Reviewer comment (Experiments):
In practical scenarios involving table data, recommendation systems present a more realistic application. Since relying solely on a highly generalizable model like XGBoost might not provide strong model performance validation, can ProgSyn consider further methodological validation (e.g., using models from other domains like ONN, xDeepFM, or more general models like SVM, GBDT)?

Author response:
**Q2: Would using different models to evaluate the quality of the data change the overall picture of ProgSyn’s strong performance compared to other methods?**

No, and for the sake of demonstration, we have re-run the fairness experiment on Adult adding more classifiers. In the table below we compare the performance of ProgSyn to the next best non-private method TabFairGAN on the classifiers Logistic Regression (LogReg, linear model), SVM, Random Forest (RF, tree ensemble), XGBoost (XGB, gradient boosted tree-based, used in the paper), and CatBoost (CB, gradient boosted tree-based):

|Classifier|ProgSyn [Acc. / Dem. Parity]|TabFairGAN [Acc. / Dem. Parity]|  
|:---|:--:|:--:|  
|LogReg|82.1 / 0.01|76.1 / 0.01|  
|SVM|80.0 / 0.04|76.0 / 0.00|  
|RF|81.4 / 0.01|78.8 / 0.02|  
|XGB|82.1 / 0.01|79.8 / 0.02|  
|CB|82.0 / 0.01|80.5 / 0.01|

As we can see, the overall picture is unchanged: ProgSyn prevails as the best method in terms of fairness-accuracy trade-off. Note that for LogReg and SVM, TabFairGAN is unable to produce data that is suitable for training and therefore results in classifiers that largely only predict the majority class for every instance, irrespective of any feature (including the protected feature, therefore such classifiers appear as fair, however are not useful as they are only able to assign every instance to the same class).

Further, note that [5] asserts that for evaluating synthetic tabular data, one should use a state-of-the-art classifier, as weaker models are unable to exploit the edge in distributional detail that better synthesization methods provide, painting a false picture.

Finally, we do not test on recommendation models, as, again in line with prior work, the datasets used in this paper are classification datasets. Note that this is representative, especially in the context of our work, as none of the referenced methods in which ProgSyn is positioned examine recommendation datasets, but focus mainly on classification datasets (e.g., [1, 2, 3, 4, 5, 6, 7, 8, 9]). Also, as table 10 in [10] highlights, albeit in the context of federated learning, only 2 out of 61 tabular data applications concern recommendation systems.

Reviewer comment (Novelty):
The main approach of fine tuning and adding differentiable constraints is relatively straightforward.

Author response:
We would like to thank the reviewer for their acknowledgment of our timely and strong empirical contribution and quality of writing, their informative feedback, the informative resource provided, and their favorable overall assessment of the paper. Below we address their questions and points raised:

**Q1: What novel technical and conceptual contributions does ProgSyn make?**

Additionally to ProgSyn’s empirically demonstrated strong performance and high level of versatility, we firmly believe that ProgSyn brings both significant technical and conceptual novel contributions to the field of synthetic tabular data generation.

Conceptually, it is the first work to facilitate extended customizability of the generated data, allowing for row-wise, distributional, and downstream-dependent specifications. Our framework cannot only be used to improve the synthetic data quality, but also to hand-tailor its distribution and other properties to specific, custom use-cases defined by the user. At the same time, ProgSyn effectively captures the capabilities of prior works across different aspects of tabular data generation.

Technically, ProgSyn introduces several novel elements in the differentiable computation of the specification loss-terms, tackling several difficult challenges on the way. The common key challenge of each specification type arises from the fundamentally non-differentiable nature of categorical features in the produced tables. Once the conversion from complex constraints involving discrete features is achieved, using them as a regularizer in a two-staged optimization pipeline is indeed a natural approach for manipulating the modeled distribution to our liking. Nevertheless, note that ProgSyn makes its key technical contributions addressing the challenge of arriving at these differentiable loss terms (something that is for example in [1] in the domain of time series data is treated as a given). For instance, for logical constraints and conditions, we address this through our masking scheme introduced in Section 4. Further, we are the first to offer a technical solution to incorporate complex formulas involving conditional statistical expressions acting directly on the distribution of the data, going beyond row-wise constraints. We are also first to provide a general framework of incorporating downstream classifiers in the synthetic data generation training process; tackling the difficult additional challenges of training-in-the-loop, and uniting it with the rigorous requirements of differential privacy. Notably, this allowed us to set a new state-of-the-art on fair synthetic data generation, both in the non-private and the differentially private setting, while this objective not being the main focus of our work.

Reviewer comment (Experiments):
How would you generate a variety of synthetic datasets with varying constraint specifications without retuning your model?

Author response:
**Q2: Discuss varying the constraints after the model has already been tuned for a different set of constraints.**

Yes, in principle adaptation to a new constraint set without tuning is possible, although it would be ill-advised. For constraints that apply row-wise (like logical constraints), rejection sampling can be used, however, it will come with the limitations already discussed in the paper. For the rest of the constraint types it is also possible, as the model is differentiable with respect to the input noise as well, so one may optimize the noise at sampling time to adapt to new constraints.

However, it is ill-advised, due to several factors: (i) the number of parameters in the noise is orders of magnitudes larger than the number of parameters in the model, making such optimizations slow and memory costly, and (ii) one of the most important features of ProgSyn are constraints that act on a distributional level and **not** on a row level, therefore ruling out the possibility of any guiding method relying on judging single samples at generation time being sufficient (such as for example the method used in [1], linked by the reviewer).

Nevertheless, we believe that in the domain of ProgSyn there is **no need** for tuning-free adaptation, as the framework and as such the tuning, is extremely light-weight.

Nevertheless, as we have demonstrated in Q2, as the framework is light-weight, effective adaptation can be conducted in just a few seconds, which, in our eyes, addresses the concerns coming with the necessity of potential frequent needs of adaptation. For instance, to adapt to the constraint R1 listed in Table 2, at the current parameter setting on Adult, we required around 3.5 minutes of fine-tuning at the employed hyperparameter settings.

As we did not see this as a concern during development, we did not make further optimizations. However, note that adaptation time can be linearly reduced by: (i) reducing the batch size, (ii) increasing the chunking size at marginal calculation, or (iii) decreasing the number of epochs. For the sake of the argument, we repeated the mentioned experiment optimizing for fine-tuning time, and achieve **84.7% accuracy at 100% CSR** (equal to what is reported in the paper) in a mere **13.7 seconds** of fine-tuning resulting in a **15.8x speedup**, by only adjusting three hyperparameters in favor of execution speed: reducing the number of epochs to 100, reducing the batch-size to 2000, and increasing the chunk-size to 2000.

Therefore, we believe that re-tuning the model in order to adapt it to new constraints is feasible.

**References:**

[1] A Coletta et al. On the Constrained Time-Series Generation Problem. To appear at NeurIPS 2024. https://arxiv.org/abs/2307.01717