Paper: How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data (ICLR-2024)

Reviewer comment (Writing):
From reading the paper, it is not immediately clear how the training procedure with CLs works, and whether CLs can be applied to other generative models, such as Variational Auto Encoders, Normalizing Flows or Diffusion Models.

Author response:
> **Question:**
> *How does the CL training procedure work? And how does CL affect the training?* 

**Answer:**
The basic intuition behind our layer is very simple: given a set of constraints expressed as linear inequalities, we compile our constraints into a differentiable layer (CL), which then gets added to the topology of the network itself right after the DGM layer that generates the samples. At training time, the network is trained with CL injected in the topology and the gradients backpropagate seamlessly through it.  We have added a paragraph to explicitly state that the gradients can backpropagate through CL at the beginning of Section 3.  



> **Question:**
> *Can CL be applied to other generative models?* 

**Answer:**
Yes, we applied it to TVAE [1] and GOGGLE [2], and we added the results in Table 2. As can be seen from the results, adding CL at training time results is an improvement in all metrics but one for TVAE and in all metrics for GOGGLE. As we stated in the general answer, note that the results on GOGGLE are computed on 5 out of 6 datasets. We will update the results with the final values as soon as possible.

Reviewer comment (Evaluation):
A metric to compare the data distribution from the generated distribution is missing from the experiments. Would it be possible to include a metric such as negative log-likelihood or Wasserstein distance?

Author response:
> **Question:**
> *How does the application of CLs shift the distribution of the generated data? Does it result in an "overpopulation" of the regions on the boundaries? If that's the case, the resulting distribution would be skewed from the true distribution especially when the baseline model generates many samples which violate the constraints. Adding other metrics (see weaknesses) would help with investigating this matter.*

**Answer:**
We thank the reviewer for this comment, which allowed us to study more in-depth the behavior of our C-DGMs at the boundaries. 

To conduct this study,  following the suggestion given by the reviewer and the approach of [3], we performed a comparative analysis of the distributions of data generated by the models under study and the real data by measuring the distance between the real data distribution and the generated samples' distribution. We used the Wasserstein distance for the continuous features and Jensen-Shannon divergence for categorical features. We report the results in Appendix C.2, in Tables 11 and 12. As we can see from the Tables, there is almost no difference in terms of Jensen-Shannon divergence (Table 12) between the samples generated by the DGMs, the P-DGMs, and the C-DGMs. The differences are more accentuated when studying the continuous features in terms of the Wasserstein distance, where the C-DGMs achieve the lowest distance in 12 cases (out of a total of 22 comparisons). 

While interesting, the above did not give us much insight into the behaviour of the models at the boundary. To specifically study this phenomenon, we did the following: 
- We considered the datasets and constraints for which we plotted Figures 3, 4, 5, and 6. These examples are particularly interesting, because these were the constraints for which we got the highest number of violations, and thus we should expect the overpopulation problem (if we had it) to show its worst effects. 
- We defined a band around the boundary whose width $w$ we set to be proportional to the range of the values of the considered features in the real dataset. As in all the considered constraints, only two features appear, we set $w = p\sqrt{r_1^2+r_2^2}$, where $r_1$ (resp. $r_2$) represents the range of the first (resp. second) feature, while $p$ represents the proportion of the range of values each feature can assume. For the sake of this study, we considered $p=0.01, p=0.05$ and $p=0.1$. For example, for the constraint *MaxHemoglobingLevel - MinHemoglobinLevel* $\ge 0$, we have that the feature *MaxHemoglobingLevel*'s values range between 6.8 and 17.2 (thus $r_1 = 10.4$), while feature *MinHemoglobingLevel*'s values range between 5.3 and 16.7 (thus $r_2=11.4$). The width of the band in this case for $p=0.01$ is thus $w \sim 0.154$. 
- We then calculated the percentage of data points falling in the defined band for (i) the real dataset, (ii) the DGMs and (iii) the C-DGMs. 
- We reported the results in Appendix C.1 in Table 10.

As we can see from the results, there is no overpopulation happening. Indeed, for each example considered we have that in **every single case** the average number of samples generated by the C-DGMs falling in the band is closer to the number of real data points falling in the band than the number of samples generated by the standard DGMs.
This is due to the fact that the layer is added at training time, and thus the DGMs are aware of the presence of the layer (teaching the model to distance its outputs from the invalid regions). 

Notice that the same cannot be said when the layer is used at post-processing time. In this case, the DGMs do not have access to the background knowledge during training and thus all the data points for which the DGMs commit a violation will simply end up on the boundary. While we can argue that having points on the boundary is still better than having completely unrealistic data points, and the results are still very good for a post-processing step, it is also interesting future work how to create post-processors (that thus cannot be added at training time) that do not suffer from this shortcoming.

Reviewer comment (Reproducibility):
Perhaps a point that I missed from the paper, but how do CLs affect the training? Can gradients backpropagate through these layers? Can CLs be applied to other Generative Models?

Author response:
> **Question:**
> *How does the CL training procedure work? And how does CL affect the training?* 

**Answer:**
The basic intuition behind our layer is very simple: given a set of constraints expressed as linear inequalities, we compile our constraints into a differentiable layer (CL), which then gets added to the topology of the network itself right after the DGM layer that generates the samples. At training time, the network is trained with CL injected in the topology and the gradients backpropagate seamlessly through it.  We have added a paragraph to explicitly state that the gradients can backpropagate through CL at the beginning of Section 3.  



> **Question:**
> *Can CL be applied to other generative models?* 

**Answer:**
Yes, we applied it to TVAE [1] and GOGGLE [2], and we added the results in Table 2. As can be seen from the results, adding CL at training time results is an improvement in all metrics but one for TVAE and in all metrics for GOGGLE. As we stated in the general answer, note that the results on GOGGLE are computed on 5 out of 6 datasets. We will update the results with the final values as soon as possible.

Reviewer comment (Reproducibility):
Can CLs impose constraints between categorical variables, or between numerical and categorical variables? For example, if x1 = "category 1", then x2 > 5, or similar?

Author response:
> **Question:**
> *Can CLs impose constraints between categorical variables, or between numerical and categorical variables? For example, if x1 = ``category 1'', then x2 $>$ 5, or similar?* 

**Answer:**
CL can impose constraints expressed as systems of linear inequalities. As reviewer DAiC pointed out, and as we have added in Section 2, linear constraints enjoy multiple desirable properties that make them an interesting class of constraints. Among such properties there is the fact that systems of linear inequalities always define a convex space. We use this assumption in Equation (2), where we define the lower and upper bound associated to each variable. Notice that if we allow for constraints of the type ``if $x_1 =$ *'category 1', then* $ x_2 > 5$'', then they would no longer define a convex space, and thus our method would not be applicable. Additionally, adding such constraints would ultimately correspond to allowing for disjunctions in the constraints and, as far as we know, it is still an open problem how to compile such constraints in a backtrack-free tree representation (see comments to Reviewer ikQ7).

Reviewer comment (Theory):
Are there cases in which assigning a valid variable ordering is not feasible?

Author response:
> **Question:**
> *Are there cases in which assigning a valid variable ordering is not feasible?* 

**Answer:**
 All variable orderings are always allowed, with some leading to better performance than others (as shown in Appendix C.3).

Reviewer comment (Theory):
**(Important)** The paper mentions the choice of $\lambda(i) = i$ is for ease of notation, while different orderings can drastically change the performance of a C-DGM. As an illustrative example, consider tabular data with three outputs $\langle x_1, x_2, x_3 \rangle$ with constraints: $x_2 \le 10$ and $x_1 \le x_2 \le x_3$. Now assume the generative model only produces $\langle 10, 5, 5 \rangle$. In this case, if the ordering $\lambda = \langle 1, 2, 3 \rangle$ is considered, then $CL(\tilde{x})$ would be $\langle 10, 10, 10 \rangle$ and if the ordering $\lambda = \langle 2, 1, 3 \rangle$ is considered, then $ CL(\tilde{x}) = \langle 5,5,5 \rangle$ with one being significantly closer to the generated distribution than the other. In fact, the notion of “optimality” is not well defined, as each ordering can produce a different optimal with none of them being comparable for defining an optimum. Moreover, the notion of optimality should also consider the discrepancy between the $CL(\cdot)$ outputs and the generative samples to be minimal.

Author response:
We articulate our answer in three points: 
- In a tabular dataset, the features are not ordered (contrarily to e.g., natural language datasets), and thus we can associate e.g., the variable $x_0$ to any feature. Thus, our choice of notation does not affect our results in any way.
-  The optimality is well-defined and takes into account the discrepancy between the final CL outputs and the generative samples. Let us consider the example proposed by the reviewer. As it can be seen, both the final outputs $\langle 10, 10, 10 \rangle$ and $\langle 5,5,5 \rangle$ are optimal according to our definition. Indeed, for both the final outputs there does not exist another sample such that (i) it satisfies the constraints and (ii) for each $i=1,2,3$ it associates a value closer to the original output than CL.     The very fair question the reviewer asks is essentially why are both final outputs considered optimal. To answer this question, suppose that we are given a DGM that is not able to capture the distribution of the features $x_2$ and $x_3$, while it is particularly good at capturing the distribution of $x_1$. In this scenario, the output $\langle 10, 10, 10 \rangle$ is much more desirable than $\langle 5,5,5 \rangle$. Thus, ruling out a priori the solution $\langle 10, 10, 10 \rangle$ as ``non-optimal'' would be a mistake. The fact that closeness to the original output is not the only factor to take into account when correcting a deep learning model has already been shown in the neuro-symbolic community (see e.g., [1]), where it has been shown that selecting the closest solution leads to worse performance than selecting the solution taking into account the confidence in the predictions.  We understand though that this might be counter-intutive, and for this reason we have added a paragraph in Section 3 (right above Theorem 3.5).
-  As a consequence of the point above, in Appendix C.3 a study on the impact of the variable orderings can be found, where we also propose two heuristics to decide favourable variable orderings. We could not include such a discussion in the main paper due to the lack of space. A clear reference to this section has been added to the above-mentioned paragraph that has been included in the main body of the paper.

Reviewer comment (Experiments):
**(Important)** I might have missed something but the reduction defined for constraints $\Pi$ can easily turn the number of constraints exponential, meaning that $|\Pi_1| \in \mathcal{O}(exp(|\Pi|))$. The only reason the current experiments do not hinder the performance is that the number of conditions is fairly small, to begin with. I would require datasets with a much larger number of conditions to be convinced that the post hoc method does not impact the sample generation time.

Author response:
The reviewer is right in saying that the algorithm is exponential in the number of constraints. **However**, even though the compiled representation of the constraints, in the worst case, can have an exponential size in $|\Pi|$, such representation allows us to correct the prediction with a single forward pass both at training and at inference time. Further, the representation has a depth equal to the number of features, and it is possible to parallelize the operations made at each level (as they are $max$, $min$, etc.). Given the above, it should be clear that we designed our method keeping in mind the specific requirements that arise when working with neural networks. 
Indeed, notice that the above considerations match the ``Efficiency" requirement explicitly stated in [2], whose approach (which also embeds given constraints into the topology of standard neural networks) may lead to an exponential structure but can compute a prediction in linear time in the size of the predictor. Two last observations:
- The exponential blow-up happens in the worst case, and indeed it happens when the constraints have many variables in common.
- In many cases the constraints are elicited directly from domain experts and/or from the underlying physics, and thus it is extremely unlikely for them to be deeply intertwined and involve very high numbers of variables. 

In practice, in all the domains that we considered (which have up to 31 constraints, each including up to 17 different features), we did not experience any problem with the constraints' representation, and the overhead introduced by the constraint layer was always negligible.

Reviewer comment (Novelty):
The paper only considers linear constraints. Even though it is pointed out as a limitation some extensions to non-linear constraints are quite simple. For example, by introducing polynomial features, one can add polynomial constraints to the current approach. Having one entire paper on linear constraints seems rather limited in novelty. I would suggest adding simple experiments as proof-of-concept for such extensions.

Author response:
We thank the reviewer for raising this point, which allows us to discuss the many amenable properties of linear inequalities and the difficulty of extending this line of work to constraints involving polynomial inequalities. 

Regarding the positive properties of linear constraints: 
- Linear constraints are already very expressive and are able to capture many different relations among the features. The importance of this class of constraints is underlined by the existence of entire fields dedicated to its study (see, e.g., linear programming).
- As pointed out by Reviewer DAiC, the unsatisfiability of linear constraints can be decided in polynomial time using Farkas' lemma [3].
- As shown in [4], it is possible to compile the linear inequalities in our tree representation of the constraint, which allows for a backtrack-free computation of a prediction "close" to the original one.
- A system of linear inequalities always defines a convex space, a property that we exploit in Equation (2), in order to compute a unique lower and upper bound.

We added a discussion on these properties in Section 2 below Example 2.1.  

Regarding the difficulty of extending our work to polynomial constraints:
- Even in the case of a single variable, it is not possible to find solutions in radicals for polynomials of degree higher than 4 (Abel-Ruffini theorem),
- To the best of our knowledge, no data structures allow for a backtrack-free representation (in the sense outlined above) of constraints with polynomials,
- Polynomials allow defining a non-convex space of solutions (e.g., $x^2 - 1 \geq 0$).

Reviewer comment (Experiments):
Even though GANs have achieved popularity for image generation, they are known to fail drastically for tabular data generation. That said, is there any reason why other DGMs such as TVAEs, STaSy, and TabDDPM are not considered in this study given the current limitations of GANs? For a more compelling story, it would be good to include other types of generative models as well. Especially in the post hoc experiments.

Author response:
We added our constraint layer to TVAE and included in the revised paper an experimental study for the post-hoc case, but also for the case where our layer is added at training time.
The new results are in Tables 2 and 3. As it can be seen from Table 2, adding our layer at training time improves the utility according to all metrics and the detection according to 2 out of 3 metrics. On the other hand, adding it at post-processing time results in a slight decrease in the performance according to all metrics but one, thus underlying the importance of using our layer at training time.

Reviewer comment (Reproducibility):
The reported charts and tables are thorough, but I wasn’t able to find any source code for reproducibility and a footnote claims that the code will be released upon publishing; however, I didn’t find anything to run in the supplementary material.

Author response:
We are currently trying to anonymize our code and share a link with the reviewers (as we gave priority to running all the requested experiments). We will share a link as soon as possible.

Reviewer comment (Experiments):
Tabular data synthesis methods used for the experiments are outdated. Please consider [1] GOGGLE, [2] GReaT, [3] STaSy, [4] CoDi, and [5] TabDDPM if possible.

Author response:
We included two new models in our experimental results: the suggested model GOGGLE [1] and TVAE [2] (which was requested by both reviewers ikQ7 and 72Ff). The results for both models can be found in Table 2.



Regarding GOGGLE, we reported the results for 5 out of 6 datasets in green, and we can see that our method gives an overall improvement in both utility and detection metrics, when using the constraint layer during training.
We are currently running the last experiments on the remaining dataset, along with the post-processing experimental analysis.

Regarding TVAE, we can see that adding our layer at training time improves the utility according to all metrics and the detection according to 2 out of 3 metrics.


**References:**

[1] Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. GOGGLE: Generative modelling for tabular data by learning relational structure. In Proceedings of International Conference on Learning Representations, 2022.

[2] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Proceedings of Neural Information Processing Systems, 2019.

Reviewer comment (Evaluation):
How was the performance improvement of the state-of-the-art methods after applying the proposed method?

Author response:
We included two new models in our experimental results: the suggested model GOGGLE [1] and TVAE [2] (which was requested by both reviewers ikQ7 and 72Ff). The results for both models can be found in Table 2.



Regarding GOGGLE, we reported the results for 5 out of 6 datasets in green, and we can see that our method gives an overall improvement in both utility and detection metrics, when using the constraint layer during training.
We are currently running the last experiments on the remaining dataset, along with the post-processing experimental analysis.

Regarding TVAE, we can see that adding our layer at training time improves the utility according to all metrics and the detection according to 2 out of 3 metrics.


**References:**

[1] Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. GOGGLE: Generative modelling for tabular data by learning relational structure. In Proceedings of International Conference on Learning Representations, 2022.

[2] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Proceedings of Neural Information Processing Systems, 2019.

Reviewer comment (Theory):
The issue with the paper is the proofs can be difficult to follow.

Author response:
We have reworded all the proofs to make them more readable.

Reviewer comment (Writing):
This also makes it a bit hard to flow and check.

Author response:
We have reworded all the proofs to make them more readable.

Reviewer comment (Presentation):
I think the authors can potentially frame the use of linear constraints in a better light to highlight how its logical completeness can be useful when specifying background knowledge.

Author response:
Thanks for pointing out  the logical completeness of linear inequalities, which further emphasizes
the advantage of utilizing linear constraints in our work. We have added a paragraph in Section 2 (below Example 2.1) highlighting all the positive properties of linear inequalities that we exploit in our paper.