Paper: Structured Evaluation of Synthetic Tabular Data (ICLR-2024)

Reviewer comment (Evaluation):
The representation in Figure 1B, specifically regarding the spectrums, may benefit from further context or an enriched explanation. This raises a query: Are the authors implying that model-free evaluations, such as those estimating marginal or pairwise relationships, may not provide a holistic perspective? Is there an inherent advantage in adopting model-based techniques, like PCC, to act as proxies for real data while assessing the same metrics? Moreover, given that PCC operates as a synthetic model, does its role in the evaluation process imply a comparison between synthetic models through another synthetic standard? Gaining clarity on these nuances would greatly enhance understanding.

Author response:
**Elaboration of the PCC-based metric:**
We agree that it is helpful to have a deeper elaboration on the distinction between model-free and the PCC-based metric, as well as an enriched explanation on why one would like to have the model-based metrics. To achieve that, we will add a summary of our responses below to the Introduction and Section 3. The response below is also our point by point response to the specific questions raised by the reviewers.

Whether the evaluation is holistic or not depends on whether the metrics provide coverage of the *full joint distribution*. This means that evaluating only marginal and pairwise relationships would not provide a holistic perspective, because higher order statistics such as 3-way interactions would not be captured by the evaluation. Using metrics that target the full joint distributions (i.e., LOO or full-joint) could in principle provide a holistic picture. This is true regardless of whether the metrics are model-free, model-based, or a combination of the two. The issue with the model-free metrics is that they are often noisy, especially towards the RHS of the spectrum for the LOO and full-joint metrics, as shown by the large error bars and non-monotonicity in panels A and C of Figures S1–S3. The large error bars and non-monotonicity originate from (1) the use of multiple ML models with different expressiveness, as well as (2) the different types of estimators and scores used for different distributional properties and data types. For example, the full-joint metrics were evaluated indirectly via a discriminator, which also included a linear version (i.e., logistic regression) and a simple nonlinear one (i.e., SVC). In other words, the model-free metrics are *incoherent* in terms of the estimators (e.g., the underlying ML models used) and scores. In contrast, the model-based metrics neither have the incoherence problem, nor exhibit the noisy and non-monotonicity, as evidenced in panels B and D of Figures S1–S3. Thus, the inherent advantage of the model-based technique is the coherence in methodology, that is, all the metrics can be derived from the same type of estimator and score.

The reviewer is correct that the model-based technique puts the evaluation on another synthetic standard. The worry is then whether this synthetic standard would bias the evaluation. The degree of bias depends on how much the surrogate model would mask statistical properties because of a lack of expressiveness. To illustrate, if the surrogate model captures only the marginal distributions but not the pairwise and higher-order dependencies, it would assign similar likelihoods to the PERM baseline and the real dataset. We have two pieces of evidence that PCC does not suffer from such lack of expressiveness. First, PCC is consistently a top data synthesizer as evaluated by the model-free metrics (panel A in Figure S1–S3). Second, the model-based evaluation derived from PCC does not always put itself as the highest performing model but can capture superiorities of other techniques (panel B in Figure S1–S3). In fact, the ordering of the performance of the different data synthesizers is consistent between the PCC-based and model-free evaluations.

Reviewer comment (Novelty):
It would be illuminating to discern how this work either mirrors or deviates from established frameworks in previous literature. While the authors' initiative to broaden the metrics spectrum and introduce a surrogate model approximating real-data probability distribution is commendable, elaborating the distinct facets or innovative insights of the author's proposal, especially vis-à-vis findings in [1, 2] referenced in the Questions section, could accentuate the originality and significance of the research amidst prevailing knowledge.

Author response:
**Compare and contrast with refs [1,2]**:
We thank the reviewer for bringing ref [2] to our attention. (Ref [1] was cited already.) Below we make a vis-à-vis comparison between our approach and refs [1,2].

What we and ref [1] have in common:
- The “attribute fidelity” and “bivariate fidelity” in ref [1] match the semantics of our marginal group and pairwise group, respectively.
- Two of the “population fidelity” in ref [1], namely “Distinguishability type metrics” and “The log-cluster metric,” can fall under our joint-distribution group.
What we have that ref [1] does not have:
1. Ref [1] does not define an overarching mathematical objective nor an explicit spectrum of evaluation coverage. It is only with a clearly defined objective and spectrum that one knows whether the evaluation covered the objective in full.
2. Ref [1] places ML efficacy in its own category as a Narrow application-specific measure, whereas we show that ML efficacy can be interpreted as a Broad resemblance-based measure targeting a leave-one-out conditional distribution. Without this interpretation, the evaluation may require trading off between multiple objectives. With the reinterpretation, all the metrics can be ordered according to their structural complexity and be used together in the same space, as shown in our Figure 2.
3. Ref [1] places both the “cross-classification metric” and ““Distinguishability type metrics” under the population fidelity, whereas we would identify the former as targeting the LOO distribution and the latter as targeting the full-joint. This distinction may be useful for metric selection and improvement.
4. Ref [1] places the “Difference in Empirical distributions type metrics” under population fidelity, but our framework points out that this metric is really about the type of estimator used rather than the distributional property targeted. Again, this distinction may be useful for metric selection and improvement.
5. Ref [1] places the “likelihood metrics” under population fidelity, whereas our framework has identified it as an orthogonal dimension to distributional property, and has formed an entire model-based spectrum of metrics out of it.

What we and ref [2] have in common:
- The use of the words “marginal,” “pair,” and “joint” in ref [2] coincides with how we use them in our paper. Some of the “marginal” and “pair” metrics also match the metrics we use in the marginal and pairwise groups.
What we have that ref [2] does not have:
- Same as points 1 and 2 above.
- Ref [2] does not have model-based metrics. Model-free metrics avoid the issues of needing different estimators and scores for different data types and distributional properties, and are hence more coherent.
What ref [2] has that we do not have:
- Their joint measure of “distance to closest record” is an interesting reinterpretation of what is typically used to evaluate privacy. Their measure of “likelihood approximation” as an extension of “distance to closest record” to the test dataset is an interesting idea for testing the joint distribution. We did not include distance-based metrics because their natural range is not [0,1]. This makes it hard to combine them with the other metrics through an average.

In summary, our work is unique in that we provide a clear objective and spectrum of evaluation which, for the first time, allows us to:
- reason about the completeness of the metrics;
- reason about the coherence of the all the metrics, not as a multidimensional evaluation, but as a structured evaluation where we can order, visualize, and combine all the metrics in a principled way;
- develop a spectrum of coherent model-based metrics that are free from the need to use different estimators and scores for different data types and distributional properties.

We will summarize these points related to refs [1,2] in the Related Work section with a pointer to a new Appendix E, work the above summary points into the Introduction, and update Figure 1 to show that the “Population metrics” in ref [1] spreads across LOO and the full-joint group.

Reviewer comment (Writing):
In section 3, the authors mentioned that “the objective of the data synthesiser is \( Q=P \)”. While the underlying objective might be to highlight the close similarity between the distributions, stating it in this manner might lead some readers to interpret this as \( Q \) being an exact replica of \( P \). Given the paper's central theme of using \( Q \) as a more private alternative to \( P \), such an interpretation could be seen as contradictory. It might be clearer to emphasize that \( Q \) is intended to be statistically analogous or mirrors the distribution of \( P \), signifying that while \( Q \) captures the broader statistical characteristics of \( P \), individual data points might differ, ensuring privacy. A more detailed description or clarification in this section could be beneficial for enhancing the reader's understanding and mitigating potential misconceptions.

Author response:
**Clarifying Q = P but S $\neq$ X**:
Following the reviewer’s suggestion, we have changed the first sentence in the first paragraph of Section 3 to say: “The goal of a data synthesizer is to produce samples that are statistically analogous to the real data but not direct copies of them.” We have also revised the paragraph to put “Q = P” and “S $\neq$ X” right beside each other. Note that in the original paper we already tried to make this point. For example, in the Introduction we say “Our framework stems from a formal objective that a synthesizer should produce *samples from the same joint distribution* as the real data.” In the privacy subsection, we say “While we would like to have Q = P , we do not want the synthesizer to simply memorize the training data, resulting in S = X.”

Reviewer comment (Evaluation):
The presentation of the leave-one-out (LOO) metric seems to bear a resemblance to the dimension-wise prediction performance metric as described in references [3, 4], as well as the all model's test metric outlined in [2]. The authors should clarify whether these are synonymous or if there's a discernible distinction between them.

Author response:
**On refs [3,4]**:
We thank the reviewer for pointing out the column-wise prediction in refs [3,4]. Column-wise prediction as presented in [3] is surely a kind of LOO. The difference between dimension-wise prediction and ML efficacy is the following: Let the model trained on the synthetic data be $M_S$, and the model trained on the real data be $M_R$. In column-wise prediction the comparison is between the predictions of $M_S$ and $M_R$, whereas in ML efficacy the comparison is between the predictions of $M_S$ and the corresponding column of the real data. Also, ML efficacy does not require splitting the data into a train and a test set, while the dimension-wise prediction uses a split. We will be happy to include column-wise prediction into the paper if this is crucial for increasing the rating of the paper.

Reviewer comment (Experiments):
Rather than depending on a surrogate model to estimate ground truth, it might be more reliable to employ a distinct hold-out test set, ensuring it retains the same distribution as the real (observed) data. While this approach might pose challenges when dealing with limited samples, methodologies like k-fold validation could be explored to compute an average score over several iterations. Alternatively, having a baseline that shows the performance of the surrogate on a hold-out test set could serve as the acceptable error threshold.

Author response:
**On hold-out test set**:
We are not sure how a hold-out test set would function in place of the surrogate model. The main difficulty in evaluating synthetic data lies in the difficulty to compare two datasets directly, but the PCC as a surrogate model can summarize any subset of a dataset with a number (the subset’s likelihood) to allow direct comparison between datasets. If the reviewer is referring to the dimension-wise prediction of [3,4], we agree that having a hold-out test set is a valid approach for evaluating the LOO distribution. Or perhaps the reviewer is referring to the “likelihood approximation” in [2], which is surely a valid model-free approach for evaluating the joint distribution. In fact, the “likelihood approximation” may be a more bonafide model-free approach, as it relies only on computations of Euclidean distances instead of training a discriminative model. The performance of the surrogate on a hold-out set can be seen in Figures S1–S3’s panel D with the HALF baselines. Please let us know if we are misunderstanding the question.

Reviewer comment (Writing):
The current presentation of details incorporates a variety of symbols, which, while comprehensive, can sometimes add complexity to the narrative without necessarily enhancing clarity. To improve readability and facilitate a deeper understanding for readers, a dedicated subsection early on could be introduced to familiarize readers with the notation. Within the main text, notation should be used only when it brings forth novel information, and plain language descriptions should be relied upon when the content permits. For example, the passage: "We then use the surrogate model to compute   \{ \hat{P(X_i) \mid i=1,..,n \}, which is the likelihood of  X_i…" could be more intuitively conveyed as: "We use the surrogate model to determine the likelihood of the real data samples under this model." If the precise mathematical formulation is essential, placing it in a distinct equation block for reference could be beneficial.

Author response:
**On improving readability and clarity**:
We thank the reviewer very much for the suggestion to improve the readability paper with such a detailed example! Following the suggestion, we will separate as much as possible the plain English description from the mathematical notation, and lead with the plain English description.

Reviewer comment (Novelty):
In section 3.3, the discussion surrounding the pp-plot could benefit from further clarity. There is a question as to whether the likelihood estimate method introduced is akin to the "Distance to Closest Record" concept mentioned in [5], where a Nearest Neighbours model is employed to gauge the likelihood of each synthetic data originating from the real data distribution. The primary distinction might be the use of the Probabilistic Cross-Categorisation model. Elucidation on this comparison would be invaluable for readers familiar with the referenced methodology.

Author response:
**On likelihood estimate via Distance to Closest Record**:
We thank the reviewer for asking this interesting question. There are several major distinctions between our model-based likelihood estimate and that based on distance to closest record (DTCR). Empirically, being close to a real data point and being a likely real data point is often not the same. Consider a normal distribution. If a synthetic data point *s* is very close to a real data point at the tail of the normal distribution, DTCR would assign *s* a high likelihood, whereas a gaussian model would assign it low likelihood. Conversely, if a synthetic data point is very close to the mode but there are no real data points as close to the mode, the DTCR would assign a lower likelihood than the gaussian model would. From a more theoretical perspective, the likelihood constructed from DTCR with Euclidean distance is similar to that constructed from a model built using kernel density smoothing over the training data with a Gaussian kernel of standard deviation 1. In contrast, the likelihood constructed from PCC follows Equation (1) in the paper. For continuous variables, PCC learns a Dirichlet Process Gaussian Mixture model with Gaussian prior on the mean and scaled inverse chi-squared prior on the variance (see Appendix B), which can be interpreted as performing Bayesian Occam razor on the kernel density smoothing approach. We now mention this in the Related Work with a pointer to the more detailed discussion in Appendix E.

Reviewer comment (Evaluation):
Given that the evaluation encompasses a diverse range of metrics within the same family, such as marginal, pairwise-based, leave-one-out conditionals, full-joint-based, missingness, and privacy, it might be insightful for readers if a correlation plot is provided. Such a plot could help elucidate potential correlations among metrics both within the same group and across different groups, offering a comprehensive perspective on the interplay of these metrics and their potential overlaps or distinctions.

Author response:
**Others**:
Following the reviewer’s suggestion, we will show the correlation plots of the individual metrics within each structural group as well as correlation plots of the group metrics across the structural groups. These will be added to Appendix D and referred to in Section 5.

Reviewer comment (Novelty):
Potential missing reference: There appears to be an omission in the paper's review of related literature. In particular, ref. [2] in its section 3 emphasizes the significance of evaluating synthetic tabular data generators across various metrics, including marginal-based, column-pairs, joint, and utility considerations. The thrust of these discussions in [2] bears a strong resonance with the core objectives of this paper. It is surprising and noteworthy that such pertinent work is not cited or discussed in the current paper's related work section.

Author response:
**Others**:
We now include references [2,3,4,5] in the paper. Reference [1] was in the paper already.

Reviewer comment (Evaluation):
Many of the metrics presented in Table 1 are not covered at all in the main text.

Author response:
**On rewriting and implementation feasibility**:
To help with readability, we will revise Section 3 such that the intuitive description in plain English is well separated from complex mathematical notation. We now also make sure the main text mentions all the metrics in Table 1 with references to the Appendix. Our intention in presenting Table 1 was not to provide the recipe for constructing a structured evaluation but to provide a summary of the structured framework and what it covers. Upon acceptance, we will open source our implementation, as mentioned in the list of contributions, so that any reader can use and build on it, and to ensure feasibility. To emphasize this point, we add in the caption to Table 1 that the code is at <url TBA>. Given this readily available codebase, we believe it reasonable and important to use ALL the metric groups instead of a subset of them. Section 5 and Figure 2 show precisely how all the metrics can work together to form a coherent and complete picture of the evaluation via the naturally ordering induced by the structured framework.

Reviewer comment (Theory):
Questions regarding how the ML efficacy belongs to the substructure "leave-one-out conditional distribution": leaving-one-out assumes that the target is a single column, but in many cases the target might not be a single column. How would this affect the thesis?

Author response:
**On ML efficacy and LOO**:
We thank the reviewer for bringing up these questions. If the target includes multiple columns, we can simply generalize the leave-one-out (LOO) to leave-n-out (LNO). The implication on how LNO is related to the objective of Q = P is the same as LOO, because all the dependencies described by the chain rule factorization would still be covered. Its natural ordering would be after the LOO and before the full joint when added to Figure 2. The reviewer is correct that if ML efficacy is tested on a single column, it would not be sufficient. This is why in our implementation we loop through all the columns. We now make a note that this is a slight generalization from how people usually use ML efficacy in the Leave-one-out subsection. Lastly, the binary problem is a special case of the classification problem with 2 classes, so the argmax would cover the binary case. Appendix A.3 lists the ML models used for both the binary and categorical classification. The reviewer is correct that the argmax would not be used for a regression problem. We did not include ML regressors because their measure of accuracy (i.e., RMSE) does not fall inside the range of [0,1], and hence, not in the right scale to be averaged with the other metrics. To fill this lack, the model-based LOO metrics is agnostic to column types and do cover all the columns. This is a strength of the model-based metrics that we now highlight explicitly in various parts of the paper.

Reviewer comment (Theory):
The implication is stated to hold if this is done for all $j$s (i.e., leaving out all columns). The ML efficacy test leaves out a single column, so how is sufficiency obtained in this case?

Author response:
**On ML efficacy and LOO**:
We thank the reviewer for bringing up these questions. If the target includes multiple columns, we can simply generalize the leave-one-out (LOO) to leave-n-out (LNO). The implication on how LNO is related to the objective of Q = P is the same as LOO, because all the dependencies described by the chain rule factorization would still be covered. Its natural ordering would be after the LOO and before the full joint when added to Figure 2. The reviewer is correct that if ML efficacy is tested on a single column, it would not be sufficient. This is why in our implementation we loop through all the columns. We now make a note that this is a slight generalization from how people usually use ML efficacy in the Leave-one-out subsection. Lastly, the binary problem is a special case of the classification problem with 2 classes, so the argmax would cover the binary case. Appendix A.3 lists the ML models used for both the binary and categorical classification. The reviewer is correct that the argmax would not be used for a regression problem. We did not include ML regressors because their measure of accuracy (i.e., RMSE) does not fall inside the range of [0,1], and hence, not in the right scale to be averaged with the other metrics. To fill this lack, the model-based LOO metrics is agnostic to column types and do cover all the columns. This is a strength of the model-based metrics that we now highlight explicitly in various parts of the paper.

Reviewer comment (Theory):
To define the score, the function argmax is used. What happens if the problem is a regression problem or binary?

Author response:
**On ML efficacy and LOO**:
We thank the reviewer for bringing up these questions. If the target includes multiple columns, we can simply generalize the leave-one-out (LOO) to leave-n-out (LNO). The implication on how LNO is related to the objective of Q = P is the same as LOO, because all the dependencies described by the chain rule factorization would still be covered. Its natural ordering would be after the LOO and before the full joint when added to Figure 2. The reviewer is correct that if ML efficacy is tested on a single column, it would not be sufficient. This is why in our implementation we loop through all the columns. We now make a note that this is a slight generalization from how people usually use ML efficacy in the Leave-one-out subsection. Lastly, the binary problem is a special case of the classification problem with 2 classes, so the argmax would cover the binary case. Appendix A.3 lists the ML models used for both the binary and categorical classification. The reviewer is correct that the argmax would not be used for a regression problem. We did not include ML regressors because their measure of accuracy (i.e., RMSE) does not fall inside the range of [0,1], and hence, not in the right scale to be averaged with the other metrics. To fill this lack, the model-based LOO metrics is agnostic to column types and do cover all the columns. This is a strength of the model-based metrics that we now highlight explicitly in various parts of the paper.

Reviewer comment (Theory):
Regarding the full joint substructure, the authors write "Sufficiency is hard to show because...". The reviewer asks whether it is sufficient, and whether it is necessary.

Author response:
**On sufficiency**:
We thank the reviewer for pointing out the vagueness about the sufficiency related to the joint-distribution metric. Because we cannot show sufficiency, we now say it is likely not sufficient because there may be multiple ways that the implicit $f_q$ and $f_p$ can produce a discriminator to achieve perfect ROC AUC. The condition is necessary for Q = P, which we mention in the main text just before the text quoted. The approach we show necessity and/or sufficiency was outlined in the “Sketch of analysis” subsection in the beginning of Section 3. We now add a few sentences in that sketch to describe in more detail our approach to show necessity and/or sufficiency.

Reviewer comment (Theory):
For all the substructures, the reviewer would have liked to see a much more structured approach to showing sufficiency and necessity.

Author response:
**On sufficiency**:
We thank the reviewer for pointing out the vagueness about the sufficiency related to the joint-distribution metric. Because we cannot show sufficiency, we now say it is likely not sufficient because there may be multiple ways that the implicit $f_q$ and $f_p$ can produce a discriminator to achieve perfect ROC AUC. The condition is necessary for Q = P, which we mention in the main text just before the text quoted. The approach we show necessity and/or sufficiency was outlined in the “Sketch of analysis” subsection in the beginning of Section 3. We now add a few sentences in that sketch to describe in more detail our approach to show necessity and/or sufficiency.

Reviewer comment (Theory):
The reviewer asks whether there is proof of the upper and lower bounds provided by the baselines.

Author response:
**On the HALF baseline**:
The idea of the HALF baseline is to split the real dataset into two, and then run the evaluation to compare the two datasets. It is useful because this ensures that both datasets are from the real data generating distribution, yet not direct copies of each other. In other words, the HALF baseline is the closest thing to meeting the proposed objective of Q =P and S $\neq$ X. Thus, it should provide the desirable target to the metrics. We agree that “upper bound” is probably not the right word for the HALF baseline, because the SELF baseline beats it. We now change the “upper bound” associated with the HALF baseline to the “target value” in the text. For the SELF baseline, the upper bound on fidelity metrics follows from the fact that the synthetic and real distributions are equal (Q=P), and the lower bound on the privacy metrics follows from the fact that the synthetic dataset is a direct copy of the real dataset (S=X). How the metrics relate to Q=P is summarized in the Implementation column in Table 1. Also, the second row of Figure 2 shows that empirically, the SELF baseline is an upper bound, and the HALF baseline is also higher or equal to the methods.

Reviewer comment (Writing):
On page 6, the authors write "We then use pp-plot to compare these two lists of probabilities." The reviewer asks what a pp-plot is and why it is used.

Author response:
**On pp-plot**:
A pp-plot is a plot that plots two cumulative distributions against each other. It is used to detect how different the two distributions are. If the two distributions are the same, the plot will follow the x=y line. We will revise Section 3.3 to clarify this.

Reviewer comment (Evaluation):
In Table 2, Quality is not reported for 2 models on the census dataset. The reviewer asks why.

Author response:
**Others**:
We omit GReaT on the census dataset because we were not able to finish training given our computational resources. We omit DDPM on the census because of its poor quality (always generating the same category for categorical columns and extreme values for continuous columns). We mention this in the caption of Table 2, but now also mention it in the caption of Figure 2.

Reviewer comment (Writing):
There is a typo on page 5 in the Missingness paragraph: $Q_v(c_j) = P_v(c)j)$.

Author response:
**Others**:
We have fixed the typo on page 5. Thank you.

Reviewer comment (Presentation):
The reviewer suggests adding a bird-eye view of subsection 3.2 rather than just starting to describe baselines one by one.

Author response:
**Others**:
We have added an introductory sentence to Section 3.2 as suggested.

Reviewer comment (Presentation):
The reviewer suggests putting tables in the right format (i.e., as large as text).

Author response:
**Others**:
We have restructured Table 1 to fit the text width by removing the Target substructure column and placing it as rows. We have also combined some metrics into one and simplified the estimates to describe f_q and not f_p for brevity. To make Table 2 fit the text width, we removed the decimals if the figure is greater than 3 digits.