Paper: On the Inadequacy of Similarity-based Privacy Metrics: Reconstruction Attacks against ``Truly Anonymous Synthetic Data'' (ICLR-2024)

Reviewer comment (Writing):
A thorough description of the attack is lacking in section 4 which significantly weakens the comprehensiveness of the paper. While the full Algorithm is relegated to the appendix, including a paragraph explaining the logic of the attack and where the information is being extracted would significantly improve the paper. Given the vulnerability of models trained with DP to this attack, the information is presumably coming from the metrics, but an intuitive explanation of how that occurs is missing.

Author response:
We added an extended description of the attack in Sec. 4 and provided more details in App. F. Furthermore, we updated Algorithm 1 to make it cleaner/easier to follow.

Reviewer comment (Presentation):
Figure 2 should be presented in a way where it is possible to distinguish between the different training algorithms and datasets. Enlarging the figure is suggested despite the constraint on space, since it is a key result for the paper. This is viewed as more useful than Figure 4, which is given much more space.

Author response:
We tried to improve the presentation of Fig. 2 (now Fig. 3). Basically, the top 5 lines are from Adult Small, the bottom 5 from Adult, while the middle 5 are from Census. We also reduced the space allocated to Fig. 4 (now Fig. 5).

Reviewer comment (Presentation):
The takeaway from Figure 3 is unclear. The figure is referred to only in passing and never explained. More generally, the description of the results would benefit from more thorough explanation.

Author response:
The main takeaway is that all models generate MNIST digits that are further away from the train data compared to the distance between test data and train data (min distance is 6). This is the main reason $SampleAttack$ performs so poorly (0% for all models). We explain this in Sec. 5.1.1.

Reviewer comment (Evaluation):
The term 'outliers' is never clearly defined, yet it is the denominator for all success rates reported. It is important to explain what this set of data points is and how much of the training data they make up.

Author response:
We added more details about how we define outliers for every dataset in App. C.2. While there are various definitions of outliers (Carlini et al., 2019a; Meeus et al., 2023), we opt for a simple and intuitive selection of roughly 10% of the train data as outliers, aiming to cover various scenarios. We also explain better how the adversary fits the $OutlierDetector$ and decides on the outliers regions in App. F.

Reviewer comment (Theory):
Stating that the attack works on models trained with DP in the abstract/intro is true in a literal sense, but feels somewhat disingenuous because, as noted later in the paper, the information comes from statistics released without DP. Any model formally satisfying DP would never be able to look directly at the private data in this way.

Author response:
We agree and do state that training the generator while satisfying DP but allowing unperturbed queries to the metrics breaks the end-to-end DP pipeline -- please see Sec. 5.2.1. We rephrased our statements to make this point more clear in the Abstract and Sec. 1. Our motivation for combining DP model training and direct/unperturbed metrics access is to emulate the product offerings of some companies in practice. We also made this point more clear in Sec. 5.2.1 and App. 1.

Reviewer comment (Presentation):
The paper has a strong attack, but the results could be better presented to effectively communicate the key message to the synthetic data community. Ideally, there could be a figure that makes it impossible to ignore the weaknesses of synthetic data generation, but the current figures are overly information-loaded, causing the key takeaway to be lost.

Author response:
We introduced Fig. 1, which hopefully manages to demonstrate the success of our attack as well as the weakness of the privacy metrics.

Reviewer comment (Presentation):
The attack needs to be described in the main paper. One suggestion to make room for this could be to cut the incredibly verbose evaluation section down to summarize the key points over all datasets with a few interesting anomalies. As the paper is currently written, we read the evaluation with little idea of what is being evaluated.

Author response:
In the revised paper, we add an overview of the main algorithmic steps describing how the attack works in the main paper in Sec. 4.

Reviewer comment (Reproducibility):
The algorithm needs to be described more formally as to the exact set operation that takes place during filtering and the exact branching conditions based on the metrics.

Author response:
We added more details of the algorithmic steps and clarified Algorithm 1 in App. F.

Reviewer comment (Theory):
Assuming the adversary can query any dataset with the metric seems far-fetched. Although the paper argues this is the case in practice, it would be easy for companies to restrict this.

Author response:
First, we would like to clarify that generating unlimited synthetic data from a trained generator is essentially a universal selling point of synthetic data providers. Second, for many providers, similarity-based metrics are the only way they demonstrate/guarantee the privacy of the product to their end users, which makes access to the metrics crucial. We clarified this in App. A. As confirmed by our direct experience with these companies (details had to be omitted to preserve double-blind submission), in commercial settings, restricting access to these metrics would severely limit the desirability of these products.

We would also like to highlight that there are numerous research papers and whitepapers by companies like AWS that rely entirely on these/similar metrics to demonstrate the privacy of their models (we reference several such studies in Sec. 1 and Sec. 2). Finally, unlimited querying to a model/dataset is, for better or worse, common in production settings -- like for the Diffix system (Pyrgelis et al., 2018; Gadotti et al., 2019; Cohen & Nissim, 2020a), as discussed in Sec. 6.

Reviewer comment (Experiments):
In Section 5, it seemed the attack was adaptive (The search attack is deployed based on the performance of the Sample attack). However it was not clear how the attack could know the reconstruction rate of the previous component without access to the training data.

Author response:
We now added more details in Sec. 4 and App. F. As discussed in Example 2 in App. E and explained in App. F, we can use $SampleAttack$ to trick the metrics to output the exact distance of a given synthetic record to its closest train record. Thus, we can detect exact matches in the first phase of the attack through the metrics without having access to the train data.

Reviewer comment (Novelty):
While the attack is useful for a proof of concept, it doesn't have any merit in the case that the privacy metrics are not leaked. Thus, the argument that it outperforms related work in terms of its lack of assumptions is a little misleading.

Author response:
We rephrased the introduction of the attack to mention that it is a proof of concept attack designed to specifically expose the weaknesses of the metrics in Sec. 1.

Reviewer comment (Theory):
The section about the effect of DP is somewhat a moot point as it can not be DP while allowing unperturbed queries about the training dataset (the metrics). I somewhat get the idea to show that even if they decided to use DP on the generator, it still wouldn't be enough with the metrics being present.

Author response:
We agree and do state that training the generator while satisfying DP but allowing unperturbed queries to the metrics breaks the end-to-end DP pipeline -- please see Sec. 5.2.1. We rephrased our statements to make this point more clear in the Abstract and Sec. 1. Our motivation for combining DP model training and direct/unperturbed metrics access is to emulate the product offerings of some companies in practice. We also made this point more clear in Sec. 5.2.1 and App. 1.

Reviewer comment (Experiments):
Perhaps more interesting would be to show what happens if the metrics are released with DP (an actual defence). Although epsilon already gives an indicator of how private the synthetic data is, the best defence would be removing the metrics entirely.

Author response:
We are not sure this will scale as querying the metrics in a DP way would eventually deplete the DP budget. This means that no more synthetic datasets would be able to be generated, which, as mentioned, contradicts one of the main selling points of synthetic data providers.

In principle, we agree. Indeed, the best defense would be to train the model in a DP way and not accompany data releases with metrics. However, as mentioned earlier, for many companies (and research papers), these metrics are the only way to demonstrate/guarantee privacy to customers and users and that is why in practice they are released in the first place.

Reviewer comment (Writing):
Consider using parenthesis around citations, as some sentences are hard to parse when filled with citations that are not distinguished from normal text.

Author response:
We updated all citations.

Reviewer comment (Presentation):
2D gauss was missing from Table 1 despite section 5.1.1 referring to it.

Author response:
We included 2d Gauss in Table 1.

Reviewer comment (Evaluation):
No mention of how the ground truth outliers are computed in the evaluation.

Author response:
We added more details about how outliers are defined for every dataset in Sec. 2 and App. C.2.

Reviewer comment (Presentation):
Figure 5 is very hard to read. The shading is subtle and changes the colour to make it even more confusing. Perhaps shapes for each epsilon and make them shaded or hollow for the two attacks?

Author response:
We followed the advice and updated Fig. 5.

Reviewer comment (Experiments):
The attack results are contingent on the definition of outliers. Both the targets of the attack (outliers) and the test records (over which the attack success rate is computed) are constructed using similar clustering methods. This may lead to over-optimistic results. Have the authors considered (1) evaluating their attack on a dataset that has not been pre-processed by them to add outliers or (2) identifying outliers (to be used as the test records) with a state-of-the-art approach such as https://arxiv.org/abs/2306.10308?

Author response:
> The attack results are contingent on the definition of outliers. Both the targets of the attack and the test records are constructed using similar clustering methods. This may lead to over-optimistic results.

We add more details about how we define outliers for every dataset in App C.2 (including new references to (Carlini et al., 2019a; Meeus et al., 2023)). While the adversary follows the same steps as the provider to determine the outliers regions, we would like to clarify two points: 1) the targets of the attack are the outliers from the train data, not the test data; 2) there is no leakage from the provider to the adversary, i.e., the adversary fits $OutlierDetector$ on their own sample of synthetic data (using the fitted generative model) without having access to any of the train/test data. Therefore, if the adversary does not do a good job with $OutlierDetector$, they will reduce the attack’s performance. All these, also combined with our experiments in Sec. 5.2.2 and App. G (in which the adversary reconstructs large proportions of any train data), lead us to believe that our results could actually serve as a lower bound on the number of records that could potentially be reconstructed.

Reviewer comment (Evaluation):
It’s unclear which of the similarity-based metrics is used in the empirical evaluation.

Author response:
> It’s unclear which of the similarity-based metrics is used in the empirical evaluation.

The adversary in Sec. 5 has access to the 3 privacy metrics (IMS, DCR, NNDR). In the revised paper, we highlight this in Sec. 2 and Algorithm 1 in App. F. We do so to emulate the real-world product offerings of some companies.

Reviewer comment (Novelty):
Missing (conceptual and/or empirical) comparison to the reconstruction method of Dick et al. Can the authors describe (add a paragraph on) how their reconstruction method relates to Dick et al.? Even though the methods operate on different objects (aggregates in Dick et al. as compared to synthetic data here), they both involve repeated generation of datasets of “candidate” records and filtering of these records in some way to only keep the most likely records.

Author response:
> Can the authors describe (add a paragraph on) how their reconstruction method relates to Dick et al.?

To initiate their successful attack, Dick et al. (2023) need access to the publicly known data distribution and true answers to a number of queries. They use this information to generate multiple synthetic datasets using a RAP variant (Aydore et al., 2021), ranking them based on the number of duplicated rows. In contrast, our attack does not require such any information, except for the privacy metrics. Additionally, our method has a black-box access to a single generative model which does not depend on preserving the actual statistics of the data (which is often the case for the models we evaluate).

Reviewer comment (Writing):
Unfinished sentence on page 3: The test passes if [...]-5th percentile is larger or equal.”

Author response:
> Unfinished sentence on page 3

We fix the sentence.

Reviewer comment (Evaluation):
Is it possible, using the authors' method, to assign a confidence or likelihood to each reconstructed record?

Author response:
> Is it possible, using the authors' method, to assign a confidence or likelihood to each reconstructed record?

Since the metrics are deterministic and unperturbed (as discussed in Sec. 3), the adversary can be 100% confident for each reconstructed record. We tried to allude to that in the take-aways in Sec. 4.

Reviewer comment (Experiments):
Why do the authors focus on outliers only? Intuitively, an average record is more likely to be generated by the synthetic data model than an outlier record. The other leading reconstruction method based on repeated dataset generation, developed by Dick et al. against aggregate statistics, doesn’t explicitly target outliers.

Author response:
> Why do the authors focus on outliers only? Intuitively, an average record is more likely to be generated by the synthetic data model than an outlier record.

We agree that the average record is more likely to be generated and thus probably easier to reconstruct. To confirm this hypothesis, we run new experiments, which we discuss in App. G. In short, reconstructing any train records is indeed much easier.

However, in the main body of the paper, we focus on outliers because they likely correspond to the most vulnerable individuals, they are more challenging to attack (which puts our attack to the test), are more susceptible to memorization, and are specifically highlighted by regulators (we also discuss this in Sec. 4). Therefore, if we manage to reconstruct them successfully, this will constitute a clear and serious privacy violation.

Reviewer comment (Evaluation):
What do the authors mean by “all three tests pass” (relating to the metrics)?

Author response:
> What do the authors mean by “all three tests pass” (relating to the metrics)?

Every metric is associated with a statistical test. The passing criteria are either the average or 5th percentile comparison between ($D_{train}, D_{test}$) and ($D_{train}, D_{synth}$). We made this more clear in Sec. 2.