Paper: Black-Box Privacy Attacks Against GANs via Detector Networks (ICLR-2024)

Reviewer comment (Novelty):
The proposed detector-based attack is the same as case 1 of the discriminative setting in Section 4.4 of LOGAN (Hayes et al. 2018).

Author response:
We thank the reviewer for bringing this to our attention. We have updated the discussion in the draft to reflect this relationship to prior work. Although this diminishes the algorithmic novelty of our approach, we actually think that this **strengthens the impact of this particular paper.** There are two components to this, the algorithmic novelty, and the experimental finding. 

**Experimental Findings relative to LOGAN**: LOGAN studies training a detector on generated samples on CIFAR10 as we do in our Image GANs section and finds: “In Fig. 7, we plot the accuracy results for both settings, showing that the attack fails with both datasets when the attacker has only test set knowledge, performing no better than random guessing.” So the verdict on these black-box detector attacks from LOGAN is that they do not work. This stands in contrast to our findings, which is that in some settings, and particularly on genomic data, they do work very well. One reason is that since the LOGAN paper in 2018, our understanding of how to properly evaluate MIAs has evolved significantly since https://arxiv.org/abs/2112.03570 — rather than evaluating the accuracy of the MIA as in LOGAN, recent work including ours focusing on plotting the full ROC curve, and evaluating the TPR at low fixed FPR, which corresponds better to actual privacy risk for a small subgroup. By this metric on genomic data, the ADIS attack achieves a 10x improvement over the random baseline in some settings! On CIFAR10 our findings match LOGAN; although we do beat the random baseline at low FPRs for some detector-based methods, our improvements are modest, and worse than distance-based methods. **Since prior work concludes detector based attacks are not effective, the fact that our work shows they work very well on tabular data adds importance nuance.**

**Novelty relative to LOGAN:** In addition to the important experimental findings above, relative to the LOGAN paper, we introduce the ADIS variant of the detector based attack, which performs much better in practice on genomic data. We also provide theoretical analysis of the detector success in Theorem 1 and in Equation 1, which may provide some insight on why the detector-based attacks perform worse in certain settings. Finally, we evaluate on new types of GAN architectures: BigGAN, Progressive GAN, and ContraGAN, and find differences in attack success across architectures.

Reviewer comment (Novelty):
The proposed one-way distance-based attack is the same as Section 5.2 of GAN-leaks (Chen et al. 2020).

Author response:
The distance-based attacks are similar to the attacks proposed in GAN-leaks, which we state in the contributions section: 
``We compare our Detector-based methods to an array of distance and likelihood-based
attacks proposed in prior work Chen et al. (2020).'' These distance based attacks are implemented to provide comparison to our other black-box attacks. We also note that a small contribution of this work is executing these distance-based attacks with different distance-metrics, like LPIPS (https://arxiv.org/abs/1801.03924). We also provide the most extensive evaluation of these distance-based attacks across tabular and image data.

Reviewer comment (Evaluation):
Many plots in Figure 1 and Figure 2 show that the proposed approaches can perform worse than random guesses in some settings (e.g., Figure 1a, 1b, 1d, 1e, 1f, 1g, 1j, 1k). Do you have explanations for that?

Author response:
On the genomic data, only the one-way distance attack really seems to perform worse than random guessing.

 Recall that the one-way distance attack thresholds based on the distance between $x$ and the generated samples. We have added discussion of these results to Section 5.2: 
"Interestingly, the only method that consistently fails to beat the random baseline (AUC $< .5$) in all settings is the One-way distance-based attack. Paired with the moderate success of two-way distance attacks, this shows that for training points while the \emph{relative difference} between their distance to generated samples and their distance to test samples is larger than the difference in distances for test points, but the actual distance to the generated samples alone is not uniformly larger."

Reviewer comment (Evaluation):
The claim that "... ADIS consistently outperforms the Detector ..." is not supported, as this is not the case in Figure 1j.

Author response:
We have updated the language to reflect that occasionally the detector outperforms ADIS.

Reviewer comment (Experiments):
Because many of the lines in Figure 1 and 2 have significant overlap and the rankings of the methods are not consistent across the entire range of FPR, it would be more convincing to conduct the experiments multiple times and report the average and standard deviation of the scores.

Author response:
All reported results are averaged over 11 training runs, and in Figure $1$ we report the AUC $\pm$ the standard deviation in the legend. We've added this to the caption of the figure to make this more clear.  We also added table $7$ to the Appendix that also contains the average AUCs and standard deviations.

Reviewer comment (Theory):
The attack assumes that the adversary can get fresh samples from the same distribution as the underlying private data. However, it is unclear whether such an assumption is reasonable in practice.

Author response:
We note that although almost all prior work on MIAs assumes that adversary has access to the true distribution P, we agree that in some situations where data access is very limited for privacy reasons, it is reasonable to assume the adversary only has access to a proxy distribution. We’ve added new experiments evaluating how the attack success of our detector attack changes as the proxy distribution that the adversary has access to shifts away from $\mathcal{P}$. We find that under both models of distribution shift that we study, as the distribution shifts way from $\mathcal{P}$, attack success decreases. Surprisingly we find that attack success decreases less than we might have thought, showing that the detector is somewhat robust to distribution shift. We summarize theses results in the distribution shift section of the draft.

Reviewer comment (Experiments):
In real-world settings, oftentimes the adversary may only be able to get data samples from a slightly different distribution (e.g., to attack synthetic data released from one hospital, the adversary can only access data from another hospital). It is unclear how the proposed attack would perform under such distributional shifts. Some empirical results on this would be nice to have.

Author response:
We note that although almost all prior work on MIAs assumes that adversary has access to the true distribution P, we agree that in some situations where data access is very limited for privacy reasons, it is reasonable to assume the adversary only has access to a proxy distribution. We’ve added new experiments evaluating how the attack success of our detector attack changes as the proxy distribution that the adversary has access to shifts away from $\mathcal{P}$. We find that under both models of distribution shift that we study, as the distribution shifts way from $\mathcal{P}$, attack success decreases. Surprisingly we find that attack success decreases less than we might have thought, showing that the detector is somewhat robust to distribution shift. We summarize theses results in the distribution shift section of the draft.

Reviewer comment (Writing):
The paper could benefit from a further round of proofreading to fix typos (e.g., that to random points -> than to, we trained average -> we average). Also, some notations in the Appendix are not consistent with the main paper (e.g., G = βP + (1 − β)T -> G = βT + (1 − β)D).

Author response:
Thank you for the found typos, we will make sure to fix all of the above prior to the camera ready, and we have fixed the typo with the lipschitz constant in the Appendix (it was correct in the main draft) thank you!

Reviewer comment (Theory):
In the proof of Lemma A.1, E_G[f(x)] should be 2TPR_M(f). Also, in the proof the Lipschitz constant is on the denominator which is not consistent with Eq. 1.

Author response:
Thank you for the found typos, we will make sure to fix all of the above prior to the camera ready, and we have fixed the typo with the lipschitz constant in the Appendix (it was correct in the main draft) thank you!

Reviewer comment (Theory):
How does the mode-collapse phenomenon of GAN motivate expressing G as a mixture of T and P?

Author response:
First we remark that Equation 1 still holds for arbitrary $G$, which says we can bound the error of the detector attack by $W^1(G, T)$. Now Theorem $1$ attempts to sharpen this result by showing how even if $G$ is actually quite far from $T$, and $W^1(G, T)$ is quite large,  if $G$ is a convex combination of $\mathcal{P}, T$, the attack is still approximately optimal. We are not arguing that in practice $G$ is actually a convex combination of $\mathcal{P}, T$. We are more arguing that under this simplified representation of the generator, the attack is approximately optimal. While we do not expect that generators trained in practice will be an exact mixture distribution, the convex combination has some merits: (i) it reflects the fact that if properly trained G learns the true distribution $\mathcal{P}$ a $1-\alpha$ fraction of the time. Generative models have been observed to memorize their training data (e.g. https://arxiv.org/abs/2301.13188) and so the $\alpha$ component represents this memorization probability. Without a simplified model of the generator, obtaining theoretical results on attack optimality seems out of reach. We have updated the draft to provide this discussion in Section $4.1$.

Reviewer comment (Experiments):
How would the proposed attack perform if P is shifted from T?

Author response:
We note that although almost all prior work on MIAs assumes that adversary has access to the true distribution P, we agree that in some situations where data access is very limited for privacy reasons, it is reasonable to assume the adversary only has access to a proxy distribution. We’ve added new experiments evaluating how the attack success of our detector attack changes as the proxy distribution that the adversary has access to shifts away from $\mathcal{P}$. We find that under both models of distribution shift that we study, as the distribution shifts way from $\mathcal{P}$, attack success decreases. Surprisingly we find that attack success decreases less than we might have thought, showing that the detector is somewhat robust to distribution shift. We summarize theses results in the distribution shift section of the draft.

Reviewer comment (Experiments):
Overall, while the experiments conducted are very thorough, they should be complemented with at least one differentially-private data synthesis approach to assess in particular if such protection mechanism would impact the success of MIAs in a similar manner.

Author response:
In response to this suggestion, we trained two DP GANs, one as per https://arxiv.org/abs/1802.06739 which uses DP-SGD to privately update the discriminator. We track the privacy loss using the RDP accountant in tensorflow (https://github.com/tensorflow/privacy/blob/979748e09c416ea2d4f85e09b033aa9aa097ead2/tensorflow_privacy/privacy/analysis/compute_dp_sgd_privacy.py). We also trained a DP GAN using PATE-GAN (https://openreview.net/forum?id=S1zk9iRqF7) although this did not work as well as the DP-SGD GAN, since the sample splitting required resulted in either too few samples to train the target GANs, or too much noise when privately aggregating. As a result we report results for the DP-SGD GAN, at epsilon = 1.7 in the new DP GAN training section of the draft. 

On 805 dbGaP the Detector attack achieves AUC of .57 against the non-private WGAN. Against the DP-WGAN trained using DP-SGD, the attack AUC is only .50, which is the same as random guessing. We note that the fact training with DP destroys MIA attack success isn't that surprising, as differential privacy provides theoretical defense guarantees against $\textit{any MIA}.$ See Theorem 3.1 in https://arxiv.org/pdf/2010.12112.pdf which says that if the training algorithm is $(\epsilon, \delta)$-DP, then for any attack the $ \text{TPR} \leq \text{FPR}+ (e^\epsilon - 1 + 2\delta)/(e^\epsilon + 1)$. The difficulty in use DP as a defense is that historically training private classifiers that achieve competitive accuracy has been challenging, let alone DP generative models. While this result is encouraging, we note that it is possible a more complete battery of GAN convergence/utility tests may find that our private GAN is less useful than the non-private GAN even at $805$ SNPs, and we anticipate difficulty scaling these results to higher dimensions. We leave this as an intriguing direction for future work!

Reviewer comment (Writing):
Typo correction: « why detectors can be approximately optimal membership inference attacks » -> « why detectors are approximate optimal membership inference attacks».

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Writing):
Typo correction: The following reference is repeated twice: Boris van Breugel, Hao Sun, Zhaozhi Qian, and Mihaela van der Schaar. Membership inference attacks against synthetic data through overfitting detection, 2023b.

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Writing):
Typo correction: « Absent any additional information » -> « In the absence of any additional information ».

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Writing):
Typo correction: « hamming » -> « Hamming ».

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Writing):
Typo correction: « we only report those results here . » -> « we only report those results here. ».

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Writing):
Typo correction: « features improved attack success » -> « features mproves the attack success ».

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Writing):
The following sentence is not complete: « computation of reconstruction losses ?? ».

Author response:
Thank you very much for your feedback, and the great suggestion about training a private GAN. We are incorporating all of the found typos to improve the readability of the paper, including unifying some notation between the draft and the Appendix, and adding more explanation of the Detector attack from the Appendix to the main draft.

Reviewer comment (Experiments):
The main question is how a protection measure such as differential privacy would impact the conclusions drawn from the experiments, and it would be good if the authors could do additional experiments to evaluate this issue.

Author response:
In response to this suggestion, we trained two DP GANs, one as per https://arxiv.org/abs/1802.06739 which uses DP-SGD to privately update the discriminator. We track the privacy loss using the RDP accountant in tensorflow (https://github.com/tensorflow/privacy/blob/979748e09c416ea2d4f85e09b033aa9aa097ead2/tensorflow_privacy/privacy/analysis/compute_dp_sgd_privacy.py). We also trained a DP GAN using PATE-GAN (https://openreview.net/forum?id=S1zk9iRqF7) although this did not work as well as the DP-SGD GAN, since the sample splitting required resulted in either too few samples to train the target GANs, or too much noise when privately aggregating. As a result we report results for the DP-SGD GAN, at epsilon = 1.7 in the new DP GAN training section of the draft. 

On 805 dbGaP the Detector attack achieves AUC of .57 against the non-private WGAN. Against the DP-WGAN trained using DP-SGD, the attack AUC is only .50, which is the same as random guessing. We note that the fact training with DP destroys MIA attack success isn't that surprising, as differential privacy provides theoretical defense guarantees against $\textit{any MIA}.$ See Theorem 3.1 in https://arxiv.org/pdf/2010.12112.pdf which says that if the training algorithm is $(\epsilon, \delta)$-DP, then for any attack the $ \text{TPR} \leq \text{FPR}+ (e^\epsilon - 1 + 2\delta)/(e^\epsilon + 1)$. The difficulty in use DP as a defense is that historically training private classifiers that achieve competitive accuracy has been challenging, let alone DP generative models. While this result is encouraging, we note that it is possible a more complete battery of GAN convergence/utility tests may find that our private GAN is less useful than the non-private GAN even at $805$ SNPs, and we anticipate difficulty scaling these results to higher dimensions. We leave this as an intriguing direction for future work!

Reviewer comment (Novelty):
The authors do not cite several relevant papers which have looked at black box membership inference attacks e.g. [1,2]

Author response:
Thank you very much for your comments. 

**Minor Comments:** We will add the relevant citations on black-box MIAs, move the description of ADIS up to the main draft, and added a description of how the detector pipeline works, so that Figure 4 is not essential to read the paper, as it is in the Appendix. Train and test sizes for the detector attack were chosen so that there are enough training points to train (i) a target GAN that converges (ii) a Detector that is accurate on a test set and (iii) points left over to accurately evaluate the MIA attack success. Typically we need a few thousand points each to train the generative models and the detector, and then at least $500$ points to evaluate MIA attack success.

Reviewer comment (Reproducibility):
Please include a more detailed explanation of the ADIS attack in the main paper since it is a novel contribution of this paper and has been used in the experiments.

Author response:
Thank you very much for your comments. 

**Minor Comments:** We will add the relevant citations on black-box MIAs, move the description of ADIS up to the main draft, and added a description of how the detector pipeline works, so that Figure 4 is not essential to read the paper, as it is in the Appendix. Train and test sizes for the detector attack were chosen so that there are enough training points to train (i) a target GAN that converges (ii) a Detector that is accurate on a test set and (iii) points left over to accurately evaluate the MIA attack success. Typically we need a few thousand points each to train the generative models and the detector, and then at least $500$ points to evaluate MIA attack success.

Reviewer comment (Presentation):
Figure 3 is cited in the main paper but is not a part of the main paper. Please at mention clearly that the figure is a part of the appendix and try to write the paper such that the appendix is purely optional reading.

Author response:
Thank you very much for your comments. 

**Minor Comments:** We will add the relevant citations on black-box MIAs, move the description of ADIS up to the main draft, and added a description of how the detector pipeline works, so that Figure 4 is not essential to read the paper, as it is in the Appendix. Train and test sizes for the detector attack were chosen so that there are enough training points to train (i) a target GAN that converges (ii) a Detector that is accurate on a test set and (iii) points left over to accurately evaluate the MIA attack success. Typically we need a few thousand points each to train the generative models and the detector, and then at least $500$ points to evaluate MIA attack success.

Reviewer comment (Theory):
In theorem 4.1, why is it reasonable to assume that G is a convex combination of P and T? Please clarify/explain in the paper.

Author response:
**Major Comments**: 

**Q:** Why is it reasonable to assume that $G$ is a convex combination of $P$ and $T$, $G = \alpha P + (1-\alpha)T$. 

**R:** First we remark that Equation 1 still holds for arbitrary $G$, which says we can bound the error of the detector attack by $W^1(G, T)$. Now Theorem $1$ attempts to sharpen this result by showing how even if $G$ is actually quite far from $T$, and $W^1(G, T)$ is quite large,  if $G$ is a convex combination of $P, T$, the attack is still approximately optimal. We are not arguing that in practice $G$ is actually a convex combination of $P$, $T$. We are more arguing that under this simplified representation of the generator, the attack is approximately optimal. While we do not expect that generators trained in practice will be an exact mixture distribution, the convex combination has some merits: (i) it reflects the fact that if properly trained G learns the true distribution P a $1-\alpha$ fraction of the time. Generative models have been observed to memorize their training data (e.g. https://arxiv.org/abs/2301.13188) and so the $\alpha$ component represents this memorization probability. Without a simplified model of the generator, obtaining theoretical results on attack optimality seems out of reach. We have updated the draft to provide this discussion in Section $4.1$.

Reviewer comment (Experiments):
In section 5.1, explain how the different train/test set sizes were chosen. Seems a little arbitrary.

Author response:
Thank you very much for your comments. 

**Minor Comments:** We will add the relevant citations on black-box MIAs, move the description of ADIS up to the main draft, and added a description of how the detector pipeline works, so that Figure 4 is not essential to read the paper, as it is in the Appendix. Train and test sizes for the detector attack were chosen so that there are enough training points to train (i) a target GAN that converges (ii) a Detector that is accurate on a test set and (iii) points left over to accurately evaluate the MIA attack success. Typically we need a few thousand points each to train the generative models and the detector, and then at least $500$ points to evaluate MIA attack success.