Paper: Privacy at Interpolation: Precise Analysis for Random and NTK Features (ICLR-2024)

Reviewer comment (Evaluation):
The paper seems to address an extremely limited scenario: linear model with squared error and invertible kernel. The relevance of this kind of model in the current era is unclear. The attack also appears quite specific; while a narrow example is given (a person in a compromising environment), it is uncertain whether the conclusions of the method apply to more general forms of realistic attacks, and if so, how.

Author response:
The study of generalized linear models in the over-parameterized regime constitutes a well-established line of research in deep learning theory. In particular, the behavior of RF and NTK models in this regime ($N = o(p)$) has been the object of extensive study e.g. in the context of optimization [1, 2, 3], generalization [4, 5] and robustness [6, 7].


While it is true that state-of-the-art deep learning models may be far from the NTK regime, the theoretical analysis of such objects still provides useful insights also in practice [8, 9]. Furthermore, even if our formal attack description requires a clear distinction between useful information $x$ and background noise $y$, our analysis provides a solid theoretical foundation to the family of attacks in which the attacker has partial knowledge about the training samples, and aims to retrieve information about the rest.


This is particularly relevant in the context of hallucinations in language models. As mentioned at the end of Section 3, sensitive information ($x$) about an individual ($y$) could be stored in a textual dataset. The attacker could guess that $y$ was mentioned in the training dataset, and can try to recover $x$ by querying the model with the prompt


“The address of $y$ is…”


These experiments have been considered e.g. for question-answering tasks [10]. Specifically, in [10], the authors mask the tokens containing the information which is relevant to solve the task, and they verify that the trained model can still hallucinate the correct answer on the “masked” training samples (see Table 1 of [10]).


While the precise formulation of more realistic problems might differ, our approach offers a solid ground to mathematically formulate different instances of this general problem in machine learning.

Reviewer comment (Novelty):
Differential privacy has been generally accepted as the gold standard for privacy, with carefully reasoned arguments supporting this choice. The paper uses the term "privacy guarantee". It is unclear why this new notion of privacy should be of interest, and whether the ideas could be more closely related to differential privacy.

Author response:
We agree with the reviewer that our work does not provide any formal privacy guarantee according to the formalism of Differential Privacy (DP). To better represent our contribution, we have modified what before we were addressing as “privacy guarantees” to “resistance to the attack” in the revised version of our work. In this regard, as suggested by Reviewer 4po4, we have also modified the first part of the title, replacing “Privacy at Interpolation” with “Stability and Information Retrieval”, as well as several sentences in the abstract, introduction, conclusions and right after the statements of the main theorems. We believe that these modifications better frame our technical contributions. We consider the analysis of DP training methods using the formalism developed in this work as an exciting avenue for future research.

Let us conclude by addressing directly the question of the reviewer: “Why should we be interested in this new notion of privacy?” The problem of information retrieval in over-parameterized ML models is pressing in practice. Our analysis aims to characterize those situations in which DP training is not deployed, but the application would still benefit from being resistant to black-box information retrieval attacks. This is prominent in the case of LLMs [10, 11] (see also our response to the previous point) and in individual recognition in computer vision (see also Section 3 of the paper).

Reviewer comment (Novelty):
The introduction states "the accuracy of the attack grows with generalization error". If this is the main point of the paper, it appears, while perhaps not trivial, not especially surprising. A model that perfectly fits the data and has high generalization error can memorize arbitrary points easily, implying low privacy and an easy attack.

Author response:
The reviewer is right in pointing out that a model that can perfectly fit the data and has high generalization error could be simply memorizing arbitrary points. This, in fact, also suggests that the attack *should* be easy, and it is in line with the empirical evidence in [10], where overfitting is seen to correlate with the success of the recovery attack. The results of our work are a **formal mathematical description of this phenomenon**, through the analysis of the concentration of the feature alignment to a strictly positive quantity, in the settings of RF and NTK regression.


As described by Theorems 5.4 and 6.3 and shown in Figures 3-5, our analysis also links the power of the attack to the specific choice of the model and its activation function. This connection (exemplified in the second subplot of Figures 3 and 4, as well as in Figure 5 added in the revision) is highly non-intuitive, and it comes purely from our analysis.


To conclude, as mentioned in Section 7, characterizing the concentration of the feature alignment of more complex feature maps would allow to compare different models and establish which of those are the most resistant to recovery attacks, thus allowing a principled design of privacy-preserving ML models.

Reviewer comment (Evaluation):
The paper would probably benefit from having a section that describes and discusses the results of the experiments. Though the results in sections 5 and 6 are pretty interesting, they are also quite similar. Maybe the authors can consider cutting the proof sketch of section 6 and adding some discussion on the results of the experiment.

Author response:
As suggested, we have included the following paragraph at the end of Section 6, providing more intuition both on our experimental and theoretical results:

“As (12) suggests, if the value of $\mathcal F_\varphi(z_1, z_1^m)$ converges to 0, no information can be recovered by the attacker. 
However, our Theorems 5.4 and 6.3 show that, for RF and NTK models respectively, a strictly positive geometric overlap ($\alpha > 0$) guarantees a strictly positive feature alignment. This in turn guarantees the attacker the possibility to recover information about the original data, as long as the generalization error is not 0, as discussed in (10).
Our experiments both on synthetic and real datasets confirm these findings. First, in Figures 3, 4 and 5, we see a decrease in the attack accuracy as generalization improves, as discussed in Section 4. This behaviour is displayed also by neural networks, as shown in Figure 1. Additionally, our experiments confirm the characterizations of $\gamma_{\rm RF}$ ($\gamma_{\rm NTK}$) in terms of $\alpha$ and the Hermite coefficients of the activation (its derivative). The expected dependence is displayed also on real datasets (see Figure 5), where functions with higher order Hermite coefficients lead to models that are more resistant to the attack.”

To make space for it, as suggested by the reviewer, we have moved the sketch of the proof for NTK regression at the beginning of Appendix E.

Reviewer comment (Theory):
I also think some discussion on the results of Theorem 5.4 and Theorem 6.3 are needed. At first glance, the main conclusion that we can get from the results is the feature alignment value would converge to some constant that depends on the norm of the background $d_y$ (which is easily interpretable) and the $l-$th Hermite coefficient of the activation function (which is not that intuitive).

Author response:
As suggested, we have included the following paragraph at the end of Section 6, providing more intuition both on our experimental and theoretical results:

“As (12) suggests, if the value of $\mathcal F_\varphi(z_1, z_1^m)$ converges to 0, no information can be recovered by the attacker. 
However, our Theorems 5.4 and 6.3 show that, for RF and NTK models respectively, a strictly positive geometric overlap ($\alpha > 0$) guarantees a strictly positive feature alignment. This in turn guarantees the attacker the possibility to recover information about the original data, as long as the generalization error is not 0, as discussed in (10).
Our experiments both on synthetic and real datasets confirm these findings. First, in Figures 3, 4 and 5, we see a decrease in the attack accuracy as generalization improves, as discussed in Section 4. This behaviour is displayed also by neural networks, as shown in Figure 1. Additionally, our experiments confirm the characterizations of $\gamma_{\rm RF}$ ($\gamma_{\rm NTK}$) in terms of $\alpha$ and the Hermite coefficients of the activation (its derivative). The expected dependence is displayed also on real datasets (see Figure 5), where functions with higher order Hermite coefficients lead to models that are more resistant to the attack.”

To make space for it, as suggested by the reviewer, we have moved the sketch of the proof for NTK regression at the beginning of Appendix E.

Reviewer comment (Experiments):
I think the experiments right now mainly support that generalization improves privacy but it's quite hard to see where the feature alignments part comes into play here. Maybe the authors can try varying the level of alignment or $\alpha = d_y/d$ to see how it affects the attack's efficiency.

Author response:
We would like to thank the reviewer for the question and the helpful suggestion. First of all, let us point out that the role of $\alpha$ and of the Hermite coefficients of the activation can be appreciated in the second subplot of Figures 3 and 4: $\alpha=1/2$ (red lines) leads to a more successful attack than $\alpha=1/4$ (black lines); an activation function with dominant low-order Hermite coefficients leads to a more successful attack; neither $\alpha$ nor the specific choice of the activation influence much the test accuracy. This experimentally confirms the findings of our main theoretical results.

In Figure 5 of the revision (also discussed above), the effect of the activation on the accuracy of the attack is displayed for MNIST and CIFAR-10. 

Finally, to address the reviewer’s comment, we have performed an additional experiment with synthetic data in which we plot the test and attack accuracies as a function of $\alpha$. The results are presented in Figure 6 contained in Appendix G: the test accuracy does not depend on $\alpha$, while the accuracy of the attack monotonically grows with $\alpha$. This is again in agreement with the results of Theorem 5.4 and 6.3.

Reviewer comment (Theory):
The main weakness of the paper is its attempt to characterize "privacy". This is discussed in the paragraph "Generalization and privacy", and the only argument made there is that the attacker's recovery loss can be bounded below (eq. 12) by a quantity that depends on $\gamma_\phi \sqrt{R_{Z_{-1}}}$ (where $R$ is a generalization term, and $\gamma_\phi$ is the feature alignment term that the paper proceeds to bound). The issue is that this lower bound ignores several terms, some of which are important (in particular the variance of $f(z_1^m, \theta^*)$). The lower bound of eq (12) is inadequate to really characterize the attacker's power.

Author response:
While it is true that the bound in (12) is not necessarily tight (because of the variance of $f(z_1^m, \theta^*)$), it still provides a worst-case estimate of the recovery power of the attacker. Specifically, if the generalization error is small, then provably the loss of the attacker cannot be better than random guessing. 

To address the comment of the reviewer, in the revision we also provide another bound that does not neglect the variance of $f(z_1^m, \theta^*)$. By assuming that both the labels and the predictor are balanced ($\mathbb E_{x_1}[g_1]=0$ and $\mathbb E_x[f(z_1^m, \theta^*_{-1})]=0$), we have 

$$
\mathbb E_x[\mathcal L]\ge \mathbb E_x[\mathcal L_0] +\gamma_\varphi^2\mathcal R_{Z_{-1}}-2\gamma_\varphi\sqrt{\mathcal R_{Z_{-1}}}\sqrt{{\rm Var}(g_1)},
$$

where $\mathcal L_0=\mathbb E_{x_1, y_1}[(f(z_1^m, \theta^*_{-1})-g_1)^2]$ denotes the loss of an attacker that has no information available on the true label $g_1$. We note that, when the generalization error is small, the RHS is close to $\mathbb E_x[\mathcal L_0] -2\gamma_\varphi\sqrt{\mathcal R_{Z_{-1}}}\sqrt{{\rm Var}(g_1)}$, which is similar to the lower bound in eq (12). We have added this new bound at the end of Section 4 and deferred its proof to Appendix F.

Reviewer comment (Theory):
There is not a single formal result on privacy stated in the paper, despite many claims made about privacy guarantees. Quoted examples include: "Our analysis provides the first theoretical privacy guarantees for ERM algorithms interpolating the data.", "The combination of Theorem 5.4 and Lemma 4.1 shows the proportionality between stability and privacy.", and "We provide a quantitative characterization of the accuracy of a family of powerful information recovery attacks". The reviewer disagrees with all of these claims, stating that the paper only gives an informal, qualitative argument about attack success, together with experiments, and provides no formal guarantees about privacy. The reviewer concludes that this is a good paper on stability analysis, but not on privacy.

Author response:
Yes, we agree with the reviewer and have revised accordingly. In particular: 
- We have removed “Privacy” from the title, replacing it with “Stability” and “Information Retrieval”.
- We have removed the sentence “Our analysis provides the first theoretical privacy guarantees for ERM algorithms interpolating the data".
- We have rephrased the sentence “The combination of Theorem 5.4 and Lemma 4.1 shows the proportionality between stability and privacy” as “The combination of Theorem 5.4 and (10) shows that the more the algorithm is stable (and, therefore, capable of generalizing), the smaller is the statistical dependency between the query of the attacker and the true label”.
- We have rephrased the sentence “We provide a quantitative characterization of the accuracy of a family of powerful information recovery attacks” as “We study the accuracy of a family of powerful information recovery attacks for models trained via empirical risk minimization, in the interpolation regime”.
- Several other changes in this direction have been made to the abstract, introduction, discussion after the main theorem statements, and conclusions.


All these changes are in red color to facilitate the reviewing process. 

We would like to highlight that these modifications only concern how to interpret our technical results. The technical content of the paper remains unchanged. The only exception is the additional bound on the loss of the attacker discussed above.

Reviewer comment (Novelty):
The paper should improve its connection to prior work. There is only a brief mention of related work on stability in Section 2, with no further attempt to compare or contrast the results in this paper, in particular Lemma 4.1. Stability in linear regression is a classic topic, with work going back much further, for example [1]. The projector matrix defined in the paper is known as the hat matrix, and its role in stability is well studied. For example, [1] gives a similar expression of stability involving the hat matrix. Although not identical to Lemma 4.1, they are closely related, and it is important to make that connection and discuss differences. The reviewer invites the authors to do a more careful review of the topic.

Author response:
We thank the reviewer for pointing the related work [1], which describes the role of the Hat matrix to estimate the influence of a single data point in linear regression.

We remark that the Hat matrix $H := X (X^\top X)^{-1} X^\top$ can be defined when $X^\top X$ is invertible, which requires (in our notation) $p \leq N$. As we consider the overparameterized regime ($p > N$), this condition cannot be satisfied.

In the underparameterized regime, following 5.5 in [1], we get (using our notation)

$$f(z_1, \theta^*) - f(z_1, \theta^*_{-1}) = h_1 r_1 / (1 - h_1),$$

where $h_1$ is the first element in the diagonal of $H$, and $r_1$ is the residual on the first sample, i.e.,

$$r_1 = g_1 - f(z_1, \theta^*).$$

At interpolation, $r_1 = 0$ and the leave-one-out stability has to be derived alternatively.

We agree with the reviewer that our analysis leverages projectors having a similar form, such as $P_{\Phi} = \Phi^\top (\Phi \Phi^\top)^{-1} \Phi$. However, Lemma 4.1, which uses $P_{\Phi_{-1}}^\perp$ to describe the leave-one-out stability of overparameterized generalized linear models, is – to the best of our knowledge – novel.

In the revision, we mention the related work [1] in Section 2, and then we discuss it more in detail right after the statement of Lemma 4.1.

Reviewer comment (Theory):
Would the authors be willing to recast the paper as a study of stability rather than privacy? The reviewer would be more inclined to accept the paper in that case, particularly if all claims about quantitative or formal privacy guarantees were removed, privacy was removed from the title, etc., while retaining the empirical results.

Author response:
Yes, we agree with the reviewer and have revised accordingly. In particular: 
- We have removed “Privacy” from the title, replacing it with “Stability” and “Information Retrieval”.
- We have removed the sentence “Our analysis provides the first theoretical privacy guarantees for ERM algorithms interpolating the data".
- We have rephrased the sentence “The combination of Theorem 5.4 and Lemma 4.1 shows the proportionality between stability and privacy” as “The combination of Theorem 5.4 and (10) shows that the more the algorithm is stable (and, therefore, capable of generalizing), the smaller is the statistical dependency between the query of the attacker and the true label”.
- We have rephrased the sentence “We provide a quantitative characterization of the accuracy of a family of powerful information recovery attacks” as “We study the accuracy of a family of powerful information recovery attacks for models trained via empirical risk minimization, in the interpolation regime”.
- Several other changes in this direction have been made to the abstract, introduction, discussion after the main theorem statements, and conclusions.


All these changes are in red color to facilitate the reviewing process. 

We would like to highlight that these modifications only concern how to interpret our technical results. The technical content of the paper remains unchanged. The only exception is the additional bound on the loss of the attacker discussed above.

Reviewer comment (Novelty):
Can the authors carefully review work on stability in linear regression, in particular connections to [1] and related work?

Author response:
We thank the reviewer for pointing the related work [1], which describes the role of the Hat matrix to estimate the influence of a single data point in linear regression.

We remark that the Hat matrix $H := X (X^\top X)^{-1} X^\top$ can be defined when $X^\top X$ is invertible, which requires (in our notation) $p \leq N$. As we consider the overparameterized regime ($p > N$), this condition cannot be satisfied.

In the underparameterized regime, following 5.5 in [1], we get (using our notation)

$$f(z_1, \theta^*) - f(z_1, \theta^*_{-1}) = h_1 r_1 / (1 - h_1),$$

where $h_1$ is the first element in the diagonal of $H$, and $r_1$ is the residual on the first sample, i.e.,

$$r_1 = g_1 - f(z_1, \theta^*).$$

At interpolation, $r_1 = 0$ and the leave-one-out stability has to be derived alternatively.

We agree with the reviewer that our analysis leverages projectors having a similar form, such as $P_{\Phi} = \Phi^\top (\Phi \Phi^\top)^{-1} \Phi$. However, Lemma 4.1, which uses $P_{\Phi_{-1}}^\perp$ to describe the leave-one-out stability of overparameterized generalized linear models, is – to the best of our knowledge – novel.

In the revision, we mention the related work [1] in Section 2, and then we discuss it more in detail right after the statement of Lemma 4.1.