Paper: Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation (ICLR-2024)

Reviewer comment (Theory):
For Theorem 1, one can intuitively understand its meaning, but it is hard to interpret the English sentence into a formal mathematical statement. The proof is also vague, with many descriptions but few math formulas and equations. It is hard to rigorously understand the mathematical meaning of this theorem, and it is also hard to check the correctness of the proof. The authors need to rewrite Theorem 1 (maybe leave an informal description in the main paper and postpone the full theory statement in the appendix) and provide a more rigorous proof.

Author response:
We thank the reviewer for their thoughtful critiques regarding the theory aspect of this paper, as well as recognizing the sound empirical findings. We would like to clarify the questions raised by the reviewer in this rebuttal, and these changes will make their way into the paper in future revisions.

**Theorem 1**

We will reword the theorem as follows:

**Theorem 1**
 If $L_\textrm{reconstruction} \to 0$ (from Eq. 7), as $w\to \infty $, then $X_R \to X_T $ in probability for training data, $X_T$ on the unit hypersphere.

The proof follows closely with the outline provided in the text as is, but to clarify some bits which are possibly unclear, we rewrite the proof steps here:
\begin{align}
   L_{recon} =  &\quad \Big\|\Delta \theta - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j \nabla_{\theta_f} f_{\theta_f} (x_j)\Big\|^2_2 =\Big\|\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i \nabla_{\theta_0}f(x_i)} - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j \nabla_{\theta_f} f_{\theta_f} (x_j)\Big\|^2_2 \\
    &= \Big\|\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i k_{\theta_0}(x_i, \cdot)} - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j k_{\theta_f}(x_j, \cdot)\Big\|^2_2 
&\to \Big\|\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i k_{NTK}(x_i, \cdot)} - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j k_{NTK}(x_j, \cdot)\Big\|^2_2 \mathbf{i.p.}
\end{align}
Define 
\begin{align}
P_T = \sum_{\alpha_i, x_i \in \alpha^T, X_{T}}{\alpha_i\delta(x_i)},
P_R = \sum_{\alpha_j, x_j \in \alpha^R, X_{R}}{\alpha_j\delta(x_j)}
\end{align}

Which are both signed measures over $\Omega = S^{d}$.

Then we can rewrite 
\begin{align}
\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i k_{NTK}(x_i, \cdot)}
\end{align} as 
\begin{align}
\int_{\Omega} k_{NTK}(x, \cdot) dP_T(x)
\end{align}, and likewise for $P_R$. Then, we have

\begin{align}
L_{recon} \to \Big\|\int_{\Omega} k_{NTK}(x, \cdot) dP_T(x) - \int_{\Omega} k_{NTK}(x, \cdot) dP_R(x)\Big\|^2_2 = MMD_{k_{NTK}}(P_R, P_T)
\end{align}

This is the maximum mean discrepancy described in [1]. From [2], we have that $k_{NTK}$ being universal over $S^{d}$ [3] implies that i $MMD_{k_{NTK}}(P_R, P_T) = 0$ implies that $P_R = P_T$, i.e. $X_R = X_T$. $L_{recon} \to 0$, as assumed, so $MMD_{k_{NTK}}(P_R, P_T) \to 0$, in probability. As $k_{NTK}$, and the MMD are continuous in their inputs, this implies that $X_R \to X_T$ in probability as well.

We hope that this clarifies the proof. We acknowledge that the description in the text is rather space constrained and in future revisions we will have the more informal statement in the main text, and defer the proof to the appendix, as suggested by the reviewer.

[1]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Sch ̈olkopf, and Alexander Smola.
A kernel two-sample test.

[2] Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural Tangent Kernel: Convergence and Generalization in Neural Networks.

[3] Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanckriet. Universality, characteristic
kernels and rkhs embedding of measures.

[4] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-
regression.

Reviewer comment (Theory):
For Theorem 2, the derivation in the appendix is readable, but the theorem statement in the main paper needs to be more clear. However, when considering infinite width of the neural network, some derivations are missing: for example, in Page 19, the first "->" needs more details. Although k_{\theta_0} -> k_{NTK}, the error terms are needed in the later derivation to show that a negligible |k_{\theta_0}-k_{NTK}| really leads to a negligible error term in the first "->" in Page 19.

Author response:
Yes, these are results in previous literature. Specifically, $k_{\theta_0}$ converging to $k_{NTK}$ is given by Theorem 1 in [1], and corollary 2.4 in [3]. Likewise $k_{\theta}$ staying constant over the course of training (so that $k_{\theta_f}$ also converges to $k_{NTK}$) is given in Theorem 2 in [1] and theorem 2.1 in [2]. We will cite these theorems more specifically in future revisions.

[1] Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural Tangent Kernel: Convergence and Generalization in Neural Networks.

[2] Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2020). Wide neural networks of any depth evolve as linear models under gradient descent


[3] Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation.

Reviewer comment (Writing):
The authors claim that “outliers” are more vulnerable to reconstruction attacks, but this notion of “outlier” is not well defined up front. I believe that your technical definition for outliers is points that have high $\alpha$ values (i.e. points that are “hard to fit”), but this does not necessarily mean that these are points that are distant e.g. under the Euclidean metric in the input space.

Author response:
**Definition of outliers**
We define an outlier to be a datapoint which is harmful/uninformative to the training procedure, has a strange/unusual visual appearance or is far away from other training data in terms of training dynamics. Moreover, there is no agreed-upon definition in the literature. We show that easily reconstructed images fit these three properties as follows:
1. Harmful to the training procedure/uninformative - see fig. 6, in which we show that removing these data points do not adversely affect training
2. Strange visual appearance - see figures 25-40 in the appendix, where we see that the more easily reconstructed images tend to have more “unusual” features (i.e. solid-colored backgrounds for CIFAR-10, faint lines for MNIST)
3. Difficulty to fit - see the discuss of alpha values
We do not believe that L2 distance in the euclidean space is a good definition for an “outlier” data point when working with neural networks, as it does not take into account data structure, or the training procedure. We believe that the qualities that we looked at in terms of defining an outlier are suitable properties and reflect a more nuanced view which takes into account training dynamics and dataset properties. We are happy to discuss this further.

Reviewer comment (Presentation):
The text on most of the figures is so small it makes them hard to read, even on a screen.

Author response:
**Various presentation points**
We apologize for the poor readability of some of the figures. Indeed we were very constrained in terms of space, and consequently had to reduce figure sizes, as well as move certain results to the appendix, which is why the results on convolutional architectures were moved there. We will address these concerns, along with the other minor concerns pointed out by the reviewer in future revisions.

Reviewer comment (Experiments):
Can you make claims about the effect of self-supervised learning on the effectiveness of your reconstruction attacks? Does your method naturally extend to structured outputs, not just scalar outputs?

Author response:
**Self Supervised Learning and output structure**
We note that our attack works with vector outputs, as seen by the results on multiclass classification. Extended our work to self-supervised learning may be more difficult as self-supervised learning typically occurs outside of the NTK regime, i.e. there is a large amount of representation learning, which is not modeled by the NTK. However, we believe that our work fills a vital first-step in understanding reconstruction attacks in a more simplified regime.

Reviewer comment (Theory):
Do you have any speculation how informative your results will be for networks trained with different losses than MSE? Is this a simplifying assumption to enable your proofs or a factor that could significantly change the structure of the empirical NTKs your method uses?

Author response:
**Non-MSE losses**
Our method should work on non-MSE losses such as cross-entropy. We include a discussion in appendix G.2. The main requirement of our attack to work is that the change in network parameters lies in the span of the datapoint gradients, and that these gradients do not change significantly over the course of training. This is fulfilled by wide neural networks under any loss (see [1], appendix B for more details). We chose to focus on MSE loss in this work as it allows a closed for solution for the $\alpha$ values, simplifying analysis, but the analysis still applies. In the proof for Theorem 1, we do not explicitly require MSE loss, so this proof still holds for non-MSE losses in the kernel regime.

Reviewer comment (Theory):
Is there an underlying assumption in (1) that there is a prior (i.e. weight decay) on the parameters of the network?

Author response:
**Weight Decay**
We do not assume weight decay, but including weight decay would not affect the theoretical results in our paper. Like in the answer to the previous question, weight decay does not affect whether the weight changes remain in the span of the dataset gradients, and thus our analysis still holds. Note that we can have a similar effect to weight decay with early stopping, and we carefully ablate the effects of early stopping in appendix G.1.