Paper: Neural Collapse meets Differential Privacy:  Curious behaviors of NoisySGD with Near-Perfect Representation Learning (ICLR-2024)

Reviewer comment (Evaluation):
The results are not meaningful.

Author response:
Thank you for your feedback. Could the reviewer possibly offer further explanation as to why our theoretical and experimental observations regarding non-robustness may not be meaningful? Additionally, would you be able to suggest any other references that might provide more significant insights? Your opinions will be appreciated.

Reviewer comment (Experiments):
How about $f_W(x)=Wx+b$?

Author response:
In the case of perfect collapse, incorporating an offset term $b$  is unnecessary. For binary classification problem with $\alpha$-class imbalance, straightforward computation reveals that the output of NoisyGD is given by $(n e_1,(1 - 2\alpha)) + \mathcal{N}(0,\sigma^2 I)$ where the $1 - 2\alpha$ component arises from the offset $b$ and the direction associated with $b$ is simply Gaussian noise. This noise could potentially detract from performance.

For a non-perfect collapse scenario, a similar computation indicates that an offset $b$ does not eliminate the perturbation. Consequently, the outcome retains its dimension-dependence.

Reviewer comment (Theory):
All of the proofs only analyze one-step Noisy-GD algorithm under a very strong neural collapse assumption. This setting is too simple and might not be reflecting what is happening in De at al (2022). If perfect neural collapse holds, then there is no need for further training. For example, in theorem 2, you can set the clipping threshold G to be very small (near zero) to get near zero error rate. This suggests that you don't have to train on the private data if the neural collapse is assumed. This bound might not be very useful.

Author response:
We have emphasized our main contributions in the official comments in details and we would like to emphasize them again here shortly.

1. In addition to examining the ideal scenario of perfect neural collapse, our research also explores the concept of approximate collapse, which is more prevalent in practical situations. Our results suggest that the extraordinary dimension-independent properties observed during perfect neural collapse is inherently fragile. This non-robustness is also evident in our empirical studies for both NoisyGD and NoisySGD using real datasets!

2. We wish to clarify for the reviewer that the example where the $G$ is near zero, as mentioned by the reviewer, does not align with our theory. In fact, in the proof, we choose $G$ to be the sensitivity of the gradient, and our theory is valid only when $G\geq 1$ for perfect collapse, or when $G \geq \sqrt{1 + \beta^2 p}$ for imperfect collapse.  It is worth noting that even if we clip the data point with a small $G$, this does not alter our results. In fact, in the case where $K=2$, as detailed in the proof of Theorem 3 (similar results hold for the more general case, Theorem 2, and the imperfect case in Section 3.2 and Section 3.3) in Appendix A.2, $n/2$ is scaled to $Gn/2$ if $G<1$, and a smaller $G$ does not imply a smaller mis-classification error.

3. Existing analysis of DP-GD and DP-SGD focuses on suboptimality in the surrogate losses.  The sample complexity is polynomial  $d/\gamma$  or $d/ \sqrt{\gamma}$ (in the strongly convex case) to achieve $(1-\gamma)$-accuracy.  We actually directly studied the ``0-1" loss!  And we showed that the sample complexity is  $\log(1/\gamma)$. This is an exponential improvement and shows an exponential benefit of strong representation learning.

4. We agree that our analysis is simple, but in light of those points in this response and the official comments above, we believe the simplicity of the analysis is a feature rather than a bug! After all, we believe the reviewer agree that our results are interesting and new.  Isn’t it better to have a simpler proof of a fundamental result than a highly technical (and harder to verify) proof?

Reviewer comment (Novelty):
The proof is simple and the technical contribution is limited.

Author response:
We have emphasized our main contributions in the official comments in details and we would like to emphasize them again here shortly.

1. In addition to examining the ideal scenario of perfect neural collapse, our research also explores the concept of approximate collapse, which is more prevalent in practical situations. Our results suggest that the extraordinary dimension-independent properties observed during perfect neural collapse is inherently fragile. This non-robustness is also evident in our empirical studies for both NoisyGD and NoisySGD using real datasets!

2. We wish to clarify for the reviewer that the example where the $G$ is near zero, as mentioned by the reviewer, does not align with our theory. In fact, in the proof, we choose $G$ to be the sensitivity of the gradient, and our theory is valid only when $G\geq 1$ for perfect collapse, or when $G \geq \sqrt{1 + \beta^2 p}$ for imperfect collapse.  It is worth noting that even if we clip the data point with a small $G$, this does not alter our results. In fact, in the case where $K=2$, as detailed in the proof of Theorem 3 (similar results hold for the more general case, Theorem 2, and the imperfect case in Section 3.2 and Section 3.3) in Appendix A.2, $n/2$ is scaled to $Gn/2$ if $G<1$, and a smaller $G$ does not imply a smaller mis-classification error.

3. Existing analysis of DP-GD and DP-SGD focuses on suboptimality in the surrogate losses.  The sample complexity is polynomial  $d/\gamma$  or $d/ \sqrt{\gamma}$ (in the strongly convex case) to achieve $(1-\gamma)$-accuracy.  We actually directly studied the ``0-1" loss!  And we showed that the sample complexity is  $\log(1/\gamma)$. This is an exponential improvement and shows an exponential benefit of strong representation learning.

4. We agree that our analysis is simple, but in light of those points in this response and the official comments above, we believe the simplicity of the analysis is a feature rather than a bug! After all, we believe the reviewer agree that our results are interesting and new.  Isn’t it better to have a simpler proof of a fundamental result than a highly technical (and harder to verify) proof?

Reviewer comment (Experiments):
Is there any empirical improvement by using the proposed data normalization and PCA tricks? I am curious because DP-PCA would also needs privacy-utility trade-off that needs to be accounted.

Author response:
DP-PCA has been widely adopted in DP-SGD to enhance empirical performance, as evident in ``Deep Learning with Differential Privacy" (https://arxiv.org/abs/1607.00133). We provide a perspective from neural collapse theory to explain the efficacy of PCA.
Due to the time constraints of the rebuttal period, we have not conducted experiments using DP-PCA at this stage.

The reviewer is correct that DP-PCA requires a privacy-utility trade-off. However, the impact of the privacy budget on PCA's accuracy is not the focus of our paper.
It's important to note that the recent theory on DP-PCA, such as the one described in ``https://arxiv.org/abs/2205.13709", cannot be directly applied to our setting since their accuracy is measured by the angle between eigenvectors.
In contrast, our investigation into the 0-1 loss highlights the importance of the infinity norm between eigenvectors. 
Therefore, it would be interesting to establish theoretical guarantees for DP-PCA under the infinity norm.

Reviewer comment (Writing):
**It may be difficult for some readers to understand:** Given the topic, I imagine many readers will be familiar with differential privacy but less familiar with neural collapse. As a result, the third paragraph of the introduction and the corresponding figure 1 will be meaningless to them without more explanation. Some of the introductory material is present in section 2.2, but it is a bit technical and not well-suited to newcomers. I would recommend giving a high level explanation of neural collapse in the introduction to help readers.

Author response:
Thank you for these comments. We have added a high-level explanation of the neural collapse and the $K$-simplex equiangular tight frame (ETF) to the manuscript in the introduction. Neural collapse is the phenomenon where features corresponding to different classes are separable enough. An illustrative example for the case where $K=2$ is as follows: an ETF $M$ can be represented as $M = [-m_1, m_1]$ for some vector $m_1 \in \mathbb{R}^{p}$, where the features corresponding to the label $y=1$ are $m_1$, and the features corresponding to the label $y=-1$ are $-m_1$.

Reviewer comment (Writing):
Introduction, paragraph 1, last line: do you mean "no-privacy utility tradeoff"

Author response:
Thank you for pointing out this typo. We have corrected it.

Reviewer comment (Presentation):
Bottom of page 7, Sigma is missing a backslash.

Author response:
Corrected.

Reviewer comment (Theory):
The dimension independence of Theorem 5 requires $\beta_0$ to scale with $p$, but the non-robustness results in 3.2 and 3.3 would also become dimension independent if $\beta$ chosen to vary with $p$ in a similar way. Because of this, it's not clear what we are gaining from dimension reduction in section 4.2.

Author response:
As detailed in Section 4.3, for PCA we have established that $\beta_0 \leq \frac{1}{\sqrt{m}}$ with high probability  where $m$ represents the sample size used in PCA, effectively scaling $p$ to $p/\sqrt{m}.$
Our theoretical framework, at this stage, does not encompass DP-PCA, which is actually implemented in DP-SGD. Should DP-PCA be taken into account,
$\beta_{0}$ would be influenced by the privacy budget. While our current paper does not concentrate on this aspect, understanding the convergence of the eigenvector in DP-PCA, particularly in terms of the infinity norm, necessitates a separate and thorough analysis.

Reviewer comment (Theory):
It's not clear to me how this analysis of neural collapse applies to full fine tuning. The success of DP full fine tuning is most surprising because of the many total parameters of the networks (not just in the last layer). Neural collapse may explain some of the dynamics of last layer fine-tuning as presented in this paper, but clearly something interesting must be happening at intermediate layers in the full fine-tuning setting. I think it would be very helpful to mention whether there is any way that these results might shed light on the dynamics of intermediate layers.

Author response:
Thank you for drawing attention to this intriguing topic. Our empirical findings suggest that, except in the final layer, neural collapse is typically not observed—even in settings without Differential Privacy. This is also corroborated by another paper "Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training" (https://arxiv.org/pdf/2101.12699.pdf), which explains that, from a mathematical standpoint, an Equiangular Tight Frame (ETF) is the optimal solution for Empirical Risk Minimization (ERM) when considering the features of the last layer.

Recent research, such as "The Tunnel Effect: Building Data Representations in Deep Neural Networks" by Masarczyk et al., presented at NeurIPS 2023, has made some novel observations. These include the discovery that the last few layers of deep neural networks exhibit low-rank structures, and their performance is resemblance to that of linear models. Within this context, neural collapse in the final layer can be seen as a special case of these low-rank structures, which are also present in transfer learning scenarios.

Given these developments, it would be a valuable line of inquiry for our future research to examine whether fine-tuning the last few layers with NoisySGD could reveal low-rank structures, like neural collapse --- or perhaps a partially neural collapse influenced by the noise. Due to time limitations, we will defer this investigation to subsequent studies.