Paper: Relating Implicit Bias and Adversarial Attacks through Intrinsic Dimension (ICLR-2024)

Reviewer comment (Experiments):
The contributions are somewhat limited, as the authors do not provide sufficient evidence to support their arguments for the strong correlations between implicit bias and adversarial attacks. From Tab.2, the cosine similarity is not sharp, and many more results (such as the adversarial success rates against non-linear correlations) should be presented to clarify this.

Author response:
**On the cosine similarity:** The fact that the cosine similarity between masks is not sharp is one of the crucial points of our work. We find that the correlation between sets of masks is highly non-linear and thus not detectable by cosine similarity. This is the reason why we designed and applied a novel non-linear correlation method based on intrinsic dimension estimation.

**On additional results:** We thank the Reviewer for suggesting to inspect how correlations are related to adversarial success rates. We ran an experiment using PGD with variable perturbation budgets and found that correlation appears in any case. Results are available in the Appendix of the revised pdf (section A.8).

Reviewer comment (Novelty):
The authors need to give a clear presentation about why it is necessary to relate implicit bias and adversarial attack together.

Author response:
**Why we need to relate the implicit bias and adversarial attack:** The problem of adversarial vulnerability of neural networks remains largely unresolved and, to a certain extent, not fully understood. We believe that our work represents a step towards understanding adversarial attacks and, potentially, defending against them.

Reviewer comment (Reproducibility):
The proposed method described in Sec.3.2 is limited and incomplete, making it challenging to apply to practical applications.

Author response:
**The proposed method is limited and incompleted:** We are aware that there are limitations to our ID-based correlation approach, mainly related to the computational issues of ID estimation in the case of very high-dimensional or high-ID data sets. We clearly state these limitations in the paper (sections 3.2 and 4.6). However, we do not fully understand what the Reviewer refers to with the term ‘incompleted’.

Reviewer comment (Reproducibility):
It is unclear how we know the adversarial class. Since untargeted adversarial attacks are usually adopted, the procedure to compute Eq.(1) for adversarial examples needs clarification.

Author response:
**On the adversarial class:** The Reviewer is right in saying that the attacks we employ are untargeted. We state this point explicitly in the revised version of our pdf file. The adversarial class is simply known by feeding the adversarial input to the classifier.

Reviewer comment (Novelty):
This paper does not propose new methods or strategies on top of their findings, leaving unclear what can be done next after finding the correlations.

Author response:
**New methods or strategies on top of our findings:** We believe that this work can be a starting point towards new strategies for adversarial defence, as indicated in the future directions (section 5). Moreover, we sketch what we believe to be a promising idea in this direction in the last lines of the section that deals with class-specific masks (section 4.7).

Reviewer comment (Evaluation):
The Z-test is not used in practice that much. Moreover, it is not appropriate if the statistics of the population is not known beforehand. I think that it should be replaced with a t-test at the very least. Moreover, the distribution of intrinsic dimension cannot be normal, as it is a positive number, so a Poisson/Gamma distribution is more appropriate. Even though the issue of the distribution does not appear to be detrimental to the results of the paper, the issues of Z-test should be clearly stated in the paper.

Author response:
The Reviewer is right when assessing that the true distribution of the ID cannot be Gaussian, due to the impossibility of obtaining negative values. However, in all our tests, the distribution of the IDs computed after shuffling was nearly Gaussian, somehow justifying the choice of the Z-test. In any case, we coincide with the Reviewer's statement that, although the results of the paper are not affected by this issue, it should be at least commented on, so we added a sentence to the paper on the limitations of the Z-test (section 3.2).

Reviewer comment (Experiments):
The conventional $\epsilon$ for $\ell_\infty$ attacks of CIFAR is $\frac{8}{255} \approx 0.03$, whereas in the experiments it is $0.01$ (refer to https://robustbench.github.io/).

Author response:
As we detail in the paper (section 4.3), we chose $\epsilon=0.01$ for PGD to maintain consistency with the perturbation sizes found by the other attacks. This choice does not harm the efficacy of the attack, which succeeds in almost 99% of the cases (please see Table 4, in the Appendix). However, motivated by this suggestion and by other points raised in the Reviews, we decided to test our method on other perturbation magnitudes, including $\epsilon=0.03$. The results confirm our findings and they are reported in the Appendix (Section A.8) in the revised version of our pdf.

Reviewer comment (Theory):
The proposal is poorly supported and the original concepts that are introduced in the paper does not have any rigorous or concrete definition.

Author response:
The strong connection between correlations among data features and the intrinsic dimension of a dataset is a widely acknowledged concept in the field of unsupervised learning. For instance, in the context of hyper-planar manifolds, these correlations manifest in the PCA spectrum, creating a discernible gap that establishes the correct projection dimension. In the case of non-linear manifolds, where correlations exhibit non-linear characteristics, this connection persists, although its mathematical definition becomes somewhat elusive and hinges on the specific method employed for determining the intrinsic dimension. Since our method is designed to be agnostic to the intrinsic dimension determination technique, we purposefully refrain from providing a detailed mathematical description, opting instead to showcase the intuitive aspects of the method and its application to both toy and realistic models. We hold the firm belief that the tests we present are sufficient to support the conclusions outlined in the paper. This confidence is rooted in our verification that artificial uncorrelated by-design datasets with the same macro-features as those investigated in the paper yield clearly different results than those that we found correlated.

Reviewer comment (Writing):
The literature review does not clear up what exactly "implicit bias" is, or how it is related to Fourier transform. The paper just assumes that the reader has the required background knowledge and only describes these relations as some "deep connection".

Author response:
We could only provide a compact literature review due to space constraints. However, in section 2.1 we provide pointers to a number of papers that have analysed implicit bias in Fourier space (e.g. Rahaman et al., 2019 and Fridovich-Keil et al., 2022). The connection between implicit bias and adversarial attacks has not been as thoroughly investigated and it is, in fact, the focus of our work.

Reviewer comment (Novelty):
Please elaborate on the significance of the proposal from the perspective of robustness in machine learning. Specifically, I don't see the significance of providing "empirical evidence that the network bias in Fourier space and the target frequencies of adversarial attacks are closely tied". The literature on this issue is pretty big as evident from a simple search in Google Scholar.

Author response:
We believe that understanding the phenomenon of adversarial vulnerability is a fundamental milestone towards building more robust and reliable models. As anticipated in the previous point, the literature on the relationship between implicit bias and adversarial attacks is, to the best of our knowledge, quite limited at the moment. We cite one paper by Faghri et al. (2021) as the main inspiration four our work. However, it is a purely theoretical paper and it deals with linear networks. Our work extends their results to more complex and useful models using an empirical approach.

Reviewer comment (Theory):
I don't find the verbal description of the proposed correlation test convincing at all. Even if the concept is sound, no formal statement or expression of the method is present. I cannot see how it is possible to verify the arguments made in the paper in its current form.

Author response:
The strong connection between correlations among data features and the intrinsic dimension of a dataset is a widely acknowledged concept in the field of unsupervised learning. For instance, in the context of hyper-planar manifolds, these correlations manifest in the PCA spectrum, creating a discernible gap that establishes the correct projection dimension. In the case of non-linear manifolds, where correlations exhibit non-linear characteristics, this connection persists, although its mathematical definition becomes somewhat elusive and hinges on the specific method employed for determining the intrinsic dimension. Since our method is designed to be agnostic to the intrinsic dimension determination technique, we purposefully refrain from providing a detailed mathematical description, opting instead to showcase the intuitive aspects of the method and its application to both toy and realistic models. We hold the firm belief that the tests we present are sufficient to support the conclusions outlined in the paper. This confidence is rooted in our verification that artificial uncorrelated by-design datasets with the same macro-features as those investigated in the paper yield clearly different results than those that we found correlated.

Reviewer comment (Novelty):
Among the technical contributions of the paper, the only novel idea is that of testing correlation using intrinsic dimensionality and variable shuffling. In other respects, the authors draw upon the Fourier mask framework of Karantzas et al. for analyzing the robustness of ANNs.

Author response:
The Reviewer correctly references Karantzas et al. (2022) as the main inspiration for our Fourier mask framework. However, we would like to note that, while the concept of essential frequency mask had already been explored in the aforementioned paper, the adversarial frequency masks are a novel contribution of our work.

Reviewer comment (Theory):
The authors have not accounted for the potential for bias in their Z-score test. The variance of the ID (and therefore the significance of the Z-score) itself strongly depends on the dimensionality within which it is assessed. Setting a threshold for hypothesis testing that is uniform across all dimensions may not be appropriate here. Estimation of ID also has its own biases that may confound hypothesis testing of this type. Accounting for (and if necessary, adjusting for) dimensional bias would greatly improve both the importance and novelty of the results.

Author response:
Indeed, it is crucial to consider the variance associated with ID estimation. Employing a maximum-likelihood approach, one can obtain an expression for the variance and incorporate this insight as a correction term in the Z-score test. This would result in a slight modification of the p-value while maintaining unchanged conclusions. Additionally, addressing the bias in ID estimation could likely enhance the results and enable the extension of the test to datasets with a higher intrinsic dimension. However, tackling this remains an open problem in the unsupervised learning community, and its solution falls beyond the scope of this work. It is noteworthy, though, that the method proposed here remains independent of the ID estimator used (that is, if one had a perfect ID estimator, it would equally work on the method).  Moreover, by comparing measures of the ID estimated on the same embedding dimension, one could expect that many of the potential biases vanish – or, at least, are mitigated – by error compensation.