Paper: Improving Robustness in Vision Transformers with Nullspace Noise Augmented Finetuning (ICLR-2024)

Reviewer comment (Theory):
As underlined by the authors, the nullspace is a concept from linear algebra. When moving to the nonlinear world, its counterpart is the zero set. This is a common tool in convex optimization [1] which the authors may want to investigate. It seems however that rather than looking for a set of zeros, the authors are more interested in looking for invariance of the ViT with respect to some additive subspaces (this is equation (5)). It may be interesting to change the narrative to motivate it by invariant properties instead of null space properties.

Author response:
Thank you for the insightful comment. Although the common definition of nullspace doesn’t apply to the nonlinear transformer encoder, we empirically find a set of vectors in the input space of the transformer encoder which preserves one property of nullspace that is important to robustness, i.e., invariance of the mapping’s output when added to any input, with good precision on the data distribution. Additionally, our experiment in Appendix A.5 suggests that the $\epsilon$-approximate version of such a set satisfies the properties of closure under addition and scalar multiplication in a considerable scope, similar to those of a vector space. Thus, we extend the definition of nullspace in linear algebra to “encoder-level nullspace” in Equation (5), to highlight the similarity between the explored set and nullspace in linear algebra. 

We’ve considered some well-established concepts in mathematics, such as “zero set” or “invariant set”, as alternative terminology, but they carry their own meanings and have different implications from our work. In Section 2.3 of our manuscript, we delineated the nuances of our extended definition with a focus on the invariance property. 

To address your concern, we have refined our writing in Section 2.3 and the conclusion to make the distinction more explicit.

Reviewer comment (Theory):
The authors state in the conclusion that "Our findings demonstrate that a non-trivial nullspace indeed exists" but it is unclear if the paper shows that. The authors show that a nullspace exists for the patch embedding layer, but then the full study revolves more around invariance properties of the ViT than its nullspace.

Author response:
Thanks for this comment. The complete sentence in our paper was “Our findings demonstrate that a non-trivial nullspace indeed exists for Vision Transformers, a direct consequence of the patch embedding layer.” In section 2.2, we first showed that a non-trivial nullspace exists within the patch embedding layer. Then, because vectors in this set would not affect the entire model’s output when added to any input, this nullspace is a subset of the nullspace of the transformer model, according to our extended definition in Equation (5).

Our discussion on the nullspace of the patch embedding layer, and on the nullspace of the nonlinear encoder according to our extended definition, are concrete definitions of invariance of the transformer model to input perturbations, in contrast to the broadly used “invariance”, which is often vaguely defined as model’s output does not change with certain (or even any) perturbations of input data.

Reviewer comment (Theory):
The statement "We ascertain that employing noise sampled from this approximate nullspace for data augmentation significantly enhances model robustness..." suggests that the training of the ViT performance can improve when trained on data from its nullspace; however, if the data shown to the network belong to its nullspace, it is unclear how it can influence its output.

Author response:
Thanks for this question. We hope to clarify that our original claim quoted in the review is about “approximate nullspace”, for which our definition is the set of vectors which causes a mean $\ell_2$ distance of less than $\epsilon$ on the output logits when added to the input over the data distribution], as in Equation (7). It can have effects on the model by definition.  The goal is to reduce the impact of the identified approximate noise on the network and by doing this we make the model more robust.

Reviewer comment (Theory):
Loss (6) appears to not be lower bounded because no constraint is added. Thus, the minimum of (6) is reached by any vector of infinite norm. Why not add a constraint on the norm of $\tilde{v}$? This may be why early stopping is needed (page 18).

Author response:
Thank you for the observation. For \lambda < \infty, whether a vector of infinite norm is a minimizer of the entire loss function will also depend on the 1st term. For example, if we consider a simple case where both f in (6) are identity functions, then a vector of infinite norm is instead the maximizer. 

We agree this can potentially help prevent the discussions of extreme cases, however, in practice, we did not find it necessary. We noticed that the learning of $\tilde{\mathbf{v}}$ can always converge with the different values of $\lambda$ in our experiments. 
In addition, we had considered adding a bound constraint, but since we don’t have prior knowledge about how large the noise should at least be, we later decided to use a logarithm term to prevent a trivial solution. 

In our work, early stopping was used as a heuristic to find vectors close to the boundary of the $\epsilon$-approximate nullspace. It is used in combination with Equation (6), which minimizes the influence of the noise with gradient descent until it reaches the $\epsilon$ region. Here, we no longer use the regularization term to prevent trivial solution.

Additionally, the effectiveness of our data augmentation method does not rely on the early stopping heuristic. To validate this, we experimented on removing early stopping, and kept other settings the same as our experiment for row 2 of Table 1. We learn the nullspace noise with a fixed number of iterations T. We set T to be large enough so that the learned noise can enter the $\epsilon$ area every time after the fine-tuning starts. The below table shows the performance on various benchmarks, and compared it with the early stopping setting in row 2 of our Table 1.

|  Early stop | clean | FGSM  | DamageNet |   A   | C (⭣) |  V2   |   R   | Sketch | Stylized |
|:---------:|:-----:|:-----:|:---------:|:-----:|:-----:|:-----:|:-----:|:------:|:--------:|
| Yes | 77.47 | 25.95 |   32.43    | 20.77  | 55.98 | 66.5   | 41.61 | 25.67  |  16.02   |
|  No  | 77.01 | 28.84 |   30.26   | 19.68  | 57.68 | 66.77 | 44.21 | 29.40  |  17.31   |

It can be seen that the performance of these two settings are comparable. We have added both the variants to the results table.

Reviewer comment (Evaluation):
While it is interesting to look for vectors of relatively large norms, showing that very different inputs can provide the same output, the resulting vector $u+\tilde{v}$, where $u$ is a natural image, is not a natural image. What are the potential applications in this context? Is it better for a model to be robust to nullspace variations than to common adversarial attacks?

Author response:
Thank you for this inspiring question. When we find vectors of relatively large norms, the resulting vector $\mathbf{u} + \tilde{\mathbf{v}}$ can easily go beyond the pixel range of [0, 255]. However, there are still interesting applications. For example, we can consider these vectors a characteristic of the model being trained, and use it to identify the models, which is related to the application of model patenting. Another interesting aspect that can arise is that of gaining customer trust with regards to privacy. If a server-side ML model can support “unnatural” inputs, a user can effectively hide(transform) their data while preserving the intended inference output. Making it challenging to recover original “natural” images if/when leaked or stolen.

Reviewer comment (Presentation):
Section 3 is clearly unfinished, with unrelated sentences and equations. The narrative from (7) to (10) is unclear.

Author response:
We did complete the writing of Section 3, and we are sorry for any confusion brought to readers. We have added a few transition sentences and explanations in our paper, to improve the coherence of discussion. 
In addition to the text edited in the main paper, here are some additional explanations of Section 3. 

Equation (7) defines the space of additive perturbations that have minute influence to the model’s output. We use $\epsilon$ as the upper bound of the influence measured by MSE of the output logits, so that the space is specified with a given $\epsilon$ and a given model;

Equation (8) defines the problem of finding the largest vector in this space as a reflection of its size;

Equation (9) specifies the loss function used to learn the nullspace noise, which is used in combination with the early stopping method to heuristically solve the problem in Equation (8);

Equation (10) specifies the loss function used in our data augmentation method, in which we used the noise learned by Equation (9) and early stopping to conduct data augmentation.

Reviewer comment (Theory):
In section 3, the authors state that "we demonstrated that there may exist a non-isotropic space". It is unclear if it was really shown that it is non-isotropic.

Author response:
Yes, in Section 2.3, we measured the influence of the learned vectors in terms of MSE probability and percentage of matching predictions. We also experimented with resetting their directions by randomly shuffling their elements. The real and dashed lines in Fig. 2 shows that while the learned vectors are benign to the model, their shuffled version significantly degrades the model’s predictions at different values of $\lambda$. This shows that the noise vectors produce far less influence on the model’s output than they would if they were in a random direction with the same norm, which indicates that the space is non-isotropic.

Reviewer comment (Evaluation):
Section 4.4 is very difficult to follow. The aim of the experiment presented there is unclear. For instance, the authors claim "the noise was always able to enter the $\epsilon$ region". How can this be confirmed? Does any metric check that? This is not what Fig. 3 is showing, or there may be a misunderstanding.

Author response:
The aim of this experiment is to use a data augmentation approach to enlarge the $\epsilon$-approximate nullspace of the model and see its effect on the model’s robustness. This is to explore the relationship between the nullspace and robustness.

The claim "the noise was always able to enter the region" was indeed not reflected in Fig. 3, but we carefully verified from the output of our experiment scripts that early stopping was triggered, which means the noise entered the $\epsilon$-approximate region. We have added the noise influence evaluation in Appendix A.6.

The early stopping for learning noise our method (Section 3) is based on the condition that the noise has an influence less than $\epsilon$ in terms of MSE probability. We carefully verified from our experiment logs that early stopping was always triggered after the fine-tuning started, which means the noise entered the $\epsilon$-approximate region. 

We hope to clarify the misunderstanding. Fig. 3 did not show this piece of information, but only showed the change trend of the noise norm and several performance metrics with the fine-tuning steps. We have updated the draft in Section 4.4 to improve clarity.

Reviewer comment (Writing):
The writing style is often inappropriate and the authors should adopt a more neutral tone. Examples include:  
- "Our work marks an important stride in the understanding of [something]..." — sounds presumptuous, readers will judge if it is truly "an important stride". A better phrasing could be "Our work provides an explanation of [something] by...". Similarly, "we offer a fresh narrative" sounds odd.  
- "This observation leads to a provocative question: Can the inherent properties of ViTs be harnessed to bolster their robustness?" — the question is not provocative; it is a typical research question for this conference, and could be reformulated in a less theatrical way, such as "This observation raises the following question: Can we leverage the inherent properties of ViTs to enhance their robustness?"

Author response:
Thank you for the suggestions. We have examined the writing style of our whole paper, and modified multiple places for a more neutral and humble tone, including the examples you provided.

Reviewer comment (Writing):
The paper's discussion on societal impact and practical applications could benefit from further elaboration and real-world examples to strengthen the argument.

Author response:
Below we provide some examples about the applications of nullspace in ViTs in various societal and technological contexts.

**Enhancing ViT's Resilience to Minor Noises:** This has practical implications in real-world scenarios where visual data might be subject to various types of noise, such as in low-light conditions or with low-quality sensors. Understanding and leveraging nullspace properties can enhance the robustness of ViTs in these noisy environments.

**Model Patenting:** The unique nullspace property of a ViT, determined by its specific set of weights, can be used for patenting purposes. It allows the identification of a specific ViT model based on its nullspace characteristics, which can be particularly useful in legal scenarios involving intellectual property rights and model authentication.

**Image Watermarking:** Nullspace noise can be used for watermarking images in a way that does not alter the output or operability of ViTs. By superimposing marks in the form of nullspace noise onto input images, users can protect their data from misuse or unauthorized redistribution. This application is particularly relevant in fields like digital media, where copyright protection is critical, and in sensitive applications like personal data security.

Reviewer comment (Theory):
While the nullspace concept is intriguing, the limitations of its applicability due to non-linearity in ViTs should be addressed more explicitly.

Author response:
There is inherent non-linearity in ViTs due to elements like activation functions and self-attention mechanisms, posing challenges to the exact analysis on the encoder-level nullspace.  In our work, we address this through approximate nullspace computation, employing deep-learning based optimization to identify minimally impactful vectors on the model’s output. This approach offers a meaningful approximation within the ViT framework, providing an understanding of the robustness of transformer models. Further research could delve into a more detailed analysis of ViTs, to study the nullspace within each layer and their cascading effect, or employ advanced optimization techniques to refine this approximation in non-linear systems.

Reviewer comment (Theory):
The paper mentions the non-linearity in ViTs as a limitation to the exact calculation of nullspace. Could the authors discuss potential methods or approaches to address this limitation or provide more insights into the nature of this non-linearity?

Author response:
There is inherent non-linearity in ViTs due to elements like activation functions and self-attention mechanisms, posing challenges to the exact analysis on the encoder-level nullspace.  In our work, we address this through approximate nullspace computation, employing deep-learning based optimization to identify minimally impactful vectors on the model’s output. This approach offers a meaningful approximation within the ViT framework, providing an understanding of the robustness of transformer models. Further research could delve into a more detailed analysis of ViTs, to study the nullspace within each layer and their cascading effect, or employ advanced optimization techniques to refine this approximation in non-linear systems.

Reviewer comment (Experiments):
For adversarial attacks, the authors only adopt FGSM and DamageNet. However, FGSM is a light-weight adversarial attack. Stronger attacks like PGD should also be considered. The performance of the model under stronger adversarial attack is a better measurement of the robustness of the model.

Author response:
Thank you for the suggestion. Acknowledging the feedback, we have now incorporated CW, MIM and PatchFool. The results are shared at the end.

Reviewer comment (Experiments):
The baseline methods listed in the paper are insufficient. The author should also incoporate some other important baselines[1-5].

[1]Li, Yanxi, and Chang Xu. "Trade-Off Between Robustness and Accuracy of Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2]Zhou, Daquan, et al. "Understanding the robustness in vision transformers." International Conference on Machine Learning. PMLR, 2022.

[3]Paul, Sayak, and Pin-Yu Chen. "Vision transformers are robust learners." Proceedings of the AAAI conference on Artificial Intelligence. Vol. 36. No. 2. 2022.

[4]Wu, Boxi, et al. "Towards efficient adversarial training on vision transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[5]Chefer, Hila, Idan Schwartz, and Lior Wolf. "Optimizing relevance maps of vision transformers improves robustness." Advances in Neural Information Processing Systems 35 (2022): 33618-33632.

Author response:
We appreciate the recommendations of the reviewer. We have added these baseline results into our Table 1.
We appreciate your other comments about supplementary experiments. However, it would take us some time to address them. We have started doing these experiments, and will let you know as soon as we get some results.

Reviewer comment (Experiments):
The proposed method seems independent on the model architecture. Would CNNs also experience improved robustness for the proposed method?

Author response:
Thank you for your insightful question regarding the applicability of our nullspace data augmentation method to CNN architectures. We appreciate the opportunity to clarify this aspect.

Our initial experiments with CNN models, specifically ResNet-18 and ResNet-50, have yielded mixed results. We observed a decrease in performance for ResNet-18 and a marginal improvement for ResNet-50 when applying our nullspace augmentation method. This variance in effectiveness is likely due to the architectural differences between CNNs and ViTs.

|  Model   | nullspace | clean | FGSM  | DamageNet |   A   | C (↓) |  V2   |   R   | Sketch | Stylized |  Avg  |
|:--------:|:---------:|:-----:|:-----:|:---------:|:-----:|:-----:|:-----:|:-----:|:------:|:--------:|:-----:|
| ResNet18 |    No     | 71.55 | 6.37  |   6.40    | 2.45  | 77.42 | 59.45 | 34.64 | 23.65  |   7.72   | 25.75 |
| ResNet18 |    Yes    | 67.38 | 3.06  |   6.70    | 1.69  | 78.03 | 54.15 | 31.20 | 16.49  |   8.48   | 23.46 |     
| ResNet50 |    No     | 80.14 | 23.92 |   17.49   | 11.92 | 63.56 | 68.66 | 39.71 | 28.77  |   8.23   | 34.82 |
| ResNet50 |    Yes    | 78.16 | 28.00 |   19.61   | 8.88  | 62.82 | 66.35 | 40.33 | 30.37  |  10.07   | 35.44 |

Our method, particularly the learning of nullspace vectors, is tailored for the input space of the non-linear encoder of transformers. In contrast, CNN models lack distinct counterparts to the patch embedding and encoder components found in ViTs. In our experiment with CNNs, we adapted our method to learn nullspace vectors at the input space of the CNN. While our method has shown effectiveness for transformer models, its direct application to CNNs is not as straightforward and requires further investigation and adaptation to account for the architectural differences.

Reviewer comment (Theory):
Why using nullspace as augmentation method could boost adversarial robustnes is unclear. As paper said, "We hypothesize that the model’s tolerance to approximate nullspace noise is indicative of its robustness under a variety of distribution shifts." It's quite confusing, as the method is trying to find vectors that does not change model output, then regularize model to show no output change when adding such vector to input. Meanwhile, adversarial training finds perturbs that changes ouput most significantly and optmize the model to tolerate such perturb. The idea is somhow opppsite. Please justify why your method could work.

Author response:
In our approach, we identify approximate nullspace vectors, which does not alter model outputs in theory but still introduces a small error, and then we train our models to make the output the same. This is based on the assumption that a robust model should be insensitive to perturbations that do not modify the semantic content of an image. On the other hand, adversarial training identifies the perturbation that changes the output the most under a norm constraint, and then trains the model to have the same output. This is also based on the assumption that a robust model should be insensitive to small perturbations. Both methods follow the logic of making the model tolerant to additive perturbations that should not alter its predictions, so our method shows the effect of improving the adversarial robustness similar to adversarial training.

Reviewer comment (Experiments):
The experiments are weak. For white-box robustness, please use Auto-attack, CW attack, etc to evaluate. For black-box attack, please evaluate transferability from other models, using more well-known approaches such as MIM series attacks.

Author response:
Thank you for the suggestions. We share below the results on CW, PatchFool[1] and MIM (using deit-tiny) for the ViT-Base. We have also added these results and experimental details to the adversarial robustness experiments of our paper which we will be uploading in a shortwhile.

|               |  CW  |  MIM(fooling rate, lower better)  | PatchFool[1] |
|---------------|:----:|:-----:|:---------:|
| ViT-Base      | 0.56 | 81.72 |   15.92   |
| ViT-Base + NS | 2.36 | 74.72 |   23.52   |
| DAT           | 0.76 | 70.28 |   22.64   |
| DAT + NS      | 0.88 | 71.18 |   24.14   |

**References:**

[1] Fu, Y., Zhang, S., Wu, S., Wan, C., & Lin, Y. (2021, October). Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?. In International Conference on Learning Representations.