Paper: Setting the Record Straight on Transformer Oversmoothing (ICLR-2024)

Reviewer comment (Experiments):
Oversmoothing is primarily a problem that prohibits/hampers training at initialization (see e.g. Noci et al.). The authors make the same observation, while also noting that trained models exhibit different behavior. In that sense, it would make more sense to show if/how their new parametrization enables training in severe cases of oversmoothing, as e.g. when the networks become much deeper. Although I do find the applications of low-data and corruption interesting, their motivation is less clear. It could be interesting to analyse what is happening during the early phase of training.

Author response:
Thank you for raising this point, we agree. To answer this we have added an experiment where we double the depth of the ViT-Ti models on CIFAR100. We have also added a parameterization variant where the first $L/2$ layers sharpen (as in ViT-Ti$^-$) and the remaining $L/2$ layers are left normal (as in ViT-Ti), which we call ViT-Ti*. Finally we have added a comparison FeatScale by Wang et al., 2022. All results are averaged over 3 trials with standard deviations (we bold the best result in each column and any method whose mean plus standard deviation is greater than or equal to mean of the best result):

[new] Increased Depth (Table 3)

| Depth       	| 12       	| 24       	|
|----------------|--------------|--------------|
| ViT-Ti     	| 66.78 ± 0.1  | 67.22 ± 0.1  |
| ViT-Ti + FeatScale | 66.57 ± 0.5  | 67.71 ± 0.5  |
| ViT-Ti$^+$ 	| 64.34 ± 0.3  | 65.65 ± 0.3 |
| ViT-Ti$^-$ 	| 66.62 ± 0.3  | 66.76 ± 0.3  |
| ViT-Ti$^*$ 	| **67.84 ± 0.6**  | **69.05 ± 0.1**  |

Here ViT-Ti$^+$ and ViT-Ti$^-$ both underperform ViT-Ti, and we suspect this is because ViT-T$^-$ sharpens too much, amplifying noise in the data (whereas ViT-Ti$^+$ oversmooths the data). On the other hand ViT-Ti* significantly outperforms all other models not just for larger depths in Table 3, but also in corruption robustness (Table 4), and data efficiency (Table 2). This suggests that both oversmoothing and oversharpening are detrimental to performance, and that a careful balance is key to an accurate model. We have updated the paper to describe these results and look forward to investigating this further in future work.

Reviewer comment (Novelty):
Oversmoothing is less of a problem in ViT for image classification, compared to other scenarios. In image classification, labels are sparse -- 1 per sequence of tokens -- and some oversmoothing in deeper layers is expected -- in fact modern ViT just use the mean activations of the last layer to make predictions [1]. In that sense, a task in vision or language that requires a higher rank output, requiring a different prediction per token, could be more desirable.

Author response:
This is a very good point. We are interested to try out the approach in a more label-dense vision task such as segmentation, where oversmoothing poses much larger problems than image classification as you suggest.

Reviewer comment (Theory):
The authors analyse a simplified Transformer architecture, making some non-trivial assumptions along the way. It is not clear how these findings generalize to a more general scenario. In more detail:  
- They analyze a 1-head attention layer.  
- They assume the same attention and weights are repeated across layers.  
- They ignore the existence of LayerNorm.  
- Attention weights change depending on the data.

Author response:
We were also interested in understanding how our theoretical results generalize to a more general scenario with changing parameters and Pre-LN architectures. To test this, in our experiments we allow both attention matrices $\mathbf{A}$ and weights $\mathbf{H}$ to change every layer, and we use the same architecture as ViT-Ti (a Pre-LN architecture). We were excited to see that even though these models don’t follow the assumptions of our theoretical results, we can still use them to control the smoothing/sharpening behavior of these models. Thank you for brining this up. We have clarified these details in the paper.

Reviewer comment (Experiments):
Most notably, Pre-LN architectures [2] have been shown to effectively counteract some of the oversmoothing in Transformers. In the experiment sections, ViTs used by the authors seem to include LN as far as I can tell. The authors should make this clear.

Author response:
We were also interested in understanding how our theoretical results generalize to a more general scenario with changing parameters and Pre-LN architectures. To test this, in our experiments we allow both attention matrices $\mathbf{A}$ and weights $\mathbf{H}$ to change every layer, and we use the same architecture as ViT-Ti (a Pre-LN architecture). We were excited to see that even though these models don’t follow the assumptions of our theoretical results, we can still use them to control the smoothing/sharpening behavior of these models. Thank you for brining this up. We have clarified these details in the paper.

Reviewer comment (Experiments):
Can you comment on how your findings will change in the presence of different weights per layer and in the presence of pre-ln layers?

Author response:
We were also interested in understanding how our theoretical results generalize to a more general scenario with changing parameters and Pre-LN architectures. To test this, in our experiments we allow both attention matrices $\mathbf{A}$ and weights $\mathbf{H}$ to change every layer, and we use the same architecture as ViT-Ti (a Pre-LN architecture). We were excited to see that even though these models don’t follow the assumptions of our theoretical results, we can still use them to control the smoothing/sharpening behavior of these models. Thank you for brining this up. We have clarified these details in the paper.

Reviewer comment (Experiments):
If oversmoothing is the problem, what about other ways to mitigate it? I am talking about scaling the residuals (Noci et al, [1, 2]) or initializing the attention layers differently, e.g. [3] or different initializations per layer, e.g. [4]. There is a long list of proposed techniques in the literature from the signal propagation perspective. Since you are proposing a new parametrization, it makes sense to compare what you are achieving compares to what they are trying to achieve. Ensuring that the Transformer is not a low-pass filter, does not necessarily mean that any of the meaningful signal is preserved or that feature learning can take place.

Author response:
Thanks for bringing this up. One of the strengths of our approach is that it is complementary to these approaches (scaling residuals [1 ,2], different initializations [3, 4]) and so we can apply them at the same time as our approach. Because of this we opted to compare against an approach that isn’t complementary, specifically FeatScale by Wang et al., 2022. Alongside the above increased depth experiment we also compare its data efficiency and corruption robustness below:

Data Efficiency (Table 2)

| Data Kept          	| 100%     	| 50%      	| 10%      	|
|------------------|--------------|--------------|--------------|
| ViT-Ti       	| 66.78 ± 0.1  | 56.38 ± 0.5  | 32.47 ± 0.9  |
| ViT-Ti + FeatScale | 66.57 ± 0.5  | 57.08 ± 0.3  | 32.50 ± 0.4  |
| ViT-Ti$^+$   	| 64.34 ± 0.3  | 55.21 ± 0.5  | 28.67 ± 0.5  |
| ViT-Ti$^-$   	| 66.62 ± 0.3  | 56.58 ± 0.3  | **33.99 ± 0.4**  |
| ViT-Ti$^*$   	| **67.84 ± 0.6**  | **57.78 ± 0.4**  | **33.64 ± 0.5**  |


Corruption Robustness (Table 4)

| Corruption Intensity | 0          	| 1          	| 2          	| 3          	| 4          	| 5          	|
|----------------------|----------------|----------------|----------------|----------------|----------------|----------------|
| ViT-Ti           	| 66.78 ± 0.1	| 61.59 ± 0.2	| 57.75 ± 0.2	| 53.12 ± 0.1	| 49.32 ± 0.1	| 39.17 ± 0.2	|
| ViT-Ti + Featscale   | 66.57 ± 0.5	| 61.25 ± 0.5	| 57.36 ± 0.6	| 52.64 ± 0.6	| 48.87 ± 0.5	| 38.77 ± 0.4	|
| ViT-Ti$^+$       	| 64.34 ± 0.3	| 58.36 ± 0.4	| 53.83 ± 0.5	| 49.03 ± 0.6	| 45.15 ± 0.6	| 35.86 ± 0.6	|
| ViT-Ti$^-$       	| 66.62 ± 0.3	| 61.62 ± 0.2	| 58.00 ± 0.3	| 53.56 ± 0.2	| 49.70 ± 0.3	| 40.07 ± 0.2	|
| ViT-Ti$^*$       	| **67.84 ± 0.6**	| **62.49 ± 0.5**	| **58.82 ± 0.7**	| **54.32 ± 0.7**	| **50.40 ± 0.6**	| **40.38 ± 0.7**	|

Reviewer comment (Theory):
Before section 5, should the superscript $^+$ model be initialized as $\text{diag}(\Lambda_H) = + (\psi^2)$?

Author response:
This is correct! Thank you for catching this typo!

Reviewer comment (Evaluation):
Can you comment on the stability of your new parametrization? Especially what (if any) are the differences in the early stage of training.

Author response:
This is also interesting! To better understand what is happening early in training we have added a plot of HFC/LFC for the final layer against training iterations in Figure 4 in Section 5. Interestingly, we see that the last layers of both ViT-Ti$^*$ and ViT-Ti have the same filtering properties from the beginning to the end of training, despite the fact that the first $L/2$ layers of ViT-Ti$^*$ are sharpening. Further at the beginning of training, the last layer of ViT-Ti$^-$ becomes less sharp, but by the end of training it becomes sharper than at initialization. Thank you for asking about this and for these citations, we will add them to the paper.

Reviewer comment (Novelty):
While the theory extends previous work in some aspects, it is not clear if this difference is significant. Other preventive measures such as AttnScale/FeatScale from Wang et al. need comparison and more discussion. Several details are also missing from the paper.

Author response:
Thanks for this! We agree that it would be enlightening to compare with Wang et al., 2022 and so we have implemented their FeatScale method. Here are the new accuracy results. We have added another a new experiment where we double the model depth (Table 3) and a parameterization variant where the first $L/2$ layers sharpen (as in ViT-Ti$^-$) and the remaining $L/2$ layers are left normal (as in ViT-Ti), which we call ViT-Ti*. All results are averaged over 3 trials with standard deviations (we bold the best result in each column and any method whose mean plus standard deviation is greater than or equal to mean of the best result):

Data Efficiency (Table 2)

| Data Kept          	| 100%     	| 50%      	| 10%      	|
|------------------|--------------|--------------|--------------|
| ViT-Ti       	| 66.78 ± 0.1  | 56.38 ± 0.5  | 32.47 ± 0.9  |
| ViT-Ti + FeatScale | 66.57 ± 0.5  | 57.08 ± 0.3  | 32.50 ± 0.4  |
| ViT-Ti$^+$   	| 64.34 ± 0.3  | 55.21 ± 0.5  | 28.67 ± 0.5  |
| ViT-Ti$^-$   	| 66.62 ± 0.3  | 56.58 ± 0.3  | **33.99 ± 0.4**  |
| ViT-Ti$^*$   	| **67.84 ± 0.6**  | **57.78 ± 0.4**  | **33.64 ± 0.5**  |


[new] Increased Depth (Table 3)

| Depth       	| 12       	| 24       	|
|----------------|--------------|--------------|
| ViT-Ti     	| 66.78 ± 0.1  | 67.22 ± 0.1  |
| ViT-Ti + FeatScale | 66.57 ± 0.5  | 67.71 ± 0.5  |
| ViT-Ti$^+$ 	| 64.34 ± 0.3  | 65.65 ± 0.3 |
| ViT-Ti$^-$ 	| 66.62 ± 0.3  | 66.76 ± 0.3  |
| ViT-Ti$^*$ 	| **67.84 ± 0.6**  | **69.05 ± 0.1**  |


Corruption Robustness (Table 4)

| Corruption Intensity | 0          	| 1          	| 2          	| 3          	| 4          	| 5          	|
|----------------------|----------------|----------------|----------------|----------------|----------------|----------------|
| ViT-Ti           	| 66.78 ± 0.1	| 61.59 ± 0.2	| 57.75 ± 0.2	| 53.12 ± 0.1	| 49.32 ± 0.1	| 39.17 ± 0.2	|
| ViT-Ti + Featscale   | 66.57 ± 0.5	| 61.25 ± 0.5	| 57.36 ± 0.6	| 52.64 ± 0.6	| 48.87 ± 0.5	| 38.77 ± 0.4	|
| ViT-Ti$^+$       	| 64.34 ± 0.3	| 58.36 ± 0.4	| 53.83 ± 0.5	| 49.03 ± 0.6	| 45.15 ± 0.6	| 35.86 ± 0.6	|
| ViT-Ti$^-$       	| 66.62 ± 0.3	| 61.62 ± 0.2	| 58.00 ± 0.3	| 53.56 ± 0.2	| 49.70 ± 0.3	| 40.07 ± 0.2	|
| ViT-Ti$^*$       	| **67.84 ± 0.6**	| **62.49 ± 0.5**	| **58.82 ± 0.7**	| **54.32 ± 0.7**	| **50.40 ± 0.6**	| **40.38 ± 0.7**	|

We have added these results to the paper. We suspect the reason that FeatScale underperforms is because it is still motivated assuming high frequencies are always suppressed, specifically they (Wang et al., 2022) say “According to our analysis in Section 2.2, MSA module will indiscriminately suppress high-frequency signals, which leads to severe information loss. Even though residual connection can retrieve lost information through the skip path, the high-frequency portion will be inevitably diluted (Theorem1). To this end, we propose another scaling technique that operates on feature maps, named Feature Scaling (FeatScale).”
We agree with you that their analysis suggests that residual connections might prevent high frequencies from collapsing to zero, but Wang et al., 2022 disagree: “Although multi-head, FFN, and skip connection all help preserve the high-frequency signals, none would change the fact that MSA block as a whole only possesses the representational power of low-pass filters”. 
We believe the reason they argue this is because when they analyze the update equations that include residual connections (Proposition 5) they do not account for the eigenspectrum of those equations. Our goal with this work is to set the record straight with a new analysis that uncovers the relationship between the eigenspectrum of the weights and the suppression (or not) of high frequencies in the update equations.

Reviewer comment (Evaluation):
While Figure-2 shows that asymmetry degrades with more epochs, it is unclear if 0.7 is small enough to suggest symmetry, given that it starts at 0.95. Questions also remain about outliers in the off-diagonal entries and how this could be connected to the condition number.

Author response:
Thank you for bringing this up. It’s true that Figure 2 does not suggest perfect symmetry. Our original motivation for using a symmetric parameterization was inspired by [Hu et al., 2019. Exploring weight symmetry in deep neural networks.] who show that weight symmetry in NNs does not affect their universal approximation capabilities. However, we agree that it would be interesting to test a non-symmetric parameterization. Based on your suggestion we have implemented a new parameterization for $\mathbf{H}$ as $\mathbf{H} = \mathbf{V}_H \Lambda \mathbf{V}_H^{-1}$. Specifically, we take gradients to $\mathbf{V}_H$ in the backwards pass, and compute $\mathbf{H}$ using the above equation in the forwards pass. We use the suffix “-NS” for the non-symmetric parameterizations. Here are the new accuracy results: 

We have added these results to the paper. The non-symmetric parameterization does improve upon the symmetric parameterization in some cases (particularly for increased depth and corrupted data). We appreciate your suggestion to investigate this.

> [..Additionally, can you connect it to the condition number?...How should I interpret the asymmetry metric? How does it relate to condition number?..]

This is an interesting question. To investigate this we computed the average condition number for all H matrices used in ViT-Ti for every epoch in training on CIFAR100. We have added the results to Appendix C. The condition number fluctuates during training but gradually increases throughout training. We do not know of a connection between the condition number and the asymmetry of H, but agree it would be an interesting direction for future work.

Reviewer comment (Reproducibility):
Clarification is needed on how $\Theta$ is updated in $QR(\Theta)=[V_H, R]$. Is QR decomposition performed after every gradient step, or is $\Theta R^{-1}=V_H$ used with gradients backpropagated directly to $\Theta$?

Author response:
We backpropagate through the QR decomposition, which requires a QR decomposition for each gradient step. This is because we require $V_H$ to be orthogonal. If we instead parameterized $V_H = \Theta R^{-1}$ and used gradient descent to update $\Theta$ then this could break the orthogonality of $V_H$. Thank you for this, we will add this detail to make this clearer in the paper.

Reviewer comment (Evaluation):
Clarification is needed on how HFC/LFC evolves in training steps.

Author response:
Good question. To answer this we plot the HFC/LFC of the final layer for every epoch during training for the best performing symmetric models (ViT-Ti$^-$ and ViT-Ti$^*$), as well as ViT-Ti, and present the results in Figure 4 in Section 5. Interestingly, we see that the last layers of both ViT-Ti$^*$ and ViT-Ti have the same filtering properties throughout training, despite the fact that the first $L/2$ layers of ViT-Ti$^*$ are sharpening. Further at the beginning of training, the last layer of ViT-Ti$^-$ becomes less sharp, but by the end of training it becomes sharper than at initialization. Thank you for this suggestion!

Reviewer comment (Theory):
At the end of Section 4, page 6, clarification is needed on why $V_H^{-1}$ appears after "clip".

Author response:
Thank you for catching these! There should be no $V_H^{-1}$, no negative sign, and additional ||.

Reviewer comment (Theory):
On the same page, last paragraph, it appears that it should be $diag(\Lambda_H)=\Psi^2$ — without a negative sign.

Author response:
Thank you for catching these! There should be no $V_H^{-1}$, no negative sign, and additional ||.

Reviewer comment (Writing):
The paper should define Q in the main statement of Theorem-2.

Author response:
Good call, we have moved the definition from the proof to the theorem statement.

Reviewer comment (Writing):
On page 6, "||" is missing in the definition of asymmetry.

Author response:
Thank you for catching these! There should be no $V_H^{-1}$, no negative sign, and additional ||.

Reviewer comment (Evaluation):
The metrics for Table-2 and Table-3 should be described.

Author response:
Sorry, these are test accuracies, thank you for asking!

Reviewer comment (Theory):
The proposed method for avoiding oversmoothing is based on the claim that H becomes more symmetric as the training progresses. However, it is not confirmed that H eventually becomes a symmetric matrix. Therefore, parameterizing H as a symmetric matrix might restrict the expressive power of the Transformer model as a whole.

Author response:
Thanks for this comment. Our original motivation for using a symmetric parameterization was inspired by [Hu et al., 2019. Exploring weight symmetry in deep neural networks.] who show that weight symmetry in NNs does not affect their universal approximation capabilities. However, we agree that it would be interesting to test a non-symmetric parameterization. Based on your suggestion we have implemented a new parameterization for $\mathbf{H}$ as $\mathbf{H} = \mathbf{V}_H \Lambda \mathbf{V}_H^{-1}$. Specifically, we take gradients to $\mathbf{V}_H$ in the backwards pass, and compute $\mathbf{H}$ using the above equation in the forwards pass. For both the symmetric and non-symmetric versions we test out 2 variants: 1. (-) i.e., the sharpening model: $\mbox{diag}(\Lambda_H ) := −(\psi^2)$ and a new variant 2. (*) where the first L/2 layers are parameterized as the sharpening model above, and the remaining L/2 layers are left normal (as in ViT-Ti). We use the suffix “-NS” for the non-symmetric parameterizations. We have also included a new baseline, FeatScale [Wang et al., 2022], as well as a new experiment where we double the depth of the network. Here are the new accuracy results, averaged over 3 trials with standard deviations (we bold the best result in each column and any method whose mean plus standard deviation is greater than or equal to mean of the best result): 

Data Efficiency (Table 2)

| Data Kept          	| 100%     	| 50%      	| 10%      	|
|------------------|--------------|--------------|--------------|
| ViT-Ti       	| 66.78 ± 0.1  | 56.38 ± 0.5  | 32.47 ± 0.9  |
| ViT-Ti + FeatScale | 66.57 ± 0.5  | 57.08 ± 0.3  | 32.50 ± 0.4  |
| ViT-Ti$^+$   	| 64.34 ± 0.3  | 55.21 ± 0.5  | 28.67 ± 0.5  |
| ViT-Ti$^-$   	| 66.62 ± 0.3  | 56.58 ± 0.3  | **33.99 ± 0.4**  |
| ViT-Ti$^*$   	| 67.84 ± 0.6  | **57.78 ± 0.4**  | **33.64 ± 0.5**  |
| ViT-Ti-NS$^-$	| 67.80 ± 0.2  | 57.54 ± 0.1  | **33.78 ± 0.3**  |
| ViT-Ti-NS$^*$	| **68.53 ± 0.6**  | **57.67 ± 0.3**  | 33.75 ± 0.2  |


[new] Increased Depth (Table 3)

| Depth       	| 12       	| 24       	|
|----------------|--------------|--------------|
| ViT-Ti     	| 66.78 ± 0.1  | 67.22 ± 0.1  |
| ViT-Ti + FeatScale | 66.57 ± 0.5  | 67.71 ± 0.5  |
| ViT-Ti$^+$ 	| 64.34 ± 0.3  | 65.65 ± 0.3 |
| ViT-Ti$^-$ 	| 66.62 ± 0.3  | 66.76 ± 0.3  |
| ViT-Ti$^*$ 	| 67.84 ± 0.6  | 69.05 ± 0.1  |
| ViT-Ti-NS$^-$  | 67.80 ± 0.2  | 67.09 ± 0.1  |
| ViT-Ti-NS$^*$  | **68.53 ± 0.6**  | **69.64 ± 0.3**  |


Corruption Robustness (Table 4)

| Corruption Intensity | 0          	| 1          	| 2          	| 3          	| 4          	| 5          	|
|----------------------|----------------|----------------|----------------|----------------|----------------|----------------|
| ViT-Ti           	| 66.78 ± 0.1	| 61.59 ± 0.2	| 57.75 ± 0.2	| 53.12 ± 0.1	| 49.32 ± 0.1	| 39.17 ± 0.2	|
| ViT-Ti + Featscale   | 66.57 ± 0.5	| 61.25 ± 0.5	| 57.36 ± 0.6	| 52.64 ± 0.6	| 48.87 ± 0.5	| 38.77 ± 0.4	|
| ViT-Ti$^+$       	| 64.34 ± 0.3	| 58.36 ± 0.4	| 53.83 ± 0.5	| 49.03 ± 0.6	| 45.15 ± 0.6	| 35.86 ± 0.6	|
| ViT-Ti$^-$       	| 66.62 ± 0.3	| 61.62 ± 0.2	| 58.00 ± 0.3	| 53.56 ± 0.2	| 49.70 ± 0.3	| 40.07 ± 0.2	|
| ViT-Ti$^*$       	| 67.84 ± 0.6	| 62.49 ± 0.5	| 58.82 ± 0.7	| 54.32 ± 0.7	| 50.40 ± 0.6	| 40.38 ± 0.7	|
| ViT-Ti-NS$^-$    	| 67.80 ± 0.2	| **63.03 ± 0.2**	| **59.65 ± 0.1**	| **55.05 ± 0.3**	| **51.30 ± 0.1**	| **41.33 ± 0.1**	|
| ViT-Ti-NS$^*$    	| **68.53 ± 0.6**	| 63.01 ± 0.1	| **59.44 ± 0.3**	| **54.90 ± 0.4**	| **51.12 ± 0.4**	| 40.90 ± 0.4	|

We have added these results to the paper. The non-symmetric parameterization does improve upon the symmetric parameterization in some cases (particularly for increased depth and corrupted data). We appreciate your suggestion to investigate this.

Reviewer comment (Theory):
There is no definite answer to why the ViT-Ti^- version improves robustness and data efficiency. The intuition provided by the authors is not enough to answer this because robustness and data efficiency are more nuanced than simply choosing higher-frequency features.

Author response:
Thanks. This seems to be a very deep question. We originally suspected a link between sharpening and data efficiency because of the eigenvalue distribution of the Data Efficient Image Transformer (DeiT-B/16) in Table 1: compared to all other baseline models, it has the highest distribution of sharpening eigenvalues. This is what led us to test data efficiency. We also hypothesized that it may help with input corruption robustness as many of the corruption techniques in Hendrycks & Dietterich, 2019 blur or reduce the resolution of the image: any further smoothing would likely make classification even harder. As far as we are aware we are the first to investigate the relationship between sharpening and data efficiency and corruption robustness. We agree this is interesting to further investigate. If you have any suggestions for additional experiments we will try to implement them as quickly as possible.

Reviewer comment (Writing):
The writing of the paper is not polished, which creates confusion in key details of the paper. For example, given the redundancy in the cases in Theorem 1, maybe Theorem 1 could be rewritten to make the main result easier to read.

Author response:
Thank you for pointing out this redundancy! We have fixed this.

Reviewer comment (Experiments):
In Table 1, the authors show the distribution of the dominating eigenvalues but do not show whether the features oversmooth when \lambda_1^A dominates. This must be verified because the Theorems presented only work when the attention matrix A is fixed across layers, which does not hold for all practical settings. Therefore, showing the distribution of the dominating eigenvalues without showing how they affect the final features is vacuous.

Author response:
We believe there is a small confusion here. For ViT-Ti$^+$ we have that (a) Figure 3 shows that the features do oversmooth more and more, and (b) Table 1 shows that $\lambda_n^A$ dominates. On the other hand, for ViT-Ti$^-$ we have that (a) Figure 3 indicates that the features do not oversmooth, and (b) Table 1 shows that $\lambda_1^A$ dominates. This shows that even when the attention matrix $\mathbf{A}$ is not fixed across layers the results agree with the Theorem statements. Thank you for letting us clarify this, we will add these details to the text to make this clearer.

Reviewer comment (Writing):
The precise definition of dominating eigenvalues should be given to make Theorem 1 easier to read.

Author response:
We agree. We have pulled this definition out of the proof and placed it into the main paper.

Reviewer comment (Writing):
“To show a case where over-smoothing is guaranteed, we also define a model diag(ΛH) := −(ψ^2), which we refer to using the superscript +.” I think the authors mean diag(ΛH) := +(ψ^2).

Author response:
Thank you for catching this! We have fixed this.

Reviewer comment (Experiments):
It would be good if the authors could compare the runtime of their ViT-Ti^+ with other methods.

Author response:
Thanks. The training time of ViT-Ti versus our original proposed methods is as follows (averaged over 3 runs):
|     |  Training time |
|----------|---------------|
| ViT-Ti  |   5h 34min |
| ViT-Ti$^+$|  6h 03min |
| ViT-Ti$^-$ |  6h 03min |

The throughput of each method (in images/second, averaged over 10 runs) is:

|     |  Throughput |
|----------|---------------|
| ViT-Ti  |   1683.0 |
| ViT-Ti$^+$|  1642.6 |
| ViT-Ti$^-$ |  1639.0 |

We will add these timings to the text.

Reviewer comment (Experiments):
The authors showed the distribution of dominating eigenvalues of their - version on CIFAR10, but not on ImageNet. It would be interesting to see if the distribution of the dominating eigenvalues is still the same on ImageNet.

Author response:
We think there is a small confusion. The dominating eigenvalue distribution of our $\mbox{}^-$ version is always 0%: $(1 + \lambda^H_j \lambda^A_n)$ and 100%: $(1 +  \lambda^H_j \lambda^A_1)$, regardless of the data, as we control the eigenvalues via bounding $\Lambda^H$ in our parameterization. We will clarify this in the paper.