Paper: Unitention: Attend a sample to the dataset (ICLR-2024)

Reviewer comment (Experiments):
While the methods approach the visual recognition problem through a novel perspective, i.e., the contextual information of the dataset, I suspect this is not very fair for the testing scenario, as testing should be performed via individual images, one by one.

Author response:
A: We thank the reviewers for raising concerns about improving the fairness of our methodology. Despite this concern, we believe that Unitention's test scenario is still *fair*. The reason for this is as follows:

- 1. It is generally acknowledged that during the testing, the use of training data is fair, whereas the use of test data (e.g., using **multiple** test samples in an inference, **not one by one**) is a data leakage. For instance, many machine learning models like k-NN, Decision Tree, SVM, Gaussian Process, are all based on training data's features. At the time of testing, these models already have information about the entire training dataset. They are considered fair, because no features of test data are memorized during the testing.

- 2. When testing, Unitention behaves quite similarly to these traditional machine learning models. It also uses multiple features from the training data (like what k-NN and Decision Tree do) to gain some understanding at the dataset level. So this is also a fair testing.

- 3. We conclude that the reason why Reviewer Ubis has such concerns is Unitention uses a new inference process that is fundamentally different from mainstream deep learning models: it uses a universal-individual feature fusion mechanism, whereas typical deep models have no such explicit dataset-level (universal) features. This reaffirms the originality and uniqueness of our approach.

We have updated our manuscript to explain more on the fairness of Unitention's testing phase. We hope our replies can address your concerns.

Reviewer comment (Experiments):
If one intends to include the contextual information from datasets, one also should not include the extra data in the dataset during the testing phase, otherwise, it is not fair.

Author response:
A: We thank the reviewers for raising concerns about improving the fairness of our methodology. Despite this concern, we believe that Unitention's test scenario is still *fair*. The reason for this is as follows:

- 1. It is generally acknowledged that during the testing, the use of training data is fair, whereas the use of test data (e.g., using **multiple** test samples in an inference, **not one by one**) is a data leakage. For instance, many machine learning models like k-NN, Decision Tree, SVM, Gaussian Process, are all based on training data's features. At the time of testing, these models already have information about the entire training dataset. They are considered fair, because no features of test data are memorized during the testing.

- 2. When testing, Unitention behaves quite similarly to these traditional machine learning models. It also uses multiple features from the training data (like what k-NN and Decision Tree do) to gain some understanding at the dataset level. So this is also a fair testing.

- 3. We conclude that the reason why Reviewer Ubis has such concerns is Unitention uses a new inference process that is fundamentally different from mainstream deep learning models: it uses a universal-individual feature fusion mechanism, whereas typical deep models have no such explicit dataset-level (universal) features. This reaffirms the originality and uniqueness of our approach.

We have updated our manuscript to explain more on the fairness of Unitention's testing phase. We hope our replies can address your concerns.

Reviewer comment (Evaluation):
How is including extra data during the testing phase fair for evaluation?

Author response:
A: We thank the reviewers for raising concerns about improving the fairness of our methodology. Despite this concern, we believe that Unitention's test scenario is still *fair*. The reason for this is as follows:

- 1. It is generally acknowledged that during the testing, the use of training data is fair, whereas the use of test data (e.g., using **multiple** test samples in an inference, **not one by one**) is a data leakage. For instance, many machine learning models like k-NN, Decision Tree, SVM, Gaussian Process, are all based on training data's features. At the time of testing, these models already have information about the entire training dataset. They are considered fair, because no features of test data are memorized during the testing.

- 2. When testing, Unitention behaves quite similarly to these traditional machine learning models. It also uses multiple features from the training data (like what k-NN and Decision Tree do) to gain some understanding at the dataset level. So this is also a fair testing.

- 3. We conclude that the reason why Reviewer Ubis has such concerns is Unitention uses a new inference process that is fundamentally different from mainstream deep learning models: it uses a universal-individual feature fusion mechanism, whereas typical deep models have no such explicit dataset-level (universal) features. This reaffirms the originality and uniqueness of our approach.

We have updated our manuscript to explain more on the fairness of Unitention's testing phase. We hope our replies can address your concerns.

Reviewer comment (Theory):
The relation between Unitention and codebook/k-NN is somehow overclaimed.

Author response:
A: We highly appreciate the helpful advice to avoid misleading readers. Traditional codebook motivates our high-level idea of using some universal "bank" to complement the deep feature. This high-level idea of Unitention does not limit the specific bank structure to be implemented. As mentioned in Section 2.2, there are a lot available ways to implement the bank, e.g., **label-free** FIFO queue, or **label-based** average class centers. In practice, we choose the class center only for its performance, and this does not contradict our original high-level motivation. We have updated the statements to mitigate any potential misunderstanding.

Reviewer comment (Reproducibility):
Typically, codebook is constructed using unsupervised methods, often employing K-means. However, in this work, it seems that the codebook is dependent on labels.

Author response:
A: Many thanks to the valuable advice. Besides adjusting the according statements in our manuscript, we further clarify about the **migration ability** of Unitention.

- 1. Unitention is not a pretraining method, and it does not aim to give some pretrained backbones. We tend to think of Unitention as an end application: if someone wants to tackle the classification task on domain B, with a backbone pretrained on domain A, we would suggest to load an existing pretrained backbone (without Unitention), and train it with Unitention directly on domain B (so that the class prototypes are built from scratch on B). This is what we actually do on iNaturalist benchmark, as introduced in Section 3.3 and Appendix D. We will refine Section 3.3 and Appendix D to make this clearer.

- 2. If there is a real need to provide migration capabilities for Unitention, we could use a FIFO queue bank to achieve this. However, this may not perform as well as class centers. This can be explained by the "no free lunch" theorem: generality often comes at the expense of performance.

Reviewer comment (Experiments):
The migration ability for annotated universal bank is limited. The universal bank essentially serves as a class prototype, requiring label annotations for aggregation. Such mechanism, which relies on annotation information, faces a challenge when migrating to other downstream tasks with different categories.

Author response:
A: Many thanks to the valuable advice. Besides adjusting the according statements in our manuscript, we further clarify about the **migration ability** of Unitention.

- 1. Unitention is not a pretraining method, and it does not aim to give some pretrained backbones. We tend to think of Unitention as an end application: if someone wants to tackle the classification task on domain B, with a backbone pretrained on domain A, we would suggest to load an existing pretrained backbone (without Unitention), and train it with Unitention directly on domain B (so that the class prototypes are built from scratch on B). This is what we actually do on iNaturalist benchmark, as introduced in Section 3.3 and Appendix D. We will refine Section 3.3 and Appendix D to make this clearer.

- 2. If there is a real need to provide migration capabilities for Unitention, we could use a FIFO queue bank to achieve this. However, this may not perform as well as class centers. This can be explained by the "no free lunch" theorem: generality often comes at the expense of performance.

Reviewer comment (Presentation):
The reliance on annotation information for the universal bank should be clearly outlined in section 2.2. Moreover, it is advisable to move the updating principle for universal banks to section 2.2.

Author response:
A: Many thanks to the valuable advice. Besides adjusting the according statements in our manuscript, we further clarify about the **migration ability** of Unitention.

- 1. Unitention is not a pretraining method, and it does not aim to give some pretrained backbones. We tend to think of Unitention as an end application: if someone wants to tackle the classification task on domain B, with a backbone pretrained on domain A, we would suggest to load an existing pretrained backbone (without Unitention), and train it with Unitention directly on domain B (so that the class prototypes are built from scratch on B). This is what we actually do on iNaturalist benchmark, as introduced in Section 3.3 and Appendix D. We will refine Section 3.3 and Appendix D to make this clearer.

- 2. If there is a real need to provide migration capabilities for Unitention, we could use a FIFO queue bank to achieve this. However, this may not perform as well as class centers. This can be explained by the "no free lunch" theorem: generality often comes at the expense of performance.

Reviewer comment (Reproducibility):
The algorithm is not clearly demonstrated. As clarified in the paper, the training-free approach is like a variation of k-NN. But how does the classifier come about for the test phase? By aggregating training samples?

Author response:
A: We first train a classification model using a standard supervised learning configuration. To obtain a k-NN classifier, we remove the linear classification head of the model, to get a pure backbone $f(\cdot)$.The deep features $f(x\_i)$ of all training samples $x\_i$ are used to build the k-NN classifier.

Reviewer comment (Experiments):
There is a lack of some empirical studies.

Author response:
A: We take ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s as an instance. Unitention adds around 5% extra overhead. Note this can be further optimized by operators such as FlashAttention, as Unitention performs a naive single-query cross-attention:

| Model | GPU hours &nbsp; | Peak GPU Mem |
| --- | --- | --- |
| ResNet-50 &nbsp; | 242.0 | 13.2 |
| ResNet-50 w/ Unitention &nbsp; | 255.1 | 13.8 |

Reviewer comment (Evaluation):
It is recommended to analyze Unitention training/inference efficiency, in particular the computational and storage demands for the additional 1000xC class-level feature bank.

Author response:
A: We take ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s as an instance. Unitention adds around 5% extra overhead. Note this can be further optimized by operators such as FlashAttention, as Unitention performs a naive single-query cross-attention:

| Model | GPU hours &nbsp; | Peak GPU Mem |
| --- | --- | --- |
| ResNet-50 &nbsp; | 242.0 | 13.2 |
| ResNet-50 w/ Unitention &nbsp; | 255.1 | 13.8 |

Reviewer comment (Experiments):
There is a lack of evaluation on other popular vision tasks, e.g., detection or segmentation.

Author response:
A: We highly agree (also stated in the old Limitation section of paper) that evaluation on detection or segmentation tasks can make our empirical conclusion more robust. We have done these experiments and reported the performance in the appendix, where we tested Mask R-CNN ResNet50-FPN on MS-COCO, and the UperNet Swin-T on ADE20k. The results are:

| Object Detection &nbsp; | Image Resize &nbsp; | Schedule &nbsp; | $\text{AP}^\text{box}$ | $\text{AP}^\text{box}\_{50}$ | $\text{AP}^\text{box}\_{75}$ |
| --- | --- |  --- |  --- |  --- |  --- |
| ResNet-50 &nbsp; | (384, 600) | 1$\times$ | 38.9 | 59.6 | 42.7 |
| ResNet-50 w/ Unitention &nbsp; | (384, 600) | 1$\times$ | 40.1 (+1.2) &nbsp; | 59.9 (+0.3) &nbsp; | 44.2 (+1.5) &nbsp; |

| Instance Segmentation &nbsp; | Image Resize &nbsp;  | Schedule &nbsp; | $\text{AP}^\text{mask}$ | $\text{AP}^\text{mask}\_{50}$ | $\text{AP}^\text{mask}\_{75}$ |
| --- | --- |  --- |  --- |  --- |  --- |
| ResNet-50 &nbsp; | (384, 600) | 1$\times$ | 35.4 | 56.5 | 38.1 |
| ResNet-50 w/ Unitention &nbsp; | (384, 600) | 1$\times$ | 36.3 (+0.9) &nbsp; | 57.1 (+0.6) &nbsp; | 39.0 (+0.9) &nbsp; |

| Semantic Segmentation &nbsp; |  Crop Size &nbsp; | Schedule &nbsp; | $\text{mIoU}$ |
| --- | --- |  --- |  --- |
| Swin-T &nbsp; | (512, 512) | 160k | 44.5 |
| Swin-T w/ Unitention &nbsp; | (512, 512) | 160k | 45.2 (+0.7) |

Unitention brings similar gains across these three dense prediction tasks, which showcases its effectiveness.

Reviewer comment (Experiments):
About the hyperparameter analysis on $\tau$: It seems that tuning $\tau$ obtains positive gains with momentum 0.8 setting, i.e., row 7, 9, 10 and 11 in Table 4. Why is $\tau$ fixed as 1 finally?

Author response:
A: Thanks for pointing out the typo. In row 9, 10, and 11 in Table 4, $\tau$ should be 1.0. We have corrected this.

Reviewer comment (Experiments):
In Table 1, is the k-NN accuracy obtained by the model trained with only a k-NN classifier?

Author response:
A: As explained earlier in the response to (W3), we first train the classification model using standard supervised learning. We then  remove the learned linear classification head, and perform k-NN inference on the pure backbone. We have updated the according statements in Section 2.1.

Reviewer comment (Evaluation):
Parameters, flops and inference speeds are not provided. It shows that an Unitention module contains multiple FC layers and a feature bank. When applying an attention module to a backbone, it will improve its performance but also increase its parameter and flop usually. The authors should provide the parameters, flops and inference speeds for models with and without Unitention module, so that we can make sure whether Unitention can make a good trade-off between performance and cost. For example, the parameters, flops and inference speeds of ResNet50, ResNet101, ResNet152, ViT-Small, Swin-Tiny with and without Unitention.

Author response:
A: We have reported these in our updated version, hoping they can help the readers better see the efficiency-effectiveness trade-off in our Unitention. For instance, the increases in parameters and flops are:

| Model | ResNet-50 | ResNet-101 | ResNet-152 | ViT-S &nbsp; | Swin-T &nbsp; |
| --- | --- | --- | --- | --- | --- |
| Para (M) &nbsp;&nbsp; | 25.6 | 44.5 | 60.2 | 22 | 29 |
| $\Delta$Para (M) &nbsp;&nbsp; | +1.4 | +1.4 | +6.9 | +1.4 | +1.4 |
| FLOPS (G) &nbsp;&nbsp; | 4.1 | 7.9 | 11.6 | 4.6 | 4.5 |
| $\Delta$FLOPS (G) &nbsp;&nbsp; | +0.3 | +0.3 | +1.2 | +0.3 | +0.3 |

Overall, we believe the extra overhead is theoretically acceptable. We also measure the wall-clock time and memory footprint of ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s:

| Model | GPU hours &nbsp; | Peak GPU Mem |
| --- | --- | --- |
| ResNet-50 &nbsp; | 242.0 | 13.2 |
| ResNet-50 w/ Unitention &nbsp; | 255.1 | 13.8 |

Where Unitention introduces about 5% additional cost. Note this can easily be further reduced by operators such as FlashAttention, as Unitention  uses a naive single-query cross-attention.

Reviewer comment (Experiments):
The authors considered different models and modalities in the experiments. While Unitention was only tested in single-label global-feature-based classification tasks in the experiments. I think the authors could provide some experimental results of Unitention for different tasks such as multi-label classification to show its generalization ability.

Author response:
A: We highly agree to benchmark Unitention on more challenging tasks beyond the single-and-global classification, e.g., the per-pixel-classification (semantic segmentation). As we also expressed in the Limitation section, we would like to test Unitention on dense prediction tasks including detection and segmentation. Now we've finished these and reported the results in appendix, where we tested a CNN (Mask R-CNN ResNet50-FPN) on MS-COCO, as well as a transformer (UperNet Swin-T) on ADE20k:

| Object Detection &nbsp; | Image Resize &nbsp; | Schedule &nbsp; | $\text{AP}^\text{box}$ | $\text{AP}^\text{box}\_{50}$ | $\text{AP}^\text{box}\_{75}$ |
| --- | --- |  --- |  --- |  --- |  --- |
| ResNet-50 &nbsp; | (384, 600) | 1$\times$ | 38.9 | 59.6 | 42.7 |
| ResNet-50 w/ Unitention &nbsp; | (384, 600) | 1$\times$ | 40.1 (+1.2) &nbsp; | 59.9 (+0.3) &nbsp; | 44.2 (+1.5) &nbsp; |

| Instance Segmentation &nbsp; | Image Resize &nbsp;  | Schedule &nbsp; | $\text{AP}^\text{mask}$ | $\text{AP}^\text{mask}\_{50}$ | $\text{AP}^\text{mask}\_{75}$ |
| --- | --- |  --- |  --- |  --- |  --- |
| ResNet-50 &nbsp; | (384, 600) | 1$\times$ | 35.4 | 56.5 | 38.1 |
| ResNet-50 w/ Unitention &nbsp; | (384, 600) | 1$\times$ | 36.3 (+0.9) &nbsp; | 57.1 (+0.6) &nbsp; | 39.0 (+0.9) &nbsp; |

| Semantic Segmentation &nbsp; |  Crop Size &nbsp; | Schedule &nbsp; | $\text{mIoU}$ |
| --- | --- |  --- |  --- |
| Swin-T &nbsp; | (512, 512) | 160k | 44.5 |
| Swin-T w/ Unitention &nbsp; | (512, 512) | 160k | 45.2 (+0.7) |

The consistent performance gains from Unitention showcase its generalization ability.

Reviewer comment (Experiments):
From my point of view, Unitention is an enhanced design of classification head focusing on the information of classes, since it uses feature vectors after the global average pooling operation. I think the authors should compare Unitention with other classification head designs, for example, iSQRT-COV[1]. Or the authors could show that Unitention is complementary with them, which can still demonstrate performance gains on the models with other heads.

Author response:
A: Thanks for this valuable advice. Since Unitention offers a way to improve heads that is orthogonal to existing methods (i.e. through a cross-data individual-universal fusion mechanism), it is worth investigating whether it can complement those existing designs. In addition to iSQRT-COV [r1] on ResNet, we also test another different head design (SoT [a1] on Vision Transformers, suggested by reviewer zCBo) to make the results more convincing. The results are as follows and are also included in the appendix:

| Model | baseline (our impl.) &nbsp; | w/ [r1,a1] (our impl.) &nbsp; | w/ [r1,a1] and Unitention |
| --- | --- | --- | --- |
| ResNet-50 &nbsp; | 78.9 | 80.1 (w/ [r1]) | 80.8 (w/ [r1] and Unitention) |
| Swin-T | 81.3 | 82.6 (w/ [a1]) | 83.2 (w/ [a1] and Unitention) |

From the results one can see that Unitention can further improve the performance of other head designs due to its new universal-individual mechanism.

Reviewer comment (Experiments):
In the ablation study, how does Unitention update the feature bank when the number of class centers is less than number of classes?

Author response:
A: At the begging of training, we random select 100/1000 classes and only maintain the class centers of them during the whole training. We have updated this detail in ablation study.

Reviewer comment (Experiments):
I'm wondering about the performance of Unitention when the feature bank is replaced by trainable parameters. That is, the parameters in the feature bank is updated according to the gradient but not proposed method.

Author response:
A: Thanks for such a creative ablation. Following the setting of Section 3.5, we tried it out and found it achieved an accuracy of 79.4%, close to a FIFO queue. We feel that a trainable feature bank would still be hopeless to beat class centers, as class centers directly use the information from labels. This ablation is added to our study and we hope it can help the readers better understand our Unitention.

Reviewer comment (Experiments):
Will the backbones with Unitention still show performance gains when fine-tuned on the downstream tasks such as detection and segmentation?

Author response:
A: See the response to (W2) above.

A: We highly agree to benchmark Unitention on more challenging tasks beyond the single-and-global classification, e.g., the per-pixel-classification (semantic segmentation). As we also expressed in the Limitation section, we would like to test Unitention on dense prediction tasks including detection and segmentation. Now we've finished these and reported the results in appendix, where we tested a CNN (Mask R-CNN ResNet50-FPN) on MS-COCO, as well as a transformer (UperNet Swin-T) on ADE20k:

| Object Detection &nbsp; | Image Resize &nbsp; | Schedule &nbsp; | $\text{AP}^\text{box}$ | $\text{AP}^\text{box}\_{50}$ | $\text{AP}^\text{box}\_{75}$ |
| --- | --- |  --- |  --- |  --- |  --- |
| ResNet-50 &nbsp; | (384, 600) | 1$\times$ | 38.9 | 59.6 | 42.7 |
| ResNet-50 w/ Unitention &nbsp; | (384, 600) | 1$\times$ | 40.1 (+1.2) &nbsp; | 59.9 (+0.3) &nbsp; | 44.2 (+1.5) &nbsp; |

| Instance Segmentation &nbsp; | Image Resize &nbsp;  | Schedule &nbsp; | $\text{AP}^\text{mask}$ | $\text{AP}^\text{mask}\_{50}$ | $\text{AP}^\text{mask}\_{75}$ |
| --- | --- |  --- |  --- |  --- |  --- |
| ResNet-50 &nbsp; | (384, 600) | 1$\times$ | 35.4 | 56.5 | 38.1 |
| ResNet-50 w/ Unitention &nbsp; | (384, 600) | 1$\times$ | 36.3 (+0.9) &nbsp; | 57.1 (+0.6) &nbsp; | 39.0 (+0.9) &nbsp; |

| Semantic Segmentation &nbsp; |  Crop Size &nbsp; | Schedule &nbsp; | $\text{mIoU}$ |
| --- | --- |  --- |  --- |
| Swin-T &nbsp; | (512, 512) | 160k | 44.5 |
| Swin-T w/ Unitention &nbsp; | (512, 512) | 160k | 45.2 (+0.7) |

The consistent performance gains from Unitention showcase its generalization ability.

Reviewer comment (Novelty):
The motivation, the idea, and the cross-attention design of the proposed Unitention are not new. As a fundamental research topic, there already exist a large number of previous research works that focus on designing a better head structure or enhancing feature encoding for the classification head. Unfortunately, the authors totally ignore this line of research, failing to acknowledge or compare with representative prior works such as those by Mircea Cimpoi et al. (CVPR 2015), Relja Arandjelovic et al. (CVPR 2016), Yang Gao et al. (CVPR 2016), Feng Zhu et al. (CVPR 2017), Mengran Gou et al. (CVPR 2018), Shilong Liu et al. (arXiv 2021), Jiangtao Xie et al. (arXiv 2021), Ke Zhu et al. (CVPR 2021), and Chao Li et al. (ICLR 2023).

Author response:
We appreciate Reviewer zCBo’s constructive feedback, especially the additional related work which is very helpful in highlighting the uniqueness and novelty of our Unitention. However, the judgement of *“the motivation, the idea and the cross-attention design of the proposed Unitention are not new”* seems to misunderstand the core difference between Unitention and related work [r1-r9], thus may lead to a *misjudgment* of the value and novelty of Unitention. We are sorry about this and will clarify this later. We'd like to response to each of concerns as follows.

----------------

**[To additional related literature required]**

Thanks for pointing out the work **[r1-r9]** which helps a lot to consolidate our work. **[r1-r2] are Bag-of-Words** methods which we have already discussed in Section 4.1/4.3, where we've shown their fundamental difference to Unitention. **For head-enhancing work [r3-r9]**, we did a thorough review and summarized it in a new Related Work subsection in the revised pdf. Now we're addressing the *core concern* on the novelty of Unitention compared to [r3-r9], as follows.

&nbsp;

**[To concerns on novelty]: “given [r1-r9], the motivation, the idea and the cross-attention design of the proposed Unitention are not new”**

(i) &nbsp;The "not new" is somewhat excessive and could be a misjudgment. First, [r3-r5, r7-r9] **didn't use either** a cross-attention operator **or** a cross-data fusion (universal-individual) mechanism.

(ii) &nbsp;Although [r6] uses cross-attention operator (the same as in DETR [a1] and Seq2Seq Transformers [a2, a3]), it **does not use** cross-data fusion mechanism at all (which is the core design of Unitention), thus is still fundamental different from Unitention, as explained at the bottom of page 4. We repeat here that 1) the [r6, a1-a3] uses Q, K, V from one *single* data point and model parameters, whereas 2) Unitention uses key-value pairs from *multiple different* data points. This difference makes cross-attention operator act as two **radically different functions** in [r6, a1-a3] vs. in Unitention: one to select different regions of a *single* image for different recognition targets (one cat's face, two cat's claws, etc.), the other to compare one *single* data with all other data in the *whole* training set and make some *holistic* understanding.

(iii) &nbsp;A full comparison between Unitention and [existing r3-r9, extra ones we surveyed a1-a5] is summarized as follows:

| Methodology | What features to fuse | Cross-attention used? | Cross-data fusion? |
| --- | --- | --- | --- |
| Bilinear pooling and improvements [r3, r5] | the feature of one single data point, form one or two models. | $\times$ | $\times$ |
| Improved global pooling [r7, r8, a5] | the feature of one single data point (r7: different token embedding) | $\times$ | $\times$ |
| Spatial attention [r4, a4] | the feature of one single data point | $\times$ | $\times$ |
| Channel-split attention [r9] | non-overlapping feature groups of one feature from one data point | $\times$ | $\times$ |
| Query-based transformer decoder (DETR, translation model, etc.) [r6, a1-a3] | the feature from one data point | $\checkmark$ | $\times$ |
| Unitention (ours) | features from different data points | $\checkmark$ | $\checkmark$ |

(iv) &nbsp;In summary, [r1-r9, a1-a5] can be regarded as additional related work to consolidate our work. They won't devalue the main contribution and novelty of Unitention. *Instead, they would* reinforce the originality and uniqueness of Unitention, which is evident from the comparison above and also acknowledged by Reviewer CKwM as well as Reviewer Ubis. We have updated our Related Work to discuss and explain this, hopefully clearing up any possible misunderstandings.

&nbsp;

(to be continued)

Reviewer comment (Experiments):
In the experimental section, the authors fail to provide comprehensive comparisons of the proposed method to existing works that also focus on designing a better head structure or enhancing feature encoding for the classification head. Such comprehensive comparisons are necessary to validate the contributions.

Author response:
**[To other advice/questions about experiments]**

> More comparisons with existing works [r1-r9]

A: Unitention shows a *cross-data* mechanism to improve classification heads, different from existing *single-data-based* head-enhancing modules like [r3-r7]. We highly agree that it is valuable to verify whether Unitention can complement other methods, and this is also suggested by reviewer yV5u. Here, we apply Unitention on two open-sourced model that also experimented on ImageNet (SoT [r7] and iSQRT-COV [a5]), and evaluate the validation accuracy:

| Model | baseline (our impl.) &nbsp; | w/ [r7,a5] (our impl.) &nbsp; | w/ [r7,a5] and Unitention |
|---|---|---|---|
| ResNet-50 | 78.9 | 80.1 (w/ [a5]) | 80.8 (w/ [a5] and Unitention)|
| Swin-T | 81.3 | 82.6 (w/ [r7]) | 83.2 (w/ [r7] and Unitention)|

The further performance improvements demonstrate that Unitention can complement existing advanced head designs. We have included these results in appendix as extra empirical assessments.

Reviewer comment (Evaluation):
The authors do not report or discuss the extra training cost of the proposed Unitention method compared to the baseline.

Author response:
> How about the extra training cost of the proposed method against the baseline?

A: Take ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s as an example. Unitention introduces about 5% additional overhead (note there is still room for efficiency improvement, e.g., by using FlashAttention, as Unitention performs a naive single-query cross-attention):

|Model| GPU hours &nbsp; |Peak GPU Mem|
|---|---|---|
|ResNet-50|242.0|13.2|
|ResNet-50 w/ Unitention &nbsp; |255.1|13.8|