Paper: ZeroI2V: Zero-Cost Adaptation of Pre-Trained Transformers from Image to Video (ICLR-2024)

Reviewer comment (Writing):
I am not fond of the term "zero-cost adaptation." In reality, the adaptation process, which involves re-training, cannot be considered zero-cost. The zero-cost aspect only applies during inference after the linear adapter has been merged with the existing weights. Referring to it as zero-cost adaptation may be an overstatement.

Author response:
We sincerely apologize for any confusion. We have repeatedly emphasized in the abstract and the main paper that our meaning of “zero-cost adaptation” is that there is no additional computational and parameter cost during inference.  We would change the name  "zero-cost adaption" to "adaption with zero extra inference cost"  or "zero-extra cost adaption" in the final version for clarity.

Reviewer comment (Evaluation):
In my opinion, the full adaptation diagram still requires a significant amount of computational resources and memory for backpropagation, as there are a bunch of tunable parameters located in the shallow layers.

Author response:
We have supplemented the comparison of ZeroI2V and the previous PETL method regarding training time and training GPU memory usage in ***Table 14 of Appendix B***. It can be seen that although our method’s training efficiency is not as prominent as the inference efficiency compared to the previous PETL method, it is also not much more cumbersome. Compared to AIM that impose an additional computational burden for temporal modeling, our STDHA method, which does not introduce extra learnable parameters, ensures that ZeroI2V maintains superior training efficiency. And we believe that it is worthwhile and acceptable to exchange some training cost for a reduction in inference cost.

Reviewer comment (Experiments):
Is there a training wall clock time comparison with prior works? What is the total training parameter of linear adapters used during training in Table 2/3?

Author response:
I presume you might be referring to Table 3/4? As in Table 2, we have already listed the number of tunable parameters. As for the total training parameters of ZeroI2V in Table 3/4, we provide the final setting that we used in Table 6 in Appendix A.1. For most cases, we use 4 adapters per ViT block (14M for ViT-B and 50M for ViT-L).

Reviewer comment (Evaluation):
I noticed that the best figure in Table 2 for SSv2 Top-1 is 66.3. However, in Table 4, the corresponding number is 67.7. Which setting accounts for this improvement?

Author response:
The result of 66.3 was obtained using a single view test (center crop), while the result of 67.7 was obtained using a multi-view test (three crops). This evaluation method is consistent with previous work.

Reviewer comment (Experiments):
How do you select a specific head for different $\Delta t$? Has there been any ablation study conducted?

Author response:
Due to the existence of serial learnable adapters, we believe that it is unnecessary to deliberately select heads for different $\Delta t$. This is because different heads actually correspond to different feature dimensions, which in turn correspond to $W_{qkvo}$ in the attention layer. We believe that the learnable linear adapter can adaptively adjust the weights of $W_{qkvo}$ corresponding to each head during low-rank adaptation.

Reviewer comment (Experiments):
The authors claim ZeroI2V is a general method to adapt image models to the video domain, but the experiments are only done on the action recognition task, which does not require fine-grained spatial understanding as opposed to tasks like video segmentation. In order to properly support this claim there need to be experiments on another video task.

Author response:
**Q2: There need to be experiments on another video task**

Thank you for your valuable suggestions. To understand the capability of our method in fine-grained spatial understandingWe evaluate our method on action detection dataset AVA. Following the setting of VideoMAE. As shown in ***Table 9 of Appendix B*** , compared to using the original image CLIP features, our ZeroI2V achieved a significant performance improvement (26.4 vs 18.3) with the same number of parameters and computation. It’s noteworthy that our method was not pre-trained on action recognition datasets such as Kinetics. Instead, we directly applied image-to-video transfer on the AVA dataset. Remarkably, our method still managed to achieve performance on par with full-finetuning methods and self-supervised methods that underwent pre-training using the Kinetics dataset, even when using only 8 frames as input. In summary, our ZeroI2V demonstrates outstanding potential in video tasks beyond recognition.

Reviewer comment (Novelty):
Can you clarify the novelty of ZeroI2V compared to the Patch Shift Transformer introduced in [1]?

Author response:
**Q3: Can you clarify the novelty of ZeroI2V compared to the Patch Shift Transformer introduced?**

The TPS operation in the Patch Shift Transformer is placed before the self-attention layer. It achieves efficient modeling by mixing information from frames at different times, which is more like an extension of the channel shift proposed in TSM, rather than a profound improvement on the original MHSA operation. Our STDHA, on the other hand, starts from the redundancy of heads in MHSA. By assigning some heads that originally performed spatial modeling to complete temporal modeling tasks, we believe it maintains the purity of information within the same head (in contrast, in TPS, each head contains a portion of channels from other frames when calculating attention, i.e., all heads are simultaneously performing spatio-temporal modeling). This allows us to ultimately achieve better spatio-temporal modeling capabilities. This point is also demonstrated in Table 13 of Appendix B, where under the same settings, Swin-B w/ STDHA achieved significantly better performance than PST-B (70.0 vs 67.4).

Additionally, we understand that the reason you may find this confusing is because we use inter-frame attention for temporal modeling in STDHA, which also requires us to use a shift operation in our implementation. However, we believe that the key contribution of our proposed STDHA is not just in our use of inter-frame attention for temporal modeling. Rather, it lies in demonstrating the effectiveness and efficiency of using different heads for simultaneous spatial and temporal modeling, and in providing an efficient implementation based on inter-frame attention. In the future, we will explore more temporal modeling operations for the temporal head.

Reviewer comment (Experiments):
Which configuration/hyperparameter setting is chosen for ZeroI2V, based on the ablations, for the experiments in Section 4.3 that compare it to the state of the art?

Author response:
**Q5: Which configuration/hyperparameter setting is chosen for ZeroI2V**

In Appendix A.1, Table 6, we provide the final setting that we used in Section 4.3.

Reviewer comment (Writing):
What is the benchmark mentioned on page 2 in “establish a benchmark” on page 2? Usually this means there is a new dataset introduced.

Author response:
> **What is the benchmark mentioned on page 2 in “establish a benchmark” on page 2?**

We agree with your statement that a benchmark typically introduces a new dataset. In this case, we use the existing Something Something v2 dataset as our dataset for evaluating the temporal modeling capabilities of our model after transfer. We compared common image-to-video adaptation methods under fair comparison conditions, so we refer to it as “establishing a benchmark”. Similar expressions are also used in the introduction of ST-Adapter.

Reviewer comment (Writing):
How is “offering powerful capabilities” a "key perspective" of temporal modeling? It's unclear what idea the authors are trying to get across.

Author response:
> **How is “offering powerful capabilities” a "key perspective" of temporal modeling?** 

What we wanted to express here is that the “Capability of capturing the temporal dynamics” is a “key perspective” of temporal modeling. We have corrected this in the revised paper.

Reviewer comment (Writing):
Saying ZeroI2V is “reducing the difficulty” of image-to-video adaptation is also vague. It would be better to specifically mention reducing the inference cost.

Author response:
> **Saying ZeroI2V is “reducing the difficulty” of image-to-video adaptation is also vague**

Here, we are not just aiming to reduce the inference cost, but we are also concerned with reducing the training cost in image-to-video adaptation. STDHA, by not introducing additional learnable parameters and retaining the original spatial head for image modeling, also reduces the difficulty of training to a certain extent.

Reviewer comment (Writing):
Page 1 paragraph 1 sentence 2: Missing “and” between “CLIP” and “DINO”

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Writing):
Page 1 paragraph 1 sentence 4: Remove “the” from before “parameter efficient transfer learning”.

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Writing):
Page 2 paragraph 2 sentence 2: Why is the word “images” treated as a proper noun?

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Writing):
Page 2 paragraph 4 sentence 5: Incomplete sentence. Remove “find” from beginning of sentence.

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Writing):
Page 3 paragraph 2 sentence 3: Replace “image” with the plural “images”

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Writing):
Page 3 paragraph 3 sentence 1: Use past tense “was” instead of “is”. In sentence 3, missing definite article “the” before “video domain”.

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Writing):
Page 5 paragraph 4 sentence 1: Use “Assume“ instead of “assuming”.

Author response:
Thank you for your suggestions regarding grammar. We have corrected these issues in the revised paper. Additionally, we would like to address your questions about the content of the writing here.

Reviewer comment (Evaluation):
The results are convincing but need a bit more illustration. For example, MViTv2-L works better on SSv2 — is that from the model design, fully supervised training, or something else? Similarly, for AIM performing better on K400, where does the performance gap come from — the design or something else? What are the advantages and disadvantages of the proposed STDHA compared to commonly used action recognition models (not only the fine-tuned CLIP models)?

Author response:
**Q1: The results are convincing but needs a bit more illustration**

Thank you for your valuable suggestions. Due to space constraints, we did not elaborate in detail on the comparison of results. We will supplement it here and update it in our paper after adjusting the layout:

- **Why MViTv2-L works better on SSv2？** Firstly, from the perspective of model design, MViTv2 introduces more efficient downsampling designs compared to the original ViT, thereby resulting in lower FLOPs. This allows it to use a higher spatial resolution (312) and sampling frame number (40). Secondly, our method is based on image pre-training, while MViTv2 is based on video weights pre-trained on K400. As SSv2 places more emphasis on temporal modeling capabilities, MViTv2-L can achieve a better performance trade-off.
- **Why AIM better on the K400？** We understand that the K400 dataset places more emphasis on spatial modeling capabilities. Our STDHA, by making some heads that originally performed spatial modeling complete temporal modeling, has to some extent reduced the model’s spatial modeling capabilities. On the other hand, AIM, through the stack of temporal attention and spatial attention, fully retains the model’s original spatial modeling capabilities. Therefore, it achieves slightly stronger performance on K400, but at the cost of nearly 30% additional computational overhead.

**Q2: The advantage and disadvantages of the proposed STDHA comparing against the commonly used action recognition models**

- **advantage**：The first obvious advantage is that STDHA is highly efficient, requiring no additional learnable parameters or computational overhead. In addition, compared to other efficient temporal modeling methods, our STDHA is specifically designed for the characteristics of the attention mechanism and has stronger spatio-temporal modeling capabilities (we have discussed this in Section 3.2 of the main paper). Finally, due to the popularity of the attention mechanism, our method has a very wide range of applications and is plug-and-play (we have shown in Table 13 of Appendix B that our method can also achieve good performance when using Swin Transformer as the backbone).
- **disadvantage**：The main drawback of STDHA is that it makes some heads, which originally performed spatial modeling, complete temporal modeling. This to some extent impairs the model’s spatial modeling capabilities. This may result in slightly poorer performance in scenarios that require high spatial information modeling (for example, its performance on K400 is slightly inferior to AIM).

Reviewer comment (Novelty):
The novelty is a bit limited or not well highlighted, as the inter-frame attention is not originally from this paper. While the paper still has a solid idea on adapters at no additional cost, the authors are encouraged to give the intuition more illustration, e.g., why choose the inter-frame attention as part of the proposed STDHA.

Author response:
**Q3: Why pick the inter frame attention as part of the proposed STDHA:** 

- **Motivation of the choice of inter-frame attention:** Our method is designed for image-to-video adaptation, so we start from the perspective of trying to reduce the difficulty of image model transfer. A direct idea is to tile video frames into images for spatio-temporal modeling, but this is not ideal in terms of both computational efficiency and performance. Then we considered inter-frame attention. Since only the key and value of the current frame are replaced with those of other frames, it can be assumed that the data distribution of the key and value has not changed much when calculating attention. From this perspective, we replaced the temporal attention in AIM with inter-frame attention. We found that it has stronger temporal modeling capabilities in the image-to-video scenario:

| Method         | Spatial Modeling      | Temporal Modeling      | FLOPs | SSv2 Top1 |
| -------------- | --------------------- | ---------------------- | ----- | --------- |
| AIM       | spatial attention     | temporal attention     | 624   | 66.4      |
|                | spatial attention     | inter-frame attention  | 640   | 68.4      |
| ZeroI2V (ours) | spatial head in STDHA | temporal head in STDHA | 422   | 67.7      |

Moreover, the implementation of inter-frame attention is very simple and has the same computational cost as the original spatial attention, we can implement STDHA in a highly efficient manner.

- **The inter-frame attention is not originally from this paper：** We believe that the key contribution of our proposed STDHA is not just in our use of inter-frame attention for temporal modeling. Rather, it lies in demonstrating the effectiveness and efficiency of using different heads for simultaneous spatial and temporal modeling, and in providing an efficient implementation based on inter-frame attention.

Reviewer comment (Presentation):
The visualizations are not clear enough; consider including a CLIP ViT activation map for comparison.

Author response:
**Q4: CLIP VIT activation map there for comparison.**

Thank you for your valuable suggestions. We have added the CLIP VIT activation map for a more intuitive comparison in ***Figure 3 of Appendix C*** in our revised paper. It can be seen that STDHA pays more attention to the interaction of objects in the video (such as the touch between hands and cups or Rubik’s cubes), while CLIP only focuses on individual objects and does not capture the spatio-temporal dynamics in the video well.

Reviewer comment (Reproducibility):
I am curious about how is the STDHA implemented. Because it needs to split the heads into spatial and temporal, I am assuming it will introduce some other operations, although they may not contribute to FLOPs, but may still slow down the latency. However, in Table 2, the proposed method has exactly the same latency as the baseline.

Author response:
**Q1: how is the STDHA implemented?**

Our choice of inter-frame attention for temporal modeling in STDHA allows us to directly apply the shift operation to the key and value tensors associated with the temporal head. Shift operation is very lightweight and can be almost ignored compared to other operations that are counted into FLOPs (such as matrix multiplication). Specifically, we directly modified the `nn.MultiHeadAttention` in PyTorch. It’s worth mentioning that our STDHA does not conflict with some commonly used efficient attention implementations such as FlashAttention, and we can combine them to achieve higher computational efficiency.

Reviewer comment (Presentation):
In Table 1 (b), what is the meaning of 1/2 head?

Author response:
**Q2:  In Table 1 (b), what is the meaning of 1/2 head?**

$\Delta t=1/2$ means that half of the channels in this temporal head have the time offset of 1 and the other half have -1. This experiment actually violates our original intention of keeping the purity of the head, so we just treat it as an extreme case of ablation experiments.

Reviewer comment (Experiments):
In Table 4, ST-Adapter ViT-L/14 has a performance of 72.3/93.9, which is higher than the proposed method. I think it would be better to show the full comparison, and I don’t think it will degrade the significance of the work.

Author response:
**Q3: It would be better to show the full comparison with ST-Adapter**

Thank you for your valuable suggestion. Due to space constraints, we choose one method for comparison at different model sizes. We will add a more comprehensive comparison after adjusting the layout in the future.