Paper: TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale (ICLR-2024)

Reviewer comment (Novelty):
I mainly concern about the technical contribution. As mentioned in the draft, the proposed degradation-free training strategy freezes the shallow layers while tuning the deeper layer of text encoder. This training strategy is more like an engineering trick by tuning parts of the large-scale model, which can be commonly used for practical large-scale training.

Author response:
**A:** Thanks, we are willing to highlight the technical merits of our work, both from **experimental results** and the **underlying motivation**.

**(i)** Experimentally, to demonstrate that our degradation-free training strategy goes beyond a mere engineering workaround, we introduced an alternative baseline. This baseline involves a ViT-B/32 model equipped with adapters [1] (M$ _{\text{adapter}}$) inserted between layers of the text encoder, where only the adapter-introduced parameters are adjustable during training. The retrieval results on MSR-VTT, detailed in Appendix D Table 16, show that this approach is inferior to our method. This reinforces our assertion that end-to-end tuning can lead to style overfitting, even when the text encoder remains unchanged, proving that our partially frozen strategy is not a simplistic fix but a considered solution.

**(ii)** From a motivational standpoint, scalability remains a significant and challenging research area, as recognized by Reviewer hjUh. This field is marked by several pioneering works featured in leading conferences, like VideoMAE V2 [2] and FLIP [3]. In our view, it is the simplicity and efficacy of these methods that render them ideal as foundational models, requiring minimal code adjustments for integration into existing frameworks. Upon acceptance, we plan to release all relevant codes, pre-trained models, and well-crafted APIs for immediate application. We are confident that these foundational models will greatly benefit the broader community, particularly academic institutions with limited resources for large-scale experiments. Our contributions, we believe, merit recognition at prestigious conferences such as ICLR.

| Method | R@1 | R@5 | R@10 | MdR |
| :--- | :---: | :---: | :---: | :---: |
| CLIP | 30.6 | 54.4 | 64.3 | 4.0 |
| M$ _{\text{ours}}$ | 34.5 | 58.5 | 67.7 | 3.5 |
| M$ _{\text{adapter}}$ | 28.8 | 54.8 | 65.7 | 4.0 |

> [1] Parameter-Efficient Transfer Learning for NLP, ICML 2019

> [2] VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking, CVPR 2023

> [3] Scaling Language-Image Pre-Training via Masking, CVPR 2023

Reviewer comment (Novelty):
Different tuning: The key difference between the two versions is how to tune the text encoder. It is an interesting finding but may not be a qualified novelty for a new conference paper. And the poor performances caused by the weak initialization (Appendix D)?

Author response:
**W1.1: The partially frozen tuning strategy lacks novelty.** 

**A:** Thanks, we are willing to emphasize the novelty of the proposed partially frozen tuning. Note that it plays a vital role (rather than a trick) in training on the easy-to-scale-up ASR-video pairs, which is not trivial given that few previous papers consider tackling the text distortion issue. We carefully break down the experiment step and pinpoint the problem. We identify the problem via comprehensive experiments and figure out a simple yet effective solution that is in favor of practical use and can be well transferred to different datasets (Table 6) and vision models (Table 7). The logic, motivation and ablation studies are mostly acknowledged by all reviewers.

To further underscore our contribution, we introduce a ViT-B/32-based alternate baseline, integrating adapters [1] between text encoder layers, denoted as M$_{\text{adapter}}$. During training, only the adapter-introduced parameters are trainable. The MSR-VTT retrieval results, detailed in the subsequent table, show that this alternative method yields comparatively lower performance. This outcome reinforces our assertion that end-to-end tuning can cause style overfitting, even when the text encoder remains unaltered. It also confirms that our approach of partially frozen tuning is not just a superficial fix, but a substantive and effective solution to the identified problem. We have added this discussion in our revised manuscript (Appendix D Table 16).

| Method | R@1 | R@5 | R@10 | MdR |
| :--- | :---: | :---: | :---: | :---: |
| CLIP | 30.6 | 54.4 | 64.3 | 4.0 |
| M$_{\text{ours}}$ | 34.5 | 58.5 | 67.7 | 3.5 |
| M$_{\text{adapter}}$ | 28.8 | 54.8 | 65.7 | 4.0 |

We believe that a paper's contribution should not be limited to proposing novel methods. Extensive empirical studies (i.e., analyzing why performance degrades), useful findings/insights (i.e., the overfitted text model), scalable training schemes (i.e., the simple yet practical partial freezing), and foundation models (i.e., TVTSv2 models that can generalize to various tasks) are also essential to the community, especially in the era of big models. Such contributions are also acknowledged by Reviewer hjUh (Task-Agnostic Focus, Novel Pre-training Strategy, etc.) and Reviewer A6Wr (Effective Pre-training Strategy).

> [1] Parameter-Efficient Transfer Learning for NLP, ICML 2019

**W1.2 TVTSv1's poor performances may be caused by weak initialization (Appendix D).**

**A:** We admit that a strong initialization yields better performance, as reported in Appendix D Table 20. **However, naively relying on stronger initialization leads to degraded performance when pre-training on large-scale ASR-paired videos.** As shown in Table 1 in our initial manuscript (the subsequent Table) , after equipping CLIP weights, all fully tuned variants still lag behind the partially tuned competitors (L3 vs L7, L5 vs L8), and they even degrade compared to the CLIP baseline (L1 vs L3, L5), indicating the importance of the proposed partial frozen tuning strategy for producing general-purpose video representations.

| No | Method | ASR | Alt-text | Kinetics-400 (ZS AR) | DiDeMo (ZS T2V) |
| :--- | :--- | :---: | :---: | :---: | :---: |
| L1 | CLIP | $\times$ | $\times$ | 42.7 (+0.0) | 24.7 (+0.0) |
| L3 | M$_{\text{ASR-Full}}$ | YT-Temporal | $\times$ | 45.7 (+3.0) | 19.3 (-5.4) |
| L5 | M$_{\text{All-Full}}$ | YT-Temporal | WebVid-2.5M | 47.9 (+5.2) | 24.2 (-0.5) |
| L7 | M$_{\text{ASR-Partial}}$ | YT-Temporal | $\times$ | 48.1 (+5.4) | 26.8 (+2.1) |
| L8 | M$_{\text{All-Partial}}$ | YT-Temporal | WebVid-2.5M | 50.8 (+8.1) | 29.8 (+5.1) |

Reviewer comment (Novelty):
Same objectives/architecture/masking: The transcript sorting task and masking techniques have also been used in TVTSv1, though the masking strategies are different. And the architectures are the same (as Frozen[1]), where the residuals are skip-connected.

Author response:
**W1.3: TVTSv1 and TVTSv2 share similar objectives/architecture/masking strategy.**

**A:** Thanks. We would like to first emphasize that the central focus of our research is on scalable pre-training, not on introducing new methods or modules. Scalability is a significant and complex area of study, as acknowledged by Reviewer hjUh. There is a wide range of cutting-edge research in this field that doesn't necessarily involve new network architectures, as seen in works like VideoMAE V2 [1] and FLIP [2]. In line with this, we plan to release all our codes and pre-trained models, along with user-friendly APIs, upon acceptance. We believe that these foundational models will be a valuable asset to the community, particularly for academic labs with limited resources for large-scale experiments. The substantial contributions of our work are worthy of recognition by top-tier conferences like ICLR.

In addition, we would like to point out that the differences between TVTSv1 and TVTSv2 are significant, for two primary reasons:

**(i)** Our approach in TVTSv2 involves a thorough and logical analysis for selecting the best attention mechanism for video representation learning. We start with a key question: which type of attention is most effective for this purpose? While TVTSv1 uses joint attention, we introduce a new baseline in Appendix D Table 19 for comparison. This new baseline shows that simply extending the attention range can actually degrade spatial understanding. Furthermore, we explore the optimal method to initialize temporal weights. As demonstrated in Appendix D Table 19, initializing temporal weights with zeros beats its random initialization competitor, indicating that growing temporal reasoning ability out of nothing avoids hurting the well-learned spatial prior, thus more suitable for training video foundation models.

**(ii)** We also pioneer the exploration of masked contrastive learning in large-scale pre-training in the video domain. Previous methods, like MAEs [1, 3, 4], focused on pixel-level rather than semantic-level learning, which limited their utility in learning versatile features, as evidenced by weak linear classification results in Appendix D Table 13. In contrast, TVTSv2 employs direct contrastive learning without pixel reconstruction, leading to promising results that underscore the benefits of semantic-level pre-training based on masked representations.

> [1] VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking, CVPR 2023

> [2] Scaling Language-Image Pre-Training via Masking, CVPR 2023

> [3] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, NeurIPS 2022

> [4] Masked Autoencoders As Spatiotemporal Learners, NeurIPS 2022

Reviewer comment (Experiments):
For DiDeMo, the code shows that it was tested on test split. Should it be tested on validation split?

Author response:
**Q1: For DiDeMo, the code shows that it was tested on test split. Should it be tested on validation split?**

**A:** Thanks for your concern. We follow the prior works [1, 2, 3] to conduct paragraph-to-video retrieval on the DiDeMo test split for a fair comparison. And we have double-checked the split used in their official code implementation to avoid any unfairness.

> [1] CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval, Neurocomputing 2022

> [2] CLIP-ViP: Adapting Image-Text Pre-training to Video-Language Representation Learning, ICLR 2023

> [3] Bridging Video-text Retrieval with Multiple Choice Questions, CVPR 2022

Reviewer comment (Presentation):
In Table 2, why the single-stream models are de-emphasized?

Author response:
**Q2: In Table 2, why are single-stream models de-emphasized?**

**A:** The single-stream model refers to those using a fusion module to encode multimodal entangled representations. Besides the much higher computational complexity during indexing (i.e., $O(N*M)$ for single-stream models and $O(N+M)$ for ours), the single-stream models are incapable of out-of-the-box usage, deviating from the motivation of this work. For instance, these models cannot serve for linearly probing action recognition (Appendix D Table 13) since they cannot encode video-only representations, while our pre-trained foundation models are capable of various downstream tasks without any adaptation. We have revised our manuscript for a clearer statement.

Reviewer comment (Experiments):
In Table 2, why the authors do not consider ViT-L and directly scale it to ViT-H?

Author response:
**Q4: In Table 2, why do the authors not consider ViT-L and directly scale it to ViT-H?**

**A:** Our initial focus was to assess the efficacy of our proposed pre-training framework on base-sized models. Subsequently, we sought to evaluate its effectiveness when scaled up to larger models. However, this scaling presents significant challenges in terms of time and resource consumption. For instance, training the ViT-H/14 model required a substantial investment of time and resources, taking 10 days on 80 V100 GPUs (detailed information can be found in Appendix B Table 10). Given our limited computational resources, it was impractical to train both the ViT-L/14 and ViT-H/14 models simultaneously within our submission deadline. Consequently, to more effectively demonstrate scalability, we opted to train the larger ViT-H/14 model.

We have since completed training on the ViT-L/14 variant and its performances are presented in the following table. These comprehensive results are now included in the updated Table 2. In summary, the ViT-L/14 model's performance aligns with the expected scaling rule, positioning itself performance-wise between the ViT-B/16 and ViT-H/14 models.

| Method | MSR-VTT R@1 | MSR-VTT R@10 | DiDeMo R@1 | DiDeMo R@10 | LSMDC R@1 | LSMDC R@10 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Ours-B/32 | 34.5 | 67.7 | 31.2 | 68.3 | 16.1 | 38.7 |
| Ours-B/16 | 35.9 | 71.3 | 33.4 | 70.6 | 16.9 | 38.2 |
| Ours-L/14 | 36.9 | 72.9 | 33.9 | 71.0 | 17.1 | 40.6 |
| Ours-H/14 | 38.2 | 73.2 | 34.6 | 71.5 | 17.3 | 41.4 |