Paper: EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens (ICLR-2024)

Reviewer comment (Experiments):
The results on SSv2 and K400 was compared only for 200 epochs. Results on K400 and SSv2 with more epochs?

Author response:
####  $\rightarrow$ We first respectfully hope that you could understand the limited budget of our research group. We want to emphasize this work is on academic paper to cut down the immense scale of video unsupervised learning, rather than industrial paper using extreme resources. 

Additionally, We validated the strengths of our proposed EVEREST through extensive experiments on multiple architectures (ViT-S/B/L), benchmarks (UCF101, HMDB51, SSv2, K400, Ego4D), and tasks (action recognition, OSCC) with sufficient analyses in our main paper that consistently show better efficiency and comparable performance to VideoMAE, so we expect our method to also be effective in longer training period settings.


And, we politely want to say that **we provided very long training epochs results in our main paper**: We validated our method by training 4800 and 3200 epochs** on the UCF101 and HMDB51 datasets (Please see Table 7), respectively, and achieved superior fine-tuning accuracy along with memory/training time/computational efficiency by a large margin (Please see Table 2 and Figure 5 in the paper).

Reviewer comment (Evaluation):
Looks like the results of VideoMAE and EVEREST of ViT-L at 200 epochs are about the same.

Author response:
#### $\rightarrow$ We would like to emphasize that the **goal** of this work is to make the masked video modeling (MVM) more **_efficient_** without performance drop rather than just achieving state-of-the-art performance.  Several works [1,2,3] reducing redundant tokens of ViT in the image domain have achieved its efficiency but sacrificed performance drop. However, we elaborate on how to effectively reduce redundant tokens in the spatiotemporal domain and find a way to achieve efficiency without performance drop. Therefore, we would like to highlight that it is meaningful to increase efficiency without losing performance. Furthermore, the computation and memory efficiency of our EVERST enables us to train MVM models using only **single node** with 8 GPUs, which helps the vision community to **lower the barrier to further SSL for video research.**

Reviewer comment (Experiments):
The paper mostly compare with VideoMAE, but there have been multiple developments in MAE for videos. I would encourage the authors to compare the approach with more baselines e,g, AdaMAE [1], MME [2], MVD [3], ST-MAE [4] for videos (using random masking), Omnimae [5].

Author response:
#### $\rightarrow$ First, we want to politely emphasize **we already provided the comparison with MME[2] and MVD[3] in Table 4 and 5 in our submission!**

ST-MAE [4] has a very similar architecture to VideoMAE and also performs similarly (For example, VideoMAE outperforms ST-MAE in [3], and ST-MAE shows on-par performance In [2].). So, we use VideoMAE as our counterpart baseline, instead of ST-MAE, given the limited resources. 

#### OmniMAE [5] can learn versatile representation compatible with image and video domains. However, they require massive computational costs (similar GFLOPs and Params with MVD [3]) while achieving a **_lower_** action recognition performance than VideoMAE and MVD [3]. 

#### MVD [3] distills knowledge from pre-trained image and video teachers during pre-training. MVD improves performance on top of VideoMAE due to knowledge transfer from **rich and heavy teacher models**, but **it suffers from nontrivial computational costs and slow training time**. MME [2] predicts the motion trajectory of the masked video contents, which is encoded by HOG and optical flow for modeling the shape and the position of moving objects. MME strategy can capture motion more accurately. Yet, it **requires computing HOG and optical flow** of all input video data before training, resulting in another **heavy computation burden**.

####  AdaMAE [1] introduces an auxiliary token sampling network to MAE, which estimates categorical distribution over spacetime-patch tokens and then samples higher spatiotemporal tokens from the distribution. Due to the adaptive sampling network, it can reduce the masking ratio from 90% (VideoMAE) to 95% while achieving better performance than VideoMAE. However, its computation reduction rate is not significant and it needs to further train the auxiliary network. However, Unlike MME and AdaMAE, our EVEREST can capture motion information **_in an online manner without an external module or additional training_**. Furthermore, our method can be applied even when **the fine-tuning phase** (all other methods cannot), which demonstrates that our EVEREST is a more general and flexible method. To support this claim, we already showed the finetuning computation and memory usage comparison in **Table 5**.

Reviewer comment (Experiments):
It would be great to add a comparison of the masking strategy against multiple masking strategies.

Author response:
#### $\rightarrow$  We have already compared multiple masking strategies by visualizing the masked results (Please see Figures 7 and 10.), and it shows our ReRo token selection successfully captures the informative tokens and removes redundant tokens from the given videos.

Reviewer comment (Evaluation):
Although the approach seems to be computationally less expensive, the results are about the same as VideoMAE, sometimes even degrades the performance of pre-trained masked autoencoder methods for videos (c.f, Table 5).

Author response:
#### $\rightarrow$ We would like to emphasize that the **goal** of this work is to make the masked video modeling (MVM) more **_efficient_** without performance drop rather than just achieving state-of-the-art performance.  Several works [1,2,3] reducing redundant tokens of ViT in the image domain have achieved its efficiency but sacrificed performance drop. However, we elaborate on how to effectively reduce redundant tokens in the spatiotemporal domain and find a way to achieve efficiency without performance drop. Therefore, we would like to highlight that it is meaningful to increase efficiency without losing performance. Furthermore, the computation and memory efficiency of our EVERST enables us to train MVM models using only **single node** with 8 GPUs, which helps the vision community to **lower the barrier to further SSL for video research.**

Reviewer comment (Evaluation):
The paper could benefit from confidence intervals for the reported accuracies in Tables 1 and 2.

Author response:
$\rightarrow$ Thank you for your constructive comments! We respectfully hope that you understand the limited short rebuttal period. We could train our model repeatedly with different random seeds to provide confidence intervals. However, we only have a little time to offer it. Instead, we have provided the implementation code already in the supplementary material, and anyone can easily reproduce it and see similar results in Tables 1 and 2. We will make sure to report the confidence interval for our main table in our final revision.

Reviewer comment (Writing):
In equation (3), following the notation in section 4.1., should not $\tilde{k}^n$ be $(\tilde{k}')^n$ since $m^n$ is a mask over $\tau \times [J \cdot \rho_{pre}]$? Also, $\tilde{v}$ is not defined (I assume it’s parts of the video that contain $(\tilde{k}')^n$).

Author response:
$\rightarrow$ Sorry for the confusion. We understood the point you were confused about and clarified the explanations of notations in our revision. Here, we briefly correct some notations.

Let $\widetilde{\boldsymbol{\mathit{k}}}$ be the token embeddings of the top-$[\tau\cdot\rho_{pre}]$ highest importance based on Eq. (2) and $\widetilde{\boldsymbol{\mathit{k}}}’$ be the randomly sampled token embeddings from $\widetilde{\boldsymbol{\mathit{k}}}$ with the ratio of $\rho_{post}$. Then, an input forwarded by $f_{\mathcal{W}}$ is $\widetilde{\boldsymbol{\mathit{k}}}’$, and the reconstructed target is $\{\boldsymbol{\mathit{m}}} \otimes {\boldsymbol{\mathit{v}}}$. Please see the revision for details.

Reviewer comment (Presentation):
It would be interesting to see the full curves for VideoMAE (75% and 90%) in Figure 5, not just at 3200 pre-training epochs.

Author response:
$\rightarrow$ Thanks for suggesting such a great idea. It would show the effectiveness and efficiency of our model clearly. However, [1] only reports the results of 3,200 pre-training epochs in their paper, and we don't have enough time and resources to implement their method during this short rebuttal period. After this rebuttal period, we'll update the full curves in our final revision.

Reviewer comment (Experiments):
A comparison to VideoMAE2 could also be interesting.

Author response:
$\rightarrow$ Thank you for the great suggestion. VideoMAE-V2 simply attaches running cell masking [3] for decoder masking in VideoMAE. Running cell masking is an input-independent ad hoc approach; the model constructs square cells based on the spatial location of the frame (i.e., meta-patch), and then randomly samples tokens from **unselected regions in the previous frame** for masking. That is, **VideoMAE-V2** simply aims to mask different regions for each frame (i.e., mask redundancy) and **cannot capture visual redundancy** according to the input videos. To see clear differences and limitations in running-cell masking used in VideoMAE-V2, **we additionally visualized real masking examples** in **Figure 8** in our latest revision. As shown, while EVEREST can focus on informative spatial-temporal tokens in incoming videos, running-cell masking just avoids the overlap of masks between adjacent masking and **cannot capture visual redundancy ($\neq$ masking redundancy) or semantic importance**.

For quantitative analysis, during the rebuttal period, we have compared the effect of our redundancy-robust token masking with running-cell masking used in MAR [3] and VideoMAE-V2. We pre-trained with these masking strategies and performed conventional full-finetuning. As shown in the table below, **our EVEREST clearly outperforms running-cell masking**, demonstrating our effectiveness against other selection strategies given the same architecture.

| method |UCF101|HMDB51|
|---------------|------------|-----------|
| Running cell | 91.0 | 61.4 |
| EVEREST (Ours)| **93.4** | **68.1**| 

[3] Qing et al., MAR: Masked Autoencoders for Efficient Action Recognition, IEEE Transactions on Multimedia, 2023

Reviewer comment (Novelty):
The novelty of the proposed method is limited. First, I would say the similarity-based token selection method has been studied before. An important baseline is K-centered Patch Sampling in [1]. The core idea is close, and the method both highly rely on the rank of the similarity matrix. Second, the conclusion or observation of this paper is somewhat novel but has also been showed in previous work. In particular, the VideoMAE-V2 [2] has already been shown in VideoMAE pretraining and fine-tuning. There is plenty of redundancy, so they introduce dual-masking. The conclusion both show us that the efficiency of training video models can be further improved. Since the high-mask ratio reconstruction (90%~95% in the video) has told us the key of VideoMAE pre-training, which has been discussed very well, the further improvement in efficiency can be viewed as incremental, and such improvement has been shown in [2]. So, I would say the observation is not that novel as well.

Author response:
> ***W1-1. The novelty of the proposed method is limited: the similarity-based token selection method has been studied before. An important baseline is K-centered Patch Sampling in [1]. The core idea is close, and the method both highly rely on the rank of the similarity matrix.***

[1] Park, Seong Hyeon, et al. "K-centered patch sampling for efficient video recognition." ECCV 2022  

$\rightarrow$ We respectfully disagree that the idea of our method is close to [1], and we believe the reviewer misunderstands our method in some sense - ***we do not compute any form of the similarity matrix***. 

K-centered patch sampling [1] maps all patch embeddings in the vector space and sample K patches by **running the greedy farthest point sampling algorithm** (Please see Figure 2 in the [1]). This sampling strategy has a critical limitation in that they cannot avoid sampling redundant patches. Suppose two video patches $A_1$ and $A_2$ are identical, and there is another patch $B$ that is the farthest with $A_1$ and $A_2$. If the sampling algorithm picks $A_1$ (or $A_2$), it will pick $B$ (the most distant from $A$). Next, the algorithm will pick $A_2$ (or $A_1$), which is the farthest from $B$. But, as shown, *selecting the farthest tokens* based on the algorithm [1] doesn't mean *minimizing redundancy*.

Unlike [1], our EVEREST guarantees to prevent redundant sampling with our intuitive yet powerful redundant-robust token selection strategy. Furthermore, we suggest on-the-fly information-intensive frame selection that adaptively determines semantic dense (i.e., informative) frames based on the occupancy of our selected redundancy-robust patches.

Therefore, K-centered [1] and ours seem to share similar intuitions, but our proposed method is much more efficient and practical. This is obviously observed in Table 1 in the submission.

(This is a partial table from Table 1 in the submission.)
| method | PT-GFLOPs | FT-GFLOPs | Top-1 Acc |
|---------------|------------|-----------|-----------|
| K-centered [1] |  67.4| 369.5| 74.9| 
| EVEREST (Ours)| **6.3 ($10.7 \times\downarrow$)**| **29.1 ($12.7 \times\downarrow$)** | **75.9 ($1.0$ %p $\uparrow$)**| 
| EVEREST (Ours)| **21.5 ($3.13 \times\downarrow$)**| **98.1 ($3.77 \times\downarrow$)** | **79.2 ($4.3$ %p $\uparrow$)**| 

Our proposed EVEREST surpasses these baselines by significant margins in terms of Top-1 accuracy and computation efficiency (GFLOPs) on the K400 dataset, demonstrating that our approach contains sufficient originality in design, and clearly differs from [1].

Reviewer comment (Experiments):
The significance is limited. Since this is an experimental-based paper, the significance mainly relied on the results or improvement over current state-of-the-art or strong baselines. First, the performance is compared under a very short pre-training period in Table 1. I am concerned about this since, in the original paper, at least an 800-epoch training should be conducted for a fair comparison. Second, when measuring the wall-clock time memory consumption to show efficiency, different devices are used. To make an accurate claim, I believe they should be benchmarked on the same device. Finally, the proposed method only achieved comparable performance under limited budgets, so "state-of-the-art" performance should be carefully used in the abstract.

Author response:
> ***W2. The significance is limited since this is an experimental-based paper.***    

$\rightarrow$ We politely disagree with your claim that the significance is limited due to our main contributions being from the experiments, yet strongly believe that **a simple yet effective method could make a meaningful contribution to the research community.** Many of the high-impact work such as dropout [1], prompt-tuning [2], chain-of-thought prompting [3], and parameter-efficient Transfer Learning Techniques (Adapter [4], LoRA [5], etc.) are all impressively simple, but the simplicity is rather an advantage as it promotes more widespread adaptation of the method. On the other hand, there are many complicated and esoteric architectures and algorithms that have not been adapted as widely due to the difficulty in implementation and reproduction of the results.

As you already commented that our intuitive and efficient approach shows really surprising/novel results, we hope you focus more on the advantage of the method, as it has a **great potential to lower the barrier for video-related research by significantly reducing the computations, memory usage, and training time while preserving the models' performance, which will save time and cost of the researchers as well as reduce the carbon emission.**


[1] Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, JMLR 2014  
[2] Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, EMNLP 2021  
[3] Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, NeurIPS 2022  
[4] Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, ICML 2019  
[5] Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR 2022

Reviewer comment (Experiments):
The presentation can be improved. The ablation study is disorganized since the author uses different datasets and different models, even different devices. I would see a clear ablation for the proposed method is needed for better presentation. In Table 1, I cannot understand the comparison of the computation saving since it only happened when compared with VideoMAE. I cannot understand Tables 3–5 as well since there is insufficient consistency in the author's experiments.

Author response:
> ***W2-2. & W3. Measuring the wall-clock time memory consumption to show efficiency is not an accurate claim since different devices are used. The ablation study is disorganized since the author uses different datasets and different models, even different devices.***   

$\rightarrow$ This is a clear misunderstanding. Throughout Table 3, 4, and 5, we aim to show ours **consistently and stably outperforms baselines on efficiency regardless of hardware and experimental setups**, and we do believe **these multiple evaluations with diverse scenarios make our claim stronger**, rather than weaker.

We confirm that we measured memory/wall-clock time/performance for the comparison in each table with identical experimental setups with the same devices for all baselines and ours. **There is no doubt that our results are accurate and fair**.

However, we expect that an important point of confusion for the reviewer is that memory usage can vary due to differences in hardware. We want to politely emphasize that **it is a very well-known commonsense that different architectures have varying efficiencies in memory utilization, instruction sets, and internal scheduling, caching \& memory management. This means they handle processes differently, which can lead to **different memory usage patterns even for the same model**. As we already clarified, for Tables 3 and 4, the reported result is about the memory size of the **pre-training**. But one difference is that we deploy models on A6000 in Table 3 and A100 in Table 4, where A100 is probably a newer generation than A6000. 
- Therefore, there could be variations in their specific implementations. These differences may impact how memory is utilized or managed, even under similar workloads. 
- Both GPUs have tensor cores optimized for AI workloads, but there might be differences in the number or efficiency of these cores between the two models. This can affect how computations are performed and, consequently, how memory is used. 
- GPUs employ various caching strategies and memory management techniques to optimize performance. Differences in these strategies between the A100 and A6000 could result in different patterns of memory usage.

On the other hand, for Table 5, the reported result is about the memory size of the **fine-tuning**, as clarified in its caption. 

We hope you now could understand that the different values in Tables 3,4, and 5 are not our weaknesses but are **strong evidence demonstrating the robustness of our EVEREST across different hardware**.

Reviewer comment (Experiments):
How do you choose the hyper-parameters for the proposed token selection method?

Author response:
> ***Q1. How do you choose the hyper-parameters for the proposed token selection method?***

$\rightarrow$ As described in experimental settings in Section 5, we follow the same training protocols as VideoMAE. For hyperparameters like $\rho_{pre}$ for our EVEREST, we select them through a standard hyperparameter tuning strategy via the validation set.