Paper: VideoGLUE: Video General Understanding Evaluation of Foundation Models (ICLR-2024)

Reviewer comment (Evaluation):
VideoGLUE Score: (As the authors mentioned in Limitation Section) The proposed VideoGLUE score focuses solely on the computation-performance tradeoff, overlooking other valuable aspects, e.g., memory usage, model size, sample efficiency, model architecture difference. Given the significant influence of datasets and evaluation protocols on research directions, there is a concern that the current benchmark and scoring system might lead practitioners to overlook these crucial aspects of comprehensive video AI development.

Author response:
* Re W1 "The proposed VideoGLUE score focuses solely on the computation-performance tradeoff."

As reviewer 1CiA commented, “The suggested VideoGLUE score can be an interesting **initial** way to start a comparison and discussion of the strength of foundation models”. We acknowledge that it is only an initial attempt to score FMs by their video understanding capabilities, and it is not comprehensive as we discussed the limitations (more details in Appendix A). 

Nonetheless, it captures the essence and can help the design of new metrics in future work. In particular, we design VideoGLUE score (VGS) as the model performance weighted by the number of tunable parameters, which are arguably the two most important factors when evaluating an FM [1, 2]. 

[1] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E. and Brynjolfsson, E., 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[2] Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y.J. and Gao, J., 2022. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35, pp.9287-9301.

Reviewer comment (Experiments):
Selection of tasks: The benchmark is limted to three tasks, which may not fully capture the generalizability of FMs in video understanding. The inclusion of additional tasks, e.g., repetition counting, (long-term) action anticipation, human-object interaction, and so on, could provide a more comprehensive evaluation.

Author response:
* Re W2 "Selection of tasks are limited to three tasks."

 We appreciate the reviewer’s suggestion, and we have to leave them to future work from the community. As you may notice, completing the evaluation on one set of metrics in our paper requires more than one hundred experiments (six foundation models, four adaptation methods over eight datasets), not to mention the hyperparameter tuning and other explorations. Hence, we indeed made hard decisions about which to include in VideoGLUE. 

We are working on a new video task/dataset. Should bandwidths permit, we will work on the second version of VideoGLUE by including the new task and investigating the tasks the reviewer suggested.

Reviewer comment (Experiments):
Selection of motion-focused benchmarks: Athough SSv2 is a widely acknowledged motion-focused dataset, existing literature [a, b] indicates that strong performance is achievable without motion modeling on SSv2 due to strong correlation between objects and action classes in SSv2. Utilizing Something-Else [a] as an alternative could offer a purer evaluation of motion modeling capability by breaking those spurious correlation. Furthermore, FineGym [c] provides hierarchical action labels allowing for a more granular assessment of a model modeling capability.

Author response:
* Re W3 "Selection of motion-focused benchmarks."

We did consider Something-Else against SSv2 and FineGym against Diving and decided to go with SSv2 and Diving because they are well received by the community. We can swap them in the future if their drawbacks become an obstacle for fair evaluations.

Reviewer comment (Evaluation):
Is there any reason why image-native FM CoCa outperforms other FMs on D48, which mitigates static contextual biases but focuses on motion?

Author response:
* Re Q1 "Is there any reason why image-native FM CoCa outperforms other FMs on D48"

We note that although image-native CoCa and CLIP perform well on D48 in the end-to-end fine-tuning setting and the adapter setting, respectively, video-native FMs perform better in the frozen feature setting and the multi-layer attention pooler setting. Because D48 is small, it is unlikely to provide sufficient supervision for end-to-end fine-tuning the spatiotemporal weights in the video-native FMs. However, it still signifies the video-native FMs’ motion understanding capabilities when the backbones are frozen.

Reviewer comment (Evaluation):
This paper introduces a scalar VideoGLUE score to assess FM's efficacy and efficiency by averaging performance scores across four adaptation methods. Yet, the rationale behind the metric's design appears arbitrary. It's unclear why this particular weighted score, derived from the specified datasets and adaptation methods, is indicative of FMs' video understanding capabilities.

Author response:
As reviewer 1CiA commented, “The suggested VideoGLUE score can be an interesting **initial** way to start a comparison and discussion of the strength of foundation models”. We acknowledge that it is only an initial attempt to score FMs by their video understanding capabilities, and it is not comprehensive as we discussed the limitations (more details in Appendix A). Nonetheless, it captures the essence and can help the design of new metrics in future work. In particular, we design VideoGLUE score (VGS) as the model performance weighted by the number of tunable parameters, which are arguably the two most important factors when evaluating an FM [1, 2]. 

[1] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E. and Brynjolfsson, E., 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[2] Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y.J. and Gao, J., 2022. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35, pp.9287-9301.

Reviewer comment (Evaluation):
In Figure 3, CoCa achieves an average score of approximately 36 under the MLAP adaptation. However, Table 5 lists the average score for CoCa as 45.9. Why aren't they consistent?

Author response:
Thank you for catching this! We made a typo in Table 5. Figure 3 is the correct one. We have updated the manuscript accordingly.

Reviewer comment (Evaluation):
Different datasets emphasize varied aspects in video tasks; for instance, SSV2 focuses on motion, while Kinetics is more context-centric. How does VideoGLUE address the differences among these diverse datasets?

Author response:
* Re "How does VideoGLUE address the differences among these diverse datasets?"

We choose a diverse set of video datasets/tasks on purpose. These diverse video datasets/tasks help us probe into FMs’ video understanding capabilities from different perspectives. For example, SSv2 and Kinetics emphasize motion and appearance, respectively. Video classification and temporal action localization test an FM’s holistic understanding versus temporal reasoning.

Reviewer comment (Experiments):
While the study delves into transformer-based Foundation Models (FMs), is there any comprehensive analysis or comparison involving 3D-CNN or even 2D-CNN based FMs?

Author response:
* Re "is there any comprehensive analysis or comparison involving 3D-CNN or even 2D-CNN based FMs?"

We followed the following definition of foundation models: “any model that is trained on broad data that can be adapted (e.g., finetuned) to a wide range of downstream tasks” [1] to choose the most representative image- and video-based foundation models. Interestingly, the resultant FMs happen to be all transformer-based, potentially because the model developers chose the transformer architecture considering its better scalability than CNNs [2, 3].

[1] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E. and Brynjolfsson, E., 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. and Uszkoreit, J., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

[3] https://twitter.com/giffmana/status/1717995379034071483

Reviewer comment (Novelty):
Despite the contribution of suggesting a VideoGLUE score, there are no novel models, modifications, or datasets presented in the paper. In my opinion, it would be very beneficial to develop at least one of them as an additional contribution to make the strongest possible paper.

Author response:
We appreciate the reviewer for this invaluable suggestion as we are actively working on a new video task/dataset. We cannot include it here because of the high workload involved. As one may notice, completing this work's evaluation on one set of metrics in our paper requires more than one hundred experiments (six foundation models, four adaptation methods over eight datasets), not to mention the hyperparameter tuning and other explorations. Including new models, modifications, or datasets would be great complements to this project but also require extensively more work. We will highlight this limitation in the revised paper.

Reviewer comment (Experiments):
The list of studied foundation models is not as comprehensive as potentially can be. I understand, that not models are publicly available but it would be very interesting and important to make even stronger conclusions by including some of video-based models developed on top of CLIP paradigm. Some of the examples are: "VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding", Xu et al., EMNLP 2021; "Expanding Language-Image Pretrained Models for General Video Recognition", Bolin et al., ECCV 2022.

Author response:
Regrettably, we have lost the priority of computing resources for this project, and we are unable to provide further results for other models. Nevertheless, the paper has included two “video-based models developed on top of CLIP paradigm”: InternVideo and VATT. Especially, VATT is more or less a “superset” of VideoCLIP suggested in the review because VATT’s pretraining data and method cover VideoCLIP’s. Similarly, InternVideo can be viewed as a “superset” of the second paper mentioned in the review.

Reviewer comment (Experiments):
Also, there is a rapid development of adaptation techniques that could be also very nice to be included in the analysis. Some of the recent examples are: "AIM: Adapting Image Models for Efficient Video Action Recognition", Yang et al., ICLR 2023; "Frozen CLIP Models are Efficient Video Learners", Lin et al., ECCV 2022; "ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning", Pan et al., NeurIPS 2022. The listed papers are mostly adapters from images to videos, but it would be still great to have them in the analysis for some of the applicable foundation models.

Author response:
Thanks for pointing these out. In our exploration phase, we tried multiple adapters, including the “ST-Adapter” suggested by the reviewer. However, we were unable to reproduce ST-Adapter’s results reported in the original paper. In contrast, LoRA was the most stable and widely embraced one with superior performance, so we chose to report LoRA in the paper.