Paper: FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition (ICLR-2024)

Reviewer comment (Experiments):
The authors touch upon "Text rephrasing" in Sec. 4.2, yet there's an absence of any ablation study related to it.

Author response:
**R:** The impact of text rephrasing is shown in the last two rows of Tab. 4 of the paper, which is presented below. It can be found that rephrasing the class names into descriptions can improve performance since they provide more detailed information about the actions. The corresponding experimental analysis is given in Sec. 4.3 of the paper.

**Table 1: Impacts of text rephrasing under the cross-dataset evaluation setting.**
| Method | UCF101 | HMDB51 | K-600 |
| :------: | :------: | :------: | :------: |
| Ours w/o text rephrasing | 84.4 | 54.4 | 73.0 $\pm$ 0.7 |
| Ours | **85.0** | **54.5** | **74.8** $\pm$ 0.9 |

Reviewer comment (Experiments):
In the caption of Table 1, the authors mentioned that 'The results of most of the other papers are taken from Open-VCLIP and ViFi-CLIP.', where are the results for Open-VCLIP? From Table 2 and Table 3, if my understanding is correct, the reported results use the Open-VCLIP as the backbone. Hence, showcasing Open-VCLIP's results in Table 1 becomes critical to underscore the effectiveness of the suggested method.

Author response:
**R:** Please refer to Tab. 1 of our paper. Thanks for the suggestion and we have included the suggested comparison. Below, we present the results of VCLIP. The results demonstrate that our method can improve the performances for both the base and novel categories for all datasets.

**Table 2: Performance (top1-acc (\%)) under the base-to-novel evaluation setting. B, N, and HM denote the base set, novel set, and harmonic mean, respectively.** 
| Method | K400(B) | K400(N) | K400(HM) | UCF(B) | UCF(N) | UCF(HM) | HMDB(B) | HMDB(N) | HMDB(HM) | SSv2(B) | SSv2(N) | SSv2(HM)$ |
| :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |
| VCLIP | 76.5 | 62.6 | 68.9 | 70.3 | 50.4 | 58.7 | 94.8 | 77.5 | 85.3 | 16.0 | 11.0 | 13.0 |
| **VCLIP w/ours** | **77.8** | **64.3** | **70.4** | **74.1** | **58.0** | **65.1** | **95.3** | **80.0** | **87.0** | **18.3** | **12.2** | **14.6** |

Reviewer comment (Experiments):
It would be better to report the finetuned results on all datasets (including K-400) for a better comparison.

Author response:
**R:** Thanks for the kind suggestion and we have included the additional results in Tab. 5 of Appendix A.3. Since this paper focuses on the open-vocabulary recognition setting, we did not include the closed-set recognition results (fine-tuning and evaluation are conducted on the same dataset) in the initial submission. In Tab. 1 below, we include the closed-set evaluation results on K-400. We can see that our method achieves comparable performance to the state-of-the-art closed-set models (AIM and ViFi-CLIP). At the same time, our method consistently outperforms them by a large margin under the cross-dataset evaluation, i.e., UCF101, HMDB51, and K-600 datasets. The results indicate that the state-of-the-art closed-set methods fit the seen categories well but cannot be satisfactorily generalized to unseen ones. To evaluate the overall performance under the closed-set and open-vocabulary settings, we report the harmonic mean of them in Tab. 2 below. From Tab. 2, we find that our method achieves the best trade-off on both the closed-set and open-vocabulary settings. 

By the way, inspired by this suggestion, developing a more comprehensive protocol to evaluate overall performance on both closed-set and cross-dataset scenarios is a promising direction for our future work.

**Table 1: Performance (top1-acc (\%)) of different methods under the closed-set (K-400) and cross-dataset (UCF,HMDB, and K-600) evaluation settings. Except for ER-ZASR (TSM), all the methods are based on the CLIP-ViT-B/16 model.** 

| Method | K-400 | UCF101* | UCF101 | HMDB51* | HMDB51 | K-600 |
| :------: | :------: | :------: | :------: | :------: | :------: | :------: |
ER-ZASR | - | - | 51.8 $\pm$ 2.9 | - | 35.3 $\pm$ 4.6 | 42.1 $\pm$ 1.4 |
Frozen CLIP | 57.2 | 74.2 | 73.8 $\pm$ 0.6 | 46.3 | 47.9 $\pm$ 0.5 | 68.1 $\pm$ 1.1 |
ActionCLIP | 82.6 | 77.4 | 77.5 $\pm$ 0.8 | 48.0 | 48.2 $\pm$ 1.5 | 62.5 $\pm$ 1.2 |
X-CLIP | 83.8 | - | 72.0 $\pm$ 2.3 | - | 44.6 $\pm$ 5.2 | 65.2 $\pm$ 0.4 |
VPT | 76.9 | - | 69.3 $\pm$ 4.2 | - | 44.3 $\pm$ 2.2 | 55.8 $\pm$ 0.7 |
Text4Vis | 82.9 | 76.4 | 76.4 $\pm$ 0.6 | 44.5 | 46.1 $\pm$ 0.2 | 60.1 $\pm$ 0.5 |
AIM | **83.9** | 79.0 | 79.4 $\pm$ 1.0 | 49.5 | 50.3 $\pm$ 0.8 | 66.7 $\pm$ 0.5 |
ST-Adapter | 82.0 | 77.9 | 77.6 $\pm$ 0.7 | 50.3 | 51.1 $\pm$ 0.6 | 60.2 $\pm$ 1.8 |
Vita-CLIP | 81.8 | - | 75.0 $\pm$ 0.6 | - | 48.6 $\pm$ 0.6 | 67.4 $\pm$ 0.5 |
ViFi-CLIP | **83.9** | - | 76.8 $\pm$ 0.7 | - | 51.3 $\pm$ 0.6 | 71.2 $\pm$ 1.0 |
Open-VCLIP | 81.8 | 83.5 | 83.4 $\pm$ 1.2 | 53.2 | 53.9 $\pm$ 1.2 | 73.0 $\pm$ 0.8 |
**Ours** | 82.0 | **85.0** | **84.8** $\pm$ 1.1 | **54.5** | **54.8** $\pm$ 1.3 | **74.8** $\pm$ 0.9 |

**Table 2: Harmonic mean (top1-acc (\%)) of different methods under the closed-set (K-400) and cross-dataset (UCF,HMDB, and K-600) evaluation settings.**

| Frozen CLIP | Action CLIP | X-CLIP | VPT | Text4Vis | AIM | ST-Adapter | Vita-CLIP | ViFi-CLIP | Open-VCLIP | **Ours** |
| :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |
| 60.1 | 64.8 | 62.9 | 58.9 | 63.0 | 67.4 | 65.3 | 65.6 | 68.4 | 70.8 | **71.9** |

Reviewer comment (Reproducibility):
The results in Table 1 are a little bit confusing, as the "Top1 accuracy" is only 77.8 on base categories. It would be better to further clarify the training and evaluation protocol, and the number of clips/crops used.

Author response:
**R:** Under the base-to-novel evaluation setting, each class consists of only 16 samples in the training set. This setting was first introduced by ViFi-CLIP [1], where the model is trained on base (seen) classes in a few-shot manner and evaluated on a set of novel (unseen) classes. Due to the limited training samples, the performance on the base categories is relatively low. 

For the training and evaluation protocol, we use videos with 8 frames for training and sample 3 clips (8 frames each) with 1 crop (3x1 views) for evaluation. 

The above protocol details have been given in Appendix A.1 and A.2 of the paper, please refer to them.

Reviewer comment (Novelty):
I have concerns on the novelty. Knowledge distillation on CLIP is not new in this field. The authors didn't discuss related references as follows. Besides, the authors almost follow the classic knowledge distillation framework except for residual feature distillation. [a] CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval, CVPR 2023. [b] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation, ACL 2022.

Author response:
**R:** Thanks for pointing out the CLIP-based distillation method (CLIPPLING [1] and VLKD [2]), which helps improve the literature analysis in our paper. Below, we compare our method with CLIPPLING [1] and VLKD [2]:

- **(a)** CLIPPING focuses on model compression, which aims to fully transfer the knowledge of the Clip4clip (a big teacher model) to the MobileViT-v2 (a small student model). In contrast, we aim to efficiently adopt a pre-trained CLIP model, such that the resulting model maintains the strong generalization capacity of a pre-trained CLIP while being video-specific. **(b)** The key technical contributions of CLIPPING are a layer-wise alignment scheme and a distillation strategy based on video-text distributions. Both the two components are to ensure the student model can fully absorb the knowledge of the CLIP model. Differently, we design a residual feature distillation module emphasizing the balance between generalizability maintenance and video-specific feature learning, which goes beyond just mimicking the CLIP model. **\(c\)** Besides, CLIPPING is only designed for a specific student model (MobileViT-v2), while our method can be widely used for student models with different architectures (refer to Tab. 3 of our paper).

- **(a)** VLKD concentrates on aligning the features of the language model (BART) to the CLIP model and subsequently integrating them to be a multi-modal generator. In contrast, our method is more than feature alignment. Since the tuned model in our framework is well-aligned with the CLIP model at the beginning of fine-tuning, our goal is to maintain its generalizability while learning video-specific knowledge. **(b)** The key technical contribution of VLKD is proposing to align the features of BART to CLIP using feature distance minimization and contrastive loss. On the contrary, we propose to balance feature learning in two different objectives (generalizability and video-specific learning) by a residual feature distillation module.

To demonstrate the superiority of our method, we evaluate the effectiveness of the VLKD method for knowledge distillation in our task. After carefully reading the CLIPPING paper, we found that its design choice is highly optimized for the specific MobileViT-V2 model in the conventional knowledge distillation setting, making it non-trivial to be modified to handle the open-vocabulary action recognition task. In contrast, our method can be easily integrated with different methods to improve their performance on open-vocabulary action recognition (please refer to Tab. 3 in the main paper).

As shown in Tab. 1 below, our method achieves superior performance than the VLKD method.

**Table 1: Performance (top1-acc (\%)) of different distillation methods under the cross-dataset evaluation setting.**
| Distillation Method | UCF101 | HMDB51 | K-600 |
| :------: | :------: | :------: | :------: |
| VLKD | 82.6 | 52.3 | 71.8 $\pm$ 1.2 |
| **Ours** | **85.0** | **54.5** | **74.8** $\pm$ 0.9 |

We have included the above analysis in the section of related work of our revision.

Besides, we agree that knowledge distillation has been a widely studied topic in the field, which is part of the motivation of our method to study open-vocabulary action recognition from a knowledge distillation perspective. However, we would like to highlight that conventional knowledge distillation methods aim to distill the same knowledge from one model to another, while differently, in our case, we have two objectives, one is maintaining the generalization capability of a pre-trained CLIP and the other is effectively adapting from image to video task. Simply applying the conventional distillation methods is not sufficient for our tasks (refer to Tab. 4 of our paper), while our method achieves superior results.

Reviewer comment (Experiments):
The authors didn't employ compact student model as in normal knowledge distillation. I am wondering if the performance is better if we use ViT-B/32 as teacher and ViT-B/16 as student.

Author response:
**R:** Thanks for the suggestion. We suppose the reviewer meant using ViT-B/16 as the teacher and ViT-B/32 as the student, because '/32' and '/16' indicate the patch size for patch embedding, making ViT-B/32 a more compact model than ViT-B/16. We conduct the comparison in Tab. 1 below. We observe that using a compact student model can also consistently improve performance, which further demonstrates the effectiveness of our proposed method.

**Table 1: Performance (top1-acc (\%)) of using different models for distillation under the cross-dataset evaluation setting. --- indicates only conducting model finetuning but not involving knowledge distillation.**

| Student | Teacher | UCF101 | HMDB51 | K-600 |
| :------: | :------: | :------: | :------: | :------: |
| ViT-B/32 | --- | 78.2 | 46.8 | 62.3 $\pm$ 0.9 |
| ViT-B/32 | ViT-B/32 | 80.0 ($\uparrow$ **1.8**) | 51.6 ($\uparrow$ **4.8**) | 68.3 $\pm$ 0.5 ($\uparrow$ **6.0**) |
| ViT-B/32 | ViT-B/16 | 81.0 ($\uparrow$ **2.8**) | 51.2($\uparrow$ **4.4**) | 69.5 $\pm$ 0.8 ($\uparrow$ **7.2**) |

Reviewer comment (Evaluation):
In Fig. 1, why CLIP performs more stable on larger Kinetics dataset than HMDB51 or UCF101?

Author response:
**R:** We are not completely sure what the reviewer means by "stable" here. We suspect that it refers to the fact that the frozen CLIP achieves better performance than the fine-tuned methods on the Kinetics-600 dataset, instead of the HMDB51 and UCF101 datasets. We clarify it as follows.

The experiments in Fig. 1 of the paper were conducted under a cross-dataset evaluation setting, where the models were fine-tuned (except for the frozen CLIP) on the Kinetics-400 dataset and then evaluated on the UCF101, HMDB51, and Kinetics-600 datasets. Please note that in the Kinetics-600 dataset, we excluded the classes that are identical to those in the Kinetics-400 dataset. In this way, the average text-feature similarity (see Fig. 1 of our paper) of all categories between Kinetics-600 and Kinetics-400 (0.721) is smaller than that between UCF101 and Kinetics-400 (0.805) and that between HMDB51 and Kinetics-400 (0.771). Therefore, compared to the UCF101 and HMDB51 datasets, recognition performance on the Kinetics-600 dataset relies more on generalizability. Since the frozen CLIP has not been fine-tuned on video data, its pre-trained generalizability is better maintained than the fine-tuned methods. Hence, the frozen CLIP achieves better performance than fine-tuned methods on the Kinetics-600 dataset, instead of the HMDB51 and UCF101 datasets.

Reviewer comment (Experiments):
What is the contribution of text rephrasing in Section 4.2?

Author response:
**R:** In the last paragraph of Sec. 3.1.1 in the paper, we mention that the text rephrasing is introduced to rephrase the action names into descriptions for enriching the text diversity, which serves as text data augmentation during training, due to the limited template text provided.

The impact of text rephrasing is evaluated in the last two rows of Tab. 4 of the paper, which is shown below.

**Table 2: Impacts of text rephrasing under the cross-dataset evaluation setting.**
| Method | UCF101 | HMDB51 | K-600 |
| :------: | :------: | :------: | :------: |
| Ours w/o text rephrasing | 84.4 | 54.4 | 73.0 $\pm$ 0.7 |
| Ours | **85.0** | **54.5** | **74.8** $\pm$ 0.9 |

We have added the above description in 3.1.1 in the paper to further clarify.