Paper: EZ-CLIP: EFFICIENT ZERO-SHOT VIDEO ACTION RECOGNITION (ICLR-2024)

Reviewer comment (Experiments):
The used baseline models are two adapters AIM and LoRA-FA (the last paragraph of Section 3.2). The paper's baseline model utilizes the spatial adapter only, which makes it less comparable to methods like AIM that incorporate both spatial and temporal adapters. Including both spatial and temporal adapters in the baseline would provide a more accurate basis for comparing the proposed temporal visual prompting.

Author response:
**W3.2: Spatial and Temporal Adapter.**

**Answer:**  
Thank you for your thoughtful comments and suggestions. Your observation regarding the comparison of the baseline models is valid, and we would like to provide further insights into our choices.

**1)** AIM as the reviewer rightly pointed out, is a fully supervised approach that excels in supervised classification. In our preliminary experiments, we conducted experiments including both spatial and temporal adapters in the baseline model. However, we observed that this approach tends to overfit to the source dataset, resulting in poor performance on the target dataset, particularly in zero-shot scenarios.



|               Method                              |HMDB|UCF|K-600|
|-----------------------------------------------|---------|-------|--------|
| CLIP(temporal adapters)                |46.0    |70.5 | 61.0  |
| EZ-CLIP(Temporal visual prompt)  |52.9    |79.1 | 70.1  |


**2)** Apart from this performance issue, temporal adapters also suffer from scaling challenges. For a more detailed understanding, we have provided the breakdown of trainable parameters for different configurations, highlighting the scalability challenges associated with temporal adapters, especially with larger models like ViT-14.
| Model                 | Component                    | Trainable Parameters (M) |
| ----------------------| -------------------------------- | ---------------------------------- |
|  CLIP ViT-16      | Temporal Adapters         |                   3.5                   |
|  CLIP ViT-14      | Temporal Adapters         |                  12.6                  |


| Model                 | Component                        | Trainable Parameters (M) |
| ----------------------|------------------------------------ | ---------------------------------- |
|    CLIP ViT-16    | Temporal Visual Prompts    |                  0.07                 |
|   CLIP ViT-14     | Temporal Visual Prompts    |                 0.19                  |



Additionally, we explored the impact of incorporating temporal visual prompts alongside spatial adapters, and the results indicated challenges in scalability, especially with larger models.

We hope this clarifies our rationale and the considerations we took into account during the experimentation. If you have further questions or suggestions, please feel free to let us know. Your feedback is invaluable in enhancing the quality and clarity of our work.

Reviewer comment (Evaluation):
Table 5 demonstrates the substantial positive impact of the LLM-generated action class descriptions on action recognition. It suggests that the improved performance of EZ-CLIP may primarily stem from the use of these descriptions rather than the introduction of temporal visual prompting. For example, EZ-CLIP wo LLM-description only achieves 77.3% HM result on UCF-101, while ViFi CLIP which does not use LLM (I checked their paper, if I did not miss something) obtains the higher result of 78.3%. As such, the better performance of EZ-CLIP may mainly come from the utilization of LLM.

Author response:
**W3.3 Impact of the LLM-generated action class descriptions.**

**Answer:**

---

Thank you for your insightful observation. We appreciate your careful examination of our results. The role of LLM-generated action class descriptions is indeed crucial in our approach, and we acknowledge their positive impact on action recognition, as demonstrated in Table 5.

While the use of LLM-generated descriptions contributes to the improved performance of EZ-CLIP, it is not the sole factor. Although the use of LLM-generated descriptions in this context is a novel idea, we would like to emphasize that our approach involves a joint training strategy that incorporates both temporal visual prompting and motion loss. The ablation studies in our paper demonstrate that the introduction of temporal visual prompts and motion loss leads to significant performance gains across various datasets. The use of LLM-generated descriptions without temporal visual prompting and motion loss is not as strong as it is with these components.

*Zero-shot Ablation on LLM-generated Action Class Descriptions:*

                             
| TVP  |Motion loss | LLM-description   | HMDB-51 | UCF-101 | K-600  |
|--------|----------------|-------------------------|---------------|--------------|-----------|
|    ❌ |       ❌        |     ✔️                     |   49.2        |   75.5     |     63.1  |
|     ✔️|           ✔️    |   ❌                       |  52.0         |   77.8      |    67.2  |
|    ✔️ |       ✔️        | ✔️                        |    52.9        |   79.1      |   70.1   |


                                                       |  HMDB-51                      |  UCF-101                      |    SSv2                            |
| TVP|Motion loss|LLM-description| K=2 | K=4 | K=8 | K=16 | K=2 | K=4 | K=8 | K=16 | K=2 | K=4 | K=8 | K=16 |
|-------|---------------|---------------------|-------|-------|--------|--------|-------|--------|-------|--------|-------|--------|-------|---------|
|  ❌  | ❌            |           ✔️          | 53.7 |57.9 |61.7  | 63.6  |81.2  |85.5  |86.6  |89.8  |5.7    |6.7    |7.9   |  10.8  |
| ✔️   |✔️             |             ❌        | 57.0 | 59.1| 65.1 | 65.2  |81.8  |86.6  |88.8  |90.2  |6.1    |8.0    |9.3   | 11.9    |
| ✔️   |✔️             | ✔️                    |57.3  |61.1 |65.4  |67.7   |84.7  |88.3  |91.3  |92.8  |6.8    |8.7    |9.6   |13.2     |



For instance, when training the model jointly with both temporal visual prompts and motion loss, we observed substantial improvements, particularly in the case of the SSv2 dataset. This indicates that our proposed techniques play a complementary role in enhancing temporal modeling in videos.

Regarding the comparison with ViFi CLIP, we would like to clarify that our goal is to improve efficiency with a focus on zero-shot action recognition. The use of LLM is indeed a novel idea, and we acknowledge its application in other contexts, as seen in ["VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS"](https://arxiv.org/pdf/2210.07183.pdf). However, even without LLM, our approach achieves comparable performance to existing works while utilizing very few parameters.

We also appreciate your attention to the breakdown of how motion loss and temporal prompts contribute to our results. The attention visualizations in Figure 4 illustrate the shift in focus from appearance to motion in EZ-CLIP, particularly evident in datasets like UCF-101 and SSv2. This shift aligns with our goal of capturing complex relative motion features, especially beneficial in scenarios where motion cues are critical, as observed in the SSv2 dataset.

We hope this clarifies the contributions of our proposed approach, and we are open to further discussions or clarifications.

Thank you for your valuable feedback.

Reviewer comment (Experiments):
The proposed EZ-CLIP employs 8 frames for efficiency, while other methods use 32 frames (a 4x difference). As such, the comparison in Table 6 may not be appropriate.

Author response:
**W.3.4: Frame selection strategy.**

**Answer:**
Thank you for your observation regarding the frame count difference between EZ-CLIP and other methods. We appreciate your attention to detail.

In Table 6, throughput is calculated based on the number of frames processed per second. Therefore, we think that the efficiency comparison in Table 6 is appropriate, irrespective of whether the model employs 8 frames per video or 32 frames per video. Throughput provides a standardized measure that allows for a fair comparison of efficiency across different methods, accounting for the processing speed regardless of the specific frame count used by each method.

If you have further questions or if there are additional aspects you would like us to address, please feel free to let us know. We value your input and are committed to ensuring clarity in our methodology and results.

Reviewer comment (Reproducibility):
The proposed overall framework seems to be heavily relied on additional training modules and tricks like LLM based prompt ensembling and spatial-language adaptors. However, there is very little detail given on these modules. For example, what kind of adaptors are used, at which place they are incorporated in the model, how many prompts are being used from LLM, illustrations of LLM prompts? It will be better to provide a high level figure diagram which also shows the usage of these additional components.

Author response:
We thank the reviewer for their valuable comments and detailed questions. Please refer to our responses below:

**W/Q 2.1: Detail about spatial and text adapters?**

**Answer:**
We appreciate your inquiry regarding the details of additional training modules and components in our proposed framework. We understand the importance of providing a comprehensive understanding of these elements.

To address your concerns:

1. **Text and Spatial Adapters:**
   - The details on the placement of text and spatial adapters can be found in Figure 6, specifically in the Lth block BL, as highlighted in Section A.3 of the revised paper. This illustrates the incorporation of these adapters into our model.

2. **LLM-Based Prompt Ensembling:**
   - The usage of prompts from LLM is an integral part of our approach. In Table 7, available in Section A.2, we provide examples of action class descriptions generated by LLM using a template of "describe [category] as an action performed by humans." This demonstrates the effectiveness of LLM-based prompt ensembling in generating diverse and informative descriptions.

We recognize the importance of clarity in visual representation. In response to your suggestion, we will consider incorporating a high-level figure diagram in the revised version of the paper to provide a more intuitive overview of the overall framework, including the usage of these additional components.

Thank you for your understanding, and we welcome any further questions or suggestions.

Reviewer comment (Experiments):
It will be good to see effectiveness of the proposed approach on (i) larger CLIP models like ViT Large or Huge (ii) and on any other CLIP variant (e.g EVA-CLIP) which would confirm the generalization of the proposed approach towards other model scales and recent VL models other than CLIP.

Author response:
**W/Q 2.2: Experiments on other CLIP models**

**Answer:**
Thank you for your suggestion. We have indeed conducted experiments on two additional CLIP architectures: CLIP ViT-32 and CLIP ViT-14. These experiments cover three scenarios: (a) Zero Shot, (b) Base to Novel, and (c) Few Shot. Here are the details:

**CLIP ViT-32:**
- Number of Tunable Parameters:
  - TVP: 0.07M
  - Visual Adapters: 3.5M
  - Text Adapters: 1.5M

**CLIP ViT-14:**
- Number of Tunable Parameters:
  - TVP: 0.19M
  - Visual Adapters: 12.6M
  - Text Adapters: 3.5M

Zero-Shot experiment

|               Method  | Input size     | HMDB-51  | UCF-101 | K-600 |
|------------------------|-------------------|---------------|--------------|---------|
| EZ-CLIP(ViT-32) | 8x224x224   |     50.0       |     77.5     | 67.0  |
|------------------------|-------------------|---------------|--------------|---------|
| EZ-CLIP(ViT-16) |  8x224x224  |     52.9       |      79.1    | 70.1  |
|------------------------|-------------------|---------------|--------------|---------|
| EZ-CLIP(ViT-16) | 16x224x224 |     53.3       |      80.0    | 71.1  |
|------------------------|-------------------|---------------|--------------|---------|
| EZ-CLIP(ViT-14) |  8x224x224  |     55.2       |      82.6    |  72.1 |

Base to Novel experiment
                              | Kinetics-400            |         HMDB-51       |     UCF-101            |        SSv2               |
|------------------------|---------|----------|-------|---------|----------|------|---------|----------|------|---------|----------|------|
|        Method         | Base | Novel  | HM  | Base  | Novel | HM | Base | Novel  | HM | Base | Novel  | HM |
|------------------------|---------|----------|-------|---------|----------|------|---------|----------|------|---------|----------|------|
| EZ-CLIP(ViT-32) | 80.3  | 58.16   | 67.4 | 77.0  | 55.3    |64.3 | 95.4  | 70.0    |80.7| 53.1   | 19.1    |28.0|
|------------------------|---------|----------|-------|---------|----------|------|---------|----------|------|---------|----------|------|
| EZ-CLIP(ViT-16) | 83.7   | 62.5    | 71.5 | 79.8  | 62.2    |69.9| 95.9   | 76.5    |85.1| 54.0   | 20.6   |29.8 |
|------------------------|---------|----------|-------|---------|----------|------|---------|----------|------|---------|----------|------|
| EZ-CLIP(ViT-14) | 85.6   | 67.2    |75.2 | 81.0   | 64.5    |71.8| 96.4   | 79.5    |87.1| 60.5   | 23.0   | 33.3|

Few shot experiment

                             |HMDB-51                       |         UCF-101               |        SSv2                |
|-----------------------|---------------------------------|--------------------------------|----------------------------|
| Method                |K=2   | K=4 | K=8|K=16  | K=2 | K=4  | K=8 |K=16| K=2|K=4|K=8  |K=16|
|------------------------|--------|-------|-------|--------|-------|--------|--------|-------|------|------|------|-------|
| EZ-CLIP(ViT-32) | 52.4 | 56.9 | 60.4 | 63.7 | 79.7 | 83.7 | 87.5 | 89.8 | 5.8 | 6.7 | 8.0 | 12.1 |
|------------------------|--------|-------|-------|--------|-------|--------|--------|-------|------|------|------|-------|
| EZ-CLIP(ViT-16) | 57.3 | 61.1 | 65.4 | 67.7 | 84.7 | 88.3 | 91.3 | 92.8 | 6.8 | 8.7 | 9.6 | 13.2 |
|------------------------|--------|-------|-------|--------|-------|--------|--------|-------|------|------|------|-------|
| EZ-CLIP(ViT-14) | 62.4 | 65.3 | 68.4 | 70.7 | 85.2| 89.7  | 92.4 | 93.3 | 8.9 |10.1| 11.1|15.2 |

These experiments provide valuable insights into the effectiveness of our proposed approach across various CLIP model scales. If you have any further inquiries or suggestions, please feel free to let us know.

Reviewer comment (Experiments):
I am not completely sure but I think there might be something wrong in the ssv2 results for base-to-novel generalization setting. EZ-CLIP is achieving around 54% and 20.6% accuracy for 16 shots samples. But the same EZ-CLIP is performing relatively poor in few-shot setting where it also uses 16 shot samples. Can the authors revisit the experiments and confirm that they are using the correct split for base-to-novel setting for ssv2? I am afraid it can be the case that complete ssv2 training data is used instead of 16 shots. Otherwise the few-shot results should have similar scale results for the proposed approach.

Author response:
**W2.3: Few-shot vs base-to-novel comparison on SSv2.**

**Answer:**
Thank you for your careful examination of our results. To clarify, we employed the same protocol as in existing works for the base-to-novel setting. Specifically, during base-to-novel training, we use all samples from the base classes (not just 16), creating a split between base and novel categories. In the few-shot learning setting, a fixed number of samples (e.g., 16-shot) is utilized.

The observed differences in accuracy between the base-to-novel and few-shot settings, even with the same number of samples (16 shots), are expected due to the distinct training strategies and the resulting differences in the number of training classes between these two setups. We appreciate your attention to detail and hope this explanation clarifies the discrepancy you noticed.

If you have any further questions or suggestions, please feel free to let us know.