Paper: VIDEOPROMPTER: AN ENSEMBLE OF FOUNDATIONAL MODELS FOR ZERO-SHOT VIDEO UNDERSTANDING (ICLR-2024)

Reviewer comment (Experiments):
The efficiency of VideoPrompter hasn't been thoroughly examined. Given that VideoPrompter appears to require generating 10 times the number of samples and the use of an additional text-to-video model, it could substantially raise the inference costs, both for evaluating existing VLMs and in practical applications.

Author response:
Kindly note that the VideoPrompter offers a flexible framework, allowing
for the substitution of various modules, including the video-2-text
model (VGPT), text-2-text model (GPT-3.5), and vision-language model
(CLIP). During our initial submission, Video-ChatGPT (VGPT) \[1\] was
chosen as the video-2-text model due to its competitive performance when
compared to Video-Chat and VideoLLaMA. However, the recent introduction
of Video-LLaVA \[2\], *a model released this week*, has set a new
state-of-the-art (SOTA) in zero-shot question-answer and video-based
generative benchmarks. We performed an experiment where we replaced VGPT
with Video-LLaVA and generated video-textual descriptions for HMDB-51
and UCF-101 benchmarks. Conducting a single inference (as opposed to the
10 inferences with VGPT) without any filtering applied (as now we have
only one description per video), we observed a 0.21% and 0.75%
improvement on previous scores for HMDB-51 and UCF-101 benchmarks,
respectively, as shown in the table below. The adoption of Video-LLaVA
over VGPT reduced the number of required inferences from 10 to 1,
leading to a significant decrease in computational cost. Additionally,
the per-video inference time for Video-LLaVA is approximately 6 times
less than that of VGPT, with 3.79s per video, while, for reference, the
inference time of the CLIP vision encoder is 0.74s per video. We
anticipate that ongoing advancements in multi-model methods will further
reduce the inference cost of VideoPrompter. We will include the updated
results with Video-LLaVA on all benchmarks and models in our final
manuscript.

|         **Dataset**          | **Baseline (CLIP)** | **Video-Prompter (VGPT)** | **Video-Prompter (LLaVA)** |
|:---------------------------:|:-------------------:|:--------------------------:|:---------------------------:|
|           HMDB-51            |         37.5        |           52.51            |          **52.72**          |
|           UCF-101            |         61.72       |           73.88            |          **74.63**          |

Reviewer comment (Experiments):
The selection of Video-ChatGPT as the video-to-text model seems arbitrary. Alternative models, such as Video-LLaMA [A], should be considered and discussed.

Author response:
Please note that our choice of video-to-text generative model is defined
by the competitive performance \[1\] *(at the time of submission)* of
Video-ChatGPT compared to Video-Chat and VideoLLaMA. However, the recent
introduction of Video-LLaVA \[2\], *a model released this week*, has set
a new state-of-the-art (SOTA) in zero-shot question-answer and
video-based generative benchmarks. We used it as an alternative model
and show the results below for HMDB-51 and UCF-101 benchmarks. Note
that, for Video-LLaVA single inference is performed (as opposed to the
10 inferences with VGPT) and no filtration is applied (as now we have
only one description per video). Thank you for suggesting this analysis.
We will include this ablation in our final manuscript.

|         **Dataset**          | **Baseline (CLIP)** | **Video-Prompter (VGPT)** | **Video-Prompter (LLaVA)** |
|:---------------------------:|:-------------------:|:--------------------------:|:---------------------------:|
|           HMDB-51            |         37.5        |           52.51            |          **52.72**          |
|           UCF-101            |         61.72       |           73.88            |          **74.63**          |

Reviewer comment (Novelty):
The idea of generating more descriptions for class names and using high-level context is not new in prompting large foundation models (e.g. the prior works cited in this paper). This is model ensembling for enhancing zero-shot performance. Can the authors justify the main novelty of this paper?

Author response:
**Q1: *Can the authors justify the main novelty of this paper?***

Kindly note that our work is the first one to study the role of
video-to-text models for video understanding and highlights how video descriptions can
further enhance visual representations. So far, the existing methods
\[1,2\] are only limited to text-to-text generative models for enhancing
the text classifier representations only. We summarize our contributions
as follows:

1.  So far, only text-to-text generative models (e.g. GPT) are used to
    enhance the representations of the text classifier. Our work
    discusses the impact of having video-to-text generative models (e.g.
    VGPT) to enrich the visual representations as well, in addition to
    only text-based models (Section 2.2).

2.  We propose a novel way of querying LLMs called action context
    (Section 2.3.2). Action context can classify semantically close
    videos under a single high-level context (category). In the related
    work, we discuss the limitations of the existing methods \[3,4\] and
    discuss how these methods are only applicable to image benchmarks
    and are unsuited for video benchmarks where the classes can be
    fine-grained as well can have a diverse range.

3.  We test our framework in three different video settings namely:
    action recognition, video-to-text, and text-to-video retrieval, and
    time-aware video tasks. We show results with 4 different models on 7
    benchmarks.

Reviewer comment (Experiments):
VGPT is used to generate the text description of the query video, and which is then converted to an image-like text embedding. Why not just prompting VGPT for the downstream applications (e.g. action classification)? Comparison to this baseline is an important justification to the proposed method.

Author response:
**Q2: *Why not just prompting VGPT for the downstream applications?***

As recommended, we designed an experiment where only VGPT is directly
used to classify the input video. We found this setting to degrade the
results by a large margin. We relate this to the training of VGPT as it
is trained on generating long contexts about the input, rather than
specific categories \[5\]. We also present a more extended setting,
where the descriptions from VGPT are given to GPT-3.5 to predict the
class name. Here, the GPT-3.5 is also provided by the list of class
names. It can be seen that VideoPrompter outperforms both of
the aforementioned settings which endorses its design choice. Thank you
for suggesting this analysis. We will include this ablation in our final
manuscript.
|     **Datasets**     | **Vanilla-CLIP** | **VGPT only** | **VGPT+GPT** | **VideoPrompter** |
|:--------------------:|:----------------:|:-------------:|:------------:|:------------------:|
|       HMDB-51        |       37.5       |      16.43    |      46.0      |      **52.51**     |
|       UCF-101        |       61.5       |      39.21    |      68.0      |      **73.88**     |

Reviewer comment (Experiments):
The claim for the comparison to CUPL (Pratt et al., 2022) is not very clear (section 3.1.4). The authors claim that VideoPrompter only requires 3 text descriptions instead of 50 descriptions adopted in CUPL. However, VideoPrompter adopts a VGPT model while CUPL does not. Is using VGPT a better choice in terms of the cost?

Author response:
**Q3: *The comparison with CUPL.***

We regret any confusion, it's important to note that a direct comparison
with CUPL may not be straightforward, as it can be viewed as a
complementary approach to our method. As discussed in \[3\], the
performance of such methods benefits from averaging over a large number
of descriptors. When we combined CUPL with
VideoPrompter i.e. weights of language descriptors are averaged,  performance
increased by  0.12% for UCF-101. We will further update
this discussion in our final manuscript.

Reviewer comment (Evaluation):
Since at least 3 foundation models have been used (CLIP, GPT, VGPT), how do we know if those models are trained with examples overlapped with the downstream datasets (e.g., HMDB-51, UCF101, SSv2, K400, MSR-VTT, Charades).

Author response:
**Q1: *Leaking examples from downstream datasets to foundation
models*.**

Kindly note that the CLIP is trained on 400M image-text pairs and the
VGPT employs video-text pairs and is trained on a subset of
ActivityNet-200. While the GPT is only trained on the text data. To
investigate any overlapping with the downstream datasets we design the
following experiment and present results on HMDB-51 and UCF-101
benchmarks:

1.  We show zero-shot results of vanilla-CLIP on HMDB-51 and UCF-101
    benchmarks to understand the extent of knowledge the CLIP model
    contains about these datasets.

2.  Second, to understand any data overlap of VGPT, we only employ VGPT
    directly to classify the input video. We found this setting to
    degrade the results by a large margin. We relate this to the
    training of VGPT as it is trained on generating long contexts about
    the input, rather than specific categories \[1\].

Kindly, from the table below, it can be seen that the individual
performance of these modules (vanilla-CLIP and VGPT) is substantially
low, which shows minimum data overlap. Thank you for suggesting this
analysis. We will include this ablation in our final manuscript.

|   **Datasets**   | **CLIP** | **VGPT only** | **VideoPrompter** |
|:----------------:|:--------:|:-------------:|:------------------:|
|     HMDB-51      |   37.5   |     16.43     |      **52.51**     |
|     UCF-101      |   61.5   |     39.21     |      **73.88**     |

Reviewer comment (Novelty):
The novelty seems moderate if not low. As the paper mentions the main contributions are 1) introducing video-to-text to enhance visual embeddings and 2) applications to videos.

Author response:
**Q2: *The novelty seems moderate if not low.***

Kindly note that our work is the first one to study the role of
video-to-text models for video understanding. Our work highlights how
the recent advances in the video-to-text models can be used in the
general video-understanding pipeline, and how video descriptions can
further enhance the visual representations. So far, the existing methods
\[2,3\] are only limited to text-to-text generative models for enhancing
the text classifier representations only. We summarize our contributions
as follows:

1.  So far, only text-to-text generative models (e.g. GPT) are used to
    enhance the representations of the text-classifier. Our work
    discusses the impact of having video-to-text generative models (e.g.
    VGPT) to enrich the visual representations as well, in addition to
    only text-based models (Section 2.2).

2.  We propose a novel way of querying LLMs called action context
    (Section 2.3.2). Action context can classify semantically close
    videos under a single high-level context (category). In the related
    work, we discuss the limitations of the existing methods \[4,5\] and
    discuss how these methods are only applicable to image benchmarks
    and are unsuited for video benchmarks where the classes are
    fine-grained as well as have diverse ranges.

3.  We test our framework in three different video settings namely:
    action recognition, video-to-text, and text-to-video retrieval, and
    time-aware video tasks. We show results on 7 benchmarks including
    HMDB-51, UCF-101, SSv2, K400, MSRVTT, Charades, and the time-aware
    synthetic dataset. Moreover, as our framework offers plug and play
    module, we show results with various models such as CLIP, ViFi-CLIP,
    Action-CLIP, and AIM.

Reviewer comment (Presentation):
The written presentation could be further improved:  
1) section 2.1 could be renamed to "Overview" and try to capture the big picture of the framework. The author(s) can refer back to Fig. 1 for the big picture (in the current flow of presentation, there is no big picture and it flows in with overwhelming many details and notations). Then sections 2.2 and 2.3 can be further followed up from 2.1 to provide detailed of components.

Author response:
**Q3 (a): *The written presentation could be further improved.***

Please note that, sections 2.1, 2.2, and 2.3 have been updated. Now, for
more clarity we have re-named section 2.2 from video-specific language
descriptors to class-specific language descriptors. The revised text is
highlighted in red.

Reviewer comment (Presentation):
The written presentation issue also includes that table 6 is presented in page 7, yet never been referred from the text.

Author response:
**Q3(b): "*table 6 is presented in page 7, yet never been referred from
the text?***

Kindly note that table 6 is referred in video action recognition section
on page 6.

Reviewer comment (Experiments):
My main concerns are the leaking examples from downstream datasets to foundation models.

Author response:
**Q1: *Leaking examples from downstream datasets to foundation
models*.**

Kindly note that the CLIP is trained on 400M image-text pairs and the
VGPT employs video-text pairs and is trained on a subset of
ActivityNet-200. While the GPT is only trained on the text data. To
investigate any overlapping with the downstream datasets we design the
following experiment and present results on HMDB-51 and UCF-101
benchmarks:

1.  We show zero-shot results of vanilla-CLIP on HMDB-51 and UCF-101
    benchmarks to understand the extent of knowledge the CLIP model
    contains about these datasets.

2.  Second, to understand any data overlap of VGPT, we only employ VGPT
    directly to classify the input video. We found this setting to
    degrade the results by a large margin. We relate this to the
    training of VGPT as it is trained on generating long contexts about
    the input, rather than specific categories \[1\].

Kindly, from the table below, it can be seen that the individual
performance of these modules (vanilla-CLIP and VGPT) is substantially
low, which shows minimum data overlap. Thank you for suggesting this
analysis. We will include this ablation in our final manuscript.

|   **Datasets**   | **CLIP** | **VGPT only** | **VideoPrompter** |
|:----------------:|:--------:|:-------------:|:------------------:|
|     HMDB-51      |   37.5   |     16.43     |      **52.51**     |
|     UCF-101      |   61.5   |     39.21     |      **73.88**     |