Paper: ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The form of the benchmark is still relatively simple, just let models choose the correct answer from two candidate sentences. However, simple two-choice questions are not enough to fully measure the ability of the model.

Author response:
We understand your concern regarding the simplicity of our benchmark. However, we argue that this simplicity is intentional and beneficial. The two-choice question format was chosen to precisely evaluate specific capabilities of video-language models (VidLMs) without the confounding effects of more complex task structures. This straightforward approach allows us to directly assess whether a model possesses a particular capability, avoiding confusion that could arise from more intricate tasks. Moreover, our foiling setup offers several advantages: it is task-agnostic, requires no additional fine-tuning or specific heads for evaluation tasks, and allows direct assessment of multimodal pre-training benefits without the risk of shortcut learning. This simplicity also facilitates faster and more straightforward testing, making our benchmark more accessible and easily adoptable by the research community.

Reviewer comment (Evaluation):
With the success of large language models, the latest video-language models, e.g., BLIP2 can output text with variable length and free content. The community may be more concerned about how to properly evaluate these open outputs.

Author response:
We acknowledge the growing interest in multimodal large language models and think evaluating them in a generative setting is interesting. That said, our primary aim is to evaluate specific capabilities of VidLMs in a clear and straightforward manner, rather than tackling the broader, unsolved problem of natural language generation evaluation.

Reviewer comment (Experiments):
Do you consider doing more analysis on the video itself, such as exploring the sensitivity of the model's temporal understanding ability to the length of the video? What happens to the model if you insert noise frames into the video?

Author response:
Your query about the model's sensitivity to video length and the impact of noise frames is intriguing. We are currently working on a length-based performance analysis, as preliminary observations suggest that performance deteriorates with longer videos. This is likely due to the sampling methods used by some VidLMs and the increased complexity of modeling longer temporal dependencies. Please, check the appendix of the updated manuscript to see our analysis (D. Further Analysis).

Reviewer comment (Experiments):
There are some of the most recent VidLMs with good performances missing in the evaluation. E.g. InternVideo [1], mPLUG2 [2], Uniformer v2 [3], etc. The necessity of the proposed dataset needs more evaluation to validate.

Author response:
We acknowledge the importance of including recent video-language models (VidLMs) in our evaluation. To this end, we have adapted the InternVideo and mPLUG2 models. These models will be featured in the final version of our paper. Regarding UniFormer-v2, we excluded it due to its inherent design as a vision-only model, lacking the necessary capabilities for language processing, which is crucial for our benchmark's objectives. In the following, we present the results of InternVideo and mPLUG2 models (P+T). We will update our manuscript accordingly.

|Method|Action Counting|Situ. Awareness|Change of State|Rare Actions|Spatial Relations|
|-|-|-|-|-|-|
|InternVideo|48.7|29.5|55.1|92.7|45.3|
|mPLUG-2|27.7|21.5|20.8|24.0|26.5|

Reviewer comment (Novelty):
Next-QA [1] also requires the model to perform temporal understanding and reasoning. It seems the main difference between the proposed dataset and Next-QA is the format. Can the author explain the core challenges posed by the proposed dataset?

Author response:
Our benchmark differs from NExT-QA in several fundamental aspects. 
1. Unlike NExT-QA, which primarily focuses on event recognition and localization, our benchmark is designed to challenge pretrained VidLMs with minimally edited counterfactual captions. This novel approach assesses various linguistic and spatio-temporal capabilities, such as understanding relational dynamics, counting repetitive actions, and observing state changes. 
2. Our benchmark employs a counterfactual caption generation technique that provides a more nuanced and challenging test of model capabilities. This stands in contrast to NExT-QA’s multiple-choice format, where incorrect choices are often drawn from unrelated questions, leading to irrelevant options. For instance, consider this example from the NextQA paper:
“Q: Why did the girl in blue stop and turn around at the start? Options: 0. Pour ingredient in, 1. Scared, 2. Waiting for the lady, 3. Take out vegetable, 4. Dance” (Correct Answer: 2). 
This format can unintentionally simplify the task for models by including distinctly irrelevant options, thus diminishing the quality of the assessment.
3. Our benchmark is specifically tailored as a zero-shot, test-split-only foiling benchmark, eliminating the need for additional finetuning or task-specific prediction heads. This design allows for a direct assessment of the visio-linguistic competencies acquired by models during their pre-training phase, thus providing a novel and robust measure of temporal understanding and reasoning in video-language models.

Reviewer comment (Evaluation):
The gap between P+T suggests that there exists inherent dataset bias in VILMA, which can be exploited to achieve good performance. This phenomenon seems contradictory to the claim on page 24 that “the biases are not significantly present”.

Author response:
The performance gap between T and P+T is attributable to a plausibility bias, as also noted by Reviewer f3MM. Our methodology to mitigate this bias is outlined in our response to f3MM. In particular, we have taken careful measures to ensure that our human validation process does not reintroduce lexical distributional shifts between captions and foils. This is achieved by aligning the lexical distributions of captions and foils, as indicated in Appendix B.1, Bias Check (pages 22-24 of the revised version).

Reviewer comment (Experiments):
The evaluation could be more comprehensive by including recent Video Large Language Models (Video LLMs), e.g., VideoChat [1], Otter [2], Video-LLaMA [3].

Author response:
Acknowledging the importance of including recent video-language models (VidLMs) in our evaluation, we have integrated all the methods you mentioned: We have successfully concluded all experiments related to the Otter model and are currently awaiting the completion of experiments for the other models. Their results, which demonstrate the efficacy and relevance of our approach, will be included in the final version of our manuscript. This inclusion will not only enhance the comprehensiveness of our study but also it will ensure that our findings remain relevant and applicable to current advancements in the field.