Do Video Language Models really understand the video contexts? (2025)

Abstract: This paper examines how well visual language models (VLMs) understand video question answering (VideoQA) tasks and generate responses accordingly. Recently, VLMs based on Large Language Models (LLMs) have shown remarkable performance, but the processes of understanding and reasoning in VLMs remain under-explored. To tackle this challenge, we propose Video Understanding and Response Consistency Assessment, VURCA, a framework that incorporates a fine-grained question generation and answering process to measure how well the responses generated by VLMs align with what the model understands. In addition, we introduce an extended benchmark dataset, FgNExT-QA, which builds upon NExT-QA by incorporating more fine-grained VideoQA tasks. FgNExT-QA is designed to evaluate fine-grained understanding in video question answering. Through experiments, we found that despite the strong overall QA performance of VLMs, their understanding of both the video content and the question remains limited. In particular, they exhibit poor video comprehension in fine-grained VideoQA tasks.

## introduction video question answering (videoqa) @xcite serves as a critical benchmark for evaluating the capabilities of foundational visual language models (vlms) @xcite , particularly those trained on large-scale multi-modal datasets @xcite . despite recent advancements in videoqa performance, several fundamental concerns remain underexplored. a key question is whether these models accurately comprehend video and question to enable robust multi-modal reasoning, or if they merely mimic learned patterns from the training dataset @xcite . responses based on * equal contribution. ## related work in videoqa tasks, a primary objective is to ensure that the model accurately comprehends video data and generates appropriate responses. previous research has focused on building models for video action and dynamics recognition @xcite . however, most of these efforts fall under the category of simple perceptuallevel understanding, such as handling straightforward video @xcite . recent advancements in transformer-based language models @xcite have been accompanied by substantial progress in visual-language models (vlms), leading to significant improvements in video question answering performance. @xcite integrated visual encoders and llama-adapter @xcite into llms to enable video understanding, training the model to process both textual and visual inputs effectively. @xcite and @xcite demonstrated remarkable performance improvements by first generating image captions using a vlm, selecting frames directly relevant to the question from the video, and then integrating these captions with the reasoning process of an llm, such as chatgpt @xcite . @xcite extended this approach by applying chain-of-thought (cot) @xcite reasoning capabilities from llms to vlms. recently, @xcite critically questioned the degree to which the answers generated by such techniques are truly grounded in the relevant visual content. however, research verifying the alignment between understanding and response in vlms has yet to be extensively explored. ## video understanding and response consistency assessment to investigate the understanding of vlms, we present vurca, a framework designed to quantify the consistency between video understanding and responses by integrating videoqa with finegranined questions generation. as illustrated in figure 2 , the process of the proposed framework consists of four main steps: video question answering, generation of verification question, verification question answering and evaluating consistency ## conclusion this paper explores how visual language models understand videoqa tasks and generate appropriate responses. however, evaluating whether these models truly understand both video and language inputs remains a challenging task. to address the challenge of evaluating vlms comprehension, we propose vurca, a framework to assess the alignment between the initial responses and vlms understanding. vurca achieves this by generating verification questions and comparing the subsequent responses with its initial answers. additionally, we introduce fgnext-qa, a benchmark dataset designed for fine-grained videoqa tasks, which offers more fine-grained assessment scenarios. our experimental results indicate that despite their impressive performance in qa tasks, vlms often fail to adequately understand video content and the corresponding questions. these results provide valuable insights for the development of advanced evaluation frameworks, the design of more robust model architectures, and the refinement of training methodologies. future research should aim to enhance the reasoning capabilities of vlms through improved pre-training strategies that inte-grate a more comprehensive understanding of video content and question semantics.