T emp C ompass: Do Video LLM s Really Understand Videos? (2024)

Abstract: Recently, there is a surge in interest surrounding video large language models (Video LLMs). However, existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs. On the one hand, most of them are unable to distinguish between different temporal aspects (e.g., speed, direction) and thus cannot reflect the nuanced performance on these specific aspects. On the other hand, they are limited in the diversity of task formats (e.g., only multi-choice QA), which hinders the understanding of how temporal perception performance may vary across different types of tasks. Motivated by these two problems, we propose the TempCompass benchmark, which introduces a diversity of temporal aspects and task formats. To collect high-quality test data, we devise two novel strategies: (1) In video collection, we construct conflicting videos that share the same static content but differ in a specific temporal aspect, which prevents Video LLMs from leveraging single-frame bias or language priors. (2) To collect the task instructions, we propose a paradigm where humans first annotate meta-information for a video and then an LLM generates the instruction. We also design an LLM-based approach to automatically and accurately evaluate the responses from Video LLMs. Based on TempCompass, we comprehensively evaluate 9 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs, and reveal the discerning fact that these models exhibit notably poor temporal perception ability.

## introduction the development of video understanding systems has long been a popular topic in artificial intelligence research. inspired by the unprecedented progress of large language models (llms), a line * equal contribution. of initial efforts @xcite @xcite @xcite have been devoted to build llms with video understanding ability. these video llms can serve as versatile multi-modal solvers for video and language tasks, demonstrating strong potential across various real-world applications. with the rapid development of video llms, a compelling question arises: "do video llms really understand the temporal dynamics of videos" despite the importance of this question, current benchmarks fail to provide a satisfactory answer. firstly, a majority of them neglect differentiating between various temporal aspects (e.g., type of action, speed and direction), thereby failing to offer a comprehensive view to diagnose the temporal perception ability. secondly, while some video llm benchmarks @xcite have categorized various temporal aspects, they are restricted in task format variety (e.g., only multi-choice qa). consequently, they are not optimally suited for assessing video llms, which are expected to generalize across diverse tasks and instruction formats. in response to the above issues, this work proposes the tempcompass, a benchmark to comprehensively evaluate the temporal perception ability of video llms. tempcompass introduces five basic temporal aspects (action, speed, direction, attribute change and event order) and ten finegrained sub-aspects, as shown in figure 1 . additionally, tempcompass involves four distinct types of task formats (multi-choice qa, yes/no qa, caption matching and caption generation), as shown in figure 2 , which allows us to investigate how the temporal perception ability of video llms varies across different task formats. the videos in tempcompass are originated from the shutterstock 1 platform. these open-domain videos cover a diverse variety of contents, ranging from human activities to natural scenarios, among others. to prevent video llms from leveraging single-frame bias or language priors to complete the tasks, we construct conflicting video pairs/triplets, within which the videos share the same static content but differ from each other in a specific temporal aspect. given the collected videos, we derive 7,540 task instructions for the four types of tasks, using a collaboration of human annotated meta-information and llm generation. due to the diverse task formats in tempcompass and the free-form nature of video llm responses, it is non-trivial to automatically evaluate the performance of video llms. to address this challenge, we resort to the language understanding ability of llms for evaluation. for each type of task, we use tailored evaluation prompts for chatgpt (gpt3.5-turbo) to assess whether the video llm response is correct. to balance the cost and accuracy of evaluation, we also adopt some rule-based assessment methods, which are implemented prior to utilizing chatgpt. based on our tempcompass benchmark, we evaluate 12 sota multi-modal llms (mllms), including 9 video llms and 3 image llms. the evaluation results reveal that the video llms demonstrate a deficiency in temporal perception skills, failing to surpass their image llms counterparts. we also find that the temporal perception ability of mllms indeed varies a lot across different task formats, which emphasizes the need to incorporate diverse task formats in the assessment process. the main contributions of this work are summarized as follows: (1) we present a benchmark with diverse temporal aspects and task formats to comprehensively evaluate the temporal perception ability of video llms. (2) we introduce conflicting videos that prevent video llms from exploiting singe-frame bias or language priors. (3) we combine rule-based and llm-based methods to efficiently and accurately evaluate the responses from video llms. (4) our empirical results reveal the weak temporal perception ability of sota video llms. ## tempcompass benchmark tempcompass is a dataset of videos and task instructions intended to test the temporal perception ability of video llms. this section will introduce the temporal aspects, task formats and static contents included in tempcompass (section 3.1), how to collect the videos and task instructions (section 3.2) and how to automatically evaluate video llms on tempcompass (section 3.4). ## conclusions in this work, we propose the tempcompass benchmark to evaluate the temporal perception ability of video llms. our benchmark introduces ten temporal aspects and four distinct types of task formats, which offers a comprehensive view to investigate the temporal perception capability. two innovative strategies are devised in the data collection process, including (1) the construction of conflicting videos to mitigate the influence of single-frame bias and language-priors and (2) the collaboration of human annotation and llm generation to efficiently collect high-quality task instructions. we also propose an automatic evaluation method based on chatgpt, which is able to accurately assess the free-form video llm responses. based on temp-compass, we extensively evaluate 9 sota video llms and 3 image llms. our evaluation results reveal the pressing need to enhance the temporal perception ability of video llms.