V ideo INSTA : Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLM s (2024)

Abstract: In the video-language domain, recent works in leveraging zero-shot Large Language Model-based reasoning for video understanding have become competitive challengers to previous end-to-end models. However, long video understanding presents unique challenges due to the complexity of reasoning over extended timespans, even for zero-shot LLM-based approaches. The challenge of information redundancy in long videos prompts the question of what specific information is essential for large language models (LLMs) and how to leverage them for complex spatial-temporal reasoning in long-form video analysis. We propose a framework VideoINSTA , i.e. INformative Spatial-TemporAl Reasoning for zero-shot long-form video understanding.VideoINSTA contributes (1) a zero-shot framework for long video understanding using LLMs; (2) an event-based temporalreasoning and content-based spatial reasoning approach for LLMs to reason over spatial-temporal information in videos; (3) a self-reflective information reasoning scheme based on information sufficiency and prediction confidence while balancing temporal factors.Our model significantly improves the state-of-the-art on three long video question-answering benchmarks: EgoSchema, NextQA, and IntentQA, and the open question answering dataset ActivityNetQA. Code is released: https://github.com/mayhugotong/VideoINSTA.

## introduction large language models (llms) have demonstrated remarkable reasoning abilities, even in longcontext situations @xcite @xcite . these advancements have spurred interest in video reasoning. previous works bridging video and text modalities depend on meticulously designed models suffering largescale pretraining. this challenge is pronounced with videos, a data format characterized by a vast volume of information scaling with length. consequently, these models exhibit limited generalizability across datasets and struggle to scale to long video within a single model @xcite . more recent models have gradually integrated llms' reasoning abilities by introducing lightly tuned adaptation layers @xcite @xcite . however, they still struggle with the length of the videos. recently, to avoid expensive training costs, early attempts have proposed a zero-shot solution by reasoning over semantic representations of video content using llms @xcite @xcite . these approaches have become strong competitors to earlier end-toend models. nonetheless, long-form video understanding, which demands advanced reasoning over extended timespans, remains challenging even for llm-based methods. even in light of these tryouts, many challenges remain unsolved: (1) information quality. videos contain vast information even with some redundancy due to minor visual changes. identifying the most crucial piece of information and extracting it effectively is essential to enhance the quality of data within the context window manageable by llms. how can we achieve this extraction? (2) neglect of spatial and temporal characteristics. videos inherently exhibit temporal and spatial characteristics. how can we effectively preserve and convey this spatial-temporal information to support llm reasoning? especially, how do llms process temporal dynamics in videos? (3) complexity of reasoning with unbalanced information over temporal span. in long videos, the significance of information along the video temporal axis varies greatly. llms' implicit "intuition" to process all the information is insufficient. how do we develop an explicit reasoning algorithm for unbalanced information considering temporal factor? to address these challenges, we propose a framework videoinsta, i.e. informative spatial-temporal reasoning for zero-shot long-form video understanding, aiming to build a compound system extracting essential information from long-form videos -leveraging spatial-temporal reasoning and temporal-aware self-reflective reasoning to handle complex information with llms. videoinsta is a zero-shot framework for reasoning with llms, augmented with visuallanguage tools. first, this framework emphasizes event-based temporal reasoning by proposing an automatic temporal segmentation method c-dpcknn, which segments long videos into multiple events. besides, it derives the global temporal information with the help of a unified temporal representation tool univtg @xcite and utilizes a temporal grounding scheme allowing the event to inherit the local temporal information. second, this framework emphasizes content-based spatial reasoning by improving video captions with various visual-language captioning tools to extract richer spatial information. specifically, event captioning is compensated by object detection and action caption as spatial information. a follow-up summarization serves as implicit spatial reasoning in a chain-of-thought manner. third, this framework proposes iterative information reasoning with llms, which iteratively merges the temporal and spatial information derived in the previous stages based on the selfevaluation of llms on the information sufficiency and prediction confidence. experiments have showcased remarkable improvements in existing long-form video questionanswering tasks compared to end-to-end videolanguage models as well as other zero-shot llmbased video understanding compound systems. besides, videoinsta handles long videos with an average length of 3 minutes and is easily extensible for longer videos in a zero-shot manner. this framework also shows excellent results both on multi-choice and open-question answering tasks. the main contributions are summarized as follows: spatial-temporal reasoning on video spatialtemporal reasoning in video has been a topic of continuous discussion @xcite @xcite @xcite @xcite @xcite due to the dual characteristics of video data. most previous approaches compress information and perform reasoning within the embedding space. additionally, recent works have highlighted llms' capabilities in temporal @xcite @xcite @xcite and spatial reasoning @xcite @xcite @xcite . however, applying llms' spatialtemporal reasoning abilities to video remains underexplored. our work innovatively harnesses these abilities, augmenting them with spatial-temporal reasoning methods as tools, to effectively analyze long-form videos both spatially and temporally. ## videoinsta: informative spatial-temporal reasoning with large language models in this section, we explain our videoinsta framework shown in figure 1 following its three-phase methodology: event-based temporal reasoning, content-based spatial reasoning, and self-reflective information reasoning with llms. ## extensibility of the framework extensible api tools videoinsta is a general framework for informative spatial-temporal reasoning on videos and maintains the extensibility to improve both, the temporal reasoning and spatial reasoning phases by acquiring informative prompts from different expert tools through apis. for example, expert temporal segmentation models can be utilized for better event parsing in the temporal reasoning phase in videoinsta. expert spatial models like high-fidelity captioning models and object detectors can provide more accurate informative prompts for the spatial reasoning phase. open question answering apart from singlechoice question answering, videoinsta can also be easily adapted to open question answering. we tested videoinsta on acitivitynet-qa @xcite , which is a dataset for open-ended question answering over complex web videos. following @xcite , we also conduct evaluation in a zero-shot manner, employing llm-assisted evaluation to assess the predictions' accuracy of videoinsta. ## experimental setup in this section, we describe the experimental setup of the videoinsta framework. we present quantitative results and a qualitative analysis on the egoschema @xcite , next-qa @xcite , and intent-qa @xcite benchmarks. egoschema egoschema is a benchmark for long-form video understanding, featuring 5,000 single-choice questions derived from egocentric videos. a distinctive feature of this dataset is the length of its videos, each lasting 180 seconds. egoschema comprises only a test set, with a subset of 500 questions having available labels. nextqa the next-qa dataset includes 5,440 natural videos that feature object interactions in daily life, accompanied by 48,000 single-choice questions. the average length of the video is 44 seconds. in line with standard practices, our zeroshot evaluation is focused on the validation set. intentqa intentqa focuses on intent reasoning. it contains 4,303 videos and 16k single-choice question-answer pairs focused on reasoning about people's intent in the video. the videos are more than 44 seconds in average length. we perform a zero-shot evaluation on the test set. evaluation metrics since each dataset features single-choice questions and videoinsta generates option predictions directly, we utilized accuracy as the evaluation metric. baselines the baselines include recent representative llm-based zero-shot video understanding methods -including llovi, videoagent, proviq and morevqa -and other baselines include supervised end-to-end models, see experiment design to comprehensively analyze videoinsta, there are two research questions. rq1: how is the performance of the proposed videoinsta framework compared to the existing end-to-end models and llm-based compound systems? rq2: how do the components of the videoinsta affect its effectiveness? implementation details following llovi and videoagent, we utilize the lavila model retrained on ego4d, filtering out videos that overlap with egoschema to ensure zero-shot evaluation. ## conclusion this work focuses on understanding long-form videos with llms -particularly emphasizing information quality, spatial-temporal reasoning, and explicit complex reasoning across unbalanced distributed information. the proposed trainingfree framework videoinsta for long-form video understanding showcases exceeding performance over state-of-the-art end-to-end and zero-shot llm-based methods. it further reveals the potential on open question answering and the extensibility of various visual-language tool-augmented spatial-temporal reasoning approaches.