Let’s Think Frame by Frame with VIP : A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought (2023)

Abstract: Despite exciting recent results showing vision-language systems’ capacity to reason about images using natural language, their capacity for video reasoning remains underexplored. We motivate framing video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language while alleviating the computational complexities of processing videos. To evaluate this novel application, we introduce VIP, an inference-time challenge dataset designed to explore models’ reasoning capabilities through video chain-of-thought. Inspired by visually descriptive scene plays, we propose two formats for keyframe description: unstructured dense captions and structured scene descriptions that identify the focus, action, mood, objects, and setting (FAMOuS) of the keyframe. To evaluate video reasoning, we propose two tasks: Video Infilling and Video Prediction, which test abilities to generate multiple intermediate keyframes and predict future keyframes, respectively. We benchmark GPT-4, GPT-3, and VICUNA on VIP, demonstrate the performance gap in these complex video reasoning tasks, and encourage future work to prioritize language models for efficient and generalized video reasoning.

## introduction constituting 65% of all internet traffic in 2023, videos are an area of huge potential for the next chapter of leveraging artificial intelligence @xcite @xcite . for example, video question answering @xcite and video summarization @xcite are two existing datasets that empirically evaluate video models. yet, they do not assess more challenging tasks, such as reasoning through specific relationships between multiple frames. just like how humans understand videos by processing frames across time steps, ai's ability to accomplish multi-frame reasoning is a core task for video understanding. multi-frame video reasoning is bottlenecked by the sheer computing resources needed to process videos, which typically contain 24 frames per second and can vary widely. however, intelligible videos tend to have little variation from frame to frame. akin to selecting principle components on the axes containing the highest orthogonal variance, picking a small sample of keyframes can capture much of the video's meaning. understanding a video via extracted keyframes poses the challenge of multi-hop reasoning, which requires stronger language model capabilities. to evaluate this challenging multi-hop multiframe video reasoning, we elicit video chain-ofthought (videocot). we propose two tasks to test such capabilities in existing models -video infilling and video prediction. in the video infilling setting, the goal is to predict the masked keyframes' descriptions when given the previous and next keyframes' descriptions in sequence as context, following the spirit of masked language modeling. in the video prediction, the goal is to predict the descriptions of the next keyframes in the sequence when given a set of previous keyframes' descriptions, similar to the next token prediction task. these tasks can help analyze video generation and whether video models truly understand the dynamic relations between subsequent video frames, given the variable context gap between keyframes. to benchmark our proposed video infilling and prediction tasks, we construct a dataset by proposing an automated method to extract keyframes. in addition to the frame itself, we propose two textual representations -unstructured dense captions and famous structured scene descriptions (figure 1 ). the unstructured captions are intended to extract more significant and visually descriptive informa- ## vip dataset construction to construct the dataset, we first outsource the video corpus to stem from the youtube-8m dataset @xcite , whose diverse, realistic videos with human-labeled topics align well with our desiderata. to effectively enable multi-frame video reasoning, we downsample visually static categories such as weather, which may not contain much change throughout the video. the weights of each category 2 is described in figure 2 . then, to reduce the computational complexity of video processing, we reduce a video into a set of keyframes that seek to capture the video's meaning ( §3.1). to accommodate the limitations of existing models, we also generate two forms of visually-descriptive, textual representations-unstructured dense captions and famous scene descriptions ( §3.2). figure 3 summarizes this automated pipeline, which reduces the cost ordinarily spent to manually construct such a dataset. 2 videos may be in multiple categories. proportion of videos covered video categories hobbies & leisure food & drink business & industrial pets & animals travel autos & vehicles home & garden jobs & education law & government sports people & society reference science games health real estate 0.00 0.05 0.10 0.15 0.20 ## video reasoning tasks taking inspiration from existing natural language tasks, we propose two tasks for videos that explore a model's multi-frame reasoning capabilities. the video infilling task requires models to predict a set of keyframes given the preceding and following frames, akin to masked language modeling for keyframes. video prediction tasks models to predict the most likely sequence of frames to follow a given series of frames -parallel to the text completion task. video infilling and prediction of keyframes are two general tasks with several downstream contexts that can benefit from video understanding and completion. to concretely define the tasks below, we represent the sequence of chronological keyframes as k 1 , . . . , k n , their respective unstructured dense captions as u 1 , . . . , u n , and famous structured scene descriptions as s 1 , . . . , s n . ## future work in this paper, we aim to lay the groundwork for exploring the challenging topic of multi-frame, multihop reasoning within the existing capabilities of deep learning models. naturally, this opens several directions for exciting future work. in the language model space, we benchmark the performance of several leading models using textual representations of keyframes for video-related reasoning following a standard in-context inference procedure with few-shot demonstrations. this invites the opportunity to discover more targeted inference-time techniques using language models or vision-language models to improve the performance of video reasoning tasks beyond a general paradigm. similarly, additional training-time ef-fort could be worthwhile through fine-tuning or a more traditional train-validate-test paradigm to learn skills beyond the general pre-trained learnset. in this vein, collecting additional data samples could improve the feasibility of these research threads. beyond the language modality, bridging the video reasoning task end-to-end with video is a longer-term research direction with immediate benefits in animation. our paper reduces the video reasoning task into a language task with a textual output. image synthesis would be an immediate step to reconstruct the keyframe image. then, video synthesis from a set of images would naturally follow. finally, unifying these disjoint tasks could benefit from error reduction and improved usability. as video reasoning is a new space, developing robust evaluation metrics would be a valuable contribution. some desirable but difficult properties to consider in this area include the ability to capture both the spatial and temporal invariance that could occur through videos, as multiple interchangeable actions are plausible within different areas of the frame and sequences. finally, our general video reasoning tasks pose the prospect of efficient transfer learning where improving on such a task could benefit several new applications, similar to the contemporary boom of language technologies. ## conclusion we present the inference-time video infilling and prediction dataset to evaluate models' video reasoning abilities by performing a video chain-ofthought on video keyframes. to collect this dataset, we introduce a novel pipeline to systematically extract keyframes and generate corresponding textual representations -unstructured dense captions and structured famous scene descriptions. we benchmark state-of-the-art language models on vip tasks and evaluate their ability to generate these textual representations of keyframes through a video chain-of-thought. these models display potential to perform video-related reasoning yet have significant potential to improve. by testing multi-hop, multi-frame reasoning abilities on sparse keyframes, we hope to promote research into developing models that support real-world video understanding and video generation while being resource efficient.