Question-Instructed Visual Descriptions for Zero-Shot Video Answering (2024)

Abstract: We present Q-ViD, a simple approach for video question answering (video QA), that unlike prior methods, which are based on complex architectures, computationally expensive pipelines or use closed models like GPTs, Q-ViD relies on a single instruction-aware open vision-language model (InstructBLIP) to tackle videoQA using frame descriptions. Specifically, we create captioning instruction prompts that rely on the target questions about the videos and leverage InstructBLIP to obtain video frame captions that are useful to the task at hand. Subsequently, we form descriptions of the whole video using the question-dependent frame captions, and feed that information, along with a question-answering prompt, to a large language model (LLM). The LLM is our reasoning module, and performs the final step of multiple-choice QA. Our simple Q-ViD framework achieves competitive or even higher performances than current state of the art models on a diverse range of videoQA benchmarks, including NExT-QA, STAR, How2QA, TVQA and IntentQA.

## introduction recently, vision-language models have shown remarkable performances in image questionanswering tasks @xcite @xcite , with models such as flamingo @xcite , blip-2 @xcite , instructblip @xcite and mplug-owl @xcite showing strong reasoning capabilities in the vision-language space. image captioning @xcite is one of the tasks in which these models truly excel, as they can generate detailed linguistic descriptions from images. different works have leveraged this capability in many ways for zero-shot image-question answering, such as giving linguistic context to images @xcite , addressing underspecification problems in questions @xcite , coordination of multiple image captions to complement information @xcite , or by combining captions with other type of linguistic information from the image @xcite . in this manner, the reasoning capabilities of llms can be directly used to reason about the linguistic image descriptions and generate an answer for the given visual question. this approach has been successful for images, but in the case of video-question answering tasks @xcite @xcite @xcite this is more challenging. video possesses multiple image frames that have relationships between each other and involve the recognition of objects, actions, as well as the inference about semantic, temporal, causal reasoning and much more @xcite . thus, some works @xcite have focused on using chatgpt to either ask visual questions to image-language models like blip-2 or to respond and retrieve useful information from large datasets with detailed linguistic information from the video. similarly, @xcite have leveraged the reasoning capabilities of gpt-3.5 to create textual summaries from the video, and later perform video qa using only textual information. while others @xcite combine linguistic information from multiple sources such as captions, visual tokenization or even subtitles of input speech. in summary, current methods for video qa rely on any combination of closed llms, expensive training regimes, and complex architectures with multiple modules @xcite @xcite @xcite . in contrast, we introduce q-vid a simple question-instructed visual descriptions for video qa approach that relies on an instructionaware vision-language model, instructblip @xcite , to automatically generate rich specific captions from video frames. in this manner, we effectively turn the video qa task into a text qa task. more specifically, given an input video v we sample n number of frames, then, we generate question-specific instructions to prompt the multimodal instruction tuned model to generate captions for each frame. afterwards, we form a video description by concatenating all the generated question-dependent captions from instruct-blip, and use it along with the question, options and a question-answering instruction prompt as input to the llm-based reasoning module that generates an answer to the multiple-choice question about the video. we demonstrate the effectiveness of q-vid on five challenging multiple choice video question answering tasks (next-qa, star, how2qa, tvqa, intentqa), showing that this simple framework can achieve strong performances comparable with more complex pipelines. our contributions are summarized as follows: ## method recently, vision-language models trained with instruction tuning @xcite @xcite have shown impressive capabilities to faithfully follow instructions and extract visual representations adapted to the task at hand. thus, with q-vid (figure 1 ),we propose to leverage these capabilities for multiple-choice video qa, and turn this task into textual qa using in-structblip @xcite . we use a questiondependent captioning prompt as the input instruction, to guide instructblip to generate video frame descriptions that are more relevant for the given question. afterwards, we reuse the llm from instructblip and use it as our reasoning module. this llm (flan-t5) takes a question-answering prompt as input, that consists of a video description formed by the concatenation of all the questiondependent frame captions, the question, options and a task instruction. considering that flan-t5 is also originally trained with instructions, we aim to leverage its reasoning capabilities to correctly answer the question given only the text we just described as input. our simple approach does not rely on complex pipelines or closed gpt models, which makes it easy, cheaper and straight forward to use for zero-shot video qa. on the other hand, q-vid is flexible and model agnostic, which means we can use any multimodal models available. this section presents our approach in detail. first, we in-troduce some preliminary information on instruct-blip, which serves as the foundation of our work, and then we provide a detailed overview for all components from our q-vid framework. ## experiments in this section, we present our experiments for zeroshot video qa. first, we describe the datasets we used and the implementation details. then, we evaluate our approach, compare q-vid with other state of the art models for video qa and provide a comprehensive analysis of the model's performance. lastly, we conduct some ablation studies of q-vid regarding the instructions prompt design. ## ablation studies in this section, we perform some ablation studies related to the instruction prompt selection for q-vid. for these experiments, we chose next-qa and star as our benchmarks, and report results on the validation sets on each dataset. specifically, we test two model variations, using instructblip-flant5 xl (q-vid xl ) and the one used to report our main results, instructblip-flant5 xxl (q-vid xxl ), we test different prompts to analyze and ## conclusion this information is later used by a reasoning module with a question-answering instruction prompt to perform multiple-choice video qa. our simple approach achieves competitive or even higher performances than more complex architectures and methods that rely on closed models like the gpt family. in our ablation studies we show that using dedicated instructions to get question-dependent captions works better than common prompts to get general descriptions from frames to perform video qa using captions.