Self-Adaptive Sampling for Accurate Video Question Answering on Image Text Models (2024)

Abstract: Image–text models (ITMs) is the prevalent architecture to solve video question–answering tasks, which requires only a few input frames to save huge computational cost compared to video–language models.However, we find existent ITM video question–answering solutions either 1) adopt simplistic and unintentional sampling strategies, which may miss key frames to offer the answer clues; or 2) sample a large number of frames into divided groups, which the computational sources can not accommodate. In this work, we aim at an efficient sampling method towards the few-frame situations.We first summarize a family of prior sampling methods based on question–frame correlation into a unified one, dubbed *Most Implied Frames* (MIF). Through some primary results and analysis, Through analysis, we form a hypothesis that question-aware sampling is not necessary, from which we further propose the other method *Most Dominant Frames* (MDF).Experimental results on four public datasets and three advanced ITMs demonstrate that our proposed strategies can boost the performance for image–text pretrained models, and have a wide application scenario in terms of model architectures and dataset types. Our code is available at https://github.com/declare-lab/Sealing https://github.com/declare-lab/Sealing .

## introduction with the advancement in computer vision technology, we are witnessing an explosive surge of visual data. together, research in vision-language understanding has progressed significantly in the past decade, challenging a wide variety multimodal application tasks @xcite @xcite @xcite , such as image captioning, visual question answering and multimodal retrieval. with the continuing improvement in computation, researchers have extended conventional image-text models (itms) to video-text ones, mainly by sub- on-disk video system ram input frames/clips stituting image encoders with their video counterparts @xcite @xcite . this learning paradigm achieves decent performance on numerous video-text tasks, as it incorporates temporal features into modeling. nevertheless, 3d convolution, the core technique adopted in these video-text pretrained models, demands tremendous computational power in terms of both time and memory, limiting models' deployment on consumer-level devices. a straightforward solution to reduce overhead is to extract solely those keyframes that describe the main content or are related to the task from a given video, so that image-text models can preprocess them @xcite @xcite . contemporary auto-regressive itms manage to adapt themselves to video-text tasks with a few frames sampled from those videos and yield promising results @xcite . in this family of approaches, image frames or clips (consecutive frames, as shown in fig. 2a ) are sampled from raw videos, cut into patches, and then encoded through a visual encoder (e.g., resnet @xcite and @xcite ). x-clip @xcite further inserts cross-frame communication modules to construct connections across timestamps. despite performing well, we observe that the sampling strategies employed in these models are simplistic: they are blind to the video and question and only base on statistical probability distributions (fig. 2a ). these data-agnostic approaches inevitably limit the performance when finetuning and inferring on these itms, since they may cause keyframe omission (fig. 3 ). on the other hand, recent works @xcite introduce learningbased sampling methods. assisted by the gumbel-softmax trick @xcite , they build a parametric sampling network and concatenate that to the backbone. then, as an auxiliary module, the parametric sampling strategy is jointly optimized with the main video-qa task. although these frameworks gain competitive performance, they have the following drawbacks. first, they sacrifice efficiency owing to the additional overhead and the slow convergence speed caused by the devised sampling network, compared to direct few-frame finetuning on itms (from less than 10 epochs to more than 50 epochs) @xcite . secondly, it also undermines flexibility-the intervention touches the preprocessing stage in these works @xcite . they encode the pre-sampled clips with customized pretrained video encoders, like 3d resnet101 @xcite or clip @xcite , leading to incompatibility with itms which only accept raw images as input. additionally, the sampling network must be optimized along with the backbones on such clip features, which deters them from being directly applied to itms. to address these issues, we first explore the correlation between model's performance and the frames output from captioning-based samplers. specifically, we propose a learning-free sampling method, dubbed most implied frames (mif), which we show is an simplified and unification of previous v(isual) q(uestion)-aware methods. it utilizes lightweight pretrained models to annotate frames and grades each of them with a captionquestion score. the selected frames are those with highest scores, or the best captions that imply the answer. then, we conclude from empirical studies on mif that capturing the most questionrelated frames is not a prerequisite for better accuracy. based on our analysis, we hypothesize that question-aware sampling is not necessary and propose another self-adaptive sampling strategy-most dominant frames (mdf). the underlying logic is to diversify the input frames to minimize the dominant scenes in that video, because most of the answers can be answered from static scenes instead of dynamic segments. to this end, we first define a goal function that measures the dynamics in videos whose input is the visual feature encoded by the backbone model's inherent image encoder. then we devise a search algorithm to quickly locate the frames where features move slowest in that video. since question content no longer participates in the sampling process, mdf is a v-aware q-agnostic method. in implementation, both mif and mdf are executed in an offline fashion figure 1 , enhancing the training efficiency compared to those online sampling algorithms. we further conduct experiments on three itms (clip @xcite , git @xcite and all-in-one @xcite ) using four widely tested video qa datasets. the results show that both methods are feasible solutions towards video-qa tasks on itms, among which mdf can provide better efficiency, and indirectly substantiating the correctness of our hypothesis. the contributions in our paper are as follows: ## method in this section, we first briefly recap the background of the video-qa task on itms. then we introduce the most implied frames (mif), a generalization to previous question-aware sampling approaches. we report some primary results and describe our key findings to the statistics. finally, based on these discoveries we introduce the more efficient most dominant frames (mdf). ## conclusion in this paper, we focus on the frame sampling issue inhering in the task of video question-answering and propose two simple and effective methods-most implied frames (mif) and most dominant frames (mdf). mif streamlines a set of sampling methods in the textual space by projecting heterogeneous inputs (question and video) to a common space through pretrained itms. it then identifies frames with the highest matching scores generated from a scoring model. based on the insights and analysis derived from mif, we further propose most dominant frames (mdf), which exploits a more concise, self-adaptive formulation for sampling. the success on these sampling strategies from clip to all-in-one demonstrates the broad applicability of our proposed methods across a spectrum of general scenarios.