Generative Frame Sampler for Long Video Understanding (2025)

Abstract: Despite recent advances in Video Large Language Models (VideoLLMs), effectively understanding long-form videos remains a significant challenge. Perceiving lengthy videos containing thousands of frames poses substantial computational burden. To mitigate this issue, this paper introduces Generative Frame Sampler (GenS), a plug-and-play module integrated with VideoLLMs to facilitate efficient lengthy video perception. Built upon a lightweight VideoLLM, GenS leverages its inherent vision-language capabilities to identify question-relevant frames. To facilitate effective retrieval, we construct GenS-Video-150K, a large-scale video instruction dataset with dense frame relevance annotations. Extensive experiments demonstrate that GenS consistently boosts the performance of various VideoLLMs, including open-source models (Qwen2-VL-7B, Aria-25B, LLaVA-Video-7B/72B) and proprietary assistants (GPT-4o, Gemini). When equipped with GenS, open-source VideoLLMs achieve impressive state-of-the-art results on long-form video benchmarks: LLaVA-Video-72B reaches 66.8 (+4.3) on LongVideoBench and 77.0 (+2.7) on MLVU, while Aria obtains 39.2 on HourVideo surpassing the Gemini-1.5-pro by 1.9 points.

## introduction recent advances in large multimodal models (lmms) @xcite @xcite @xcite @xcite have shown remarkable progress, yet understanding long videos remains a significant challenge @xcite . current video-oriented lmms @xcite @xcite @xcite , known as videollms, typically employ an im-age encoder such as clip @xcite or siglip @xcite to encode individual video frames as an initial step in video perception. when processing hours-long videos containing thousands of frames, a critical challenge emerges: how to efficiently sample representative frames from the original video sequence? existing videollm assistants primarily employ two approaches for sampling lengthy videos: 1) uniform sampling based on the videollm's maximum context length, which leads to significant visual information loss due to limited fixed-interval sampling; 2) frame-per-second (fps) sampling, as implemented in long-context models like gemini @xcite , which can capture frames at 1 fps for comprehensive visual coverage. however, it obtains thousands of frames for hours-long videos, resulting in booming memory consumption and slow inference speed. intuitively, for videollm assistants, most frames in long videos are redundant when addressing a specific user instruction (i.e., query). to mitigate visual redundancy, several works propose language-guided frame sampling via clip to retrieve query-aware frames efficiently @xcite . however, clip-based frame samplers have three major limitations. for visual side, its frame-by-frame matching fails to capture temporal relationships implied by successive frames, as depicts in figure 1 (a) . for textual side, it is constrained by limited language capabilities, only able to process concise and simple user queries. additionally, it embeds frames and textual queries separately to calculate cosine similarity, which hinders sufficient vision-language interaction to achieve complex multi-hop reasoning @xcite @xcite . to mitigate these limitations, we present generative frame sampler (gens), a videollmbased approach to retrieve relevant frames through question: what did the male protagonist in the video do immediately after finishing his personal report at the meeting? answer: "watch a friend unbox a watch." answer: "cook and have lunch." ## method we introduce the novel gens method that effectively selects instruction-aware frames from long-form videos. to address the challenge of insufficient training data, we first construct gens-video-150k, a video instruction dataset with dense relevant frame annotations (section 2.1). we then present the gens architecture, focusing on an efficient generative format for videollm-based frame retrieval (section 2.2). finally, we demonstrate how to integrate gens with existing videoqa assistants to enhance long-form video perception (section 2.3). ## extension applications coarse-to-fine hybrid sampling. we propose a coarse-to-fine hybrid approach that combines a performance indexing format discrete numbers (relevance order) 55. 7 continuous spans (relevance order) 56. 1 continuous spans (temporal order) 53. 2 text labels 57.7 text labels + visual labels 57.5 output format input format lightweight clip @xcite sampler with our gens to improve sampling efficiency for extremely long videos. specifically, we first adopt clip to densely sample frames from the 1 fps candidate pool and return top 256 most relevant frames, then apply gens to re-sample the most informative frames within a single 256-frame temporal window. results in ## conclusion this paper presents gens, a novel generative frame sampling method and a high-quality video instruction dataset gens-video-150k. our extensive experiments show that gens brings consistent improvements across different videoqa models' architectures and sizes and achieve new state-of-the-art results on longvideobench (66.9), mlvu (77.0), and hourvideo (40.7). it suggests that efficient frame sampling is a promising direction for advancing long-form video understanding.