Grounded- V ideo LLM : Sharpening Fine-grained Temporal Grounding in Video Large Language Models (2025)

Abstract: Despite their impressive performance in coarse-grained video understanding, Video Large Language Models (Video-LLMs) still face challenges in fine-grained temporal grounding, including ineffective temporal modeling and inadequate timestamp representations. In this work, we introduce Grounded-VideoLLM, a novel Video-LLM designed to perceive and reason over specific video moments with fine-grained temporal precision. Our model features (1) a two-stream encoder that explicitly captures inter-frame relationships while preserving intra-frame visual details and (2) discrete temporal tokens enriched with structured time knowledge for timestamp representation. Besides, we propose a multi-stage training strategy tailored to such grounding-specific architecture. The model is initially trained on simple video-caption tasks and progressively introduced to complex video temporal grounding tasks, ensuring a smooth learning curve and temporal alignment. We further strengthen Grounded-VideoLLM’s temporal reasoning by constructing a VideoQA dataset with grounded information using an automated annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only surpasses existing models in fine-grained grounding tasks but also exhibits strong potential as a general video understanding assistant.

## introduction multi-modal large language models (mllms) have made remarkable progress in image-level understanding @xcite @xcite . however, extending their capabilities to the video domain poses distinct challenges. unlike static images, the temporal nature of videos challenges models to process not only visual content but also the sequence and timing of events. while current video-llms @xcite @xcite are capable of capturing global visual semantics and generating coarse-grained captions for short clips, they struggle with fine-grained video understanding @xcite , which requires decomposing the video along the temporal axis to accurately perceive and reason over specific moments, such as subtle actions, transitions, and events that unfold over time. previous research efforts @xcite @xcite @xcite have explored temporal grounding to improve fine-grained video understanding. however, two main challenges impede their potential for achieving effective finegrained temporal grounding: (1) models like video-chatgpt @xcite , p-llava @xcite , and video-llama @xcite typically sample multiple frames from a video and encode each frame independently using an image encoder, followed by a feature projector (e.g., sliding q-former @xcite , visual adapter @xcite ). this focuses primarily on spatial details while potentially neglecting the temporal relationships between frames since their visual encoders are solely trained on images. (2) current models also struggle with timestamp representation, which is crucial for pinpointing specific moments in time for fine-grained understanding. models such as timechat @xcite and vtimellm @xcite represent timestamps as plain texts, for example, ["from 102.3 to 120.1 seconds"]. despite being straightforward, this needs to tokenize continuous floating-point values, which is inefficient for llms since their next-token prediction paradigm struggles with handling numerical data @xcite . although there have been some previous works @xcite @xcite using spe- ## related work video large language models have caught a growing interest for general video understanding @xcite . however, they struggle with temporal perception @xcite and exhibit hallucination @xcite ) when asked about specific moments. they typically encode each frame independently using an image encoder to create video embeddings, resulting in video representations lacking inherent temporal information and heavily relying on the position embeddings of llm for temporal understanding, limiting the model's capability to perform fine-grained temporal grounding. norton @xcite employs video-paragraph and clip-caption contrastive losses to capture long-term dependencies for fine-grained alignment. in this work, we employ a two-stream architecture that integrates a video expert to extract motion features to complement the appearance features during the early encoding process. this is different from traditional two-stream networks @xcite since we don't rely on heavy extraction of optical flows. additionally, we employ a progressive training strategy that gradually adapts an image-based mllm for fine-grained video understanding. although concurrent works such as slowfast-llava @xcite and videogpt+ @xcite also introduce an additional stream, slowfast-llava relies on a single image encoder to process each video frame without training, missing crucial temporal relationships between frames. videogpt+ merely arranges video tokens as a prefix to image tokens using sparse frames. instead, our approach is specifically designed for fine-grained temporal grounding, leveraging a unique encoding, pooling, and training strategy tailored for dense frames, along with a dedicated grounding mechanism. video temporal grounding (vtg) tasks usually include temporal sentence grounding @xcite , dense video captioning @xcite , and grounded videoqa @xcite . given the emerging capabilities of video-llms @xcite @xcite , many studies have investigated how to adapt them for vtg tasks. for example, timechat @xcite and vtimellm @xcite perform temporal grounding using a fully text-to-text approach through instruction-tuning datasets. momentor @xcite introduces a temporal perception module to address the quantization errors, and vtg-llm @xcite incorporates absolute-time tokens to handle timestamps. compared to these, we avoid textual representation of timestamps and instead introduce discrete temporal tokens for timestamp encoding. different from previous methods that also use special tokens @xcite @xcite , our model is more efficient by continuing training based on an established image mllm with a two-stream architecture and a progressive training strategy. ## model architecture given that current mllms already exhibit strong image-understanding capabilities, our architecture aims to sharpen temporal awareness by capturing motion dynamics across frames, which serve as a vital supplement to spatial content. as shown in figure 2 , we develop grounded-videollm upon a well-established mllm for spatial comprehension and integrate an expert video encoder for temporal comprehension. additionally, to avoid tokenizing numerical texts, we incorporate temporal tokens into the llm's vocabulary for efficient and unified timestamp representation. ## progressive training different from previous methods @xcite ) that train models from scratch using mixed image and video datasets, we start with a pre-trained image-based mllm (microsoft, 2024) and progressively enhance its finegrained temporal grounding capabilities. this strategy can be applied to any off-the-shelf mllm and is more efficient. stage-1: video-caption alignment. we leverage approximately 1.28 million video-text pairs sampled from diverse sources @xcite @xcite to align video encoder's features with the mllm. this alignment allows the mllm, which was pre-trained solely on images, to gain a foundational understanding of videos. since 2d down-sampling has been applied to the visual features, only the two projectors f (•) and g(•) are trainable, while the video encoder, image encoder, and llm remain frozen. as this stage does not involve any temporal grounding tasks, the temporal tokens in sec.3.2 are not yet incorporated. stage-2: temporal token alignment. while video-caption alignment connects videos and the mllm at a coarse level, a gap persists between this alignment and fine-grained temporal grounding. to address this, we introduce the temporal tokens described in sec.3.2 and continue pre-training the model on a diverse range of grounding datasets @xcite @xcite , focusing on tasks such as temporal sentence grounding, dense video captioning, and temporal referring, which enables the model to refer to and localize temporal information effectively. since new tokens are introduced, we additionally make the word embedding matrix and the final classifier head of the llm trainable. this stage enhances the model's ability to comprehend multiple events and aligns the temporal tokens with both the video timelines and the llm's semantic space as shown in figure 4 and stage-3: multi-task instruction tuning. following the initial two stages, the model has developed a basic understanding of video content and can locate specific timestamps. in this stage, we 3-103.7s) (0-9.3s) (68.9-72.0s) (74.6-79.3s) the other swimmers get to the finish line. prompt for questionanswer generation q: what does the girl do after taking off goggles? a: wave at the camera and smile t: (74.6-79.3s) q-a pairs of other videos cos sim retrieve and sample 4 distractors (-) start swimming again (-) get out of the pool (-) dive back in (-) hold the breath further enhance the model's fine-grained temporal grounding while improving its responsiveness to diverse instructions. to achieve this, we gather two types of datasets: (1) we compile a wide range of datasets for temporal grounding tasks, similar to time-it (ren et al., 2024) and vtg-it (guo et al., 2024), which include tasks of dense video captioning (remove activitynet (caba heilbron et al., 2015)), temporal sentence grounding, and grounded videoqa. (2) we incorporate a subset of instructional datasets from videochat2 (li et al., 2024) and sharegpt-4video (chen et al., 2024a) to further enhance the model's ability to generate detailed video captions. by utilizing these diverse datasets, which encompass both temporal grounding and video instruction tasks, grounded-videollm excels in temporal referring, localization, reasoning, and general comprehension of video content. in this stage, the trainable parameters remain the same as in stage 2, with the addition of lora parameters (hu et al., 2022) for the llm. ## grounded videoqa dataset generation grounded videoqa requires the model to not only answer questions but identify timestamps that support predicted answers, demonstrating the temporal reasoning abilities. next-gqa @xcite was manually developed by extending next-qa @xcite with temporal labels for start and end timestamps. however, annotating these labels is labor-intensive and time-consuming, limiting next-gqa only to qa pairs for the validation and test sets. to create a scalable training dataset, as depicted in figure 3 , we utilized openai @xcite to assist in constructing training sets for the grounded videoqa task. these sets were built on public datasets that already contain temporal labels, such as qvhighlights @xcite . we framed the task as a multiple-choice videoqa using a two-round conversational format as depicted in appendix a.3. tasks and benchmarks. we assess grounded-videollm across three video temporal grounding tasks: temporal sentence grounding, dense video captioning, and grounded videoqa, utilizing benchmarks such as charades-sta @xcite , activitynet-captions @xcite , and next-gqa @xcite . we also show its reasoning capability by the task of open-ended videoqa with benchmarks including msvd-qa, msrvtt-qa @xcite , and acticitynet-qa @xcite . additionally, to evaluate the model's general video understanding capabilities, we benchmark grounded-videollm against existing models using vcg-bench @xcite and mvbench @xcite . the evaluation details are in appendix a.4. ## conclusion we present grounded-videollm, a novel video-llm that incorporates a two-stream encoding for temporal modeling, along with the temporal tokens for timestamp representation. we employ a multi-stage training scheme, starting with an imagebased mllm and progressively equipping it with fine-grained temporal grounding capabilities. additionally, we curate a grounded-videoqa dataset to further enhance the model's temporal reasoning ability. extensive experiments demonstrate that grounded-videollm not only excels in video temporal grounding tasks but also performs strongly on general video understanding benchmarks.