Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge (2024)

Abstract: Despite progress in multimodal large language models (MLLMs), the challenge of interpreting long-form videos in response to linguistic queries persists, largely due to the inefficiency in temporal grounding and limited pre-trained context window size. In this work, we introduce Temporal Grounding Bridge (TGB), a novel framework that bootstraps MLLMs with advanced temporal grounding capabilities and broadens their contextual scope. Our framework significantly enhances the temporal capabilities of current MLLMs through three key innovations: an efficient multi-span temporal grounding algorithm applied to low-dimension temporal features projected from flow; a multimodal length extrapolation training paradigm that utilizes low-dimension temporal features to extend the training context window size; and a bootstrapping framework that bridges our model with pluggable MLLMs without requiring annotation. We validate TGB across seven video benchmarks and demonstrate substantial performance improvements compared with prior MLLMs. Notably, our model, initially trained on sequences of four frames, effectively handles sequences up to 16 longer without sacrificing performance, highlighting its scalability and effectiveness in real-world applications. Our code is publicly available.

## introduction a fundamental aspect of human intelligence is to effortlessly perceive, memorize, and comprehend daily multi-modal information such as events, observations, and videos that span hours and days. such capacity of long-form multi-modal understanding, seamlessly integrating prolonged visual dynamics with textual cues, poses considerable challenges for contemporary machine perceptual systems. a wide range of research works in computer vision and multi-modal tasks has extensively accuacy (%) 30 40 50 60 a b figure 1: training efficiency and length extrapolation of tgb. a. results of parameters on agqa (grunde- @xcite our method demonstrates the best performance with less trainable parameters. b. results of frame extrapolation on egoschema @xcite under zero-shot setting. t-num indicates the number of training context window size. by training with four-frame videos, our model shows consistent performance on extended video length. some researchers have proposed scaling up the amount of vision data fed into larger models @xcite , following the scaling law observed in llms. however, the scarcity of high-quality, long video-language datasets makes this approach difficult. others have explored sampling-based methods to reduce input overhead by selecting relevant frames at either the frame level @xcite @xcite or token level @xcite . these methods have three main limitations: first, they are computationally inefficient with slow training and inference speeds due to the large number of tunable parameters; second, the sampling strategy may miss important motion features, especially when there's misalignment between the video segment and the language query; and third, the complexity of the specialized vision encoder complicates the adaptation to long-video understanding. to address these challenges, we present a novel framework, temporal grounding bridge, which enriches image-language models with temporal priors, significantly improving the understanding of long videos. tgb distinguishes itself in the following key aspects: efficient and adaptable video compression: tgb features a bridge that is both lightweight and adaptable. to achieve this, we introduce a learnable multi-span algorithm capable of simultaneously extracting multiple relevant segments from lowdimension motion features. subsequently, we can compress the entire video into several keyframes. this method efficiently balances performance and resource consumption when processing long-form videos, as demonstrated by our results on the agqa @xcite , with a relatively low parameter count (see fig. 1a ). ## related work long-form video understanding the computational demands of processing long-form videos have led to research exploring various methods to address the challenge. a common approach involves sampling-based techniques that aim to reduce the computational load by selectively choosing relevant frames. research @xcite @xcite integrate sparse sampling within the framework of video-language pretraining. @xcite introduce an atemporal probe (atp) model that seeks to distill a single image representation from a video clip for more details. despite these advancements, there's a risk that sparse sampling may lead to an insufficient representation of visual information, which may not be relevant to corresponding language queries. mist @xcite attempts to address this by leveraging the inherent structure of videos to iteratively select and sample spatial-temporal information within a transformer architecture. nonetheless, these methods often suffer from reduced computational efficiency and prolonged training and inference times due to the extensive tunable parameters required for processing either spatial or temporal dimensions. more recent studies are exploring the utilization of llms for enhancing longform video understanding. these approaches include a range of techniques such as incorporating temporal embeddings @xcite , applying prompt-based strategies @xcite , condensing video frames through a similarity metric @xcite , compressing visual tokens with resampling methods @xcite @xcite , and employing retrieval-based methods that integrate visual features @xcite . to overcome the constraints of current methods, which usually depend on human-provided annotations for time alignment or require intricate encoding of context, our proposed approach employs a novel bootstrapping framework. this framework enhances a temporal grounding bridge, using mllms. this bridge is designed to simultaneously capture multiple granular pieces of key information by leveraging multi-span sampling, which it then integrates with low-dimensional motion features for a more efficient and effective representation. bootstrapping large language models for visual tasks capitalizing on the success of large language models (llms) in nlp, there is a growing trend of applying them to computer vision tasks, such as vqa @xcite @xcite @xcite , image generation @xcite , and visual instruction following @xcite . the research mainly progresses along three avenues: (i) leveraging llms' reasoning for visual tasks @xcite @xcite ; (ii) adapting transformer or linear networks to equip llms with visual perception @xcite @xcite @xcite ; (iii) merging llms with video and audio inputs @xcite @xcite . recently, sevila's @xcite self-chained videoqa framework uses a twostep approach: selecting keyframes with a tailored prompt and applying them to tasks. however, it faces three issues: time-consuming keyframe localization, static frames missing motion details, and incomplete video representation by sampled frames. addressing these, we introduce a tgb that incorporates both static and dynamic features for video-language understanding. ## methodology in the subsequent sections, we begin with a detailed formulation of the video-language understanding problem in section §3.1. next, in section §3.2, we outline the core components for efficient length extrapolation of our tgb. section §3.3 explains the process of jointly tuning tgb with pluggable mllms on new video-language datasets within our bootstrapping framework. the overall architecture of tgb is illustrated in figure 2 . ## experiments in this section, we utilize the tgb on 5 mllms, across encoder, encoder-decoder, and decoderonly three types of architectures. we demonstrate the effectiveness of our approach on three tasks: long-form videoqa and zero-shot open-domain videoqa (section 4.1), temporal question grounding on video (section 4.2). furthermore, we provide a detailed analysis to showcase the effectiveness of our framework in length extrapolation (fig. 1b ), the effectivness of different components (section 4.3), and compare its computational efficiency with other state-of-the-art models on a similar scale (section 4.4). ## conclusion in this work, we propose a pluggable framework tgb for long video-language understanding tasks, which comprises a tgb and a spatial prompt solver to combine spatial-temporal-language alignment and temporal grounding. experiments on long-form video question answering and temporal question grounding on video demonstrate a consistent improvement over various types of mllms. comprehensive analysis verifies the effectiveness, efficiency, and generality of our framework.