Enhancing Temporal Modeling of Video LLM s via Time Gating (2024)

Abstract: Video Large Language Models (Video LLMs) have achieved impressive performance on video-and-language tasks, such as video question answering. However, most existing Video LLMs neglect temporal information in video data, leading to struggles with temporal-aware video understanding. To address this gap, we propose a Time Gating Video LLM (TG-Vid) designed to enhance temporal modeling through a novel Time Gating module (TG). The TG module employs a time gating mechanism on its sub-modules, comprising gating spatial attention, gating temporal attention, and gating MLP. This architecture enables our model to achieve a robust understanding of temporal information within videos. Extensive evaluation of temporal-sensitive video benchmarks (i.e., MVBench, TempCompass, and NExT-QA) demonstrates that our TG-Vid model significantly outperforms the existing Video LLMs. Further, comprehensive ablation studies validate that the performance gains are attributed to the designs of our TG module. Our code is available at https://github.com/LaVi-Lab/TG-Vid.

## introduction the advancement of large language models (llms) @xcite has greatly fueled multi-modal research, such as image llms @xcite @xcite which have achieved success on image-and-language downstream tasks @xcite . inspired by image llms, many recent efforts manage to empower llms to understand video data @xcite @xcite . the typical architecture of these video llms comprises a pretrained vision encoder @xcite , a pretrained llm @xcite , and a connection module in between @xcite . despite the impressive performance demonstrated by video llms @xcite @xcite , a recent study @xcite reveals that most video llms perform comparably to, or even worse than, image llms. this discrepancy arises because existing video benchmarks can often be adequately addressed by single-frame bias @xcite , without the need for capturing the temporal dynamics of videos. to better evaluate the temporal modeling capability, multiple temporal-sensitive benchmarks have been developed @xcite @xcite that cannot be solved by simply relying on single-frame bias as a shortcut. in this paper, we aim to enhance the temporal modeling ability of video llms and evaluate our model on the temporal-sensitive benchmarks. specifically, we propose a temporal-aware video llm (tg-vid) in this work, featuring a novel time gating module (tg) to enhance temporal modeling. this tg module comprises three submodules, gating spatial attention, gating temporal attention, and gating mlp, simultaneously capturing spatial and temporal information. a recent relevant work st-llm @xcite also tries to enhance temporal modeling, by directly utilizing bt-adapter @xcite which applies spatio-temporal attention in parallel to the vision encoder. in contrast, our work builds gating spatiotemporal attention on top of the vision encoder, and our gating mechanism imposes effective modulespecific control over each sub-module of the tg module. as validated by experiments, our design achieves better performance on temporal-sensitive benchmarks. we conduct comprehensive experiments on three temporal-sensitive video benchmarks (i.e., mvbench @xcite , tempcompass (liu large language model instruction: approach the video by thinking about the reasons behind the actions and their order in time, and choose the most relevant option. question: what did the brown dog do when the black dog first ran away? options: (a) chase the dog. (b) fight back. (c) mount the black dog. (d) touch its face. (e) bite and push. text input "a" response qformer … t gating mlp gating temporal attention gating spatial attention tg … … … … … … … … … … text input flatten n × @xmath0, l, d) tg vision encoder … … … frozen trainable video patch embeddings @xmath1, lq, d) @xmath2, l, d) l video patch embeddings figure 1: model architecture of tg-vid. given a video with t frames, the vision encoder extracts t frame-level embeddings. our tg employs a novel time gating mechanism to enhance video temporal modeling, thereby enhancing the frame-level video modeling ability of the qformer. moving forward, the qformer compresses each frame-level video embedding from l patch tokens to l q query tokens, followed by llms. et al., 2024e) and next-qa (xiao et al., 2021)). the results show that our tg-vid significantly outperforms the existing video llms across all benchmarks and demonstrate the effectiveness of our tg-vid on temporal-aware video understanding. the thorough ablation studies further emphasize that the performance gains are attributed to the designs of our tg module. ## related work video large language models. benefited from the reasoning power of large language models (llms) @xcite @xcite @xcite , video llms @xcite @xcite @xcite @xcite have shown impressive performance on video-and-language tasks, such as video question answering @xcite @xcite @xcite . however, most existing video llms inherit the design of image llms @xcite @xcite and overlook the temporal modeling that is critical for video data, leading to unsatisfactory capability on temporalaware video understanding @xcite . for example, tempcompass @xcite reveals that the temporal understanding ability of most video llms is on par with or even weaker than image llms. in this work, we propose a temporal-aware video llm, featuring a new architecture of time gating module to enhance video temporal modeling. ## methodology in this section, we introduce our time gating video llm (tg-vid). fig. 1 provides an overview of our model architecture. to enhance the temporal modeling of a video llm (comprising an llm, a vision encoder, and a connection module), we propose a time gating (tg) module with a novel module-specific time gating mechanism. ## experiments compared with existing video llms, we evaluate our tg-vid on three temporal-sensitive video understanding benchmarks (i.e., mvbench @xcite , tempcompass @xcite , and next-qa @xcite ). more details of datasets, implementation, experiment results and visualization are provided in the appendix.