Encoding and Controlling Global Semantics for Long-form Video Question Answering (2024)

Abstract: Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets.

## introduction videoqa has been extensively studied to develop systems to assist humans in daily activities @xcite , e.g., remind users of their past actions, help users locate their belongings, and provide assistance with complex tasks. to implement these functions, we expect videoqa systems to understand and extract relevant information from long-form videos with diverse objects and complex spatial-temporal interactions. compared with short clips, long-form videos pose more challenges for videoqa. they consist * corresponding to: thong, e0998147@u.nus.edu of a higher number of objects and events. as such, comprehensively encoding information from them requires expensive computations. moreover, a high amount of information may be unrelated to the posed question. to address these problems, recent studies @xcite @xcite adaptively select a subset of video frames and visual regions associated with the question. nevertheless, if a question necessitates a reasoning of the entire sequence of events (e.g. video 1's figure 1 ), or an understanding of the overall video narration (e.g. video 2's figure 1 ), a mere handful of selected frames or regions might not sufficiently encapsulate necessary details. to tackle these problems, we introduce a state space layer (ssl). before forwarding video frames to selection modules, ssl fixes long-term dependency patterns for integrating global information into visual representations. such global information offers the selected frames the global context within the video, so that they can relate to other frames even though those frames are not selected for attention computation. however, a considerable amount of unrelated global information may flow into visual representations. therefore, we first equip ssl with a gating mechanism to provide more controllability over the flow of global semantics into visual representations, resulting in our gated state space multi-modal transformer (gsmt) architecture. furthermore, we promote global semantics that is more aligned with the question. in particular, we introduce cross-modal compositional congruence (c 3 ) objective that compares visual attention with its version transitioned to the language basis via cross-modal attention, effectively measuring cross-modal congruence between intra-modal relations. our rationale behind focusing on intra-modal relations is because videoqa models often need to understand spatial and temporal relationships between entities and events posed by the question @xcite , question: what central theme best describes activities throughout the video? answer: routine household management and maintenance video 1: 3037_iron_man2 -video length: 2.08 hours person's health deteriorates replace the ingot of palladium wearing the armored suit activate the rt unit question: what if the person did not have the ingot of palladium to replace the smoking one in the rt unit? answer: the rt unit would degrade, potentially causing the person's health to deteriorate or the armored suit to malfunction. figure 1 : long-form videoqa examples, with videos taken from mad @xcite and ego4d @xcite datasets, respectively. question in video 1 requires the model to reason about the relation chain of replacing ingot of palladium to activate the rt unit that powers the armored suit and protects person's health. question in video 2 necessitates an understanding of the overall theme in video 2. to sum up, our contributions are as follows: ## related work video question answering (videoqa). videoqa datasets @xcite @xcite @xcite mostly focus on short video clips about daily activities, e.g. sports or household work. recent works @xcite @xcite focus on complex spatialtemporal reasoning ability over longer temporal lengths with causal and transition questions. regarding methodology, early works @xcite present lstm-based and gnn-based architectures to capture cross-modal and motion-appearance interaction. for example, @xcite incorporate graph modeling into transformer to explicitly encode the object relations in videos. due to outstanding performance of pre-trained vision-language transformers, bain et al. (2021); fu et al. (2023); wang et al. (2023) utilize pre-trained transformer models on downstream videoqa tasks. long-form video modeling. to improve the applicability of vision-language systems, various works have focused on long-form video modeling for videoqa tasks. wu and krahenbuhl (2021) introduce short-term feature extraction technique and long-term memory mechanism to alleviate redundant video frame processing. lin et al. (2022b) propose to compensate sparsely extracted video frames with audio cues. islam and bertasius (2022) design structured multi-scale temporal decoder. gao et al. (2023) utilize question as a guide to select question-related visual segments to mitigate computation. inspired by @xcite in language modeling domains, we introduce a method to encode global semantics of videos to tackle information degradation problem in long-form video modeling. cross-modal alignment. alignment approaches, such as contrastive learning @xcite @xcite , optimal transport @xcite @xcite , tree-based fusion @xcite , and energy-based modeling @xcite , have been investigated to compare different input samples. for multimodal learning, recent cross-modal alignment methods focus on establishing a shared latent space in which samples of different modalities can be compared readily @xcite @xcite . for example, with a contrastive learning formulation, clip @xcite and align @xcite learn generalizable imagetext representations from millions of image-text pairs. to exploit relation for cross-modal alignment, @xcite use hard bijective correspondences between words and objects via an argmax over the cross-modal attention matrix to optimize the cross-modal alignment. @xcite generalize the hard alignment with their soft alignment approach, but they focus on objects within an image rather than a long video. ## methodology the formulation of video question answering (videoqa) is to predict the answer y for a question q about a video v as follows: ## ego-qa and mad-qa benchmarks to more rigorously evaluate long-form videoqa models, we construct two new datasets, i.e. ego-qa and mad-qa. ## conclusion we introduce a gated state space multi-modal transformer (gsmt) with a state space layer (ssl) to integrate global semantics of video into visual representations to tackle long-form videoqa. we further incorporate a gating unit to provide more controllability over the integrated global semantics and a cross-modal compositional congruence (c 3 ) objective to encourage the semantics aligned with the question. to comprehensively evaluate longform videoqa, we curate two long-form videoqa datasets with excessively long video lengths and long-natured questions. extensive experiments on these and standard datasets validate the superiority of our framework.