Mulan: A Multi-Level Alignment Model for Video Question Answering (2023)

Abstract: Video Question Answering (VideoQA) aims to answer questions about the visual content of a video. Current methods mainly focus on improving joint representations of video and text. However, these methods pay little attention to the fine-grained semantic interaction between video and text. In this paper, we propose Mulan: a Multi-Level Alignment Model for Video Question Answering, which establishes alignment between visual and textual modalities at the object-level, frame-level, and video-level. Specifically, for object-level alignment, we propose a mask-guided visual feature encoding method and a visual-guided text description method to learn fine-grained spatial information. For frame-level alignment, we introduce the use of visual features from individual frames, combined with a caption generator, to learn overall spatial information within the scene. For video-level alignment, we propose an expandable ordinal prompt for textual descriptions, combined with visual features, to learn temporal information. Experimental results show that our method outperforms the state-of-the-art methods, even when utilizing the smallest amount of extra visual-language pre-training data and a reduced number of trainable parameters.

## introduction video question answering (videoqa) is a task that focuses on answering questions related to the visual content of a video. it serves as a representative task that showcases the fusion of visual and linguistic modalities. it demands the ability to comprehend and integrate data from both modalities to learn complex spatiotemporal information. given the videos are a continuous sequence of images, the mainstream models @xcite @xcite @xcite object-level alignment ## conclusions in this paper, we introduce mulan: a multi-level alignment model for video question answering, which enhances temporal and spatial information learning through multi-level alignment. additionally, we conduct experiments on public videoqa datasets and achieve state-of-the-art results. finally, we provide ablation studies to demonstrate the complementary effects of multi-level alignment and the effectiveness of our method.