Multi-Scale Progressive Attention Network for Video Question Answering (2021)

Abstract: Understanding the multi-scale visual information in a video is essential for Video Question Answering (VideoQA). Therefore, we propose a novel Multi-Scale Progressive Attention Network (MSPAN) to achieve relational reasoning between cross-scale video information. We construct clips of different lengths to represent different scales of the video. Then, the clip-level features are aggregated into node features by using max-pool, and a graph is generated for each scale of clips. For cross-scale feature interaction, we design a message passing strategy between adjacent scale graphs, i.e., top-down scale interaction and bottom-up scale interaction. Under the question’s guidance of progressive attention, we realize the fusion of all-scale video features. Experimental evaluations on three benchmarks: TGIF-QA, MSVD-QA and MSRVTT-QA show our method has achieved state-of-the-art performance.

## introduction video question answering (videoqa) is a popular vision-language task, which focuses on predicting the correct answer to a given natural language question based on the corresponding video. videoqa task entails representing video features in both spatial and temporal dimensions. compared with the visual features of a picture in visual question answering, it requires a more complex attention. therefore, @xcite employed appearance features and motion features as video representation, and designed a dual-lstm network based on spatio-temporal attention to fuse visual and text information. next, memory networks are widely used to capture long-term dependencies. for example, @xcite applied feature augmented memory to strengthen the information augmentation of video and text. complex relational reasoning is important for videoqa task. consequently, a conditional relationship network @xcite question: who is cleaning in a kitchen while wearing gloves ? answer: woman time video: many methods complete this task from a certain aspect, however, none of them have a fine-grained understanding of video information. when looking for the answer in a question-based video, the video frames corresponding to different objects in the question are of different lengths. as shown in fig. 1 , when asked "who is cleaning in a kitchen while wearing gloves?", we need to find the keywords "cleaning", "a kitchen" and "wearing gloves" from different levels of clips. previous methods searched for the answer on the same level of clips in a video, leading to insufficient or redundant information. firstly, we construct clips of different lengths from the frame sequence, and regard the length of a clip as its scale information. then, multi-scale graphs are generated separately for clips of different scales. the nodes in the multi-scale graphs indicate video features corresponding to different clips. for implementing relational reasoning, the nodes in each scale graph are first updated by using graph convolution. most importantly, under the guidance of the question, progressive attention has been utilized to enable the fusion of multi-scale features during cross-scale graph interaction. in detail, each graph is gradually updated in top-down scale order, followed by updating each graph in bottom-up scale order. finally, node features of a graph are fused with question embedding, and a classifier is employed to find the answer. question: how many times does the cat lick? video representation question representation question embedding appearance net motion net 1 q d v n d max pool @xmath0: 7 times gcn gcn gcn fusion progressive attention multi-scale graphs cross-scale feature interaction 2 method an overview of the proposed mspan is shown in fig. 2 . the input is a short video and a question sentence, while the output is the produced answer. ## ablation studies to explore the potential of our network, ablation experiments are performed on tgif-qa dataset. default hyperparameters are: @xmath1. we study the effectiveness of our network in the next two aspects, as shown in ## conclusion we introduce a multi-scale learning method to achieve a fine-grained understanding of the video. compared with existing spatio-temporal attention, we use progressive attention to realize cross-scale feature interaction. the top-down and bottom-up structures we have designed are conducive to learning all-scale visual information of the video. for longer videos, we plan to use dilated max-pools with different strides to reduce the size of graphs. in general, we consider the videoqa task from the perspective of multi-scale information interaction, and the proposed network is instructive.