Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering (2021)

Abstract: Video Question Answering is a task which requires an AI agent to answer questions grounded in video. This task entails three key challenges: (1) understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, causality), and (3) cross-modal grounding between language and vision information. We propose Motion-Appearance Synergistic Networks (MASN), which embed two cross-modal features grounded on motion and appearance information and selectively utilize them depending on the question’s intentions. MASN consists of a motion module, an appearance module, and a motion-appearance fusion module. The motion module computes the action-oriented cross-modal joint representations, while the appearance module focuses on the appearance aspect of the input video. Finally, the motion-appearance fusion module takes each output of the motion module and the appearance module as input, and performs question-guided fusion. As a result, MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets. We also conduct qualitative analysis by visualizing the inference results of MASN.

## introduction recently, research in natural language processing and computer vision has made significant progress in artificial intelligence (ai). thanks to this, visionlanguage tasks such as image captioning @xcite , visual question answering (vqa) @xcite , and visual commonsense reasoning (vcr) @xcite have been introduced to the research community, along with some benchmark datasets. in particular, video question answering (video qa) tasks @xcite @xcite @xcite have been proposed with the goal of reasoning over higher-level visionlanguage interactions. in contrast to qa tasks based on static images, the questions presented in the video qa dataset vary from frame-level questions regarding the appearance of objects (e.g., what is the color of the hat?) to questions regarding action and causality (e.g., what does the man do after opening a door?). there are three crucial challenges in video qa: (1) understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, and causality), and (3) crossmodal grounding between language and vision information. to tackle these challenges, previous studies @xcite @xcite have mainly explored this task by jointly embedding the features from the pre-trained word embedding model @xcite and the object detection models @xcite . however, as discussed in @xcite , the use of the visual features extracted from the object detection models suffers from motion analysis since the object detection model lacks temporal modeling. to enforce the motion analysis, a few approaches @xcite have employed additional visual features @xcite (i.e., motion features) which were widely used in the action recognition domain, but their reasoning capability is still limited. they typically employed recurrent models (e.g., lstm) to embed a long sequence of the visual features. due to the problem of long-term dependency in recurrent models @xcite , their proposed methods may fail to learn dependencies between distant features. in this paper, we propose motion-appearance synergistic networks (masn) for video question answering which consist of three kinds of modules: the motion module, the appearance module, and the motion-appearance fusion module. as shown in figure 1 , the motion module and the appearance module aim to embed rich cross-modal representations. these two modules have the same architecture except that the motion module takes the motion features extracted from i3d as visual features and the appearance module utilizes the appearance features extracted from resnet. each of these modules first constructs the object graphs via graph convolutional networks (gcn) to compute the relationships among objects in each visual feature. then, the vision-question interaction module performs cross-modal grounding between the output of the gcns and the question features. the motion module and the appearance module each yield cross-modal representations of the motion and the appearance aspects of the input video respectively. the motion-appearance fusion module finally integrates these two features based on the question features. the main contributions of our paper are as follows. first, we propose motion-appearance synergistic networks (masn) for video question answering based on three modules, the motion module, the appearance module, and the motionappearance fusion module. second, we validate masn on the large-scale video question answering datasets tgif-qa, msvd-qa, and msrvtt-qa. masn achieves the new state-of-the-art performance on tgif-qa and msvd-qa. we perform ablation studies to validate the effectiveness of our proposed methods. finally, we conduct a qualitative analysis of masn by visualizing inference results. ## related work visual question answering (vqa) is a task that requires both understanding questions and finding clues from visual information. vqa can be classified into two categories based on the type of the visual source: image qa and video qa. in image qa, earlier works approach the task by applying attention between the question and the spatial dimensions of the image @xcite @xcite . in video qa, since a video is represented as a sequence of images over time, recognizing the movement of objects or causality in the temporal dimension should also be considered along with the details from the spatial dimension @xcite . there have been some attempts @xcite @xcite to extract motion and appearance features and integrate them on a spatio-temporal dimension via memory networks. @xcite , @xcite , @xcite proposed better performing models using attention in order to overcome the long-range dependency problem in memory networks. however, they do not represent motion in-formation sufficiently since they only use features pre-trained on image or object classification. to better address this, we model spatio-temporal reasoning on multiple visual information (i.e., resnet, i3d) while also solving the long-range dependency problem that occurred in previous studies. action classification is a task of recognizing actions, which are composed of interactions between actors and objects. therefore, this task has much in common with video qa, in that the model should perform spatio-temporal reasoning. for better spatio-temporal reasoning, @xcite introduced c3d, which extends the 2d cnn filters to the temporal dimension. @xcite proposed i3d, which integrates 3d convolutions into a state-of-the-art 2d cnn architecture, which now acts as a baseline in action classification tasks @xcite . @xcite introduced slowfast, a network which encodes images in two streams with different frame rates and temporal resolutions of convolution. this study based on a two-stream architecture inspired us in terms of assigning different inputs to each encoder module. however, our method differs from the former studies in two aspects: (1) we utilize language features as well as vision features, and (2) we expand the two-stream structure to solve more than motion-oriented tasks. attention mechanism explicitly calculates the correlation between two features @xcite , and has been widely used in a variety of fields. for machine translation, the transformer architecture first introduced by @xcite , utilizes multi-head selfattention that captures diverse aspects in the input features @xcite . for video qa, @xcite ; @xcite use self and guidedattention to encode temporal dynamics in video and ground them in the question. for multi-modal alignment, @xcite apply the transformer to merge cross-modal time series between vision, language, and audio features. we utilize the attention mechanism to capture various relations between appearance and motion and to aggregate them. ## model in this section, we introduce a detailed description of our masn network. first, we explain how to obtain appearance and motion features in section 3.1. then, we describe the appearance and motion modules, which encode visual features and com-bine them with the question in section 3.2. finally, the motion-appearance fusion module modulates the amount of motion and appearance information utilized and integrates them based on question context. ## experiments in this section, we evaluate our proposed model on three video qa datasets: tgif-qa, msvd-qa, and msrvtt-qa. we first introduce each dataset and compare our results with the state-of-the-art methods. then, we report ablation studies and include visualizations to show how each module in masn works. ## conclusion in this paper, we proposed a motion-appearance synergistic networks to fuse and create a synergy between motion and appearance features. through the motion and appearance modules, masn manages to find motion and appearance clues to solve the question, while modulating the amount of information used of each type through the fusion module. experimental results on three benchmark datasets show the effectiveness of our proposed masn architecture compared to other models.