Video Question Answering is a task which requires an AI agent to answer questions grounded in video . This task entails three key challenges : ( 1 ) understand the intention of various questions , ( 2 ) capturing various elements of the input video ( e.g. , object , action , causality ) , and ( 3 ) cross-modal grounding between language and vision information . We propose Motion-Appearance Synergistic Networks ( MASN ) , which embed two crossmodal features grounded on motion and appearance information and selectively utilize them depending on the question 's intentions . MASN consists of a motion module , an appearance module , and a motion-appearance fusion module . The motion module computes the action-oriented cross-modal joint representations , while the appearance module focuses on the appearance aspect of the input video . Finally , the motion-appearance fusion module takes each output of the motion module and the appearance module as input , and performs question-guided fusion . As a result , MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets . We also conduct qualitative analysis by visualizing the inference results of MASN . The code is available at https://github.com/ ahjeongseo / MASN-pytorch . Recently , research in natural language processing and computer vision has made significant progress in artificial intelligence ( AI ) . Thanks to this , visionlanguage tasks such as image captioning ( Xu et al . , 2015 ) , visual question answering ( VQA ) ( Antol et al . , 2015 ; Goyal et al . , 2017 ) , and visual commonsense reasoning ( VCR ) ( Zellers et al . , 2019 ) have been introduced to the research community , along with some benchmark datasets . In particular , video question answering ( video QA ) tasks ( Xu et al . , 2016 ; Jang et al . , 2017 ; Lei et al . , 2018 ; Yu et al . , 2019 ; Choi et al . , 2020 ) have been proposed with the goal of reasoning over higher-level visionlanguage interactions . In contrast to QA tasks based on static images , the questions presented in the video QA dataset vary from frame-level questions regarding the appearance of objects ( e.g. , what is the color of the hat ? ) to questions regarding action and causality ( e.g. , what does the man do after opening a door ? ) . There are three crucial challenges in video QA : ( 1 ) understand the intention of various questions , ( 2 ) capturing various elements of the input video ( e.g. , object , action , and causality ) , and ( 3 ) crossmodal grounding between language and vision information . To tackle these challenges , previous studies ( Li et al . , 2019 ; Jiang et al . , 2020 ; Huang et al . , 2020 ) have mainly explored this task by jointly embedding the features from the pre-trained word embedding model ( Pennington et al . , 2014 ) and the object detection models ( He et al . , 2016 ; Ren et al . , 2016 ) . However , as discussed in ( Gao et al . , 2018 ) , the use of the visual features extracted from the object detection models suffers from motion analysis since the object detection model lacks temporal modeling . To enforce the motion analysis , a few approaches ( Xu et al . , 2017 ; Gao et al . , 2018 ) have employed additional visual features ( Tran et al . , 2015 ) ( i.e. , motion features ) which were widely used in the action recognition domain , but their reasoning capability is still limited . They typically employed recurrent models ( e.g. , LSTM ) to embed a long sequence of the visual features . Due to the problem of long-term dependency in recurrent models ( Bengio et al . , 1993 ) , their proposed methods may fail to learn dependencies between distant features . In this paper , we propose Motion-Appearance The output from the fusion module is used to derive answers . For question features , the word-level representation F Q is integrated with the visual features in the VQ interaction submodule . The last hidden units q from the bi-LSTM are used to combine appearance and motion features . Synergistic Networks ( MASN ) for video question answering which consist of three kinds of modules : the motion module , the appearance module , and the motion-appearance fusion module . As shown in Figure 1 , the motion module and the appearance module aim to embed rich cross-modal representations . These two modules have the same architecture except that the motion module takes the motion features extracted from I3D as visual features and the appearance module utilizes the appearance features extracted from ResNet . Each of these modules first constructs the object graphs via graph convolutional networks ( GCN ) to compute the relationships among objects in each visual feature . Then , the vision-question interaction module performs cross-modal grounding between the output of the GCNs and the question features . The motion module and the appearance module each yield cross-modal representations of the motion and the appearance aspects of the input video respectively . The motion-appearance fusion module finally integrates these two features based on the question features . The main contributions of our paper are as follows . First , we propose Motion-Appearance Synergistic Networks ( MASN ) for video question answering based on three modules , the motion module , the appearance module , and the motionappearance fusion module . Second , we validate MASN on the large-scale video question answering datasets TGIF-QA , MSVD-QA , and MSRVTT-QA . MASN achieves the new state-of-the-art performance on TGIF-QA and MSVD-QA . We perform ablation studies to validate the effectiveness of our proposed methods . Finally , we conduct a qualitative analysis of MASN by visualizing inference results . In this paper , we proposed a Motion-Appearance Synergistic Networks to fuse and create a synergy between motion and appearance features . Through the Motion and Appearance modules , MASN manages to find motion and appearance clues to solve the question , while modulating the amount of information used of each type through the Fusion module . Experimental results on three benchmark datasets show the effectiveness of our proposed MASN architecture compared to other models .

Challenge: Existing methods for Video Question Answering lack temporal modelling, reasoning capability, or an ability to learn dependencies between distant features.
Approach: They propose a motion module and an appearance module which embeds cross-modal representations and a motion-appearance fusion module to integrate them based on question features.
Outcome: The proposed method achieves the new state-of-the-art performance on TGIF-QA and MSVD-QA datasets and they also present qualitative analysis with visualizations.