MMFT-BERT : M ultimodal F usion T ransformer with B ERT E ncodings for V isual Q uestion A nswering (2020)

Abstract: We present MMFT-BERT(MultiModal FusionTransformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an isolated diagnostic subset of TVQA, which strictly requires the knowledge of visual (V) modality based on a human annotator’s judgment. This set of questions helps us to study the model’s behavior and the challenges TVQA poses to prevent the achievement of super human performance. Extensive experiments show the effectiveness and superiority of our method.

## introduction in the real world, acquiring knowledge requires processing multiple information sources such as visual, sound, and natural language individually and collectively. as humans, we can capture experience from each of these sources (like an isolated sound); however, we acquire the maximum knowledge when exposed to all sources concurrently. thus, it is crucial for an ideal artificial intelligence (ai) system to process modalities individually and jointly. one of the ways to understand and communicate with the world around us is by observing the environment and using language (dialogue) to interact with it @xcite . a smart we treat input modalities as a sequence. [fuse] is a trainable vector; h q 0 j , h v 0 j , and h s 0 j are fixed-length features aggregated over question-answer (qa) pairs, visual concepts, and subtitles. using a transformer encoder block, @xcite attends all source vectors and assigns weights based on the importance of each input source. training end to end for vqa enables the mmft module to learn to aggregate input sources w.r.t. the nature of the question. for illustration purposes, we show that for a single head, mmft collects more knowledge from the visual source h v 0 j (green colored) than from the qa and subtitles. best viewed in color. tvqa @xcite , a video-based ques-tion answering dataset, is challenging as it provides more realistic multimodal question answers (qa) compared to other existing datasets . to answer tvqa questions, the system needs an understanding of both visual cues and language. in contrast, some datasets are focused either visually: moviefib @xcite , video context qa @xcite , tgif-qa @xcite ; or by language: movieqa @xcite ; or based on synthetic environments: mari-oqa @xcite and pororoqa @xcite . we choose tvqa because of its challenges. the introduction of transformers @xcite has advanced research in visual question answering and shows promise in the field of language and vision in general. here, we adopt the pretrained language-based transformer model, bert @xcite to solve the vqa task. the human brain has vast capabilities and probably conducts processing concurrently. like humans, an intelligent agent should also be able to process each input modality individually and collectively as needed. our method starts with independent processing of modalities and the joint understanding happens at a later stage. therefore, our method is one step forward toward better joint understanding of multiple modalities. we use separate bert encoders to process each of the input modalities namely q-bert, v-bert and s-bert to process question (q), video (v), and subtitles (s) respectively. each bert encoder takes an input source with question and candidate answer paired together. this is important because we want each encoder to answer the questions targeted at its individual source input. thus, pairing up the question and candidate answers enables each stream to attend to the relevant knowledge pertinent to the question by using a multi-head attention mechanism between question words and a source modality. we then use a novel transformer based fusion mechanism to jointly attend to aggregated knowledge from each input source, learning to obtain a joint encoding. in a sense, our approach is using two levels of question-to-input attention: first, inside each bert encoder to select only relevant input; and second, at the fusion level, in order to fuse all sources to answer the common question. we show in our experiments that using q-bert, a separate bert encoder for question and answer is helpful. our contribution is three-fold: first, we propose a novel multi-stream end-to-end trainable architecture which processes each input source separately followed by feature fusion over aggregated source features. instead of combining input sources before input to bert, we propose to process them individually and define an objective function to optimize multiple berts jointly. our approach achieves state-of-the-art results on the video-based question answering task. second, we propose a novel multimodal fusion transformer (mmft) module, repurposing transformers for fusion among multiple modalities. to the best of our knowledge, we are the first to use transformers for fusion. third, we isolate a subset of visual questions, called tvqa-visual (questions which require only visual information to answer them). studying our method's behavior on this small subset illustrates the role each input stream is playing in improving the overall performance. we also present detailed analysis on this subset. ## related work image-based question answering. image-based vqa @xcite @xcite @xcite has shown great progress recently. a key ingredient is attention @xcite @xcite @xcite . image based vqa can be divided based on the objectives such as generic vqa on real world images @xcite , asking binary visual questions @xcite and reasoning based vqa collecting visual information recurrently @xcite @xcite @xcite to answer the question both in synthetic @xcite @xcite as well as real image datasets @xcite . video-based question answering. video-based qa is more challenging as it requires spatiotemporal reasoning to answer the question. @xcite introduced a video-based qa dataset along with a two-stream model processing both video and subtitles to pick the correct answer among candidate answers. some studies are: grounding of spatiotemporal features to answer questions @xcite ; a video fill in the blank version of vqa @xcite ; other examples include @xcite @xcite . representation learning. bert has demonstrated effective representation learning using selfsupervised tasks such as masked language modeling and next sentence prediction tasks. the pretrained model can then be finetuned for a variety of supervised tasks. qa is one such task. a singlestream approach takes visual input and text into a bert-like transformer-based encoder; examples are: visualbert @xcite , vl-bert @xcite , unicoder-vl @xcite and b2t2 @xcite . two-stream approaches need an additional fusion step; vilbert @xcite and lxmert @xcite employ two modality-specific streams for images. we take this a step further by employing three streams. we use a separate bert encoder for the question-answer pair. we are specifically targeting video qa and do not need any additional pre-training except using pre-trained bert. ## approach our approach permits each stream to take care of the questions requiring only that input modality. as an embodiment of this idea, we introduce the mul-timodal fusion transformer with bert encodings (mmft-bert) to solve vqa in videos. see fig. 1 for the proposed mmft module and fig. 2 for illustration of our full architecture. ## dataset in tvqa, each question (q) has 5 answer choices. it consists of 152k qa pairs with 21.8k video clips. each question-answer pair has been provided with the localized video v to answer the question q, i.e., start and end timestamps are annotated. subtitles s have also been provided for each video clip. see supplementary work for a few examples. ## conclusion our method for vqa uses multiple bert encodings to process each input type separately with a novel fusion mechanism to merge them together. we repurpose transformers for using attention between different input sources and aggregating the information relevant to the question being asked. our method outperforms state-of-the-art methods by an absolute ∼2.41% on q+v and ∼1.1% on q+v+s on tvqa validation set. our proposed fusion lays the groundwork to rethink transformers for fusion of multimodal data in the feature dimension. linchao zhu, zhongwen xu, yi yang, and alexander g hauptmann. 2017. uncovering the temporal context for video question answering. international journal of computer vision, 124(3):409-421. yuke zhu, oliver groth, michael bernstein, and li fei-fei. 2016. visual7w: grounded question answering in images. in proceedings of the ieee conference on computer vision and pattern recognition, pages 4995-5004.