Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for V ideo QA (2022)

Abstract: We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA). Our model combines two original contributions”:” A multi-modal fast-learning feature fusion (FLF) block and a mechanism that uses self-attended language features to separately guide neural attention on both static and dynamic visual features extracted from individual video frames and short video clips. When trained from scratch, VLCN achieves competitive results with the state of the art on both MSVD-QA and MSRVTT-QA with 38.06% and 36.01% test accuracies, respectively. Through an ablation study, we further show that FLF improves generalization across different VideoQA datasets and performance for question types that are notoriously challenging in current datasets, such as long questions that require deeper reasoning as well as questions with rare answers.

## introduction video question answering (videoqa) has emerged as a challenging task at the intersection of natural language processing and computer vision. in contrast to image-based visual question answering @xcite @xcite , videoqa takes dynamic visual content (a video) as input @xcite @xcite . this poses new challenges given that generating correct answers requires models to analyze spatial, appearance-based features of individual video frames jointly with the temporal, motion-based dynamics across multiple frames @xcite . however, there still is a semantic gap between the visual and language channels @xcite @xcite that prior work has tried to close by leveraging external memory @xcite @xcite . while external memory allows models to cache sequential information and retrieve relevant multimodal content @xcite , latest models still suffer from decreased performance, for example on ambiguous questions that require deeper reasoning abilities. moreover, current deep neural models for videoqa are limited in that they only gradually learn during training. in contrast, human cognition leverages two different learning systems: a gradual and a fast-learning system @xcite .the interplay between the fast and gradual duel learning systems is essential for humans to learn new representations, hence to generalizing @xcite . current networks lack a similar fast-learning mechanism, which impedes their ability to efficiently reason and generalize to unseen data since the fast learning system acts as an encoder of new information which is then transferred to the gradual learning system for referencing and consolidating @xcite . to address these limitations, we propose the video language co-attention network (vlcn) -a novel memory-enhanced model for videoqa. vlcn implements a video language co-attention module that uses self-and guided-attention to align language features of the question with static and dynamic visual features extracted from videos. as such, the module offers complementary information that our network attends to, independently of each other, when visually grounding a question. furthermore, vlcn features a novel multimodal fast-learning fusion (flf) block that helps the model to deal with challenging questions that need deeper reasoning and understanding. inspired by the cognitive fast-learning system @xcite , we leverage the differentiable neural computer (dnc) to incorporate an external memory which the network learns how to use by freeing and reusing its memory slots. we seamlessly integrate our novel video language co-attention module as well as the fast-learning feature fusion approach in the recent transformerbased mcan network @xcite . we show that our model achieves competitive results with the state of the art on two challenging datasets -msvd-qa and msrvtt-qa. our results further show that our model performs better on ambiguous questions and can better reason not only about questions with rare answers but also longer questions that require a deeper understanding of both the question and the visual input. in addition, we show that flf facilitates generalization across different videoqa datasets via transfer learning. ## related work our work is related to previous works on 1) attention mechanisms in videoqa, and 2) memoryenhanced networks. attention mechanisms in videoqa. neural attention mechanisms have become the de-facto standard in machine comprehension tasks @xcite @xcite . in videoqa, attention mechanisms are particularly important given that the information necessary to generate correct answers is scattered across frames -many of which are redundant or even irrelevant to the question at hand @xcite . @xcite introduced the attributeaugmented attention network that learned temporally attended video representations according to semantic attributes. @xcite reported new state-of-the-art performance by applying questionguided attention over both the appearance and motion features of individual as well as multiple video frames. motivated by the challenge to capture long-range dependencies, @xcite used a transformer-based co-attention network to exploit the global dependencies of the text and the temporal dynamics of the videos. @xcite leveraged bert @xcite to obtain richer contextual feature representations over the question. more recently, @xcite proposed a two-stream multimodal video transformer based architecture (comvt) that jointly attends over words in text and visual objects and scenes to learn visualdialogue context. although comvt achieves stateof-the-art results on multiple downstream videoqa datasets, it requires a computationally-demanding pretraning stage on 1.2m instructional videos. these previous methods have used question features to guide attention over either frame or cliplevel visual features, and some applied self and co-attention to individual frames. our work, however, is the first to use self-attention on the question which then separately guides the attention over both individual video frames and clips. memory-enhanced networks. in parallel, other works have focused on augmenting models with external memory components to improve their reasoning capabilities particularly over long-range data that are common in many visiolinguistic tasks, e.g. images with many objects or videos with a large number of frames. one of the first methods introduced a memory component over simple facts for question answering @xcite . the introduction of end-to-end trainable models popularized the use of external memory components @xcite . driven by the insight that memory access is similar to neural attention @xcite , other works integrated attention mechanisms to allow networks to better interact with their external memory through read and write operations, such as the neural turing machine (ntm) @xcite or the differential neural computer (dnc) @xcite . the latter includes a dynamic memory allocation scheme that enables it to learn how to effectively free and reuse memory slots. several works aimed to leverage the potential of memory-enhanced networks for videoqa. @xcite applied memory over the video frames using multi-layered cnns read and write networks to capture richer temporal dynamics of frame-level sequence information. @xcite obtained syntax parse trees over questions and then stored these into memory, allowing their model to perform better on more complex questions. @xcite used one memory component to effectively learn global context information from appearance and motion features in combination with another question-memory to help understand the complex semantics of questions and highlight queried subjects. @xcite used a co-memory attention mechanism to generate attention from motion and appearance cues. more recently, @xcite achieved new state-of-the-art results on msvd-qa @xcite by using a dnc @xcite to encode the textual information of the question and the visual information of the video. while previous works used memory-enhanced networks to extract linguistic and visual features, we propose a memory-augmented block adapted from the dnc to potentially emulate the human-like fast-learning capabilites @xcite and use it to fuse multimodal features previously attended by an encoder-decoder transformer-based co-attention module instead. ## method we propose the video language co-attention network (vlcn) that integrates two original contributions (see figure 1 ): first, we propose to use self-and guided-attention to separately align the language features with the static and dynamic visual features extracted from single video frames and frame sequences (clips). second, we introduce fast-learning fusion (flf) -a novel memory-enhanced multimodal block that learns a single fused representation of all features (i.e. language, static and dynamic visual features). ## conclusion in this work, we proposed the video language co-attention network (vlcn) for videoqa. at its core are two distinct novel contributions: stacked co-attention layers in an encoder-decoder framework to separately guide self-attended language features over both static video frame and dynamic clip features; and fast-learning fusion (flf) -a memory-enhanced multimodal block to efficiently fuse the reduced features. we demonstrated that the combination of both results in significant improvements and competitive performance with state-of-the-art models on the challenging msvd-qa and msrvtt-qa datasets. we also demonstrated the particular advantage of our model in dealing with long questions that require deeper reasoning or questions with rare answers. finally, further experiments showed that our flf block allows our model to generalize better across different datasets via transfer learning.