Rethinking Multi-Modal Alignment in Multi-Choice V ideo QA from Feature and Sample Perspectives (2022)

Abstract: Reasoning about causal and temporal event relations in videos is a new destination of Video Question Answering (VideoQA). The major stumbling block to achieve this purpose is the semantic gap between language and video since they are at different levels of abstraction. Existing efforts mainly focus on designing sophisticated architectures while utilizing frame- or object-level visual representations. In this paper, we reconsider the multi-modal alignment problem in VideoQA from feature and sample perspectives to achieve better performance. From the view of feature, we break down the video into trajectories and first leverage trajectory feature in VideoQA to enhance the alignment between two modalities. Moreover, we adopt a heterogeneous graph architecture and design a hierarchical framework to align both trajectory-level and frame-level visual feature with language feature. In addition, we found that VideoQA models are largely dependent on languagepriors and always neglect visual-language interactions. Thus, two effective yet portable training augmentation strategies are designed to strengthen the cross-modal correspondence ability of our model from the view of sample. Extensive results show that our method outperforms all the state-of the-art models on the challenging NExT-QA benchmark.

## introduction given a video and a question about its content, video question answering (videoqa) aims to answer the question through multi-modal reasoning. since it can benefit numerous multi-modal applications such as video retrieval @xcite and interaction with robot vision, videoqa has received increasing attention in recent years. compared to image question answering (imageqa) @xcite , videoqa requires more complex reasoning. a naive extension of imageqa methods @xcite figure 1 : an illustration of videoqa. existing videoqa models utilize objects as finer clues. we further track the objects with same classes across time and first leverage video trajectory for better alignment. as computing systems are more frequently intervening to improve people's work and daily lives, it is critical for machines to correctly comprehend the complex relationships between events such as causal and temporal relations. however, conventional videoqa benchmarks @xcite are constituted by descriptive questions, e.g., asking the number, color or single action of the video elements. recently, a benchmark called next-qa @xcite ) was elaborately designed to challenge videoqa models to reason about causal and temporal actions and understand the rich object interactions in daily activities. instead of a fixed answer set, next-qa requires the models to pick the correct answer out of five candidates. since this multi-choice setting allows concatenating question-answer pair to form a holistic language query and aligning it with videos, it can be easily generalized to any other multi-modal tasks besides question answering. multi-modal alignment, namely finding the correspondences between two modalities, is the foundation of multi-modal tasks. one of the major challenges to align two modalities is to fill in the semantic gap between language and video due to different levels of abstraction. intuitively, the key to better alignment is to identify the direct relations between sub-elements (e.g., words, frames, etc.) of instances from two different modalities. as a highly abstract vehicle of expression, almost every word in a natural language sentence can express a complete meaning, e.g., an entity, an action or a time. in contrast, a video also contains a number of visual concepts but they are naturally indivisible unlike the word. primitive videoqa models @xcite utilize frame-level appearance feature as video representation and align it with language feature. these methods simply regard the video as a series of frames. to better capture dynamic change in videos, others @xcite @xcite ) incorporate frame-level motion feature together with appearance feature. recently, object feature as more granular information has been leveraged to strengthen the semantic correspondence ability @xcite @xcite . although bringing in object information results in a better ability to reason about complex interactions in videos, there are still two unsolved problems: (1) static object information is hard to model temporal-related relations. (2) same object may take different actions during time and different objects with the same label can cause a mismatch. as shown in figure 1 , there are two boys and two balls in the video and both boys hold a ball. in this case, videoqa models that only use object-level information may end up with misalignment, which leads to a wrong answer. therefore, tracking objects across time is of vital importance. in this paper, we explore the multi-modal alignment in videoqa from feature and sample perspectives to better reason over videos' causal/temporal actions. from feature perspective, we decouple trajectories as salient entities from video and first leverage video trajectory feature in videoqa. in order to model the rich interactions between trajectories, we propose a trajectory encoder using multi-head self-attention with temporal and semantic embeddings. video trajectories are the essential ingredient for video relation detection task @xcite , which require track-ing the same object from different frames along the temporal axis. specifically, we first apply a pretrained object detector to obtain bounding boxes. then, an association algorithm named improved sequence nms @xcite ) is applied to obtain video trajectories that contain spatial-temporal information of the visual elements. we further align trajectory-level and frame-level feature with language feature by a cycle-attention module and adopt a heterogeneous graph architecture for implicit relation reasoning. in addition, from sample perspective, we design two training strategies in order to enhance multimodal alignment in feature space. to be specific, we first increase negative candidate answers when computing the matching score. this strategy forces the model to focus on the discriminative regions within a question-answer pair. we then add negative question-answer pairs that are attached to other videos. by doing so, the video and its affiliated language are drawn closer in feature space and the mismatched pairs are pulled away. moreover, we found that these strategies can also solve the problem that videoqa models are largely dependent on language priors and neglect visual-language interactions. together with the proposed model, our method achieved state-of-the-art performance. in summary, the main contributions of our work are listed as follows: (1) we first leverage video trajectory features in videoqa to capture richer causal and temporal relations in the video. (2) we design two training strategies to strengthen the cross-modal correspondence ability of our model and further boost the performance. (3) we conducted extensive experiments on next-qa and the results demonstrate the effectiveness of our model. ## approach formally, suppose we have a video @xmath0. meanwhile, we have a natural language question @xmath1, where w l denotes the l-th word in the sentence and l represents the question length. videoqa aims to predict the correct answer a p to the question according to the relevant video content. in the multi-choice setting, the goal is to choose the correct answer a p from n candidate answer set in this section, we sequentially introduce each component of our proposed model. the video and language encoding procedures are presented in section 3.1. the alignment and reasoning modules are introduced in section 3.2. the answer predictor is introduced in section 3.3. in section 3.4, we introduce two sample augmentation strategies. ## experiments 4.1 experimental details dataset. next-qa @xcite ) is a recently designed challenging videoqa benchmark which advances video question answering from describing to reasoning. the dataset contains 5,440 videos where 3870 for training, 570 for validation and 1,000 for testing. the videos are selected from the relation dataset vidor @xcite which contains natural videos of daily life such as outdoor activities and social scenes. thus they are richer in objects and interactions. it consists of 47,692 questions where 34,132, 4,996 and 8,564 for training, validation and testing, respectively. almost half of the questions are causal questions which contain questions starting with "why" and "how", which is a great challenge for videoqa models to reason about causality. temporal questions of inferring temporal actions compose 29% of the dataset. apart from causal and temporal questions, others are descriptive questions that focus on describing attributes, location and main events in videos. for multi-choice task that is to select one out of the five candidate answers, next-qa sampled four qualified candidates as distracting answers for each question to enhance the hard negatives. in a word, next-qa goes beyond descriptive qa to benchmark causal and temporal action reasoning in realistic videos and is also rich in object interactions. in addition, several recent state-of-theart methods are examined on it. evaluation metric. we report the accuracy of our model in all experiments which represents the percentage of correctly answered questions. implementation details. for the training process, we set the number of hidden units d to 256. the batch size is set to 64 and adam optimizer is used for optimization. the learning rate is set to 0.00005 for glove setting and 0.0001 for bert-ft setting, respectively. for better performance, we reduce the learning rate when a metric has stopped improving. the dropout rate is set to 0.3. we set balance factors λ 1 and λ 2 to 0.5 for all the experiments. we randomly sample 5 negative samples from the training set for each strategy. we utilize both ## conclusion in this paper, we explored multi-modal alignment in videoqa from feature and sample perspectives. from the view of feature, we first leverage video trajectory features in videoqa to bridge the semantic gap between the sub-components of the video and the language. moreover, in order to better utilize the trajectory feature, we propose a graph-based model which is capable of alignment and reasoning over heterogeneous representations. from the view of sample, we propose two sample augmentation strategies to further enhance the cross-modal correspondence ability of our model. the promising results on challenging next-qa dataset have exhibited the causal and temporal reasoning ability of our method. in the future, we will further explore a better way to take advantage of trajectory information considering its significant potential.