Dual Capsule Attention Mask Network with Mutual Learning for Visual Question Answering (2022)

Abstract: A Visual Question Answering (VQA) model processes images and questions simultaneously with rich semantic information. The attention mechanism can highlight fine-grained features with critical information, thus ensuring that feature extraction emphasizes the objects related to the questions. However, unattended coarse-grained information is also essential for questions involving global elements. We believe that global coarse-grained information and local fine-grained information can complement each other to provide richer comprehensive information. In this paper, we propose a dual capsule attention mask network with mutual learning for VQA. Specifically, it contains two branches processing coarse-grained features and fine-grained features, respectively. We also design a novel stackable dual capsule attention module to fuse features and locate evidence. The two branches are combined to make final predictions for VQA. Experimental results show that our method outperforms the baselines in terms of VQA performance and interpretability and achieves new SOTA performance on the VQA-v2 dataset.

## introduction in recent years, visual question answering (vqa) has received extensive research attention in the fields of computer vision and multimedia computing. the goal of vqa is to answer questions related to the content of images correctly @xcite . it has a wide range of practice applications, such as helping people with visual impairment and human-computer q&a. in the early stage, most vqa models extract features from images and questions independently @xcite @xcite . these methods fail to capture the fine-grained key features and include much unnecessary information. afterward, the attention mechanism becomes popular and is introduced in many fields @xcite . the vqa models with the attention mechanism extract critical information from one modality guided by another modality @xcite @xcite . consider the question related to figure 1 (left) "what is on the little girl's head?" the attention method needs to encourage the model to focus on the "girls head" in the question and related regions in the image to produce the correct answer as "helmet." in this case, local fine-grained input(features which have critical information with attention processing) can help the model eliminate distractions and generate correct answers. however, the attention mechanism is not a panacea. some questions may mislead visual attention and lead to wrong answers. for example, in figure 6 (line 3, left), the question word "boxes" makes the model focus on the printer(which looks like a box) in the picture and leads to the wrong answer. also, in some scenarios, the model needs to focus on multiple objects for reasoning, but the question cannot explicitly remind which object require its attention. for example, consider the ques-tion related to figure 1 (right) "could this be a multi-purpose room?" the model needs to take the bed, chair, computer, and printer into account for reasoning, but there are no words in the question that can help identify the relevant objects. in these cases, global coarse-grained input(features that have all information without attention processing) can provide more comprehensive information for generating answers. the challenge is to make the model focus on key features while maintaining a reference to the global information. to overcome the above limitations, we propose a novel dual capsule attention mask network (dcamn) with mutual learning to process multimodal information at different granularities. we believe that coarse-grained information and finegrained information can complement each other to provide richer comprehensive information for answer reasoning. inspired by mutual learning and its variants @xcite , we design a two-branch network. the first branch processes the entire features of visual and language, analyzes global information at the coarsegrained level and fuses the features to produce predictions. we also design a stackable dual capsule attention module (sdcam) to model cross-modal deep interactions between the image and the question. the second branch masks visual features and language features with the attention weights generated by the sdcam of the first branch to get fine-grained features, which enables the network to focus on key regions of the image and keywords of the question. finally, we combine results from two branches to get final predictions. in contrast to other multi-granularity work @xcite , our dcamn does not introduce additional information such as predicates, while utilizes the attention weights from the first branch to filter features for the second branch. as a novel method of multi-modal fusion, sd-cam can output precise attention weights, which not only mask fine-grained features but also help locate evidence(grounds for the answer). by analyzing evidence, we can learn what information the network concerns more and how it makes decisions. also, the stacking strategy for sdcam improves attention accuracy and the fusion of visual and language representations. in dcamn, there is only one language module to encode the questions, while images are presented to the two branches for processing separately. compared with using two independent peer networks, sharing the language module between two branches can reduce the burden of parameters and calculation. moreover, early blocks acquire gradients from both branches during backpropagation, which reduces the risk of gradient vanishing @xcite . the knowledge of two branches at different perspectives and granularities is learned by another branch for information supplement and regularization, which can improve the generalization capability and vqa performance of dcamn. the main contributions of this paper are as follows: (1) a novel dual capsule attention mask network with mutual learning is proposed, which can process coarse-grained features and fine-grained features separately. two branches can learn from each other, and their combination can improve the vqa performance. (2) a stackable dual capsule attention module is proposed, which provides precise co-attention weights for masking out features and locating evidence. (3) we propose to share the language module between two branches of different granularities, which can reduce parameter requirements and the risk of gradient vanishing. (4) extensive experiments are conducted to evaluate the proposed method. our method has significant advantages over the baselines in terms of interpretability and accuracy and achieves state-ofthe-art performance on the vqa-v2 dataset. ## method given an image i and a question q, the purpose of vqa is to output the correct answer @xmath0, where a indicates the candidate word list for answers. we follow the transformer design of mcan @xcite and trar @xcite and use them as our backbones with their encoder-decoder units. the overall framework of dcamn is shown in figure 2 .