We propose a novel attention based deep learning architecture for visual question answering task (VQA). Given an image and an image-related question, VQA returns a natural language answer. Since different questions inquire about the attributes of different image regions, generating correct answers requires the model to have questionguided attention, i.e., the attention on the regions corresponding to the input question's intent. We introduce an attention-based configurable convolutional neural network (ABC-CNN) to locate the question-guided attention based on input queries. ABC-CNN determines the attention regions by finding the corresponding visual features in the visual feature maps with a "configurable convolution" operation. With the help of the question-guided attention, ABC-CNN can achieve both higher VQA accuracy and better understanding of the visual question answering process. We evaluate the ABC-CNN architecture on three benchmark VQA datasets: Toronto COCO-QA, DAQUAR, and VQA dataset. ABC-CNN model achieves significant improvements over state-of-the-art methods. The question-guided attention generated by ABC-CNN is also shown to be the regions that are highly relevant to the questions' intents.
Question-guided attention mechanism REF uses semantic representation of a question as query to search for the regions in an image that are related to the answer.
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
ArXiv
ArXiv
Computer Science