In this paper, we propose to employ the convolutional neural network (CNN) for learning to answer questions from the image. Our proposed CNN provides an endto-end framework for learning not only the image representation, the composition model for question, but also the intermodal interaction between the image and question, for the generation of answer. More specifically, the proposed model consists of three components: an image CNN to extract the image representation, one sentence CNN to encode the question, and one multimodal convolution layer to fuse the multimodal input of the image and question to obtain the joint representation for the classification in the space of candidate answer words. We demonstrate the efficacy of our proposed model on DAQUAR and COCO-QA datasets, two datasets recently created for the image question answering (QA), with performance substantially outperforming the state-of-the-arts.
Ma et al. in REF take an interesting approach and use three convolutional neural networks to represent not only the image, but also the question, and their common representation in a multimodal space.
Learning to Answer Questions From Image using Convolutional Neural Network
ArXiv
ArXiv
Computer Science