Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multimodal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multimodal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a 'co-attention' mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-theart performance on the real-world VQA dataset. Code available at https://github.com/yuzcccc/mfb. 4 ZKDW FRORU DUH WKH FDWV H\HV $ \HOORZ 3 \HOORZ ZKDW FRORU DUH WKH FDWV H\HV " 4 ZKDW FRORU LV WKH FDWFKHUV SDQWV $ EODFN 3 ZKLWH ZKDW FRORU LV WKH FDWFKHUV SDQWV 4 KRZ PDQ\ ELUGV DUH IO\LQJ $ 3 KRZ PDQ\ ELUGV DUH IO\LQJ 4 KRZ PDQ\ IODJV DUH VKRZQ $ 3 KRZ PDQ\ IODJV DUH VKRZQ LV WKH ER\ RQ WKH ERWWRP SOD\LQJ OHIW KDQGHG 4 LV WKH PDQ VPLOLQJ $ \HV 3 \HV LV WKH PDQ VPLOLQJ 4 LV WKH ER\ RQ WKH ERWWRP SOD\LQJ OHIW KDQGHG $ \HV 3 QR 4 ZKDW LV RQ WKH IORRU $ FDW 3 FDW ZKDW LV RQ WKH IORRU 4 ZKDW DUH WKH UHG WKLQJV $ PHDW 3 WRPDWRHV ZKDW DUH WKH UHG WKLQJV
Whereas the work in REF proposes a bilinear pooling method based on matrix factorization to combine multi-modal features.
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering
2017 IEEE International Conference on Computer Vision (ICCV)
2017 IEEE International Conference on Computer Vision (ICCV)
Computer Science