Augmenting Visual Question Answering with Semantic Frame Information in a Multitask Learning Approach. Visual Question Answering (VQA) concerns providing answers to Natural Language questions about images. Several deep neural network approaches have been proposed to model the task in an endtoend fashion. Whereas the task is grounded in visual processing, if the question focuses on events described by verbs, the language understanding component becomes crucial. Our hypothesis is that models should be aware of verb semantics, as expressed via semantic role labels, argument types, andor frame elements. Unfortunately, no VQA dataset exists that includes verb semantic information. Our first contribution is a new VQA dataset (imSituVQA) that we built by taking advantage of the imSitu annotations. The imSitu dataset consists of images manually labeled with semantic frame elements, mostly taken from FrameNet. Second, we propose a multitask CNNLSTM VQA model that learns to classify the answers as well as the semantic frame elements. Our experiments show that semantic frame element classification helps the VQA system avoid inconsistent responses and improves performance.

OCRVQA: Visual Question Answering by Reading Text in Images The problem of answering questions about an image is popularly known as visual question answering (or VQA in short). It is a wellestablished problem in computer vision. However, none of the VQA methods currently utilize the text often present in the image. These texts in images provide additional useful cues and facilitate better understanding of the visual content. In this paper, we introduce a novel task of visual question answering by reading text in images, i.e., by optical character recognition or OCR. We refer to this problem as OCRVQA. To facilitate a systematic way of studying this new problem, we introduce a largescale dataset, namely OCRVQA200K. This dataset comprises of 207,572 images of book covers and contains more than 1 million questionanswer pairs about these images. We judiciously combine wellestablished techniques from OCR and VQA domains to present a novel baseline for OCRVQA200K. The experimental results and rigorous analysis demonstrate various challenges present in this dataset leaving ample scope for the future research. We are optimistic that this new task along with compiled dataset will openup many exciting research avenues both for the document image analysis and the VQA communities.