Visual Question Generation from Radiology Images (2020)

Abstract: Visual Question Generation (VQG), the task of generating a question based on image contents, is an increasingly important area that combines natural language processing and computer vision. Although there are some recent works that have attempted to generate questions from images in the open domain, the task of VQG in the medical domain has not been explored so far. In this paper, we introduce an approach to generation of visual questions about radiology images called VQGR, i.e. an algorithm that is able to ask a question when shown an image. VQGR first generates new training data from the existing examples, based on contextual word embeddings and image augmentation techniques. It then uses the variational auto-encoders model to encode images into a latent space and decode natural language questions. Experimental automatic evaluations performed on the VQA-RAD dataset of clinical visual questions show that VQGR achieves good performances compared with the baseline system. The source code is available at https://github.com/sarrouti/vqgr .

## introduction vqg refers to generating natural language questions based on the images contents. it is a new and exciting problem that combines both natural language processing @xcite and computer vision techniques @xcite . the motivation for the vqg task is two-fold: (1) generating large scale visual question answering (vqa) pairs to produce more training data at little cost @xcite and (2) improving efficiency of human annotation for vqa datasets construction @xcite . in addition to the aforementioned motivations, medical vqg could also benefit both doctors and patients. for example, patients could use questions provided by vqg systems to better understand medical images and start a conversation with their doctors. moreover, such systems could support medical education, medical decision, and patient education @xcite . a few recent works have attempted to generate questions from images in the open domain. however, the task of vqg in the medical domain has not been studied or explored. one major problem with medical vqg is the lack of large scale labeled training data which usually requires huge efforts to build. in this paper, we introduce vqgr, a vqg system that is able to generate natural language questions when shown radiology images. in summary, this paper makes the following contributions: the paper is organized as follows: section 2 surveys related work. section 3 describes the proposed vqg approach. section 4 presents experimental results and discussion. ## related work question generation, an increasingly important area, is the task of automatically creating natural language questions from a range of inputs, such as natural language text @xcite @xcite , structured data @xcite and images @xcite . in this work, we are interested in generating questions from medical images. vqg in the open-domain benefited from the available large annotated datasets @xcite @xcite . there is a variety of work studying generative models for generating visual questions in the open domain @xcite . recent vqg approaches have used autoencoders architecture for the purposes of vqg @xcite @xcite . the successes of these systems have primarily been a result of variational autoencoders (vaes) @xcite . conversely, vqg in the medical domain is still a challenging and under-explored task @xcite @xcite . although a high-quality manually created medical vqa dataset exists, vqa-rad @xcite , this dataset is too small for training and there is a need for vqg approaches to create training datasets of sufficient size. generating new training data from the existing examples through data augmentation is an effective approach that has been widely used to handle the data insufficiency problem in the open domain @xcite . due to the problem of data scarcity in medical vqg, we automatically generate new training data. in this paper, we present vqgr, a vqg system capable of generating questions about radiology images. the system is based on the vae architecture and data augmentation. ## methods the goal of this study is to generate natural language questions based on radiology image contents. the overview of vqgr is shown in figure 1 . ## conclusion and future work we presented the first attempt to generate visual questions in the medical domain. we first presented a data augmentation method to generate new training questions and images from the vqa-rad dataset. we then introduced the vqgr model that generates questions from radiology images. the results of the automatic and manual evaluations showed that vqgr outperforms the baseline model by generating fluent and relevant questions. image generated questions vs. ground truth what type of mri is used to acquire this image ? mri imaging modality used for this image? what is seen in the lung apices ? what abnormalities are in the lung apices ? is a ring enhancing lesion present in the right lobe of the liver? is the liver normal ? generated questions to advance vqa in the medical domain.