Leveraging Visual Question Answering to Improve Text-to-Image Synthesis (2020)

Abstract: Generating images from textual descriptions has recently attracted a lot of interest. While current models can generate photo-realistic images of individual objects such as birds and human faces, synthesising images with multiple objects is still very difficult. In this paper, we propose an effective way to combine Text-to-Image (T2I) synthesis with Visual Question Answering (VQA) to improve the image quality and image-text alignment of generated images by leveraging the VQA 2.0 dataset. We create additional training samples by concatenating question and answer (QA) pairs and employ a standard VQA model to provide the T2I model with an auxiliary learning signal. We encourage images generated from QA pairs to look realistic and additionally minimize an external VQA loss. Our method lowers the FID from 27.84 to 25.38 and increases the R-prec. from 83.82% to 84.79% when compared to the baseline, which indicates that T2I synthesis can successfully be improved using a standard VQA model.

## related work initial t2i approaches @xcite adopted the conditional gan (cgan) @xcite and ac-gan @xcite ideas to replace the conditioning variable by a text embedding which allows to condition the generator on a textual description. analog to many current approaches, our approach is also based on attngan @xcite , which incorporates an attention mechanism on word features to allow the network to synthesize fine-grained details. in terms of using vqa for t2i, to our knowledge the only other approach is vqa-gan @xcite . however, in contrast to their approach, our architecture is simpler, we do not use layout information, work on individual qa pairs, and produce higher resolution images (256x256 vs. 128x128). ## approach we extend the attngan @xcite architecture to leverage question and answer (qa) pairs from the vqa 2.0 @xcite dataset by appending a vqa model @xcite . in addition to generating images from image descriptions, our model is also trained to produce images given a qa pair and minimize an external vqa loss. see figure 2 for an overview. after revisiting the individual components, we explain our extension in more detail. ## conclusion in this paper we proposed a simple method to leverage vqa data for t2i via a combination of attngan, a well-known t2i model, and a standard vqa model. by concatenating quesion answer (qa) pairs from the vqa 2.0 dataset we created additional training samples. images generated from the qa pairs are passed to the vqa model which provides an additional learning signal to the generator. our results show a substantial improvement over the baseline in terms of fid, but requires additional supervision. possible future research directions could be to investigate whether our extension also boosts other t2i models, and the influence of more training data and the external vqa loss separately.