Diversity and Consistency: Exploring Visual Question-Answer Pair Generation (2021)

Abstract: Although showing promising values to downstream applications, generating question and answer together is under-explored. In this paper, we introduce a novel task that targets question-answer pair generation from visual images. It requires not only generating diverse question-answer pairs but also keeping the consistency of them. We study different generation paradigms for this task and propose three models: the pipeline model, the joint model, and the sequential model. We integrate variational inference into these models to achieve diversity and consistency. We also propose region representation scaling and attention alignment to improve the consistency further. We finally devise an evaluator as a quantitative metric for consistency. We validate our approach on two benchmarks, VQA2.0 and Visual-7w, by automatically and manually evaluating diversity and consistency. Experimental results show the effectiveness of our models: they can generate diverse or consistent pairs. Moreover, this task can be used to improve visual question generation and visual question answering.

## introduction teaching a machine to generate question-answer pairs (qaps) from images can benefit a lot of downstream applications such as child education @xcite , visual dialog @xcite , generating verification code for websites, visual question generation (vqg) @xcite and visual question answering (vqa) @xcite . for example, training vqa models requires large scale labelled data, which is usually labour intensive and expensive to construct. meanwhile, bias still exists in large qa datasets @xcite , including domain coverage, question and answer types, and linguistic style. therefore, as an alternative to constructing datasets manually, qap generation can promote vqa further. in this paper, we first study a novel task that aims to generate qaps from visual images, namely vqapg (visual question answer pair generation). as shown in figure 1 , it has two challenges: diversity and consistency. on the one hand, even a simple image will contain various content that could be asked. the focus of questions could range from appeared objects and their features to the global attributes, such as the image background and photo time. on the other hand, generating consistent qaps is also critical. when we say a qap is consistent with the image, we mean two points: the question is answerable with the image, and the answer is correct for the question. therefore, keeping the consistency is difficult because it requires the generation model to simultaneously guarantee the correctness of both questions and answers. there are two related tasks with vqapg, but neither serves as suitable prior art. the first is vqg @xcite @xcite , which produces questions given answers or other knowledge. another similar task is vqa that aims to answer given questions @xcite @xcite . they can be viewed as two subtasks of vqapg. however, simply combining the two is not an ideal substitute for vqapg since they condition another's output for learning. we study several generation paradigms to perform vqapg. the first is a pipeline model that generates questions and answers one after another. next, inspired by the non-autoregressive text generation @xcite , we propose a joint model that generates questions and answers in parallel. to reduce the model size, we also propose a sequential model that concatenates the two targets into one. we integrate latent variable(s) into these models through variational inference @xcite to improve diversity. if fed with different latent variables sampled from the prior distribution, the model can generate various qaps. besides, we observe that if we grid an image into multiple regions, the target question and answer are only related tightly with only part of them. as the latent variable contains information of the target qap, we use it to make the model concentrate on those related regions by scaling their representations. to improve the joint model's consistency, we align the attentions of the question decoder and the answer decoder to make them focus on similar regions. a remaining issue of vqapg is how to measure the consistency automatically in addition to manual inspection. traditional popular metrics, such as bleu @xcite and me-teor @xcite , only tell the overlapping degree between generated qaps and the target ones, being insufficient to indicate the consistency. targeting this issue, we devise a consistency evaluator trained with an adversarial strategy. we assume the samples in the dataset as consistent and construct inconsistent samples through shuffling images, answers, questions, or all of them. this evaluator will give a high score if the generated qap is consistent with the image and a low score if not. we conduct experiments on two datasets, vqa2.0 @xcite and visual-7w @xcite , in both of which each image has multiple human-created qaps. the quantitative results indicate that each model has its strengths. the pipeline model achieves the best diversity, while the sequential model achieves the best consistency. however, they can not boost both together. in contrast, the joint model improves both diversity and consistency impressively through variation. more ablation studies illustrate the effectiveness of our proposed methods: both the region representation scaling mechanism and the attention alignment mechanism improve consistency significantly. we also evaluate generated qaps manually and give case studies. moreover, to prove the effectiveness of vqapg on downstream applications, we use vqapg to generate pre-training samples for vqg and vqa. results indicate vqapg can enhance the performance of both vqg and vqa models. in short, our contribution mainly includes four parts: i) we propose a novel task, vqapg, that targets to generate diverse and consistent questionanswer pairs from images. ii) to perform vqapg, we study multiple generation paradigms and propose three models. we also design a consistency evaluator. iii) we incorporate variational inference into models and propose a series of techniques, including region representation scale and attention concentration to improve diversity and consistency. iv) we conduct comprehensive experiments on two large scale datasets. the results show the effectiveness of our approach to generate diverse and consistency qaps and benefit other applications. ## related works visual question generation is an interesting task emerged in recent years. question generation is firstly studied on text @xcite @xcite @xcite @xcite . while related studies on images has received little attention. existing methods in this field are typically based on learning algorithms @xcite . such methods are often incorporated with the variational process @xcite . visual question generation is also conducted together with visual question answering @xcite . visual question answering has received more interest thanks to available public datasets such as vqa2.0 @xcite and visu-algenome @xcite . through finetuning on large pre-trained models @xcite @xcite , the performance has been improved considerably. however, it still requires large scale labeled datasets, which is too consuming to annotate manually. therefore, a successful vqapg system would be beneficial to reduce such costs. ## vqapg task given an image denoted as i, vqapg aims to produce diverse qaps, each of which contains an answerable question q and its correct answer a under i. both the question @xmath0, q 2 , • • • , q m and the answer @xmath1, a 2 , • • • , a n are sequences, in which q i and a i represent tokens. figure 4 compares vqapg with typical vqg and vqa task. our final goal is to obtain a model to approximate the true data distribution p (q, a|i) so that we can sample questions and answers from it. because vqapg is a one-to-many task, we also expect this model to support sampling of diverse and consistent qaps. image question answer the vqg and vqa task the vqapg task image answer question image question answer ## approach we propose three models to perform vqapg, including the pipeline model, the joint model, and the sequential model. as shown in figure 2 , all of them adopt the encoder-decoder arcitecture. ## experiment setup in this section, we give a description of datasets and evaluation metrics. other settings including implementation details are in the appendix. ## conclusion in this paper, we propose a novel task, vqapg, which generates question-answer pairs from images. we also propose three models to perform this task. targeting on diversity and consistency, we integrate variational inference to these models and propose a series of actions, including region representation scaling and attention alignment. to evaluate the consistency automatically, we devise an evaluator. we evaluate our models on two datasets: vqa2.0 and visual-7w. the results show each model has its own merits. overall, they perform well to generate diverse and consistent qaps. on the other hand, there are still limitations in our works. for example, there is a trade-off between diversity and consistency; the generated question is typically one-hop, requiring no extra reasoning; the latent variable is uncontrollable and could introduce unexpected linguistic features to the decoder, bringing inconsistent qaps; the consistency evaluator needs more robust training strategy. in future works, we will explore generating consistent deep question-answer pairs.