Exploring Union and Intersection of Visual Regions for Generating Questions, Answers, and Distractors (2024)

Abstract: Multiple-choice visual question answering (VQA) is to automatically choose a correct answer from a set of choices after reading an image. Existing efforts have been devoted to a separate generation of an image-related question, a correct answer, or challenge distractors. By contrast, we turn to a holistic generation and optimization of questions, answers, and distractors (QADs) in this study. This integrated generation strategy eliminates the need for human curation and guarantees information consistency. Furthermore, we first propose to put the spotlight on different image regions to diversify QADs. Accordingly, a novel framework ReBo is formulated in this paper. ReBo cyclically generates each QAD based on a recurrent multimodal encoder, and each generation is focusing on a different area of the image compared to those already concerned by the previously generated QADs. In addition to traditional VQA comparisons with state-of-the-art approaches, we also validate the capability of ReBo in generating augmented data to benefit VQA models.

## introduction visual question answering (vqa) @xcite @xcite represents a burgeoning research domain that necessitates the development of algorithms capable of responding to arbitrary natural language questions of a given image. a specific subset of vqa, known as multiple-choice (mc) vqa @xcite @xcite , involves the algorithm choosing the correct answer from a predefined list of distractors. mc-vqa, which requires vision-language understanding and cross-modality reasoning, is the representative benchmark for large vision-language models (lvlms) @xcite @xcite . in the era of large models, the imperative for large-scale, high-quality mc-vqa datasets has become increasingly pronounced. the traditional process of manually generating data is both labor-intensive and error-prone. many automated methods are available today to independently generate questions @xcite @xcite , answers @xcite , and distractors @xcite ) (qads) by machines based on images. however, these machine-generated qads are often created independently, making it challenging to ensure intrinsic dependencies between them. to address this issue and enhance the capabilities of large models in vision-language understanding and cross-modality reasoning, our work focuses on the unified generation of qads. in the process of jointly generating qads, how to comprehensively understand an image and diversify its generated qads is rarely touched. as illustrated in figure 1 , the three bounding boxes focused on by gpt-4o are significantly intersected, inducing redundant questions such as "who is in the photo" and "what animal is in the photo". in contrast, the qads generated by our model, rebo, are semantically rich and comprehensive for comprehending the image, as a broad union region with small intersections is concentrated on. in the long run, addressing the above challenge comes down to how to align image understanding across qads. we tackle this issue in two folds. first, we automate the generation of qads in a unified manner, ensuring a consistent image understanding from questions to answers and distractors. next, we research the generation of a series of qads by diversifying their focuses across image regions, which prevents information redundancy and provides a comprehensive understanding of the entire image. from the methodological point of view, we introduce a recurrent multimodal encoder to generate groups of qads considering the bounding boxes (rebo) of the given image. rebo takes the qads generated in previous steps as part of the input to generate qad in the next step. in addition, rebo considers the union and intersection of image bounding boxes, ensuring that each group of qads focuses on diverse regions. in this way, rebo disperses its attention on a broad area of the image and boosts the diversity of the generated qads. we conduct extensive experiments to validate the performance of rebo in different scenarios. moreover, a further experimental analysis suggests that the qads generated by rebo can be used to promote existing vqa models in vqa tasks. our main contributions are listed as follows: ## related work most prior research focused on generating a part or parts of qads, that is, question, answer, or distractors. for instance, the studies of visual question generation aim at generating questions related to an image or a video. zhang et al. (2016) took images and captions as inputs to generate questions with different types. johnson et al. (2016) introduced densecap to produce region captions, providing additional context to steer the process of question generation. krishna et al. (2019) formulated a visual question generation framework by optimizing the mutual information between the generated question and the pair of image and anticipated answer. shen et al. (2020) explored a visual question generation approach based on a double hint strategy concerning textual answers and regions of visual interests. on the other hand, the studies of vqa deploy attention on generating correct answers by understanding images, questions, and their interactions. for example, li et al. (2018) proposed iqan by taking visual question generation as a dual task to improve vqa performance. xiong and wu (2020) designed question-generating mechanisms and encouraged collaborative learning interactions among question-answering agents. changpinyo et al. (2022) used neural models to generate textual questions and question answering. in recent years, some research has broken into the joint generation of question-answer pairs. yang et al. (2021) employed variational inference to generate question-answer pairs considering diversity and consistency. su et al. (2021) presented an end-toend generator-pretester network, which generated question-answer pairs from videos. in contrast to visual question generation and vqa, visual distractors generation is a newly rising research field, which targets to generate challenging distractors according to the image, question, and answer. for example, @xcite introduced a reinforcement learning approach to generate distractors in the context of visual images. in this study, we explore a joint generation of groups of qads as well as take into account their diversified discriminative correlations. our proposed framework is capable of capturing the information from a broad region of the image, thereby enhancing the diversity and contextuality of the generated qads. ## our method: rebo we propose the unified framework rebo to generate qads as diverse as possible. in this section, we first introduce the model architecture in section 3.1. then, we describe the recurrent multimodal encoder in section 3.2, followed by the details of the diversifying qad generations in section 3.3. ## implementation details we adapt our model based on the modular architecture of instructblip @xcite . we retain the image encoder and the llm decoder while adapting the q-former into a recurrent multimodal encoder. we implement our model with the image encoder vit-g/14 @xcite and the large language model flant5-xl @xcite , which is an instruction-tuned model based on the encoder-decoder transformer t5 @xcite . we refer @xcite to employ an extra contrastive learning loss function to normalize the embeddings of prediction results and ground truth. for the hyper-parameters, we set the maximum text length to 60 and the minimum text length to 20 for the recurrent generation type and 60 to 180 for the concatenation generation type. the image size in all models is resized to 224. we use the batch size 8 and 32 for training and testing and fine-tune the datasets for 10 epochs. other parameters are set according to the original articles. for large language models, we calculated the mean and variance of the results over three runs. for large vision-language models, we report only one result due to consistent outputs. for our model and all other baselines, we divided the training and testing data into ten splits and calculated the mean and variance of the results over ten runs. we use the huggingface foot_0 transformers library implementation for llms and lvlms. our experiments are run on 1 nvidia a40 48g gpu. the source code is available at https://github.com/wenjianding/rebo . ## conclusion in this paper, we propose a novel framework with a recurrent multimodal encoder and bounding box scores to generate a series of qads. the multimodal encoder recurrently generates different qads for an image, utilizing the previous qads as part of the input to generate current qads. the bounding box scores consider the intersection over union and the union over total image, which can facilitate the generation of qads that attend to as large and diverse areas as possible for one image. we conduct experiments on the benchmark to demonstrate a significant advantage of our model in the evaluation metrics. additionally, our generated qads, as supplementary data to the original dataset, exhibit the capability to promote the performance of existing vqa models.