Guiding Visual Question Generation (2022)

Abstract: In traditional Visual Question Generation (VQG), most images have multiple concepts (e.g. objects and categories) for which a question could be generated, but models are trained to mimic an arbitrary choice of concept as given in their training data. This makes training difficult and also poses issues for evaluation – multiple valid questions exist for most images but only one or a few are captured by the human references. We present Guiding Visual Question Generation - a variant of VQG which conditions the question generator on categorical information based on expectations on the type of question and the objects it should explore. We propose two variant families: (i) an explicitly guided model that enables an actor (human or automated) to select which objects and categories to generate a question for; and (ii) 2 types of implicitly guided models that learn which objects and categories to condition on, based on discrete variables. The proposed models are evaluated on an answer-category augmented VQA dataset and our quantitative results show a substantial improvement over the current state of the art (over 9 BLEU-4 increase). Human evaluation validates that guidance helps the generation of questions that are grammatically coherent and relevant to the given image and objects.

## introduction in the last few years, the ai research community has witnessed a surge in multimodal tasks such as visual question answering (vqa) @xcite , multimodal machine translation @xcite @xcite , and image captioning (ic) @xcite @xcite . visual question generation (vqg) @xcite @xcite , a multimodal task which aims to generate a question given an image, remains relatively under-researched despite the popularity of its textual counterpart. throughout the sparse literature in this domain, different approaches have augmented and/or incorporated extra information as input. for example, @xcite emphasised that providing the ground truth answer to a target question is beneficial in generating a non-generic question. @xcite pointed out that requiring an answer to generate questions violates a realistic scenario. instead, they proposed a latent variable model using answer categories to help generate the corresponding questions. recently, @xcite incorporated a pretrained language model with object features and image captions for question generation. in this work, we explore vqg from the perspective of 'guiding' a question generator. guiding has shown success in image captioning @xcite and @xcite ), and in this vqg work we introduce the notion of 'guiding' as conditioning a generator on inputs that match specific chosen properties from the target. we use the answer category and objects/concepts based on an image and target question as inputs to our decoder. we propose our explicit guiding approach to achieve this goal. we additionally investigate an implicit guiding approach which attempts to remove the dependency on an external actor (see more below). the explicit variant (section 3.1) is modelled around the notion that an actor can select a subset of detected objects in an image for conditioning the generative process. depending on the application, this selection could be done by a human, and algorithm or chosen randomly. for example, imagine either a open-conversation chat-bot or a language learning app. in the chat-bot case, a human may show the bot a picture of something. the bot may use randomly sampled concepts from the image (e.g. an object-detected tree) to ask a human a question upon. in the language learning case, the human may wish to select certain concepts they want the generated question to reflect. for exam-ple, they might select a subset of animal-related objects from the whole set of detected objects in order to generate questions for teaching the animalrelated vocabulary in a language learning setting. alongside the objects, the actor may also provide, or randomly sample, an answer category to the question generator. the implicit variant (section 3.2), on the other hand, is motivated by removing the dependency on the aforementioned actor. we provide two methodologies for our proposed implicit variant. the first uses a gumbel-softmax @xcite to provide a discrete sample of object labels that can be used for generating a question. the second method employs a model with two discrete latent variables that learn an internally-predicted category and a set of objects relevant for the generated question, optimised with cross-entropy and variational inference @xcite . human evaluation shows that our models can generate realistic and relevant questions, with our explicit model almost fooling humans when asked to determine which, out of two questions, is the generated question. our experiments and results are presented in section 5. to summarise, our main contributions are: 1) the first work to explore guiding using object labels in visual question generation; 2) a novel generative transformer-based set-to-sequence approach for visual question generation; 3) the first work to explore discrete variable models in visual question generation; and 4) a substantial increase in quantitative metrics -our explicit model improves the current state of the art setups by over 9 bleu-4 and 110 cider. ## methodology we introduce the shared concepts of our explicit and implicit model variants, before diving into the variant-specific methodologies (section 3.1 & 3.2). for both variants, we keep the vqg problem grounded to a realistic scenario. that is, during inference, we can only provide the model with an image, and data that can either be generated by a model (e.g. object features or image captions) and/or trivially provided by an actor (i.e. answer category and a selected subset of the detected objects). however, during training, we are able to use any available information, such as images, captions, objects, answer categories, answers and target questions, employing latent variable models to minimise divergences between feature representations of data accessible at train time but not inference time. this framework is inspired by @xcite . in appendix a, we discuss the differences of input during training, testing and inference. formally, the vqg problem is as follows: given an image ĩ ∈ ĩ, where ĩ denotes a set of images, decode a question q. in the guided variant, for each ĩ, we also have access to textual utterances, such as ground truth answer categories and answers. the utterances could also be extracted by an automated model, such as image captions @xcite , or object labels and features @xcite . in our work, answer categories take on 1 out of 16 categorical variables to indicate the type of question asked. for example, "how many people are in this picture?" would have a category of "count" (see @xcite for more details). text encoder. for encoding the text, we use bert @xcite as a pre-trained lan-guage model (plm). thus, for a tokenised textual input s of length t , we can extract a d-dimensional representation for @xmath0: @xmath1 image encoder. given an image ĩ, we can extract object features, @xmath2, and their respective normalized bounding boxes, @xmath3, with the 4 dimensions referring to horizontal and vertical positions of the feature bounding box. following the seminal methodology of @xcite , k o is usually 36. subsequent to obtaining these features, we encode the image using a transformer @xcite , replacing the default position embeddings with the spatial embeddings extracted from the bounding box features @xcite . specifically, given f, b from image ĩ: text decoder. we employ a pretrained transformer decoder for our task @xcite . following standard sequence-to-sequence causal decoding practices, our decoder receives some encoder outputs, and auto-regressively samples the next token, for use in the next decoding timestep. our encoder outputs are the concatenation (; operator) of our textual and vision modality representation: @xmath4; i] ∈ r (t +ko)×d , and our decoder takes on the form: @xmath5, where q is the predicted question. in this work, we primarily focus on a set-tosequence problem as opposed to a sequence-tosequence problem. that is, our textual input is not a natural language sequence, rather an unordered set comprising of tokens from the answer category, the object labels, and the caption. how this set is obtained is discussed in following section. due to the set input format, we disable positional encoding on the plm encoder (text encoder in figure 1 ). ## results we present quantitative results in ## conclusions we presented a guided approach to visual question generation (vqg), which allows for the generation of questions that focus on specific chosen aspects of the input image. we introduced three variants for this task, the explicit, implicit, and variational implicit. the former generates questions based on an explicit answer category and a set of concepts from the image. in contrast, the latter two discretely predict these concepts internally, receiving only the image as input. the explicit model achieves sota results when evaluated against comparable models. qualitative evaluation and human-based experiments demonstrate that both variants produce realistic and grammatically valid questions.