Visual Question Generation in B engali (2023)

Abstract: The task of Visual Question Generation (VQG) is to generate human-like questions relevant to the given image. As VQG is an emerging research field, existing works tend to focus only on resource-rich language such as English due to the availability of datasets. In this paper, we propose the first Bengali Visual Question Generation task and develop a novel transformer-based encoder-decoder architecture that generates questions in Bengali when given an image. We propose multiple variants of models - (i) image-only: baseline model of generating questions from images without additional information, (ii) image-category and image-answer-category: guided VQG where we condition the model to generate questions based on the answer and the category of expected question. These models are trained and evaluated on the translated VQAv2.0 dataset. Our quantitative and qualitative results establish the first state of the art models for VQG task in Bengali and demonstrate that our models are capable of generating grammatically correct and relevant questions. Our quantitative results show that our image-cat model achieves a BLUE-1 score of 33.12 and BLEU-3 score of 7.56 which is the highest of the other two variants. We also perform a human evaluation to assess the quality of the generation tasks. Human evaluation suggests that image-cat model is capable of generating goal-driven and attribute-specific questions and also stays relevant to the corresponding image.

## introduction visual question generation (vqg) is an emerging research field in both computer vision and natural language processing. the task of vqg simply uses an image and other side information (e.g. answers or answer categories) as input and generates meaningful questions related to the image. tasks like cross-modal visual question answering (vqa) @xcite , 2019; @xcite @xcite , video captioning (vc) @xcite , image captioning (ic) @xcite @xcite , and multimodal machine translation @xcite @xcite are the recent advances in the ai community. while the majority of visuolingual tasks tend to focus on vqa, a few recent approaches have been proposed, focusing on the under-researched multi-modal task of vqg. vqg is a more creative and particularly challenging problem than vqa, because the generated questions need to be relevant, semantically coherent and comprehensible to the diverse contents of the given image. existing studies on visual question generation (vqg) have been primarily focused on languages that have ample resources, such as english. while some vqa research have been conducted in lowresource languages like hindi @xcite , bengali @xcite , japanese @xcite , and chinese @xcite , limitations have been identified specifically in the context of bengali language. while bengali language has some recent work on reading comprehension based question answering @xcite and visual question answering @xcite , there has been no research conducted for vqg task specifically in bengali language. to obtain meaningful questions, some vqg methods have either augmented the input including additional information such as answer categories, objects in image and expected answers @xcite @xcite . @xcite used ground truth answer with the image as an input, underscoring it to be an effective approach to produce non-generic questions. @xcite stated that knowing the answers beforehand simply defeats the purpose of generating realistic questions since the main purpose of generating a question is to attain an answer. instead, they introduced a variational auto-encoder model, which uses the concept of latent space, providing answer categories to generate relevant questions. @xcite , recently, proposed a guiding approach with three variant families that conditions the generative process to focus on specific chosen properties of the input image for generating questions. inspired by previous work, we also use additional information such as answer and answer categories in our experiments. to summarize, the main contributions of our paper are the following: ## related works the advent of visual understanding has been made possible due to continuous research in question answering and the availability of large-scale visual question answering (vqa) datasets @xcite @xcite . in the past few years, many methods have been proposed to increase the model's performance for a vqg task. earlier studies @xcite @xcite @xcite have explored the task of visual question generation through recurrent neural network (rnn), generative adversarial network (gan), and variational auto-encoder (vae) which either followed algorithmic rule-based or learning-based approach. in the visual-language domain, the first vqg paper proposed by @xcite introduced question-response generation that takes meaningful conversational dialogues as input to generate relevant questions. @xcite used an lstm-based encoder-decoder model that automates the generation of meaningful questions with question types to be highly diverse. motivated by the discriminator setting in gan, @xcite formulated a visual natural question generation task that learns two non-generic textual characteristics from the perspective of content and linguistics producing non-deterministic and diverse outputs. whereas, @xcite followed the vae paradigm along with lstm networks instead of gan to generate large set of diverse questions given an image-only input. during inference, their obtained results nevertheless required the use of ground truth answers. to defeat this non-viable scenario, @xcite proposed a vae model that uses the concept of latent variable and requires information from the target, i.e. answer categories, as input with the image during inference. similarly, @xcite follows the concept of latent variable, however, their proposed model architecture explores vqg from the perspective of guiding, which involves two variant families, explicit and two types of implicit guiding approach. our work is closely related to their explicit guiding method excluding the use of latent space. recently, @xcite proposed a bert-gen model which is capable of generating texts either in mono or multi-modal representation from out of the box pre-trained encoders. ## methodology in this section, we introduce our transformer based bengali visual question generation models which can generate meaningful non-generic questions when shown an image along with additional textual information. our vqg problem is designed as follows: given an image ĩ ∈ i, where i denotes a set of images, decode a question q. for each image ĩ, we also have access to textual utterances, such as ground truth answer and answer categories. note, we will use terms "answer category" and "category" interchangeably throughout the paper. in our work, we used answer categories from @xcite that take 1 out of 16 categorical variables to indicate the type of questions asked. for example, if our model wants to understand answers of "ro k (color)" category, then it should generate a question "eta ik ro er bas ? (what color is the bus?)". our baseline is an imageonly model with no additional textual information like answer or category. we present further two variants both of which shares the same architecture but takes different inputs in training. we feed two different textual information to our model during training. the first model is imageanscat that feeds the concatenated ground truth answer and category to the encoder and is concatenated with the image features. the second model is imagecat that takes only the relevant answer category as input to the encoder. in both of the versions, the input image is reconstructed to maximize information between the image and encoded outputs. vocabulary: we construct vocabulary considering all the textual utterances: questions, answers and answer categories. our vocabulary has a total of 7081 entries including the special tokens. we use word level tokenization. we set a default length of 20 token to each of the questions and 5 to each of the answers. in image encoder: given an image ĩ, we can extract image features, @xmath1. our image encoder is a resnet-18 pretrained cnn model, which is a convolutional neural network with 72-layer architecture consisting 18 deep layers @xcite . once obtaining these features, they are passed to a fully connected layer followed by a batch normalization layer. specifically, given f from image ĩ: encoder: we build a transformer encoder @xcite and use bengali pretrained glove (global vectors for word representation) word vectors @xcite as the embedding layer of the text encoder. next, we provide answer or answer categories and image features f as input to the text encoder. note that, image-cat variant only takes answer category c as its input during training and image-ans-cat takes concatenated version of answer and category, [a; c] (; operator represents concatenation) as seen in figure 2 . for image-ans-cat variant, a concatenated ver-sion of answer and category [a; c] is passed through the embedding layer and projected out as context, c img+ans+@xmath2; c]) ∈ r b×t×300 where, b is batch size and t is the length of the [a; c]. for the image-cat variant, we only pass the category, c and similarly generate a context c img+@xmath3, where t is the length of c. additionally, we generate padding masks on answer and category [a; c] @xmath4; c]) ∈ r b×1×t to @xmath5. same operation is performed on category input c and a masked category is generated c m . the imagecat model takes context, c and masked category c m as input to the encoder to encode textual feature representation: @xmath6, c m ) ∈ r b×t×300 . we follow the same procedure for the image-ans-cat model, where now encoder takes the context, c img+ans+cat and masked concatenated answer and category, [a; c] m . these textual feature representation s from the encoder are then concatenated to the input image features @xmath7, thus, representing our final encoder outputs as the concatenation (; operator) of textual and vision modality: @xmath8; i] ∈ r b×t×300 where b is the batch size and t is length of s. decoder: our decoder is a transformer decoder that also uses glove embeddings. following sequence-to-sequence causal decoding practices, our decoder receives encoder outputs from text encoder and ground truth questions during training. we, initially, @xmath9. each target question is concatenated with @xmath10, forming a tensor. in our decoder we follow similar steps as we did in our text encoder. we take ground truth questions q and generate target context: c @xmath11: q @xmath12. before, we pass the target context, c q to the decoder, we concatenate it with the same image features i that were passed as input to the encoder previously. the final target context can be denoted by @xmath13; i] ∈ r b×t×300 . finally, the decoder takes the encoder outputs x from the text encoder, the concatenated target context q and the source mask ([a; c] m or c m ) depending on the model variant and target question q m in the form of a tuple. our decoder is represented as following: @xmath14, q) where the decoder outputs a generated question q. ## conclusion we proposed the first vqg work in bengali and presented a novel transformer based encoderdecoder architecture that generates questions in bengali when shown an image and a given answer category. in our work, we presented two variants of our architecture: image-cat and image-ans-cat that differs from what input they receive during training. both of the variants generate a question based on answer category as guiding information from an image. however, due to having two different input combinations, image-cat performs marginally better in terms of quantitative scores, however, generates goal driven, specific questions conditioned towards the categorical information it receives. in contrast, the image-ans-cat model although generating grammatically valid questions fail to learn about answer categories. future work could analyze the impact of using more modern cnn architectures and newer pretrained models to generate questions from images.