Multi- VQG : Generating Engaging Questions for Multiple Images (2022)

Abstract: Generating engaging content has drawn much recent attention in the NLP community. Asking questions is a natural way to respond to photos and promote awareness. However, most answers to questions in traditional question-answering (QA) datasets are factoids, which reduce individuals’ willingness to answer. Furthermore, traditional visual question generation (VQG) confines the source data for question generation to single images, resulting in a limited ability to comprehend time-series information of the underlying event. In this paper, we propose generating engaging questions from multiple images. We present MVQG, a new dataset, and establish a series of baselines, including both end-to-end and dual-stage architectures. Results show that building stories behind the image sequence enables models togenerate engaging questions, which confirms our assumption that people typically construct a picture of the event in their minds before asking questions. These results open up an exciting challenge for visual-and-language models to implicitly construct a story behind a series of photos to allow for creativity and experience sharing and hence draw attention to downstream applications.

## introduction the popularity of image-sharing behavior in chats and social media applications shows that this is a natural way to increase participant engagement @xcite . in response, asking questions based on these photos is a straightforward method to promote awareness, sustain attention, and acquire useful information. an obvious example is that when we see someone share a photo of a car accident on facebook, commenting "was anyone injured in the crash?" draws more attention and replies from both the author and other readers than "oh my goodness, that is serious." furthermore, from the author's aspect, posting photos on social media with an engaging, image-related question helps the author to hear the public voice of their feelings and thoughts, and keeps the author connected with the world @xcite . however, not all the questions have the same effect. questions in traditional text-based qa datasets such as squad @xcite , narrativeqa @xcite , or fairy-taleqa @xcite are for educational purposes and language understanding, which do not seek to encourage people to reply. meanwhile, questions in the vqa dataset @xcite ) usually ask about the color of an object or its position, to which the answers are too obvious for humans to respond. in fact, these two kinds of questions are rarely seen in daily chat and social media posts. @xcite state that humans consider engaging and effective captions those that "avoid stating the obvious." as with image captions, an engaging and effective question asks about things behind the scenes and is usually open-ended. moreover, the input information for question generation also matters. images are more straight-forward for humans than text, and they provide plenty of room for imagination. in addition, images shared on social media are often sequential instead of solitary, as a single image gives readers only a limited understanding of the experience being shared. to this end, despite the existence of question generation (qg) datasets such as those created by @xcite , @xcite , or @xcite , which contain engaging questions, their text-based or single-image settings limit the usage of current qg models in popular applications. to facilitate downstream applications involving a set of shared images, e.g., accompanying robots, social media robots, automatic assistants, or reminiscence therapy, we propose generating an engaging question from multiple images. we create mvqg, a new dataset, by asking workers to write down a question following instructions based on a sequence of photos from vist @xcite , a dataset consisting of five sequential images and a story about those images. for a better illustration of the task, figure 1 shows two examples of mvqg. unlike the instruction of vqa's data collection @xcite asked workers to imagine "a smart robot" that "understands a lot about images," such as objects, scenes, or color, or texture, and come up with questions to "stump this smart robot." our instruction, on the other hand, asked workers to imagine that they want to have a conversation with people on twitter and hence to write a question to start that conversation. the data analysis shows that our instructions help collect more engaging questions than vqg @xcite , the benchmark dataset for visual question generation. furthermore, we establish a series of baselines, including both end-to-end and dualstage architectures. the experimental results show that information about stories behind the image sequence helps baselines generate engaging questions, which confirms our assumption that humans typically construct stories in their heads before asking questions. these results open up an exciting challenge for visual-and-language models: implicitly constructing a story behind a series of photos to allow for creativity and experience sharing, hence drawing attention to its downstream applications. the contributions of our paper are threefold: first, we introduce a novel task multi-vqg and mvqg, a new dataset: given a sequence of relevant images, generate a corresponding engaging question; second, we propose several baselines and show that story information helps baselines to generate engaging questions from image sequences; third, we propose five aspects for human evaluation as benchmarks to better evaluate the engagement of generated questions. ## related work user engagement has received much recent attention in the nlp community. @xcite created the first visual question generation dataset comprised of natural and engaging questions. however, engagement is not well-stated in this work; it simply means "the first question that comes to mind". @xcite present an engaging image captioning task to improve the ability of machines to communicate with humans; engaging captions are defined as captions that "avoid stating the obvious." @xcite develop a dataset for poll-question generation for social media posts. this work demonstrates that the poll question is an engaging question that can be utilized to help us hear the public voice for decision-making and thus better understand our society. @xcite state that an engaging and attractive question may incorporate additional details or emotional phrases. such questions are more likely to be answered. images and questions are two prominent elements in these works, indicating that visual stimulation and inquiry are typical means to communicate awareness and sustain connections. however, these studies primarily consider single images, limiting the use of current qg models in popular applications because individuals typically share multiple photos to express more comprehensive experiences. in our study, we propose generating engaging questions over an image sequence and creating a dataset comprised of five photos and human-written questions. a visual-and-language (vl) model is typically used to generate engaging questions from images. after the development of bert @xcite , various bert-based vl models were proposed. these vl models are designed to integrate information from both vision and language modalities via an encoder, and are categorized into fusion encoders and dual encoders based on how input from distinct modalities is aggregated @xcite . fusion encoder models such as @xcite , xlmert @xcite , soho @xcite , and vl-t5 @xcite ) encode text embeddings and image features in the same model with different fusion approaches. following self-or cross-attention, the hidden state of the last layer is treated as a fused representation of different modalities. because fusion encoder models require image and text pairings as input, the model must input all possible pairs in image-text matching tasks, resulting in poor inference speed. dual encoder models such as clip @xcite , on the other hand, use two single-modal encoders to encode the two modalities separately and use the dot product to project the image embedding and text embedding to the same semantic space to compute vl similarity scores. although dual encoder models are lighter, they frequently fail in difficult vl understanding tasks. as a result, we continue to employ fusion encoder models as baselines in our work. we choose vl-t5 @xcite as the backbone in particular because it treats all vl tasks as text-generating tasks, which is appropriate for our question generation scenario. inspired by @xcite , we propose an additional baseline model by employing the visual encoder of clip @xcite instead of the self-trained image feature extractor in our fusion encoder, so that the image features are better projected into the semantic space. ## baselines we propose both end-to-end and dual-stage vl baselines for mvqg. we introduce each baseline here and provide the details in appendix a. for the end-to-end baselines, we chose the vl-t5 model @xcite as the backbone . vl-t5 inputs contain the visual embedding v and the visual semantic grounding g. each image v i is handled as a sequence of visual embeddings consisting of the whole image embedding and its object region embeddings. as visual embeddings from roi features lack semantic meaning, we inject visual semantic grounding into vl-t5 to facilitate semantic understanding and cross-image reasoning. we adopt grounded situation recognition (gsr) @xcite and the corresponding jsl model to produce structured semantic summaries of images. for each image v i , jsl outputs a verb representing the salient activity of v i and its 1 to 6 corresponding semantic roles. the predicted verb and nouns are combined as the visual semantic grounding g i of each image. we propose three fine-tuned versions of vl-t5 respectively pretrained on vcr @xcite (vl-t5 f_vcr ), vist (vl-t5 f_vist ), and vqg (vl-t5 f_vqg ), and fine-tuned on mvqg. he et al. (2019) show that after standard finetuning, the model forgets important language generation skills acquired during pretraining. therefore, we propose the adapt-tuned version of vl-t5 by adding the adapter layer to each transformer block of the baseline, and replacing the fine-tuning stage with adapt-tuning. in the adapttuning stage, we update only the parameters of the adapter layer and freeze all other parameters. we pretrain our model on vist (vl-t5 a_vist ) and vqg (vl-t5 a_vqg ), and adapt-tune on mvqg. moreover, inspired by @xcite , which for the dual-stage baselines, we first used an image captioning model to generate a description from an image sequence, after which we used a question generation model to generate a question from the description. the image captioning model used was a vl-t5 model pretrained on vcr, and the question generation model was a t5 model @xcite pretrained on squad @xcite . we provided three types of text as descriptions: (1) captions from the vist dataset (cap2q), (2) stories from the vist dataset (sty2q), and (3) summaries from q2 in mvqg (sum2q). the vl-t5 image captioning model and question generation model were fine-tuned on these three description types, respectively. as the end-to-end baselines used clip to encode visual input, we adopted the clip visual encoder in our dual-staged baselines. we replaced the t5 model in the second stage with vl-t5 and then used the result of the clip visual encoder as visual input and the descriptions as textual input. for the different types of descriptions, we propose cap2q clip , sty2q clip , and sum2q clip . ## experiment and discussion we randomly divided mvqg into the training (70%), val (20%), and test (10%) sets, and eval-uated the models introduced earlier with human and automatic metric evaluation. ## conclusion we propose a novel task: given a sequence of images, generate an engaging question. this task extends visual question generation by enabling reasoning across images to comprehend a complete story. we collect mvqg by asking workers to write down five obvious objects, a summary of the image sequence, as well as an engaging question that they would want to post on social media. we establish several baselines for this task. experimental results reveal that image-related stories help models generate engaging questions, and that using multiple images as input helps models understand the overall picture of the current situation, leading to a better question. the task, dataset, and ex- case 1 ground truth can someone tell me what the name of the bright pink flowers are? vl-t5f_vist what color is the flower? vl-t5a_vist have you been interested in learning about bugs and bird life? case 2 ground truth when you look at these pictures, what else do you think might be sold? vl-t5f_vist do you like to go to places that have a crowd of people? sty2q clip what kind of food would you like to buy at this festival? case 3 ground truth how does everyone think the red sox are going to do this year? vl-t5a_vist are you fascinated by football? sty2q clip have you ever been to a costume party before?