ECIS - VQG : Generation of Entity-centric Information-seeking Questions from Videos (2024)

Abstract: Previous studies on question generation from videos have mostly focused on generating questions about common objects and attributes and hence are not entity-centric. In this work, we focus on the generation of entity-centric information-seeking questions from videos. Such a system could be useful for video-based learning, recommending “People Also Ask” questions, video-based chatbots, and fact-checking. Our work addresses three key challenges: identifying question-worthy information, linking it to entities, and effectively utilizing multimodal signals. Further, to the best of our knowledge, there does not exist a large-scale dataset for this task. Most video question generation datasets are on TV shows, movies, or human activities or lack entity-centric information-seeking questions. Hence, we contribute a diverse dataset of YouTube videos, VideoQuestions, consisting of 411 videos with 2265 manually annotated questions. We further propose a model architecture combining Transformers, rich context signals (titles, transcripts, captions, embeddings), and a combination of cross-entropy and contrastive loss function to encourage entity-centric question generation. Our best method yields BLEU, ROUGE, CIDEr, and METEOR scores of 71.3, 78.6, 7.31, and 81.9, respectively, demonstrating practical usability. We make the code and dataset publicly available.

## introduction question generation (qg) aims at generating a valid and fluent question according to a given passage and an optional target answer. qg systems can be used to create interactive learning materials, exams, quizzes @xcite , interview questions, clarification questions @xcite and study aids for students @xcite . in customer support, qg systems can be helpful in generation of frequently asked questions (faqs) pages which can help train better chatbots, thereby improving customer support services. lastly, generating thought-provoking questions can stimulate critical thinking and can play a vital role in formulating research questions, surveys, and interview protocols. most of the previous studies in qg are based on text passages @xcite . in contrast, qg is rather underexplored in images and videos. in video qg @xcite @xcite , the goal is to generate meaningful questions about a video, optionally targeting an answer. although under-explored, video qg has multiple applications in video-based e-learning systems, recommending questions for "people also ask" module on search engines, video-based chatbots, and fact-checking from videos. fig. 1 shows a video clip (chapter) being shown as an answer to the second question in the "people also ask" module. there are a very few studies on video-based qg, even those studies either generate questions only from transcripts @xcite @xcite (and hence do not use visual information from videos) or generate questions about common objects (in a frame of a video) and attributes (such as colour of clothing) present in the video @xcite @xcite @xcite . they focus on generating questions like "what did the person get a plastic bowl from?", "what will i show you?", "what is the nationality of the celebrities highlighted in the last one third of the video?", etc. such research studies involving questions whose answers are too contextually dependent on the video content, do not provide generally applicable information-seeking questions. note that these questions do not contain entity mentions. unlike such studies, we focus on a novel video qg problem setting: generation of entity-centric information-seeking (ecis) questions from videos. we define an entity as something with an actual existence or a distinct identity. it could refer to objects of focus in a video, encompassing places, buildings, persons, things, or organizations-for example, a book, the company microsoft, and the president of the usa. our problem definition can be established as follows: let v denotes a video context consisting of a sequence of frames tf 1 , f 2 , ..., f n u, where each frame f i represents a snapshot of the video at time t i . each frame contains visual and possibly textual information. given a video context v , the goal is to generate a set of questions q " tq 1 , q 2 , ..., q m u that specifically targets entities within that context. these entities are selected based on their potential to pique a user's interest, and the generated questions aim to extract detailed information about these entities from the video content. each question q i is formulated to elicit information about a particular entity e j and can be represented as a function qpe j q that maps the entity e j to a natural language question. as illustrated in fig. 2 , in the first example, the traditional qg model (t5 fine-tuned on squad @xcite ) generated the question "is the food really cheap?". this question is not self-complete and needs more context like specific "place" or the subject (food item). on the other hand, the question generated by our proposed system, "what is special about outdoor dining area at khaja ghar?" is self-complete and informationseeking. the corresponding video provides a relevant answer to this question. one of the core challenges of generating entity-centric questions lies in effectively extracting suitable features from diverse video content, encompassing a wide array of topics, contexts, and presentation styles. solely relying upon pre-determined patterns or templates or identifying objects in a video, etc. @xcite , is not enough. in this study, we aim to exploit video chapter titles to generate ecis questions for videos. however, not all chapter titles are question-worthy. hence, we train classifiers to find question-worthy chapter titles. also, chapter titles cannot be directly used as questions; they need to be modified appropriately. hence, we also fine-tune transformer @xcite -based encoderdecoder models like t5 @xcite and bart @xcite on our manually annotated dataset. to generate high-quality ecis questions, besides chapter title, we supply additional input based on richer context like video title, video transcript, and visual information like frame captions and frame embeddings. our models are trained using a contrastive loss that motivates the generation of ecis questions rather than questions about common objects or attributes. lastly, we also explore effectiveness of prompt engineering with alpaca lora @xcite , qwen-vl @xcite , gpt-3.5-turbo @xcite and @xcite for this ecis-vqg task. overall, we make the following contributions. (1) we propose a novel problem setting for video qg: entity-centric information-seeking questions. link: https://www.youtube.com/watch?@xmath0: their cosy outdoor dining area transcript: they started this khaja ghar because ramesh wanted to … food is coming ... existing qg model: is the food really cheap? our model: what is special about outdoor dining area at khaja ghar? link: https://www.youtube.com/watch?@xmath1: arm rest transcript: armrest is a very important feature of a chair because it helps to get rid from having any shoulder pain.... existing qg model: what are the pros and cons of the chair? our model: how is the 'arm rest' in 'gaming chair im-19'? … … figure 2 : two examples of ecis qg task. for example-1, although the existing qg model @xcite generates a grammatically sound question, it lacks key context information like a place (where is the food cheap?) or subject (which food item?). in example-2, without the particular chair's name, the question generated by the existing qg model is too broad. ## videoquestions dataset data curation and pre-processing: we curate a dataset of videos from youtube with manually annotated questions as follows. we first gather metadata for 50k foot_2 youtube videos using youtube-dl foot_3 . we only retain videos less than 10 minutes in duration to manage the overall dataset size. also, we retain a video only if it has an english transcript. our goal is to select videos that are highly information-centric and relevant to real-world topics. hence, we choose videos belonging to these categories: categorization of chapter titles: one way to generate ecis questions from a video is to exploit video chapter titles (demarcated by the uploaders themselves). however, not all chapter titles are question-worthy. hence, we train classifiers to find question-worthy chapter titles. according to the richness of the information in chapter titles, we categorize them into the following four types. (1) useless (ul): chapter titles that provide no meaningful information or are not conducive to generating coherent and meaningful questions. examples: "intro", "like-share-subscribe." (2) self-complete questions (scq): chapter titles that, by themselves, form proper questions related to the specific topics being discussed in the respective video chapter. examples: "what is a wormhole?" or "what is servant leadership in business?". (3) not self-complete questions (nscq): chapter titles that are appropriate questions but require additional information or context to make sense. examples: "when was she born?", "what makes each place different?". in the context of the respective videos, their complete question equivalents would be "when was the la model elizabeth turner born?", "how is lakewood ranch different from palmer ranch?" (4) not self-complete phrases (nscp): chapter titles that contain key information but cannot be used directly as questions. examples: "gives you online visibility" or "aerial roots." we asked two male indian annotators in their late 20s to manually classify the 2789 chapter titles in videoquestions using the above categorization scheme. both annotators have a bachelors in computer science and engineering, and have a good grasp of the english language with prior knowledge in the similar tasks. the annotation process resulted in a substantial kappa score of 0.65, indicating a significant level of agreement between our annotators. conflicts were resolved by mutual discussions between the annotators. overall, chapter title type distribution is as follows: 524 ul, 133 nscq, 189 scq and 1943 nscp. further, for nscq and nscp chapter titles, the annotators were also asked to write out corresponding questions ensuring that the questions were grammatically sound, information-seeking, entity-centric and highly relevant to the respective video chapter. ignoring the ul type chapter titles, overall the videoquestions dataset contains questions corresponding to each of the 133 (nscq) + 189 (scq) + 1943 (nscp) = 2265 chapters. specifically, the annotators were provided these guidelines to generate questions. (1) questions should be broadly related to the existing chapter title and not unrelated. (2) ensure that questions are sensible, capable of generating interest in users, are information-seeking, centered around some real world entity, and are directly related to the content in the corresponding video chapter. (3) avoid questions that require additional context or refer to topics beyond the specific chapter. (4) avoid questions with relative time references like "current us president" or "current google ceo" or "conversion rate usd/jpy", etc. obtaining frame captions: while transcripts and video titles offer insight into the linguistic content of the video, they cannot capture the rich visual context that videos inherently possess. hence, we obtain captions for individual video frames. using frame captions along with other inputs like chapter title, video title, transcript for question generation chapter title video title chapter video chapter transcript chapter frame captions frame extraction using blip/clipcap bert-based classifier chapter title type? chapter video embedding clip/ resnext gpt3.5-turbo summary t5/bart/ alpaca traditional qg non-ecis question generated question frame 1 : caption 1 frame 2 : caption 2 … frame n : caption n 𝐿𝐿 = 𝐿𝐿 𝐶𝐶𝐶𝐶 + 𝜆𝜆𝐿𝐿 𝐶𝐶 contrastive loss 𝐿𝐿 𝐶𝐶 cross-entropy loss 𝐿𝐿 𝐶𝐶𝐶𝐶 cross-attention transformer video title discard ul scq nscq/ nscp prompt for alpaca gold question can ensure that the generated questions retain information from the spoken as well as visual context of the video. we obtain frame captions as follows. we first obtain the clip corresponding to a chapter based on its start and end timestamp. next, we sample one frame per second from the chapter clip. each extracted frame is then passed through the blip @xcite /clipcap @xcite model, to generate frame captions. all frame captions corresponding to the same chapter are concatenated. overall, in our dataset, average lengths of chapter title, video title, transcript and frame captions are around 4.6, 10.3, 132.4 and 173.6 words, respectively. ## ecis question generation from videos given a video, at inference time, our proposed system generates ecis questions from the video as follows. we first extract chapter titles from the video description. next, using a chapter title classifier (discussed next in this section), we classify each chapter as ul, scq, nscq, or nscp. ul chapter titles are discarded. scq chapter titles directly serve as relevant ecis questions. nscq and nscp questions must be processed by a ecis questions generator module (discussed later in this section) to generate appropriate questions. we illustrate the proposed method in fig. 3 . ## conclusion in this paper, we have proposed an interesting problem for generating ecis questions from videos. since there was no readily available dataset, we create a new dataset, videoquestions, with 411 videos and more than 2k manually annotated questions. we proposed a system which consumes rich text and visual signals to arrive at an effective video representation. this, in turn, helps train a transformer encoder-decoder model with cross entropy and contrastive loss. extensive experiments show that our best model generates high quality ecis questions which can be leveraged for several practical applications like "video-based people also ask", "online tutoring systems", etc. in the future, we aim to expand this system to locate relevant clips within videos and broadening our approach to support multiple languages to increase the model's global relevance.