M ulti QG - TI : Towards Question Generation from Multi-modal Sources (2023)

Abstract: We study the new problem of automatic question generation (QG) from multi-modal sources containing images and texts, significantly expanding the scope of most of the existing work that focuses exclusively on QG from only textual sources. We propose a simple solution for our new problem, called MultiQG-TI, which enables a text-only question generator to process visual input in addition to textual input. Specifically, we leverage an image-to-text model and an optical character recognition model to obtain the textual description of the image and extract any texts in the image, respectively, and then feed them together with the input texts to the question generator. We only fine-tune the question generator while keeping the other components fixed. On the challenging ScienceQA dataset, we demonstrate that MultiQG-TI significantly outperforms ChatGPT with few-shot prompting, despite having hundred-times less trainable parameters. Additional analyses empirically confirm the necessity of both visual and textual signals for QG and show the impact of various modeling choices. Code is available at https://anonymous.4open.science/r/multimodal-QG-47F2/

## introduction automatic question generation has the potential to enable personalized education experiences for subjects such as reading comprehension at a large scale @xcite @xcite and improve standardized tests by reducing the costs and the test length @xcite . most, if not all, existing question generation (qg) methods operate only on text: they take a textual paragraph @xcite or story @xcite as input and generate a textual question. these methods' focus on text-based qg is limiting, because many interesting questions can involve, or be generated from, multiple modalities such as images, diagrams, and tables, in addition to texts @xcite . ## the multiqg-ti methodology we now describe the four modules in multiqg-ti: a question generator module, an image-to-text module, an optical character reconigion (ocr) module, and an input formatting module. the question generator module. this module generates the question and is the only trainable module in multiqg-ti. we adopt a text-based question generator such that its inputs must be all in text format. adopting a text-based question generator enables us to choose from a wide range of pre-trained text-based generative models, whose training is also often more efficient than their multimodal counterparts. in this work, we instantiate the question generator with the recent flan-t5 model @xcite that have shown to perform strongly on new downstream tasks when fine-tuned on limited task-specific data. the image-to-text and ocr modules. a textbased question generator cannot take any visual input. to solve this problem, we use the imageto-text and ocr modules to interface between the image and text modalities and extract the visual information from the image format into a textual format appropriate as input for the text-based question generator. in particular, we use the image-totext module to describe the content in the image in texts, including any objects, scenes, actions, and events. we instantiate this module with the flan-t5-xxl version of blip-2 @xcite . while the image-to-text module extracts visually rich signals, it often fails to recognize any text in the image. this is problematic if the majority of the content in the image is text, such as a table. therefore, we complement the image-to-text module with an ocr module that specializes in extracting the texts in the image. we instantiate the ocr module in multiqg-ti with paddleocr @xcite . the input formatting module. this module, g, is a simple function that concatenates the input text and the texts from the input image into one coherent textual input for the question generator model. there are many choices available and one can simply perform a string join operation. in this work, we apply input formatting with the following template: generate a question based on the following information. background: {input_text}. image: {image_description}. texts in image: {image_text}.. in this template, {input_text}, {image_description}, and {image_text} are placeholders that will be replaced with the actual input text, the output from the image-to-text model and the output from the ocr module, respectively. training and inference. during training, we only update the parameters of the qg module while keeping the other modules fixed. we use the next word prediction as the training objective, which is commonly used in modern language model training @xcite . during inference, we proceed as follows: given an input image and text, ## conclusion we have conducted a first study into automatic multi-modal qg from images and texts. our proposed solution, multiqg-ti, is simple, easy-to-use, and highly capable, as evaluated and analyzed on the scienceqa dataset. our work opens a myriad of research opportunities. some of the exciting future directions include: 1) qg with multi-modal inputs and multi-modal outputs; 2) end-to-end visionlanguage modeling approach for qg; and 3) evaluating and comparing the pedagogical utilities of questions generated from multi-modal sources in real-world educational scenarios.