Chain-of-Exemplar: Enhancing Distractor Generation for Multimodal Educational Question Generation (2024)

Abstract: Multiple-choice questions (MCQs) are important in enhancing concept learning and student engagement for educational purposes. Despite the multimodal nature of educational content, current methods focus mainly on text-based inputs and often neglect the integration of visual information. In this work, we study the problem of multimodal educational question generation, which aims at generating subject-specific educational questions with plausible yet incorrect distractors based on multimodal educational content. To tackle this problem, we introduce a novel framework, named Chain-of-Exemplar (CoE), which utilizes multimodal large language models (MLLMs) with Chain-of-Thought reasoning to improve the generation of challenging distractors. Furthermore, CoE leverages three-stage contextualized exemplar retrieval to retrieve exemplary questions as guides for generating more subject-specific educational questions. Experimental results on the ScienceQA benchmark demonstrate the superiority of CoE in both question generation and distractor generation over existing methods across various subjects and educational levels.

## introduction multiple-choice questions (mcqs) are important in education for promoting deep and extensive knowledge acquisition. research @xcite indicates that well-crafted questions with educational intent are closely linked to heightened student engagement and achievement. a key aspect in question generation is the quality of the distractors @xcite . questions with inadequate or lowquality distractors are less challenging and easier to solve. generating plausible, yet incorrect distractors is crucial in educational contexts, as effective and challenging distractors can significantly enhance students' reading comprehension and contribute to their overall academic success. however, it is costly and time-consuming to manually produce mcqs, since even professional test developers do not manage to write more than three or four good mcqs per day @xcite . to alleviate the human labour, automatic mcq generation has received extensive attention. previous research @xcite @xcite ) primarily focuses on text-based inputs for mcq generation, while generating mcqs from multimodal contexts is still relatively underexplored. such emphasis on text often leads to underutilization of visual information, which is prevalent within educational content, such as textbooks @xcite or examinations @xcite . additionally, some latest studies @xcite neglect the creation of challenging and thought-provoking distractors, which is essential for high-quality educational question generation. moreover, questions generated by current methods tend to be too general and not tailored for specific subjects or educational levels. as illustrated in figure 1 , the question generation model is tasked with creating educational questions from a biology textbook that includes both textual descriptions and visual illustrations. the example question focuses on subject-specific knowledge, and the accompanying distractors are carefully crafted to be plausible yet incorrect, enhancing the educational value of the questions. in the light of these challenges, we propose a novel framework named chain-of-exemplar (coe), which combines retrieved exemplars and chain-of-thought (cot) reasoning to generate educational questions and distractors from multimodal inputs of texts and images. specifically, we employ multimodal large language models (mllms) to encode multimodal contexts and incorporate them into a three-stage multimodal-cot framework that separates question generation, rationale generation, and distractor generation. the cot helps trigger the reasoning capability of mllms leading to the generation of plausible and confusing distractors. meanwhile, to generate more specialized questions for educational purposes in specific subjects, we leverage retrieved exemplary educational questions as demonstrations to guide the generation. finally, we adopt an easy-to-apply multi-task training strategy to finetune our generative models. the main contributions of this work can be summarized as follows: ## related works question generation question generation (qg) @xcite plays a crucial role in applications like conversational systems @xcite @xcite and intelligent tutoring systems @xcite @xcite @xcite . evolving from prior research based on syntactic trees or knowledge bases @xcite , most existing studies typically adopt deep neural networks @xcite @xcite for question generation. with the advent of pre-trained and large language models (plms/llms), recent works @xcite @xcite design various finetuning strategies to enhance the qg capabilities of language models. regarding educational purposes, multiple-choice question generation @xcite holds great importance, where distractor generation @xcite plays a crucial role. apart from text-based inputs, there is also a growing body of research focusing on qg from images @xcite . however, it is worth noting that most existing work focuses on uni-modal qg, leaving the potential of multimodal qg largely unexplored. ## method given a correct answer a and multimodal context @xmath0, t }, where i represents the image and t represents the texual paragraph, the task of multimodal educational question generation aims to generate a relevant question q and several distracting answers a ′ . the overview of the proposed coe framework is illustrated in figure 2 . ## conclusions in this paper, we present a novel framework called chain-of-exemplar (coe), which combines retrieved exemplars and chain-of-thought (cot) reasoning to generate educational questions and distractors from multimodal inputs of texts and images. specifically, we utilize mllms to encode multimodal contexts and incorporate them into a three-stage multimodal-cot framework, namely question generation, rationale generation, and distractor generation. meanwhile, we introduce a contextualized exemplar retrieval (cer) module to retrieve exemplary educational questions as demonstrations to guide the generation. we finally adopt an easy-to-apply multi-task training strategy to finetune our generative models hallucination issues another limitation of coe is that our rationale generation module may suffer from the typical flaw of hallucination issues, i.e., making fabricated intermediate reasoning that is irrelevant to the context and answer. hallucinated rationale would mislead the generation process, resulting in irrelevant and trivial distractors generation. one potential solution involves replacing the greedy decoding strategy used in chainof-thought reasoning with self-consistency @xcite by sampling diverse reasoning paths and selecting the most consistent rationale for distractor generation, which mimic multiple different ways of thinking. the greater the diversity of reasoning paths, the more authentic and reasonable rationale can be generated by cot. we believe that this future research direction will prove valuable and promising in effectively tackling hallucination issues. exemplar resource during training, our cer module retrieves domain-specific exemplars from training split of scienceqa to guide the generation. however, the generators trained under the guidance of these exemplars greatly restricts the quality and diversity of the generated questions and distractors, primarily due to the strong dependence on training data. actually, this limitation is not exclusive to our work. the educational mcq generation heavily relies on training data in specific domain to generate questions and distractors for educational purpose. therefore, we acknowledge the need for future research to explore methods for incorporating the retrieved exemplar with external knowledge to reduce the reliance on training data and enhance the generation quality.