Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integration (2024)

Abstract: In this paper, we tackle the task of distractor generation (DG) for multiple-choice questions. Our study introduces two key designs. First, we propose the concept of retrieval augmented pretraining, which involves refining the language model pretraining to align it more closely with the downstream task of DG. Second, we explore the integration of knowledge graphs and language models to further enhance the performance of DG. Our study unveils promising directions for further development in DG by showcasing the efficacy of knowledge augmentation and task-specific pretraining. These findings demonstrate the potential for leveraging both strategies to enhance the quality and performance of DG systems.

## introduction multiple-choice questions (mcqs) are widely used to evaluate a learner's knowledge. however, creating them manually requires a significant amount of time and effort from educators. well-designed mcqs help in accurately assessing learners' abilities. the most challenging part of creating mcqs is designing appropriate incorrect options, commonly called distractors. therefore, researchers have focused on the automatic distractor generation for mcqs in recent years. in this paper, we explore two directions for improving dg methods. task-specific pretraining current sota dg methods, such as utilizing bert for dg @xcite or harnessing t5 for text2text dg @xcite , are all based on pre-trained language models (lms). existing lms undergo task-agnostic pertaining. however, recently, the idea of task-specific pretraining has gained prominence. the aim is to refine the pretraining process to closely align with the downstream task. therefore, our first endeavor in this paper is to investigate task-specific pretraining for the dg task. in line with this goal, we propose retrieval augmented pretraining (rap) for task-specific pretraining. the main idea involves using mcq answers to retrieve relevant sentences/passages from a large corpus, such as wikipedia, to create pseudo questions, and then use the generated pseudo question for task-specific pertaining. figure 1 illustrates an example of a create pseudo question. assume that we are given a set of mcq options, the answer kidneys, and three distractors lungs, pancreas, liver. the idea is to use the answer option kidneys to retrieve a sentence, i.e., the kidneys are two reddish-brown bean-shaped organs. and replace the answer with the [mask] token to generate a pseudo question, i.e., the [mask] are two reddish-brown bean-shaped organs. the pseudo question and the distractors are then served as training data for rap. knowledge augmented generation recent research trends show that leveraging knowledge graphs (kgs) can enhance the performance of lmbased text generation tasks, as discussed in @xcite and @xcite . the perspective is that lms and kgs should complement each other. kgs go beyond text by offering structural information, representing entities as nodes and their relationships as edges. this structured knowledge enhances the ability for multi-step reasoning. in line with this motivation, we propose knowledge-augmented generation (kag), where we build upon the candidate augmentation strategy proposed by the state-of-the-art dg method @xcite . the idea is to retrieve knowledge triplets from a knowledge graph to serve as auxiliary information for the text2text dg model. our study unveils promising directions for further development in dg by showcasing the efficacy of knowledge augmentation and taskspecific pretraining. the rest of this paper is organized as follows. section 2 reviews the works of automatic distractor generation in the literature. in section 3 we present the proposed methods. section 4 reports the performance evaluation and section 5 concludes this work and discuss the future work. ## conclusion in this paper, we introduce the utilization of taskspecific pretraining and knowledge base. our experimental results highlight a significant performance improvement achieved through the integration. our work represents a novel contribution to the field of distractor generation for mcqs, showcasing the potential of combining retrieval augmented pretraining and kgs to achieve superior results.