Distractor Generation based on T ext2 T ext Language Models with Pseudo K ullback- L eibler Divergence Regulation (2023)

Abstract: In this paper, we address the task of cloze-style multiple choice question (MCQs) distractor generation. Our study is featured by the following designs. First, we propose to formulate the cloze distractor generation as a Text2Text task. Second, we propose pseudo Kullback-Leibler Divergence for regulating the generation to consider the item discrimination index in education evaluation. Third, we explore the candidate augmentation strategy and multi-tasking training with cloze-related tasks to further boost the generation performance. Through experiments with benchmarking datasets, our best perfomring model advances the state-of-the-art result from 10.81 to 22.00 (p@1 score).

## introduction cloze-style multiple choice question (mcq) is a common form of exercise used to assess the knowledge of learner. manual crafting of cloze questions demands significant time and effort for educator, which motivates the need for automatic cloze question generation. an important challenge in the preparation of cloze questions lies in the selection of appropriate wrong options (distractors). carefully designing distractors is crucial for enhancing the effectiveness of learner ability assessment, but it also requires significant time and effort. as a result, there has been a growing motivation to explore automatic distractor generation (dg) techniques. the paradigm for cloze dg is the candidate generating-and-ranking (cgr) framework. the cgr paradigm consists of two stages/components: (1) candidate generator and (2) candidate selector. the candidate generator is generally based on knowledge bases (such as probase @xcite or pre-trained language model @xcite ) to have a distractor candidate set, and the candidate selector ranks the candidates by linguistic features (e.g., morphological, pos, word embedding similarity). the sota methods @xcite question stem i was in a _ to reach my office options (a) hurry, (b) way, (c) dream, (d) deferral while the cgr framework shows promise, it overlooks the importance of the item discrimination index @xcite when evaluating the quality of questions. when teachers design multiple-choice questions (mcqs), it is crucial to consider the validity of the test questions by including distractors of varying difficulty levels. for example, in a four-option mcq, one option may be easily eliminated, while the remaining two options pose a greater challenge in distinguishing the correct answer, as shown in our study incorporates the following notable designs. first, we introduce a formulation that treats cloze distractor generation as a text2text task. as demonstrated in the experiment section, this approach yields a significant improvement in performance compared to traditional cgr methods. second, we propose the utilization of the "pseudo kullback-leibler divergence" technique to regulate the inter-correlation between the generated distractors. this ensures the diversity and relevance of the distractors. third, we investigate two additional strategies: the "candidate augmentation" strategy and the "multi-tasking training with cloze-related tasks" approach, both of which aim to further enhance the generation performance. distractor level answer type method type model word/phrase sentence cloze r.c. extractive generative type gao et al. 2019 y y y y rnn zhou et al. 2019 y y y y rnn araki et al. 2016 y y y non-neural model welbl et al. 2017 y y y random forests guo et al. 2016 y y y word2vec kumar et al. 2015 y y y y svm liang et al. 2017 y y y gan liang et al. 2018 y y y y non-neural model chung et al. 2020 y y y plm ren and q. zhu 2021 y y y knowledge-base peng et al. 2022 y y y plm chiang et al., 2022 y y y plm this work y y y text2text the rest of this paper is organized as follows. section 2 reviews the works of automatic distractor generation in the literatures. in section 3 we present the proposed methods. section 4 reports the performance evaluation and section 5 concludes this work and discuss the future work. ## related work in this section, we review the literature related to this work. ## methodology our approach employs a two-stage training process. in the first stage (subsection 3.1), we utilize a text2text framework to generate distractors. this involves training the model to generate plausible distractors based on a given cloze question and its corresponding answer. in the second stage (subsection 3.2), we introduce pseudo kl-divergence as a means to regulate the generation of distractors. this step is crucial for ensuring the validity of testing when designing multiple-choice questions (mcqs). by incorporating this technique, we aim to control the quality and relevance of the generated distractors. furthermore, we delve into the exploration of boosting techniques in subsections 3.3 and 3.4. these techniques are intended to enhance our overall approach. they may play a role in improving the distractor generation process or optimizing the design of mcqs. figure 2: candidate augmented sentence level generation ## conclusion in this paper, we introduce the utilization of a text2text formulation for generating clozestyle multiple-choice questions. our experimental results highlight a significant performance improvement achieved through the adoption of the text2text formulation. specifically, our approach yields a nearly two-fold increase in performance compared to the current state-of-the-art method. these results strongly suggest that the generative text2text framework represents a superior alternative to the traditional candidate generating-andranking (cgr) framework.