CDGP : Automatic Cloze Distractor Generation based on Pre-trained Language Model (2022)

Abstract: Manually designing cloze test consumes enormous time and efforts. The major challenge lies in wrong option (distractor) selection. Having carefully-design distractors improves the effectiveness of learner ability assessment. As a result, the idea of automatically generating cloze distractor is motivated. In this paper, we investigate cloze distractor generation by exploring the employment of pre-trained language models (PLMs) as an alternative for candidate distractor generation. Experiments show that the PLM-enhanced model brings a substantial performance improvement. Our best performing model advances the state-of-the-art result from 14.94 to 34.17 (NDCG@10 score). Our code and dataset is available at https://github.com/AndyChiangSH/CDGP.

## introduction a cloze test is an assessment consisting of a portion of language with certain words removed (cloze text), where the participant is asked to select the missing language item from a given set of options. specifically, a cloze question (as illustrated in figure 1 ) is composed by a sentence with a word removed (a blank space) and list of options (one answer and three wrong options). the cloze test with carefully-design distractors can improve the effectiveness of learner ability assessment. however, manually designing cloze test consumes enormous time and efforts. the major challenge lies in wrong option (distractor) selection. as a result, automatic cloze distractor generation is proposed @xcite @xcite . in this paper, we extend the candidate-ranking framework reported in @xcite by exploring the employment of plms as an alternative for candidate distractor generation. in this paper, we propose a cloze distractor generation framework called cdgp (automatic cloze distractor generation based on plms) which incorporates a serial of training and ranking strategies to boost the performance of distractor generation based on plms. the contribution of this work is as follows. ## related work the methods on cloze distractor generation can be sorted into the following two categories. the first category @xcite is to prepare cloze distractors based on linguistic heuristic rules. the problem with these methods is that the results are far from practically satisfactory. the second category @xcite is to construct candidate distractors from domain-specific vocabulary or taxonomies and employ classifiers for selecting final distractors. the results by the methods of this category are still less than satisfactory due to the domain generalization and the generation quality. to improve the quality, @xcite proposes to use knowledge bases (wordnet @xcite and probase @xcite ) to analyze the word semantic and hypernym-hyponym relations for generating candidate distractors. in this paper, we explore the employment of plms as a alternative for the knowledge bases in @xcite and also explore various linguistic features for candidate selection. ## conclusion our study indicates that plm-based candidate distractor generator is a better alternative for knowledge-based component. the experiment results show that our model significantly surpassed the sota method, demonstrating the effectiveness of plm-based distractor generation on cloze test. also, the result shows that using domain-specific plm will further boost the generation quality.