Distractor Analysis and Selection for Multiple-Choice Cloze Questions for Second-Language Learners (2020)

Abstract: We consider the problem of automatically suggesting distractors for multiple-choice cloze questions designed for second-language learners. We describe the creation of a dataset including collecting manual annotations for distractor selection. We assess the relationship between the choices of the annotators and features based on distractors and the correct answers, both with and without the surrounding passage context in the cloze questions. Simple features of the distractor and correct answer correlate with the annotations, though we find substantial benefit to additionally using large-scale pretrained models to measure the fit of the distractor in the context. Based on these analyses, we propose and train models to automatically select distractors, and measure the importance of model components quantitatively.

## introduction multiple-choice cloze questions (mcqs) are widely used in examinations and exercises for language learners @xcite . the quality of mcqs depends not only on the question and choice of blank, but also on the choice of distractors, i.e., incorrect answers. distractors, which could be phrases or single words, are incorrect answers that distract students from the correct ones. according to @xcite , distractors tend to be syntactically and semantically homogeneous with respect to the correct answers. distractor selection may be done manually through expert curation or automatically using simple methods based on similarity and dissimilarity to the correct answer @xcite . intuitively, optimal distractors should be sufficiently similar to the correct answers in order to challenge students, but not so similar as to make the question unanswerable @xcite . however, past work usually lacks direct supervision for training, making it difficult to develop and evaluate automatic methods. to overcome this challenge, @xcite sample distractors as negative samples for the candidate pool in the training process, and @xcite sample questions and use manual annotation for evaluation. in this paper, we build two datasets of mcqs for second-language learners with distractor selections annotated manually by human experts. both datasets consist of instances with a sentence, a blank, the correct answer that fills the blank, and a set of candidate distractors. each candidate distractor has a label indicating whether a human annotator selected it as a distractor for the instance. the first dataset, which we call mcdsent, contains solely the sentence without any additional context, and the sentences are written such that they are understandable as standalone sentences. the second dataset, mcdpara, contains sentences drawn from an existing passage and therefore also supplies the passage context. to analyze the datasets, we design context-free features of the distractor and the correct answer, including length difference, embedding similarities, frequencies, and frequency rank differences. we also explore context-sensitive features, such as probabilities from large-scale pretrained models like bert @xcite . in looking at the annotations, we found that distractors are unchosen when they are either too easy or too hard (i.e., too good of a fit in the context). consider the examples in when they divide, the new cells may be the same type of cell as the original cell.... plastic f ...these circumstances made it virtually impossible for salmon to mate. therefore, the number of salmon declined dramatically. we develop and train models for automatic distractor selection that combine simple features with representations from pretrained models like bert and elmo @xcite . our results show that the pretrained models improve performance drastically over the feature-based models, leading to performance rivaling that of humans asked to perform the same task. by analyzing the models, we find that the pretrained models tend to give higher score to grammatically-correct distractors that are similar in terms of morphology and length to the correct answer, while differing sufficiently in semantics so as to avoid unaswerability. ## datasets we define an instance as a tuple x, c, d, y where x is the context, a sentence or paragraph containing a blank; c is the correct answer, the word/phrase that correctly fills the blank; d is the distractor candidate, the distractor word/phrase being considered to fill the blank; and y is the label, a true/false value indicating whether a human annotator selected the distractor candidate. 1 we use the term question to refer to a set of instances with the same values for x and c. ## features and analysis we now analyse the data by designing features and studying their relationships with the annotations. ## models since the number of distractors selected for each instance is uncertain, our datasets could be naturally treated as a binary classification task for each distractor candidate. we now present models for the task of automatically predicting whether a distractor will be selected by an annotator. we approach the task as defining a predictor that produces a scalar score for a given distractor candidate. this score can be used for ranking distractors for a given question, and can also be turned into a binary classification using a threshold. we define three types of models, described in the subsections below. ## experiments we now report the results of experiments with training models to select distractor candidates. ## related work existing approaches to distractor selection use wordnet @xcite metrics @xcite , word embedding similarities @xcite , thesauruses @xcite , and phonetic and morphological similarities @xcite . other approaches consider grammatical correctness, and introduce structural similarities in an ontology @xcite , and syntactic similarities @xcite . when using broader context, bigram or n-gram co-occurrence @xcite , context similarity @xcite , and context sensitive inference @xcite have also been applied to distractor selection. based on these heuristic features, liang et al. (2018) assemble these features and apply neural networks, training the model to predict the answers within a lot of candidates. yeung et al. (2019) further applies bert for ranking distractors by masking the target word. as we have two manually annotated datasets that have different lengths of contexts, we adopt both word pair features and the context-specific distractor probabilities to build our feature-based models. moreover, we build both elmo-based and bert-based models, combining them with our features and measuring the impact of these choices on performance. ## conclusion we described two datasets with annotations of distractor selection for multiple-choice cloze questions for second-language learners. we designed features and developed models based on pretrained language models. our results show that the task is challenging for humans and that the strongest models are able to approach or exceed human performance. the rankings of distractors provided by our models appear reasonable and can reduce a great deal of human burden in distractor selection. future work will use our models to collect additional training data which can then be refined in a second pass by limited human annotation. other future work can explore the utility of features derived from pretrained question answering models in scoring distractors.