Automatic Distractor Generation for Multiple Choice Questions in Standard Tests (2020)

Abstract: To assess knowledge proficiency of a learner, multiple choice question is an efficient and widespread form in standard tests. However, the composition of the multiple choice question, especially the construction of distractors is quite challenging. The distractors are required to both incorrect and plausible enough to confuse the learners who did not master the knowledge. Currently, the distractors are generated by domain experts which are both expensive and time-consuming. This urges the emergence of automatic distractor generation, which can benefit various standard tests in a wide range of domains. In this paper, we propose a question and answer guided distractor generation (EDGE) framework to automate distractor generation. EDGE consists of three major modules: (1) the Reforming Question Module and the Reforming Passage Module apply gate layers to guarantee the inherent incorrectness of the generated distractors; (2) the Distractor Generator Module applies attention mechanism to control the level of plausibility. Experimental results on a large-scale public dataset demonstrate that our model significantly outperforms existing models and achieves a new state-of-the-art.

## introduction standard test, such as tofel and sat, is an efficient and essential tool to assess knowledge proficiency of a learner @xcite . according to testing results, teachers or its (intelligent tutoring system) services can develop personalized study plans for different students. when organizing a standard test, a vital issue is to select a suitable question form. among various question forms, multiple choice question (mcq) is widely adopted in many notable tests, such as gre, tofel and sat. mcqs have many advantages including less testing time, more objective and easy on the grader @xcite . a typical mcq consists of a stem and several candidate answers, among which one is correct, the rest are distractors. as shown in figure 1 , in addition to a stem, some tests also include a long reading passage to provide the context of this mcq. the quality of an mcq depends heavily on the quality of the distractors. if the distractors can not confuse students, the correct answer could be concluded easily. as a result, the discrimination of the question will degrade, and the test will also lose the ability of the assessment. however, it is a challenging job to design useful and qualified distractors. rather than being a trivial wrong answer, the distractor should have the plausibility which confuses learners who did not master the knowledge @xcite . a good distractor should be grammatically correct given the question and semantically consistent with the passage context of the question @xcite . meanwhile, the question composers need to enhance the plausibility of the distractor without hurting its inherent incorrectness. otherwise, the distractor easily becomes a definitely wrong answer, further making the question to be sloppy. hence, the manual preparation of distractors is time-consuming and costly @xcite . it is an urgent issue to automatically generate useful distractors, which can help to alleviate question composers' workload and relax the restrictions on experience. it could also be helpful to prepare a large train set to boost the machine reading comprehension (mrc) systems @xcite . in this paper, we focus on automatically generating semantic-rich distractors for mcqs in real-world standard tests, such as race @xcite which is collected from the english exams for chinese students from grades 7 to 12. ## zp10211059 passage: ... held on a farm, the glastonbury festival is the most well-known and popular festival in the uk. it began in 1970 and the rst festival was attended by one thousand ve hundred people each paying an admission price of ps1 ... since then the glastonbury festival has gone from strength to strength -in 2004 one hundred and fty thousand fans attended, paying ps112 each for a ticket to the three-day event. tickets for the event sold out within three hours. ... glastonbury is not unique in using live music to raise money to ght global poverty. ... ## related work in recent years, some research efforts have devoted to distractor generation. generally, the related work can be classified into the following three categories. feature-based methods. considerable research efforts @xcite @xcite have been devoted to using manual design features, such as pos features and statistic features, to generate the distractors. however, designing effective features is also labor intensive and hard to scale to various domains. differently, our work is an end-to-end framework without manual design features. similar word/entity-based methods. some works @xcite @xcite focused on finding answer-relevant ontologies or words as the distractors with the help of wordnet and word2vec. for example, @xcite leveraged an educational biology ontology to conduct the distractor generation. however, some of these works depend heavily on the well-designed ontology and they can only generate short distractors, which usually only contain one single word or phrase. nn-based methods. recently, neural network based and data-driven solutions emerge. @xcite proposed an end-to-end solution focusing on distractor generation for mcqs in standard english tests. they employed the hierarchical encoder-decoder network as the base model and used the dynamic attention mechanism to generate the long distractors. @xcite further strengthened the interaction between the question and the passage based on the model of @xcite . ## conclusion and future work in this paper, we propose a novel question and answer guided distractor generation(edge) framework to automatically generate distractors for multiple choice questions in standard english tests. in edge, we design two modules based on attention and gate strategies to reform the passage and question, which then are combined to decode the distractor. experimental results on a large-scale public dataset demonstrate the state-of-the-art performance of edge and the effectiveness of two reforming modules. in future work, we will explore two potential directions. first, since the beam search ignores the generation diversity, we will explore how to incorporate the prior generated distractor information to guide the generation of successor distractors. second, we will work on how to generate the distractors requiring multi-sentence/hop reasoning, which can further improve the plausibility.