A BERT -based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies. (2020)

Abstract: In this paper, we investigate the following two limitations for the existing distractor generation (DG) methods. First, the quality of the existing DG methods are still far from practical use. There are still room for DG quality improvement. Second, the existing DG designs are mainly for single distractor generation. However, for practical MCQ preparation, multiple distractors are desired. Aiming at these goals, in this paper, we present a new distractor generation scheme with multi-tasking and negative answer training strategies for effectively generating multiple distractors. The experimental results show that (1) our model advances the state-of-the-art result from 28.65 to 39.81 (BLEU 1 score) and (2) the generated multiple distractors are diverse and shows strong distracting power for multiple choice question.

## introduction given a passage, a question, and an answer phrase, the goal of distractor generation (dg) is to generate context-related wrong options (i.e., distractor) for multiple-choice questions @xcite . pioneering research @xcite @xcite have demonstrated the feasibility of generating distractors based on deep learning techniques. while significant advances for dg were reported in the literature, we find that the existing dg results are still far from practical use. in this paper, we investigate the following two issues for distractor generation: (1) dg quality improvement and (2) multiple distractor generation. dg quality improvement there is still room to be improved for high-quality distractor generation. by manually examining the dg results generated by the existing method, we find that the results are still far from ideal for practical use. thus, one example 1 context omitted. (see appendix) question • why did mr.king want to send henry away? answer • because henry was too lazy. gen. distractors •d 1 : because henry didn't want to go. example 2 context omitted. (see appendix) question • which of the following women would look most attractive? answer • a short red-haired woman who wears a purple hat. gen. distractors •d 1 : a young woman who wears a white hat. for the quality issues, in this paper, we explore bert model's employment for performance improvement. as known, employing transformerbased language models has shown to be useful for improving nlp tasks. thus, we investigate the bert model's application for dg and report our design in this paper. ## related work the dg research can be categorized from different perspectives. first, for dg task type, there are two main task categories for dg: cloze-style distractor generation and reading comprehension (rc) distractor generation. in cloze-style dg task, it is viewed as a word filling problem. in general, the first step is to extract distractor candidates from context or some knowledge base, and then the next step is to rank the extracted distractors as a final result. along this direction, the models are mainly based on similarity heuristic @xcite @xcite or supervised machine learning way @xcite . the distractors generated for cloze-style dg are mainly word/phrase level. on the other hand, the rc-type qg focuses on generating sentence-level distractors for reading comprehension level testing, such as summarizing article or understanding author opinion @xcite . for the sentence-level distractors, neural models are commonly employed as it is difficult to generate a semantic rich and fluent distractor from question, content, and answer. in this paper, we also focus on generative sentence-level dg for rc task. however, as mentioned in the introduction, we find the existing dg results are still far from human level. the best sota result (in terms of bleu 1 score) is 29, which is far from the ideal result for practical use. aiming at this point, we explore the employment of transformer-based pre- example 1 context omitted. (see appendix) question • why did mr.king want to send henry away? answer • because henry was too lazy. bdg •d 1 : because henry didn't want to go. example 2 context omitted. (see appendix) question • which of the following women would look most attractive? answer • a short red-haired woman who wears a purple hat. bdg •d 1 : a young woman who wears a white hat. ## conclusion we present a state-of-the-art neural model based on a pre-trained transformer-based model for dg. we introduce two techniques, answer negative regularization and multi-task with parallel mlm, to boost the dg performance. in addition, we also introduce bdg ensemble with an entropy maximization mechanism to enhance the dg quality by