BERT -based distractor generation for S wedish reading comprehension questions using a small-scale dataset (2021)

Abstract: An important part when constructing multiple-choice questions (MCQs) for reading comprehension assessment are the distractors, the incorrect but preferably plausible answer options. In this paper, we present a new BERT-based method for automatically generating distractors using only a small-scale dataset. We also release a new such dataset of Swedish MCQs (used for training the model), and propose a methodology for assessing the generated distractors. Evaluation shows that from a student’s perspective, our method generated one or more plausible distractors for more than 50% of the MCQs in our test set. From a teacher’s perspective, about 50% of the generated distractors were deemed appropriate. We also do a thorough analysis of the results.

## introduction multiple-choice questions (mcqs) are widely used for student assessments, from high-stakes graduation tests to lower-stakes reading comprehension tests. an mcq consists of a question (stem), the correct answer (key) and a number of wrong, but plausible options (distractors). the problem of automatically generating stems with a key has received a great deal of attention, e.g., see the survey by @xcite . by comparison, automatically generating distractors is substantially less researched, although @xcite report that manually finding reasonable distractors was the most time-consuming part in writing science mcqs. indeed, reasonable distractors should be grammatically consistent and similar in length compared to the key and within themselves. given the challenges above, we attempt using machine learning (ml) to aid teachers in creating distractors for reading comprehension mcqs. the problem is not new, however most of the prior work has been done for english. in this paper we propose the first such solution for swedish (although the proposed method is novel even for english, to the best of our knowledge). the key contributions of this work are: proposing a bert-based method for generating distractors using only a small-scale dataset, releasing swequad-mc foot_0 , a dataset of swedish mcqs, and proposing a methodology for conducting human evaluation aimed at assessing the plausibility of distractors. ## data we have collected a swedish dataset, henceforth referred to as swequad-mc, consisting of texts and mcqs for the given texts. the dataset was created by three paid linguistics students instructed to pose unambiguous and independent questions. they were also asked to identify the key with at least two distractors, all of which are contiguous phrases in a given text. additionally, as the distractors were required to be in the same grammatical form as the key (e.g., both in plural), the students were allowed to change the grammatical form of phrases if they constituted plausible distractors after this change. the exact instructions given to the students along with more details on the used texts are provided in appendix a. each datapoint in swequad-mc consists of a base text and an mcq, i.e. a stem, the key and at least two distractors. the same text can be reused for different mcqs, but the sets of texts in training (∼ 80%), development (∼ 10%) and test (∼ 10%) datasets are disjoint. however, some overlap in sentences is possible, since the texts might come from the same source. descriptive statistics of all swequad-mc splits is provided in ## method given the small scale of swequad-mc we have decided to fine-tune a pretrained bert foot_1 for swedish @xcite on the task of distractor generation (dg). for achieving this, we have added on top of bert two linear layers with layer normalization @xcite in the middle to be trained from scratch (see architecture in figure 1 ). the last linear layer is followed by a softmax activation giving probabilities over the tokens in the vocabulary for each position in the text. we trained the model using cross-entropy loss only for tokens in masked positions. recall that each mcq consists of a base text t, the stem q based on t, the key a and (on average) two distractors d1 and d2. the dg problem is then to generate distractors conditioned on the context, consisting of t, q and a. we provide all context components as input to the bert model, separated from each other by the special separator token @xcite . given that bert's maximum input length is 512 tokens, we trim t to the first 384 tokens (later referred to as t 384), since that is the average text length of the training set. we have explored two different solution variants of dg. the first variant aims at generating distractors autoregressively, left to right. at generation time, the input to bert consists of a context ctx (t 384, q and a separated by [sep] token), a [sep] token, and a [mask] token at the end. after a forward pass through bert, the [mask] token gets replaced by the word with the highest softmax score, which becomes the first word of the first distractor (dubbed d11). the generation of the first distractor continues by appending a [mask] token after each forward pass until the network generates a separator token @xcite , which concludes the generation of the first distractor d1. the next distractor d2 is generated in the same way, except that the ctx is extended by d1. at training time, we use the same procedure, but with teacher forcing, allowing us to use the correct distractor tokens as targets for the cross-entropy loss (see example training datapoints for one mcq in the second variant is inspired by u-pmlm, and aims at generating distractors autoregressively, but in an arbitrary word order. at generation time, the input to bert consists of a context ctx, a [sep] token, and a predefined number of [mask] tokens (see section 6.1). the generation proceeds by unmasking the token at the position where the model is most confident. this differs from unmasking a random position, proposed by @xcite . the training procedure largely follows a masking scheme employed by u-pmlm by drawing the masking ratio from the uniform distribution (see example training datapoints for one mcq in linear gelu linear softmax layernorm bert (b, 512, 768) (b, 512, 768) (b, 512, 768) (b, 512, v) [cls] t [sep] q [sep] a [sep] d11 d12 [mask] [cls] t [sep] q [sep] a [sep] [mask] d12 [mask] left to right u-pmlm d13 d11 d13 left to right u-pmlm cross-entropy loss ## experimental setup we have used huggingface's transformers library @xcite for implementing the dg model. the training hardware setup included 16 intel xeon cpu e5-2620 v4 (2.10ghz), 64 gb of ram and 1 nvidia geforce rtx 2080 ti (11 gb vram). for this setup, we have fixed the random seed to 42, the number of training epochs to 6, the batch size to 4 (for both training and dev sets) and max maskings to 20 (for u-pmlm variant only). with these settings, training took about 3.67h for the left-to-right and 3h for the u-pmlm variant. ud trees for the baseline were obtained using stanza package @xcite and convolution partial tree kernels on the ud trees were calculated using udon2 library @xcite . baseline requires no training and running our implementation of the baseline takes about a minute on the development or test set. ## evaluation following the analysis of rodriguez (2005), we generate three distractors per mcq for each model. due to prohibitively high costs of human evaluation, we have divided the evaluation process into two stages. the first stage is quantitative evaluation, which gives limited information about the model's quality, but is sufficient for model selection. the second stage is human evaluation of the best model, selected during the first stage. ## related work we employed a systematic process to get a comprehensive overview of dg methods (see appendix e for more details). out of the resulting 28 articles (see an overview in in this paper we work on reading comprehension mcqs, which makes only 12 papers, dealing with factual questions, relevant. two of these used rule-based approaches. majumder and saha (2015) generated mcqs for cricket domain and used a number of hand-crafted rules based on gazeteers and wikipedia entries to generate distractors. @xcite proposed to generate distractors for mcqs on electronic instructional documents using wordnet. six of these relied on extractive approaches. @xcite , @xcite , and ha and yaneva (2018) formulated choosing a distractor as a ranking problem from the given candidate set. in the first two articles the candidate set constituted all distractors from the available mcq dataset. the authors then trained ml-based ranker(s) for choosing the best distractors. in the last one, the candidate set was created using content engineers. distractors with a high similarity of their concept embeddings (summed for multiple words) and appearing in the same document as the key are ranked higher. @xcite and @xcite worked in the domain of biology. the former used an ontology and the latter employed event graphs containing information about coreferences to generate distractors. @xcite used thesaurus and tf-idf to identify key concepts in the given text and then select as distractors those having the same semantic type as the key. the remaining four employed neural methods and are most relevant among the surveyed. qiu et al. (2020) trained a sequence-to-sequence (seq2seq) model with a number of attention layers. zhou et al. (2020) also employed a seq2seq model, but with a hierarchical attention to capture the interaction between a text and a question, as well as semantic similarity loss. both articles used a beam search combined with filtering based on jaccard coefficient at generation time. offerijns et al. (2020) trained a gpt-2 model to generate 3 distractors for a given mcq, and used bert-based question answering model for quantitative evaluation (along with human evaluation). finally, chung et al. (2020) proposed a bertbased method for english with answer-negative regularization, penalizing distractors for containing problem/method property # extractive 14 generative, rule-based 7 generative, neural 7 only automatic evaluation 5 only human evaluation 19 automatic and human evaluattion 4 cloze-style, single-word answers 14 cloze-style, continue the sentence 2 factual questions 12 the same words as the key, and training a sequential and a parallel mlm model simultaneously. at generation time, they generate one distractor, and then create a distractor set of the predefined size based on sampling from the probability distribution returned by bert for each token of the distractor. then they rank every triple of distractors based on the entropy of a separately trained qa model. our method also relies on bert, but has a number of differences beyond being applied to swedish. firstly, we did not include answer-negative regularization, since it is not always a good strategy. for instance, given the stem "when should you pay a fee if you apply for a visa?" and a key "before you have submitted the application", the best distractor would be "after you have submitted the application", which shares most of the words with the key. secondly, we generate distractors in arbitrary word order compared to left-to-right generation in @xcite . thirdly, at generation time, we use previously generated distractors as input for generating next ones, and always take tokens with a maximum probability. this lowers the risk of generating ungrammatical distractors. finally, our training set is 100 times smaller compared to the training set used by @xcite . ## conclusion we have collected swequad-mc, the first dataset of swedish mcqs, and showed the possibility of training usable bert-based dg models, despite the small scale of the dataset. we have showed that a u-pmlm variant of the bert-based dg model performs best on the dataset, and proposed a novel methodology of evaluating the plausibility of generated distractors. around half of the generated distractors were found acceptable by the majority of teachers, and more than 50% of mcqs had at least one plausible generated distractor, judging by the entropy of students' responses. bearing in mind that the aim of the proposed method is to support (not replace) teachers, we deem that our method works well for mcqs in swedish (and potentially in other languages with a pretrained bert and a dataset of a similar scale). furthermore, we have presented a baseline applicable to any language with a ud treebank (currently about 100 languages). although its performance is nowhere near the u-pmlm variant, we believe that it can serve as a good point of comparison to emerging neural methods for other languages. system demonstrations, pages 38-45, online. association for computational linguistics. xiaorui zhou, senlin luo, and yunfang wu. 2020. coattention hierarchical network: generating coherent long distractors for reading comprehension. in proceedings of the aaai conference on artificial intelligence, volume 34, pages 9725-9732.