Automatic Generation of Distractors for Fill-in-the-Blank Exercises with Round-Trip Neural Machine Translation (2022)

Abstract: In a fill-in-the-blank exercise, a student is presented with a carrier sentence with one word hidden, and a multiple-choice list that includes the correct answer and several inappropriate options, called distractors. We propose to automatically generate distractors using round-trip neural machine translation: the carrier sentence is translated from English into another (pivot) language and back, and distractors are produced by aligning the original sentence and its round-trip translation. We show that using hundreds of translations for a given sentence allows us to generate a rich set of challenging distractors. Further, using multiple pivot languages produces a diverse set of candidates. The distractors are evaluated against a real corpus of cloze exercises and checked manually for validity. We demonstrate that the proposed method significantly outperforms two strong baselines.

## introduction a cloze (fill-in-the-blank) exercise is a common method of teaching vocabulary, as well as assessing non-native speaker performance in a foreign language: a passage (sentence) is presented to the learner with one word (target) being removed. the target word is presented along with a list of distractors (usually 3), and the task is to correctly identify the target word from that list. a valid distractor is a word that does not fit the context. for example, "great" and "utmost" are invalid distractors, since they both fit the context. given a carrier sentence and the target word, the problem is to generate challenging distractors. in typical high-stakes tests, such as test of english as a foreign language (toefl), distractors are generated manually by educational testing experts, a time-consuming procedure. an automated method to generate distractors would be extremely valuable. the problem becomes more challenging once the exercises are aimed at high-proficiency learners, since distractors that are not semantically close to the target word or grammatically unfit will be too easy for advanced speakers @xcite . to address this, previous work used context-sensitive inference rules @xcite , common collocation errors from largescale learner corpora @xcite , cooccurrence likelihoods @xcite , and word embeddings @xcite . in this work, we propose to generate distractors using round-trip neural machine translation (mt). word choice errors are commonly affected by the speaker's first language, and even advanced learn-ers struggle with word usage nuances and may inappropriately use semantically related words @xcite . our assumption is that lexical challenges common with non-native speakers will also manifest themselves in the round-trip machine translation as back-translated words that are semantically close to the target. such words should therefore serve as challenging distractors for advanced learners. unlike previous work, this method also opens up a possibility of customizing the cloze task for speakers of different languages. we focus on exercises aimed at advanced english as a second language (esl) learners. a carrier sentence is translated from english into another pivot language, where top n translation hypotheses are generated. for each hypothesis, top m backtranslations into english are generated. the backtranslated words aligned to the target are treated as potential distractors. we use five round-trip mt systems and show that using multiple pivot languages encourages diversity in the distractor generation, as the distractors produced with different pivot language systems are often unique. using a corpus of cloze exercises for advanced esl learners, we demonstrate that the proposed method retrieves over 31% of the gold distractors used in the exercises and over 70% percent of cloze items have at least one gold distractor retrieved with our approach. evaluation shows that the proposed method outperforms two strong baselinesthe word embeddings approach (word2vec) and bert. manual evaluation of the distractor validity indicates that over 72.3% of all distractors are valid with our approach compared to 56.1% and 38.0% using word2vec and bert, respectively. our contributions are as follows: (1) we propose to use round-trip machine translation to generate challenging distractors for cloze exercises and tests. we use hundreds of round-trip translations and multiple pivot languages, and generate challenging diverse distractors; (2) we validate our approach using a dataset of real cloze exercises for advanced esl learners and show that it significantly outperforms the word2vec and bert baselines both in automatic and manual evaluation; (3) unlike previous work, we find that different pivot languages provide rather unique distractors for the same item, thereby allowing for customizing the exercises on the basis of the native language of the student. the next section presents related work. section 3 describes the dataset of cloze exercises. sec-tion 4 describes the baseline methods, and section 5 presents our approach. section 6 presents the results of the automatic and manual evaluation of the generated distractors. section 7 further discusses the results, while section 8 concludes. ## related work the general approach to automatic distractor generation can be broken down into candidate generation (identification), and candidate ranking. candidate generation most of the work on automatic distractors focuses on generating distractor candidates. these include word frequency, phonetic and morphological similarity, and grammatical fit @xcite @xcite . for advanced speakers, distractors should be picked more carefully, so that they are reasonably hard to distinguish from the target. consider, for example, the target word "error" in the carrier sentence: "it is often only through long experiments of trial and error that scientific progress is made." the word "mistake" is semantically close to it but is not appropriate in the sentence context, and thus could serve as a valid distractor. however, note that "mistake" can be substituted for "error" in the context of "he made a lot of mistakes in his test." and would therefore not be a valid distractor. thus, on the one hand, challenging distractors should be semantically close to the target word, yet, on the other hand, a valid distractor should not produce an acceptable sentence. most of the approaches to generating challenging distractors rely on methods of semantic relatedness, such as n-grams and collocations @xcite , thesauri @xcite , or wordnet @xcite . @xcite use semantic contextsensitive inference rules. @xcite propose generating distractors using errors mined from a learner corpus. the approach, however, assumes an annotated learner corpus, and is quite limited, as both the choice of the target word and of the distractors are constrained by the errors in the corpus. several recent studies showed that word embeddings are effective in distractor generation: jiang and lee (2017) and @xcite generated distractors using semantically similar words obtained from word2vec @xcite . we propose to use round-trip neural machine translation to generate distractors. the only previ-ous mention of using mt is that of @xcite who aim at correcting esl collocation errors using a statistical machine translation technique. to the best of our knowledge, ours is the first dedicated study that uses state-of-the-art nmt systems with 5 pivot languages and large sets of back-translations for generating distractors. several studies, while they do not generate distractors, address the complexity of the cloze task for language learners. @xcite buttery (2019 et al. (2020) show that bert is helpful in measuring the fit of the distractor in the context, and thus can be used for estimating distractor difficulty. finally, we also note that there is a significant body of work on a task of generating reading comprehension (rc) items, that test a different set of examinee abilities, such as inference. that work @xcite deals with generating phrases and complete sentences for distractors. rc item generation is a distinct problem from vocabulary item generation that is addressed in this work. candidate ranking can be used as an additional step to (re-)rank the candidates produced during candidate generation. one reason for this is that context is typically not taken into account when generating candidates. yeung et al. (2019) used bert @xcite to re-rank the candidate distractors generated with word2vec for chinese. we show that bert is not effective at generating or re-ranking candidate distractors. ## data it is important to note that there is no benchmark dataset for the task. previous studies evaluate either on artificially created items with random words as targets or proprietary data. in contrast, we obtain cloze exercises from a reputable test preparation website, esl lounge. 2 the website contains study materials and preparatory exercises for esl tests, such as fce first certificate, toefl, and international english language testing system (ielts). there was significant effort put into the development of the exercises, which were manually curated for esl students, and the exercises are of high quality. this is the first dataset that can be used by researchers working on the task. 3 since we wish to generate distractors for advanced learners, we use the c1 advanced level multiple choice cloze exercises. foot_3 c1 level is part of cefr scale. 5 it is used to prove high-level achievement in english and is designed for learners preparing for university or professional life. we extract a total of 142 cloze items. 6 each item consists of a carrier sentence with the target word removed and is accompanied by four word choices that include the target word and three distractors. we show two sample items in ## the baselines we compare the round-trip mt method against word2vec and bert. both word2vec embeddings and bert can be used to generate candidates, and to rank candidates generated with mt. here, we describe how we generate candidates with word2vec and bert. in section 5.3, we describe how we use the two methods for candidate ranking. using word2vec, we generate words that have the highest similarity to the target word and use these as potential distractors. we use the 300-dimensional word2vec embeddings trained on google news. for a given target word, we find k nearest neighboring words using cosine similarity in the word embedding space. with bert, we produce a set of candidates by passing the carrier sentence with the target word replaced by a masked token. bert returns a list of words that best fit the context of the carrier sentence at the position of the masked token. each word is associated with probability; we select the top k candidates with the highest scores. the candidates are filtered out using the same filtering algorithm applied in round-trip mt (see section 5.2). in addition, we filter out misspellings by using a wordlist of about 130,000 english wordforms. sentence: much of the neighbourhood was demolished in the 1940s when living ______ had deteriorated. choices: situations, conditions*, circumstances, states sentence: scientists are yet to understand the full nutritional ______ of the humble olive. choices: favours, helps, goods, benefits* ## generating distractors with neural mt formally, given a sentence @xmath0, x 2 , . . . , x n } and a position @xmath1, n] of the target word, the task is to generate a set of candidate distractors d such that @xmath2. since challenging distractors should be more similar to the target word @xcite , and because many word sense nuances are challenging for non-native speakers due to the differences between word usage in their native language and in english, we expect that candidates generated with roundtrip mt that uses the target word together with the surrounding context will make good distractors for advanced esl learners. ## evaluation we evaluate the generated distractors using both automatic and manual evaluation. ## analysis and discussion we further analyze the distractors generated with round-trip mt. first, we examine the gold distractors that have not been identified with the mt approach. we find that some gold distractors are not semantically close to the target. next, we focus on the differences between the distractors generated with word2vec, bert, and mt, and show an example that demonstrates the ability of round-trip mt to model sentential context. first example in the second example compares bert-generated and mt-generated distractors: while not all of the mt distractors are valid, bert is more likely to generate candidates that are synonymous with the target, and thus are invalid as distractors. in fact, @xcite successfully use bert for the task of lexical substitution, while @xcite use bert for lexical simplification. the idea of using bert in such tasks is to provide good substitutes that are close synonyms in the given context. this is precisely the opposite of our goal: difficult distractors for a gap-filling task should not be substitutes of the target word. finally, the example below demonstrates that mt systems are capable of generating unique pivotdependent distractors. consider the carrier sentence "despite being such a frequent visitor to paris, sam never bored of exploring it." with the sentence: the ______ of thieves had been captured. target word: band; gold distractors: bunch, crew, range top-5 word2vec distractors: keyboardist, vocalist, drummer, quintet, guitarist round-trip mt distractors: crew, group, orchestra, gang, squad sentence: the ______ of the report have yet to be analysed by the government so they can formulate new policies. target word: findings; gold distractors: invenstions, discoveries, rulings top-5 bert distractors: recommendations, assertions, observations, results, conclusions round-trip mt distractors: outcomes, familiarities, shows, results, achievements ## conclusion we present a novel approach to generating challenging distractors for cloze exercises using round-trip neural machine translation. we show that using multiple pivot systems and a large set of round-trip translations produces diverse candidates, and each pivot language contributes unique distractors. this opens up a possibility of customizing the cloze generation task for speakers of different languages (groups), an interesting promise that bert-based and other models cannot do. we conducted a thorough evaluation of the distractors, using a set of real cloze exercises for advanced esl learners. comparison with word2vec and bert showed that the round-trip mt retrieves substantially more gold distractors given the same size of the candidate set. for future work, we will focus on customizing distractors based on the learner's native language, by prioritizing that language as pivot for mt. we will also conduct a study with language learners to determine whether the automatic distractors produced with our approach result in cloze items of the same difficulty as those that use gold distractors. for the current work for english, we used highquality machine translation systems. however, for many language pairs that do not include english as one of the languages, high-quality mt systems are not available. further, high-quality mt systems are also rarely available for low-resource languages paired with english. the future work will also focus in determining whether and how translation quality might affect the quality of generated distractors. we hypothesize that the proposed method might require special approaches when used to develop exercises for languages other than english and when generating english distractors using lowresource pivots. this is another exciting direction for future work.