Using Neural Machine Translation for Generating Diverse Challenging Exercises for Language Learner (2023)

Abstract: We propose a novel approach to automatically generate distractors for cloze exercises for English language learners, using round-trip neural machine translation. A carrier sentence is translated from English into another (pivot) language and back, and distractors are produced by aligning the original sentence with its round-trip translation. We make use of 16 linguistically-diverse pivots and generate hundreds of translation hypotheses in each direction. We show that using hundreds of translations allows us to generate a rich set of challenging distractors. Moreover, we find that typologically unrelated language pivots contribute more diverse candidate distractors, compared to language pivots that are closely related. We further evaluate the use of machine translation systems of varying quality and find that better quality MT systems produce more challenging distractors. Finally, we conduct a study with language learners, demonstrating that the automatically generated distractors are of the same difficulty as the gold distractors produced by human experts.

## introduction a common challenge for language learners involves understanding how to appropriately use words that may have similar meanings but are used in different contexts. for instance, "main" and "vital" are semantically related but "main importance" is not an acceptable expression while "vital importance" is. this subtle language knowledge is not explicitly available to learners. for this reason, word usage (collocation) errors are some of the most common types of errors even for advanced non-native speakers @xcite . in this work, we develop exercises for mastering vocabulary use for second (foreign) language learners. we focus on cloze (fill-in-the-blank) exercises. a cloze exercise is a common method of teaching vocabulary, as well as assessing non-native speaker performance in a foreign language: a sentence is presented to the learner with one word (target) hidden. the target word is presented along with a list of distractors (usually 3), and the learner should identify the target word from that list. given a carrier sentence and the target word, the problem is to generate distractors. distractors are typically created manually by educational testing experts, a time-consuming procedure. the problem becomes more challenging once the exercises are aimed at high-proficiency learners, since distractors that are not semantically close to the target word or are grammatically unfit will be too easy for them @xcite . we propose to generate distractors using roundtrip neural machine translation (mt). robust machine translation systems exist today for many language pairs. while translations produced with modern automated systems are reasonably good, these are not perfect, and, while a round-trip translation may preserve the sentence meaning, it will often not result in the exact same sentence. we use this observation to develop an approach to automatically generate distractors for cloze exercises. we focus on exercises aimed at advanced english as a second language (esl) learners. a carrier sentence is translated from english into another pivot language, where the top n translation hypotheses are generated. for each hypothesis, the top m back-translations into english are generated. each back-translation is aligned with the original sentence, and the back-translated word aligned to the target is treated as a potential distractor. the intuition behind the approach is that word choice errors are commonly affected by the learner's first language. in particular, the different meanings (or contextual uses) of an ambiguous word in the learner's native language may lead to different word choices in english. the assumption thus is that lexical challenges that are common with non-native speakers will also manifest themselves in the round-trip machine translation as backtranslated words that are semantically close to the target. such words should therefore serve as challenging distractors for advanced learners. unlike previous work, this method also opens up a possibility of customizing the cloze task for speakers of different languages. this work builds on a pilot study @xcite that made use of five round-trip mt systems. however, the pivots used in the study were closely related languages spoken in europe. in addition, the study did not evaluate the difficulty of the automatic distractors and did not test these with language learners. in this paper, we use 16 language pivots from a diverse set of linguistic families and conduct a thorough evaluation of the proposed method, using a dataset of real cloze exercises for advanced learners. our contributions are as follows: (1) we use mt systems of varying levels of quality. we show that, while poor mt systems generate a larger pool of candidate distractors, high quality systems tend to produce more challenging distractors that are semantically close to the target word; (2) we evaluate the approach as a function of using pivots from different language families and show that pivot languages that are typologically distant contribute more diverse distractors; (3) we conduct a human study with 32 advanced language learners and show that the generated distractors are of the same difficulty as distractors created by experts. the rest of the paper is organized as follows. the next section presents related work. section 3 describes the dataset of cloze exercises. section 4 describes our approach. section 5 presents the evaluation of the approach along several dimensions. section 6 describes the human study. section 7 concludes, by outlining avenues for future work and discussing the limitations of the study. ## related work previous work on distractor generation made use of word frequency, phonetic and morphological similarity, and grammatical fit @xcite @xcite . for advanced speakers, distractors should be selected more carefully, so that they are reasonably hard to distinguish from the target. consider, for example, the target word "error" in the carrier sentence: "it is often only through long experiments of trial and error that scientific progress is made." the word "mistake" is semantically close to it but is not appropriate in the sentence, and thus could serve as a valid distractor. however, note that "mistake" can be substituted for "error" in the context of "he made a lot of mistakes in his test." and would therefore not be a valid distractor in that context. thus, challenging distractors should be semantically close to the target word, yet, a valid distractor should not produce an acceptable sentence. most of the approaches to generating challenging distractors rely on semantic relatedness, computed through n-grams and collocations @xcite , thesauri @xcite , or wordnet @xcite . zesch and melamud (2014) use semantic context-sensitive inference rules. @xcite propose generating distractors using errors mined from a learner corpus. the approach, however, assumes an annotated learner corpus, and both the choice of the target word and of the distractors are constrained by the errors in the corpus. several studies showed that word embeddings are effective in distractor generation @xcite @xcite . our work builds on a study that employed five pivot languages @xcite , showing that the round-trip mt approach outperforms two strong baselines -word2vec and bert (section 5.4 and appendix b provide more detail on the comparison of the mt approach with these methods). the present study focuses on an in-depth evaluation of the mt approach to distractor generation along several dimensions. ## data we obtain cloze exercises from a reputable test preparation website, esl lounge. 2 the website contains study materials and preparatory exercises for esl tests, such as fce first certificate, toefl, and international english language testing system (ielts). there was significant effort put into the development of the exercises, which were manually curated for esl students, and the exercises are of high quality. this is the first dataset that can be used by researchers working on the task. 3 previous studies thus evaluate either on artificially created items or on proprietary data. we use the advanced level multiple choice cloze exercises, which includes 142 cloze items. 4 each item consists of a carrier sentence with the target word removed and is accompanied by four word choices that include the target word and three distractors provided by human experts. we refer to these distractors as gold distractors. we first translate the sentence x from english using a forward mt system s en-trg to obtain a set of top n f translation hypotheses @xmath0, y 2 , . . . , y n f } in the target language trg. we then translate the sentences in y using a backward mt system s trg-en and obtain a set of top n b translation hypotheses for y @xmath1. finally, we our earlier study included five indo-european languages: german, russian, italian, french, and czech. presently, we include 16 languages from a diverse set of language families. for all language pairs, we use competitive neural mt systems of @xcite . alignment computation given a round-trip translation x rt i for carrier sentence x, we compute the alignment between the two sentences. the word in x rt i that is aligned to the target word in x is considered to be the back-translation of the target and can be a potential distractor. we use simalign foot_3 @xcite ) that employs contextual word embeddings @xcite to produce an alignment model for a pair of sentences. given the original sentence x and a round-trip translation x rt i , the similarity between each token in x with each token in x rt i is computed, using contextual embeddings from multilingual bert. ## evaluation we evaluate the mt approach to distractor generation along 4 dimensions: (1) comparing the effect of using typologically diverse language pivots; (2) using mt systems of various quality; (3) using different number of translation hypotheses in the forward and backward direction; (4) evaluating the diversity of distractors produced with linguistically related versus linguistically unrelated pivots. evaluation for the distractor generation task is not straightforward, since the set of valid distractors for a given exercise item is not uniquely defined. for this reason, automatic evaluation against the set of distractors proposed by human experts does not provide a full picture of the quality of the generated distractors. thus, we conduct several types of evaluation. first, we compare the generated distractors against the set of gold distractors for each item, making the assumption that a method that retrieves a higher percentage of gold distractors among its automatic candidates is better. second, we conduct manual annotation with native english speakers to determine the percentage of valid distractors among the candidates proposed by mt: although filtering removes a majority of invalid candidates, there are still candidates that remain due to filtering errors. third, we evaluate the difficulty of the generated distractors by annotating the distractors for their semantic similarity to the target. our final test with language learners in section 6 assesses the difficulty of the automatic distractors generated using the best settings for mt, as compared to the difficulty of gold distractors. ## study with language learners to evaluate the difficulty of automatically generated distractors, we conduct a cloze exercise test with english learners. we use a pool of manually validated items from the 30.30 setting and the pivots in group 1 to create a cloze test for participants. manual validation ensured that all of the automatically generated candidates are valid. we sample 32 exercise items uniformly at random from the pool. participants our participants are adult nonnative english speakers of diverse language backgrounds. to ensure that the participants are advanced learners, we asked them to provide their gold distractors retrieved word2vec bert mt 39 (9.2%) 97 (22.8%) 136 (31.9%) toefl or ielts scores. we also gave them a sample test to complete to exclude those whose english was too good or not good enough. participants were informed that the results of their tests would be used to collect statistics for research, without disclosing personal information. participants were provided with $25 gift cards. cloze exercise setup we create two versions of a cloze test with the same set of 32 carrier sentences. each version contains 16 sentences with gold distractors and 16 sentences with automatic distractors. the sentences that come with gold distractors in the first version, come with automatic distractors in the second version of the test, and vice versa. the order of the cloze items in each version is randomized. additionally, we ensure that for each item the target always appears in the same position with both gold and automatic distractors on the multiple-choice list. each version of the test was completed by exactly 16 participants, so each cloze item was completed by 16 learners who were given gold distractors, and by another group of 16 learners who received automatic distractors. we use the first 2 cloze items as training items, to help the test takers familiarize themselves with the task. the statis- we set up the test in a user interface setting, where a participant can see the carrier sentence and the four choices on the screen and has to pick one choice. as part of the test instructions, the participants were asked not to leave the response blank. we asked the participants not to get help from external resources to solve the exercise. the participants took between 20 to 30 minutes to complete the test. paired t-test a paired t-test was used to compare the human performance on cloze items with gold and automatic distractors. for computing the paired t-test statistics, we use the 30 cloze items that were not used as training items, and compare scores of gold vs. automatic distractors used, where the score is defined as proportion of participants that correctly solved the item. there was no significant difference in the scores of gold distractors (with mean 9.57, standard deviation 3.83) and automatic distractors (with mean 10.23, standard deviation 3.47). the two-tailed p value is 0.2884. these results suggest that the scores on cloze items using gold distractors and automatic distractors are not significantly different. specifically, our results show that when automatic distractors are used in the cloze items instead of gold distractors, the difficulty of the cloze items remains the same. ## conclusion we present a novel approach to generate challenging distractors for cloze exercises with round-trip neural mt. we show that using multiple pivot systems and a large set of round-trip translations produces diverse candidates, and each pivot contributes unique distractors. the latter opens up a possibility of customizing the cloze task for speakers of different languages, by tying the pivot choice to the learner's native language, an interesting promise that bert-based and other models cannot do. we conduct a thorough evaluation of the distractors, using a set of real cloze exercises for advanced esl learners. finally, we conduct a study with language learners that demonstrates that the automatic distractors produced with our approach result in cloze items of the same difficulty as those that use gold distractors. for future work, we will focus on customizing distractors based on the learner's native language, by prioritizing that language as pivot for mt.