Distractor Generation for Fill-in-the-Blank Exercises by Question Type (2023)

Abstract: This study addresses the automatic generation of distractors for English fill-in-the-blank exercises in the entrance examinations for Japanese universities. While previous studies applied the same method to all questions, actual entrance examinations have multiple question types that reflect the purpose of the questions. Therefore, we define three types of questions (grammar, function word, and context) and propose a method to generate distractors according to the characteristics of each question type. Experimental results on 500 actual questions show the effectiveness of the proposed method for both automatic and manual evaluation.

## introduction fill-in-the-blank questions, also known as cloze tests @xcite , are one way to assess learners' english proficiency and are widely used in examinations such as toeic 1 and in school education. as shown in figure 1 , the question format generally consists of a four-choice option with one correct answer and three distractors. these require substantial costs because they are manually created by question writers with extensive language teaching experience. this study automatically generates distractors to reduce workload. most of the previous studies on the automatic generation of cloze tests @xcite @xcite @xcite have generated words that are semantically similar to the correct words as distractors. other methods have been proposed, such as those based on co-occurrence with words in the carrier sentence @xcite , considering the whole context @xcite , and considering the learner's error tendencies @xcite . however, these previous studies apply the same method to all questions, which 1 https://www.ets.org/toeic.html jeff didn't accept the job offer because of the ____ salary. leads to bias in the characteristics of the generated distractors. actual entrance examinations have multiple question types reflecting the purpose of the questions, such as grammatical knowledge and idiomatic expressions. existing methods have difficulty in flexibly changing the characteristics of distractors for each question type. in this study, we first manually classify english fill-in-the-blank questions in the entrance examinations for japanese universities foot_0 by an expert. next, we propose a method for automatic distractor generation according to the characteristics of each question type. experimental results on 500 actual questions show the effectiveness of the proposed method for both automatic and manual evaluation. ## related work previous studies have generated distractors in the following three steps: (1) candidate generation, (2) reranking, and (3) filtering. jiang and lee (2017) utilized cosine similarity with word embeddings @xcite probability with bert @xcite . they also utilize bert for filtering, eliminating candidates with too high and too low probabilities. @xcite proposed candidate generation based on round-trip machine translation. that is, the carrier sentence was first translated into a pivot language and back-translated into english. then, word alignment was used to obtain a candidate for the correct word and its corresponding word. these candidates were reranked using word embeddings and filtered by wordnet @xcite . specifically, synonyms of the correct word in word-net and words with a different part of speech from the correct word were excluded from the candidates. these existing methods have been evaluated in different ways on different datasets, making it difficult to compare their performance. we have comprehensively evaluated them and propose further improvements on top of their combinations. ## definition of question types an experienced english teacher specializing in english education has categorized the question types for english fill-in-the-blank questions. the analysis covers 500 randomly selected questions from the entrance examinations for japanese universities in the five-year period from 2017 to 2021. as shown in question type number of questions grammar 66 (13.2%) function word 195 (39.0%) context 239 (47.8%) ## generating distractors following previous studies @xcite @xcite , we also generate distractors through three steps. for candidate generation and reranking, we selected combinations of the existing methods described in section 2 that maximize performance on the validation dataset foot_1 for each question type. for filtering, we propose methods according to the characteristics of each question type, which are described below. ## experiments we evaluate the method of distractor generation on the 500 questions constructed in section 3. ## conclusion to reduce the cost of creating english fill-inthe-blank questions in entrance examinations for japanese universities, this study addressed automatic distractor generation. first, we identified three question types and constructed a fill-in-theblank corpus annotated by an expert with those question types. next, we proposed methods to generate distractors that take into account the characteristics of each question type, focusing on candidate filtering. experimental results based on automatic and manual evaluations demonstrate the effectiveness of the proposed method. specifically, our method is able to generate candidates that match the gold distractors better than existing methods and has the highest correlation with the examinees' english proficiency as assessed in actual entrance examinations. for future work, we plan to expand the corpus size by estimating question types, to generate distractors by supervised learning.