Automated Generation of Multiple-Choice Cloze Questions for Assessing E nglish Vocabulary Using GPT -turbo 3.5 (2023)

Abstract: A common way of assessing language learners’ mastery of vocabulary is via multiple-choice cloze (i.e., fill-in-the-blank) questions. But the creation of test items can be laborious for individual teachers or in large-scale language programs. In this paper, we evaluate a new method for automatically generating these types of questions using large language models (LLM). The VocaTT (vocabulary teaching and training) engine is written in Python and comprises three basic steps: pre-processing target word lists, generating sentences and candidate word options using GPT, and finally selecting suitable word options. To test the efficiency of this system, 60 questions were generated targeting academic words. The generated items were reviewed by expert reviewers who judged the well-formedness of the sentences and word options, adding comments to items judged not well-formed. Results showed a 75% rate of well-formedness for sentences and 66.85% rate for suitable word options. This is a marked improvement over the generator used earlier in our research which did not take advantage of GPT’s capabilities. Post-hoc qualitative analysis reveals several points for improvement in future work including cross-referencing part-of-speech tagging, better sentence validation, and improving GPT prompts.

## introduction vocabulary acquisition lies at the core of foreign language education, forming an essential pillar in most holistic curricula @xcite . to measure learners' vocabulary knowledge, a common testing approach is to use multiple-choice cloze questions (hereafter, mcc; @xcite : learners see a stem sentence with a blank followed by several one-word options (one correct answer as the "key" plus several distractors that are incorrect answers) and must choose the option which best fills the blank. the following is an example: this is a fairly simple process with __ steps. the quality of stems and distractors in mcc questions is crucial. according to previous studies, context clarity and relevance to keys are paramount in stems, meaning that a stem should be free from syntactic errors, with appropriate length to provide context for the key, and should show a good use of the key @xcite . meanwhile for distractors, part-of-speech (pos) and semantic considerations apply. specifically, effective distractors should fit syntactically into stems but should be semantically less appropriate than the keys @xcite . traditionally, the generation of such questions has been a manual endeavor, with pedagogical experts or individual educators crafting content tailored for their classes. although there is a clear demand for automated tools to facilitate the generation of numerous vocabulary test items, existing applications, as highlighted by studies like @xcite , @xcite , and @xcite , are either not readily accessible or lack user-friendliness. another aspect often neglected in mcc is ensuring that distractors align with students' genuine learning experiences. students usually engage with vocabulary in structured units or sublists @xcite , which indicates that distractors should be derived only from words they have previously studied. deviating from this can inadvertently provide clues, allowing students to guess based solely on unfamiliarity, thereby diminishing the test's effectiveness. in our previous initiative of a web-based vocabulary training and testing application, or "vocatt" foot_0 , we used the word quiz constructor (wqc, @xcite to automatically generate mcc questions for the general service list and academic word list @xcite . wqc incorporates various features: it tags the pos of input words, crafts a question stem around a chosen keyword from corpus resources, and identifies distractors with matching pos from the input words. it also evaluates the distractors by placing them in the blank and comparing the frequencies in google books of local tri-grams with that when the keyword is filled. if the former is lower that the latter, then the distractor is considered valid. the modified version of wqc's output has been effectively incorporated into english curricula at a japanese university and the in-house application "vocatt". however, it is not without its shortcomings. most notably, evaluations by human experts flagged quality issues with the generated content. among the 1128 question stems and 3384 distractors generated for 1128 questions targeting the academic word list, the percentages of well-formed stems and distractors were only 34.93% and 38.56%, respectively @xcite . 2 it took the reviewers much effort to manually correct the inappropriate stems and distractors before the questions were imported into the application. the advent of advanced natural language processing tools, especially models like gpt, provides new opportunities. these models can potentially enhance the quality of automated mcc question generation through the colossal-scale corpora used as their training data and their deep understanding of complex topics @xcite . this paper delves into our efforts to create a program that automates mcc question generation incorporating an llm. we evaluate its effectiveness through human validation and also provide insights into potential refinements, underpinned by a thorough qualitative analysis of both the generation mechanics and the final output. ## methodology building upon prior work with wqc, the process of automatically generating mcc questions in the program consisted of three distinct phases: preprocessing of input words, stem generation, and distractor selection. a key evolution in the program involves the integration of an llm during both 2 interestingly, this wellformedness rate may suggest the possibility that creating items manually may be more efficient. although not done in the present work, earlier work with wqc showed that the well-formedness rates of automatically generated items were actually comparable to those for manually-produced items @xcite @xcite . stem creation and distractor validation. subsequent sections will delve into the tools employed in this endeavor, followed by a detailed exposition of the program. ## data analysis the research utilized two primary data sources: the output file and the log file, and data analysis was carried out in four steps: step 1. preliminary output check: upon receiving the output file, a preliminary examination was initiated. we verified the presence of all question stems, blanks, distractors, and other essential components in the output, ensuring its integrity before progressing to the next phase. step 2. human evaluation: after the preliminary check, two seasoned reviewers were tasked with an independent evaluation of the questions. the reviewers were all english native speakers with more than 20 years of experience teaching academic english at japanese universities. they had also been involved in a similar project reviewing automatically generated mcc questions on awl words. both reviewers underwent training using an evaluation guide. they were asked to judge whether a stem or distractor was appropriate for assessing the vocabulary knowledge of university students, and if not, provide reasons in comment boxes. the criteria for judging appropriateness are as follows: there should be no grammatical errors. b. an appropriate stem should solicit an accepted use of the key and effectively highlight or emphasize the key. once the reviewers' evaluations were submitted, cohen's d and percent agreement were employed to measure inter-rater reliability. in instances where a discrepancy in evaluations arose, a third expert was consulted to deliver the final judgment, substantiated by relevant comments. step 3. human annotation: subsequent to the human evaluation, items flagged as inappropriate underwent an annotation process by two annotators, who were experienced english teachers with nearnative english proficiency working at a japanese university. drawing from thematic analysis techniques, the two annotators collaboratively classified the inappropriate stems and distractors, and identified categories and subcategories. step 4. qualitative analysis of the log file: with the annotations in place, an exhaustive qualitative examination was set into motion, leveraging the rich information contained in the log file. the primary objective of this step was to pinpoint the root causes of the identified errors. such insights are invaluable for refining and enhancing future iterations of the process. ## results fifteen minutes after its initiation, the program generated the output and log files for analysis. the subsequent sections will detail the results sequentially.