Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests (2024)

Abstract: The cloze training objective of Masked Language Models makes them a natural choice for generating plausible distractors for human cloze questions. However, distractors must also be both distinct and incorrect, neither of which is directly addressed by existing neural methods. Evaluation of recent models has also relied largely on automated metrics, which cannot demonstrate the reliability or validity of human comprehension tests. In this work, we first formulate the pedagogically motivated objectives of plausibility, incorrectness, and distinctiveness in terms of conditional distributions from language models. Second, we present an unsupervised, interpretable method that uses these objectives to jointly optimize sets of distractors. Third, we test the reliability and validity of the resulting cloze tests compared to other methods with human participants. We find our method has stronger correlation with teacher-created comprehension tests than the state-of-the-art neural method and is more internally consistent. Our implementation is freely available and can quickly create a multiple choice cloze test from any given passage.

## introduction the cloze procedure, first introduced by @xcite , is a widely used method for creating reading comprehension tests inspired by the gestalt principle of "closure." though many variations have been introduced and studied, the core concept is to mask words in prose and task the subject with providing the missing words. the fraction of words guessed correctly is used as a measure of comprehension of the document. a commonly used variant uses a multiple-choice response for each masked word, with the correct answer hidden among several "distractors." this simplifies scoring by removing ambiguity surrounding synonyms or misspellings. choosing words to be distractors for each blank, however, requires either large pilots with free-text responses or experts in the domain of the text being tested. good distractors must be incorrect, meaning they don't make sense in the blank, but still plausible enough to require comprehension of the passage to rule them out, requiring careful thought. manual creation of distractors could also be subject to bias, especially if the test creator is testing two versions of a text and has an interest in one being seen as more comprehensible. many have thus sought to automate distractor generation, using word co-occurence, lexical databases, and embeddings. more recently, pretrained masked language models have been used, both with their original training objective and finetuned on reference distractors. the training objective of mlms, which is essentially the cloze task, makes them natural choice for producing plausible words to act as distractors. however, pedagogical literature suggests that distractors should also be both incorrect and distinct @xcite @xcite . to be incorrect, they must not make sense in the blank given the entire passage, which is not necessarily satisfied by choosing words that are slightly different from the answer. to be distinct, distractors should not overlap with each semantically, which requires optimization of sets of distractors, rather than ranking and choosing the top k. prior work largely assesses the quality of distractors using information retrieval metrics vs. reference distractors, and qualitative human judgments of each distractor. there are several issues with this approach. first, these metrics do not capture interactions between distractors, such as semantic overlap (which hurts distinctiveness). second, there are only a few reference distractors for each blank, while many other possible good distractors (perhaps even better) exist. third, these metrics rely on reference distractors for blank positions that were chosen by teachers and do not characterize how the methods would perform on arbitrary blanks, which would be necessary for creating new tests. to address these issues, we first define the pedagogically motivated objectives of plausibility, incorrectness and distinctiveness in terms of embeddings and conditional probabilities from by mlms. we then optimize for these objectives using simulating annealing. this makes our method both unsupervised (reference distractors are not required) and interpretable (the balance of the three objectives for a chosen set of distractors is known). we call the resulting method ncloze, for "neural cloze." to address shortcomings with assessment, we measure how well automatically generated cloze tests from various methods actually perform their intended function, by giving tests to human readers and measuring their validity and reliability as psychometric instruments. as a reference instrument for measuring validity, we use a set of 18 teachercreated middle-and high-school-level reading comprehension passages from the cloth dataset @xcite . we test two recently reported neural distractor methods @xcite , both with mechanical deletion, and ncloze with two deletion strategies. the best-performing version of ncloze improves correlation to the reference instrument by 17% and internal consistency by 18% vs. the next best existing method. to test domain applicability, we perform another experiment using the newest vital sign (nvs) @xcite as a reference instrument for measuring health literacy. we find that ncloze tests on health-related text passages strongly correlate with performance nvs. our contributions are: (1) we define pedagogically motivated objectives, based on conditional distributions from masked language models, for ranking sets of distractors. (2) we present ncloze, an unsupervised, interpretable method for generating cloze tests based on these objectives. (3) we experimentally demonstrate the validity and reliability of ncloze tests, and neurally generated cloze tests generally, with human participants. (4) we provide open-source implementations of both our new method and the previously closed-source sota method. ## background since the cloze procedure was first introduced, many variations have been proposed @xcite . two widely used variations are (1) "rational deletion" (as opposed to "mechanical deletion"), in which the words to blank are chosen based on their importance to the passage, and (2) multiple-choice (rather than free-text) responses, for which "distractors," or incorrect answer choices, must be chosen @xcite . there has been interest in automating both of these tasks, with methods evolving along with natural language processing techniques. early methods to choose distractors @xcite , and some more recent @xcite , relied on wordnet @xcite to find words having the same part of speech as the answer and semantic similarity. for the goal of testing language proficiency, morphological modifications and orthographic similarity have been used @xcite . once word embeddings were introduced, e.g. word2vec @xcite and glove @xcite , many distractor generation methods incorporated them @xcite @xcite @xcite . frequency of n-grams has also been used @xcite . transformer-based pretrained language models, such as bert (bidirectional encoder representations from transformers) @xcite and gpt @xcite , trained to fill blanks or predict the next word, offered new opportunities for creating cloze tests. one use of such language models in this domain has been to guide rational deletion (i.e., choosing which words to blank). @xcite use language models to estimate conditional probabilities for differing contexts, hypothesizing that good blanks will have options that are likely in a local context but not a broader context. however, since they use gpt-2 @xcite , only previous, and not subsequent, context can be included. they also find that inclusion of further domain-specific context (namely, the wikipedia entry for "psychology") is required; our method needs only a provided passage and a pretrained language model. @xcite use language model distributions to ensure blanked words have unique answers. yang et al. (2021) perform additional supervised training with an mlm to identify the optimal word to blank in a passage, based on how important the word is to the overall passage. all three of these rational deletion methods assume free-text response and thus do not address distractor generation. another application of language models to cloze tests is scoring of potential distractors. @xcite , for example, use bert to rerank pools of words selected by similarity to the answer via non-contextual embeddings. @xcite use conditional probabilities from a pretrained mlm as features for supervised classification of whether words could be distractors. similarly, andersson and picazo-sanchez (2023) use bert with a restricted softmax to assess distractors, but use morphological perturbations specific to japanese generate them. @xcite use a transformerbased grammatical error correction model to predict whether a given potential distractor would require grammatical knowledge or reading comprehension too rule out. finally, neural language models have been used to create sets of distractors for given blanks, which is the main focus of our work. @xcite use transformer-based round-trip neural machine translation to generate distractors appropriate for specific language pairs for second-language learners. chiang et al. (2022) fine-tune encoder-only mlms to gap-fill with likely distractors rather than likely words, training on the cloth set. the top k can be chosen as a set, but they are not optimized jointly, which is important for nonredundant distractors and thus effective multiple choice questions. similarly, @xcite fine-tune bert to create distractors for agglutinative languages. @xcite choose distractor sets jointly by training encoder-decoder language models to produce sets of distractors. however, they find their model often produces duplicate distractors, making it impractical despite its state-of-the-art performance via automatic metrics. this underscores the need for both more advanced methods and better evaluation tools. ## methods we will first describe the process of generating a set of distractors for a single blank, which can be generalized to any number of blanks in the passage. given a passage of text @xmath0, ..., x l } of length l and the index b of the word to blank, x b , our task is to generate a set of distractors d of size n. our method has two stages: (1) select a pool of words w that are highly ranked for both plausibility and incorrectness, based the probabilities of words appearing in the blank, then (2) choose an optimal set of distractors d from w to balance the latter objectives with distinctiveness, based on the similarity of potential distractor sets. the following sections will define the three objectives, discuss the optimization of d from w , and provide a rational deletion strategy based on the objectives to be optimized. ## experiment 1 recent work has framed distractor generation as an information retrieval problem @xcite @xcite . under this paradigm, models are trained in a supervised manner and information retrieval metrics are used to evaluate them. however, ranking reference distractors highly does not necessarily translate into creating effective sets of distractors, because (1) the reference distractors may be mixed with ineffective distractors near the top of the list, (2) sets chosen from the top k may contain overlapping distractors, and (3) arbitrary blanks in unseen text may have different properties than those of the reference set. further, as our method is unsupervised and optimizes sets of distractors, there is no clear way to evaluate it using these types of metrics. we thus instead perform human experiments to characterize validity and reliability, which are standard measurements used to evaluate psychometric instruments. ## experiment 2 as a domain experiment, we use a model trained to identify medically related text @xcite to choose 5 biomedical passages and generate corresponding ncloze distractors for each one for a total of 5 cloth/ncloze pairs, with an average length of 299 words per passage. we measure correlation of each with newest vital sign (nvs) @xcite , a commonly used health literacy test that provides a nutrition label and ask 6 questions that require reading, understanding, and reasoning about the label. ## discussion and future work the experiments we have run show initial evidence of validity and reliability of our method and support the idea of using pedagogical objectives. the large discrepancies between information retrieval metrics (mrr and f1 scores of our distractors from hyperparameter tuning) and testing instrument metrics (validity and reliability) show that there are many other possible distractors, and that some can in fact be better than reference distractors. however, they also show there is much work to be done in this space, despite intense interest and progress. interestingly, rational deletion did not help our method, in fact placing it last in terms of validity, despite having the highest reliability. this is surprising considering the rational deletion method was chosen to optimize the same objectives as the distractor generation method. one possibility is that this leads to too many similar words being blanked, essentially testing similar concepts over and over. however, further exploration is warranted. another line of investigation is constructing multi-token distractors using a beam search. this could allow common phrases or out-of-vocabulary terms, which may be especially useful in technical domains with jargon. ## conclusion we have presented a distractor generation method based on pedagogical objectives that exhibits higher correlation with teacher-created distractors compared to the state-of-the-art neural method, and is more internally consistent. in addition to combining existing ideas in the field with new ideas and methods, we demonstrated the effectiveness of neurally generated distractors experimentally with human readers, which represents a large advance for the field that we hope will inspire further work. we formulate the task of distractor generation to align with the mlm pretraining objective, i.e., estimating conditional probabilities over a vocabulary given surrounding contexts. consequently, our method requires no further training or layers, using only fast, gradient-free methods to optimize further desiderata for sets of distractors, namely plausibility, incorrectness and distinctiveness. the use of pretrained models directly also allows models to be easily swapped into our system for domainspecific applications. we implement our method in an open-source tool that allows researchers to easily generate distractors from arbitrary sources of text. despite remaining unknowns, it is our hope that the work presented here is a step toward a reliable, automated method for creating reading comprehension tests for a wide variety of domains and applications. our implementation is available at https://github.com/ondovb/ncloze .