Cloze Quality Estimation for Language Assessment (2023)

Abstract: Cloze tests play an essential role in language assessment and help language learners improve their skills. In this paper, we propose a novel task called Cloze Quality Estimation (CQE) — a zero-shot task of evaluating whether a cloze test is of sufficient “high-quality” for language assessment based on two important factors: reliability and validity. We have taken the first step by creating a new dataset named CELA for the CQE task, which includes English cloze tests and corresponding evaluations about their quality annotated by native English speakers, which includes 2,597 and 1,730 instances in aspects of reliability and validity, respectively. We have tested baseline evaluation methods on the dataset, showing that our method could contribute to the CQE task, but the task is still challenging.

## introduction a cloze test @xcite is an efficient and comprehensive tool in language assessment, and thus it is widely used in language proficiency tests (passage and questions in the significantly high price of creating cloze tests by experts has prompted automatic cloze generation methods @xcite @xcite . however, automatically generated cloze tests do not always contribute well to language assessment and suffer from low quality. in language assessment, a reliable and valid cloze test shows a high ability to measure examinees' language level @xcite . cloze question creation has two steps, word deletion and distractor generation, with the latter greatly affecting the ability to measure language level. as indicated by @xcite , some cloze tests, particularly automatically generated cloze tests, are created coarsely and cause two fatal issues in language assessment: (1) these tests do not guarantee that the answer is not ambiguous, which means there is a risk that multiple options fit almost equally well into the blank; (2) the test creation process considers less about which aspect of the language phenomenon is measured in the test; hence, such tests cannot measure the examinee's language level. these two issues make the cloze test unsuitable for measuring examinees' language level. the first issue makes the test unreliable (e.g., question 4 in in this paper, we tackle the issues of evaluating the appropriateness of cloze tests for language assessment focusing on distractors. by following the test design principle @xcite , we define a zero-shot task to evaluate cloze tests for language assessment considering two aspects: reliability and validity, which is called cloze quality estimation passage: a policeman was walking along the street. in the doorway of a shop, a man was standing in the __1__ light, with an unlighted cigar in his mouth. the policeman slowed down and then walked up to the man. "i'm just waiting for a friend here," the man said "it's an appointment __2__ twenty years ago." the man struck a match and __3__ his cigar. the light __4__ a pale face with a little white scar near his right eye. (cqe). in cqe, each question in a cloze test is asked to be estimated, whether it is reliable and valid. cqe provides a cloze as input (passage and questions in the main contributions of this work are summarized as follows: ## related work language educators are capable of creating cloze tests rationally; they select words to be blanked and design distractors by their experience in language education to improve reliability and validity. cloth @xcite , scde @xcite , and cepoc @xcite are collections of human-created cloze tests, which are highly evaluated by experts in terms of measuring the english ability of examinees. however, designing cloze tests by experts is costly and difficult to generalize. automatic cloze generation methods could decrease the cost of creating cloze tests. to avoid generating useless tests in language assessment, these methods focus on distractor generation, which affects the quality of tests significantly. previous works have conducted trials designing good rules or machine learning models. @xcite described a method of generating distractors for assessing an english as second language (esl) learner's ability to distinguish semantic nuances between vocabulary words. they could generate valid questions, but these questions are domain-limited, which is not easy to generalize to cloze tests for human language assessment. the children's book test (cbt) @xcite deletes named entities, common nouns, verbs, and prepositions and then designs distractors having the same part of speech (pos) as the deleted words to measure abilities in reading comprehension and vocabulary. because of the naive distractor creation, the cbt method has the risk of generating unreliable questions. @xcite ; @xcite al. (2010); correia et al. (2012); hill and simha (2016); jiang et al. (2020) explore cloze test generation with various features, such as n-gram frequency and pos tag, and attempt to select deleted words and create distractors by using discriminative models including conditional random fields and support vector machine. advanced distractor generation methods @xcite that employ large pre-trained language models (lms) can provide more valid distractors. these lms produce better text representation that captures rich semantic information, and better text representation allows generation methods to produce more plausible distractors, which could measure language abilities better. however, designing good rules or models to improve the quality of cloze tests is not that easy. furthermore, these works claimed they could generate better distractors, but it is difficult to perform comparisons to previous work. all the works performed human evaluations using their own metrics. crowdsourcing has been used to evaluate the quality of cloze tests and explore factors that affect quality @xcite ; workers were required to fill appropriate words in an open clozestyle sentence (a sentence with a blank, but without providing options). answers from workers were used to calculate cloze easiness @xcite to indicate whether the sentence is usable for the testing an examinee's vocabulary. the association of language testers in europe provides a manual for developing a language test (alte, 2011), which specifies that the statistics of a test's results reflect its reliability and validity. it asks various examinees to answer a test and analyzes the statistics of a question such as the accuracy and the answer distribution. however, these evaluation methods require human resources or experts, which is timeconsuming and sometimes difficult to obtain. ## cloze quality estimation motivated by related work, we propose the task of evaluating the quality of the cloze test. we introduce the definition of cqe task and cela, our new dataset designed specifically for the cqe task. figure 1 shows the flow of creating cela. we collected expert-designed and automatically generated cloze tests and asked native english speakers to annotate whether these tests are reliable and valid. ## option-aware cqe method we propose two methods to tackle the cqe task, which analyze all options of cloze questions as baseline methods for the cqe task. ## experiment we conduct experiments to determine whether option-aware cqe methods ( §4) can be a good baseline to estimate the quality of cloze tests, 3 https://github.com/stanfordnlp/stanza by comparing them with option-agnostic baseline methods ( §5.2). ## conclusion and future work we proposed a novel task for evaluating cloze questions for human language assessment (cqe), which involved two important factors that affect the quality of cloze questions, reliability and validity, and also provided the cela dataset. in addition, we explored automated cqe methods that can estimate the quality of cloze tests, by designing optionaware methods. our experimental results on the cela dataset showed that imbalanced data bring challenges. in future work, we would like to investigate more factors that affect the quality of cloze questions and expand the cela dataset. for example, given the context the sun also ____. and two different sets of options [raises, rises, lifts, elevates] as well as [raises, lives, runs, lefts], although these two sets of options both measure reading comprehension ability, answering the question with the former set of options is more difficult than the latter and requires a higher language level. improving the performance of dnn models and optimizing implementation of breakgrammar(•) and breaksemantics(•) can improve the evaluation performance. for example, designing more detailed filters in the grammatical error corrector to filter out potential semantic errors or using finedesigned data to train different language models for grammatical and semantic error detection separately may boost the performance of dnn models. we hope that the task and our resource will encourage further exploration from both computational linguistics and language education.