C loz E x: A Task toward Generation of E nglish Cloze Explanation (2023)

Abstract: Providing explanations for cloze questions in language assessment (LA) has been recognized as a valuable approach to enhancing the language proficiency of learners. However, there is a noticeable absence of dedicated tasks and datasets specifically designed for generating language learner explanations. In response to this gap, this paper introduces a novel task ClozEx of generating explanations for cloze questions in LA, with a particular focus on English as a Second Language (ESL) learners. To support this task, we present a meticulously curated dataset comprising cloze questions paired with corresponding explanations. This dataset aims to assess language proficiency and facilitates language learning by offering informative and accurate explanations. To tackle the task, we fine-tuned various baseline models with our training data, including encoder-decoder and decoder-only architectures. We also explored whether large language models (LLMs) are able to generate good explanations without fine-tuning, just using pre-defined prompts. The evaluation results demonstrate that encoder-decoder models have the potential to deliver fluent and valid explanations when trained on our dataset.

## introduction cloze questions @xcite are a fundamental component of language assessment (la). they typically consist of a sentence or a passage with certain words or phrases omitted, and language learners are required to select or fill in the most appropriate word to complete the text. cloze questions in language educational settings are usually used in language educational settings to evaluate language proficiency in terms of various aspects such as grammatical knowledge @xcite and reading comprehension skills (raymond, question: the work was done ____ the rehabilitation institute of chicago under an $8-million grant from the army. (a) at (b) down (c) round (d) of explanation: based on the context of the sentence, option (a) "at" is the appropriate choice for the cloze question. "at" indicates a specific location or arrival at a particular place or position, such as "he is at the store." explanations for cloze questions play a crucial role in language learning, particularly in self-study contexts. when learners encounter challenging cloze questions, having access to clear and concise explanations after answering the question can greatly aid their understanding of the correct answers @xcite . explanations provide learners with insights into the reasoning behind the correct and incorrect choices, helping them identify and rectify their own misconceptions. the provision of high-quality explanations can empower learners, fostering deeper comprehension and longterm knowledge retention. however, despite its usefulness, there has been almost no work on generating high-quality explanations for given cloze questions. one essential reason is that no dataset for such a task is available. because of high costs in terms of time and human effort, employing experts to create such a dataset, to provide abundant data, is difficult, al-though it could guarantee the quality of the dataset. furthermore, even if a dataset could be constructed, it would be challenging to automatically generate human-like cloze explanation. to address these challenges, we propose the task clozex of generating explanations for english cloze questions. intuitively, a good explanation that helps answer a cloze question should be easy to read and provide sufficient background knowledge. therefore, fluency and informativeness should be considered in explanation generation. we also provide a large dataset comprising over 140k expertquality-assured (question, explanation) pairs, an example of which is shown in finally, to investigate the factors that contribute to addressing the clozex task, we train various models as baselines, including encoder-decoder and decoder-only architectures. we also investigated the performance of large language model (llms) in a zero-shot prediction scenario, in which we employed llms to generate explanations for given cloze questions without fine-tuning. the evaluation of baseline models indicated that both encoder-decoder and decoder-only models after fine-tuning are able to produce acceptable explanations. meanwhile, llms are generally good at generating fluent explanations, but in most cases, these explanations do not provide sufficient information for answering questions. only providing llms with questions with naive prompts is challenging to generate high-quality explanations efficiently. the main contributions of this work are summarized as follows: ## related work owing to the efficiency of cloze questions in language assessment, previous research focused on the automatic generation of cloze questions. in contrast to early-period studies that employed naive approaches such as fixed ratio word deletion and random selection of distractor options @xcite , current research places greater emphasis on ensuring the validity of generated questions. for example, @xcite created distractor options based on an english learner writing correction corpus. with this method, words that tend to be misused in a given context would be selected as the distractor options. therefore, questions generated from their methods are expected to be more discriminative in measuring the language proficiency of learners. additionally, previous research @xcite @xcite @xcite investigated what aspects of features affect the validity of cloze questions, such as part of speech (pos), n-gram frequency, and sense of words. they then generated questions based on such features. however, although question generation tasks do primarily aim to generate plausible questions based on given texts, they often do not explicitly address the generation of accompanying explanations. the ability to generate explanations alongside cloze questions is crucial for providing comprehensive support to language learners. by shifting the focus from generating questions to generating explanations, this research introduces a novel task that contributes to the advancement of language learning technologies. @xcite proposed a task of feedback comment generation (fcg) for writing learning with a corresponding dataset. the fcg task automatically generates feedback comments such as a hint or an explanatory note for writing learning for non-native learners of english. however, although this task contributes to grammar learning through writing correction, it has certain limitations in facilitating systematic grammar learning. firstly, the fcg task primarily relies on free english composition, which adopts a bottom-up approach to provide grammatical knowledge. consequently, it inevitably lacks comprehensive coverage of grammar items that learners need to master. in contrast, cloze questions are meticulously designed by experts, adhering to established learning guidelines, thereby ensuring a certain level of coverage of grammar items that learners should be familiar with. secondly, the primary focus of the fcg task lies in explaining the appropriateness of specific words within a sentence, rather than elucidating why certain plausible expressions should be avoided. furthermore, the commentaries in the fcg task stem from free composition, making it challenging to scale the production of high-quality commentaries without significant manual effort. in contrast, the clozex task builds "patterns" for each grammar item, as the cloze questions are constructed using a top-down approach. consequently, it becomes feasible to automatically generate a considerable number of high-quality explanatory (details are explained in section 3.2), as demonstrated by the generation of over 140k such sentences in this study. ## experiment to address the clozex task, we conducted an investigation into baseline models under various scenarios and architectures. to evaluate the performance of these baseline models, we conducted thorough assessments using development and test data from our dataset, encompassing both manual and automatic evaluation metrics. ## conclusion this paper introduced a novel task, clozex, aimed at generating fluent and valid explanations for english cloze questions to support esl learning. we curated a comprehensive dataset comprising more than 140k instances of cloze questions paired with explanations. the dataset is created by a pattern-based method. patterns extracted from expert-designed cloze questions and explanations ensure the quality of generated questions and explanations. expert evaluations confirmed the high quality and suitability of our dataset for the clozex task. to address this task, we fine-tuned various models, including encoder-decoder and decoder-only architectures, to generate explanations for the provided questions. additionally, we investigated the potential of llms to produce explanations in a zero-shot scenario. the experimental results highlighted the capability of the encoder-decoder models to generate high-quality explanations. however, although llms excelled in generating fluent texts, they struggled to produce valid explanations. thus, we analyzed the limitations of llms in generating satisfactory explanations without fine-tuning, shedding light on the challenges they face in this context. additionally, we explored the correlation between manual and automatic evaluation metrics, discovering that automatic metrics exhibited some degree of reliability for the clozex task.