Using Learning Analytics for Adaptive Exercise Generation (2023)

Abstract: Single Choice exercises constitute a central exercise type for language learning in a learner’s progression from mere implicit exposure through input enhancement to productive language use in open exercises. Distractors that support learning in the individual zone of proximal development should not be derived from static analyses of learner corpora, but rely on dynamic learning analytics based on half-open exercises. We demonstrate how a system’s error diagnosis module can be re-used for automatic and dynamic generation and adaptation of distractors, as well as to inform exercise generation in terms of relevant learning goals and reasonable chunking in Jumbled Sentences exercises.

## introduction supporting language learners to progress in their zone of proximal development requires exercises of different complexities @xcite . while input enhancement for implicit exposure to linguistic constructions can foster receptive skills at the lower end of the complexity range @xcite , open exercises that elicit production of linguistic constructions and the entire sentence context constitute the other extreme @xcite . in order to advance from one to the other, learners need to acquire the constructions relevant for language production in a controlled way. to this purpose, half-open exercises require learners to produce only the target form whereas closed exercise types provide a range of answer alternatives to choose from @xcite . the closedtype single choice (sc) exercises require special attention as they expose learners to incorrect linguistic material in the form of distractor options. while distractors should cover developmental misconceptions in order to be sufficiently challenging and thus relevant to learning, they should not expose learners to any misconceptions they would not have come up with on their own @xcite . given these considerations, it is not surprising that distractor generation is seen as the most challenging aspect of generating sc exercises @xcite . in order to determine pedagogically valid and plausible distractors, human judgement is often deemed best @xcite , yet even manually created distractors do often not meet these requirements @xcite . in order to automate distractor generation and at the same time increase plausibility and validity, data-driven approaches base distractors on common misconceptions of learners @xcite . this in addition allows a more learner-centered adaptation of distractors by dynamically selecting those distractors for each learner from a pool of options that target their individual misconceptions. however, abstracting learner errors into patterns that facilitate generating distractors for arbitrary target answers is not a trivial task. on the other hand, many intelligent language tutoring systems (ilts) incorporate error diagnosis mechanisms. approaches anticipating the most common correct and incorrect learner answers, henceforth referred to as answer hypotheses, and matching them to error diagnoses @xcite , are particularly interesting for distractor generation. an example that successfully pursues this approach constitutes the ilts feedbook @xcite . the process shows strong similarities to distractor generation: the most frequent learner errors constitute the most plausible distractors whereas alternative, correct answers represent unreliable distractors that need to be avoided. systems generating answer hypotheses for error diagnoses therefore inherently have the means to automatically generate distractors. this is especially valuable if sc exercises are used for remedial practice as it opens the possibility to directly associate sc exercises with learner errors and select exercises that best target the learner's misconceptions. the parallels of error analysis based on answer hypotheses and distractor generation are striking, yet these two subfields of natural language processing (nlp) have never been approached in tandem. although previous approaches to exercise generation have used learner errors solely for distractor generation, they can similarly inform chunking of jumbled sentences for word order practice, and determination of required exercise material. grammatical constructions that are not challenging for learners do not need excessive practice. on the other hand, constructions where learners make many errors should be practiced in a variety of exercises focusing on remedying these misconceptions. in order to fill the gap, we show the feasibility of using a system's error diagnosis mechanism for distractor generation, as well as for sentence chunking and learning goal definition, at the example of real learner data collected in the interact4school (i4s) study @xcite . the rest of the paper is structured as follows: section 2 presents related work on distractor generation. after outlining the research questions and the approach to answer them in section 3, section 4 introduces the data on which the approach was piloted. section 5 describes the pilot analyses and presents their results before section 6 concludes with a summary. ## related work distractor generation usually consists of candidate generation and candidate filtering and/or ranking, although they are sometimes executed in a single step. many approaches combine a number of different filtering and re-ranking approaches. for question answering and vocabulary-focused gap exercises, approaches differ in the source from which the pool of distractor candidates is compiled, as well as in the filtering and ranking strategies. the candidates are either extracted from unstructured data such as text corpora @xcite , from structured data such as databases @xcite or word lists @xcite , or else generated based on machine learning @xcite or on transformation rules @xcite . the candidate pool then comprises either a subset @xcite or all entries @xcite of the resource, or transformations thereof @xcite or of the target answer @xcite . filtering and ranking depend on the intended distractor type such as ungrammatical, nonsensical and plausible distractors @xcite , which determines for example the usefulness of grammaticality checks @xcite . for plausible distractors, the desired similarity of the distractors with the target answer constitutes an additional factor. this is on the one hand influenced by the task setup as for example synonyms may be context-inappropriate and therefore useful distractors for contextualized exercises @xcite , yet would constitute unreliable distractors if they can correctly replace the target answer @xcite . in addition, since exercise difficulty increases with distractor plausibility, target similarity can be adjusted according to the learner's proficiency @xcite @xcite . similarity can target the surface form @xcite , linguistic complexity @xcite , phonetics @xcite , morphology @xcite , syntax @xcite , or semantics @xcite and be based on nlp tools including part-of-speech taggers @xcite , latent semantic analysis @xcite and word embedding models @xcite , on external resources such as ontologies @xcite , wordnet @xcite or framenet @xcite , or else on statistical methods including classification @xcite , regression @xcite and deep learning @xcite . if the final candidate selection is not based on the ranking, it may be left to the user @xcite , or done randomly @xcite . while automatic distractor generation has been widely explored for vocabulary exercises, distractors for grammar exercises have received less attention. with closed class grammatical constructions such as prepositions, many of the approaches used for vocabulary distractors are applicable. however, this greatly underrates the importance of linking distractors to the pedagogical learning goal as good distractors characterize the space of options that a learner needs to weigh against each other. since the focus of form-based grammar exercises is not on semantics but on form, they usually rely on ungrammatical distractors @xcite . @xcite illustrate that for closed class target answers, the initial candidate pool consists of all types belonging to the class, whereas for open class target answers, transformations may produce suitable distractors. for the closed class of prepositions, @xcite start with the defined set of prepositions as candidates. for ranking, they consider co-occurrence of the candidates with either the prepositional object or the head, and their frequency as annotated errors or learner-corrected tokens in a learner corpus. suitable for open class types, @xcite use distractor generation rules for a defined set of construct patterns which introduce modifications of the target answer such as morphological or syntactic variants. @xcite present an approach to generate morphological transformations of the target answer as distractor candidates and filter out those whose morphosyntactic pattern can be found in a corpus. for verb exercises, @xcite filter the verbs from the academic word list by transitiveness, tense and person, and rank them according to semantic similarity and distributional data. @xcite apply nlp-as well as rule-based transformations to generate well-and ill-formed variations of the target answer. @xcite found distractor generation based on learner errors to yield the most plausible distractors. while their approach is closest to what we suggest, it relies on a manually annotated corpus. the resulting, statically determined distractors may be sufficiently representative for the learner population that provided the error corpus, yet they are likely to be unsuitable when more widely applied and do not allow to adapt to an individual learner's abilities. we therefore illustrate how automatic annotations obtained from a system's error diagnosis mechanism can effectively be used to generate and dynamically select valid and plausible distractors. ## approach we evaluated a dataset of learner answers to formbased grammar exercises with the aim of answering the following research questions: rq.1 can the creation of learning goals, distractors and js chunks be automated through learning analytics? rq.2 does human perception of relevant misconceptions align with relevant misconceptions derived from learning analytics? rq.3 do errors made in half-open exercises constitute plausible distractors of closed exercises? in order to answer rq.1, in the following we indicate which steps of the evaluations could not be based on automated processing of the data but instead required manual labour. in addition, we determined the ability of the system's error diagnosis module to identify relevant errors automatically. this on the one hand outlines the status quo of possible automization and on the other hand indicates future directions for extending the module in order to support the envisioned learning analytics based adaptivity. in order to answer rq.2, we first identified the most frequent errors made in half-open exercises. to this end, we determined misconceptions of interest by freely annotating the entire dataset once without any reference set of potential labels. of the thus compiled labels, those specific to questions in the simple past were included in the final label set. in order to develop an annotated learner corpus from the learner answers, we relied on two sources: (a) automatic annotations provided by the system's error diagnosis module, and (b) manual annotations. the automatic annotations provide the single most relevant error for each learner answer. they were refined into more fine-grained labels if simple string matching was sufficient and mapped to the label set. we used these annotations whenever available (@xmath0, 778) and manually annotated the remaining learner answers (n @xmath1, 058, n @xmath2, 576) if the system could not diagnose the nature of the error. five annotators with backgrounds in computational linguistics annotated the learner answers independently with an unconstrained number of labels. inter-annotator agreement (iaa) for the multi-label annotations of all annotators was calculated as krippendorff's alpha at α = .2075. for the evaluations, the union set of manual and automatic annotations was used in order to not miss any potential errors. although this might introduce some noise, it serves the purpose of identifying distractor candidates best. in a second step, we contrasted the learner errors against misconceptions judged relevant by human exercise creators. to this purpose, we analyzed the available exercises, distractors and js chunks of with respect to the errors for which they provide opportunities. we annotated the exercises with the same labels used for learner error annotations. a label was assigned if it is in principle possible to make the associated error in the exercise. errors made in half-open exercises can only inform distractor generation if learners tend to choose the associated distractors in sc exercises. similarly, separating constituents into individual chunks only supports learning if learners fail to put these chunks into the correct order in js exercises. in order to answer rq.3, we therefore analyzed whether the identified most frequent errors were also made in sc and js exercises if the exercises provided opportunities to make them. ## data the evaluations are based on data obtained in the i4s project. the study collected data from 7th grade learners of english as a second language in german secondary schools who worked with the feedbook over the course of a school year. the ilts offers practice exercises in a task based setting with intelligent feedback provided to the learners as they work on the exercises. the subset of the data used for the pilot evaluations consists of the exercises on questions in the simple past. the resulting dataset is based on 132 exercise items of the four exercise types illustrated in fig -ures 8-11 of appendix a: 27 jumbled sentences (js) whose chunks learners have to put into the correct order; 27 sc items for which learners need to select the correct option from the dropdown; 58 fill-in-the-blanks (fib) items with input fields into which learners must write the target form; and 20 short answer (sa) items which require learners to write a sentence in response to a prompt. 10 of the fib items present all correct forms to insert into the blanks as bags of words in the exercise instructions instead of giving lemmas in parentheses behind the blanks. as this renders them more similar to sc exercises, we treat them as such. fib and sa exercises constitute half-open exercise types while sc and js exercises are closed types. a total of 4,836 incorrect learner answers to an actionable element of the exercises was collected from 199 learners who submitted at least 1 of the exercises. an actionable element is defined as the blank of a fib or sc exercise, a chunk of a js exercise, or an answer to a sa exercise. all submissions were considered so that there may be multiple answers per learner and actionable element if a learner re-submitted a revised answer. while the focus of the analyses is on distractor generation, we also evaluated the feasibility of using the system's error diagnosis module to determine relevant learning goals and generate chunks of js exercises. ## conclusion we outlined a data-driven approach to determine relevant learning goals, distractors and sentence chunking for the generation of form-based gram-mar exercises. addressing our first research question, we demonstrated the feasibility of using a system's error diagnosis mechanism to automatically annotate learner errors made in half-open exercises in order to dynamically adapt distractors to a learner's misconceptions. although not all of the most frequent errors are automatically annotated in the piloted system, it is possible to extend the error diagnosis module to generate all relevant answer hypotheses. distractor generation and error diagnosis can work hand in hand to this end. we also highlight the relevance of human involvement in the selection of pedagogically valid misconceptions. pre-filtering of distractor templates should be manual and pedagogically motivated, while ranking of the candidates is best informed by learning analytics. the presented evaluations of most frequent learner errors based on the entire learner corpus serve as exemplary application to an adaptation module, and at the same time may be used as initial settings while the system still lacks learner records for individually adapted exercise configurations. with respect to the second research question, we found that while there is substantial overlap between human intuition and learning analytics based exercise generation, they also differ in the focus they put on different misconceptions. since this focus is inconsistent in human output depending on the specific task at hand, human exercise creators might benefit from explicitly specifying the learning goal in a first step. our evaluations suggest that highest pedagogic validity of exercises can be achieved by relying on human effort to define learning goals, and on learning analytics based, automatic processing for exercise generation. the third research question cannot be answered conclusively since the exercises do not cover all potential misconceptions for all exercise types. where no learner data from half-open exercises is available, no conclusions can be drawn about the pedagogical validity of learner errors as distractors. this constitutes a limitation of the presented evaluations. future work will therefore need to determine whether the errors that learners make in half-open exercises are also good distractors for sc exercises or whether learners instantly perceive them as incorrect when contrasted against the correct option. it is also yet unclear to what extent the most frequent misconceptions differ between and within learners over extended periods of time.