Learning from Partially Annotated Data: Example-aware Creation of Gap-filling Exercises for Language Learning (2023)

Abstract: Since performing exercises (including, e.g.,practice tests) forms a crucial component oflearning, and creating such exercises requiresnon-trivial effort from the teacher. There is agreat value in automatic exercise generationin digital tools in education. In this paper, weparticularly focus on automatic creation of gap-filling exercises for language learning, specifi-cally grammar exercises. Since providing anyannotation in this domain requires human ex-pert effort, we aim to avoid it entirely and ex-plore the task of converting existing texts intonew gap-filling exercises, purely based on anexample exercise, without explicit instructionor detailed annotation of the intended gram-mar topics. We contribute (i) a novel neuralnetwork architecture specifically designed foraforementioned gap-filling exercise generationtask, and (ii) a real-world benchmark datasetfor French grammar. We show that our modelfor this French grammar gap-filling exercisegeneration outperforms a competitive baselineclassifier by 8% in F1 percentage points, achiev-ing an average F1 score of 82%. Our model im-plementation and the dataset are made publiclyavailable to foster future research, thus offeringa standardized evaluation and baseline solutionof the proposed partially annotated data predic-tion task in grammar exercise creation.

## introduction while digital education tools have been increasingly developed and deployed for over a decade, the e-learning sector has definitely boomed in the wake of covid-19, even leading to a new digital education action plan from the european commission. 2 as one application in e-learning, we particularly focus on language education, and specifically on the automatic generation of gap-filling grammar exercises. this type of exercises has been shown to be very effective in language learning, with a noticeable effect of such practice tests on students progress and is generally considered as a global measure of language proficiency @xcite . furthermore, automatic generation of exercises has been shown produce relatively high quality exercises, for example, for multiple choice questions @xcite , demonstrating the potential effectiveness of reducing human effort and offering cost-effective solutions towards personalized exercise generation. in terms of technology, recent developments in natural language processing, e.g., bert @xcite , gpt-3 @xcite , instructgpt @xcite , open up new opportunities for further upscaling and improving automatic generation of such tests/exercises. in this paper we specifically propose to generate grammar exercises from existing texts, by inducing well-chosen gaps in a given input sentence, following a set of given example exercise sentences. further, we aim to create models that can be trained on the exercises themselves, without further annotations. the latter implies that we want to forgo a fully supervised learning setting, because such models would require each gap in the available exercises to be manually annotated with additional metadata, such as the particular exercise type, e.g., for gap-filling exercises, a suitable category such as a verb tense. thus, we focus on converting given input texts into gap-filling exercises, by mimicking the implicit rules underlying a given example exercise, rather than by following explicit instructions such as a prescribed exercise type. application scenario: consider a language teacher, who just introduced a particular grammatical topic (e.g., a new verb tense), and needs the students to practice. the grammar topic of interest may need to be practiced in combination with particular other topics (e.g., related tenses already stud-ied by the students). given that gap-filling questions can be completed online and automatically assessed @xcite , the teacher creates a new gap-filling exercise, covering these combined grammar topics. the goal of our model is then to support the automatic creation of new exercises, based on that example exercise, by transforming other texts provided by the teacher into additional gap-filling exercises that target the same linguistic topics to be practiced, without explicit instructions by the teacher of which topics the model should include. this would allow the teacher to rapidly create new training material for the students, potentially more diverse, for example, in terms of topics of the texts, their temporal relevance, or the inherent linguistic difficulty. learning from partially annotated data: the scenario outlined above represents a learning task in between one-shot learning (i.e., learning from one example @xcite and full supervision (i.e., based on the full annotation of all examples). on the one hand, the one-shot setting considers the example exercise as a single training instance defining the nature of the prediction task by the way it was constructed by the teacher (in this case, the included grammar topics). on the other hand, the fully supervised setting would require at least explicit knowledge of all exercise instructions (i.e., gap types per exercise). although we assume the availability of an entire corpus of such exercises, on overlapping grammar topics, we will not rely on explicit annotation of the nature of the gaps (i.e., gap type that defines the type/scope of the grammar exercise, or even just identifying the word category). thus, we do want to learn from partially annotated examples, where the annotation is limited to just the indication of the gap and the text span that constitutes the expected answer. this basically amounts to the type of information that would be available in a one-/few-shot setting, but we aim to leverage the complete corpus to train our models. note that, while creating exercises, teachers are aware of the envisioned exercise type and the gap types, and such exercise type would also be communicated (e.g., as a free-text instruction) to students. still, to keep our experiments and the gained insights transparent, we left out any exercise level instructions for our experiments. link with related research: in broad terms, the proposed work fits within the area of automatic question generation (aqg) for the educational domain. in the field of education, creating questions manually is an arduous task that demands considerable time, training, experience, and resources from educators @xcite . as a solution to this challenge, researchers have turned towards agq approaches to automatically generate homework, test, and exam exercises from readily available plain text that requires little to no human calibration. in particular, educational aqg systems have been developed for generating factoid questions covering several subjects such as history @xcite , general sciences @xcite @xcite , health and biomedical sciences @xcite , etc., as well as for language learning such as vocabulary or grammar exercises @xcite @xcite . there has been some more generic recent work, however, on finding distractors for multiple choice questions across subjects and languages @xcite . it is line with recent work on training deep neural networks for general-purpose question generation @xcite , based on large training sets. there is a clear preference for two question types that allow for automated assessment, i.e., multiple-choice questions (e.g., in @xcite @xcite ) or gap-filling questions (as in @xcite @xcite ). our work is focused on gap-filling questions, which typically require test-takers to fill in blank spaces in a text with missing word(s) omitted by test developers. the missing words can either be chosen from a set of possible answers (i.e., closed cloze questions), or generated from scratch using hints provided in the text (i.e., open cloze questions). to generate such questions, various strategies were employed, such as deleting every nth word from a text @xcite , or rationally deleting words according to specific purpose, e.g., usage of prepositions @xcite , verbs @xcite etc. previous studies have relied on selecting informative sentences @xcite from existing corpora, such as textbooks @xcite , wordnet @xcite , and example 1 example 2 1 will you work a lot? 3 i hope my favorite team won't lose any more games. 3 j'espère que mon équipe favorite ne perdra plus aucun match. a l'âge de 27 ans, le californien david blancarte a eu un grave accident de scooter. quand il s'est réveillé à l'hôpital, il ne sentait plus ses jambes. on lui a expliqué qu'il ne pourrait plus marcher. c' était une vraie catastrophe pour lui! pendant une longue période de revalidation, il a appris à se déplacer en chaise. roulante. ... at the age of 27, californian david blancarte had a serious scooter accident. when he woke up in the hospital, he no longer felt his legs. it was explained to him that he couldn't walk anymore. it was a real disaster for him! during a long period of rehabilitation, he learned . to move around in a wheelchair. more recently, @xcite , have developed sequence labeling model to automate the process of generating gap-filling exercises. another very relevant work by @xcite devised a method to adapt an elec-tra @xcite ) model for the purpose of generating open cloze grammar exercises in english. their approach involved classifying each individual token as either a gap or non-gap. however, there exist several notable distinctions between their approach and our own. firstly, unlike their method that solely focused on individual tokens, we make gap decisions based on spans. this distinction is essential as our gaps can encompass multiple words, allowing for more comprehensive and contextually accurate grammar exercises. secondly, our objective and experimental setup differ significantly. our ultimate goal is to generate multiple versions of the same text, with each version targeting a distinct grammar aspect (e.g., future tense, prepositions of time or combinations of different types). in contrast, their approach consistently produces exercises of the same type for a given input text (i.e., similar to our baseline model), lacking the versatility and adaptability our model offers. we observed a tendency in generation of gapfilling questions aiming at well-defined tasks. to the best of our knowledge, none of the prior works have proposed strategies to capture common underlying structures in terms of task definition, while training on a heterogeneous set of real-world examples (e.g., covering various grammatical topics). ## example-aware span detection model this section describes our baseline model and proposed example-aware gap detection model. figure 2 provides a schematic overview. we first detail the part indicated as baseline model, inside the smaller dashed box, followed by the part that encodes the exemplar, which leads to the full model. ## empirical validation on real-world data in this section, we first introduce the dataset that we will publicly release. then, we explain how we train our models and use them for inference. finally, we describe the strategies we adopted to evaluate the effectiveness of our models. ## experimental results in this section, we provide evidence of the effectiveness of our proposed model by reporting and discussing the experimental results. ## conclusion in this paper, we introduced a new task within the general challenge of training models to automatically create new exercises for use in education, based on existing exercises and without requiring additional manual annotations. in particular, we introduced a dataset and associated prediction task, focusing on detecting gaps within a given input text, without knowledge of the exact exercise type, by only relying on an example exercise. we proposed an example-aware neural network model designed for this task, and compared it with a baseline model that does not take into account any example of the desired exercise type. we found that our example-aware model outperforms the baseline model not only in predicting gaps, but also in disentangling gap types despite not being explicitly trained on that task. our realworld gf2 dataset of french gap-filling exercises will be publicly released together with the code to reproduce the presented empirical results. the presented work fits with our pursuit towards supporting personalized learning experiences by either suggesting existing or generating new exercises that are tailored to students' needs. teachers could also benefit from an increased efficiency in creating new exercises. for example, they could make many and diverse drill and practice exercises on chunks of text based on existing standard exercise types without having to provide extra metadata information such as instructions. we hope our benchmark dataset and task will spark new research in the cl and educational nlp community.