Parametrizable exercise generation from authentic texts: Effectively targeting the language means on the curriculum (2022)

Abstract: We present a parametrizable approach to exercise generation from authentic texts that addresses the need for digital materials designed to practice the language means on the curriculum in a real-life school setting. The tool builds on a language-aware searchengine that helps identify attractive texts rich in the language means to be practiced. Making use of state-of-the-art NLP, the relevant learning targets are identified and transformed intoexercise items embedded in the original context. While the language-aware search engine ensures that these contexts match the learner‘s interests based on the search term used, and the linguistic parametrization of the system then reranks the results to prioritize texts that richly represent the learning targets, for theexercise generation to proceed on this basis, an interactive configuration panel allows users to adjust exercise complexity through a range of parameters specifying both properties of thesource sentences and of the exercises. An evaluation of exercises generated from web documents for a representative sample of language means selected from the English curriculum of 7th grade in German secondary school showed that the ombination of language-aware search and exercise generationsuccessfully facilitates the process of generating exercises from authentic texts that support practice of the pedagogical targets.

## introduction with digital learning contexts becoming increasingly common in foreign language teaching and learning, automatic exercise generation arguably will become a crucial tool for making individualized practice materials available that are adapted to the learner's individual needs and competencies @xcite ). an ideal system for this purpose will generate exercises of parametrizable complexity for a given input text. form-focused exercises lend themselves especially well to automatic generation as their answer space is limited enough to support automatic evaluation @xcite @xcite . approaches in this domain can be subdivided into two categories: systems that generate simple exercise sentences using a rule-based approach, and tools that extract sentences which contain the targeted constructions from existing texts @xcite . working with authentic texts has been argued to have positive effects on learner motivation @xcite , especially if as much context as possible is preserved @xcite . since motivation is highest when the topic and contents of the text is of interest to the learner, allowing them to provide their own texts as input to exercise generation is advantageous @xcite . yet, authentic texts often do not include sufficient examples for the language means to be practiced @xcite . it is therefore important to assist learners in finding suitable documents that are of interest and richly represent the language means on the syllabus that are to be practiced, which has been referred to as input enrichment @xcite . more recently, an important need for automatically generated exercises is arising in the context of adaptive language tutoring systems @xcite . adaptivity comprises elements both at the micro level and at the macro level @xcite . with respect to micro-adaptivity, scaffolding feedback is used to guide the learner towards the correct answer. macro-adaptivity refers to the system capability to provide sequences of exercises at the right level for a given learner. such systems thus need to either manipulate exercise difficulty in real-time @xcite or to maintain large pools of exercises of varying complexity levels @xcite . real-time manipulation is most feasible for aspects of the exercise, such as the number of distractors or hints, but not for linguistic features of the seed sentence or the choice of the target item. this approach to macro-adaptivity in the language learning context has mainly been limited to c-tests @xcite . most educational institutions use some learning management system @xcite . to be able to integrate generated exercises into regular classes, they should be compatible with the lms system used. exercises would thus be most beneficial to instructors when provided as globally usable web components or in a format that complies with standards such as xapi foot_0 and cmi5 foot_1 . in addition, it would be important to provide interfaces that make it possible to edit exercises that were generated to be able to correct or modify them to suit the instructor's needs and preferences. summing up the requirements mentioned above and in the wider literature, an exercise generation tool should provide input enrichment mechanisms for user-selected texts, be parametrizable and editable by the educators themselves, integrate feedback into the exercises, provide the exercises in a portable format, and support use of the exercises within the original context. to our knowledge, no tool has been developed so far which complies with all these features. indeed, no fully automated exercise generation system for grammar exercises we know of even offers some of the features mentioned, such as an integrated input enrichment approach. in this paper, we thus present an exercise generation extension of the language-aware search engine flair foot_2 to address this gap. we start with section 2 introducing the research context on automatic generation of grammar exercises. section 3 describes the implementation of the exercise generation extension of flair and outlines its functionality and use. section 4 evaluates the tool before section 5 summarizes and concludes with an outlook. ## related work our approach integrates automatic exercise generation into an educational document retrieval system. therefore, we will first elaborate on previous work on educational information retrieval systems before discussing existing tools for form-based grammar exercise generation with respect to the outlined criteria we impose on such a system. similar to tutoring systems, educational docu-ment rankings systems often leverage information from a learner model to identify texts that match a learner's individual proficiency level. two examples for such an approach are reap and textfinder @xcite . the learner model is maintained within the system based on the learner's interaction with the tool. while reap can work with texts from anywhere on the web, textfinder operates on its own database of online news articles. a less automated approach relies more on user interaction. examples include the standalone tool read-x @xcite and the web extension lawse @xcite . both tools calculate readability scores for web documents. while lawse merely displays them for the analyzed documents, read-x matches the scores against a readability level which users need to specify, and filters the documents accordingly. for narrowly defined use cases, tools may filter documents without either a learner model or user input. sourcefinder constitutes an example for such a system. it applies a binary filter to its corpus of online journals and identifies texts suitable in academic contexts @xcite . systems with the highest degree of flexibility allow users to filter documents according to contained grammatical constructions. this approach is for example realised in the authoring assistance tool sakumon @xcite and the language-aware search engines flair foot_3 @xcite and kansas foot_4 @xcite . sakumon maintains information on the article's reading level as well as on contained grammatical constructions in its database. flair and kansas, the latter being based on flair and specializing on low literacy in german, analyze web texts on demand. in addition to the filtering functionality, these two systems also allow users to rank all retrieved documents according to the occurrence of linguistic constructions. this kind of re-ranking of search results allows to identify documents containing certain linguistic constructions, such as those targeted by grammar exercises. such a tool therefore lends itself well as basis into which we can integrate an exercise generation component. especially the selection of suitable documents is hardly targeted at all. only sakumon, which is also listed among the document ranking systems, offers full-fledged document filtering. since it was developed as an assistant system, exercise generation has, however, not yet been fully automated. support for preserving the authentic context varies considerably from one system to the other. rule-based systems do not rely on authentic texts at all. examples include the mgbeg exercise generator @xcite and gramex @xcite . among the tools using authentic texts, a couple use only decontextualized, single sentences. this encompasses for example lärka foot_5 @xcite , arikiturri @xcite and fast @xcite . a range of systems integrate the exercises into the base text, yet visual context such as markup elements and images are removed. this is, for instance, the case in the tutor assistant's task generator @xcite , mirto @xcite , the grammar exercise generator (geg) @xcite , the language exercise app (lea) @xcite and collie foot_6 @xcite . visual context is only preserved in those exercise generation tools implemented as web plugins. prominent examples include view foot_7 @xcite and clozefox foot_8 @xcite . most of the exercise generation tools provide the generated exercises within the system and do not offer any export functionalities. noticeable exceptions include killerfiller, arikiturri and the lea. the web plugins view and clozefox are also portable by nature. the degree to which users can influence the exercises to be generated is generally rather low. although in most systems, users can upload their own texts, they often have only rudimentary influence on the properties of the generated exercises. the most highly configurable applications include mirto, sakumon, language muse @xcite and the lea. since sakumon is an assistant system, instructors can and must select the target items and distractors manually from among the tool's suggestions. the lea also allows to specify target constructions, bracket contents and distractors. mirto in addition lets users specify interactive supportive elements such as links to lookup pages. language muse generates a range of different activities for each text from which the user can choose the one which best suits their needs. these can be edited to allow further customization. our approach aims to combine the strengths of these systems into a single application. ## system description we integrate the exercise generation functionality into the language-aware search engine flair. while the exercise generation is fully integrated into this application, we also considered the interface supporting integration of the generated exercises in the lms serving as deployment platforms. ## evaluation the quality of the exercise generation extension depends on its ability to identify documents which contain linguistic constructions that can successfully be transformed into exercise items. we conducted a three-step, pilot evaluation in order to determine the tool's performance in this respect. due to time restrictions, the gold standard annotations and evaluations results were produced only by one of the authors. 4.1 methodology 4.1.1 suitable document selection suitable documents need to contain the targeted grammatical constructions relevant to the selected language means. the tool's performance in identifying them depends mostly on the reliability of the base system. we evaluated all supported language means, i.e., simple present, past tenses, comparatives, conditionals, relative pronouns and passive sentences. for each language means, we determined a binary score of whether the highest-ranked search result for the search term education contained constructions which could be transformed into exercise items. we used an additional flag to indicate whether this was possible with the default settings or only with the help of flair's document ranking. when document ranking was applied, maximum weights were set for all constructions associated with the currently assessed language means. relevant construction identification relevant constructions targeted by the language means can only be used for exercise generation if they are correctly annotated. since the exercise generation extension uses a more fine-grained distinction between linguistic constructions than the base system, the performance of construction identification depends both on the base system's ability to correctly identify rather coarse-grained linguistic constructions and on the exercise generation tool's ability to correctly identify exercise targets from multiple, overlapping constructions. to this purpose, we sampled up to 10 occurrences for each type of exercise target from 100 arbitrarily selected web pages. identical occurrences of target constructions were not considered and only web pages which contained at least one construction were taken into account. we report the precision for the identified constructions since the quality of most of our exercise types depends on the correctness of the used constructions, whereas recall is less important as long as sufficient exercise opportunities are found. ## conclusion we presented a tool for automatic generation of english form-based grammar exercises from authentic web texts. it uses a language-aware search engine to address the challenge of identifying documents rich in the pedagogically targeted language means. while the integration of feedback aims at microadaptivity of the exercises, the tool also supports macro-adaptivity by allowing generation of parallel exercises at different levels of complexity. high parametrization of the exercise generation gives instructors control over the characterisitics of the generated exercises. an evaluation of the current implementation yielded promising results. the tool robustly generates functional exercises that comply with the user configurations. while the evaluation considered the performance aspects in isolation, in the future we plan to perform an end-to-end evaluation in an authentic education context. limitations of our tool arise from building on an existing system for input enrichment before performing more detailed linguistic analyses to support exercise generation. as a result, some of the language material provided by the input enrichment system is rejected during the exercise generation phase. we are thus considering to enrich the initial linguistic analysis performed in the input enrichment component to the more fine-grained level that will make it possible to use it for both the document ranking component and the exercise generation. future work also will be important to determine the effect of the parameter settings on the exercise complexity as experienced by the learner and to determine which parameter constellations are appropriate for generating developmentally proximal exercises for a given target population. this will open the path for adaptive sequencing algorithms to offer exercises optimally adapted to the learner's current proficiency level and cognitive capabilities.