AGR e E : A system for generating Automated Grammar Reading Exercises (2022)

Abstract: We describe the AGReE system, which takes user-submitted passages as input and automatically generates grammar practice exercises that can be completed while reading. Multiple-choice practice items are generated for a variety of different grammar constructs: punctuation, articles, conjunctions, pronouns, prepositions, verbs, and nouns. We also conducted a large-scale human evaluation with around 4,500 multiple-choice practice items. We notice for 95% of items, a majority of raters out of five were able to identify the correct answer, for 85% of cases, raters agree that there is only one correct answer among the choices. Finally, the error analysis shows that raters made the most mistakes for punctuation and conjunctions.

## introduction acquiring a language necessitates learning its grammar. in the united states, the common core standards for k-12 english literacy reflect this by including grammar as a learning outcome across all grade levels. 1 while both students and educators acknowledge that learning grammar is "necessary and effective" for language acquisition, it is "not something they enjoy doing" @xcite . given the sheer amount of grammar constructs and rules, creating exercises that target each of these rules can be tedious. likewise, for students, completing these exercises can become repetitive. to support teachers and more generally a growing interest in formative and computer-based assessments in the educational testing field, the demand for automatically created multiple-choice questions is growing @xcite . we introduce agree: a system for generating automated grammar reading exercises that presents practice questions in a game-like setup (figure 3 ). while gamification is not an explicit part of our system, agree offers immediate feedback, similar to a more formal incentive system that can boost user engagement and motivation by rewarding correct answers @xcite . in related research @xcite ; @xcite find that games which focus on drilling and practice have a positive effect on language acquisition. the grammar questions are embedded within a text in the form of masked sentences, and users are encouraged to solve each question to uncover the whole text. finally, agree also allows you to attach any type of reading material for practice; giving learners agency over their choice of reading material has been shown to improve learner engagement @xcite . the generated questions in agree are a variation on the cloze task @xcite . in this task, part of a sentence is removed and replaced with a blank and the goal is to recover the missing portion. while originally introduced to evaluate a text's readability, it has since been widely adapted for language practice and assessment. figure 1 illustrates an example of a practice item in agree. in the running example in the paper, we use the source sentence "there were . . . in the americas". the sentence contains the correct target word "any-figure 2 : the passage submission page accepts texts up to 1,000 characters long. since agree designs to generate items for only one construct per sentence, learners can re-order the grammar constructs as per their choice above the text box by dragging and dropping the numbered boxes. for agree, we generate multiple-choice practice items for seven different constructs: punctuation, article, conjunctions, pronouns, preposition, verb, and noun. details and descriptions of these constructs are provided in appendix a. given a sentence, we identify the token corresponding to the above constructs and mask it from the sentence to form the question item. next, we use bert to identify the distractor choices (section 3.5). note, we did not use any custom tokenizer here but instead used the default tokenizer from bert. after we design the grammar practice tool agree, we conducted a thorough user-study via the crowdsourcing platform amazon mechanical turk (mturk). we ask the crowd-raters to attempt around 4,500 grammar items and then asked them to provide feedback on the quality of the items (see section 4). we notice that in 95% of the items, a majority of crowd-raters were able to identify the correct answer whereas around 85% of responses agree that the items have only one correct answer. on the contrary, in our error analysis we found that crowd-raters often made mistakes for punctuation and conjunction items showing these two constructs are harder than the rest. the grammar practice tool agree is available online for practice and learning. foot_1 2 overview to begin interacting with our tool, a user must submit a passage from the live demo page shown in figure 2 . the available grammar constructs are shown above the free text box, and can be reordered in terms of priority from left to right. this enables the teacher or student to customize the experience based on learning goals. after the passage is submitted, the text is sent to a back-end implemented using amazon web services (aws) components. figure 3 : after a user clicks on a masked sentence on the left side, a grammar question is revealed to the right of the window. the masked sentences are color-coded depending on the grammar constructs, and the short name for each construct can be found in the banner at the top of the page. see appendix a for the full names of the grammar constructs, which also appear under the banner when hovered over. the passage shown is an excerpt from the tasa corpus we describe in section 2. after submission, the system will create grammar questions from the passage (see figure 3 ). grammar item generation happens at the sentence level. since completing a cloze task relies on filling in a blank based on the surrounding context @xcite , we create an item out of every other sentence so that the user is provided with both enough context to fill in the blank, and with enough practice as they progress through their reading. only one grammar item is generated per sentence, and this item will correspond to the first grammar construct that can be found in the sentence that our distractor generation algorithm successfully generates an item for. next, the grammar items will appear to the user in the form of masked sentences. when a sentence is masked, it becomes clickable to the user. and once clicked on, the question choices will appear on the right side of the window. if the student selects an incorrect answer they are prompted to try again. but when the correct answer is selected the sentence becomes unmasked so that they may proceed with their reading. for each correct an-swer agree provides feedback to encourage and motivate the learner. the system has been tested thoroughly on informational texts such as the touchstone applied science associates (tasa) corpus which "consists of representative random samples of text of all kinds read by students in each grade through first year of college" @xcite . note that the system itself does not have any dependency to a particular corpus, and so in theory can be used with texts from any domain. ## grammar item generation creating a grammar item from a sentence involves the following steps: ## evaluation we are interested in evaluating the quality of the multiple-choice items in agree. to that end, we conduct a large-scale mturk study where we present crowd-raters with our generated grammar exercises as shown in figure 3 . we ask the raters to select their choice for each item, and to answer followup questions along four aspects of item quality. we paid raters @xmath02.00 usd for completing our main task, which is approximately equivalent to @xmath124 usd (per hour). in total, the evaluation cost $12,575 usd. ## related work tools that enhance authentic texts in support of grammar acquisition include werti @xcite , flair @xcite , and grammartagger @xcite . except for a verb practice activity in werti, these tools provide few opportunities for immediate feedback. our system fills this gap by generating multiple-choice practice questions from authentic texts. on the other hand, there are also systems that generate multiple-choice grammar questions. fast @xcite covers nine grammatical categories and lee et al. (2016) create a system for learning preposition usage. we extend these works by not only covering a range of grammatical categories and collecting perception responses, but also conducting a large-scale evaluation on item performance (i.e., whether the item can be solved by the user with some competence in the language). ## conclusion we describe agree, a system and procedure for converting an informational passage into game-like grammar practice exercises that can be completed while reading. we find in human evaluations that nearly all the multiple-choice questions we generate for the exercises are correctly keyed, and can therefore be used to provide immediate feedback to students. we also observe for almost 95% of items that a majority of the raters were able to identify the correct target. on the contrary, raters made the most mistakes for punctuation and conjunctions. although we did not design our system to include gamification explicitly, it is set up in a way that makes it easy to incorporate in the future. the system can automatically generate a wealth of questions for which the correct answer is identified, and these questions can be used by whomever (e.g., game designers) to create a game that rewards learners when they correctly solve them. as it stands, the system allows teachers some control over the generated items; the reading material itself and the ranking of grammar constructs can be customized. however, teachers do not have control at the level of individual items. we may want to build in this finer-grained control in the future so that the exercises can be adapted more closely to their needs. currently, the number of questions generated for a given construct (e.g., preposition) depends on that construct's ranking, its frequency in the text, and how well it is covered in the manually created list of distractor choices. for example, there may be fewer preposition items generated than expected due to the fact that we use a reduced set of prepositions in agree. if we expand the set of distractor choices to cover all possible prepositions, we would likely run into latency issues. while increasing the coverage of existing constructs is one potential line of future work, it may be more important to find ways to align our constructs to existing efl (english as a foreign language) curricula if we want to create efficacious questions. since the constructs that a learner struggles with is influenced by-among other factors-aspects of their learner profile such as english level (hawkins and buttery, 2010) and language background, an efficacious system should take these aspects into account. doing so may allow agree to provide more personalized feedback and generate distractors that can be calibrated. for example, we can imagine using token-level probabilities to make filtering decisions about whether an item is appropriate for a certain language level. the distance between the target log probability and the log probability of the hardest distractor can be smaller for a learner whose language level is higher, assuming that as language ability improves, so does the ability to discriminate between a correct choice and a plausible yet incorrect one. this raises the question about the threshold at which a token can be considered grammatical versus not, as illustrated in the coordinating conjunction example in section 4.2. according to @xcite , one way to think about grammar is to see it as an interaction between form/structure, meaning/semantics, and use/pragmatics. @xcite also argue that, when it comes to learner english, there is no "clear dichotomy between innovation and error". following this line of thinking, we view the threshold for grammaticality as context-specific and tied to pedagogical goals. more experimentation is also needed to determine how the quality of the questions changes when individual components are altered or replaced. to increase the flexibility of the system, we can think of replacing the token matching and syntactic pattern matching components with a more specialized model that identifies the gap locations, such as the one described in @xcite . as such, making the system more functional would involve building in the ability to evaluate the output of individual components in addition to the final output.