Automated Essay Scoring Using Grammatical Variety and Errors with Multi-Task Learning and Item Response Theory (2024)

Abstract: This study examines the effect of grammatical features in automatic essay scoring (AES). We use two kinds of grammatical features as input to an AES model: (1) grammatical items that writers used correctly in essays, and (2) the number of grammatical errors. Experimental results show that grammatical features improve the performance of AES models that predict the holistic scores of essays. Multi-task learning with the holistic and grammar scores, alongside using grammatical features, resulted in a larger improvement in model performance. We also show that a model using grammar abilities estimated using Item Response Theory (IRT) as the labels for the auxiliary task achieved comparable performance to when we used grammar scores assigned by human raters. In addition, we weight the grammatical features using IRT to consider the difficulty of grammatical items and writers’ grammar abilities. We found that weighting grammatical features with the difficulty led to further improvement in performance.

## introduction automated essay scoring (aes) is a task that automatically grades essays. essay assignments are widely used in language tests and classrooms to assess learners' writing abilities, while grading them takes time and effort for human raters. maintaining inter-and intra-rater reliability is another issue associated with human scoring. aes can help alleviate these problems and has been attracting more attention in recent years. the grading methods for essays can be roughly categorized into two types: holistic scoring and analytic scoring. the former assigns a single score to an essay based on its overall performance, while the latter assigns different scores to various aspects 1 the code is publicly available at https://github.com/ ahclab/aes-grammar-mtl-irt . among those aspects, we focus on grammatical features, inspired by the research on criterial features for the levels of the common european frameworks of references (cefr) @xcite in l2 english @xcite . the cefr, one of the influential frameworks in language teaching, describes language abilities in functional terms (i.e., can-do statements, such as "can write short, simple essays on topics of interest"). however, it is grammatical items and lexis that realize the functions written in can-do statements. to fully develop and elaborate their ideas in essays, they need to use a wide range of grammatical items. in fact, grammar plays an important role in essay scoring. researches on writing in the second language acquisition field have been focusing on syntactic complexity 2 and accuracy (see @xcite . @xcite identified grammatical items that learners at a certain level and higher can use correctly and items that learners at a certain level are prone to making mistakes in. it is known that human raters look for those features consciously or unconsciously when they evaluate learners' performance, and explicit use of grammatical features in aes will improve model performance. grammatical features have been used in many feature-engineering aes models (see @xcite as well as in hybrid models, which incorporate handcrafted features into deep neural network aes models @xcite @xcite . in @xcite , features representing grammatical structures were used together with other linguistic features. however, in many previous studies (e.g., @xcite , grammatical items used correctly were aggregated into measures of grammatical complexity (e.g., ratio of dependent clauses per clauses; see @xcite rather than individual grammatical items (e.g., adverbial clause if, adverbial clause so that) even though the difficulties of individual grammatical items are different. in this paper, we propose to use individual grammatical items as inputs to hybrid aes models that predict holistic scores, and leverage the models to incorporate the variety of grammatical items in grading essays. we also use frequencies of grammatical errors corrected by a modern grammatical error correction model (gector-large; @xcite as model inputs. the grammatical features are combined with an essay representation and passed into a fully connected feed-forward neural network to predict the score of an input essay. our models used bert @xcite to learn essay representations following the current state-of-the-art aes models @xcite @xcite . to utilize grammatical features more effectively, we develop a multi-task learning framework that jointly learns to predict holistic scores and grammar scores of essays. we use two types of grammar scores: (1) scores assigned to essays by human raters and (2) writers' latent abilities estimated based on patterns of grammar usage using item response theory (irt) @xcite . note that teacher labels are not necessary for estimating the latent abilities using irt. irt estimates not only each writer's ability but also the characteristics of each item (i.e., individual grammatical item), such as discrimination and difficulty parameters. therefore, we use these irt parameters to weight grammatical items (e.g., award writers who use a difficult grammatical item; see section 3.1.2). in summary, the contributions of this paper are as follows: our method shows a significant advantage on some essay assignments in the automated student assessment prize (asap) dataset foot_1 . ## conclusions this study examined the effectiveness of using grammatical features in aes models. specifically, we fed two kinds of features: (1) grammatical items that writers used correctly in essays (pfs), and (2) the number of grammatical errors (nfs). we showed that both pfs and nfs improved the model performance, but combining them did not result in further improvement. the experimental results suggest that multi-task learning would be effective to take advantage of the information that the grammatical features have. one of the future directions could be exploring effective ways to combine pfs and nfs to improve the model performance since the way in this study was a simple concatenation of the two vectors (e.g., to learn representations for pfs and nfs in different networks and combine them). another direction would be to examine the effectiveness of adding our grammatical features in aes using a large language model. it potentially improves the scoring performance in zeroand/or few-shot settings @xcite . furthermore, in order to have more interpretable models, it would be beneficial to analyze how much individual grammatical features contribute to model's score prediction. the insights delivered by interpretable models can help practitioners in education. we also weighted pfs in several ways using irt parameters and found that considering the difficulties of grammatical items would improve the model performance. in addition, we used the ability parameter θ as teacher signals for the auxiliary task in multi-task learning. although no humanannotated labels are required to estimate the irt parameters, the model trained with the ability parameter achieved comparable performance to the model trained with grammar scores annotated by human raters. in this study, irt parameters were estimated based on grammatical items that writers used in their essays. in the future, we will apply irt to both pfs and nfs to model writers' grammar abilities.