Can Language Models Guess Your Identity? Analyzing Demographic Biases in AI Essay Scoring (2024)

Abstract: Large language models (LLMs) are increasingly used for automated scoring of student essays. However, these models may perpetuate societal biases if not carefully monitored. This study analyzes potential biases in an LLM (XLNet) trained to score persuasive student essays, based on data from the PERSUADE corpus. XLNet achieved strong performance based on quadratic weighted kappa, standardized mean difference, and exact agreement with human scores. Using available metadata, we performed analyses of scoring differences across gender, race/ethnicity, English language learning status, socioeconomic status, and disability status. Automated scores exhibited small magnifications of marginal differences in human scoring, favoring female students over males and White students over Black students. To further probe potential biases, we found that separate XLNet classifiers and XLNet hidden states weakly predicted demographic membership. Overall, results reinforce the need for continued fairness analyses as use of LLMs expands in education.

## introduction as large language models (llm)s are increasingly used for automated essay scoring (aes), it is crucial that we thoroughly analyze these systems for biases @xcite . given that llms are pretrained on large corpora, they have the potential to inherit biases embedded in the functions that predict word probabilities @xcite . if the potential biases are not monitored carefully with fairness in mind, they risk perpetuating and amplifying existing societal biases against vulnerable populations. rigorous demographic analysis of aes systems help ensure they live up to principles of equity and fairness. the persuasive essays for rating, selecting, and understanding argumentative and discourse elements (persuade) corpus provides a valuable re-source for analyzing bias in aes system @xcite . persuade contains over twentyfive thousand persuasive student essays that were annotated for argumentative elements in addition to holistic grades assigned by human raters. what makes this corpus ideal for the analysis of bias is the rich metadata about the students including gender, race, and other demographic indicators. this allows for in-depth analysis of an automated scoring system's performance on essays written by students of diverse demographic affiliations. our goal is to investigate potential biases in llms trained using conventional techniques that aim to replicate human-assigned holistic scores. after training the llm scoring model, we evaluate whether or not automated scores introduce (or exacerbate) biases relative to human-assigned scores @xcite . after evaluating bias, we determine whether the set of features that the llm uses for scoring also contains information relevant to demographic membership. in other words, can the llm guess certain demographic characteristics based on the scoring model? linear modeling using these features was recently used as evidence of model validity in aes @xcite . the novelty of this study lies in applying these techniques and showing their relevance to the analyses of bias in llms. broadly, our research aims are as follows: these aims help us determine if llms are able to score students' essays fairly, and if demographic affiliations are an implicit feature of the scoring model. ## results we organize our findings around three foci. first, we evaluate the performance of xlnet to ensure it meets operational standards. second, we assess the fairness of xlnet's automated scores by determining if there are any discrepancies, based on students' demographic affiliations, as compared to human-assigned scores. finally, we determine the extent to which the scoring model has demographic features embedded within it.