Analyzing Bias in Large Language Model Solutions for Assisted Writing Feedback Tools: Lessons from the Feedback Prize Competition Series (2023)

Abstract: This paper analyzes winning solutions from the Feedback Prize competition series hosted from 2021-2022. The competition sought to improve Assisted Writing Feedback Tools (AWFTs) by crowdsourcing Large Language Model (LLM) solutions for evaluating student writing. The winning models are freely available for incorporation into educational applications, but the models need to be assessed for performance and other factors. This study reports the performance accuracy of Feedback Prize-winning models based on demographic factors such as student race/ethnicity, economic disadvantage, and English Language Learner status. Two competitions are analyzed. The first, which focused on identifying discourse elements, demonstrated minimal bias based on students’ demographic factors. However, the second competition, which aimed to predict discourse effectiveness, exhibited moderate bias.

## introduction assisted writing feedback tools (awfts) are a promising example of educational applications using natural language processing (nlp) algorithms that can innovate and accelerate student learning @xcite . recent advances in large language models (llms) have increased awfts' capabilities to process and provide feedback on student writing with human-like sophistication @xcite . the feedback prize competition series, hosted on kaggle in 2021-2022, was an important step in advancing awfts potential by crowdsourcing innovative llm solutions for assessing and evaluating student writing that were open science @xcite . the competitions were a success with over 6,000 teams participating and over 100,000 opensource algorithms developed. (the learning agency lab, n.d.) however, these algorithms have not been reported outside of the kaggle interface, limiting knowledge of their use and minimizing potential adoption into educational applications. additionally, the algorithms have not been assessed for bias, which may limit their effectiveness in a classroom setting, especially if that bias is aimed towards student populations that have been historically marginalized. the purpose of this study is to report initial performance for the winning feedback prize models and to disaggregate performance accuracy in demographic factors including race/ethnicity, economic disadvantage, and english language learner (ell) status. ## persuade corpus the first two competitions in the feedback prize series were based on the persuade (persuasive essays for rating, selecting, analyzing, and understanding discourse elements) corpus, a collection of ~25,000 argumentative essays written by students in the u.s. in grades 6 through 12 @xcite . the essays were annotated by experts for discourse elements and the effectiveness of the discourse elements. discourse elements refer to a span of text that performs a specific rhetorical or argumentative function, while discourse effectiveness is a rating of the quality of the discourse element in supporting the writer's overall argument. the effectiveness scale included ineffective, adequate, and effective ratings. the annotation scheme for discourse elements is based on an adapted or simplified version of the toulmin argumentative framework @xcite . ## feedback prize 1.0 models the first feedback prize competition, (feedback prize 1.0: evaluating student writing) was hosted on kaggle and involved the tasks of segmenting essays into smaller sections and assigning each section a discourse label such as lead, position, claim, and evidence. to evaluate performance, submissions were assessed based on the word overlap between ground truth and predicted outputs. a model prediction was considered correct (true positive) if there was at least a 50% word overlap between the machinesegmented section and the human-segmented section, as well as a match between their discourse label. false negatives were unmatched ground truths, and false positives were unmatched predictions. the final score was calculated by determining the number of true positives, false positives, and false negatives for each class (i.e., discourse label) and taking the macro f1 score across all classes. the analysis in this paper examines the secondplace, third-place, and sixth-place winning solutions from this competition. overall, the winning solutions were broadly based on ensembles of large-scale, pre-trained transformers, paired with custom pre-processing and postprocessing techniques to improve accuracy. the first-place model was not analyzed because its complexity made it difficult to replicate and impractical in educational settings. the overall macro f1 score did not differ significantly between the second-place, third-place, and sixth-place solutions, with values of .740, .740, and .732, respectively. to assess potential bias in the models, performance accuracy was further disaggregated by demographic factors (race/ethnicity, english language learner status, and economic disadvantage) and discourse effectiveness (ineffective, adequate, effective). specifically, ttests and anovas indicated that the average true positive rate (tpr) per essay of the second-place, third-place, and sixth-place models significantly varied based on demographic factors, but the effect sizes were small (see tables 1 2 3 ). none of the ttests or anova tests reported any results with a @xmath0.01 and a cohen's @xmath1.2. for instance, the t-test comparing tpr differences between ell and non-ell writing showed a p-value of 0.03 and cohen's d of 0.103 for the second-place model, ## feedback prize 2.0 models the second feedback prize competition (feedback prize 2.0: predicting effective arguments) also hosted on kaggle required models to predict the effectiveness rating of discourse labels, using multi-class logarithmic loss as the evaluation metric. more specifically, for each discourse label, the model had to submit the probabilities (or the likelihood) that the label belongs to each of the three effectiveness ratings (ineffective, adequate, effective). the closer the predicted probabilities were to the actual true label, the higher the model score would be. feedback prize 2.0 also prioritized computationally efficient algorithms, with a prize-incentivized "efficiency track" that evaluated submissions for both accuracy and speed. feedback prize 2.0 comprised a smaller subset of the data from the first competition (around 6,900 out of the 26,000 essays), due to a need for greater balance in effectiveness scores. in the complete persuade corpus, only 4% of discourse elements were labeled ineffective while 80% were labeled adequate and 16% were labeled effective. the subset used in feedback prize 2.0 corpus had a distribution of 18% ineffective, 24% effective, and 58% adequate, resulting in greater balance. the analysis presented in this paper examines the performance of the winning models (first, second, and third place) in the efficiency track on the competition test set. a common trend among winning solutions from the efficiency track was to fine-tune a single pre-trained transformer model on the competition dataset to minimize space and runtime requirements. the authors did not analyze the winners from the non-efficiency track because performance was similar, but computational demands were much higher. the analysis consists of two parts. the first part examines the accuracy of the models in predicting the three original effectiveness ratings (ineffective, adequate, effective). in the second part, the winning models' predictions were evaluated by grouping ineffective and adequate labels into a non-effective label, creating a binary outcome variable (effective, non-effective). this analysis recoded the labels 'post hoc,' after the model submitted probabilities for all three original ratings. in both analyses, the model's predicted label was determined as the label with the highest predicted likelihood among the outputted probabilities.