Unraveling Downstream Gender Bias from Large Language Models: A Study on AI Educational Writing Assistance (2023)

Abstract: Large Language Models (LLMs) are increasingly utilized in educational tasks such as providing writing suggestions to students. Despite their potential, LLMs are known to harbor inherent biases which may negatively impact learners. Previous studies have investigated bias in models and data representations separately, neglecting the potential impact of LLM bias on human writing. In this paper, we investigate how bias transfers through an AI writing support pipeline. We conduct a large-scale user study with 231 students writing business case peer reviews in German. Students are divided into five groups with different levels of writing support: one in-classroom group with recommender system feature-based suggestions and four groups recruited from Prolific – a control group with no assistance, two groups with suggestions from fine-tuned GPT-2 and GPT-3 models, and one group with suggestions from pre-trained GPT-3.5. Using GenBit gender bias analysis and Word Embedding Association Tests (WEAT), we evaluate the gender bias at various stages of the pipeline: in reviews written by students, in suggestions generated by the models, and in model embeddings directly. Our results demonstrate that there is no significant difference in gender bias between the resulting peer reviews of groups with and without LLM suggestions. Our research is therefore optimistic about the use of AI writing support in the classroom, showcasing a context where bias in LLMs does not transfer to students’ responses.

## introduction large language models (llms) have proven useful for improving the adaptivity and individualization of educational technology @xcite . researchers and practitioners have been developing a plethora of educational writing tools based on natural language processing (nlp), for example for writing suggestions, @xcite conversational interaction @xcite , or to support writing skills and learning at scale. nevertheless, the increasing use of llms for personalized support (especially in education) bears issues of critical concern. research has found that these models can carry and propagate significant bias @xcite . llms, at any stage in their development pipeline, can harbor and propagate biases (e.g., gender, racial, or conceptual) and thus reflect the data they are trained on @xcite @xcite . such biases, specifically gender bias, in llms, can reinforce harmful stereotypes in automated writing support applications, for example in automated essay scoring @xcite or individual writing support @xcite , and thus can inadvertently influence students' writing styles and perspectives @xcite . previous research has explored the existence of bias in different language models and their representations, focusing on the english language @xcite . however, a growing body of literature advocates for the necessity of conducting comprehensive bias analyses, i.e. analyzing the impact of language models when embedded in human educational tasks @xcite @xcite . especially for gender stereotypes, recent studies such as @xcite and @xcite have demonstrated the effects of such stereotypes on cv screening and child welfare programs, respectively. nevertheless, detailed examinations of the effects of llm-based writing suggestions on students and their use of gender stereotypes have been rather sparsely investigated, especially in other lan-guages than english (baker and hawn, 2021; @xcite . given the expanding use of these models in educational settings and for writing assistance in general, (e.g., @xcite , addressing this literature gap is of utmost importance. understanding the nuanced ways in which biases in llms can seep into educational tasks can help researchers create safer and more effective learning environments that promote equitable outcomes. in this paper, we analyze how bias transfers through an ai writing support pipeline and we investigate whether bias in llms translates into bias in human writing. we use an educational context, namely german peer reviews collected from 231 students divided into five groups: one group of 52 students in a university classroom setting receiving feedback from a traditional feature-based recommender system (g0) and four groups (179 students in total) recruited through prolific. groups g1-g4 include a control group receiving no writing support (g1), two groups with suggestions from gpt-2 (g2) and gpt-3 models (g3) fine-tuned on an extended version of the non-biased corpus of @xcite containing 11, 925 peer reviews in german, and one group with suggestions from pre-trained gpt-3.5 (g4). we apply the genbit gender bias test @xcite , the german adaptation kurpicz-briki (2020) of the word embedding association tests (weat) @xcite , and sentence embedding association test (seat) @xcite translated to german to the collected peer reviews as well as to the suggestions provided by the different llms and the embeddings of finetuned gpt-2 model. with our analyses, we aim to answer the following two research questions: our results reveal a promising trend: groups receiving suggestions from llms (g2-g4) exhibit no significant measurable difference in gender bias in their writing compared to the control group without writing support (g1) and the in-classroom students receiving recommender-based feedback (g0). furthermore, none of the genbit gender bias, weat tests and seat tests reveal biases in the provided suggestions from any of the llms, despite that the analysis of gpt-2 embeddings detects significant gender bias for the gpt-2 model. our results therefore demonstrate that llm-based writing support can be positively used for specific educational scenarios without bias. ## methodology to investigate the impact of llms on gender biases in human writing within an educational context, we performed a bias analysis in three steps (see figure 1 ). in a first step, we trained a feature-based recommender system and fine-tuned gpt-2 and gpt-3 models on a corpus of 11, 925 student-written business peer reviews in german to be able to provide automated suggestions in a peer review exercise. in a second step, we conducted a large-scale user study with 231 students in a classroom and an online context, providing students with different levels of writing support (feature-based recommender, no support, suggestions by fine-tuned gpt-2 or gpt-3, suggestions by pre-trained gpt-3.5) in a peer review writing exercise in german. in the third and final step, we used the genbit gender bias score @xcite , weat, and seat tests to analyze the gender bias of the resulting reviews, the model suggestions, as well as the gpt-2 embeddings. ## results to evaluate whether bias transfers from ai assistants to students for a real-world writing support scenario (rq1) and to investigate where in the pipeline the bias persists (rq2), we applied the genbit bias score as well as the weat tests to the five different subsets from our user study: four from the prolific user study (control group g1, gpt-2, gpt-3, and gpt 3.5 assisted reviews g2-g4) and one from the classroom study with recommender system suggestions (g0). to answer our first research question, we computed the genbit bias score for the peer reviews written by the students. the goal was to identify which biases in the models transfer to humans through collaboration. we experimented with both a fixed context for windows sizes ranging from 10 to 60 as well as an infinite context. since there was no significant difference between the results from the fixed context and infinite context, and the optimal window size was determined to be 20 by previous work @xcite , in this section, we only present results for a fixed context with a window size of 20. the resulting genbit bias scores are illustrated in figure 2 . the total number of reviews from each group and statistical summary of bias scores for all groups are presented in to analyze differences between the bias scores of the four prolific groups (g1-g4), we first aggregated the mean bias scores of each review from these groups. the results of a shapiro-wilk test (shapiro and wilk, 1965) then showed that our data was not normally distributed. additionally, a levene test @xcite confirmed heteroscedasticity within the data. therefore, we chose the non-parametric mann-whitney u test (mwu) to analyze the difference in mean bias scores between any pairing of two groups and the kruskal-wallis test @xcite for the difference among all four groups. ## discussion and conclusion llms are increasingly used in educational settings, despite that they harbor inherent biases which may have a negative impact on learners. our work analyzed how bias transfers through an ai writing support pipeline in an educational context and whether the bias in llms translates to bias in students' writings. to do so, we conducted a large-scale user study, providing students with different levels of writing support in a peer review writing exercise. our analysis of data collected from in-classroom and prolific participants yielded positive results that provide optimism for the field of nlp and its application within educational contexts. the most notable finding was that students who received writing suggestions from llms exhibited the same degree of gender bias in their written text as students who received no suggestions. the group receiving suggestions from a recommender system with human interpretable features also showcased a similar amount of bias in student responses. our results seem initially promising, showing that by measuring gender bias through multiple tests (gen-bit, weat, and seat) at each stage of the pipeline (model embeddings, model suggestions, student output), llms do not inadvertently foster and perpetuate gender bias. one possible explanation for these results is that the biases present in the original training data and embeddings of the llms were not transferred to the model suggestions, as indicated by the lack of bias in model suggestions. unbiased suggestions led to positive learning outcomes for students without an inadvertent bias transfer. our post-hoc deep dive into the bias dimensions of the suggestions and the gpt-2 embeddings revealed the inherited bias in the fine-tuned models. this analysis provided valuable insight into how biases can become ingrained in llms and also identified where in the pipeline the bias stops. it suggests that even when models are fine-tuned with the intention of reducing bias, their original training on vast amounts of data from the internet can leave an indelible imprint of bias. an important takeaway from our study is that the applied domain of education, although it is often a sensitive context, is moving towards integration with llm assistants. before we can be fully confident of model impacts on sensitive young minds for downstream tasks, we need to strive for not only more sophisticated bias detection and mitigation techniques but also more transparency in how these models are trained and fine-tuned. also, future research is needed to better investigate the impact of potentially biased llms in different educational settings in addition to writing, for example, language learning @xcite , stem education @xcite , and legal education @xcite . in conclusion, our study contributes towards a more nuanced understanding of how llms interact with bias for educational tasks, and produces the indication that although models contain bias, personalized downstream applications might not. we hope that our findings will stimulate further research, contributing to the un's fourth sustainability goal of ensuring quality education for all.