Bias at a Second Glance: A Deep Dive into Bias for G erman Educational Peer-Review Data Modeling (2022)

Abstract: Natural Language Processing (NLP) has become increasingly utilized to provide adaptivity in educational applications. However, recent research has highlighted a variety of biases in pre-trained language models. While existing studies investigate bias in different domains, they are limited in addressing fine-grained analysis on educational corpora and text that is not English. In this work, we analyze bias across text and through multiple architectures on a corpus of 9,165 German peer-reviews collected from university students over five years. Notably, our corpus includes labels such as helpfulness, quality, and critical aspect ratings from the peer-review recipient as well as demographic attributes. We conduct a Word Embedding Association Test (WEAT) analysis on (1) our collected corpus in connection with the clustered labels, (2) the most common pre-trained German language models (T5, BERT, and GPT-2) and GloVe embeddings, and (3) the language models after fine-tuning on our collected data-set. In contrast to our initial expectations, we found that our collected corpus does not reveal many biases in the co-occurrence analysis or in the GloVe embeddings. However, the pre-trained German language models find substantial conceptual, racial, and gender bias and have significant changes in bias across conceptual and racial axes during fine-tuning on the peer-review data. With our research, we aim to contribute to the fourth UN sustainability goal (quality education) with a novel dataset, an understanding of biases in natural language education data, and the potential harms of not counteracting biases in language models for educational tasks.

## introduction in recent years, natural language processing (nlp) and machine learning (ml) have been extensively used for improving adaptivity and individualization of educational technology @xcite . researchers and practitioners have been developing a plethora of writing support systems @xcite and conversational agents @xcite . more generally, there has been a rise in intelligent tutoring systems for educational purposes which provide learners adaptive feedback, e.g., on grammatical structures @xcite @xcite , language learning @xcite , argumentation @xcite , or even empathy skills @xcite . the technology for language-based personalization in education comes with a cost; a large body of research has been investigating and revealing biases in nlp systems @xcite . bias has been found in multiple steps along the general nlp pipeline including the task setting, training data, pre-trained models (e.g. word embeddings), and fine-tuned algorithms @xcite @xcite , shedding a darker light on the simple usage of these models for human-centered applications, especially in education. nlp systems containing bias in any of these parts of the modeling pipeline can produce gender, racially, or conceptually biased predictions and amplify biases present in the underlying training sets (e.g., @xcite ; @xcite ; @xcite ). the propagation of gender bias in nlp algorithms poses the danger of reinforcing damaging stereotypes in downstream applications, e.g., for automatic essay scoring @xcite . while prior research on bias in education has mostly focused on non-language based interaction data, several recent reviews have called for extending the investigations of fine-grained bias analysis on educational corpora (e.g., @xcite ; blodgett et al. ( 2020 )). recent work, for figure 1 : overview of evaluating biases in educational natural language data along the nlp pipeline for pedagogical downstream tasks following @xcite . we analyzed a data set of 9,165 german peer-reviews in combination with the most common pre-trained language models (t5, gpt-2, bert) and glove embeddings before and after fine-tuning with the weat analysis for conceptual, racial, and gender biases. example, has shown negative impact of gender bias on cv screening @xcite or of algorithmic racial bias in child welfare programs @xcite . there are only few works looking at detailed bias in educational natural language data outside of english language corpora and north american context (baker and hawn, 2021). for instance, baker and hawn (2021) states the need to investigate "the differences in the performance of essay scoring algorithms for different racial groups". however, as they found, "this possibility has not yet been systematically investigated in the published literature" (baker and hawn, 2021). hence, our objective is to address this gap in research and to take a deep dive into one exemplary pedagogical scenario which includes heavy language data: student peer-reviews. student peerreviewing is a modern domain-independent pedagogical scenario which has been increasingly used to annotate corpora, analysis of feedback texts with trained models, and provide students feedback with adaptive applications @xcite , e.g., for argumentation skill training @xcite or empathy skills @xcite . in order to conduct a rigorous bias analysis, we collected a novel corpus of 9,165 german student peer-reviews of business model feedback. we relied on word embedding association test (weat) analysis @xcite and the german adaptation of weat (kurpicz-briki, 2020) as a commonly used methodology to assess conceptual, racial and gender bias in different parts of the nlp pipeline @xcite . our methodology for analysing the bias is three-fold (see figure 1 ): (1) we analyse the collected corpus for different bias dimensions to find out if the student-writings already come with bias towards the perceived helpfulness of a review, (2) we assess the most common german language models (t5, bert and gpt-2) as well glove embeddings before fine-tuning them on our data, (3) we fine-tune t5, bert and gpt-2 on our collected data-set and repeat the weat analysis to investigate how the representations have been changed. contrary to our expectations, we found that our collected corpus does not reveal many biases in using weat co-occurrence analysis or glove models; however, the pre-trained german language models not only come with substantial conceptual, racial, and gender bias but also seem to increase the bias when fine-tuning on our corpus. our results suggest to (1) do more fine-grained analyses of bias for subsets of data that are significant, (2) examine the bias in pre-trained models before using them, and (3) investigate multilingual data bias more precisely. hence, we contribute to literature on bias of educational language data by providing a detailed analysis of one particular but increasingly used pedagogical scenario (peer-reviewing). we contribute our collected corpus of peer-reviews in german for further analysis and hope to provide researchers and practitioners with a detailed analysis and discussion of bias in nlp for education. finally, we aim to contribute to the un sustainability goal four for a high quality education and fair (digital) education for all. ## results on bias analysis we present results across three stages of the bias pipeline (highlighted in figure 1 ): (1) a weat co-occurrence analysis examining bias directly in the peer-review corpora, (2) an embedding space analysis using a glove architecture trained on the peer-review data, (3) an analysis of the three most @xcite . ## discussion and conclusions are results add insights to the literature about gender bias in educational data modelling (e.g., anderson et al. (2019) ), in embedding spaces (e.g., bolukbasi et al. (2016) ), and in language modelling (e.g., @xcite ). in our research, we built on the findings from these different perspectives along the nlp pipeline and conducted a fine-granular analysis for german peer-reviews by analyizing the texts, the qualitative review scores, demographics (i.e, gender), embeddings, and the most common pre-trained language models before and after finetuning. our results suggest three main directions. first, bias can emerge and change along the nlppipeline. detecting a certain bias in the corpora or in pre-trained language models does not necessi-tate a connection to bias in fine-tuned models for downstream tasks. thus, it is necessary to have more in-depth analyses of bias not only along the entire nlp pipeline but also for subsets of data that are significant. second, it is important to examine the bias in pre-trained models before using them. and third, more investigations on multilingual data bias are necessary. therefore, we contribute to literature on bias of educational language data by providing a fine-grained analysis of one particular but increasingly used pedagogical scenario (peerreviews). we contribute our collected corpus of peer-reviews in german for further analysis and hope to provide researchers and practitioners with a detailed analysis and discussion of bias in nlp for education. finally, we aim to contribute to the un sustainability goal four for a high quality education and fair (digital) education for all.