A Fairness Analysis of Human and AI -Generated Student Reflection Summaries (2024)

Abstract: This study examines the fairness of human- and AI-generated summaries of student reflections in university STEM classes, focusing on potential gender biases. Using topic modeling, we first identify topics that are more prevalent in reflections from female students and others that are more common among male students. We then analyze whether human and AI-generated summaries reflect the concerns of students of any particular gender over others. Our analysis reveals that though human-generated and extractive AI summarization techniques do not show a clear bias, abstractive AI-generated summaries exhibit a bias towards male students. Pedagogical themes are over-represented from male reflections in these summaries, while concept-specific topics are under-represented from female reflections. This research contributes to a deeper understanding of AI-generated bias in educational contexts, highlighting the need for future work on mitigating these biases.

## introduction reflection is an effective metacognitive technique that promotes student learning @xcite . reflections can be used in a classroom setting to gather feedback from students on their comprehension and help both students and instructors identify topics of confusion. given the substantial amount of reflection data in large classes, ai-based summarization techniques have been developed to summarize these reflections @xcite @xcite . automatic summarization @xcite is a popular nlp technique used to create or sample a smaller text that represents the most important or relevant information within the original content. this process inevitably involves decisions about which is the most important or relevant information. ai bias is a well-discussed topic in recent years. efforts to identify and mitigate bias in ai and nlp systems have been applied to tasks such as language modeling @xcite @xcite @xcite , coreference resolution @xcite , and machine translation @xcite . specifically within nlp research for education, bias has been investigated in educational technologies like automated essay scoring @xcite and intelligent tutoring systems @xcite . reflection summarization is an important use case as it help instructors uncover student misconceptions, empowering them to adapt their instruction and create targeted learning opportunities that address knowledge gaps in subsequent lectures @xcite . since the goal of reflection summarization is to save teaching staff time and reduce the need to read through so many reflections, biases in whose reflections are represented by the summaries can have a direct impact on whose concerns are addressed by teaching staff. this concern motivates our study to measure biases in summarization of student reflections. specifically, we scope our research to identifying if there are differences by student gender in a dataset of classroom reflections and if the summaries of these reflections exhibit bias toward any gender. we are particularly interested in representation from female reflections due to a history of exclusion of women in stem classes @xcite . unfortunately, we are only able to compare the representation of reflections students with those of male students within a gender binary, as we do not have sufficient data on the experiences on nonbinary students, an important topic for future work. using the structural topic model (stm; @xcite , we are able to model variation in topics within reflections along with the gender of the authors of these reflections. we also apply stm to measure how closely topics in summaries match those in reflections from male and female genders. we evaluate gender bias in several types of ai-generated summaries and contrast these with human-annotated summaries. we define our research questions as follows: rq1 what differences, if any, are there between reflections from male or female students? rq2 are summaries biased towards any specific gender? rq3 if so, what is the nature of the gender bias in reflection summaries? using stm, we find subtle differences between reflections of male and female students, particularly a stronger emphasis on course logistics (such as projects) from female students. measuring differences between summary topic distributions and those of male and female reflections, we find that ai abstractive summarization models exhibit bias toward reflections from male students, while summaries from humans and ai extractive models do not show a consistent bias. we find that ai abstractive summaries appear to under-represent specific course concepts that are brought up in reflections from female students, while over-representing pedagogical themes such as teamwork from male student reflections. ## dataset description so each lecture has two summaries corresponding to each reflection prompt. for both human annotation and ai generation, three types of summaries were annotated or generated: extractive, phraselevel extractive, and abstractive. while creating human annotations, annotators were asked to extract five reflections and five phrases that best represent all student reflections for extractive and phraselevel extractive summaries respectively. they were also asked to write an abstractive summary to summarize the major points of student reflections. in the case of automatic summarizing, we evaluate a selection of models presented by @xcite , including those that are fine-tuned on human annotations as well as those that use causal language models like chatgpt in a zero-or fewshot setting 1 . among these, we have selected the two best performing approaches, from findings by @xcite , to collect summaries for each summary type: 1. extractive summary: matchsum @xcite and gpt-reflect @xcite . matchsum uses a re-ranker, and follows a two-stage paradigm to achieve state-of-the-art extractive summarization. gpt-reflect uses chatgpt (gpt-3.5 turbo) to generate zeroshot extractive summaries from reflections. for our analysis, we remove reflections by students who do not disclose gender information, leave it blank, or self-describe their gender. we hope to analyze non-binary genders in future work with more data available. reflectsumm has annotations and summarizations for all student reflections, including those who do not provide demographic information. we considered summaries where at least 80% of the reflections they summarize are from students who indicated male and female gender. this gave us a collection of 250 summaries. ## analysis methodology our aim is to analyse any gender bias present in reflection summaries. in order to achieve this goal we apply topic modeling. topic modeling learns a distribution over a set of topics for a given text document in an unsupervised fashion. we aim to capture what topics are reflected in summaries and measure their variance according to document metadata, in our case the gender of students who wrote the reflection. stm is designed for just this: to associate topics with document metadata. stm brings out the latent topics in a corpus of text and allows the use of additional covariates to alter the prior distribution used to estimate the latent topics better. this feature sets apart stm from other topic models like latent dirichlet allocation (lda) and bertopic @xcite . we first used topic modeling to learn the topics present in the student reflections we have collected and provide the topic distribution for each reflection. we analyzed this topic distributions of reflection across genders to address rq1 and explore differences between reflections from male and female students. to address rq3 on the nature of any bias in summaries, we examine topics in the summaries that are over-represented from the reflections of some groups while under-represented from the reflections of other groups. in order to identify the nature of bias, we compute a discrepancy measure between the topic probabilities in summaries and the genders involved. ## rq2: are summaries biased towards any specific gender? to measure how closely summaries represent the reflections of male or female students, we estimate the distance in topics captured in summaries from those presented in reflections from both genders. ## conclusion in this work, we present the results from our fairness analysis of reflectsumm @xcite , a benchmark student reflection summarization dataset. we structured our analysis around three research questions: what topics differ between student reflections between male and female students (rq1), are different types of summaries of those reflections biased toward any gender (rq2), and if so, what is the nature of that bias (rq3)? we found slight topical differences between male and female reflections, such as female students being more likely to mention course logistics and refer explicitly to their own learning than male students. we also found that ai-generated abstractive summaries were biased towards male reflections, irrespective of whether the model was trained on human annotations or used generative causal model like chatgpt. human-generated summaries and extractive ai summaries did not exhibit consistent patterns of gender bias. for the abstractive ai summaries, we found that topics with a pedagogical theme in the summaries are over-represented from male reflections while concept-specific topics were under-represented from female reflections. such biases caution the use of popular llm-based abstractive summarization techniques with educational reflection data. this work could be extended to other educational datasets such as oulad @xcite , which has more demographic data, however there are not many student reflections available. some issues with working with reflections data is hence the size and availability of these datasets. our work could also be extended to analyze other demographic information present in reflectsumm, such as race and ethnicity, as well as reflections from students identifying with genders outside of the gender binary. we find stm to be a useful approach for analyzing bias in our case. tracing where this bias could have originated in different training datasets with other tools @xcite and across other abstractive summarization models would help illuminate possible sources of this bias.