Does BERT Exacerbate Gender or L 1 Biases in Automated E nglish Speaking Assessment? (2023)

Abstract: In English speaking assessment, pretrained large language models (LLMs) such as BERT can score constructed response items as accurately as human raters. Less research has investigated whether LLMs perpetuate or exacerbate biases, which would pose problems for the fairness and validity of the test. This study examines gender and native language (L1) biases in human and automated scores, using an off-the-shelf (OOS) BERT model. Analyses focus on a specific type of bias known as differential item functioning (DIF), which compares examinees of similar English language proficiency. Results show that there is a moderate amount of DIF, based on examinees’ L1 background in grade band 912. DIF is higher when scored by an OOS BERT model, indicating that BERT may exacerbate this bias; however, in practical terms, the degree to which BERT exacerbates DIF is very small. Additionally, there is more DIF for longer speaking items and for older examinees, but BERT does not exacerbate these patterns of DIF.

## introduction pretrained large language models (llms) present new opportunities for english speaking assessments, yet they are prone to perpetuating and, in some cases, exacerbating social prejudices @xcite . in educational assessment, researchers have shown that pretrained llms can replicate human scoring, including english speaking assessment, with a high degree of accuracy @xcite . studies of biases of these automated scoring systems, however, is uncommon @xcite . considering how widespread and high stakes english speaking assessments are at both the primary and secondary education levels @xcite , it is imperative that these assessments be fair for all students, regardless of gender or l1 backgrounds. this study addresses the need for deeper analyses of bias in llm-based automated english speaking assessments.