Assessing Gender Bias of Pretrained B angla Language Models in STEM and SHAPE Fields (2025)

Abstract: Gender bias continues to shape societal perceptions across both STEM (Science, Technology, Engineering, and Mathematics) and SHAPE (Social Sciences, Humanities, and the Arts for People and the Economy) domains. While existing studies have explored such biases in English language models, similar analyses in Bangla—spoken by over 240 million people—remain scarce. In this work, we investigate gender-profession associations in Bangla language models. We introduce Pokkhopat , a curated dataset of gendered terms and profession-related words across STEM and SHAPE disciplines. Using a suite of embedding-based bias detection methods—including WEAT, ECT, RND, RIPA, and cosine similarity visualizations—we evaluate 11 Bangla language models. Our findings show that several widely-used open-source Bangla NLP models (e.g., sagorsarker/bangla-bert-base) exhibit significant gender bias, underscoring the need for more inclusive and bias-aware development in low-resource languages like Bangla. We also find that many STEM and SHAPE-related words are absent from these models’ vocabularies, complicating bias detection and possibly amplifying existing biases. This emphasizes the importance of incorporating more diverse and comprehensive training data to mitigate such biases moving forward. Code available at https://github.com/HerWILL-Inc/ACL-2025/ .

## introduction textual representations play a powerful role in reinforcing gender biases, particularly in how professional roles are described and associated with gender. in both stem and shape (social sciences, humanities, and the arts for people and the economy) @xcite domains, written content often reflects implicit assumptions-depicting roles like "receptionist" as female-coded and "scientist" as male-coded @xcite . such patterns are not merely descriptive but normative; they help entrench gendered expectations about who belongs in which fields. these biases contribute to the marginalization of shape disciplines and those who pursue them, often women, by diminishing their public visibility and perceived value. over time, consistent exposure to gendered language in text influences how individuals internalize societal roles and professional aspirations @xcite . recognizing and addressing gender bias in text is therefore essential to creating better representations across disciplines. studies exploring the biases of english language models @xcite do not explicitly assess gender bias in shape fields. for example, therapist, educationalist, economist, lobbyist, archaeologist, journalist, actor, dancer, cartoonist etc. are absent in the stereoset dataset used in the study by @xcite . although gender bias detection of nlp systems is a well-studied task for the english language @xcite , it remains largely unexplored for the low-resource language, bangla (often referred as 'bengali'). while several open-weight bangla language models are available in public repositories, their gender bias remains largely unexplored. this leaves room for these models to be deployed while leaving the risk of exhibiting gender bias in real-world scenarios; where over 131 million bangla-speaking internet users (dhaka tribune, 2023) can experience and be influenced by gender bias. in bangla, there is no grammatical gender; instead, the gender system relies on semantics @xcite . gender distinctions in bangla are indicated by specific lexical choices reflecting the gender of the entity. for example, in terms of grammatical gender in bangla language, "es" (se) is a pronoun and it can refer to both "she" or "he". "so úao n" (sont ""an) signifies "child" and can represent either a son or a daughter. in terms of semantic gender, bangla has separate words for both the genders. for example: "puru p" (pu:ru:s) refers to "man" and "mo o ihla" (mo-hi-la) means "woman", "o iso xo k" (sik h ok) specifies "male teacher" and "o iso o ixo ka" (sik h ika) translates to "female teacher". although the "o iso xo k" and "o iso o ixo ka" nouns are lexically similar, their usage does not affect sentence structure, verb conjugation, and adjective agreement; resulting in the absence of grammatical gender in bangla. due to the absence of grammatical gender in bangla, it is difficult to analyze implicit gender bias in stem and shape fields. to address the critical gap in evaluating gender bias in bangla nlp, our primary contribution lies in a comprehensive empirical assessment of 11 pretrained bangla language models (pblms) using five established bias evaluation metrics. despite the growing use of these models, their implicit gender associations-particularly in relation to stem and shape domains-remain largely unexplored. given the absence of gendered pronouns in bangla, we analyze word embeddings to uncover latent biases, hypothesizing that unbiased models would exhibit similar distances between stem/shape terms and male/female word embeddings. our findings reveal that several popular models display measurable gender bias, with stronger biases observed in shape-related vocabulary. to assist in this evaluation, we introduce pokkhopat, a curated dataset comprising gender-categorized bangla words across stem and shape fields. this resource is designed to support future bias evaluations, especially by enriching the currently underrepresented shape domain in bangla gender bias research, as illustrated in figure 1 . these findings have significant implications for the responsible deployment of bangla nlp systems, particularly in educational, hiring, and content generation tools where gender neutrality is critical. by uncovering these hidden biases, our work not only sets a precedent for fairness audits in lowresource languages but also encourages the development of more equitable and inclusive language technologies for diverse linguistic communities. ## bias statement this paper examines how gender bias relates to stem and shape professions by analyzing word clustering in bangla language embeddings. although bangla lacks grammatical gender (mukherjee, 2018), biases in language embeddings may reflect stereotypes about stem and shape fields. some biases are harmless, but others can be damaging. biased language models can unfairly reinforce gender roles @xcite . for example, if bangla embeddings cluster engineering-related words with male-associated words, it suggests a bias linking stem with males. conversely, if words related to psychotherapy cluster with femaleassociated words, it may reflect the stereotype that women are more suited for shape roles @xcite . such biases can limit diversity in education and the workplace @xcite . ideally, bangla embeddings should avoid reinforcing gender stereotypes in stem and shape fields. the dataset used here represents only two genders: male and female, which may harm those identifying outside this binary @xcite , particularly in bangladesh, where 12,629 identify as "third gender" @xcite . we present this study to encourage future research that is more inclusive of diverse gender identities. ## methodology our overall methodology is outlined in figure 2 . we assessed the gender bias of 11 pblms using the pokkhopat dataset and 5 bias evaluation metrics to obtain a clear picture of how biased pblms are. ## discussion we evaluated gender biases in pblms using multiple metrics and a diverse dataset. results revealed both stereotypical biases (males associated with stem, females with shape) and counterstereotypical biases (males associated with shape, females with stem). ripa and ect metrics detected biases that weat missed, providing a comprehensive view of gender bias in pblms regarding professions. our observed phenomenon of oov words affecting gender bias detection is a crucial insight. when attribute words, like those related to stem and shape, are missing from a model's pretraining data, their embeddings are either underrepresented or significantly different from those seen in the training corpus. this discrepancy is evident in our analysis, where shape-related words absent in the bangla2b+ corpus are embedded distinctly from those present in the vocabulary. the embedding gap between oov and in-vocabulary shape words suggests that models trained on incomplete corpora may fail to capture nuanced relationships between gender and profession categories, leading to a distorted or incomplete bias evaluation. this further complicates the identification of gender bias, as models may exhibit little to no bias for the words they are familiar with, despite biases potentially existing in the oov terms. the tokenizers used in our study (electratokenizerfast, t5tokenizerfast, gpt2tokenizerfast, etc.) have mechanisms to handle oov tokens through subword tokenization. yet the morphological richness and lexical complexity of bangla results in fragmented representations which affect bias detection. many of the professional terms in pokkhopat are transliterated. although common transliterated terms such as engineer, doctor, computer etc. are successfully tokenized by bangla tokenizers, less common terms such as pharmacist, forensic, physiologist etc. are over-fragmented. the survey by stanczak and augenstein (2021) presents evidence from various studies that lack of completeness in lexica and datasets limit the scope of bias analysis, particularly in occupational domains where gender stereotypes are prevalent, thus undermining the effectiveness of gender bias detection methods in nlp. one way to address this is to analyze only in-vocabulary terms; however, the vocabulary varies across the 11 pblms, making comparisons unfair. while alternative tokenization strategies could be explored, we used each model's default tokenizer to reflect typical usage patterns. a more comprehensive solution would be to fine-tune models on a corpus that includes the full pokkhopat vocabulary-an effort that would require developing a high-quality, context-rich bangla text corpus which by itself is an avenue for future research. consequently, our finding highlight the importance of comprehensive, diverse training data in the development of more fair and reliable language models, especially in underrepresented languages like bangla. ## conclusion in this paper, we attempted to analyze the gender bias of bangla language models with regards to stem and shape. to the best of our knowledge, no other previous work tackles this specific issue. statistically significant gender bias in bangla language models were not detected in many cases in our study, most likely due to the lack of diversity in the bangla corpuses. we expect that in the future, bangla corpuses will contain a larger number of words, including the ones that appear in the pokkhopat dataset, so that a better evaluation of the gender bias of bangla language models can be made. employing the pokkhopat dataset's english word lists to quantify gender bias of popular english large language models can be an interesting avenue of future research.