Probing Gender Bias in Multilingual LLM s: A Case Study of Stereotypes in P ersian (2025)

Abstract: Multilingual Large Language Models (LLMs) are increasingly used worldwide, making it essential to ensure they are free from gender bias to prevent representational harm. While prior studies have examined such biases in high-resource languages, low-resource languages remain understudied. In this paper, we propose a template-based probing methodology, validated against real-world data, to uncover gender stereotypes in LLMs. As part of this framework, we introduce the Domain-Specific Gender Skew Index (DS-GSI), a metric that quantifies deviations from gender parity. We evaluate four prominent models, GPT-4o mini, DeepSeek R1, Gemini 2.0 Flash, and Qwen QwQ 32B, across four semantic domains, focusing on Persian, a low-resource language with distinct linguistic features. Our results show that all models exhibit gender stereotypes, with greater disparities in Persian than in English across all domains. Among these, sports reflect the most rigid gender biases. This study underscores the need for inclusive NLP practices and provides a framework for assessing bias in other low-resource languages.

## introduction large language models (llms) have seen rapid adoption across languages and domains, from ev eryday use to complex industrial tasks. ensuring these technologies are fair and unbiased is essen tial. gender bias, in particular, can lead to harm ful stereotypes and representational harm @xcite . despite advancements in multilingual llms, most research focuses on highresource lan guages, especially english, leaving lowresource languages underexplored @xcite . persian is a lowresource language in the multi lingual llm landscape, largely due to the scarcity of structured, diverse training corpora. most avail able data come from unstructured sources like so cial media, and opensource resources and nlp tools for persian are limited. despite these chal lenges, persian offers a unique case for studying gender bias, given linguistic features such as the absence of gendered pronouns, which may affect how bias appears. however, there are currently no standardized benchmarks or tools for evaluating gender bias in llms for persian. to address this gap, we propose a novel template based probing method to uncover implicit gender biases in multilingual llms applied to persian. our approach targets four semantic domains, academic disciplines, professions, colors, and sports, cho sen to span a spectrum from professional identity to cultural concepts, where stereotypes are well documented in the sociological literature @xcite . we evaluate four prominent, publicly accessible multilingual llms, gpt4o mini, deepseek r1, gemini 2.0 flash, and qwen qwq 32b, developed by different organizations and representing a di verse range of architectures and training data @xcite . all four models are capable of handling persian, making them suitable for our evaluation. 1 this study investigates the following research questions: rq1: to what extent do prominent mul tilingual llms exhibit gender bias when prompted in persian across various semantic domains? rq2: are gender biases in llms more pronounced or expressed differently in persian (a lowresource lan guage) compared to a highresource language like english? our results show that llms reflect strong gender stereotypes across all four domains in persian. gen erated names for academic fields and professions display clear gender gaps, while associations with colors and sports mirror cultural gender roles. im portantly, these gender differences are much more pronounced in persian than in english. sports, in particular, stand out as the domain where traditional gender stereotypes are most strongly maintained. we also find that llms behave more consistently regarding gender bias in english than in persian. ## related work several prior studies have investigated the pres ence of gender bias in llms. for instance, thakur (2023) examined gender bias in gpt2 and gpt3.5 within the context of professions, finding that these models tend to generate male pronouns and names more frequently. similarly, @xcite in troduced a testing framework to evaluate gender bias and demonstrated that llms are more likely to associate occupations with the gender that aligns with societal stereotypes. additionally, @xcite developed an indirect probing approach to prompt llms to reveal potential gender bias. their findings indicate that llms can exhibit both ex plicit and implicit gender bias, even in the absence of gender stereotypes in the input. in another study involving high to mediumresource multilingual languages, @xcite designed a dataset to measure gender stereotypes and broader societal biases in llms. previous studies have employed various ap proaches to measure gender bias in llms. @xcite used different sentence processing methods, including masked tokens, unmasked sen tences, and sentence completion, to assess gen der bias in llms at the occupational level. they found that model outputs largely aligned with gen der distributions observed in u.s. labor force statis tics. similarly, @xcite applied persona based prompts to examine gender bias across a wide range of professions. their results revealed dis crepancies in gender representation, underscoring how architectural design, training data composition, and token embedding strategies influence bias in llms. additionally, soundararajan and delany (2024) generated gendered sentences using llms to assess bias at both the sentence and word levels, further confirming the presence of gender bias in these models. despite growing interest in multilingual llms, there has been limited research on how bias mani fests in languages with scarce highquality training data. @xcite introduced a multilin gual tool for bias assessment and explored whether lowresource languages are more prone to biases compared to highresource counterparts. their find ings revealed that biasdetection scores for low resource languages tend to vary more, especially in subtle categories like political views and racial atti tudes. similarly, @xcite lever aged chatgpt to translate texts from lowresource languages into english, aiming to evaluate implicit gender bias in relation to professions and actions. they observed gender bias in both aspects, with actions potentially exerting a stronger influence on gender inference in translated content. while ini tial studies like @xcite have included persian in broader multilingual gender bias bench marks, our work provides a deeper, more focused investigation. we use a templatebased probing method across four distinct semantic domains (aca demic disciplines, professions, colors, and sports) to reveal granular stereotypes that may not be cap tured by sentencecompletion tasks alone. ## methodology 3.1 prompting strategy to examine gender bias in llms, we use data from 66 academic fields (grouped under 10 major disciplines), as well as 10 professions, 10 colors, and 10 sports (see tables 1 and 2 for the full list). each prompt consists of two parts: an instruction defining the task and output format, and an input sentence describing a hypothetical person with the given domain, asking the model to suggest a name. for "academic discipline" and "profession," the model answers personal information questions; for "color" and "sport," it helps writers choose names for fictional characters. in all cases, the model must respond with an appropriate persian name without further explanation. example prompts (persian and english) are provided in tables 3 and 4 . some prompts, such as those beginning with "my friend is…," may sound like they refer to real indi viduals. this phrasing is intentional, reflecting the natural way people interact with language models. the ambiguity is a feature: it allows us to observe the assumptions and associations the model defaults to when gender and identity are unspecified. all prompts describe fictional scenarios and do not re fer to real people. for ground truth comparison, we also run the english translations of all prompts, asking for an appropriate english name. prompts are intention ally underspecified to force models to rely on their internal associations rather than factual knowledge. our goal is descriptive: to map these biases, not to assess the models' factual accuracy. we use 96 unique prompts (66 academic disci plines + 10 professions + 10 colors + 10 sports), each run 100 times per domain value. each model generates 9,600 names for the english prompts; for persian prompts, the total generations per model are: gpt4o mini: 9,557; deepseek r1: 9,598; gemini 2.0 flash: 9,561; qwen qwq 32b: 9,407. if a model fails to produce a valid name, by omit ting a name or generating a nonhuman one, we retry up to two additional times. last names are removed, as the focus is on gender identification. below is an english translation of one sample input sentence: color: "i'm writing a story about a char acter who likes the color green. suggest a name for the character." ## conclusion this study explores gender bias in multilingual llms when prompted in persian, a lowresource language. using a templatebased method, we iden tify implicit biases in four popular llms across aca demic disciplines, professions, colors, and sports. all models exhibit stereotypical gender associa tions, with disparities consistently greater in per sian than english. bias scores also show more consistency across models in english, while vari ability is higher in persian. academic discipline and profession domains reflect systematic gender imbalances, linking male and femaledominated roles to corresponding genders. the color and sport domains reveal culturally influenced stereotypes, with sports showing the strongest binary patterns. among the models, gemini 2.0 flash demonstrates the most pronounced biases, while gpt4o mini and qwen qwq 32b offer more balanced outputs. these results highlight how llms may reproduce or amplify gendered assumptions, especially in low resource settings.