Measuring Gender Bias in Language Models in F arsi (2025)

Abstract: As Natural Language Processing models become increasingly embedded in everyday life, ensuring that these systems can measure and mitigate bias is critical. While substantial work has been done to identify and mitigate gender bias in English, Farsi remains largely underexplored. This paper presents the first comprehensive study of gender bias in language models in Farsi across three tasks: emotion analysis, question answering, and hurtful sentence completion. We assess a range of language models across all the tasks in zero-shot settings. By adapting established evaluation frameworks for Farsi, we uncover patterns of gender bias that differ from those observed in English, highlighting the urgent need for culturally and linguistically inclusive approaches to bias mitigation in NLP.

## introduction gender bias influences how people of different genders are represented and understood in everyday communication @xcite . it appears in various areas, including the workplace, education, and media @xcite @xcite . in language, this bias appears through unequal treatment, stereotypes, and imbalanced representation of genders @xcite . since language shapes how we think and communicate, these biases also appear in languagebased technologies. natural language processing (nlp) tools are now widely used in daily life, especially since the release of openai's chatgpt @xcite . this makes it important to understand how gender bias manifests in these systems. many researchers have studied this issue using various models and methods; however, the majority of these studies focus on english and western cultures @xcite @xcite input output event: yelled at my parents for not buying me a car like other parents. gender: man / woman predicted emotion: @xmath0: couldn't start the board meeting at 9am because a man and a woman were late. waited for the woman (meeting lead), man (note taker). condition: disambiguated question: who was the secretary? answer: woman sentence: <@xmath1. identity: men / women filled words (@xmath2: @xmath3, man, genius, murderer, devil @xmath4, whore, devil, guilty, criminal we adapt and apply established englishlanguage frameworks to farsi: emotional bias detection @xcite , bbq @xcite , and honest @xcite . our results reveal patterns that diverge from those observed in english, emphasizing the importance of language-and culture-specific evaluations. our contributions are: 1) we present the first systematic study of gender bias in farsi across three distinct tasks. 2) we propose a unified process for translating gender bias resources in farsi. foot_0 3) we provide a detailed cross-task analysis that reveals unique, language-specific bias patterns. ## bias statement in this paper, we systematically investigate gender bias in language models in farsi across emotion analysis, question answering, and hurtful sentence completion tasks. our work is motivated by the recognition that language technologies, when trained on data reflecting societal stereotypes and inequalities, can perpetuate and amplify harmful biases. specifically, we focus on representational harms, where models may reinforce or propagate stereotypical associations between gender and emotions, abilities, or social roles. we define gender bias as the systematic linking of emotions, abilities, or harmful traits to one gender over another, as well as the disproportionate generation of toxic content targeting men or women. our study is constrained by a binary view of gender, which we acknowledge as a representational harm in itself. we also note that adapting english-centric frameworks and using machine translation may introduce additional biases. despite these limitations, we advocate for nlp systems that treat all users fairly and transparently, and we present this work as a step toward more inclusive and responsible bias research in underrepresented languages like farsi. ## bias in farsi emotion analysis following previous work conducted in english @xcite , we tested gender bias in farsi through emotion analysis. the task is to investigate whether llms exhibit gendered emotion attribution when prompted with farsi text and gendered personas. we prompted the models to adopt a gendered persona (e.g., a "woman" or a "man") and then asked them to identify the main emotion that persona would feel when experiencing a specific event described in farsi (e.g. "when i had an accident with damage to the car body."). by analyzing the patterns of emotions generated for male and female personas across various events, we investigated whether these models exhibited gendered stereotypes in their emotion attributions within a farsi linguistic context. this enables us to examine the presence and nature of gendered emotional stereotypes in farsi, as reflected in llms. ## bias in farsi qa following prior work by @xcite conducted in english, we investigate gender bias in farsi via question answering. our goal is to assess whether llms exhibit gender bias in their responses when presented with farsi qa prompts. specifically, we examine model behavior across two types of contexts: those that are under-informative and those that provide clear, distinguishing details. we refer to the former as ambiguous contexts, where there is insufficient information to draw a conclusion, and to the latter as disambiguated contexts, where relevant details allow for a definitive, correct answer. for example, consider the following ambiguous context: "after the first day of middle school math, a boy and a girl stayed after class to talk to the teacher." when asked "who is good at math?" with the answer choices "the boy", "the girl", and "not enough information", the correct answer is clearly "not enough information", as the context does not indicate any difference in ability. in contrast, a disambiguated version of this context might read: "after the first day of middle school math, a boy and a girl stayed after class to talk to the teacher. the girl thought the class was too easy and asked to be moved up to advanced math, while the boy was scared of failing because math is too hard." in this case, there is sufficient information for the model to answer correctly. ## bias in farsi hurtful completions we also extended to farsi the multilingual hon-est evaluation framework @xcite , which systematically assesses hurtful sentence completions in encoder-based language models. ## conclusions this paper introduced a comprehensive evaluation of gender bias in language models in farsi by reproducing three established frameworks focused on emotion analysis (plaza-del-arco et al., 2024), question answering @xcite , and hurtful completions @xcite . through this multi-task approach, we demonstrated that gender bias is not limited to high-resource languages like english but also affects less-resourced languages such as farsi, often manifesting in more subtle and culturally specific ways. our results show that gender stereotypes are consistently present in model outputs, with their expression influenced by task type, prompt design, and model architecture. importantly, even the most recent models continue to exhibit biased behavior, suggesting that improvements in general performance do not automatically lead to greater fairness. moreover, while some bias patterns appear across languages, such as the association of anger with men, others are modulated by farsi's linguistic and cultural context. overall, this study highlights the need for fairness evaluations beyond english and calls for more inclusive approaches in the development of large language models. addressing these biases is essential to ensure that nlp systems serve diverse linguistic communities equitably and responsibly.