F ilipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from S outheast A sia (2025)

Abstract: Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets—a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models.

## introduction despite the rapid evolution of plms and efforts to minimize their social harms @xcite , recent studies still confirm the presence of biases within them @xcite @xcite . ai fairness, therefore, remains to be a critical area of focus for the research community, which bears an ethical responsibility to mitigate the potential negative impacts of the technologies it builds @xcite @xcite . scholars have developed bias evaluation benchmarks to not only establish baselines quantifying biased behavior exhibited by off-the-shelf plms, but also to measure the effectiveness of bias mitigation techniques applied on these models @xcite @xcite . most bias studies in the literature, however, use only english benchmarks to assess monolingual plms @xcite . only a few recent exceptions have emerged to examine bias in multilingual plms using datasets written in other languages-i.e., french @xcite , german @xcite , dutch @xcite . among these multilingual studies of bias, the benchmarks used often treat gender as a binary construct and do not thoroughly investigate biases against non-heterosexual identities @xcite . there is thus an absence of non-english homophobic bias evaluation benchmarks that can catalyze work in evaluating and mitigating anti-queer bias in plms deployed in non-english-speaking contexts. in this paper, we address this gap by adapting two bias benchmarks-crowdsourced stereotype pairs or crows-pairs @xcite and winoqueer @xcite -for filipino, a language that currently does not have high nlp resources @xcite . crows-pairs is a dataset widely used to probe plms for different stereotypes (e.g., race, gender, religion, age, etc.), while winoqueer is a recently released benchmark designed to assess the extent of anti-lgbtq+ bias encoded in plms. designing filipino versions of these english materials is valuable for two reasons. first, the english and filipino languages do not share the same linguistic and grammatical gender mechanisms @xcite @xcite , nor do concepts of queerness and non-heterosexuality in their corresponding cultures completely overlap @xcite . our method for culturally adapting crows-pairs and winoqueer into filipino elucidates how gen-eralizable these benchmarks are to low-resource languages and what considerations and challenges need to be accounted for in translating them. our corpus development procedure can serve as a template or guide for future endeavors creating bias benchmarks in other languages. second, the integration of ai into the southeast asian landscape is growing. reports highlight both the rapid uptake of language-based ai technologies in the area @xcite and local nlp practitioners' deployment of plms trained with higher proportions of southeast asian textual data @xcite @xcite . designing contextually appropriate bias benchmarks in southeast asian languages-especially filipino, which has 83 million speakers @xcite -stands as a crucial first step in mitigating the societal harms of such plms used in the region. we demonstrate our filipino benchmarks' ability to contribute to this regard by evaluating both sexist and homophonic bias in off-the shelf multilingual plms, including causal ones specifically developed for the southeast asian context. to the best of our knowledge, we are the first to use non-english benchmarks in assessing causal and southeast asian models. our work can thus serve as a baseline for future work aiming to reduce bias in such models. our contributions are threefold: ## conclusion in this paper, we outlined our process for culturally adapting existing bias evaluation benchmarks into filipino, a low-resource language from southeast asia. the process revealed challenges in extending gender-and sexuality-related english datasets into another culture, namely differences in linguistic gender systems, concepts of queerness, and cultural practices and ideologies. our solutions to these challenges helped design filipino crows-pairs and filipino winoqueer-the latter of which is the first non-english benchmark specifically designed to assess homophobic bias. we then used these benchmarks to establish baseline bias evaluation results for multilingual plms, including those from southeast asia. these results show that the models behave with bias. this behavior can be linked to the models' exposure to more filipino data in pretraining and the english etymological origins of some filipino non-heterosexual labels (i.e., tomboy and lesbiyana). we hope that these insights can guide future work investigating how multilingual plms learn and reproduce bias across different languages. we also hope that our filipino benchmarks and bias evaluation results can accelerate work on both multilingual bias evaluation in other languages and debiasing of multilingual plms to make them less harmful towards marginalized gender and sexuality groups across the globe.