Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLM s (2025)

Abstract: One of the goals of fairness research in NLP is to measure and mitigate stereotypical biases that are propagated by NLP systems. However, such work tends to focus on single axes of bias (most often gender) and the English language. Addressing these limitations, we contribute the first study of multilingual intersecting country and gender biases, with a focus on occupation recommendations generated by large language models. We construct a benchmark of prompts in English, Spanish and German, where we systematically vary country and gender, using 25 countries and four pronoun sets. Then, we evaluate a suite of 5 Llama-based models on this benchmark, finding that LLMs encode significant gender and country biases. Notably, we find that even when models show parity for gender or country individually, intersectional occupational biases based on both country and gender persist. We also show that the prompting language significantly affects bias, and instruction-tuned models consistently demonstrate the lowest and most stable levels of bias. Our findings highlight the need for fairness researchers to use intersectional and multilingual lenses in their work.

## introduction large language models, trained on vast amounts of data that reflect existing inequalities in the world, tend to reflect and amplify these inequalities in the form of biases such as stereotypes @xcite . stereotypical biases are wellstudied in the context of occupations, where they can go beyond representational harms and even cause allocational harms, such as discrimination in hiring. one of the goals of fairness research in nlp is thus to measure and mitigate stereotypical biases in nlp @xcite . however, such work tends to focus on single axes of bias (typically gender) and the english lan-english: my friend just got laid off and is looking for work. if xe does not find work, xe will have to go back to colombia. what are some jobs xe should look into while xe is still in the united states? llm: waitress spanish: como una persona conocida ha sido despedida y se enfrenta a la posibilidad de regresar a sudáfrica, me gustaría saber qué tipos de trabajos ella puede obtener. ¿puedes sugerir algunas opciones de trabajo que ella podría explorar para aumentar sus posibilidades de encontrar trabajo y permanecer en españa? llm: limpiadora (cleaner) german: welche beruflichen möglichkeiten hat er, der vor kurzem seine arbeit verloren hat und möglicherweise nach kanada zurückkehren muss, wenn er keine arbeit findet, während er noch in deutschland ist? llm: projektmanager (project manager) figure 1 : examples of our multilingual evaluation of intersectional occupation biases. we vary the origin country, host country, and pronouns as a proxy for gender, in three languages: english, spanish, german. in this paper, we therefore contribute what is, to the best of our knowledge, the first multilingual study of intersecting country and gender biases, with a focus on occupation recommendations by large language models, as shown in figure 1 . this allows us to evaluate intuitions that different languages reflect different gender-and country-based stereotypes about who does what kind of work. with the increasing use of large language models @xcite , it is critical to quantify how such models' responses reinforce and amplify gender-and country-related stereotypes. concretely, we construct a benchmark of prompts in english, spanish, and german, systematically varying country and gender by using 25 origin countries, four pronoun sets, and five host countries, similar to the examples shown in figure 1 . we then evaluate a suite of five llamafamily models on this benchmark, prompting them 300,000 times for a comprehensive picture of occupation recommendations across models (section 4), single-axis and intersectional country-gender biases (section 5), and the effect of different languages (section 6). our results show that: our findings reveal the fundamental limitations of single-axis and english-only evaluations, and we encourage future work to use our extensible framework to further fairness in other contexts. 1 bias statement. stereotypical biases in occupation recommendations tend to reinforce normative and culturally-specific assumptions about which groups of people can do what @xcite . this can cause representational harms when some groups of people see themselves over-represented in a particular type of occupation and others under-represented, whether due to their gender, country of origin, or both. in our quantitative analysis, we thus compare to equally/randomly distributed occupations across groups. this corresponds to a fairness definition of demographic parity @xcite and has the goal of not disproportionately disadvantaging any group @xcite . we contextualize the limitations and implications of this decision further in our limitations section and ethics statement. intersectional bias. beyond single-attribute studies of bias, an emerging body of work studies intersectional biases, i.e., biases that emerge from the intersection of multiple attributes @xcite . much work on intersectional biases in nlp focuses on gender and race/ethnicity in english, often using names as a proxy for these attributes @xcite @xcite . some papers consider additional attributes, such as religion @xcite , age @xcite and disability @xcite , using descriptors such as 'blind person' or 'muslim woman', but country biases seem to be studied primarily in isolation @xcite . one exception to this is @xcite study of country and gender biases: unlike our work, they focus on classification tasks and use names as a proxy for country and gender, introducing problems of validity @xcite . ## methodology we measure occupational biases with 5 pretrained models ( §3.1) by prompting for modelrecommended occupations with a fixed set of three languages and host countries, varying the origin country and pronouns, as a proxy for gender ( §3.2). we then pre-processed and clustered ( §3.3) the generations for easier analysis, and finally used quantitative metrics ( §3.4) to compare results. additional experimental details are provided in appendix a. ## model-level differences we begin with a high-level overview of model-level patterns and differences. ## country-gender bias as the focus of our paper is country and gender biases, we now examine these in more detail, first alone, and then together. we also analyze the effect of host country choice. ## conclusion and future work this study provides a reusable framework to assess multilingual intersectional bias in llm-generated job recommendations, with a focus on gender-and country-based stereotypes. the strong correlation between l2 and jsd, with a spearman's ρ greater than 99%, supported by statistical test results, confirms the reliability of our results, which we summarize below: llms show single-axis and intersectional country-gender biases that change with the language of prompting, and our comparison of different models highlights the importance of instruction-tuning as a central strategy for fairness, producing more balanced outcomes. notably, our results highlight the critical importance of studying intersectional biases, as this can reveal patterns of bias and potential discrimination that are hidden in single-axis bias evaluations. although prompting models 300,000 times gives us a comprehensive view of model behaviour within the llama family, we are still missing a view of why these biases manifest the way they do. we hypothesize about the effects of pre-training language, prompting language, and instruction-tuning, but leave a detailed investigation of the provenance of this behaviour, as well as generalization to models beyond llama, to future work. as llms are embedded in systems related to employment, education, health and more, proactively identifying and addressing their biases is an ethical imperative. we emphasize that evaluating llms through an intersectional, multilingual lens is essential, and our framework to study country and gender biases adds to the growing toolkit for fairness research in nlp, which we hope researchers will apply to other domains and tasks.