Mind the Gap: Gender-based Differences in Occupational Embeddings (2025)

Abstract: Large Language Models (LLMs) offer promising alternatives to traditional occupational coding approaches in survey research. Using a German dataset, we examine the extent to which LLM-based occupational coding differs by gender. Our findings reveal systematic disparities: gendered job titles (e.g., “Autor” vs. “Autorin”, meaning “male author” vs. “female author”) frequently result in diverging occupation codes, even when semantically identical. Across all models, 54%–82% of gendered inputs obtain different Top-5 suggestions. The practical impact, however, depends on the model. GPT includes the correct code most often (62%) but demonstrates female bias (up to +18 pp). IBM is less accurate (51%) but largely balanced. Alibaba, Gemini, and MiniLM achieve about 50% correct-code inclusion, and their small (< 10 pp) and direction-flipping gaps could indicate a sampling noise rather than gender bias. We discuss these findings in the context of fairness and reproducibility in NLP applications for social data.

## introduction occupational coding-the task of assigning standardized occupational categories to free-text job descriptions-is a cornerstone of labor market statistics, informing policy in areas such as employment, migration, and public health. this task is inherently challenging: individuals often describe their work in ambiguous or incomplete terms, and coders must map these descriptions to one of hundreds (or 1,300 in germany) of possible categories. historically a manual process, occupational coding has evolved with the rise of automatic solutions. more recently, large language models (llms) have been proposed as tools to further automate and advance this process by leveraging their semantic capabilities to match job titles with occupational codes. this paper examines gender disparities in the coding suggestions made by llm-based occupational coding. using german survey and the official german classification of occupations (kldb 2010), we analyze how often male and female forms of job titles receive divergent codes (see figure 1 for the research pipeline). these differences are not only prevalent but occasionally substantial-pointing to potential downstream harms in labor statistics and policy. ## background occupational coding-the classification of freetext job titles into standardized categories-has long been recognized as susceptible to gender bias. in manual coding systems, biases can arise from historical taxonomy and human judgment. for example, earlier german occupation classifica-tions documented that the occupational activities of men are covered more accurately than those of women, leading to misinterpretations in labor statistics @xcite . human coders might also inadvertently rely on gendered cues or stereotypes when interpreting ambiguous job titles, though systematic evidence is limited @xcite . with the shift toward automated coding, researchers have found that algorithms often perpetuate or even amplify existing gender biases. a large-scale study of english online biographies demonstrated significant bias in occupation classification: including gender indicators (like names or pronouns) skewed predictions and yielded different true positive rates for women vs. men in genderimbalanced field @xcite . even after removing explicit gender tokens, subtle proxies in text led to residual bias favoring the majority gender in a profession. advanced large language models (llms) also reflect societal stereotypes: recent evaluations found llms three to six times more likely to assign a person an occupation stereotypical for their gender, often beyond actual labor force proportions @xcite @xcite . however, most bias studies focus on english and binary gender contexts @xcite @xcite , with less work on languages like german that feature gendered job titles. this highlights the need for further research on robust, bias-resistant coding methods and evaluation in diverse settings. ## data our empirical analysis draws on two data sources: the german classification of occupations and survey data. the primary reference taxonomy is the german classification of occupations 2010 (klassifikation der berufe 2010, kldb 2010; bundesagentur für arbeit, 2019), which defines 1,286 standardized occupational categories. each includes a description (e.g., authors and writers producing complex creative texts requiring advanced skills), typical tasks (e.g., creating and writing literary, technical, and factual text), and example job titles (e.g., authors, screenwriters) (see a key linguistic feature of german is the use of grammatical gender in occupational titles, typically marked by a masculine base form (e.g., lehrer) and a feminine suffix (e.g., lehrerin). traditionally, the masculine form has served as a generisches maskulinum (generic masculine) meant to include all genders. for instance, lehrer may refer to any group of teachers. however, research shows that such forms are not interpreted as truly neutral and often lead to male-biased mental representations @xcite . this study examines whether embedding-based occupational coding systems reflect or mitigate the semantic and social distinctions introduced by gendered job titles. to assess this, we identified 276 jobs in the dataset that differ only by grammatical gender (e.g., autor vs. autorin, ingenieur vs. ingenieurin; see ## methodology to assess the role of gender in embedding-based occupational coding, we evaluated five multilingual models on a set of gendered job title pairs. given a single gendered job title (masculine or feminine), the system retrieves the five kldb-2010 occupation codes whose reference embeddings are most similar to that title. this ranked list of five codes is our classification outcome. we evaluate it with (i) gender-based divergence-whether the male and female forms of the same title receive different top-5 suggestions-and (ii) top-5 accuracy-whether the gold code appears in the top-5 suggestions. embedding models are increasingly used in automatic text classification tasks with large label spaces, such as the categorization of industries @xcite , diseases @xcite , or international trade @xcite . they provide a scalable way to retrieve a small set of relevant categories based on linguistic similarity prior to classification. in occupational coding, embeddings help to narrow the large number of fine-grained job categories by aligning free-text job descriptions with predefined classification labels @xcite @xcite . we relied on the following models: minilm-l12-v2 (multilingual) @xcite , alibaba-nlp gte-multilingual-base @xcite ), gemini's text-embedding-004 @xcite , gpt text-embedding-3-large (openai, 2024), and ibm's granite-embedding-278mmultilingual (ibm-research, 2024). our evaluation set originates from a cati survey in which respondents named their occupation. we selected only those answers that met two criteria: (i) the job title is explicitly gendered in german (e.g., lehrer 'male teacher', autorin 'female author'), and (ii) both the masculine and feminine form appeared in the sample and were professionally coded. titles that were gender-neutral (e.g., babysitter) or represented in only one grammatical gender (e.g., soldat 'male soldier') were discarded. this selection resulted in 276 gendered responses, covering 39 distinct job title pairs (lehrer @xmath0; lehrerin -30, autor -1 and autorin -1 (see we apply the two indicators defined abovegender-based divergence and top-5 accuracyto every pair of masculine-feminine inputs. this setup allows us to test whether embeddings treat male and female occupational titles as semantically equivalent. ideally, gendered inputs for the same occupation should result in identical suggestions and be classified with equal accuracy. ## results across all five embedding models, we observed systematic gender differences in top-5 suggestions for otherwise identical job titles. the results reveal that current embedding approaches do not treat masculine and feminine occupational forms as semantically equivalent, despite their referential equivalence in context. all five models exhibited gender-based divergence ( the overall rate of gender-based divergence ranged from 54% to 82%, depending on the embedding model and the occupational information component from the reference dataset that was used for embeddings. for example, in one case, the term autor (male form of "author") was matched to occupations such as komponist (composer) and verlagskaufmann (publishing manager), while autorin (female form) yielded lektorin (editor) and notarin (notary) among its top-5 suggestions. while both forms shared a common first suggestion, three out of five recommendations differed, including assignments to distinct occupational major groups in kldb 2010. in addition to the divergence ratio, we report top-5 accuracy by gender-the proportion of masculine or feminine input titles whose gold kldb code appears among the five retrieved suggestions. moreover adding more textual fields to the reference set (e.g. @xmath2. this suggests that lexical surface forms exert an influence on embedding similarity, even when more semantic context is introduced. taken together, these findings show that gender-specific lexical variation in german occupational titles systematically affects llm-based embedding outputs. this can have downstream consequences for fairness in automated occupational classification systems, and by extension, any research or policy relying on them. ## discussion our analysis demonstrates that embedding-based occupational coding behaves differently on gendered occupational titles in german. across five state-of-the-art multilingual models and seven reference-set configurations, up to 82% of gendered pairs received divergent top-5 suggestions. these differences involved distinct occupational codes that sometimes crossed major kldb groups. such disparities highlight a critical limitation: current llm-based coding approaches fail to generalize over morphological gender, treating formally different yet semantically identical titles as distinct occupations. this means that (if placed in the survey) two respondents who perform identical work but report it with different grammatical gender therefore would face different shortlists of suggested codes, raising an obvious fairness concern. how harmful is the mismatch? that depends on whether the correct code still makes it into the list. gpt, for example, supplies the gold code for both forms in about 62% of cases on average and does so slightly more often for feminine titles (up to +18 pp). ibm has accuracy around 51% but it is almost balanced. for alibaba, gemini, and minilm the chance of seeing the gold code hovers around 50%. coupled with @xmath3. in short, divergence is pervasive, but its practical impact varies by model. the stakes are high. in germany, 1,300 standardized job categories inform labor and health statistics, and policy. even minor classification differences can skew research on employment, wages, health and gender inequality. german adds complexity by making grammatical gender overt-most job titles appear in masculine and feminine forms (e.g., anästhesist vs. anästhesistin), with the generic masculine long dominant in records. while subtle linguistically, these differences are treated as semantically distinct by lan-guage models, despite their functional equivalence. to address this, future evaluation protocols incorporate controlled tests for gender consistency, particularly in morphologically rich languages. survey infrastructures and coding systems should promote or accommodate gender-neutral occupational inputs, such as role-based terms (lehrkraft) or inclusive forms (lehrer*in), while also preparing models to interpret them reliably. embedding models used in survey contexts may benefit from finetuning or contrastive alignment that enforces gender symmetry in professional roles. ## conclusion our findings show consistent significant disparities: gendered job titles-such as autor vs. autorin -often lead to different occupation codes, despite having identical meanings. our findings underscore the importance of grounding nlp innovations in language-specific sociolinguistic knowledge. without rigorous attention to linguistic structure and social context, these tools risk perpetuating systemic biases-particularly in settings where semantic equivalence is masked by morphological variation. addressing such challenges is crucial not only for the technical refinement of nlp systems, but for ensuring that their real-world applications advance rather than hinder equity.