J ob F air: A Framework for Benchmarking Gender Hiring Bias in Large Language Models (2024)

Abstract: The use of Large Language Models (LLMs) in hiring has led to legislative actions to protect vulnerable demographic groups. This paper presents a novel framework for benchmarking hierarchical gender hiring bias in Large Language Models (LLMs) for resume scoring, revealing significant issues of reverse gender hiring bias and overdebiasing. Our contributions are fourfold: Firstly, we introduce a new construct grounded in labour economics, legal principles, and critiques of current bias benchmarks: hiring bias can be categorized into two types: Level bias (difference in the average outcomes between demographic counterfactual groups) and Spread bias (difference in the variance of outcomes between demographic counterfactual groups); Level bias can be further subdivided into statistical bias (i.e. changing with non-demographic content) and taste-based bias (i.e. consistent regardless of non-demographic content). Secondly, the framework includes rigorous statistical and computational hiring bias metrics, such as Rank After Scoring (RAS), Rank-based Impact Ratio, Permutation Test, and Fixed Effects Model. Thirdly, we analyze gender hiring biases in ten state-of-the-art LLMs. Seven out of ten LLMs show significant biases against males in at least one industry. An industry-effect regression reveals that the healthcare industry is the most biased against males. Moreover, we found that the bias performance remains invariant with resume content for eight out of ten LLMs. This indicates that the bias performance measured in this paper might apply to other resume datasets with different resume qualities. Fourthly, we provide a user-friendly demo and resume dataset to support the adoption and practical use of the framework, which can be generalized to other social traits and tasks.

## introduction large language models (llms), by their extensive training on large datasets, are particularly susceptible to learning biases present in the data @xcite . this raises significant concerns, especially as llms are increasingly considered for assisting humans in high-stakes decision-making, such as medical question-answering @xcite , resume screening @xcite , and grading @xcite . the use of llms in the hiring process has thereby prompted numerous legislative actions to protect the interests of vulnerable groups, including new york city local law 144 @xcite , and the european union's ai act (commission, 2024), among others. this evokes the extensive literature in labor economics, which defines hiring bias @xcite @xcite and proposes various tests for detecting discriminatory behaviour in real-world employment scenarios @xcite . @xcite introduced taste-based bias, where employers prefer certain types of workers. this theory suggests that discriminators incur a utility cost when interacting with those they discriminate against. expanding on this, @xcite and @xcite introduced statistical bias, where limited information about workers' ability leads firms to rely on easily observable variables like race, age, and gender, which could be used to predict educational attainment, social background and other productivity-relevant traits. distinguishing these biases is difficult, but @xcite showed that employers 'learn' about workers' true productivity over time, reducing the influence of easily observable variables. @xcite provided evidence of racial hiring bias by showing fewer callbacks for fabricated resumes with african-american names. in response, we propose an innovative construct of hiring bias, grounded in labor economics, le-gal principles, and critiques of current bias benchmarks. firstly, hiring bias aligns with the legal concept of disparate treatment, where an individual is treated less favourably, such as being passed over for a job, due to their gender (national academies of @xcite . delving deeper, we can identify two situations that are considered disparate treatment: (1) different call-back rates, job opportunities, or wages between similar groups and (2) differential degrees of uncertainty about job acquisition or wages, as proposed by @xcite . the first is termed level bias, and the second is spread bias. most llm audit studies @xcite @xcite focus on level bias using metrics like the impact ratio or the equal opportunity gap, while only a few consider spread bias. additionally, as mentioned earlier, level bias can stem from two sources: (1) tastebased and (2) statistical. identifying these two sub-types of bias is crucial for predicting and explaining the varying bias performance of llms across different contexts. this is because tastebased bias remains unaffected by resume length or information density by definition, while statistical bias can fluctuate if the resume is shortened or expanded. this distinction could potentially explain the disagreements regarding the direction of biases in the current literature (section 2). figure 1 illustrates our hierarchical construct of hiring biases, differentiating between spread bias (blue) and level bias (pink). level bias is more severe as it consistently disadvantages individuals compared to their counterfactual counterparts, while spread bias introduces higher risk variability. risk-seeking applicants may prefer facing spread bias. within level bias, taste-based bias (red) is more serious as it is unaffected by the extent of the llm's knowledge about the applicant., whereas statistical bias (pink) can be mitigated by providing more applicant information to the llm. to evaluate llms regarding hiring biases defined in our hierarchical structure (figure 1 ), we introduce the jobfair framework. based on a counterfactual approach from the rubin causal model (section 3.4) and inspired by @xcite , we fabricate genders for each resume to create male, female, and neutral versions. llms score these resumes, and scores are ranked using descending fractional ranking, enhancing comparability and assigning cardinal meanings to the outputs (section 3.5). permutation tests assess gender gaps in rank averages and variances, revealing that seven out of ten llms exhibit significant level biases against males in at least one industry, with no observed spread bias (section 4.3). regression analysis highlights pronounced male bias in the healthcare industry compared to others (section 4.3). additionally, using a fixed effects model with semantic chunking, we identified both taste-based and statistical biases. all models, except llama3-8b-instruct and claude-3-sonnet, do not exhibit statistical biases, and their level bias remains consistent despite resume length variations (section 4.4). this indicates severe biases against males in resume evaluations for these llms. ## related work as discussions about automating hiring increase, studies have started focusing on hiring biases in llms. @xcite found significant implicit biases foot_0 against males and mexicans in gpt-3.5 during job recommendation tasks with fabricated resumes. with a similar downstream task @xcite showed that models like roberta-large, gpt-3.5-turbo, and llama2-70bchat exhibit biases similar to humans. @xcite found no detectable race and gender biases in gpt-3.5, bard, and claude for the resume classification task with real resumes. recent studies have also examined biases in resume evaluation. @xcite found gpt-3.5 favoured male and white names over others using a mixed-effects model. by contrast, @xcite revealed significant bias against males and black candidates in resume scoring by gpt-3.5. another study by @xcite on resume evaluations for teaching positions found moderate, non-significant bias favouring females and racial minorities in several models. these stud-ies collectively underscore the critical need to understand and address these biases in automated hiring processes. importantly, the disagreement in the literature highlights the necessity for a reliable framework to measure hiring bias in llms, as, to our knowledge, no such framework currently exists. this motivates us to propose the jobfair. ## methodology we propose jobfair, a comprehensive statisticsbased framework for investigating hiring biases in llms. the framework is structured as follows. ## discussion and conclusion following the jobfair framework, we find that all ten llms exhibit very consistent bias results. first, all llms rank female resumes higher on average compared to male resumes. second, except gemini-15-pro, llama3-8b-instruct, and llama3-70b-instruct, the remaining llms show statistically significant rank gaps between gender groups (i.e., level bias) in at least one industry. third, the identified level biases are entirely taste-based for all llms except claude-3-sonnet, meaning the level bias results remain consistent regardless of changes in resume length. fourth, none of the llms exhibits spread bias (i.e., the rank variance is equal between gender groups). within the jobfair framework, we introduce a new method called ranking after scoring, which enhances comparability across different llms, reduces reject rates, and provides deeper insights than the scoring method used in similar studies: our findings show that the rank orders m ale ≺ f emale ∼ n eutral and n eutral ∼ m ale ≺ f emale occur most frequently across all llms (figure 4 ) when comparing the female, male, and neutral versions of each resume. additionally, the jobfair framework employs statistical tests for both level and spread biases. as demonstrated in section 4.3, the permutation tests are more sensitive to gender bias and have fewer type ii errors compared to the four-fifths rule, which only identifies four biased llms despite clear biases in other models. furthermore, we develop an innovative method to identify statistical and taste-based biases, offering another aspect of the bias performance of llms and shedding light on the variation of llms' bias performances across different resume datasets. although we primarily focus on gender bias, this framework is versatile and can be adapted to explore other social traits and downstream tasks.