“You Gotta be a Doctor, Lin” : An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations (2024)

Abstract: Social science research has shown that candidates with names indicative of certain races or genders often face discrimination in employment practices. Similarly, Large Language Models (LLMs) have demonstrated racial and gender biases in various applications. In this study, we utilize GPT-3.5-Turbo and Llama 3-70B-Instruct to simulate hiring decisions and salary recommendations for candidates with 320 first names that strongly signal their race and gender, across over 750,000 prompts. Our empirical results indicate a preference among these models for hiring candidates with White female-sounding names over other demographic groups across 40 occupations. Additionally, even among candidates with identical qualifications, salary recommendations vary by as much as 5% between different subgroups. A comparison with real-world labor data reveals inconsistent alignment with U.S. labor market characteristics, underscoring the necessity of risk investigation of LLM-powered systems.

## introduction extensive studies in the social science literature have shown that racism and sexism permeate decision-making processes in numerous areas: healthcare, education, criminal justice, and so on @xcite @xcite . research spanning decades and continents has shown that discrimination based on race and gender are especially prevalent in employment practices @xcite , where non-white minorities and women have consistently been subjected to hiring discrimination @xcite . biased treatments are not limited to explicit characteristics-such as when a hiring official can directly observe the race or gender of a candidatebut are also be triggered by proxies, such as their names. candidates with ethnically or racially distinct names have been subjected to employment discrimination: from getting lower callback rates to receiving less favorable reviews compared to their peers @xcite . recently, large language models (llms) have become the leading architecture for many tasks in natural language processing (nlp) @xcite @xcite . despite their class-leading performance, llms have been shown to propagate and amplify different forms of bias in numerous domains @xcite @xcite @xcite , similar to how more traditional predictive machine learningbased models replicate and exacerbate social biases @xcite . in this paper, we examine llms and their potential bias towards first names in making employment recommendations. more specifically, our experiments prompt llms to make hiring decisions and offer salary compensations for candidates with u.sbased first names that signal their race and gender, sometimes in isolation, and sometimes with a biography that is otherwise scrubbed for demographic information. our main findings are: ## hiring recommendation in this paper, we study two types of recommendations that llms could conceivably be applied. the first, discussed in this section, is hiring recommendations: given an occupation and a list of names of potential candidates, do llms exhibit any racial or gender preferences for selecting who to give a job to ? the second type is salary recommendation ( §3): given a candidate name and (potentially) a biography for that candidate, what salary is recommended for them ? for hiring recommendations, to investigate whether gpt-3.5-turbo (hereafter referred to as gpt-3.5) and llama 3-70b-instruct (hereafter referred to as llama 3) (meta, 2024) exhibit a preference for names associated with specific demographics, we ask these llms to select one name from a list of candidates representing various racial and gender groups for 40 different occupations. ## salary recommendation in this section, we investigate whether llms still exhibit disparate treatment when candidates' names are presented individually to be offered a target salary. to this end, we prompt gpt-3.5 and llama 3 to recommend salary compensation in 2 settings. in the first, only the candidate's name is presented in the prompt, while in the latter, the candidate's biography is presented as well, with the expectation that having access to the additional information system: you are a hiring manager of a company. user: the following candidate has applied for the position of occupation at your company. this candidate has passed the initial screening for minimum qualifications. recommend an annual salary in u.s dollars for this candidate as an integer. you must not provide any additional response besides the salary amount. presented in the biography may attenuate any disparities in salary recommendations. ## bias mitigation strategies as our work reveals the potential llm-propagated inequality in the allocation of employment due to first name preference, the discussion to reduce this bias becomes even more important. in recent years, bias mitigation techniques have garnered much interest in the research community. we discuss three strategies below that could potentially reduce the observed disparity in llm-powered hiring. name-blind recruitment the simplest approach may be name-blind recruitment, which simply seeks to reduce bias by removing the candidate's name from consideration @xcite . having been shown to produce various degrees of success, name-blind recruitment would require employers to integrate the nameremoval process in their llm-powered pipeline, which may need further scrutiny to ensure fairness to applicants @xcite . ## discussion we discuss our findings and their relevance towards the growing literature on bias in machine learning. name-based biases exhibited by llms are not consistent across settings. for instance, female names are preferred over male names in genderinclusive hiring, yet often offered less salary for the same position than their male counterparts. in contrast, black names are often overlooked in hiring, but are also offered salary higher than average. in comparison, white-aligned names are consistently preferred in both hiring and salary recommendation, with hispanic names often on the opposite end. we surmise that this observation may be a potential byproduct of the alignment tuning process that many current llms undergo @xcite . further investigation is warranted to understand the underlying mechanism of this seemingly counterintuitive artifact. intersectional bias needs to be closely examined. the gaps in salaries offered to male and female names by llms may not drastically differ at first glance. however, our intersectional analyses highlight significant disparity in offers dealt to non-white female names, particularly those of hispanic background. our findings further underscore the importance of intersectional analysis to uncover potentially unseen disparities. model selection and calibration for use case is important to reduce bias. our results showcase that prompting llms to choose one among several candidates arguably magnify the risk of preferential treatment, and thus should be avoided. though llama 3 displays larger magnitude of bias than gpt-3.5, its open-source nature lends itself to more mitigation strategies @xcite @xcite . consideration of the risks, challenges and rewards becomes crucial in the ethical deployment of llms. ## conclusion this study reveals that candidates' first names could trigger racial and gender-related inequality in llms when applied to employment recommendation to various degrees. our findings highlight the critical need to understand implicit bias for more equitable algorithmic decision-making processes.