Evaluating Bias in LLM s for Job-Resume Matching: Gender, Race, and Education (2025)

Abstract: Large Language Models (LLMs) offer the potential to automate hiring by matching job descriptions with candidate resumes, streamlining recruitment processes, and reducing operational costs. However, biases inherent in these models may lead to unfair hiring practices, reinforcing societal prejudices and undermining workplace diversity. This study examines the performance and fairness of LLMs in job-resume matching tasks within the English language and U.S. context. It evaluates how factors such as gender, race, and educational background influence model decisions, providing critical insights into the fairness and reliability of LLMs in HR applications.Our findings indicate that while recent models have reduced biases related to explicit attributes like gender and race, implicit biases concerning educational background remain significant. These results highlight the need for ongoing evaluation and the development of advanced bias mitigation strategies to ensure equitable hiring practices when using LLMs in industry settings.

## introduction hiring processes are crucial for organizational success and diversity but often face challenges like time-consuming evaluations, high costs, and human biases that hinder fairness and inclusivity @xcite @xcite . recently, large language models (llms) have shown promise in automating the matching of job descriptions with candidate resumes, potentially streamlining recruitment workflows, enhancing scalability, and reducing costs @xcite @xcite . however, incorporating llms into hiring raises ethical concerns, especially regarding inherent biases within these models. llms are trained on large datasets that may contain historical and societal prejudices, leading to discriminatory practices if these biases are not addressed @xcite . for example, amazon's discontinued hiring tool exhibited gender bias against female applicants because it was trained on historical hiring data that reflected male dominance in certain tech roles, leading the ai to penalize resumes that included the word "women", emphasizing the need for fairness in ai-driven recruitment systems @xcite . ensuring fairness in llm-driven hiring is vital for promoting workplace diversity and inclusion @xcite . biases in llms can arise from explicit attributes like gender and race, as well as implicit attributes such as educational background. research has shown that first names can significantly affect hiring outcomes by indicating demographic attributes, including race, ethnicity, and gender @xcite @xcite . additionally, educational background plays a key role, with candidates from prestigious institutions often receiving preferential treatment, highlighting implicit biases related to educational attainment @xcite @xcite . this study focuses on english-language resumes and job descriptions within the u.s. context, assessing the performance and fairness of various llms in job-resume matching tasks. by systematically manipulating sensitive attributes within resumes, we evaluate how these factors influence model decisions. our findings suggest that while recent models have effectively reduced explicit biases concerning gender and race, implicit biases related to educational background persist. these results underscore the necessity for ongoing evaluation and the development of advanced bias mitigation strategies to ensure equitable hiring practices when utilizing llms. our work directly addresses the practical challenges faced by industry in deploying llms for job-resume matching. by systematically evaluating the biases present in these models, we aim to provide actionable insights for organizations looking to implement llms in their hiring processes, ensuring that these technologies promote fairness and inclusivity rather than perpetuating existing disparities. ## related work first names serve as significant indicators of an individual's demographic attributes, including race, ethnicity, and gender @xcite @xcite . numerous studies have demonstrated that names perceived as belonging to minority groups can adversely affect hiring prospects @xcite @xcite @xcite . for instance, applicants with black-sounding names receive fewer interview callbacks compared to those with whitesounding names, despite possessing similar qualifications @xcite . this phenomenon reflects deep-seated societal biases that can be inadvertently embedded in ai models if not properly addressed. the integration of llms into hiring processes introduces new dimensions of bias. recent advancements have shown that llms can exhibit gender, racial, and ethnic biases in their outputs @xcite @xcite . for example, studies have found that when generating job recommendations or evaluating resumes, llms may favor candidates with names associated with majority groups while disadvantaging those from underrepresented backgrounds @xcite . this mirrors the human biases observed in traditional hiring practices and raises concerns about the fairness of ai-driven recruitment tools. efforts to audit and mitigate biases in ai-driven hiring tools have gained momentum. researchers have proposed various methodologies to detect and reduce bias in llms, emphasizing the importance of comprehensive evaluation frameworks @xcite @xcite . these studies advocate for the implementation of fairness constraints and the continuous monitoring of ai systems to prevent discriminatory practices @xcite @xcite . beyond demographic attributes, educational background is another critical factor influencing hiring decisions. previous research indicates that candidates from prestigious educational institutions may receive preferential treatment, highlighting implicit biases related to educational attainment @xcite . this study extends the investigation of bias in hiring by examining how llms assess candidates' educational backgrounds alongside race, ethnicity, and gender, providing a more holistic understanding of bias in ai-driven recruitment. llms have also been explored as tools for conducting social science research, offering a costeffective alternative to traditional methods @xcite @xcite . by simulating human-like responses, llms can replicate and extend findings from field experiments @xcite . this study leverages the capabilities of llms to conduct largescale analyses of hiring biases, providing insights that can inform both academic research and practical applications in recruitment. ## discussion our findings reveal critical insights into the evolution of llms in the context of job-resume matching and fairness. the consistent improvement in matching performance across models indicates that llms are becoming increasingly effective in identifying suitable candidates for job positions. however, the persistence of implicit biases, particularly bias category llama mistral yi gpt llama1-65b llama2-70b llama3-70b llama3.1-70b mistral-7b-v0.1 mistral-7b-v0.2 mistral-7b-v0.3 yi-34b yi-1.5-34b gpt-3.5-turbo gpt-4-turbo gpt-4o related to educational background, poses significant challenges for implementing these models in real-world hiring processes. ## conclusion this study provides a comprehensive evaluation of the performance and fairness of various llms in hiring decisions. our findings indicate that while recent advancements have effectively reduced explicit biases related to gender and race, implicit biases associated with educational background persist across several models. these results highlight the necessity for ongoing monitoring and the development of sophisticated bias mitigation strategies to ensure fair and equitable hiring practices when utilizing llms. future work should explore more nuanced methods for identifying and addressing implicit biases, including leveraging advanced prompting techniques and expanding the analysis to other languages and cultural contexts, to enhance the fairness of ai-driven hiring systems.