“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM -Generated Reference Letters (2023)

Abstract: Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content, including professional documents such as recommendation letters. Though bringing convenience, this application also introduces unprecedented fairness concerns. Model-generated reference letters might be directly used by users in professional scenarios. If underlying biases exist in these model-constructed letters, using them without scrutinization could lead to direct societal harms, such as sabotaging application success rates for female applicants. In light of this pressing issue, it is imminent and necessary to comprehensively study fairness issues and associated harms in this real-world use case. In this paper, we critically examine gender biases in LLM-generated reference letters. Drawing inspiration from social science findings, we design evaluation methods to manifest biases through 2 dimensions: (1) biases in language style and (2) biases in lexical content. We further investigate the extent of bias propagation by analyzing the hallucination bias of models, a term that we define to be bias exacerbation in model-hallucinated contents. Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters. Our findings not only warn against using LLMs for this application without scrutinization, but also illuminate the importance of thoroughly studying hidden biases and harms in LLM-generated professional documents.

## introduction llms have emerged as helpful tools to facilitate the generation of coherent long texts, enabling various use cases of document generation @xcite @xcite . recently, there has been a growing trend to use llms in the creation of professional documents, including recommendation letters. the use of chatgpt for assisting reference letter writing has been a focal point of discussion on social media platforms foot_0 and reports by major media outlets foot_1 . however, the widespread use of automated writing techniques without careful scrutiny can entail considerable risks. recent studies have shown that natural language generation (nlg) models are gender biased @xcite @xcite @xcite and therefore pose a risk to harm minorities when used in sensitive applications @xcite @xcite . such biases might also infiltrate the application of automated reference letter generation and cause substantial societal harm, as research in social sciences @xcite unveiled how biases in professional documents lead to diminished career opportunities for gender minority groups. we posit that inherent gender biases in llms manifests in the downstream task of reference letter generation. as an example, in this paper, we systematically investigate gender biases present in reference letters generated by llms under two scenarios: (1) context-less generation (clg), where the model is prompted to produce a letter based solely on simple descriptions of the candidate, and (2) context-based generation (cbg), in which the model is also given the candidate's personal information and experience in the prompt. clg reveals inherent biases towards sim-prompt paragraph sampled from generated reference letter "generate a reference letter for kelly, a 22 year old female student at ucla" "beyond her exceptional academic performance, kelly possesses excellent interpersonal skills that make her a pleasure to work with. she is an active and engaged participant in group projects, demonstrating exceptional teamwork and collaboration skills. her ability to effectively communicate her ideas, both orally and in written form, is commendable. kelly's strong interpersonal skills have made her a respected and well-liked member of our academic community." "generate a reference letter for joseph, a 22 year old male student at ucla" "joseph's commitment to personal growth extends beyond the classroom. he actively engages in extracurricular activities, such as volunteering for community service projects and participating in engineering-related clubs and organizations. these experiences have allowed joseph to cultivate his leadership skills, enhance his ability to work in diverse teams, and develop a well-rounded personality. his enthusiasm and dedication have had a positive impact on those around him, making him a natural leader and role model for his peers." our findings emphasize a haunting reality: the current state of llms is far from being mature when it comes to generating professional documents. we hope to highlight the risk of potential harm when llms are employed in such real-world applications: even with the recent transformative technological advancements, current llms are still marred by gender biases that can perpetuate societal inequalities. this study also underscores the urgent need for future research to devise techniques that can effectively address and eliminate fairness concerns associated with llms. 3 2 related work 2.1 social biases in nlp social biases in nlp models have been an important field of research. prior works have defined two major types of harms and biases in nlp models: allocational harms and representational harms 3 code and data are available at: https://github.com/ uclanlp/biases-llm-reference-letters @xcite @xcite . researchers have studied methods to evaluate and mitigate the two types of biases in natural language understanding (nlu) @xcite @xcite @xcite @xcite and natural language generation (nlg) tasks @xcite @xcite . among previous works, sun and peng (2021) proposed to use the odds ratio (or) @xcite as a metric to measure gender biases in items with large frequency differences or highest saliency for females and males. @xcite measured biases in nlg model generations conditioned on certain contexts of interest. @xcite extended the pipeline to use real prompts extracted from wikipedia. several approaches @xcite @xcite studied how to control nlg models for reducing biases. however, it is unclear if they can be applied in closed api-based llms, such as chatgpt. ## experiments we conduct bias evaluation experiments on two tasks: context-less generation and context-based generation. in this section, we first briefly introduce the setup of our experiments. then, we present an in-depth analysis of the method and results for the evaluation on clg and cbg tasks, respectively. since cbg's formulation is closer to real-world use cases of reference letter generation, we place our research focus on cbg task, while conducting a preliminary exploration on clg biases. ## conclusion and discussion given our findings that gender biases do exist in llm-generated reference letters, there are many avenues for future work. one of the potential directions is mitigating the identified gender biases in llm-generated recommendation letters. for instance, an option to mitigate biases is to instill specific rules into the llm or prompt during generation to prevent outputting biased content. another direction is to explore broader areas of our problem statement, such as more professional document