Beyond Binary Gender Labels: Revealing Gender Bias in LLM s through Gender-Neutral Name Predictions (2024)

Abstract: Name-based gender prediction has traditionally categorized individuals as either female or male based on their names, using a binary classification system. That binary approach can be problematic in the cases of gender-neutral names that do not align with any one gender, among other reasons. Relying solely on binary gender categories without recognizing gender-neutral names can reduce the inclusiveness of gender prediction tasks. We introduce an additional gender category, i.e., “neutral”, to study and address potential gender biases in Large Language Models (LLMs). We evaluate the performance of several foundational and large language models in predicting gender based on first names only. Additionally, we investigate the impact of adding birth years to enhance the accuracy of gender prediction, accounting for shifting associations between names and genders over time. Our findings indicate that most LLMs identify male and female names with high accuracy (over 80%) but struggle with gender-neutral names (under 40%), and the accuracy of gender prediction is higher for English-based first names than non-English names. The experimental results show that incorporating the birth year does not improve the overall accuracy of gender prediction, especially for names with evolving gender associations. We recommend using caution when applying LLMs for gender identification in downstream tasks, particularly when dealing with non-binary gender labels.

## introduction name-based gender prediction is the task of identifying the most likely gender label for a given name. this task, while not reflective of the true gender identify of the individual, is often useful for aggregate downstream analysis and as a demographic feature for predictive models. prior work has utilized name-based gender prediction to investigate gender bias in scientific productivity, citation practices, information extraction systems, personalized marketing, content recommendation, targeted advertising, gender-based sentiment analysis, and social network analysis @xcite @xcite @xcite @xcite @xcite . most prior work has utilized computational tools (e.g., genderize.io foot_0 , namsor foot_1 , gender api foot_2 , or machine learning (ml) models) or datasets (e.g., us ssn) to assign probabilities of a name (along with other features like demographics, time) likely to be a male or a female. since name-based gender is used both as a feature in downstream systems and an indicator of demographic representation, it can lead to both measurement bias and representational bias as identified in the framework proposed by @xcite . a prevalent challenge in contexts utilizing inferred gender is the practice of treating gender as a binary construct, strictly categorizing names as either male or female @xcite . this reliance on binary labels likely stems from historical and societal norms that often only recognize these two categories. binary representations can reinforce existing gender biases and exclude non-binary and gender-diverse individuals, hindering their representation and understanding @xcite @xcite in algorithm design and data annotation. the presence of gender-neutral names, as defined by @xcite , further complicates this issue. these names, frequently assigned to both genders, contradict the binary classification system, leading to potential inaccuracies and misrepresentations in data and processes reliant on gender predictions. this study aims to answer the following research questions to examine one of many aspects of gender biases in llms concerning gender prediction, especially for gender-neutral names and gender labels that change over time (figure 1 ). ## related work in the gender prediction task, models are trained to predict or classify gender labels based on various input features, such as first or last names, country information, behavioral data, or textual content from social media activity @xcite @xcite . consequently, the accuracy of gender prediction can impact the validity of research findings and derived implication, such as policies. in other words, inaccurate gender prediction can distort results and lead to misunderstandings of gender-related biases. moreover, the reliance on binary gender categorizations constrains the nuanced understanding of bias and the representation of individuals. therefore, ensuring accurate and unbiased gender prediction is essential as it can impact the fairness and effectiveness of downstream applications. previous studies found prevalent biases in nlpbased gender prediction using gender-predicting software tools @xcite , which failed to appropriately capture the fact that gender exists on a non-binary scale. while most studies of bias in gender prediction relied on binary gender labels @xcite , some studies have gone beyond binary labels by introducing an additional category for names that were not strictly associated with either female or male genders @xcite @xcite . for instance, @xcite categorized names that appeared as both female and male as "gender ambiguous". additionally, most prior work on gender prediction used names as the only input feature @xcite @xcite , while others such as @xcite and misa (2022) inferred the gender of first names using historical datasets with multilpe features. recent advances in deep learning (dl) have produced pre-trained language models like bert @xcite , charbert @xcite , and roberta @xcite , which have been widely used for gender prediction. for example, @xcite found that using the user's name achieved higher gender prediction accuracy than using other features (e.g., website page views and clicks) in both ml and dl models, while @xcite and pham and nguyen (2023) demonstrated the effectiveness of bert-based models for gender prediction for japanese and chinese names. despite these developments, few studies focused on gender prediction using autoregressive models like chatgpt (openai, 2024a) and llama 2 @xcite . the increasing application of llms for gender prediction @xcite underscores the need to evaluate the limitations of llms, particularly for genderneutral names. for example, @xcite used a prompting approach with chatgpt to predict the gender of olympic athletes, showing chat-gpt performed at least as well as common com-mercial tools (i.e., gender-api and namsor) and often outperforms them on a binary gender scale. in this paper, we conducted experiments beyond prior approaches by introducing the gender-neutral label and using three social security administration (ssa) baby name datasets to investigate gender biases by predicting non-binary gender labels. ## experiments this section discussed the datasets, pre-processing, experimental design, and how we compared various models for name-based gender prediction. ## discussion llms are poor at accurately predicting gender. gender bias occurs in llms when performing name-based gender predictions, which shows varying performance in predicting non-binary gender labels. llama 2 categorizes nearly all names as neutral genders, with first names only as input. this tendency may result from llama 2's training approach, which used reward modeling to promote more inclusive responses, where initial model outputs are adjusted based on human feedback to maximize inclusivenes and factual accuracy @xcite . the rewarding process allows the model to better align with modern datasets' nuanced and inclusive expectations. including temporal information mostly degrades accuracy. when providing dynamic gender label datasets with birth year information, the gender-prediction performance of most llms decreased, especially for gender-neutral names. however, mixtral-8x7b showed an increase in overall accuracy when birth years were added in 0-and 5-shot settings. we hypothesize that mixtral-8x7b can better use temporal data as a reference for gender prediction because it is trained with more numerical information. although llama 2 outperformed other llms in predicting gender-neutral names, it exhibited biased prediction results, often classifying most names as gender-neutral. we assume llama 2's reinforcement learning with human feedback (rlhf) approach @xcite guides the model to generate more inclusive responses. when llama 2 is unsure about a name's gender, it may default to labeling it as neutral, potentially reducing prediction accuracy for gender-neutral names. llms have worst performance on genderneutral names. we also find that most tested llms have more difficulties in predicting genderneutral first names than binary genders, which may stem from the training data of llms that primarily includes binary gender labels in the training documents @xcite . llama 3, in particular, performed poorly overall across all three datasets with different input variations (i.e., first names with or without birth years). as detailed in appendix a, the datasets used for dynamically labeling genders were imbalanced, with gender-neutral names being the majority. specifically, the total numbers of gendered names for the us, canada, and france ssa datasets were 3,996, 1,308, and 449, respectively, with around 58.1%, 82.9%, and 75.5% being gender-neutral. consequently, llama 3 underperformed in overall prediction accuracy compared to other llms due to its poor accuracy in predicting neutral genders despite performing better in predicting binary genders. llm performance is biased towards recent year patterns. based on the over-time comparison of the us ssa dataset (figure 2 ), we hypothesize that the improved prediction performance of llms for recent data can be attributed to the increased volume of training data from recent years. we assume that the training data of llms is unbalanced, predominantly consisting of recent data, potentially explaining the higher gender prediction accuracy of llms in recent years. the comparison of balanced ssa datasets and dynamic gender label datasets shown in suggestions for practitioners as we have highlighted in this work, llms have a biased and inaccurate understanding of names and hence we should be careful about using them for gender inference related tasks, even at an aggregate level. furthermore, when dealing with temporal and especially historical data, llm's name-based gender understanding may be limited and hence their usage for aggregated data analysis is likely to lead to incorrect results. ## conclusion this study underscores the limited performance of llms as classifiers in predicting gender-neutral names compared to binary genders and the challenges posed by the inherent biases in the datasets used to train llms, which may lead to unbalanced gender prediction results. by introducing a "neu-tral" category, we have taken a step towards more inclusive gender prediction. however, our findings revealed that llms may struggle recognizing gender-neutral names, especially for non-english first names. despite efforts to enhance llms' predictive capabilities by including temporal data, there were no meaningful improvements in gender prediction accuracy, especially for gender-neutral names. this suggests a fundamental limitation of current llms and training datasets when adapting to the complexities of gender identities. in future studies, we plan to expand our work by using more inclusive gender categories (e.g., cisgender and transgender) to thoroughly assess gender bias in llms across various nlp downstream tasks, including sentiment analysis and coreference resolution.