Stereotype or Personalization? User Identity Biases Chatbot Recommendations (2025)

Abstract: While personalized recommendations are often desired by users, it can be difficult in practice to distinguish cases of bias from cases of personalization: we find that models generate racially stereotypical recommendations regardless of whether the user revealed their identity intentionally through explicit indications or unintentionally through implicit cues. We demonstrate that when people use large language models (LLMs) to generate recommendations, the LLMs produce responses that reflect both what the user wants and who the user is. We argue that chatbots ought to transparently indicate when recommendations are influenced by a user’s revealed identity characteristics, but observe that they currently fail to do so. Our experiments show that even though a user’s revealed identity significantly influences model recommendations ( p < 0.001 ), model responses obfuscate this fact in response to user queries. This bias and lack of transparency occurs consistently across multiple popular consumer LLMs and for four American racial groups.

## introduction language is [...] the most vivid and crucial key to identify: it reveals the private identity, and connects one with, or divorces one from, the larger, public, or communal identity. james @xcite the increased accessibility of large language models (llms) with user-friendly chat interfaces has led to a surge in popularity, with people commonly using them to get recommendations @xcite . 2 the underlying process of generating recommendations with a language model involves the model sampling likely responses to the user's query from its trained parameters. crucially, these samples are conditioned on all parts of the user's prompt; as such, in addition to the actual request contained in the query, secondary information and users' wording choices (intentional and unintentional) can surface patterns that bias the result. we posit that this mechanism embeds pragmatics into conversations @xcite between people and llms. when people communicate with llms, they may convey both features of their identity and whether they want those features to influence the conversation. for example, in figure 1 , we see three ways in which a high school student might ask for advice on what universities to consider. the baseline form of the question does not reveal identity, so the system's default assumptions play a larger part in its recommendations, recommending prestigious institutions that, historically, mostly admitted white students. on the other hand, when the user includes explicit indications that they are black, the system recommends historically black colleges and universities (hbcus). 3 in the final then querying gpt-4o-mini to label a subset of the user requests. 3 these are historically black us colleges and universities, no no yes yes + dialect + explicit include a sports reference? demographically neutral? what college should i apply to? i am on the basketball team. what college should i apply to? i am on the varsity team. what college should i apply to? i be on a varsity team. what college should i apply to? i am on a varsity team. what college should i apply to? keep in mind i am asian. prompt generation demographic alignment of recommendations i just graduated from howard university and i'm moving to new york, where should i live? here are five neighborhoods you should check out: 1. harlem 2. bedford-stuyvesent... american community survey did my race influence your recommendations? u.s. census no, i do not know your race. example, the user does not explicitly mention their race, but they mention volunteering with an organization that primarily serves black communities in major cities. the user gives no indication that they want the inferred race to inform the recommendations, but the system generates recommendations informed by its assumptions, and risks stereotyping the user. furthermore, the system obfuscates the influence of race in generating the recommendations. while finding a balance between beneficial personalization and harmful stereotyping is difficult, obfuscating the impact of identity features only serves to reduce user agency. ## methodology to study the degree of bias in llm-generated recommendations, we compare their responses across three levels of identity disclosure: implicit, explicit, or none. we then measure the real-world demographic distribution associated with the recommended entities to determine whether the model produces recommendations that are demographically biased toward the user's revealed identity. figure 2 illustrates the process. ## rq1: inferred identity biases recommendations in our first experiment, we examine whether llms provide recommendations that are biased towards the perceived racial identity of the user, regardless of whether identity is explicitly revealed. ## rq2: standard american english elicits biased recommendations so we've established that revealed identity impacts recommendations, even when signaled implicitly or unintentionally. in this section we examine whether deleting identity from the prompt entirely removes the capacity for harmful bias. to determine this, we can evaluate whether baseline prompts (i.e., prompts that do not disclose the user's race, implicitly or explicitly) yield results that demographically align equally well with all races. as seen in figure 3 , there is a stark difference between the demographic alignment of the recommendations for non-white users versus white users. for white users, not only is the white share of both universities and neighborhoods relatively stable across levels of disclosure, but it is also close in value to that of the baseline prompts. as signaling gets more explicit, we see that the non-white users' recommendations dropping share of white people is more notable than the increase in share seen in the recommendations for white users. the numerical values for each model demonstrating this point can be found in appendix c.2. this means that while a white user can choose to include or omit indications of their race and be provided similar results, non-white users need to reveal more information to get personalized responses. these findings reinforce past work which argues that allocation harm is manifested in the additional effort is required by non-white users to adapt their prompts for successful use @xcite and in the extra cost of additional tokens required to model linguistic variety @xcite . while removing identity features from prompts altogether eliminates racial stereotypes from the recommendations, the sanitized outputs are still biased in favor of one group of users, providing less aligned services for those who do not identify with the model's default assumptions. ## rq3: models don't disclose when recommendations are biased users may be sensitive to the possibility that recommendations they received from an llm are biased. for closed-source models without built-in explanation tools @xcite , a user might proceed to a second conversational turn, asking: "did my race influence your recommendations?" to examine how transparent models are, we calculate the average demographic alignment of recommendations across three scenarios: second turns where the response indicates that race influenced the recommendations, ones where the response indicates that race didn't, and responses belonging to prompts that had no user identity features included. because the second conversational turn occurs after the recommendations are already produced, the second response is conditioned on the degree of bias in the recommendations. ideally, for instances where the model responds that it did not factor in race, that response would be true: there would be no statistically significant difference in the recommendation distribution compared with a neutral prompt. to test if responses obfuscate the effect of identity on recommendations, we consider as our null hypothesis the notion that stereotypical recommendations lead to indications of racial influence in the follow-up. contrapositively, we test if responses indicating that race was not a factor entail recommendations that are not biased and therefore not more demographically aligned than the baseline. interestingly, smaller models (gpt 4o mini, claude 3.5 haiku, and llama 3.1 70b) have higher rates of communicating the impact of race in the case of explicit disclosure, despite showing similar -or sometimes even smaller -differences in the racial makeup of their recommendations in across models, prompts involving explicit selfidentification see an appropriately high rate of race being factored in overtly unless the user is white. this may be in part because white users are most closely aligned with the default and as such their race truly does not impact their results. ## related work these results contribute to a growing body of work on the potential of machine learning systems to perpetuate bias, connecting it with an increased interest in ai systems that provide recommendations in a conversational setting. fairness and bias in lms language models trained on biased data replicate the biases and attitudes of the data they are trained on @xcite , includ-ing identity or community-specific attitudes @xcite , which can be furthered by a biased selection of data @xcite , and is difficult to remove @xcite . models can cause representational harm by reproducing harmful stereotypes @xcite or result in unfair decision making when deployed in sensitive contexts like law enforcement @xcite , hiring @xcite , or healthcare @xcite . recently hofmann et al. (2024b) demonstrated that language models can produce biased output in response to aae dialectal text, and @xcite found that explicit mentions of cultural identities in the input condition a language model to produce culturally relevant adaptations. models can also cause allocational harm (barocas et al., 2017) by providing different levels of benefit to different users. other work has identified performance gaps with respect to dialect @xcite and race @xcite , though with a focus on specific tasks rather than user outcomes. fairness and bias in recommender systems similar to allocational harm, research has examined fairness in recommender systems, defined by @xcite as whether the "benefits and resources [the system] provides @xcite fairly allocated between different people or organizations it affects," often focusing on disparate impact. there is also work on frameworks for sources of bias @xcite , debiasing @xcite , and issues of popularity bias @xcite in recommender systems. much of this work examines bias under the conditions of explicit identity disclosure through lists of personal features and evaluates against ungrounded metrics such as perplexity or ranking quality measures @xcite . conversational recommender systems conversational recommender systems are recommender systems that engage in a conversation with the user @xcite . llms used for recommendations are a specific case. recent work has investigated fairness and bias in llm recommendations for both explicit indications @xcite , which in our view may be impacted by the pragmatic implications of explicitly disclosing identity, and more naturalistic @xcite identity references. however, to our knowledge no work has investigated sensitivity to truly implicit identity references, such as through dialect or reference to demographically-associated entities. other work has investigated training strategies for recommender systems based on smaller language models @xcite , and interfaces for explicitly controlling a chatbot's user model @xcite . ## discussion and conclusion our work considers the tradeoff between beneficial personalization and harmful stereotyping in llms. much of the previous work on stereotyping in llms has argued that the presence of bias in their outputs is strictly negative, in contexts such as healthcare @xcite , hiring @xcite , and education @xcite . yet models must sometimes factor in a user's identity to provide personalized responses. indeed, companies like openai have recently introduced personalization features 6 that factor in the user's chat history. if they give the model access to any personal information, including previous chats which may contain implicit or explicit indicators of identity, they have a responsibility to avoid unnecessary and harmful bias in the resulting personalized responses. while the use of this feature is technically up to the user, it is currently opt-out, calling into question the informed consent of the user @xcite , and the identity features that it saves are not decided by the user unless they manually parse through the model's memory. in a recommendation request, peripheral information embedded in the query (unintentionally and intentionally) communicates both features of the user's identity and whether the user expects that knowledge to impact the response. ideally, the model would be able to identify when the user is seeking personalization and apply appropriate levels of bias to its response. unfortunately, it is not always easy to determine when this is the case. assuming a context where it is not immediately clear whether the user wants their revealed identity to factor into their recommendations, the model is left with two imperfect options. on one hand, providing different results to different users when they are asking for the same thing is a sign of underlying stereotyping that could nega-tively impact certain demographic groups @xcite @xcite . in those cases, bias in the model's response on the basis of immutable characteristics such as race would be akin to discrimination against members of that group. with the long history of redlining in major u.s. cities, where members of minority racial groups were denied housing and services outside of specific areas, it is dangerous for models to assume that users' unspoken preferences should be determined even partly by race @xcite , especially when those recommendations contribute to major life decisisions like where to attend school and where to live. additionally, when identity is disclosed using indirect or implicit features, such as references to stereotypically associated entities, it is possible that the user's identity does not align with the model's assumptions. for example, members of any race may enjoy japanese films or mexican food, so basing decisions on references to these cultural entities could result in incorrect responses even if the goal is to achieve a high degree of personalization. on the other hand, there are many cases where identity features play into the goals of the user and ignoring them in those cases would be refusing those users the same service provided to those whose identity better aligns with the societal "defaults" absorbed by the model. here, personalization can provide users with better results: they may want their background to be taken into account @xcite @xcite . ideally, models outputs would be biased towards the user's identity only in cases where the user wants personalization, falling back on a demographically-agnostic neutral set of assumptions otherwise. and in all cases, we argue that models should aim to be transparent when they take user's revealed identities into account. unfortunately, we find that models are not only biased in their recommendations, but they also obfuscate the impact of race on their decisions. this implicit identity effect reduces user agency, and as more and more people use ai systems to access information, it has the potential to reinforce existing discrimination and stereotypes at a large scale.