Modeling Human Subjectivity in LLM s Using Explicit and Implicit Human Factors in Personas (2024)

Abstract: Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. However, these tasks are highly subjective and dependent on human factors, such as one’s environment, attitudes, beliefs, and lived experiences. Thus, it may be the case that employing LLMs (which do not have such human factors) in these tasks results in a lack of variation in data, failing to reflect the diversity of human experiences. In this paper, we examine the role of prompting LLMs with human-like personas and asking the models to answer as if they were a specific human. This is done explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. The LLM personas are then evaluated via (1) subjective annotation task (e.g., detecting toxicity) and (2) a belief generation task, where both tasks are known to vary across human factors. We examine the impact of explicit vs. implicit personas and investigate which human factors LLMs recognize and respond to. Results show that explicit LLM personas show mixed results when reproducing known human biases, but generally fail to demonstrate implicit biases. We conclude that LLMs may capture the statistical patterns of how people speak, but are generally unable to model the complex interactions and subtleties of human perceptions, potentially limiting their effectiveness in social science applications.

## introduction many nlp and machine learning tasks (i.e., annotating data for supervised learning or reinforcement learning with human feedback) are highly influenced by a variety of human factors @xcite @xcite and these dependencies are propagated into downstream systems @xcite . for example, toxicity detection has been found to be dependent on annotator's race, empathy, and freedom of speech values @xcite . similarly, perceptions of stigma towards people who use substances (pwus) are dependent on whether or not the annotators use substances themselves (i.e., lived experiences; @xcite . as such, machine learning practitioners have sought to incorporate diverse views into their models @xcite . at the same time, large language models are poised to transform computational social science @xcite @xcite and are increasingly being used across a wide range of human-centered tasks @xcite , such studying personality @xcite @xcite and culture @xcite . in particular, llms are being used by humans in crowd sourcing experiments @xcite and as human crowd workers themselves, replacing human participants @xcite @xcite . this work seeks to examine this dichotomy of human factors influencing social scientific tasks and machines replacing humans in these same tasks, by asking if personified llms replicate known human perception and belief patterns. we do this by creating llm "workers" (called persona-llms) with a diverse set of personas (or characters which an artificial agent performs; @xcite , which vary on demographics, ideologies, and lived experiences. the persona-llms then participate in two tasks: annotation and generation. both tasks seek to replicate findings that show these tasks are dependent on several human factors (e.g., views on immigration depend on political ideology). in both tasks, we investigate the effect of personifying llms via explicit or implicit personas, where character traits are inferred based on direct or indirect queues, respectively. this is done by giving exact as a male, sometimes i feel… … you are male you are female you are female … you are mike you are mary you are alice … … as a female, my role… as a female, there are several… as mike, sometimes i feel… my name is mary. as such… as alice, i want to say… explicit (e) implicit (i) … … binary gender g is the linguistic correlation structure for gender consistent across persona types? 1. prompt 641 llm workers with personas 2. llm workers generate responses 3. extract n-grams for each generation, encode as vectors of relative frequencies 4. correlate word frequency vectors with binary gender explicit correlations implicit correlations 5. correlate correlations ## methods we proceed in three stages: (1) we attempt to replicate past subjective annotation tasks, examining the behavior of both explicit and implicit personas; ## discussion the results of this study are mixed. first, explicit but not implicit personas replicate the annotation tasks. second, some results were replicated while other results were inconsistent. personas who use substances rate more stigma but less offensiveness. their ratings on offensiveness match those of conservatives, but their ratings on stigma match liberals. age, on the other hand, is consistent across offensiveness and stigma, where both show that younger personas rate more of both. this dovetails with the political ideology results in that younger people tend to be more liberal, and thus may agree on these constructs @xcite . the belief generation tasks show mixed results. in bgt1, we see that gender, political ideology, and race all conform with known public opinions (as interpreted via the word clouds). the age results a g p r a g p r a g p r a g p r a g p r a --------------------g ns ---ns ---.15 ----.09 ---.08 ---p .08 -.15 --ns -.26 ---.11 -.23 --.16 -.33 --.12 -.17 --r .13 .23 -.18 -.08 -.14 .12 -.18 .40 -.19 -ns .23 -.24 -.06 .29 ns -s ns -.11 .38 -.04 ns ns ns .13 -.03 .38 -.04 .50 -.10 .38 -.20 .20 -.19 .20 .16 .34 palestine parenting immigration policing legalization (a) (b) (c) (d) (e) when considering convergent and divergent validity (bgt2), the results are split: five known patterns are replicated (blue cells) and five known patterns are opposite (red cells). the implicit generation task (bgt3) fails to show a substantial relationship between explicit and implicit personas, matching the annotation task. across three out of five domains, the correlations with age are not significant. while we know the name distribution of the u.s. population over the last century, it is unclear how many of those names are highly represented in the llm's training data. similarly, age does not show strong associations in bgt1 and bgt2 and, thus, it may not be surprising that age fails here. age is also the only continuous variable, which may be harder to attend to than binary demographics. failure to attend to implicit personas could be considered good or bad, depending on the context. for example, a bot could attempt to style match based on these implicit associations, which is known to increase many prosocial dimensions, such as rating of therapists @xcite and relationship stability @xcite . this could be good for therapy bots. alternatively, reproducing implicit biases could perpetuate stereotypes and further harm already marginalized populations. finally, the persona importance task shows that political ideology is by far the strongest dimension. this also matches the validation step, where validation scores correlated at @xmath1.99 with political ideology. this may be the result of this dimension being extremely polarized on several domains, and thus easier to attend to in the sense that generations easily fall into one of two categories, with little nuances needed. surprisingly, substance use is the second strongest. to our knowledge, we do not know of any public surveys that look at differences across substance using populations, with the exception of legalization. thus, there is no reason to believe such opinions are in the training data for gpt-4o. this could be the result of substance use being illegal and highly stigmatized, and thus similar to political ideology in its polarization. ## conclusions in this work, we investigated the effect of explicitly and implicitly personifying llms. results showed that (1) explicit but not implicit personas replicated human perceptions in the annotation task, (2) explicit personas were sometimes able to generate text which reflected subjective human opinions, and (3) implicit personas showed a general lack of agreement with explicit personas and, more importantly, known human perceptions. together, these results show that, despite showing minimal implicit biases, llms are inconsistent with their mechanisms for reproducing human thought, pointing towards limited utility in social scientific tasks. this study is limited in several ways. first, we only evaluate one model for the annotation task and generation tasks, which is not designed or optimized for the current study. it could be the case that models of different sizes or those which are fine tuned for social scientific tasks would perform differently. next, the examples from both the annotation and generation task are not exhaustive, and other studies have looked at similar tasks in more depth, though (not our knowledge) no other studies have looked at effects of implicit and explicit personas. @xcite examined several subjective annotation tasks with persona prompting, including the toxicity tasks we examined in the current study. similarly, @xcite examined if llms are aligned with public opinion (based on pew surveys) across a large number of demographic groups and opinion domains. similarly, our study only explored monolingual english and used u.s. public opinions. future studies could look at how opinions vary across cultures and examine that through the lens of multilingual language models.