P ersona LLM : Investigating the Ability of Large Language Models to Express Personality Traits (2024)

Abstract: Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then assess their essays with automatic and human evaluations. Results show that LLM personas’ self-reported BFI scores are consistent with their designated personality types, with large effect sizes observed across five traits. Additionally, LLM personas’ writings have emerging representative linguistic patterns for personality traits when compared with a human writing corpus. Furthermore, human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80%. Interestingly, the accuracy drops significantly when the annotators were informed of AI authorship.

## introduction with llms' impressive ability to engage in humanlike conversations, there has been a surge of interest in building personalized ai agents that interact with and support humans in various contexts. startups such as character ai 1 and replika 2 have engaged many users through virtual characters on their fast-growing platforms. meanwhile, in the academic sphere, research @xcite has also suggested that generative 1 https://character.ai/ 2 https://replika.ai/ agents can exhibit believable human behavior and could potentially be used to simulate human agents in social science studies. however, while these generative characters are becoming ubiquitous, it is a common yet unsubstantiated assumption that these agents consistently behave in a human-like manner. recent studies in the field of llms and personality have started to provide some empirical support. for example, recent research has studied personality expression in llm-generated content @xcite @xcite , created new benchmarks to measure personality expressed by llms @xcite @xcite , and proposed better prompting techniques to induce @xcite @xcite , and edit @xcite personality expressed by llms. despite these advancements, there has been little research in nlp that leverages insights from personality psychology and psychometric tools to study if llms can dutifully express personality traits. furthermore, there is little work that explores how these agents assigned with certain personality traits are perceived by humans. drawing on the extensive research of the big five personality model @xcite , we aim to investigate the capability of llms in expressing the big five personality traits -namely extraversion, agreeableness, conscientiousness, neuroticism, openness to experience. in this paper, we define an llm persona to be an llm-based agent prompted to generate content that reflects certain personality traits as defined in its initial prompt configuration. in our study, we first seek to determine whether these llm personas can accurately reflect their assigned personalities when taking a personality test. given that they show promising results on that initial exploration, we pursue the question of whether they can create narratives that are indicative of their assigned personality traits. to assess their generated stories, we extract psycholinguistic features using dictionaries that have been designed to analyze human behavior and propose to use these features to study the behavior of llm personas. we also investigate whether llm personas are proficient in using lexicons related to their assigned personality profiles and whether they can convincingly portray these personalities to human observers. in other words, can human evaluators discern the assigned personality traits of the llm personas based on their narratives? lastly, we suggest potential avenues for extending this evaluation to more real-life scenarios, such as multi-round dialogues and action planning. motivated by these inquiries, we aim to provide a comprehensive evaluation of llm personas, focusing on the following research questions (rqs): ## experiment design as shown in figure 1 , this paper investigates the behavior of llm personas through a multi-faceted approach. we start by creating llm personas with distinct personality traits and administer a personality assessment to them. subsequently, we prompt these llm personas to write stories, which we then analyze using the widely adopted linguistic inquiry and word count (liwc) framework. following this analysis, we recruit human evaluators to manually evaluate the stories and concurrently carry out an llm-based automatic evaluation. both human and llm evaluators are required to (1) assess these stories across six dimensions, namely readability, personalness, redundancy, cohesiveness, likeability, and believability, and (2) infer the personality traits assigned to the llm personas from the stories. the code, data, and annotations for our experiments are open sourced foot_0 . ## conclusion in this work, we explore the capability of llms (with a focus on gpt-3.5 and gpt-4) to consistently express a personality profile using a wellvalidated personality scale. specifically, we investigate the behavior of llm personas in completing the bfi test and story writing and run analyses with psycholinguistic features, human evaluation, and personality prediction. through psycholinguistic analysis, we find that llm personas from gpt-3.5 and gpt-4 can consistently tailor their bfi answers to match their assigned personalities and write with linguistic features characteristic of those personality traits. regarding our investigation on linguistic patterns in writing, we found that each personality trait is associated with different representative linguistic behavior of llm personas in writing. we also find a notable alignment in word usage between humans and llm personas. on llms' ability to generate stories conditioned on certain personality profiles, we find that the stories are not only linguistically fluent and structurally cohesive, but also convincingly believable. moreover, our investigation shows that human evaluators' perception of readability, redundancy, cohesiveness, likeability, and believability remains consistent regardless of whether they are aware that the content is generated by an llm. we also show that human judges are able to predict personality traits (expressed in the llm-generated content) with varying degrees across various personality traits. perhaps, the most interesting finding is that awareness of ai authorship influences the human judges' perception of personality as we notice that the accuracy (of predicting prompted personality traits) decreases (with varying degrees) when human judges are aware of ai authorship.