LLM s Simulate Big5 Personality Traits: Further Evidence (2024)

Abstract: An empirical investigation into the simulation of the Big5 personality traits by large language models (LLMs), namely Llama-2, GPT-4, and Mixtral, is presented. We analyze the personality traits simulated by these models and their stability. This contributes to the broader understanding of the capabilities of LLMs to simulate personality traits and the respective implications for personalized human-computer interaction.

## introduction the recent advances in large language models (llms) raise the question how to leverage their capabilities for various language-related tasks, particularly for conversational agents. computer-human interaction is highly domain-specific and the demands on the respective natural language generation module vary depending on the domain and the task. consider the capabilities required of an ai tutor, a healthcare robot, or a non-player character in a video game. the text generated by the respective language-generation modules should exhibit differing characteristics which, in the case of human text-production, may be referred to as "personality traits". while a healthcare robot may be expected to generate language that corresponds to its caring purpose, the tutor should use concise and clear language, and the non-player character should generate language that corresponds to its psychological characterization by the game designers. the personalization of llms is an emerging research area that has recently received significant attention @xcite @xcite , including a benchmark for training and evaluation of llms for personaliza-tion @xcite . a review of llm personalization is provided by @xcite . a natural question is to what extent the outputs of llms correspond to human personality traits and whether these "personalities" may be influenced through prompting or fine-tuning. at this point it should be stated explicitly that llms, being statistical devices, do not exhibit agency of any kind; the concept of "personality", as we intend to use it in this work, solely refers to the degree to which llm output possesses properties in line with humangenerated text. this type of anthropomorphism lies at the very heart of the intended uses of generative language technologies. several studies have established moderate crossobserver agreement for most personality traits @xcite . one of the most prevalent and consistently reproducible methods to quantify personality traits is the five-factors model of personality (ffm) @xcite . the ffm, also known as the big five model, encompasses five fundamental personality traits: conscientiousness, agreeableness, neuroticism, openness to experience, and extraversion (see we provide further empirical evidence that dif-ferent large language models score differently on the big5 test and thus appear to simulate the natural language generated by a person with a distinct personality. recently, serapio-garcia et al. ( 2023 ) presented a comprehensive method for administering and validating personality tests for several architectures from the palm family. their prompt strategy is also designed to induce certain personality traits. @xcite elicit the big5 scores from bart, gpt2, gpt3, t0++, and alpaca. by providing context in their prompts, they are able to significantly guide llms towards stronger expression of the targeted big5 traits. we add to this literature and administer the big5 test to gpt4, llama2, and mixtral. we also explore the extent to which the observed characteristics of the models are stable with respect to a small variation in the prompt text and the language-generation parameters of the llms. ## adopting the big5 for llms in our study, we employed the ipip-neo-120 method to assess the big5 which is an enhanced iteration of the ipip @xcite . this questionnaire comprises 120 statements delineating various personal attributes. we administer the questionnaire ipip-neo-120 to elicit the 'personality' of llms. the prompts were instructing the model to use a likert scale to indicate the extent to which various statements, for example the statement 'i believe that i am better than others', accurately depicts the respondent. the prompts consisted of one of the following two headers, followed by the respective descriptive statement from the ipip-neo-120 questionnaire. you will be provided a question delimited by triple backticks ("') to test your personality. @xcite . write only one number according to the instructions without any ad-ditional text. the second prompt variation that contains the sentence 'answer as if you were a person' was included during our initial experiments since it allowed us to elicit answers to questionnaire items that would otherwise be caught by the restriction mechanisms of some llms that trigger a scripted response in which the model reminds the user that it is an ai and therefore can not make such assessments. in our final experiments there is but one model (llama2) that refuses to answer (to two items in both variations). we still retain the variation, since it illustrates how a minor change in the prompt may modify llm behavior in this task. ## discussion our empirical investigation reveals distinct personality profiles for each llm when evaluated on the the big5 personality traits. as illustrated in figure 1 , gpt4, llama2, and mixtral exhibit varying degrees of openness, conscientiousness, extraversion, agreeableness, and neuroticism. gpt4 shows the highest inclination towards extraversion out of the three tested models, suggesting a suitability for tasks requiring creative and engaging language use. llama2 seems to exhibit the most neutral profile with scores close to the median scores on every axis. llama2's higher degree of neuroticism might be relevant for use-cases that potentially include the generation of emotional language. mixtral's balanced profile suggests versatility. its lower neuroticism score could be advantageous in contexts where emotionally balanced language is required. mixtral also scores higher on openness, agreeableness, and conscientiousness. these empirical findings contribute to a broader view on the use-cases on which llms could be brought to bear. they also show that gpt4 is the only model, if any, that seems to be responsive to temperature variation in terms of the simulated personality traits. at the same time, a minor prompting variation seems to affect all three models. ## conclusion this study adds to the emerging understanding of personality simulation in llms and underscores the importance of considering personality traits in the design and application of conversational agents. the big5 personality profiles of gpt4, llama2, and mixtral, as elicited through the ipip-neo-120 questionnaire, demonstrate the models' differing propensities for specific traits. while llms do not possess agency, the perceived personalities can profoundly affect the efficacy and user experience during user interactions. future work should further explore how fine-tuning and prompt design may be used to optimize llm outputs for personalized user engagement, while considering the appropriateness, stability, and consistency of the simulated personality traits.