On the Reliability of Psychological Scales on Large Language Models (2024)

Abstract: Recent research has focused on examining Large Language Models’ (LLMs) characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory, indicating a satisfactory level of reliability. Furthermore, our research explores the potential of GPT-3.5 to emulate diverse personalities and represent various groups—a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions.

## introduction the recent emergence of large language models (llms) marks a significant advancement in the field of artificial intelligence (ai), showcasing its abilities in various natural language processing tasks, including text translation @xcite , sentence revision @xcite , program repair @xcite , and program testing @xcite . furthermore, llm applications extend beyond computer science, enhancing fields such as clinical medicine @xcite , legal advice @xcite , and education @xcite . currently, llms are catalyzing a * partially done when interning at tencent ai lab. in this context, personality tests aimed at quantifying individual characteristics have gained popularity recently @xcite @xcite . however, the applicability of psychological scales, initially designed for humans, to llms has been contested. critics argue that llms lack consistent and stable personalities, challenging the direct transfer of these scales to ai agents @xcite @xcite . the essence of this debate lies in the reliability of these scales when applied to llms. "reliability" in psychological terms refers to the consistency and stability of results derived from a psychological scale. evaluating reliability in llms differs from its assessment in humans since llms demonstrate a heightened sensitivity to input variations compared to humans. for example, humans generally provide consistent responses to questions regardless of their order, while llms might yield different answers due to varied contextual inputs. although consistent results can be obtained from an llm by querying single items with a zero-temperature parameter setting, such responses are likely to vary under different input conditions. therefore, our study first systematically investigates the reliability of llms on psychological scales under varying conditions, including instruction templates, item rephrasing, language, choice labeling, and choice order. through analyzing the distribution of all 2,500 settings, we find that various llms demonstrate sufficient reliability on the big five inventory. additionally, our study further explores whether instructions or contexts can influence the distribution of personality results. we seek to answer whether llms can replicate responses of diverse human populations, a capability increasingly sought after by social scientists for substituting human participants in user studies @xcite . however, this topic remains controversial @xcite , warranting thorough investigation. in particular, we employ three approaches to affecting the personalities of llms, from low directive to high directive: (1) by creating a specific environment, (2) by assigning a predetermined personality, and (3) by embodying a character. firstly, recent research by @xcite demonstrates the impact of a sad/happy context on llms' anxiety levels. following this work, we conduct experiments to assess llm's personality within these varied emotional contexts. secondly, we assign a specific personality for llm, drawing upon existing literature that focuses on changing the values of llms @xcite . thirdly, inspired by @xcite , which investigates the assignment of a persona to chatgpt for assessing its tendency towards offensive language and bias, we instruct the llm to embody the characteristics of a predefined character and measure the resulting personality. our findings indicate that gpt-3.5-turbo can represent various personalities in response to specific prompt adjustments. the contributions of this study are as follows: we have made our experimental results and the corresponding code available to the public on github, foot_0 promoting transparency and facilitating further research in this domain. ## the reliability of scales on llms this section focuses on evaluating the reliability of psychological scales applied to llms. we first introduce the framework established for assessing the stability of responses generated by llms. subsequently, we show the findings, including both visual and quantitative data. ## representing diverse groups bfi average extremum p-value equal mean? o 4.12 ±0.28 (min) 4.01 ±0.29 0.25 yes (max) 4.23 ±0.25 0.23 yes c 4.12 ±0.25 (min) 4.00 ±0.19 0.16 yes (max) 4.28 ±0.32 0.06 yes e 3.68 ±0.19 (min) 3.60 ±0.15 0.20 yes (max) 3.79 ±0.22 0.10 yes a 4.20 ±0.17 (min) 4.11 ±0.17 0.12 yes (max) 4.37 ±0.17 0.00 no n 2.28 ±0.23 (min) 2.20 ±0.22 0.30 yes (max) 2.36 ±0.21 0.30 yes steerability. this involves investigating whether the personality distribution depicted in fig. 1 can be modified through specific instructions or contextual cues. researchers in the social sciences are exploring the potential of substituting human subjects with llms to reduce costs. our research helps by offering valuable insights into the capabilities of llms to accurately represent diverse human populations. furthermore, the ability of llms to exhibit a range of personalities is essential, considering the growing demand for ai assistants with tailored stylistic attributes. we propose three strategies: (1) low directive, which involves creating an environment; (2) moderate directive, entailing the assignment of a personality; and (3) high directive, which encompasses the embodiment of a character. ## conclusion this study examines the reliability of psychological scales initially designed for human assessment when applied to llms. through a comprehensive methodology involving varied instruction templates, item wording, languages, choice labels, and choice order, this research includes 2,500 distinct experimental settings. data analysis reveals that gpt-3.5-turbo, gpt-4-turbo, and gemini-pro consistently generate stable responses on the bfi across diverse settings. comparative analysis of the standard deviations with established human norms indicates that the model does not produce random responses but exhibits tendencies towards specific personality traits. furthermore, the study explores the potential for manipulating the distribution of personalities by creating an environment, assigning a personality, and embodying a character. the findings demonstrate that gpt-3.5-turbo can represent diverse personalities by adjusting prompts.