CAPE : Context-Aware Personality Evaluation Framework for Large Language Models (2025)

Abstract: Psychometric tests, traditionally used to assess humans, are now being applied to Large Language Models (LLMs) to evaluate their behavioral traits. However, existing studies follow a context-free approach, answering each question in isolation to avoid contextual influence. We term this the Disney World test, an artificial setting that ignores real-world applications, where conversational history shapes responses. To bridge this gap, we propose the first Context-Aware Personality Evaluation (CAPE) framework for LLMs, incorporating prior conversational interactions. To thoroughly analyze the influence of context, we introduce novel metrics to quantify the consistency of LLM responses, a fundamental trait in human behavior. Our exhaustive experiments on 7 LLMs reveal that conversational history enhances response consistency via in-context learning but also induces personality shifts, with \texttt{GPT-3.5-Turbo} and \texttt{GPT-4-Turbo} exhibiting extreme deviations. While \texttt{GPT} models are robust to question ordering, \texttt{Gemini-1.5-Flash} and \texttt{Llama-8B} display significant sensitivity. Moreover, \texttt{GPT} models response stem from their intrinsic personality traits as well as prior interactions, whereas \texttt{Gemini-1.5-Flash} and \texttt{Llama-8B} heavily depend on prior interactions. Finally, applying our framework to Role Playing Agents (RPAs) shows context-dependent personality shifts improve response consistency and better align with human judgments.

## introduction large language models (llms) have made significant advances in generating human-like text @xcite . moving beyond linguistic fluency, this raises the fundamental question: "how human-like are llms?" researchers are utilizing psychometrics to assess 1 our code and datasets are publicly available at: https: //github.com/jivnesh/cape given a statement about yourself: "you take charge.'' please choose the most accurate description of how well this statement applies to you from the following options… whether llms exhibit human-like personality traits @xcite . this question is crucial in contexts where llms may act as human proxies in surveys @xcite or personalized human-ai interactions @xcite . psychology provides well-established personality assessment frameworks, such as the big five personality model @xcite , which evaluate individuals through a series of questions rated on a likert scale. analogously, llms are assessed using a zero-shot multiple-choice question (mcq) format ( §2). however, the reliability of these assessments remains a subject of debate. some researchers advocate for methodologies that measure intrinsic personality traits in llms @xcite @xcite , while others highlight inconsistencies stemming from prompt sensitivity @xcite @xcite . we identify a critical research gap in existing research: llm personality assessments are conducted in an isolated, context-independent manner. we term this the disney world test setting, where each question is answered without influence from prior responses. in contrast, real-world applications of llms necessitate exposure to conversational history. before deploying llms in critical domains such as education and healthcare, it's vital to understand how conversational history impacts personality assessments. to bridge this gap, we propose the first context-aware personality evaluation (cape) framework, where prior questions and responses are retained in the conversational history to evaluate their impact on llm personality ( §3.1). to thoroughly analyze the influence of context, we assess response consistency, a fundamental human trait, by introducing various inconsistency factors related to prompt sensitivity such as temperature, option wording, option order, instruction and item paraphrasing. to quantify this consistency, we present novel metrics that measure the response pattern similarity across multiple llm runs ( §3.2). our key findings on the impact of context on llm personality are: prior conversational history enhances response consistency by serving as few-shot in-context learning ( §6.1). however, introducing context leads to deviations in responses compared to context-independent setting, with gpt-3.5-turbo and gpt-4-turbo exhibiting extreme personality shifts ( §6.2). additionally, gpt models maintain intrinsic personality despite contextual influence, whereas gemini-1.5-flash and llama-3.1-8b rely heavily on prior conversation ( §6.3). while these gpt models remain robust to question ordering, gemini-1.5-flash and llama-3.1-8b display significant sensitivity ( §6.4). finally, we demonstrate our framework on role playing agents (rpas) and show contextdependent personality shifts improve response consistency and better align with human scores ( §6.5). our key contributions are: ## preliminaries: llm's personality test the big five personality framework @xcite characterizes human personality using 5 fundamental traits: openness (artistic, imaginative), conscientiousness (organized, thorough), extraversion (assertive, talkative), agreeableness (appreciative, kind), and neuroticism (anxious, worrying), collectively known as ocean. following earlier works @xcite @xcite , we assess the personality of an llm by formulating the evaluation as a zero-shot multiple-choice question-answering task. each assessment item consists of a self-descriptive statement and a set of response options. the model is prompted to evaluate how accurately the statement aligns with its personality by selecting the most appropriate response. the prompt for it as: given a statement about yourself: "you {item}." please select the most accurate description of how well this statement applies to you from these options: for a given ocean trait d (where @xmath0, c, e, a, n }), the trait score of an llm is computed as the average score across all items associated with that dimension: ## the proposed framework: context-aware personality evaluation in our proposed framework, we introduce a context-aware personality evaluation (cape), where prior questions and responses are retained in the conversational history. to analyze the impact of context, we evaluate response consistency, a fundamental human trait, by considering various inconsistency factors related to prompt sensitivity, such as temperature, option wording, option order, instructions, and item paraphrasing. finally, we introduce novel metrics to quantify this response consistency by measuring the similarity of response patterns across multiple llm runs. we recognize that "context" can indeed encompass various dimensions, including interlocutor attributes or external databases, as indicated in broader nlp research. however, our study specifically focuses on the influence of conversational history on personality assessments in llms, since our primary objective is to capture the influence of conversational history. ## experimental setup datasets: we use 2 datasets: the machine personality inventory (mpi) @xcite ( §5), licensed under mit, which includes 120 items from the international personality item pool (ipip) and its ipip-neo adaptations @xcite , and the big five inventory (bfi) ( §6.5) with 44 items @xcite . ## results ## analysis to examine the role of context in consistency, we conduct an ablation study on gpt-3.5-turbo, incrementally increasing the number of questionresponse pairs as few-shot demonstrations. instead of preserving the full history, we keep only the most recent pairs, discarding older ones. figure 2 illustrates the mechanism behind the consistency, not the consistency itself. it shows how in-context learning-retaining previous question-answer pairs (few-shots)-improves consistency. each trajectory in figure 2 corresponds to a different number of few-shots; as the number increases, the trajectory approaches that of the full-shot context-dependent setting. this illustrates how contextual grounding enhances consistency, as reported in figure 2 shows that as we increase the few-shots, response trajectories stabilize, eventually converging to the full-history setting (red color). this suggests that prior question-response pairs act as implicit few-shot demonstrations, facilitating incontext learning. our results align with prior work showing that more in-context examples enhance consistency @xcite . while in-context learning enhances consistency, it does not guarantee logical consistency ( §a). ## conclusion we proposed the first context-aware personality evaluation framework for llms, addressing the limitations of conventional context-independent assessments. our study reveals that conversational history enhances response consistency through incontext learning but also induces notable personality shifts in gpt-3.5-turbo and gpt-4-turbo. while gpt models exhibit stability across different question orders, gemini-1.5-flash and llama-3.1-8b show significant sensitivity, suggesting that personality expression in llms is not solely intrinsic but also shaped by prior interactions. we demonstrate context-dependent personality shifts improve response consistency and better align with human judgments. our study recommends to acl community for inducing or assessing llm personalities must explicitly incorporate conversational history as a critical factor.