Beyond Self-Reports: Multi-Observer Agents for Personality Assessment in Large Language Models (2025)

Abstract: Self-report questionnaires have long been used to assess LLM personality traits, yet they fail to capture behavioral nuances due to biases and meta-knowledge contamination. This paper proposes a novel multi-observer framework for personality trait assessments in LLM agents that draws on informant-report methods in psychology. Instead of relying on self-assessments, we employ multiple observer LLM agents, each of which is configured with a specific relationship (e.g., family member, friend, or coworker). The observer agents interact with the subject LLM agent before assessing its Big Five personality traits. We show that observer-report ratings align more closely with human judgments than traditional self-reports and reveal systematic biases in LLM self-assessments. Further analysis shows that aggregating ratings of multiple observers provides more reliable results, reflecting a wisdom of the crowd effect up to 5 to 7 observers.

## introduction large language models (llms) have demonstrated remarkable abilities to generate human-like responses and engage in complex social interactions @xcite . in particular, llms acquire the emergent ability of role-playing to emulate designated personas, leading to applications in fields like mental health support, education, etc. @xcite . as these llm agents see wider deployment, there is a growing interest in assessing their personality traits @xcite . this task is crucial for a better understanding of their inherent characteristics and for developing more effective and appropriate human-ai interaction frameworks. human personality assessment has a long history in psychology, with various methods developed to evaluate individual traits and behaviors. among these methods, self-report questionnaires are the most common, assessing personality through individuals' responses to standardized questions about their thoughts, emotions, and behaviors. similarly, existing llm personality assessment methods rely heavily on self-report questionnaires, in which an llm is prompted to answer personality test questions. despite their extensive use, researchers have raised concerns about the reliability of self-reports for llm personality assessments, particularly about the stability of personality traits across different contexts @xcite @xcite . furthermore, there are potential risks of data contamination. it is likely that llms are exposed to discussions about personality tests during pretraining, but how this metaknowledge influences their self-report test results remains unclear. in this work, we explore alternative methodologies for llm personality assessments. we begin with the intuition that others, such as friends, family, or colleagues, can provide valuable insights into an individual's personality traits, sometimes even more accurately than the subject individual themselves. in psychology, this intuition leads to the personality assessment method of informantreport @xcite . instead of using the selfreport ratings of the subject, informants other than the subject are asked to provide ratings of the subject's personality. through this approach, it is possible to obtain a more objective assessment of the subject's personality profile. inspired by the informant-report method, we propose a personality assessment framework that leverages multiple llm agents as observers to report on the personality of a subject llm agent (figure 1 ). for a given subject agent, we first prepare n observer agents, each assigned a specific relationship with the subject, such as college classmate or cousin. based on the relationship between the subject and an observer agent, an array of interactive scenarios is generated automatically. for each scenario, we perform a simulation in which the subject and the observer agents engage in a dialogue based on the scenario. then, each observer agent is instructed to complete a personality questionnaire, rating the subject agent's personality based on the simulated dialogues. finally, personality reports from all observers are aggregated to give a final collective assessment of the subject. 1 the experiments show that observer-report ratings align more closely with human ratings, whereas self-report ratings exhibit a strong correlation with the injected personality prompt but fail to reflect actual behaviors. moreover, we find that llms display systematic biases when selfreporting their personality, particularly across dimensions such as agreeableness and conscientiousness. we also analyze the influence of the number of observers (n ) and the relationship between the observer and the subject. the analysis reveals that aggregating multiple observer ratings improves agreement between reported and latent personality, reflecting a wisdom of the crowd effect. the agreement stabilized around 5-7 observers, suggesting an optimal number of observers that can be used for practical assessment. ## methodology we now introduce our multi-observer framework for personality assessment. section 3.1 describes the configuration of the subject and observer agents. we then utilize these agents to simulate scenarios (section 3.2) before observers provide personality assessments on the subject (section 3.3). ## experimental settings we provide details on the experimental settings. llm agents for the subject and observer agents, we use gpt-4o as the llm @xcite . we also conducted experiments based on qwen2.5 and llama-3, refer to appendix a.3 for additional results. we conduct the experiments on a total of 100 subject agents. for each subject agent, we assign @xmath0. among the 15 observers, 5 have friend relationships, 5 have family relationships, and 5 have relationships within the workplace context. relationship and scenario generation we use the gpt-4o model to generate inter-agent relationships and scenarios. for each pair of subject and observer agents, we generate @xmath1. see appendix a.2 for the details of the prompts. personality questionnaire we adopt the international personality item pool (ipip) personality test @xcite , a widely used personality inventory designed for assessing the big five personality traits. the questionnaire consists of @xmath2, each corresponding to one of the big five personality dimensions. 3 ## results in this section, we present the experimental results of our personality assessment method. ## conclusions the study introduced a novel multi-observer framework for personality assessment in llms. compared with the existing self-report method, the proposed observer-report approach shows closer alignment with human ratings. additionally, it reveals systematic self-report biases in the personality dimensions of agreeableness and conscientiousness, which are likely to originate from alignment training. our approach also demonstrates that aggregating responses from multiple observer agents, optimally around 5-7, mitigates individual biases and yields more robust and context-sensitive personality evaluations. our experiments further reveal that relational context and observer diversity significantly impact rating patterns, underscoring the importance of tailored personality assessment strategies. future research will focus on refining the complexity of the scenarios and exploring alternative relationship configurations.