Rethinking Personality Assessment from Human-Agent Dialogues: Fewer Rounds May Be Better Than More (2025)

Abstract: Personality assessment is essential for developing user-centered systems, playing a critical role across domains including hiring, education, and personalized system design. With the integration of conversational AI systems into daily life, automatically assessing human personality through natural language interaction has gradually gained more attention. However, existing personality assessment datasets based on natural language generally lack consideration of interactivity. Therefore, we propose Personality-1260, a Chinese dataset containing 1260 interaction rounds between humans and agents with different personalities, aiming to support research on personality assessment. Based on this dataset, we designed experiments to explore the effects of different interaction rounds and agent personalities on personality assessment. Results show that fewer interaction rounds perform better in most cases, and agents with different personalities stimulate different expressions of users’ personalities. These findings provide guidance for the design of interactive personality assessment systems.

## introduction quantifying and benchmarking human behavior has always been an important topic in fields such as social science, philosophy, and psychology. as a core research direction, personality assessment not only helps reveal the internal mechanisms of individual behavioral patterns, thinking processes, and emotional responses, but also provides scientific evidence for mental health diagnosis @xcite , career planning @xcite , and personalized services @xcite . with the emergence of chatbots and conversational ai systems becoming seamlessly integrated into daily life, automatically assessing human personality through natural language interaction has gradually gained more attention. from early dictionary-based tools like liwc @xcite to supervised learning model methods @xcite , the rapid development of large language models (llms) provides unprecedented opportunities for dynamically capturing personality traits through natural language, such as psycot @xcite and eerpd @xcite . personality refers to a dynamic organization of psychophysical systems that creates characteristic and consistent patterns of human behavior, cognition, and feelings @xcite . personality influences the patterns of human interactions with other individuals @xcite . social penetration theory uses the "onion model" to describe personality @xcite , which suggests that personality consists of multiple layers that are gradually revealed through interaction. however, current datasets for personality assessment through natural language lack consideration of interactivity. they mainly fall into two categories: one identifies personality traits from static texts like blogs (e.g. mbti foot_0 ) and articles (e.g. essays (pennebaker and king, 1999)), which are easy to obtain but lack interactivity and struggle to reflect personality traits embedded in dynamic communication; the other uses manually annotated tv show or movie dialogues such as friendspersona @xcite and personalityevd @xcite , providing interactive contexts but limited by acted and maybe exaggerated personalities, resulting in annotations lacking ecological validity in real environments. how to naturally and stably elicit comprehensive personality expressions at the language level in real interactive situations is key to effectively building datasets. media equation theory suggests that people unconsciously apply social rules when interacting with computers @xcite . with advances in llms for human-agent interaction, combined with their excellent interactive capabilities in role-playing and personality simulation tasks @xcite @xcite , new opportunities have emerged. compared to human-to-human dialogues, interactions with agents are more stable in long, multi-round conversations @xcite , creating new chances for personality assessment. leveraging these advantages, we developed five llm agents using prompts based on the big five theory @xcite and designed game scenarios to constrain conversations and elicit personality expressions (shown in fig. 1 ). through these interactions with 42 real users, we constructed the personality-1260 dataset containing 1260 rounds of dialogues along with participants' bfi-44 personality questionnaire results. this dataset helps study personality in human-agent interactions. with personality-1260 as data support, we explored personality assessment patterns in multiround game scenarios between humans and agents with different personalities. in our research, we first validated the effectiveness of the dataset through statistical analysis and visualization. then, based on these preliminary results, we compared the ef-fectiveness of using different numbers of interaction rounds for personality assessment. finally, we conducted further experiments by comparing interactions with agents having different personalities and their impact on assessment results. building on these results, we aim to comprehensively evaluate personality assessment in human-agent interaction, focusing on the following research questions: ## conclusion in this study, we focused on personality assessment in human-agent interaction and introduced personality-1260, addressing the gap in existing datasets that lack either interactivity or authentic user labels. we validated this dataset's effectiveness through statistical analysis and visualization. based on personality-1260, we experimentally explored how different interaction rounds and agent personalities influence personality assessment. contrary to intuition, our results demonstrated that in most cases, extending the number of rounds either decreases performance or yields no improvement. additionally, we found that the interacting agent's personality influences the accuracy of personality assessment. based on these experimental findings, we proposed three design recommendations for interactive personality assessment systems. we hope these insights can provide guidance for the future design of interactive personality assessment systems.