A gent S ense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios (2025)

Abstract: Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning.

## introduction benefiting from comprehensive training data and large-scale model parameters, large language models (llms) are increasingly employed to develop autonomous agents capable of simulating human behavior @xcite @xcite . these language agents have been explored as human proxies in various fields of behavioral research, such as psychological and sociological surveys @xcite @xcite , and opinion dynamics modeling @xcite . these social science studies often assume that llms exhibit figure 1 : an illustration of challenging yet realistic social scenarios -a family gathering and an office conversation, where the characters are driven by chatgpt. while the dialogue could flow smoothly, emily is unable to achieve her goals during the family gathering and fails to deduce jordan's thoughts in an office setting. early research on the social intelligence of llms relies on static and non-interactive tests that assess commonsense reasoning about social situations @xcite ; benchmark scenario construction scenario (template) coverage interaction patterns social goal private info reasoning evaluation observation perspective sotopia (zhou et al., 2024b) top-down 90 between 2 agents ✓ ✗ subjective judge stss @xcite top-down 30 ≥ 2 agents ✓ ✗ objective n/a agentsense (ours) bottom-up 245 ≥ 2 agents ✓ ✓ mixed self, other, judge to this end, we introduce agentsense: benchmarking social intelligence of language agents through interactive scenarios. we get inspiration from dramaturgical theory @xcite , which conceptualizes social interaction as a theatrical performance in which individuals assume specific roles within various settings. we adopt a bottom-up approach, extracting scenarios from massive scripts to ensure that scenarios and social goals are diverse and grounded in real life. in agentsense, we construct 245 scenario templates from scripts, mitigating data leakage and expanding them into 1,225 scenarios. we apply erg theory @xcite , which categories human needs into existence, relatedness and growth, to show that the constructed scenarios reflect a comprehensive range of human motivations. we then situate llm-driven agents in the constructed scenarios, where each participant has social goals and may also safeguard some private information. in an interactive environment, we simulate multi-turn interactions and observe agents' social intelligence from two aspects: (1) goal completion: have the agents successfully achieved their social goals? (2) implicit reasoning: can the agents accurately deduce others' private information? these abilities are measured through interviews and multiple-choice questions, with our proposed psi metric assessing profile sensitivity. we find that llms struggle with complex social scenarios, particularly with high-level growth goals. our contributions are as follows: ## conclusion in this paper, we introduce agentsense, a benchmark evaluating llm's social intelligence via diverse and challenging social environments. extensive experiments reveal that current llms struggle with complex social scenarios and high-level goals. further analyses verify potential influence factors during evaluation to show the robustness and discriminative power of agentsense.