Towards Objectively Benchmarking Social Intelligence of Language Agents at the Action Level (2024)

Abstract: Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language agents objectively at the action level by scrutinizing the goal achievements within the multi-agent simulation.Additionally, we sample conversation scenarios to build a language-level benchmark to provide an economically prudent preliminary evaluation and align with prevailing benchmarks. To gauge the significance of agent architecture, we implement a target-driven planning (TDP) module as an adjunct to the existing agent. Our evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures. Our code is available at https://github.com/wcx21/Social-Tasks-in-Sandbox-Simulation.

## introduction large language models (llms) have become increasingly powerful owing to the escalation in model magnitude and technical advancements. recent arts such as the gpt series @xcite @xcite showcase human-level performance in diverse tasks within the natural language domain. capitalizing on the instruction-following and emerging commonsense ability of llms, the derived language agents also demonstrate human-level ability in various scenarios @xcite @xcite @xcite . the endeavor to craft comprehensive language agents to simulate humans has prompted substan-tial attention toward the evaluation of social intelligence. in recent times, numerous simulators have been developed for the social simulation of language agents @xcite @xcite . however, the challenge of quantitatively evaluating the emergent social intelligence in these simulations persists. while previous works have introduced numerous benchmarks and evaluation metrics @xcite , we contend these evaluations face two fundamental issues. firstly, these evaluations predominantly focus on the language level, assessing social intelligence through posed questions. however, it remains conceivable that an agent may claim they will perform a certain action without actually committing to it, rendering language-level evaluations inadequate for reflecting actual behaviors. secondly, many evaluations hinge on subjective metrics, diminishing the reliability of the evaluation results. we argue that designing a social intelligence benchmark that can be evaluated objectively at the action level, rather than subjectively at the language level, is necessary. in this paper, we introduce the social tasks in sandbox simulation (abbreviated as stss), an action-level benchmark that assesses task-oriented social intelligence with objective metrics based on the smallville environment @xcite . we craft 30 templates of social tasks across 5 categories with corresponding mechanisms to automatically evaluate agents within the simulation. the overarching architecture of the stss benchmark is illustrated in figure 1 , where the agent is initiated with a social task and is measured through goal achievement in the simulation. since the simulation may be economically expensive, we additionally construct a complementary language-level benchmark by capturing chat scenarios within the simulator, as a preliminary benchmark that aligns with existing language-level benchmarks such as sotopia @xcite . we conduct a comparative analysis between our proposed stss benchmark and prior works in notably, our benchmark evaluates language agents instead of merely language models, considering that the architecture of agents, i.e., how to use the language models is also important to social intelligence. to verify our hypothesis, we introduce an additional target-driven planning (abbreviated as tdp) module that aims at social tasks for the generative agents @xcite . while our framework is versatile for evaluating language models within a fixed baseline agent architecture, it concurrently serves as a testbed of language agent architectures. we conduct evaluations with various popular llms on both levels of our benchmark. the results reveal that the social tasks remain challenging even for cutting-edge models like gpt-4 (ope-nai, 2023), underscoring the potential space for improvement in language models. moreover, a well-designed tdp module demonstrates a substantial enhancement in the performance of language agents at both levels, suggesting that our benchmark can also serve as a testbed of agent architectures employing fixed llm. our contributions can be summarized as follows: ## social tasks in sandbox simulation to objectively assess the social intelligence of language agents at the action level, we suggest building benchmarks within interactive simulators. the simulator can concretize the textual output into tangible actions, facilitating a more objective and quantitative evaluation through analysis of the state and trajectory of the simulation environment. we extend the interactive sandbox environment and the generative agent architecture introduced by @xcite to accommodate the social intelligence evaluation. specifically, we devise a set of social task templates and corresponding metrics, which can be quantified by analyzing the actions of the agents. taking the example of holding a valentine's day party, the metric could be the number of participants at the correct time and location. these templates are then instantiated to specific tasks, each associated with corresponding initial states of the environment, being available for assessing language agents within the simulation. ## experiments in this section, we present the experiments conducted on our benchmarks to assess the social intelligence of language agents through performing social tasks. our experimentation centers on addressing two key questions: (1) to what extent can existing language models and agents execute social tasks within our stss benchmark? ( 2 ) what is the significance of the agent architecture in the context of social intelligence? ## conclusion in this paper, we introduce the social tasks in sandbox simulation (stss) benchmark, which is designed to assess language agents by engaging them in social tasks within a sandbox simulation. the accompanying language-level benchmark also serves as a preliminary evaluation for weaker models. additionally, we propose a target-driven planning module for generative agents to investigate the significance of designing a specialized agent architecture in social intelligence. a comprehensive evaluation involving four prominent language models validates the efficacy of our benchmark for assessing both language models and agent architectures.