SOTOPIA -π: Interactive Learning of Socially Intelligent Language Agents (2024)

Abstract: Humans learn social skills through both imitation and social interaction. This social learning process is largely understudied by existing research on building language agents. Motivated by this gap, we propose an interactive learning method, SOTOPIA-π, that improves the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement based training on filtered social interaction data according to large language model (LLM) rating. We show that our training method allows a 7B LLM to reach the social goal completion ability of an expert model (GPT-4-based agent) without the loss of more generic abilities, such as the ability to answer knowledge-based questions. We also demonstrate that this training paradigm uncovers some weaknesses in standard evaluation and safety training paradigms that (1) LLM-based evaluation of social intelligence overestimates the abilities of the language agents trained specifically for social interaction, and that (2) despite not training for better safety or question answering (QA) ability, our methods improve the safety of language agents and maintain general QA ability on the MMLU benchmark.

## introduction machine social intelligence is crucial to productive human-machine interaction @xcite . for instance, to achieve real-time social interactions with users, virtual agents should not only emulate human verbal and non-verbal social behaviors but also manage social skills such as cooperation and negotiation. however, the social intelligence of large language models (llms) still lags behind humans in various aspects, including theoryof-mind @xcite @xcite , following social norms @xcite , and navigating diverse goal-driven social scenarios @xcite . this underscores the challenge to bridge the gap and empower llm agents to navigate social situations with human-like social decision-making abilities and values. inspired by the way that humans acquire these social abilities through exploration, interaction, (3) updates agent policy based on positive data rated by gpt-4. we implement (4) human and gpt-4 evaluation on our trained agent performing tasks in sotopia with the partner agent. our training paradigms include behavior cloning and self-reinforcement. for evaluation, we use sotopia-eval and a fixed partner policy (gpt-3.5-based). note that the character profiles are omitted and the examples are shortened for demonstration. and self-reinforcement @xcite , we propose an interactive learning method, sotopia-π (figure 1 ), which improves the social intelligence of language agents through social interactions (e.g., the conversation between a seller and a buyer on craigslist). in sotopia-π, we use gpt-4 (openai, 2023) to automatically synthesize new social tasks to learn transferable social strategies, similar to openended learning @xcite ## related work social intelligence in llms large language models (llms) have led to new technologies that manage to handle common social use cases, including voice assistants, email autocomplete (chen et al., 2019), ai-assisted counseling @xcite , etc. however, human social interactions are more complicated and diverse than those restricted uses, exposing model limitations in extended contexts. @xcite study the limitations of social intelligence in current llms and conclude that current models struggle with theory of mind tasks such as socialiqa @xcite and tomi @xcite . in the avalon game setting, @xcite show that it is still challenging for llm agents to successfully deceive, deduce, and negotiate with other players, particularly in a multiagent environment. a potential method to improve the theory of mind in language agents is through meta learning the mental model of the interlocutor @xcite @xcite we leave explicity modeling theory of mind in language agents to improve the social intelligence as the future work. these studies show that the effective development of general social intelligence in model training has yet to be fully realized. studies have looked into the potential of behavior cloning from observational data for enhancing social intelligence via interaction @xcite . sotopia-π echos social science theories of inferential social learning @xcite , where models learn not only by imitating but also by making inferences about social contexts. reinforcement learning for llms reinforcement learning from human feedback (rlhf; christiano et al. (2017) ) improves the alignment of llms to human preferences @xcite . direct preference optimization @xcite 2023) ). sil sets a replay buffer and imitates state-action pairs when it is better than the current value estimation. raft generates multiple outputs and utilizes the reward model to filter out a subset. rest is a more complicated version of raft with multiple improve steps. sotopia-π applies offline self-reinforcement to training llms on social tasks and utilizes the gpt-4 to provide rewards for multi-turn social interaction. we leave investigating the effects of different offline methods on training social intelligence to future work. ## conclusion and future work in this paper, we propose an interactive learning method sotopia-π to study how to use llm ratings as a learning signal to improve the social intelligence of language agents. we first find that through optimizing the goal completion score, the general performance on sotopia @xcite , a social intelligence benchmark is improved. however, we find that the gap between llm ratings and human judgment is enlarged through this process. we also find that the sotopia-π improves social intelligence without a loss of general qa ability and with an improvement in safety. although sotopia-π demonstrates strong capabilities of improving social intelligence, several directions will improve our method further. (1) online reinforcement learning: sotopia-π is an offline training method that cannot improve iteratively. future work could study how online methods like ppo @xcite can be applied without the high cost of llm ratings. (2) learning from humans: as mentioned in §2, we use gpt-4 as the expert due to the challenge of collecting human interaction data. future work could explore using existing data including forum conversations, movies, and dialog datasets as offline data for training agents. (3) in §6, we only evaluate one social task, which allows us to dig deep into the task and create customized metrics. also, how to derive safety metrics for all social tasks is an interesting future direction. (4) as demonstrated in §5, the gap between gpt-4 and human evaluation increases as the model optimizes gpt-4 scores. future research could consider more robust evaluation and learning signals for social intelligence tasks.