Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View (2024)

Abstract: As Natural Language Processing (NLP) systems are increasingly employed in intricate social environments, a pressing query emerges: *Can these NLP systems mirror human-esque collaborative intelligence, in a multi-agent society consisting of multiple large language models (LLMs)?* This paper probes the collaboration mechanisms among contemporary NLP systems by melding practical experiments with theoretical insights. We fabricate four unique ‘societies’ comprised of LLM agents, where each agent is characterized by a specific ‘trait’ (easy-going or overconfident) and engages in collaboration with a distinct ‘thinking pattern’ (debate or reflection). Through evaluating these multi-agent societies on three benchmark datasets, we discern that certain collaborative strategies not only outshine previous top-tier approaches but also optimize efficiency (using fewer API tokens). Moreover, our results further illustrate that LLM agents manifest human-like social behaviors, such as conformity and consensus reaching, mirroring foundational social psychology theories. In conclusion, we integrate insights from social psychology to contextualize the collaboration of LLM agents, inspiring further investigations into the collaboration mechanism for LLMs. We commit to sharing our code and datasets, hoping to catalyze further research in this promising avenue.

## introduction with the prevalence of llms @xcite @xcite integral to daily social collaboration, there is a growing imperative to cultivate ai systems embodied with social intelligence. this also resonates with the society of mind (som) concept @xcite @xcite , which suggests that intelligence emerges when computational modules interact with each other, achieving collective objectives that surpass the capabilities of individual modules @xcite . previous studies @xcite @xcite @xcite @xcite have delved into strategies where llm instances, termed agents @xcite @xcite @xcite , cooperate synergistically (e.g., debate and reflect) to accomplish tasks @xcite @xcite @xcite . as illustrated in figure 1 , such collaboration fosters divergent thinking processes in llms, making them particularly effective for tasks demanding profound reflection. intuitively, reflecting on human societies @xcite @xcite , where a myriad of individuals with distinct goals and roles coexist, the som framework champions harmonious interactions @xcite . intriguingly, despite the fusion of social psychology @xcite @xcite in som with human group dynamics @xcite , which illuminates psychological patterns within social groups, its interpretation in the realm of llms is relatively uncharted @xcite . besides, our grasp of how social behaviors influence llms is still in its nascent stages. to address these issues, we delve into the machine society, probing the extent and ways that llms manifest social intelligence and collaboration capabilities @xcite . utilizing powerful llms like gpt-3.5 (openai, 2022), we build a test-bed across three datasets: math @xcite , mmlu @xcite and chess move validity @xcite . our approach incorporates four societies character-given the chess game "g2g3 f7f5 e2e3 f5f4 g1f3 d7d5 b1a3 d8d7 f3g1 d7c6 d1", give one valid destination square for the chess piece at "d1". three agents gave different and incorrect answers initially. after reflection, agent 1 and 3 converged on the same answer. the third time they debated, all agents agreed on 'e2' as the correct answer. answer e2 is correct, as shown in the figure above. after considering the suggestions from other agents and reviewing my historical answers, i acknowledge the correct destination ... is indeed e2. ... moving the queen from "d1" to "e2" allows for development, controlling the central square, and potentially supporting future positions and coordinating with other pieces. after carefully reviewing ... justifications provided by other agents and comparing them with my previous answers, i confirm that my initial response was incorrect. ... , the valid destination ... is indeed e2. this move allows the white queen to develop, control the central square, and potentially support other pieces. ized by two individual traits (easy-going and overconfident) with three agents: totally/mostly easygoing; totally/mostly overconfident. these traits are employed to emulate nuanced human society dynamics @xcite @xcite . moreover, we delve into two distinct thinking patterns under multi-round collaboration: debate @xcite @xcite @xcite and reflection @xcite @xcite . with the permutation of thinking patterns, we can constitute various collaborative strategies. to this end, we implement two patterns of collaboration in the collaborative strategies: (i) all agents adopt the same thinking pattern at each round; (ii) one agents adopts the different thinking patterns from others at each round. we then execute these multi-round collaborative strategies within different societies. through our empirical analysis, we primarily discern the following insights (further takeaways are in §3, §4 & appendix a): concretely, our findings challenge the dominant belief that mere scale is the key. we posit that smallgroup collaboration with rational strategies might present a more efficacious approach to utilizing llms. in wrapping up, we encapsulate the core contributions of this research as follows: our goal is to identify how and to what extent llms manifest social intelligence through collaboration. to enrich our inquiry, we draw upon theories from social psychology, contextualizing the behaviors and tendencies displayed by llm agents. based on responses from agent 1 and agent 3, i think this problem ...... based on responses from agent 1 and agent 2, i think this problem ...... after reflecting on my previous answer, i believe that ...... based on responses from agent 2 and agent 3, i think this problem ...... debate agent create a society round 1 round 2 round 3 ...... round n handle a task debate reflection debate reflection reflection reflection one of 8 possibilities all 8 possibilities -debate -reflection p 0 p 0 p 1 p 0 p 0 p 0 p 0 p 1 p 0 p 1 p 0 p 0 p 1 p 1 p 1 p 0 p 0 p 1 p 0 p 1 p 1 p 1 p 0 p 1 p 1 p 1 debate reflection debate i am an expert skilled in ...... and are objective ......, and i can be persuaded if other agent's answers make sense ...... i am an expert skilled in ...... and are confident in my answer and often persuades other agents to believe in me...... ## explore collaboration mechanisms with multiple llm agents in this section, we formulate and simulate the collaboration mechanisms explored within the machine society, drawing upon relevant concepts. we also illustrate the society settings in figure 2 . ## analysis of machine social collaboration our experiments are primarily driven by the following research queries: (rq1) how does problemsolving effectiveness vary under different collaborative strategies across diverse societies? (rq2) how to configure the machine society variables for optimal performance? (rq3) how does machine social collaboration mimic the human society? ## phenomena of conformity and consensus reaching to address rq3, we embark on further analysis from a social psychology view @xcite @xcite , to discern alignment between machine society collaboration and human societal dynamics @xcite . our findings indicate that machine society collaboration echoes specific human societal phenomena or theories, such as conformity @xcite @xcite and consensus reaching @xcite @xcite (more analysis are in appendix g.1). we also analyze group dynamics @xcite @xcite @xcite in multiagent collaboration at appendix g.2 as page limits. we embark on a detailed analysis, to discern the conformity and consensus-reaching phenomena in collaboration. for instance, as depicted in figure 8 (a) at appendix d.3, an agent initially responds correctly to a question. however, swayed by the misguided answers and explanations from the other two agents, eventually, the three agents conform to an incorrect answer. this phenomenon we also conduct a quantitative analysis of the prevalence of conformity and consensus-reaching phenomena. we analyze answer correctness changing at each round of collaboration in the situation of conformity, shown in figure 6 on @xcite @xcite on other llms at appendix h. we also present the ratio of consensus reaching at each round in figure 7 on chatgpt and figure 29, 38, 52, 66 on other llms at appendix h. we summarize the following obeservations: ## conclusion and future work this study has highlighted the potential of collaboration mechanisms with llms. our findings reveal the impressive collaboration capabilities of llm agents, with different individual traits, thinking patterns, and collaborative strategies. the emergence of human-like behaviors in these agents, resonating with social psychology theories, further emphasizes the potential of human-ai interaction. moving forward, a deeper exploration into the multi-agent society is warranted, focusing on collaboration behavior refinement; integrating further insights from social psychology could also guide the development of socially aware nlp systems.