Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2024)

Abstract: Human intelligence thrives on cognitive synergy, where collaboration among different minds yield superior outcomes compared to isolated individuals. In this work, we propose Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. A cognitive synergist is an intelligent agent that collaboratively combines multiple minds’ strengths and knowledge to enhance problem-solving in complex tasks. By dynamically identifying and simulating different personas based on task inputs, SPP unleashes the potential of cognitive synergy in LLMs. Our in-depth analysis shows that assigning multiple fine-grained personas in LLMs improves problem-solving abilities compared to using a single or fixed number of personas. We evaluate SPP on three challenging tasks: Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, encompassing both knowledge-intensive and reasoning-intensive types. Unlike previous works, such as Chain-of-Thought, that solely enhance the reasoning abilities in LLMs, experimental results demonstrate that SPP effectively reduces factual hallucination, and maintains strong reasoning capabilities. Additionally, comparative experiments show that cognitive synergy only emerges in GPT-4 and does not appear in less capable models, such as GPT-3.5-turbo and Llama2-13b-chat, which draws an interesting analogy to human development. Code, data, and prompts can be found at: https://github.com/MikeWangWZHL/Solo-Performance-Prompting.git.

## introduction although large language models (llms) have demonstrated impressive performance as general * work was done when interning at microsoft research asia. input output ai assistant input output ai assistant thoughts input output ai assistant domain experts / audiences personas a single llm ... task-solving agents, they still encounter challenges @xcite @xcite in various knowledgeintensive and reasoning-intensive tasks due to factual hallucination @xcite and a lack of slow-thinking @xcite capabilities. unlike humans, who can leverage the power of collaboration and information integration among different cognitive processes and individuals (referred to as cognitive synergy @xcite @xcite ), current llms are akin to "jack-of-alltrades" with a vast mixture of knowledge and characteristics. recent advancements, such as chainof-thought (cot) prompting @xcite and self-refinement @xcite , have successfully enhanced the reasoning abilities of llms by simulating slow-thinking through the generation of intermediate steps or iterative revision. however, factual hallucination remains a major challenge for llms on knowledge-intensive tasks. a cognitive synergist is an intelligent agent write a short, one-paragraph background story of an npc for the next legend of zelda game. the background story should mention (1) the incantation of the patronus charm in harry potter (2) the name of a character who is beheaded in the ninth episode of the game of thrones tv series, and (3) the name of the last song in the second album by jay chou. in the mystical land of hyrule, there lived a wise and powerful sorcerer named eddard. eddard was known far and wide for his mastery of ancient spells, including the legendary "expecto patronum" the melody resonated with his soul, unlocking a hidden power within him. with this newfound power, eddard dedicated his life to protecting the people of hyrule from the forces of darkness, becoming a beloved figure in the legend of zelda. ## solo performance prompting to unleash the power of synergizing different personas to tackle complex problems, we propose solo performance prompting (spp) which instructs a llm to perform the following the procedure for general task-solving: (1) persona identification: identify multiple participants with special personas (including a leader persona: ai assistant) that are essential for solving the particular task. given an input sequence x and a model m, let a prompt (including demonstration examples) prepended to the input to be p and the final output to be y. denote an intermediate generation before generating the final y as z. under this formulation, standard prompting and chain-of-thought (cot) prompting can be described as: cot prompting: solo performance prompting: ## experiments to explore the effectiveness of solo performance prompting (spp), we adopt an evaluation methodology similar to that of previous work @xcite . we carefully design new tasks and select tasks from existing benchmarks @xcite that are challenging even for the most capable llms @xcite . the evaluation aims to cover diverse types of tasks encompassing both knowledge-intensive and reasoning-intensive domains. tasks. we invent the trivia creative writing task ( §3.1), which requires the model to internally acquire and integrate diverse information from various fields. we observe that even gpt-4 (openai, 2023b) frequently exhibit hallucination and factuality errors in the trivia creative writing task. we also propose the codenames collaborative task ( §3.2), an extension of the codenames task from the bigbench @xcite ) that features a two-role collaboration setup. codenames collaborative demands creative reasoning across a broad range of related knowledge and challenges the model's theory of mind skills. lastly, we include a challenging pure-reasoning task, logic grid puzzle ( §3.3), from the bigbench @xcite which necessitates complex multi-step reasoning. baselines. we compare our approach with standard prompting, chain-of-thought (cot) prompting methods (outlined in §2) and self-refine @xcite . for cot, a similar prompt design to @xcite is employed, where the model is prompted to generate a plan or a series of steps before producing the final output. for self-refine, we follow @xcite to design feedback and refine prompts. we perform one self-refine iteration which requires three times more inferences than spp. full prompts for the methods can be found in appendix a.2. models. the default model we use is gpt-4 (openai, 2023b). detailed inference configurations, api versions, and full results can be found in appendices c and f. in §3.4, we further investigate the prevalence of cognitive synergy in llms with different scales and capabilities, including gpt-3.5turbo (openai, 2023a) and llama2-13b-chat (touvron et al., 2023). ## analysis spp effectively improves both knowledge and reasoning abilities in llms. as demonstrated by the results in §3, solo performance prompting (spp) not only brings significant improvements to knowledge-intensive tasks such as trivia creative writing and codenames collaborative without relying on external knowledge bases, but also achieves strong performance on reasoning-intensive tasks like logic grid puzzle. to our knowledge, spp is the first zero-shot prompting method that can enhance both knowledge and reasoning abilities on gpt-4. llms can effectively identify useful personas in a zero-shot manner. we are interested in investigating whether the identified personas are highly relevant to the tasks. we visualize the personas automatically identified by spp using a word cloud for each task in figure 7a , where a larger font indicates a higher frequency. the key observations include: (1) the identified personas are closely correlated with the particular task. for example, in logic grid puzzle, even though "logic puzzle" is not mentioned in the input, the llm frequently identifies the persona "logic puzzle expert." (2) on knowledge-intensive tasks, such as trivia creative writing, spp identifies more diverse and specific personas, while on reasoning-intensive tasks, such as logic grid puzzle, the personas are more homogeneous. we further investigate whether a detailed profile for each persona is needed for eliciting domain knowledge, as suggested by @xcite . to this end, we design a variant of spp, spp-profile, which involves generating profiles for each persona during the persona identification phase. the results in figure 7b show that spp-profile does not outperform spp. this suggests that a fine-grained persona name without a detailed description may already be sufficient for eliciting certain domain knowledge.