Simulating Expert Discussions with Multi-agent for Enhanced Scientific Problem Solving (2024)

Abstract: Large Language Models (LLMs) have shown remarkable potential across various domains, yet their application in addressing complex scientific problems remains a formidable challenge. This paper presents a novel methodology to augment the problem-solving capabilities of LLMs by assigning them roles as domain-specific experts. By simulating a panel of experts, each LLM is tasked with delivering professional and cautious responses to scientific inquiries. Our approach involves querying multiple LLMs and assessing the consistency of their responses. High agreement among the LLMs suggests greater confidence in the proposed solution, whereas discrepancies prompt a collaborative discussion among the LLMs to reach a consensus. This method emulates real-world scientific problem-solving processes, fostering a more reliable and robust mechanism for LLMs to tackle scientific questions. Our experimental results show that assigning roles to multiple LLMs as domain-specific experts significantly improves their accuracy and reliability in solving scientific problems. This framework has the potential to advance the application of AI in scientific research, enhancing its effectiveness and trustworthiness.

## introduction large language models (llms) have achieved remarkable success in a wide range of natural language processing tasks, including text generation @xcite , machine translation @xcite , and text summarization @xcite . despite their versatility and strong performance across various domains, the application of llms to solving complex scientific problems has remained a significant challenge. the primary obstacle lies not in the absence of domain-specific knowledge within these models, but rather in their * corresponding author limited ability to effectively harness this knowledge when confronted with intricate scientific problems that demand expert-level understanding and reasoning @xcite . the application of llms to scientific problemsolving presents a unique challenge due to the stringent requirements for precision and reliability in research. minor inaccuracies can have far-reaching consequences, undermining the validity and trustworthiness of results. while llms possess extensive knowledge, their current architectures often struggle to consistently apply this knowledge to meet the rigorous demands of scientific inquiry. this limitation underscores the need for innovative approaches to enhance the problem-solving capabilities of llms in specialized domains. improving the performance of llms in accurately and reliably solving complex scientific problems could significantly advance their utility in research settings and unlock new potentials for artificial intelligence in science. in this study, we introduce a novel methodology called simulating expert discussions with multiagent (sedm) that enhances the problem-solving capabilities of llms by assigning them roles as domain-specific experts, as illustrated in figure 1 . this approach involves simulating a panel of experts, where each llm is tasked with providing professional and cautious responses to scientific inquiries. by querying multiple llms and evaluating the consistency of their responses, we can gauge the confidence in the proposed solutions. high agreement among the llms indicates greater reliability, while discrepancies trigger a collaborative discussion among the models to reach a consensus. this method mirrors real-world scientific problemsolving processes, fostering a more dependable mechanism for llms to address scientific questions. we evaluate the performance of sedm on a range of problems across various scientific domains, including physics, chemistry, and mathematics. we use accuracy as the evaluation metric and compare sedm with baseline methods such as direct llm usage and few-shot learning. in summary, our contributions are: ## related work large language model reasoning large language models (llms) have demonstrated significant reasoning capabilities, especially when scaled to hundreds of billions of parameters @xcite . various techniques, such as chain-of-thought prompting @xcite @xcite and rationale engineering @xcite , have been proposed to further elicit and enhance the reasoning abilities of llms. however, despite these advancements, llms still struggle with complex reasoning tasks, particularly in the domain of scientific problem-solving @xcite @xcite . llms often struggle to provide reliable and consistent answers to intricate scientific questions @xcite , necessitating the development of novel approaches to improve their reasoning capabilities in this context. multi-model collaboration and role-playing previous studies have explored the benefits of roleplaying in llms, demonstrating that assigning distinct roles can lead to more specialized and accurate outputs @xcite @xcite . additionally, collaborative frameworks where multiple models interact and discuss to reach a consensus have shown promise in improving the robustness of the generated solutions @xcite @xcite @xcite . considering the complexity and rigor of scientific research, more effective methods are needed to stimulate the optimal intelligence of multi-agent systems. large language models in solving scientific problems recent studies have explored the potential of llms in scientific problem-solving, including theorem proof @xcite , hypothesis generation @xcite and scientific discovery @xcite . however, the understanding and reasoning capabilities of llms in fundamental stem(science, technology, engineering, and mathematics) domains remain underexplored @xcite . while llms exhibit impressive performance on high-level scientific tasks, their ability to grasp complex scientific concepts, engage in rigorous logical reasoning, and provide reliable solutions to domain-specific problems is still uncertain. these challenges necessitate a more nuanced approach to harnessing the full potential of llms. ## method: simulating expert discussions with multi-agent we propose a novel approach called simulating expert discussions with multi-agent (sedm) to enhance the scientific problem-solving capabilities of large language models (llms) by simulating expert discussions. the overall framework of our methodology is illustrated in figure 2 . we assign multiple llms with domain-specific expert roles and simulate a panel discussion among these experts on a given scientific problem. by analyzing and evaluating the consistency of the llm experts' responses, we derive reliable solutions. ## conclusion in this paper, we have introduced a novel approach called simulating expert discussions with multi-agent (sedm) to enhance the scientific problem-solving capabilities of llms. by assigning domain-specific expert roles to multiple llms and simulating a panel discussion, our method leverages the collective knowledge and diverse per-spectives of these models to tackle complex scientific problems effectively. the proposed sedm framework represents a significant step forward in harnessing the potential of llms for scientific problem-solving. by simulating expert discussions and leveraging the collective intelligence of multiple models, we can enhance the accuracy, reliability, and robustness of llm-generated solutions. this approach opens up new avenues for applying artificial intelligence in scientific research, enabling more effective and trustworthy problem-solving.