Scalability of LLM -Based Multi-Agent Systems for Scientific Code Generation: A Preliminary Study (2025)

Abstract: Recent studies indicate that LLM-based Multi-Agent Systems (MAS) encounter scalability challenges in complex mathematical problem-solving or coding tasks, exhibiting issues such as inconsistent role adherence and ineffective inter-agent communication. Moreover, the performance advantages of LLM-based MAS over a single agent employing test-time scaling methods (e.g., majority voting) remain marginal. This raises a critical question: Can LLM-based MAS scale effectively to achieve performance comparable to standalone LLMs or even Large Reasoning Models (LRMs) under optimal test-time compute?In this paper, we conduct a preliminary investigation into the scalability of LLM-based MAS for scientific code generation. We propose a simple yet scalable two-player framework based on iterative critic-in-the-loop refinement. Our experiments demonstrate that a minimalist actor-critic framework based on DeepSeek-V3 can outperform DeepSeek-R1 under equivalent computational budgets. Surprisingly, more complex frameworks fail to yield significant gains. These findings corroborate recent insights into multi-agent system limitations and highlight the importance of scalable workflows for advancing scientific code generation.

## introduction in recent years, llm-based multi-agent systems (mas) @xcite have demonstrated significant potential in complex problem-solving and system coding tasks @xcite @xcite @xcite . in such systems, each agent is assigned a specific role, working collaboratively to achieve predefined objectives, which mirrors human teamwork in real-world scenarios. ## conclusion we propose a lightweight generator-critic framework that enhances llm-based multi-agent systems for scientific code generation. our approach outperforms standalone large reasoning models while reducing computational costs, demonstrating that iterative critique inherently serves as an efficient test-time scaling strategy. experiments reveal diminishing returns with complex multi-agent setups, suggesting simplicity is key for scalability. these findings offer practical guidelines for deploying efficient llm-based systems in resourceconstrained scenarios.