Optima: Optimizing Effectiveness and Efficiency for LLM -Based Multi-Agent System (2025)

Abstract: Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through training. Optima employs an iterative generate, rank, select, and train paradigm with a reward function balancing task performance, token efficiency, and communication readability. We explore various algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs. We integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, treating conversation turns as tree nodes to explore diverse interaction paths. Evaluated on common multi-agent tasks, including information-asymmetric question answering and complex reasoning, Optimashows consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B / 3.2 3B, achieving up to 2.8x performance gain with less than 10% tokens on tasks requiring heavy information exchange. Moreover, Optima’s efficiency gains enable more effective compute utilization during inference, leading to improved inference-time scaling laws. By addressing fundamental challenges in LLM-based MAS, Optima shows the potential towards scalable, efficient, and effective MAS.

## introduction large language models (llms) have emerged as powerful tools for a wide range of tasks, from * equal contribution. ## conclusion we introduce optima, a novel framework for training llm-based mas that significantly enhances communication efficiency and task performance. experiments show optima's consistent superiority over single-agent and multi-agent baselines. we introduce key innovations such as iterative training, a balanced reward function, and mcts-inspired data generation. crucially, optima effectively improves inference-time scaling and transfers effectively to ood tasks, underscoring the importance of efficient communication in mas and llm systems. while optima marks a major step forward in multi-agent llm training, further exploration into its scalability to larger models and more complex scenarios is a promising direction for future research.