RM o A : Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation (2025)

Abstract: Although multi-agent systems based on large language models show strong capabilities on multiple tasks, they are still limited by high computational overhead, information loss, and robustness. Inspired by ResNet’s residual learning, we propose Residual Mixture-of-Agents (RMoA), integrating residual connections to optimize efficiency and reliability. To maximize information utilization from model responses while minimizing computational costs, we innovatively design an embedding-based diversity selection mechanism that greedily selects responses via vector similarity. Furthermore, to mitigate iterative information degradation, we introduce a Residual Extraction Agent to preserve cross-layer incremental information by capturing inter-layer response differences, coupled with a Residual Aggregation Agent for hierarchical information integration. Additionally, we propose an adaptive termination mechanism that dynamically halts processing based on residual convergence, further improving inference efficiency. RMoA achieves state-of-the-art performance on the benchmarks of across alignment, mathematical reasoning, code generation, and multitasking understanding, while significantly reducing computational overhead. Code is available at https://github.com/mindhunter01/RMoA.

## introduction large language models (llms) @xcite @xcite have achieved significant advancements in extensive natural language processing tasks @xcite . recently, researchers have proposed several policy-based methods that enhance model performance without model scaling. notable approaches include chain-of-thought @xcite , * corresponding author which enhances multi-step reasoning; retrieval-augmented generation (rag) @xcite , which leverages external information sources; and multi-agent systems (mas) @xcite . among these innovations, mas has garnered significant attention due to exceptional flexibility and broad compatibility. recently, iterative collaboration strategies have been shown to enhance the capabilities of mas. @xcite proposed the mixtureof-agents (moa) architecture. this architecture leverages a hierarchical processor design that enables multiple layers of agents to process queries in parallel, significantly improving computational efficiency. then, moa employs an aggregator to integrate the outputs from these agents, generating the final response. subsequently, sparse mixture-of-agents (smoa) @xcite were introduced to reduce the large number of tokens involved in parallel queries under moa, thus lowering inference costs. these approaches incorporate a judge model @xcite to evaluate the quality of responses generated by different models, thereby reducing the number of tokens processed by the aggregator. while this strategy somewhat alleviates computational overhead, it still faces challenges in ensuring the robustness of quality differentiation among responses @xcite . moreover, as the number of processing layers increases, moa may suffer from the loss of critical information during aggregation @xcite , leading to inaccurate responses and ultimately compromising the overall stability and reliability . to address these challenges, we propose rmoa, an improved moa-based architecture inspired by residual connections. unlike existing approaches, we do not employ a judge model to select the optimal response. instead, we introduce an embedding model to convert responses into vector representations and compute their similarities. a greedy strategy is then applied to select k responses with the highest diversity, ensuring greater information heterogeneity. additionally, we design a residual extraction agent to capture differences between responses at successive layers. these residuals, along with the selected diverse responses, are fed into the aggregator, preserving incremental information and mitigating the loss of key content during deep aggregation. to conserve computational resources, we incorporate an adaptive termination mechanism, which dynamically determines when to halt processing based on response variations between iterations, thereby reducing unnecessary overhead. furthermore, to foster diverse and creative reasoning, each agent is assigned a distinct role-playing persona. to comprehensively evaluate the effectiveness of our approach, we conduct extensive experiments on alignment, mathematics, code generation, and multi-task understanding. experimental results demonstrate that rmoa achieves state-of-the-art performances with lower computational costs. additionally, a series of ablation studies validate the effectiveness of each component in rmoa. finally, we investigate rmoa's performance under increased computational budgets, showing that for models with strong general capabilities, deeper architectures tend to yield improved performance across most datasets. overall, our contributions consist of three parts. ## methodology this section begins with an overview of moa, followed by a comprehensive analysis of rmoa's core components: greedy diversity embedding selection, residual agent, and adaptive termination mechanisms, as depicted in figure 2 . initially, the greedy diversity embedding selection filters out diverse and representative responses, ensuring varied inputs for further processing. next, the residual agent uses these selected responses to pinpoint key differences between dialogue rounds, integrating them into the reference material to reduce information loss. finally, the adaptive termination mechanism continuously monitors the process in real-time, deciding whether to continue based on residual detection outcomes, thus avoiding unnecessary iterations and potential hallucinations. ## analysis in this section, we conduct comprehensive experiments to thoroughly investigate the mechanisms of rmoa. the experiments are primarily divided into ablation studies, cost analysis, and case studies. ## conclusion this paper introduces the rmoa framewrok, which ultilizes iterative collaboration to improve mas capabilities. we propose greedy differential embedding selection, residual agent, and adaptive termination mechanism to achieve diversity maximization and residual compensation. the proposed rmoa alleviates the problems of high computational overhead, information loss and robustness of traditaional moa architecture. we conduct extensive evaluations across a variety of tasks and explore the potential of rmoa through ablation studies and cost analysis.