Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

An implementation of single LLM moa is available in optillm - https://github.com/codelion/optillm/blob/main/optillm/moa.py

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning (2025)
• Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks (2024)
• KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (2024)
• LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion (2025)
• Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (2025)
• Hint Marginalization for Improved Reasoning in Large Language Models (2024)
• RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Summary of "Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?"

Objective

This paper critically examines the effectiveness of the Mixture-of-Agents (MoA) approach, which ensembles output from multiple Large Language Models (LLMs). It introduces Self-MoA, an alternative method aggregating outputs from only a single top-performing model rather than multiple diverse models. The research challenges the assumption that diversity in proposers necessarily leads to better performance.

Methods

1. Comparing Mixed-MoA vs. Self-MoA:

   • MoA traditionally combines responses from multiple LLMs and aggregates them into a final output.
   • Self-MoA generates multiple outputs from the same high-performing LLM and aggregates those instead.

2. Performance Benchmarks:

   • Evaluated on AlpacaEval 2.0, MMLU, CRUX, and MATH datasets.
   • Compared quality vs. diversity trade-offs across 200+ experiments.
   • Introduced Self-MoA-Seq, a sequential version that scales up aggregation without exceeding context length constraints.

Findings

• Self-MoA consistently outperforms Mixed-MoA.

  • Achieved 6.6% better performance than Mixed-MoA on AlpacaEval 2.0.
  • Improved 3.8% on average across multiple benchmarks.

• Quality trumps diversity.

  • Mixing LLMs often reduces the average quality due to the inclusion of lower-performing models.
  • Intra-model diversity (sampling multiple outputs from the same model) is more beneficial than inter-model diversity.

• Cross-model diversity only helps in particular cases.

  • When models perform similarly, diversity can marginally improve results.
  • However, the gain is small (~0.3%), making Self-MoA the more reliable approach.

• Scaling inference compute with Self-MoA-Seq:

  • Instead of aggregating all outputs simultaneously, Self-MoA-Seq uses a sliding window method to synthesize results gradually.
  • This allows models with shorter context lengths to still benefit from large-scale ensembling.

Why Is This Paper Unique?

This research challenges a fundamental assumption in LLM ensembling—that more diverse models equal better performance. Unlike previous work emphasizing cross-model diversity, this paper provides empirical evidence that quality consistency within a single model is more effective.

Key Contributions That Make It Novel:

1. Redefines the Quality-Diversity Trade-off:

   • Past research assumes diverse models yield better outputs.
   • This paper quantifies how diversity can reduce final quality.

2. Introduces Self-MoA as a More Efficient Alternative:

   • Instead of adding new models, it reuses the best model's outputs, making it computationally efficient.
   • Lowers deployment costs by avoiding unnecessary model inference.

3. Brings a New Perspective to Scaling Compute at Inference Time:

   • Self-MoA-Seq allows ensembling without increasing memory footprint.
   • This has implications for real-world applications, where context length limits models.

Implications for AGI Development

The findings from this paper could fundamentally shape the future of AI reasoning and AGI (Artificial General Intelligence) in several ways:

1. Efficiency Over Model Expansion:

   • Instead of training bigger and more diverse models, AGI could refine knowledge within a single high-quality model.
   • This means AGI could scale compute at inference time rather than training time, making future AI systems more energy-efficient.

2. Better Decision-Making in Autonomous Agents:

   • If AGI is structured as an agent ensemble, Self-MoA suggests that multiple outputs from a strong model are preferable over diverse weaker agents.
   • This aligns with human decision-making, where experience and refined judgment matter more than sheer variety.

3. Solves the "Alignment vs. Performance" Trade-off:

   • Many AGI models face a challenge: should we prioritize alignment (safety) or performance (creativity)?
   • This research shows that alignment can be preserved without sacrificing performance, as aggregating outputs from a single model maintains quality consistency.

4. Could Enable More Reliable Self-Improving AGI:

   • If AGI is designed to learn from its past outputs, Self-MoA could enable self-critique and iteration without external feedback.
   • This removes dependence on external evaluators, making AGI more autonomous in reasoning and self-correction.

Would Self-MoA Make AGI a "Good Thing"?

Pros:

✅ More Efficient – Reduces computational waste in AI training and inference.
✅ More Reliable – Avoids the risk of bad models introducing errors in an ensemble.
✅ Scales Safely – Prevents degradation in output quality when increasing compute.
✅ Supports Alignment – Keeps control mechanisms intact without sacrificing creativity.

Cons / Risks:

⚠️ Risk of Overfitting to One Model’s Biases – If the "top model" has a flaw, Self-MoA could amplify it.
⚠️ Less Diversity in Thought – Some problems may require multiple reasoning pathways.
⚠️ Reduces Model Competition – If AGI follows this approach, fewer LLMs may be developed, consolidating power in a few key models.

Conclusion: The Future of AI with Self-MoA

This research challenges long-held assumptions about diversity in AI ensembles and introduces a more efficient, scalable, and controlled approach to AI decision-making. As AGI advances, Self-MoA could enable models to refine their reasoning processes efficiently, making future AI systems brighter, safer, and more practical.

The next steps could involve:
🔹 Testing Self-MoA in autonomous agent settings (e.g., AI assistants, robotics).
🔹 Applying Self-MoA to real-world AGI decision-making tasks.
🔹 Ensuring safeguards are in place to prevent over-amplification of model biases.

Would you like a deeper dive into specific applications of Self-MoA in AGI research? 🚀