MMA : Cross-Domain Knowledge Integration via Mixture of Multi-Domain Agents (2025)

Abstract: Rather than merely to retain previously acquired generalization, achieving synergistic improvements between generalization and domain specialization in foundation models remains a significant challenge in both pre-training and post-training. As an alternative, we propose a test-time cross-domain knowledge integration method, Mixture of Multi-domain Agents (MMA), which dynamically combines the outputs of general-purpose and domain-specific models to enhance their performance on complex, domain‐specific tasks. MMA formulates the integration process as a search problem, using Monte Carlo Tree Search (MCTS) to find the path that optimally harmonizes the respective strengths of different models in generalization and domain-specific knowledge. In addition, We design specific action spaces to control the knowledge integration between multiple models, and cross-inspection reward is introduced to fairly score strategies in different domains. Experiments in diverse domains show that MMA can effectively combine the strengths of different models to enhance their performance. For instance, in legal tests, the average performance of all tasks increased from 42.57% to 53.68%. In financial tests, it improved from 56.01% to 62.68%.

## introduction large language models (llms) demonstrate impressive capabilities in handling complex tasks that require both specific knowledge and sophisticated reasoning. however, these models tend to entangle two inherently distinct actions: memory recall and reasoning @xcite . such entanglement can lead the issue such as catastrophic forgetting @xcite during continual pre-training or domain-specific fine-tuning. as illustrated in figure 1 , we observe a clear performance divergence across task types: on reasoningintensive tasks, the general model qwen2.5-7b outperforms the domain-specific model wisdom by 24.4 points, whereas on knowledge-intensive tasks, wisdom exceeds qwen2.5-7b by 8.27 points. this phenomenon reveals a strong complementarity between general and domain-specific models, motivating the development of dynamic integration frameworks that unify the strengths of both reasoning and memory recall. to address the aforementioned challenges, we explore test-time cross-domain integration strategies, which offer two primary advantages: (1) low dependence on high-quality domain-specific data; in this paper, we propose mixture multi-domain agents (mma), which decomposes problem solving into a process of collaboration-verification among multiple models for cross-domain knowledge integration. specifically, unlike previous monte carlo tree search (mcts) research @xcite @xcite that mainly focuses on the problem formulation level, mma designs an mcts action-space strategy for model interaction and knowledge integration. this action space defines operations such as model invocation, knowledge extraction, inspection and review, enabling the dynamic exploration of diverse integration strategies. furthermore, to ensure method robustness in cross-domain knowledge integration scenarios, we cannot rely on any single model to serve as a verification model across all domains; for example, even broadly capable models like gpt-4o cannot cover every specialized field.mma addresses this challenge through cross-inspection reward. specifically, for nodes that require evaluation, mma uses models that were not involved in the final answer generation, along with part of that node's reasoning trajectory for subsequent answer generation. the consistency of the two answers for the node is then used to provide unsupervised feedback. extensive experiments using general models in conjunction with different domain-specific models separately on different domain reasoning tasks have demonstrated the effectiveness of mma. the results show that cross-domain knowledge integration can effectively combine the strengths of multiple models at different dimensions, achieving a synergistic optimization of specialized knowledge and generalization. in legal tasks, our method improves the performance of the legal domain model from 42.57% to 53.68%. in finance, it improves the performance of the financial domain model from 56.01% to 62.68%. in summary, our contributions are as follows: multi-agent collaboration. unlike deepening the capabilities of a single model to enable it to plan and reason in a human-like manner @xcite @xcite , multiagent @xcite focuses on the study of interactions between agents as a way of combining capabilities between agents. some studies @xcite @xcite assign different roles to llms, allowing them to interact and reason about the problem through collaboration or debate. multi-agent reinforcement @xcite learning enables collaboration by allowing multiple agents to interact within a shared environment based on reward signals. recently, mixture-of-agents @xcite combined the outputs of multiple llms to obtain a higher-quality final result. later, @xcite proposed self-moa, based on whether mixing different llms is truly beneficial. this method uses only the best-performing llm to generate multiple outputs through multiple samplings, and then aggregates these outputs into a high-quality final result using an aggregator. although these methods improve model performance through multiagents, they have only explored their effectiveness for signal-domain tasks. in contrast, mma effectively achieves cross-domain knowledge integration, realizing a synergistic optimization of specialized knowledge and generalization, rather than merely facilitating collaboration among models. ## preliminary as a representative approach to knowledge integration, mixture of agents (moa) aggregates the outputs of multiple models with the assumption that referencing the outputs of multiple models can generate higher quality responses. however, moa may neglects for the heterogeneity in domain expertise across different agents, which may result in knowledge conflicts and suboptimal performance. as shown in figure 2 , we make the following key observations: (1) moa outperforms the general model llama3-8b-instruct but underperforms the domain-specific model disc-lawllm, suggesting that traditional ensemble-based integration methods are not well-suited for cross-domain scenarios. ## mixture of multi-domain agents due to the confusion caused by directly mixing multi-model answers, we attempted to explore multiple integration strategies to avoid introducing erroneous knowledge. we enable models to interact and verify with each other, i.e., models can generate answers using their own knowledge or knowledge provided by other models, and then modify the provided knowledge as well as reward scoring of responses. as shown in figure 3 , we establish a framework that enables dynamic exploration of integration strategies. ## conclusion in this work, we propose mma, a method that achieve cross-domain knowledge integration via mixture of multi-domain agents. by dynamically exploring integration strategies, we find optimal harmonizes the respective strengths of different models in generalization and domain-specific knowledge. experiments across multiple domains and tests demonstrate the effectiveness of the method. extensive ablation studies and analyses further validate the efficacy of the components within the approach.