M o DEM : Mixture of Domain Expert Models (2024)

Abstract: We propose a novel approach to enhancing the performance and efficiency of large language models (LLMs) by combining domain prompt routing with domain-specialized models. We introduce a system that utilizes a BERT-based router to direct incoming prompts to the most appropriate domain expert model. These expert models are specifically tuned for domains such as health, mathematics and science. Our research demonstrates that this approach can significantly outperform general-purpose models of comparable size, leading to a superior performance-to-cost ratio across various benchmarks. The implications of this study suggest a potential shift in LLM development and deployment. Rather than focusing solely on creating increasingly large, general-purpose models, the future of AI may lie in developing ecosystems of smaller, highly specialized models coupled with sophisticated routing systems. This approach could lead to more efficient resource utilization, reduced computational costs, and superior overall performance.

## introduction domain-specific models have demonstrated encouraging performance across various fields, often surpassing state-of-the-art general models in their respective domains. in mathematics, models like qwen 2 72b math @xcite and deepseek math @xcite have shown superior performance, while in code generation, specialized models such as code llama and codemistral exhibit significant improvements over comparable general-purpose models @xcite . also, @xcite found that models with fewer than 8 billion parameters, when fine-tuned for specific tasks, can rival or even outperform larger models like gpt-4 in certain domains. despite the promise of domain-specific ai models, a significant gap exists in integrating these specialized models into a comprehensive and versatile framework. the ai community faces a crucial challenge: how to harness the power of domain-specific models across diverse tasks without sacrificing the versatility of general-purpose models. we propose modem (mixture of domain expert models) to address this. at its core, modem consists of two main components: a router and a collection of domain-specific expert models (figure 1 ). the router is designed to classify incoming prompts or queries, determining which domain they best fit into. once classified, the prompt is then directed to the expert model specialized in that particular domain. this approach allows us to harness the superior performance of domain-specific models while maintaining the ability to handle a wide range of tasks. by leveraging smaller specialized models, we achieve state-of-the-art results in various domains without the computational overhead of larger general-purpose models. this approach dramatically lowers inference costs, as only the relevant expert model is activated for each query. the result is a highly efficient system that delivers strong performance while minimizing resource utilization. modem key advantage lies in its ability to train and integrate models separately, offering significant benefits in development efficiency and system capabilities. this approach allows for independent optimization of domain experts, facilitates parallel development, and enables easy integration of new models. the modular design ultimately allows for customization across various industries and applications. to summarise, our main contributions are: ## discussion the results of our study on mixture of experts with domain-specific routing suggest a potential shift in the development and deployment of large language models (llms). this section explores the implications of our findings, their broader impact on the field of artificial intelligence, and potential directions for future research. ## conclusion this study demonstrates the effectiveness of combining domain-specific expert models with routing to enhance the performance and efficiency of large language models. our approach consistently outperformed baseline models across various benchmarks, with strong improvement in specialized domains such as mathematics. both our small and medium modem achieved superior performanceto-cost ratios compared to larger, general-purpose models, highlighting the potential for significant efficiency gains in ai deployment. this research demonstrates a promising new direction in the field of artificial intelligence: the combination of domain-specific models with intelligent routing systems. the study's findings suggest that this approach can lead to significant improvements in both performance and cost-efficiency compared to traditional large language models. these findings point to a potential shift in ai development and deployment. rather than focusing solely on creating increasingly large general-purpose models, the future may lie in developing ecosystems of smaller, highly specialized models coupled with sophisticated routing systems. this approach could lead to more efficient resource utilization, reduced computational costs, and superior performance in domain-specific tasks.