M o FE : Mixture of Frozen Experts Architecture (2025)

Abstract: We propose the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning (PEFT) and the Mixture of Experts (MoE) architecture to enhance both training efficiency and model scalability. By freezing the Feed Forward Network (FFN) layers within the MoE framework, MoFE significantly reduces the number of trainable parameters, improving training efficiency while still allowing for effective knowledge transfer from the expert models. This facilitates the creation of models proficient in multiple domains. We conduct experiments to evaluate the trade-offs between performance and efficiency, compare MoFE with other PEFT methodologies, assess the impact of domain expertise in the constituent models, and determine the optimal training strategy. The results show that, although there may be some trade-offs in performance, the efficiency gains are substantial, making MoFE a reasonable solution for real-world, resource-constrained environments.

## introduction large language models (llms) showcase significant advancements in natural language understanding and generation. llms are characterized by their immense size, often consisting of at least one billion parameters. the substantial size of llms is understandable given the scaling law suggested by @xcite , which indicates that performance on the cross-entropy loss improves predictably with increased model size, data, and computational power. however, their immense size poses a resource challenge, requiring substantial computational memory and vast amounts of data, making development and deployment difficult to afford. to address this, developing efficient llms that maintain high performance has become crucial. efforts include (1) efficient training methodologies like parameter-efficient fine-tuning (peft) and (2) efficient model scaling methodologies such as the mixture of experts (moe) architecture. in this research, we propose the mixture of frozen experts (mofe) architecture, combining both approaches for a more efficient and affordable model. mofe leverages moe's benefits while reducing computational requirements through freezing the ffn blocks. our experiments demonstrate that, despite a trade-off between performance and efficiency compared to full fine-tuning, mofe outperforms other peft methods, requiring the least training time while achieving the highest performance. additionally, mofe shows effective knowledge transfer from its constituent models, highlighting the potential for using pre-existing domain expertise models with minimal further training. ## related work primary strategies for efficient model training are peft and quantization. peft includes techniques like prompt-tuning @xcite , adapters @xcite , lora @xcite , and dora @xcite , all designed to reduce computational demands. quantization @xcite maps model weights to lower-precision formats for efficiency, and @xcite introduced qlora, combining lora with quantization. the mixture of experts (moe) architecture @xcite @xcite is another efficient scaling method that gained attention with mixtral 8x7b @xcite , which integrates eight mistral 7b models @xcite and outperforms llama-2 70b @xcite despite being smaller. following mixtral 8x7b, other moe-based models, including openmoe @xcite , jamba @xcite , bimedix @xcite , and biomistral @xcite , have been developed. ## conclusion given the enormous computational costs of training and serving llms, we propose mofe as an efficient model training and scaling strategy. while there is a trade-off between efficiency and performance, mofe significantly reduces the size of trainable parameters and training time, demonstrating superiority over other peft methods in both training time and task performance. furthermore, the transfer of domain expertise from the constituent models enables the creation of multi-domain proficient models by leveraging existing domain experts. we believe mofe presents a viable option for resource-constrained environments in real-world scenarios. to examine how performance shifts with varying numbers of frozen ffn blocks, we use a mediumsized model with four expert models. tinyllama serves as the base and expert models, as in previous experiments. five versions of the model are constructed: one with all expert models frozen, one with three frozen, one with two frozen, one with one frozen, and one with none frozen. each model is instruction-tuned on the medmcqa training dataset and evaluated on its test set. as shown in