LL a MA - M o E : Building Mixture-of-Experts from LL a MA with Continual Pre-Training (2024)

Abstract: Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters.

## introduction large language models (llms) @xcite @xcite have presented remarkable understanding and reasoning capability on a wide range of tasks. nowadays, scaling model size has become the de facto approach to augment performance efficacy further. however, the immense model size is unsustainable due to the computational costs. inspired by this, we focus on sparsely activated mixtureof-expert (moe) models that decouple model size from computation costs. training moe from scratch @xcite @xcite @xcite leads to a significant overall budget. in this work, we reduce the training costs by investigating building moe models from existing dense llms. moreover, starting from the dense model provides flexible structure design choices for moe. in other words, we can place moe in any transformer block. in this paper, we are dedicated to building a full moe model, where each layer contains an moe block. to build strong llama-moe models, we identify two important challenges. first, how to effectively construct experts from the feed-forward networks (ffns) in the existing llms. there are works exploring splitting ffn parameters to construct experts @xcite on t5 or bert model. conversely, @xcite directly copy the ffns to form experts. however, there is no existing work exploring it for decoder-only models. notably, the ffn structure of the previous t5 or bert model is based on the relu function, which shares significantly different characteristics from recent llms based on swiglu function. second, overcoming the performance decrease entailed by changing the network structure from dense to sparse remains challenging. due to the reduction in the amount of activated parameters and the newly introduced gate network for expert routing, we observe a significant performance drop between the llama-moe models and the original dense llama models. to solve the above issues, we comprehensively explore four different methods for expert construction. among them, the non-overlapping randomly splitting method achieves the best performance. subsequently, we continue training the transformed moe models and additional gate networks. in this stage, we also carefully study both dynamic and static data sampling strategies for obtaining the fastest convergence and performance improvement. finally, with a static domain weight proportion corresponding to the activated parameters, the llama-moe models can quickly converge to a decent level with 200b tokens. in summary, our contributions are as follows: ## method as illustrated in figure 1 , we construct llama-moe from llama-2-7b by first partitioning ffns into multiple experts and each token is routed to top-k experts. continual pre-training is subsequently applied to recover the moe model's language ability. the following sections describe the details of our method. ## conclusion in this paper, we build moe from a dense model by partitioning the ffn layers into experts, and continual pre-training. we comprehensively investigate different methods for expert construction and data sampling. empirically, llama-moe-3.5b significantly outperforms open-source models with similar activation parameters. meanwhile, llama-moe-3.0b achieves similar performance with open-llama-3b with less activated parameters. the instructed llama-moe models also present stronger abilities than their counterparts.