Nexus: Adaptive Upcycling to Efficiently Pretrain Mixture of Experts (2025)

Abstract: Frontier language models are increasingly based on the Mixture of Experts (MoE) architecture, boosting the efficiency of training and inference by sparsely activating parameters. Nevertheless, training from scratch on trillions of tokens remains so expensive that most users can only finetune these models. In this work, we combine parameter reuse of dense models for the MoE layers ("*upcycling*”) with a novel, *adaptive* Nexus router that can integrate new experts into an existing trained model without hurting the performance on previous domains. Our router leverages the knowledge of each expert’s training data distribution via domain embeddings to initialize the router, improving specialization and allowing it to adapt faster to new domains than a standard MoE router. Nexus overturns the strict sequential separation between training and finetuning in classical approaches, allowing more powerful improvements to existing models at a later stage through long token-horizon trainings on new pretraining data. Our experiments show that Nexus achieves a relative gain of up to 2.1% over the baseline for initial upcycling, and an 18.8% relative gain for extending the MoE to a new domain with a new expert by using limited finetuning data. This flexibility of Nexus can power an open-source ecosystem where every user continuously assembles their own MoE-mix from a multitude of dense models.

## introduction in an era of bigger and bigger models @xcite @xcite @xcite , there are several key objectives driving stateof-art progress. doing more with less by improving efficiency @xcite remains paramount, but in addition to efficiency, the deployment of these models in the wild means that the ability to adapt to new data @xcite @xcite , and specialization of compute @xcite @xcite @xcite have gained renewed focus. while all these properties are desirable, a formidable challenge is designing architectures that can fulfill all of these requirements. many recent frontier llms have chosen mixture of experts (moe) over a dense architecture as one way of doing more with less, by only activating a specific subset of total parameters for the prediction of each token. however, while this sparse activation greatly improves efficiency, experts do not appear to exhibit dedicated expertise or meaningful specialization @xcite @xcite . furthermore, moes tend to suffer from severe training instabilities @xcite . recent work has attempted to address both the training instabilities and the lack of specialization. these techniques often train completely separate experts and "upcycle" (combine) them into a single unified moe model after dense training @xcite . this further improves the efficiency during training -cross-device communication in large models can take up 20-40% of overall step time @xcite , and training multiple smaller models makes computations more local, requiring less communication @xcite . the other major advantage of these approaches is the increase in specialization with separate experts that are trained on specific domains, making them clearly responsible for their human-interpretable subset of the data. however, adaptively integrating new experts into upcycled moe models is far less studied. for most practitioners, given the scale of modern llms @xcite @xcite training moes repeatedly is an infeasible computational cost. furthermore, most model development fails to take into training data domain 1 expert 1 seed llm expert 2 expert n ffn 1 ffn 2 ffn n initialize expert n+1 domain 2 domain n new domain inherent domains domain embeddings ... ... ... embedding model ff n+1 learned projection ffn output nexus ffn input activations expert emb. a) initial training b) add new data top-k routing ffn seed 1. collect ffn layers into moe layer routed experts 2. merge non-ffn weights its training data is embedded by an embedding model and stored. the experts are combined by initializing each block's moe layer with the expert ffns, and finetuning the model on a mix of all domains. during a forward pass, the seed model ffn is used as shared expert and always activated. for the other experts, we perform top-1 routing based on the similarity of the input data with the transformed expert embeddings, which is equivalent to viewing the learned projection as a hypernetwork whose output is the router weight matrix. b) later, we can add a new expert by appending its training data embedding to the existing domain embeddings. the router function is independent of the number of experts, and therefore adapts fast to the new one. in this work, we attempt to reconcile all three desirable properties: efficiency, specialization, and adaptability. we ask "how can we adaptively combine separately trained specialized experts?" to address this, we introduce nexus, a novel moe architecture that parameterizes the router based on domain-specific data by learning to project the embedding of each data domain to an expert embedding. this learnable projection for the router allows for the easy extension of the moe model with new experts that are trained independently on new datasets of interest. this also avoids the difficulties of moe training, as our learned router scales with the number of experts without needing to be trained from scratch, which enables adding or removing experts as desired. in summary, our contributions are as follows: ## background sparse mixture of experts architectures @xcite replace the feedforward network (ffn) with an moe layer in the transformer block @xcite . an moe layer consists of a router network r and a set of n experts, e 1 , ..., e n , where each expert e i corresponds to an independent dense feed-forward network. the router network r is commonly parameterized by trainable weights w @xmath0, and followed by a softmax function which takes an intermediate token representation x as input and combines the output of each expert based on the gating scores s 1 , ..., s n . sparse moes only use the top-k experts e k based on experts gating scores s i . sparse upcycling @xcite initializes an moe model from a dense transformer model by copying ffn layers as moe experts, and the router layer is trained from scratch. btx @xcite generalize this approach to initialize each moe expert from the ffn layer of a different dense model, and all other parameters are averaged over the dense models. in nexus, we leverage upcycling specialized expert models similar to btx, however, it diverges in terms of moe training, in particular with its novel moe router, which enables to efficiently extend the moe in multiple rounds after the sparse upcycling. we describe our method in the next section. ## adaptive router for upcycling specialized experts as moe the core component of an moe model is the router, as it determines which experts to activate for any given input. in vanilla moes, the router is a learned linear layer that takes the token intermediate representations as input and computes the expert probabilities. however, this router does not necessarily learn specialization as moes are commonly trained using an auxiliary load balancing loss to improve training stability @xcite . in nexus, we propose a novel moe router where per moe block we learn a projection layer from given pre-computed domain embeddings to expert embeddings. we parametrize this projection layer p r as a two-layer mlp with a swiglu activation function @xcite : unlike the standard router, nexus's router includes a stronger inductive bias through precomputed domain embeddings * that enables expert embedding to specialize. thus, x • e i gives a high value for input tokens that are closer to the domain of the corresponding expert. notably, this router is particularly suited for the sparse upcycling setting where the dense experts are separately trained on different domains. upcycling dense experts as an moe. after training dense expert models, we merge the individual experts into a unified moe by appending their ffns along a new dimension to create an moe layer per transformer block. unlike @xcite , instead of using the original ffn of the seed model as one of the routed experts in an moe layer, we use it as the "shared expert" ffn s @xcite to better preserve the previous capabilities in the moe model. for all non-ffn parameters including the attention weights, we merge expert parameters using simple weight averaging: efficient adaptation to new domains. an important advantage of our method is that when a new data domain is present after moe training, we use the learned projection p r to compute expert embedding of the new domain as e @xmath1. this enables to enhance the trained moe model with additional dense experts, which are trained in the same way as the initial experts. the ffn parameters of the new expert are simply appended to the array of existing experts. ## related work routing variants of moes. the most common moe architecture @xcite @xcite employs a linear router with a top-k routing scheme, where k typically equals 1 or 2. in this standard routing schema, only the k experts with the highest router gate values are activated. there is substantial research proposing alternatives to top-k expert assignments @xcite @xcite @xcite . deepseek-moe @xcite introduces a routing variant where a number of experts are "shared" and always assigned to all tokens. our work also adopts this approach for our general base expert. however, these efforts primarily focus on improving the general performance and/or training stability of moes. in contrast, our work puts emphasis adaptability and extensibility. ## conclusion we propose nexus, a new llm framework that enables efficient upcycling of specialized dense experts into a sparsely activated moe model. we show that individual experts in our method retain their specialization after upcycling, and that our router based on expert embeddings outperforms previous approaches for combining the dense experts. furthermore, the model can be extended efficiently with new dense experts after the initial training phase, saving much compute compared to re-training the upcycled model or training from scratch.