Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts (2025)

Abstract: Mixture-of-Experts (MoE) models have shown remarkable capability in instruction tuning, especially when the number of tasks scales. However, previous methods simply merge all training tasks (e.g. creative writing, coding, and mathematics) and apply fixed sampling weights, without considering the importance of different tasks as the model training state changes. In this way, the most helpful data cannot be effectively distinguished, leading to suboptimal model performance. To reduce the potential redundancies of datasets, we make the first attempt and propose a novel dynamic data mixture for MoE instruction tuning. Specifically, inspired by MoE’s token routing preference, we build dataset-level representations and then capture the subtle differences among datasets. Finally, we propose to dynamically adjust the sampling weight of datasets by their inter-redundancies, thus maximizing global performance under a limited training budget. The experimental results on two MoE models demonstrate the effectiveness of our approach on both downstream knowledge & reasoning tasks and open-ended queries.

## introduction instruction tuning is a pivotal step for large language model (llm) alignment @xcite . to promote the alignment ability, llms are typically fine-tuned on a collection of instruction datasets with multiple tasks @xcite @xcite . however, dense models may be constrained by their fixed model capacities when the number of tasks grows in instruction tuning @xcite . instead, mixture-of-experts (moe) naturally incorporates multiple experts, which expands the model capacity @xcite ## preliminaries of mixture-of-experts in a typical moe structure, the layer is composed of n expert networks {e 1 , e 2 , . . . , e n } and a gating network g. different from common networks, the moe manifests itself in the design of computational strategy, characterized by inherent sparsity. given an input token x, the gating network computes a vector of routing scores g(x) ∈ r n , denoting the importance of each expert network to process the given input. the moe layer then selectively aggregates the outputs from the top-k experts, which is represented as: the cv score would be high if tokens dispatched to experts are off-balance. the aggregation of these two terms ensures a balanced dispatching among experts. the importance score vector g @xmath0. the gate load vector ## methodology in this section, we introduce our dynamic sampling strategy, which automatically adjusts the sampling weights of different instruction datasets. after every m steps of model training, we obtain the gate loads o as dataset-level representations, then calculate the differences across datasets with o and update sampling weights accordingly. the dynamic sampling algorithm is presented in alg 1. ## conclusion to combine different datasets and maximize the moe model's alignment ability, we assign different sampling weights to corresponding datasets. by incorporating the internal model state and the dataset properties, we propose to use the gate load from moe models to obtain dataset representations. based on the representations, we calculate distances between each pair of datasets, indicat-ing the inter-redundancies. we further devise an automatic algorithm to dynamically update the sampling weights. we find there is a strong correlation between the dataset-level load balance and the final performance, and the proposed dynamic sampling strategy could reach great balancing. the results also demonstrate good performance on the overall downstream tasks.