Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and M o E Models in Large Language Models (2024)

Abstract: The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferability and discrepancies of scaling laws between Dense Models and Mixture of Experts (MoE) models. Through a combination of theoretical analysis and extensive experiments, including consistent loss scaling, optimal batch size/learning rate scaling, and resource allocation strategies scaling, our findings reveal that the power-law scaling framework also applies to MoE Models, indicating that the fundamental principles governing the scaling behavior of these models are preserved, even though the architecture differs. Additionally, MoE Models demonstrate superior generalization, resulting in lower testing losses with the same training compute budget compared to Dense Models. These findings indicate the scaling consistency and transfer generalization capabilities of MoE Models, providing new insights for optimizing MoE Model training and deployment strategies.

## introduction the advent and scaling of large language models (llms), such as gpt @xcite , llama @xcite , gemini @xcite , gopher @xcite , chinchilla @xcite , and mistral @xcite , have marked a transformative era in artificial intelligence and natural language processing. characterized by their vast parameter counts and extensive training datasets, these models have significantly advanced capabilities across various domains, including machine translation @xcite @xcite , logical reasoning @xcite @xcite , and medical applications (thirunavukarasu et al., * corresponding authors. 2023; @xcite . however, their increasing complexity and parameter scale posits an urgent need for innovative scaling strategies that optimize computational efficiency without compromising performance. historically, dense transformer models have dominated due to their simplicity and scalability. and the scaling laws @xcite for dense models have been thoroughly investigated across different circumstances, such as over-training @xcite and datalimiting @xcite . despite their efficacy, the huge computational demands of these models necessitate exploration of alternative architectures like mixture of experts (moe) @xcite @xcite @xcite , which offer a promising reduction in computational load through sparse activations and dynamic expert routing. this paper delves into the analysis of scaling laws for dense and moe models within the context of llms. we extend foundational research on hyperparameters, such as compute budget, batch size, and learning rate @xcite @xcite @xcite , to explore their transferability and applicability across these architectures. our experiments involve models up to 7 billion parameters and datasets exceeding 100 billion tokens, aiming to uncover universal scaling behaviors potentially applicable to both model types. our results verify the hypothesis that certain scaling laws, particularly those related to loss and hyperparameters, may indeed be universal, bridging architectural gaps between dense and moe models. this universality suggests a simplification in hyperparameter tuning across different scales and architectures, which could significantly streamline the training processes for various llms. furthermore, we provide detailed analyses of the dif-ferential impacts of coefficient changes between moe and dense models, offering both empirical and theoretical insights into the superior data efficiency of moe models. concretely, moe models can achieve comparable performance with fewer training tokens than dense models, alleviating data constraints in llm training. our findings can be summarized as follows: ## related work large language models large language models (llms) such as gpt @xcite , llama @xcite , chinchilla (hoffmann et al., 2022), gopher (rae et al., 2021), mixtral 8x7b (jiang et al., 2024), switch transformer @xcite , glam @xcite , and deepspeed-moe @xcite have advanced significantly, categorized into dense models and mixture of experts (moe) models. dense models activate all parameters per forward pass, while moe models activate only a subset, allowing for larger model scales without proportional increases in computational costs. despite their complexity, moe models have shown potential for superior performance and efficiency. scaling laws for llms due to the significant costs associated with training process, understanding the scaling laws of large language models (llms) is crucial. studies @xcite @xcite have established a power-law relationship between model loss and factors like training tokens and compute budget. recent work @xcite has explored these relationships further in moe models, indicating cost-effective scaling benefits but also highlighting challenges such as expert selection and load balancing. however, a systematic investigation into the scaling laws of moe models' hyperparameters and the transferability of scaling laws between dense models and moe models remains lacking, which is the focus of our work. ## preliminary the scaling law of the training loss for dense models with respect to the number of training tokens and model size has been extensively studied @xcite . previous work @xcite has proposed the following scaling law (shown in equation 1 ). to introduce the concept of model scale (the flops divided by the number of training tokens) as n in our work, we denote model size (number of parameters) as p to avoid confusion. c is the compute budget with an approximation @xmath0. α, β, a and b are all coefficients. for moe (mixture of experts) models, previous research @xcite has proposed a separable scaling law (equation 2 ) for loss between model size and the number of experts. when using equation 2 to fit the training loss curve of moe models, clark et al. (2022) claim that a decrease in performance has been observed given by expert scaling. specifically, the value of b increases with model size (in equation 2) when model size is large. this suggests that as the model size increases, the benefit from increasing the number of experts e will finally decrease. to enhance the fitting ability of the scaling laws for moe models, a quadratic interaction term is added, resulting in equation 4. l(p, e) = a log(p ) + b log(e) + c log(p ) log(e) + @xmath1.1 scaling laws for training loss kaplan et al. (2020) and hoffmann et al. (2022) originally proposed a scaling law for dense models, while clark et al. (2022) extended this law to scenarios involving multiple experts (moe models). upon closer examination of equation 3 and equation 4, we observed that when the number of experts (e) remains fixed, these equations can be simplified to the first term in equation 1. motivated by these insights, we introduce a unified scaling law for both dense models and moe models, represented by equation 5. specifically, since the decrease in performance is observed only when the number of experts (e) is large, we adopt the simplified one (equation 2 ) for small e value (below 100). besides, previous work @xcite suggests replacing the model size p (number of parameters) with model scale n , which is the result of the flops divided by the number of tokens in order to fit equation 1 more accurately. finally, we get equation 5 : in order to validate equation 5, in our experiment, we fitted the training data for both 200m and 700m moe models (both with eight experts) to a curve. we then used this curve to predict the training loss scaling behavior of a 1.5b moe model. the results, shown in figure 1 , demonstrate that the formula we proposed based on previous work is applicable to moe models when the number of experts is not large. it could be observed that the reduction in benefits from increasing e is minimal and the scaling equation stands within our experiment scope. therefore, we adopted the simplified version. this formula suggests that the equation 5 could equal to equation 1, given a fixed number of experts, for moe models. this consistency could help us to compare the computing resource allo- ## generalization of scaling law we predict that moe models could have better generalization ability compared to their dense counterparts. firstly, moe models consist of multiple expert sub-networks that specialize in different aspects of the data, which enables the model to capture a broader range of features and patterns and then leads to better generalization across various tasks and datasets. additionally, the ensemble nature of moe models, where each expert contributes to the final prediction, reduces overfitting and improves robustness by combining predictions from multiple models. finally, the gating mechanisms in moe models control the contribution of each expert to the final prediction, acting as a form of to investigate the generalization performance of the scaling law for both moe models (eight experts) and dense models, we explore and compare the relationship of training loss with testing loss and compute budget for both dense models and moe models (shown in figure 6 ). we highlight the stable power-law relationship interval across different model sizes, illustrating the correlation between testing loss and compute budget. notably, moe models consistently exhibit a smaller testing loss for a given compute budget, indicating their superior generalization performance. we also explore the performance of the dense and moe models on different testing sets in ## conclusion in this paper, we investigate the transfer of traditional scaling laws from dense models to mixture of experts (moe) models. our investigation confirms that the power-law relationship extends to moe models regarding the consistency and trans-ferability of scaling strategies, including resource allocation strategy, optimal batch size / learning rate scaling, and so on. this observation indicates that the fundamental principles of training dynamics and the behavior of scaling rules are similar for both models. this means existing knowledge and practices for optimizing dense models can be easily adapted to moe models, potentially reducing the experimental burden of finding the best hyperparameters. besides, we find that moe models demonstrate approximately a 16.37% improvement in data utilization efficiency compared to dense models with a fixed compute budget. thus we suggest prioritizing increasing model scale over other factors when training moe models, highlighting their greater data efficiency. finally, we use both theoretical and empirical analysis to reveal that during the training process, moe models exhibit a lower gradient noise scale when using the adam optimizer. at the same training loss value, moe models can achieve stable training with smaller batch sizes and larger learning rates, potentially speeding up the training process and improving convergence. it shows that moe models can make more efficient use of training data and computational resources, extracting more information per training token, leading to faster training times and better utilization of available data. these results offer valuable insights for refining training and deployment strategies for moe models.