Breaking R e LU Barrier: Generalized M o E fication for Dense Pretrained Models (2024)

Abstract: As the scale of language models (LMs) continues to grow, there is a heightened interest in reducing the inference cost associated with these models. Mixture-of-Experts (MoEs) present an efficient alternative to dense models, while the existing methods to convert pretrained dense models to MoEs is limited to ReLU-based models with natural sparsity. This paper introduces G-MoEfication, applicable to arbitrary dense models, where ReLU-based activation sparsity assumptions no longer hold. For generalizations, we encounter the dilemma of needing to zero-out deactivated experts, while also avoiding excessive zeroing-out to retain dense activation information. We publicly release our code and report results conducted with mBERT, SantaCoder-1.1B, Phi-2-2.7B, and Falcon-7B demonstrating the efficacy of our approach in general scenarios: from multitask to multilingual, from fine-tuning to zero-shot evaluation.

## introduction despite the remarkable performance exhibited by recent language models (lms), the increasing scale of these models, as highlighted in @xcite , leads to a growing interest in mitigating the inference cost associated with lms. however, as many emerging lms are dense @xcite , the substantial inference cost incurred by these models arises from the necessity to activate all parameters, leading to significant communication overhead because weights are sharded across multiple accelerators. the mixture-of-experts @xcite ) (moe) emerges as a promising alternative. unlike dense models, moe sparsely activates a subset of parameters, referred to as experts. moreover, only the accelerator associated with the selected experts is utilized, resulting in a remarkable reduction in communication overhead. nevertheless, training a pretrained moe model is computationally inefficient and environmentally detrimental @xcite . furthermore, moe training presents unique challenges, such as the occurrence of expert collapse, where all experts function identically @xcite . to avoid pretraining costs, this paper explores moefication, converting existing dense pre-trained language models into moe versions while retaining the same parameters. this conversion, however, is targeted for relu-based models @xcite . as relu is naturally sparse, as shown in the activation pattern of a relu-based model (figure 1 yellow line), its conversion into an moe is rather straightforward. remarkably, this approach achieves 95% of the original performance while utilizing only a limited number of feed-forward network (ffn) parameters. unfortunately, there is a recent divergence in the trend of utilizing relu as an activation function. instead, new models opt for smoother variants for better convergence, such as gelu @xcite , where the activation pattern is no longer sparse (figure 1 blue line), thereby compli-cating moefication. a naïve approach would be, relufying the lms, by simply replacing the activation function with relu and further pretrain the model @xcite . however, this method incurs additional pretraining costs @xcite , which are prohibitive given the prevailing trend of ever-growing lm scales @xcite . in this paper, we propose generalized moefication (g-moefication) to moefy arbitrary dense lms, without pretraining. our primary challenge is addressing the conflicting goals associated with dense activations. with activations no longer sparse, we encounter the dilemma of needing to zero-out certain activations to deactivate specific experts, while also avoiding excessive zeroing-out to prevent substantial information loss in dense activations (figure 1 blue line). our contribution lies in navigating this dilemma by introducing representative values for each feature (figure 3b ). maintaining representative values instead of totally zeroing out would preserve information in activations. simultaneously, our goal is to zero-out the residuals from the representative values, aiming to minimize information loss. desirably, representatives should avoid heavy multiplication costs, retaining optimality. first, to mitigate the heavy matrix multiplication, we ensure that the retained representatives remain static, and we generate the multiplication results as an embedding. during the inference step, this replaces the heavy matrix multiplicative operation with a lightweight vector additive operation (figure 3c ). second, regarding optimality, we formulate a cost function to assess the sparsity of the residuals. our aim is to calculate the optimal representative value that minimizes this measure. our contributions can be summarized as follows: ## proposed method we first formalize moefication under hard sparsity assumption (section 3.1), then propose to overcome its limitation for generalization (section 3.2). ## conclusion this paper introduces g-moefication, a novel methodology applicable to arbitrary dense models, overcoming the limitations imposed by relubased activation sparsity assumptions. our technical contribution is identifying and overcoming the dilemma of zeroing out deactivated experts, while simultaneously preserving dense activation information to ensure optimal model performance.