M o E fication: Transformer Feed-forward Layers are Mixtures of Experts (2022)

Abstract: Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks. Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/thunlp/MoEfication .

## introduction recent years have witnessed great success of transformer-based pre-trained language models † corresponding authors part of the work was done while peng li was working at tencent. in this work, we study the activation patterns of ffns in transformer models and find a phenomenon of sparse activation, i.e., only a tiny fraction of neurons are activated for a single input. for example, when we perform inference on a finetuned t5-large model @xcite with 700-million parameters, 90% inputs only activate less than 5% neurons foot_0 . this phenomenon is similar to the sparsity in the human brain @xcite , which drives research on functional partitions of the human brain @xcite . inspired by such observation, we further raise up a question: do the functional partitions also emerge in artificial neural models, i.e., ffns in pre-trained transformer? to investigate this problem, we explore whether a transformer can be converted into an equivalent mixture-of-experts (moe) model @xcite , which regards different functional partitions in ffns as different experts conditionally activated. specially, we propose moefication to discover the functional partitions (experts) in ffns and build routers for selecting experts. it consists of two phases. (1) expert construction: split a whole feed-forward layer into multiple experts. the goal is to group those neurons that are often activated simultaneously into the same expert network. (2) expert selection: select those experts that contain as many activated neurons as possible for each input to approximate to the original results. in the experiments, we evaluate moefication on two typical kinds of downstream tasks, including glue and qa benchmarks @xcite @xcite , using t5 and bert @xcite . experimental results verify that ffns in transformers can be converted to mixtures of experts, and thus we can use only 10% to 30% of ffn parameters to maintain over 95% original performance, which verifies that the pre-trained transformers also learn the functional partitions in ffns. besides, moefication brings two advantages: (1) it can significantly speed up the inference of transformers. using 25% of ffn parameters brings 2x speedup on cpu and 1.2x speedup on gpu. (2) we can study moefied models to interpret the inner mechanism of ffns at a fine-grained level. in this work, we study their routing patterns and hope these findings can help future work on the design and training of moe models. ## moefication in this section, we will introduce the general idea of moefication and divide it into two phases: expert construction and expert selection. ## analysis in this section, we analyze the efficiency and routing patterns of moefied models. ## conclusion in this work, we verify that transformer ffns are naturally mixtures of experts and propose moefication, which utilizes the sparse activation phenomenon in ffns to convert a normal model to its moe version with the same parameters. experimental results show that moefied models can achieve comparable performance to the original models using only 10% to 30% of ffn parameters. correspondingly, it significantly reduces the flops of inference, e.g., 2x speedup with 20% of ffn parameters. besides, by studying the routing patterns of moefied models, we find that there are general and input-specific experts, which may inspire future work on training moe models. we hope moefication can benefit real-world applications of plms with better efficiency and benefit the interpretation of the inner mechanism of ffns.