Emergent Modularity in Pre-trained Transformers (2023)

Abstract: This work examines the presence of modularity in pre-trained Transformers, a feature commonly found in human brains and thought to be vital for general intelligence. In analogy to human brains, we consider two main characteristics of modularity: (1) functional specialization of neurons: we evaluate whether each neuron is mainly specialized in a certain function, and find that the answer is yes. (2) function-based neuron grouping: we explore to find a structure that groups neurons into modules by function, and each module works for its corresponding function. Given the enormous amount of possible structures, we focus on Mixture-of-Experts as a promising candidate, which partitions neurons into experts and usually activates different experts for different inputs. Experimental results show that there are functional experts, where clustered are the neurons specialized in a certain function. Moreover, perturbing the activations of functional experts significantly affects the corresponding function. Finally, we study how modularity emerges during pre-training, and find that the modular structure is stabilized at the early stage, which is faster than neuron stabilization. It suggests that Transformer first constructs the modular structure and then learns fine-grained neuron functions. Our code and data are available at https://github.com/THUNLP/modularity-analysis .

## introduction recently, pre-trained transformers have shown the potential to achieve general intelligence @xcite @xcite @xcite , which encourages researchers to explore the analogy between ## functionality evaluation in this section, we first introduce the definition of neurons and how to evaluate the functionality of neurons and experts. then, we briefly introduce the evaluation setups, including the pre-trained models. ## functional specialization of neurons in this section, we analyze the functional specialization of neurons. first, we compare the neuron predictivities of pre-trained models with those of randomly-initialized models at the layer level, which will present a general picture of neuron functions. second, we study how functions distribute on neurons in each layer. neuron predictivities of different layers. at each layer, we compute the best predictivity of neurons for each sub-function and then calculate the average best predictivity among all sub-functions in each function. for comparisons, we also evaluate randomly-initialized models. we report the results in the left part of figure 1 . from this figure, we have two observations. (1) the average best predictivities of pre-trained neurons are significantly higher than those of randomly-initialized neurons, indicating that the neurons have learned these functions from pretraining and the neurons with top-ranked predictivities indeed excel in corresponding sub-functions. distribution in each layer. we first identify the neurons with the top predictivity rankings for each sub-function as sub-functional neurons in each layer and then compute the overlap between @xmath0. formally, assuming that we identify the top k neurons for each sub-function, the overlap score is defined as from this figure, we observe that: (1) in the pretrained models, the distribution similarity of the same function is significantly larger than that of different functions, which indicates that there are some groups of neurons, each of which is good at a certain function. and, it is not mainly caused by the similarity that naturally exists between subfunctions of the same function because the results of randomly-initialized models do not show such a phenomenon as shown in appendix a.3. hence, we conclude that there are some emergent groups of functionally-specialized neurons after pretraining. (2) one neuron may be capable of multiple sub-functions even from different functions. for example, the average overlap score between the knowledge function and task function is also significantly higher than that of random models so there are some neurons good at both knowledge and task sub-functions. ## finding modular structure of neurons since moe is a promising candidate for the modular structure of transformers, we analyze the moe structure in this section. first, we verify whether the neurons specialized in a certain function are concentrated in some experts, called functional experts. second, we perturb the activations of experts corresponding to a certain function. third, we train the model on different datasets, retaining only functional experts by removing non-functional experts. the last two experiments verify the importance of the experts for their corresponding functions. ## emergence of modularity in this section, we study the emergence of modularity during pre-training. to this end, we pre-train the base version of t5 and switch transformer from scratch. we use the moefication partitioning of the last checkpoint as the moe structure of t5. more details of pre-training are in appendix a.2. emergence patterns of functional experts. we first study the changing curves of the proportion of functional experts and their modularization degree during pre-training, i.e., we apply the same analysis in section 5 to each checkpoint. the results are shown in figure 3 . we have the following observations. (1) the proportion of functional experts and their modularization degree quickly achieves a high point, and then keeps relatively stable till the end. it indicates that functional experts emerge at the early stage of pre-training. (2) the proportion of functional experts in the switch transformer fluctuates significantly at about 20k steps and its stabilization is slower than that of t5. it suggests that the emergence of the modular structure in switch transformer is surprisingly more difficult than t5. the reason may be that switch transform omits the gradients of unselected experts, which causes the optimization to be harder than that of t5 @xcite . ## discussion efficient pre-training. large-scale pre-training of transformers is very expensive @xcite . it is promising to use moe to reduce the computational cost by activating only a small part of the experts. our findings have shown the emergent modularity of experts, which demonstrates the reasonableness of the moe structure. however, as shown in section 6, switch transformer is more unstable than t5 on the modular structure at the beginning. it suggests that we should begin with a dense model and gradually make it sparse instead of directly training a sparse model from scratch, which has been explored in some preliminary works @xcite . ## conclusion in this paper, we study the modularity of pretrained transformers and validate the emergence of modularity. we also study the pre-training process to understand the emergence of modularity and find the coarse-to-fine mechanism of pre-training. we expect our evaluation framework and findings will facilitate and inspire future research in this area.