Finding Skill Neurons in Pre-trained Transformer-based Language Models (2022)

Abstract: Transformer-based pre-trained language models have demonstrated superior performance on various natural language processing tasks. However, it remains unclear how the skills required to handle these tasks distribute among model parameters. In this paper, we find that after prompt tuning for specific tasks, the activations of some neurons within pre-trained Transformers are highly predictive of the task labels. We dub these neurons skill neurons and confirm they encode task-specific skills by finding that: (1) Skill neurons are crucial for handling tasks. Performances of pre-trained Transformers on a task significantly drop when corresponding skill neurons are perturbed. (2) Skill neurons are task-specific. Similar tasks tend to have similar distributions of skill neurons. Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than fine-tuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit. We also explore the applications of skill neurons, including accelerating Transformers with network pruning and building better transferability indicators. These findings may promote further research on understanding Transformers. The source code can be obtained from https://github.com/THU-KEG/Skill-Neuron.

## introduction pre-trained language models (plms), mostly based on transformer architecture @xcite , have achieved remarkable performance on broad and diverse natural language processing (nlp) tasks @xcite . however, it remains unclear how the skills required to handle these tasks distribute among model parameters. are there specific neurons within pre-trained transformers encoding these skills? progress on this problem may help to understand the working mechanisms of pre-trained transformers @xcite @xcite , intervene model behaviors @xcite , and improve model efficiency @xcite . prompt tuning @xcite prepends some trainable embeddings, i.e., soft prompts, into the inputs and adapts plms to handle tasks by only tuning the soft prompts while freezing all the plm parameters. it has attracted wide attention recently as a promising parameterefficient fine-tuning methods @xcite . in this paper, we find that after prompt tuning for a task, the activations on soft prompts of some neurons within pre-trained transformers are highly predictive for the task. for instance, figure 1 shows the activation distribution of a specific neuron within roberta base @xcite . this neuron's activation is highly predictive of the labels of sst-2 @xcite , an established sentiment analysis dataset. when the input sentences express positive sentiments, the activations on soft prompts of this neuron tend to be much higher than when they express negative sentiments. it suggests that this neuron may encode the skill of distinguishing sentiments. we dub these special neurons skill neurons and develop a simple and effective method to find them for classification tasks via prompt tuning. for a binary classification task, we first calculate the empirical mean activation on a soft prompt token over the training set for each neuron and use it as this neuron's baseline activation. if this neuron's activation for an input sample is higher than the baseline, we regard it as predicting one label and vice versa. we aggregate the prediction accuracies on the validation set of multiple soft prompts as the neuron's predictivity score. the neurons with the highest predictivity scores are identified as skill neurons. for multi-class classification tasks, we decompose them into multiple binary classification subtasks and aggregate the skill neurons of subtasks as the skill neurons of the multi-class task. we confirm the skill neurons encode taskspecific skills with a series of experimental findings: (1) skill neurons generally and stably emerge. for all the 7 investigated tasks and 5 random trials, we can consistently find skill neurons with high predictivities close to prompt tuning. (2) skill neurons are crucial for handling tasks. when we perturb skill neurons by adding random noises to their activations, the performances on corresponding tasks drop much more significantly than when random neurons are perturbed. (3) skill neurons are taskspecific. similar tasks exhibit similar predictivity rankings of skill neurons, and skill neurons of sametype tasks are more important for handling a task than those of different-type tasks. (4) skill neurons are not from shallow word selectivity. the skill neurons typically do not selectively activate on keywords relating to the task, and their predictivities are not significantly influenced by the label words used in prompt tuning. after showing that skill neurons encode skills, we further demonstrate that skill neurons are most likely generated in pre-training rather than manufactured by the fine-tuning process of prompt tuning. this is concluded from: (1) even for randomly generated prompts and untuned hard prompts, the skill neurons still exhibit much better predictivity performance than random guesses. (2) skill neurons are also crucial for other fine-tuning methods freezing neuron weights. performance of models trained with adapter-based tuning @xcite and bitfit @xcite significantly drops when the skill neurons found with prompt tuning are perturbed. moreover, we explore the practical applications of skill neurons. first, we apply skill neurons to network pruning @xcite , which aims at removing redundant parameters to reduce memory cost and accelerate inference. experiments show that by only keeping top skill neurons active, we can reduce the pre-trained transformer to 66.6% of its original parameters and achieve about 1.4 inference speedup. then we explore building better prompt transferability indicators following @xcite . we improve their overlapping rate of activated neurons metric by only taking skill neurons into account, and this achieves significantly better performance. to summarize, our contributions are four-fold: (1) we observe the existence of skill neurons, the special neurons within pre-trained transformers, which are highly predictive for specific tasks, and develop a method to find them via prompt tuning. ## preliminary we introduce the basic knowledge about prompt tuning ( § 2.1), the definition of investigated neurons ( § 2.2), and the investigation setup ( § 2.3). ## finding skill neurons we use a simple and effective method to find skill neurons for a given pre-trained transformer m. ## conclusion and future work in this paper, we find some special neurons in pre-trained transformers whose activations on soft prompts are highly predictive of the task labels of inputs. we dub these neurons skill neurons and develop a method to find them via prompt tuning. with extensive experiments, we confirm that skill neurons encode task-specific skills required to handle these tasks and find empirical evidence showing that skill neurons are most likely generated in pre-training rather than fine-tuning. we also demonstrate some practical applications of our skill neuron finding. in the future, we will extend our prompt-based skill neuron finding method to more scenarios, such as covering non-classification tasks and other parameters in transformers like attention heads. we will also explore more fundamental problems about skill neurons and the working mechanisms of plms, including how the skill neurons emerge in pre-training, as well as the relationships between skill neurons and neurons encoding specific information found in previous works.