Analyzing Individual Neurons in Pre-trained Language Models (2020)

Abstract: While a lot of analysis has been carried to demonstrate linguistic knowledge captured by the representations learned within deep NLP models, very little attention has been paid towards individual neurons. We carry outa neuron-level analysis using core linguistic tasks of predicting morphology, syntax and semantics, on pre-trained language models, with questions like: i) do individual neurons in pre-trained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how distributed or focused is the information? and iv) how do various architectures differ in learning these properties? We found small subsets of neurons to predict linguistic tasks, with lower level tasks (such as morphology) localized in fewer neurons, compared to higher level task of predicting syntax. Our study also reveals interesting cross architectural comparisons. For example, we found neurons in XLNet to be more localized and disjoint when predicting properties compared to BERT and others, where they are more distributed and coupled.

## introduction transformer-based neural language models have constantly pushed the state-of-the-art in downstream nlp tasks such as question answering, textual entailment, etc. @xcite . central to this revolution is the contextualized embedding, where each word is assigned a vector based on the entire input sequence, allowing it to capture not only a static semantic meaning but also a contextualized meaning. previous work on analyzing neural networks showed that while learning rich nlp tasks such as machine translation and language modeling, these deep models capture fundamental linguistic phenomena such as word morphology, syntax and various other relevant properties of interest @xcite @xcite @xcite . more recently @xcite and @xcite used probing classifiers to analyze pretrained neural language models on a variety of sequence labeling tasks and demonstrated that contextualized representations encode useful, transferable features of language. while most of the previous studies emphasize and analyze representations as a whole, very little work has been carried to analyze individual neurons in deep nlp models. studying individual neurons can facilitate understanding of the inner workings of neural networks @xcite @xcite and have other potential benefits such as controlling bias and manipulating system's behaviour @xcite , model distillation and compression @xcite , efficient feature selection @xcite , and guiding architectural search. in this work, we put the representations learned within pre-trained transformer models under the microscope and carry out a fine-grained neuron level analysis with respect to various linguistic properties. we target questions such as: i) do individual neurons in pretrained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how distributed or focused is the information? and iv) how do various architectures differ in learning these properties? a typical methodology in previous work on analyzing representations trains probing classifiers using the representations learned within a neural model, to predict the understudied task. we also use a probing classifier approach to analyze individual neurons. since neurons are multivariate in nature and work in groups, we additionally use elastic-net regularization that encourages individual and group of neurons to play a role in the train-ing of the classifier. given a trained classifier, we consider the weights assigned to each neuron as a measure of their importance with respect to the understudied linguistic task. we use probes with high selectivity @xcite to ensure that our results reflect the property of representations and not the probe's capacity to learn. we choose 4 pre-trained models: elmo @xcite , its transformer variant t-elmo @xcite , bert @xcite and xlnet @xcite -covering a varied set of modeling choices, including the building blocks (recurrent networks versus transformers), optimization objective (auto-regressive versus nonautoregressive), and model depth and width. our cross architectural analysis yields the following insights: ## methodology a common approach for probing neural network components against linguistic properties is to train a linear classifier using the activations generated from the trained neural network as static features. the underlying assumption is that if a simple linear model can predict a linguistic property, then the representations implicitly encode this information. probe: we go a level deeper and identify neurons within the learned representations to carry out a more fine-grained neuron foot_0 level analysis. we use a logistic regression classifier with elastic-net regularization @xcite . the weights of the trained classifier serve as a proxy to select the most relevant features foot_1 within the learned representations, to predict a linguistic property. formally, consider a pre-trained neural language model m with l layers: {l 1 , l 2 , . . . , l l }. given a dataset @xmath0, w 2 , ..., w n } with a corresponding set of linguistic annotations @xmath1, t w 2 , ..., t w n }, we map each word w i in the data d to a sequence of latent representations: d m -→ @xmath2, . . . , z n }. the representations can either be extracted from the entire model or just from an individual layer. the model is trained by minimizing the following loss function: ## experimental setup pre-trained neural language models: we present results with 4 pre-trained models: elmo @xcite , and 3 transformer architectures: transformer-elmo @xcite , bert @xcite and xlnet @xcite . the elmo model is trained using a bidirectional recurrent neural network (rnn) with 3 layers each of size 1024 dimensions. its transformer equivalent (t-elmo) is trained with 7 layers but with the same hidden layer size. the bert model is trained as an auto-encoder with a dual objective function of predicting masked words and next sentence in auto-encoding fashion. we use base version (13 layers and 768 dimensions). lastly we included xlnet-base which is trained with the same parameter settings (number and size of hidden layers) as bert, but with a permutation based auto-regressive objective function. language tasks: we evaluated our method on 4 linguistic tasks: pos-tagging using the penn treebank @xcite , syntax tagging (ccg supertagging) foot_6 using ccgbank @xcite , syntactic chunking using conll 2000 shared task dataset @xcite , and semantic tagging using the parallel meaning bank data @xcite . we used standard splits for training, de-velopment and test data (see appendix a.1) classifier settings: we used linear probing classifier with elastic-net regularization, using a categorical cross-entropy loss, optimized by adam @xcite . training is run with shuffled mini-batches of size 512 and stopped after 10 epochs. the regularization weights are trained using grid-search algorithm. 8 for sub-word based models, we use the last activation value to be the representative of the word as prescribed for the embeddings extracted from neural mt models @xcite and pre-trained language models @xcite . linear classifiers are a popular choice in analyzing deep nlp models due to their better interpretability @xcite . @xcite have also shown linear probes to have higher selectivity, a property deemed desirable for more interpretable probes. linear probes are particularly important for our method as we use the learned weights as a proxy to measure the importance of each neuron. ## related work rise of neural network has seen a subsequent rise of interpretability of these models. researchers have explored visualization methods to analyze learned representations @xcite , attention heads @xcite of language compositionality @xcite etc. while such visualizations illuminate the inner workings of the network, they are often qualitative in nature and somewhat anecdotal. a more commonly used approach tries to provide a quantitative analysis by correlating parts of the neural network with linguistic properties, for example by training a classifier to predict a feature of interest @xcite . please refer to @xcite for a comprehensive survey of work done in this direction. @xcite used probing classifiers for investigating the contextualized representations learned from a variety of neural language models on numerous word level linguistic tasks. a similar analysis was carried by @xcite on a variety of sub-sentence linguistic tasks. we extend this line of work to carry out a more fine-grained neuron level analysis of neural language models. our work is most similar to @xcite who conducted neuron analysis of representations learned from sequence-to-sequence machine trans-lation models. our work is different from them in that i) we carry out analysis on a wide range of architectures which are deeper and more complicated than rnn-based models and illuminate interesting insights, ii) we automated the grid-search criteria to select the regularization parameters, compared to manual selection of lambdas, which is cumbersome and error-prone. in contemporaneous work, @xcite used max-pooling to identify relevant neurons (aka expert units) in pre-trained models, with respect to a specific concept (for example word-sense). a pitfall to the approach of probing classifiers is whether the probe is faithfully reflecting the property of the representation or just learned the task? @xcite defined control tasks to analyze the role of training data and lexical memorization in probing experiments. @xcite proposed an alternative that measures minimal description length of labels given representations. it would be interesting to see how a probe's complexity in their work (code length) compares with the number of selected neurons according to our method. the results are consistent at least in the elmo pos example, where layer 1 was shown to have the shortest code length in their work. in our case, most top neurons are selected from layer 1 (see figure 1d for example). @xcite discussed the complexity of the probes and argued for using highest performing probes for tighter estimates. however, complex probes are difficult to analyze. linear models are preferable due to their explainability; especially in our work, as we use the learned weights as a proxy to get a measure of the importance of each neuron. we used linear classifiers with control tasks as described in @xcite . although we mainly used probing accuracy to drive the neuron selection in this work, and selectivity only to demonstrate that our results reflect the property learned by representations and not probe's capacity to learn -an interesting idea would be to use selectivity itself to drive the investigation. however, it is not trivial how to optimize for selectivity as it cannot be controlled/tuned directly -for example, removing some neurons may decrease accuracy but may not change selectivity. we leave this exploration for future work. probing classifiers require supervision for the linguistic tasks of interest with annotations, limiting their applicability. @xcite used unsupervised approach to identify salient neurons in neural machine translation and manipulated translation output by controlling these neurons. recently, @xcite measured similarity of internal representations and attention across prominent contextualized representations (from bert, elmo, etc.). they found that different architectures have similar representations, but different individual neurons. ## conclusion we analyzed individual neurons across a variety of neural language models using linguistic correlation analysis on the task of predicting core linguistic properties (morphology, syntax and semantics). our results reinforce previous findings and also illuminate further insights: i) while the information in neural language models is massively distributed, it is possible to extract a small number of features to carry out a downstream nlp task, ii) the number of extracted features varies based on the complexity of the task, iii) the neurons that learn word morphology and lexical semantics are predominantly found in the lower layers of the network, whereas the ones that learn syntax are at the higher layers, with the exception of bert, where neurons were spread across the entire network, iv) closed-class words (for example interjections) are handled using fewer neurons compared to polysemous words (such as nouns and adjectives), v) features in xlnet are more localized towards individual properties as opposed to other architectures where neurons are distributed across many properties. a direct application of our analysis is efficient feature-based transfer learning from largescale neural language models: i) identifying that most relevant features for a task are contained in layer x reduces the forward-pass to that layer, ii) reducing the feature set decreases the time to train a classifier and also its inference. we refer interested readers to see our work presented in @xcite for more details.