Are All Neurons Created Equal? Interpreting and Controlling BERT through Individual Neurons
Abstract
While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is encoded.
Among these, the common approach is to use an external probe to rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it.
We show that this methodology confounds distinct factors—probe quality and ranking quality—and thus we separate them.
We compare two recent ranking methods and a novel one we introduce, both by probing and by causal interventions, where we modify the representations and observe the effect on the model’s output.
We show that encoded information and used information are not always the same, and that individual neurons can be used to control the model’s output, to some extent.
Our method can be used to identify how certain information is encoded, and how to manipulate it for debugging purposes.
Introduction
What linguistic information is encoded in a hidden representation?
A popular approach to tackle this question is by probing: training a simple model to predict some information from the representation [Adi et al., 2017, Conneau et al., 2018].
Most studies focus on training the classiﬁer using the entire high-dimensional representation to predict the desired attribute.
Such a method can show whether the information is encoded in the representation but not how it is encoded: in which neurons?
is it localized (concentrated in a small set of neurons) or dispersed?
A few recent studies [Elazar et al., 2021, Feder et al., 2020] have tackled a related, causal question: given that the information is encoded, is it actually being used by the language model that encoded it?
If we want to alter a model’s behaviour—for example, for debugging purposes—we would like to know both how the information is encoded and whether it is used by the model.
Such knowledge may allow us to modify speciﬁc neurons of the representation in a certain way that would change the model’s output, with respect to that attribute—as in Bau et al.
[2019]—and provide us with parameters-level explanations of the model’s decisions.
Other reasons to look at individual neurons can be to reduce the number of components in the model, using methods such as pruning [Voita et al., 2019, Sajjad et al., 2020], as well as gain a general scientiﬁc understanding of the model.
In this work, we focus on two prior neuron ranking methods, which aim to rank the neurons of a word representation according to their importance for a linguistic property, such as part of speech.
Both of these methods—L INEAR [Dalvi et al., 2019] and G AUSSIAN [Torroba Hennigen et al., 2020]—rely on an external probe to obtain a ranking: the ﬁrst makes use of the internal weights of a linear probe, while the second considers the performance of a decomposable generative probe.
Both methods evaluate the quality of their ranking according to the probe’s accuracy using only the k -highest ranked neurons.
However, using this metric, two distinct factors are conﬂated: the probe’s quality as a classiﬁer, and the quality of the ranking it produces.
We claim that we should separate the two: a good classiﬁer may provide good results even if its ranking is bad, and an optimal ranking may cause an average classiﬁer to provide better results than a good classiﬁer that is given a bad ranking.
To avoid this conﬂation, we propose a simple ranking method, P ROBELESS , which ranks neurons according to the difference in their values across labels, and thus can be derived directly from the data, with no probing involved.
To disentangle probe quality and ranking quality, we use a probe from one method with a ranking from another one.
We ﬁnd that while G AUSSIAN generally provides higher accuracy, its selectivity [Hewitt and Liang, 2019] is lower, implying that it relies more on memorization, which improves probing quality but not necessarily ranking quality.
We further ﬁnd that G AUSSIAN provides the best ranking for small sets of neurons, while L INEAR provides a better ranking for large sets.
To apply a comparison that focuses on the rankings and is free of probing, and to ﬁnd which ranking selects neurons that are used by the model, we apply interventions on the representation: we modify subsets of neurons from each ranking and observe the effect on language modeling w.r.t to the property in question.
We ﬁnd that P ROBELESS tends to select neurons that are used by the model, more so than the two probing-based rankings.
We introduce a novel intervention method that allows a ﬁne-grained control of the model’s output with respect to a linguistic attribute, to some extent.
For example, intervening on the tense attribute in
English may turn the words “using” and “think” into “used” and “thought”.
This method can be used as a neuron-level debug tool, allowing developers to understand how easy or hard it is to reduce or magnify some of the model’s properties, e.g., gender bias, by observing how the gender attribute is spread along the neurons and the success rate of changing a word’s gender.
By comparing probing results and intervention results, we deduce that there is an overlap between encoded information and used information, but they are not the same.
We experimentally analyze the M-BERT model [Devlin et al., 2019] on 9 languages and 12 morphological attributes, from the Universal Dependencies dataset [Zeman et al., 2020].
Our experiments reveal the following insights:
• We show the need to separate between probing quality and ranking quality, by showing cases where intentionally poor rankings provide better accuracy than good rankings, due to probing weaknesses.
• We present a new ranking method that is free of any probes, and tends to prefer neurons that are being used by the model, more so than existing probing-based rankings.
• We show that there is an overlap between encoded information and used information, but they are not the same. • We show the advantage of looking into individual neurons, especially for debugging purposes, to allow a ﬁne-grained control over the model’s output.
Neuron Rankings and Data
We begin by introducing some notation.
We denote the word representation space as H ⊆ R d and an auxiliary task as a function F : H → Z , for some task labels Z
(e.g., part-of-speech labels).
Given a word representation h ∈ H and some subset of neurons S ⊆ { 1 , ..., d } , we use h S to denote the subvector of h in dimensions S .
For some auxiliary task F and k ∈ N , we search for an optimal subset S ∗ such that | S ∗ | =
k and h S ∗ contains more information regarding F than any other subvector h S (cid:48) , | S (cid:48)
| =
k .
For the search task, we deﬁne neuron-ranking as a permutation Π( d ) on { 1 , ..., d } and consider the subset Π( d ) [ k ] =
{ Π( d ) 1 , ..., Π( d ) k } .
Our goal is to ﬁnd an optimal ranking Π ∗ ( d ) such that ∀ k, Π ∗ ( d ) [ k ] is the optimal subset with respect to F .
However, ﬁnding such an optimal ranking, or even an optimal subset, is NP-hard
[Binshtok et al., 2007].
Thus, we focus on several methods to produce rankings, which provide approximations to the problem, and compare them.
2.1 Rankings
The ranking methods we compare include two rankings obtained from previously suggested probingbased neuron-ranking methods, and a novel ranking we propose, based on data statistics rather than probing.
2.1. L INEAR
The ﬁrst method, henceforth L INEAR (named linguistic correlation analysis in Dalvi et al.
[2019]), trains a linear classiﬁer on the representations to learn the task F .
Then, it uses the trained classiﬁer’s weights to rank the neurons according to their importance for F .
They showed that their method identiﬁes important neurons through probing and ablation studies, and found that while the information is distributed across neurons, the distribution is not uniform, meaning it is skewed towards the top-ranked neurons.
In Dalvi et al.
[2019], after the probe has been trained, its weights are fed into a neuron ranking algorithm.
However, we observed that the proposed algorithm distributes the neurons equally among labels, meaning that each label would contribute the same number of neurons at each portion of the ranking, regardless of the amount of neurons that are actually important for this label.
Thus, we chose a different way to obtain the ranking: for each neuron, we compute the mean absolute value of the | Z | weights associated with it, and sort the neurons by this value, from highest to lowest.
In early experiments we found that this method empirically provides better results, and is more adapted to large label sets.
2.1. G AUSSIAN
The second method, henceforth G AUSSIAN [Torroba Hennigen et al., 2020], trains a generative classiﬁer on the task F , based on the assumption that each dimension in { 1 , ..., d } is Gaussiandistributed.
Then, it makes use of the decomposability of the multivariate Gaussian distribution to greedily select the most informative neuron, according to the classiﬁer’s performance, from { 1 , ..., d } at every iteration.
This way we greedily obtain a full neuron ranking after training only once, while applying this greedy method to L INEAR would require retraining the probe d ! times, which is clearly infeasible.
They found that most of the tasks can be solved using a relatively low number of neurons, but also noted that their classiﬁer is limited due to the Gaussian distribution assumption.
2.1. P ROBELESS
The third neuron-ranking method we experiment with is based purely on the representations, with no probing involved, making it free of probing limitations [Belinkov, 2021] that might affect ranking quality.
For every attribute label z ∈ Z , we calculate q ( z ) , the mean vector of all representations of words that possess the attribute and the value z .
Then, we calculate the element-wise difference between the mean vectors,
[Equation]
and obtain a ranking by arg-sorting r , i.e., the ﬁrst neuron in the ranking corresponds to the highest value in r .
In the case of a binary-valued attribute, this is simply the difference in means.
For attributes with more than two values, P ROBELESS prefers neurons that separate different labels over neurons that have a similar value across labels.
2.2 Data
Throughout our work, we follow the experimental setting of Torroba Hennigen et al.
[2020]: we map the UD treebanks [Zeman et al., 2020] to the UniMorph schema [Kirov et al., 2018] using the mapping by McCarthy et al.
We select a subset of the languages used by Torroba Hennigen et al.
[2020]: Arabic, Bulgarian, English, Hindi, Finnish, French, Russian, Spanish and Turkish, to keep linguistic diversity.
We work with morphological attributes from these languages.
Full data details are provided in Torroba Hennigen et al.
[2020] and further data preparation steps are detailed in Appendix A.1.
We process each sentence in a pre-trained M-BERT, and take word representations from layers 2, 7 and 12 of the model, to see if there are different patterns in the beginning, middle and end of the model.
We end up with a total of 156 different conﬁgs (language, attribute, layer) to test.
For words that are split during tokenization, we deﬁne their ﬁnal representation to be the average over their sub-token representations.
Thus, each word has one representation for each layer, ofdimensions.
We do not mask any words throughout our work.
Probing
Given some ranking Π( d ) , we would like to evaluate how well it sorts the neurons for the task F .
Our ﬁrst ranking-evaluation approach is the standard probing approach from previous work [Dalvi et al., 2019, Torroba Hennigen et al., 2020], where we expose the classiﬁer to a subvector of the representation and evaluate how well it predicts the task.
However, while previous work conﬂated rankings and classiﬁers, we are more careful: we separate the two, and pair each ranking with two classiﬁers, meaning that at least one of them is completely unrelated to the ranking.
Formally, for an increasing k ∈ N , we train a classiﬁer f θ : H k
→ Z with parameters θ to predict the task label, F ( h ) , solely from h Π( d ) [ k ] (the subvector of the representation h in the top k neurons in ranking Π ), ignoring the rest of the dimensions.
2 The general assumption is that the better f θ performs, the more task-relevant information is encoded in h Π( d ) [ k ] .
Yet, the behaviour of f θ itself might affect results and conclusions about the ranking.
Hence, we aim for a ﬁne-grained analysis of our results, and also measure selectivity (§ 3.1.1).
3.1 Experimental Setup
As classiﬁers, we experiment with both classiﬁers used by the ﬁrst two ranking methods (§ 2.1.1, 2.1.2).
As rankings, we experiment with the 3 ranking methods described in 2.1.
For each, we use the original ranking it produces and its reversed version, and refer to them as top-to-bottom and bottom-to-top, respectively.
To those we add a random ranking baseline, ending up with 7 different rankings overall.
We compare all classiﬁer-ranking combinations for the same k .
Since both of the ﬁrst two ranking methods are inherently tied to the classiﬁer that was used to generate them, and the third ranking is a classiﬁer-neutral ranking, it can be used for a fair comparison between the classiﬁers.
3.1.1 Metrics
Accuracy First, we measure the accuracy of the probe’s predictions.
Since we experiment with many different conﬁgs, we use the Wilcoxon signed-rank test
[Wilcoxon, 1992] as a statistical signiﬁcance test to determine whether a certain combination of a classiﬁer and a ranking is statistically signiﬁcantly better than another combination.
Selectivity We also evaluate our probes by selectivity
[Hewitt and Liang, 2019], which is deﬁned as the difference between the classiﬁer’s accuracy on the actual probing task and its accuracy on predicting random labels assigned to word types, called a control task.
Low selectivity implies that the probe is capable of memorizing the word-type–label pair, and so high accuracy in the probing task does not necessarily entail the presence of the linguistic attribute.
Thus, we prefer probes that are both accurate and selective.
3.2 Results
Fig.
1a shows a t-SNE [van der Maaten and Hinton, 2008] projection after performing K-means clustering on our 156 accuracy results from the conﬁgs we tested, where each point represents accuracy results from one conﬁg (details on clustering procedure are in Appendix A.2).
It shows three clusters of conﬁgs, which correspond to three distinct patterns, demonstrated in Figs.
1b-1d.
In these ﬁgures, each color represents a combination of a classiﬁer and a ranking, where a solid line is used for the top-to-bottom version of the ranking and a dotted line is for the bottom-to-top version of it, and a dashed line is used for random ranking.
Almost half of the conﬁgs follow the Standard pattern (Fig. 1b), in which all top-to-bottom rankings are always better than the random ranking, which is always better than all bottom-to-top rankings.
In [Figure] the G>L pattern (Fig. 1c), the G AUSSIAN classiﬁer performs exceptionally well, providing higher accuracy (after a certain point) using a random or even a bottom-to-top ranking, than the L INEAR classiﬁer using its top-to-bottom ranking. In the L>G pattern (Fig. 1d), the G AUSSIAN classiﬁer fails quickly, and thus the L INEAR classiﬁer provides higher accuracy using a random or bottom-to-top ranking than the G AUSSIAN classiﬁer using its top-to-bottom ranking.
We now turn to analyze the patterns we observe.
3.2.1 Ranking methods are inherently consistent
In most conﬁgs, each classiﬁer provides better accuracy using a top-to-bottom ranking (solid lines) compared to the bottom-to-top version of the same ranking (same color, dotted line), and the random ranking (dashed lines) is in between.
This is also seen in our statistical signiﬁcance tests (Appendix A.3).
We conclude that even if they are not optimal, all ranking methods we consider generally rank task-informative neurons higher than non-informative ones.
3.2.2 Which classiﬁer is better?
In the Standard and G>L patterns (Figs.
1b, 1c), G AUSSIAN achieves better accuracy than L INEAR when both of them use the same ranking (including top-to-bottom L INEAR ), especially when using small sets of neurons.
Our statistical signiﬁcance tests (Appendix A.3) show that G AUSSIAN performs signiﬁcantly better than L INEAR with 6 out of the 7 rankings we tried when using 10 neurons, with 5 rankings when using 50 neurons, and with 5 rankings when using 150 neurons.
We now turn to analyze what makes G AUSSIAN more successful, and show some exceptions.
G AUSSIAN is memorizing Across all conﬁgs, L INEAR provides higher selectivity than G AUSSIAN using any ranking, after a certain point (Appendix A.4).
This means that G AUSSIAN tends to memorize the word-type–label pair when solving the task.
While this is apparent in all conﬁgs, we note that speciﬁcally in the part-of-speech attribute there is a large portion of function words, i.e. closed set labels (e.g., pronouns, determiners), meaning that memorization can signiﬁcantly help solve the task.
Thus, most conﬁgs involving part of speech belong to the G>L pattern.
L INEAR is more stable On the other hand, pattern L>G shows that there are certain conﬁgs where
G AUSSIAN is struggling to model the distribution, resulting in mediocre accuracy results—which even start decreasing at some point—sometimes even below majority baseline, as seen in Fig.
1d.
This has also been mentioned in Torroba Hennigen et al.
We hypothesize that it happens because in those conﬁgs, there are only a few (or no) dimensions that are informative for the attribute and are Gaussian-distributed.
Thus, the G AUSSIAN classiﬁer tries to model these distributions with the wrong tools, and fails.
Poor modeling then leads to wrong predictions and low accuracy.
In general, L INEAR behaves similarly across conﬁgs, making it more stable.
3.2.3 Which ranking is better?
When looking at rankings, we would like to compare performance of the same classiﬁer, using different rankings.
We would expect that each classiﬁer would perform best when using the ranking it has generated.
However, this is not always the case.
As we can see in all patterns in Fig.
1, for small sets of neurons, L INEAR actually achieves better accuracy when using G AUSSIAN ’s ranking (solid green) than its own ranking (solid orange).
As the number of neurons increases, at some point its accuracy with its own ranking becomes higher than with G AUSSIAN ’s ranking.
We suggest two explanations for this phenomenon: First, due to its greediness, the G AUSSIAN ranking is not guaranteed to provide the optimal subset.
For a subset of size 1, it goes over all possibilities, but as the size grows there are more subsets that are not taken into consideration in the algorithm, so it is more likely to miss the best sets.
Second, G AUSSIAN assumes the embedding distribution to be Gaussian.
On dimensions which are not Gaussian-distributed, it makes a less accurate evaluation of the contribution of each neuron.
So, if a neuron is informative towards the attribute but is not Gaussian-distributed, its addition to the selected neurons set is unlikely to improve performance, and thus it is not selected.
This is a problem with a performance-based selection criterion, where the selection of neurons depends on the performance of the probe.
To summarize, it seems that G AUSSIAN is good at selecting speciﬁc informative neurons, but misses the rest.
While L INEAR ’s ranking is not optimal (it is deﬁnitely worse then G AUSSIAN ’s on small sets), it does seem to be more stable on different sizes.
P ROBELESS provides decent performance (and is inherently consistent), but is usually behind the other two.
Interventions
The variance of results in our probing experiments can mostly be attributed to probing limitations [Hewitt and Liang, 2019, Pimentel et al., 2020, Belinkov, 2021, Ravichander et al., 2021], and emphasizes the need to distinguish between two properties: the probe’s classiﬁcation quality, and the neuronranking quality.
To isolate the latter, and to shed light on which ranking prefers neurons that are actually being used by the model, we take a second ranking-evaluation approach: we intervene by modifying the representation in the neurons selected by the ranking, and observe how (and if) our intervention affects the language model output.
This approach is more of a causal one, inspired by similar prior work [Giulianelli et al., 2018, Elazar et al., 2021, Feder et al., 2020, Vig et al., 2020, Dai et al., 2021], and can provide neuron-level explanations to some of the model’s decisions.
We note that in this section, we use only the ranking itself, detaching it from any probes, thus removing classiﬁcation quality from ranking comparisons.
Formally, for a representation h ∈ H , ranking Π( d ) (corresponding to an attribute F ) and an increasing k ∈ N , we intervene by modifying h only in the Π( d ) [ k ] neurons, and observe the effect our intervention had on the model’s output—the word prediction (given the modiﬁed representation).
We divide the model to two components: an encoder E : V → H and a decoder D : H → V , such that for interventions in layer i , E is composed of all the layers of the model up to (including) i , and
D is composed of all of the rest of the layers.
After receiving a representation h = E ( w ) for word w , we intervene and modify h to get a new representation h (cid:48) .
If D ( h ) (cid:54) =
D ( h (cid:48) ) , then D is using the information we modiﬁed.
However, knowing that the information is being used is not enough; we would like to know to what purpose it is being used, and to verify that it only affects the speciﬁc attribute we are interested in.
Thus, we perform a ﬁner-grained analysis, and check if D ( h (cid:48) ) is similar, to some extent, to
D ( h ) .
For that, we deﬁne a lemmatizer L : V → V , which maps words to their lemmas, and an analyzer A : V → Z , which maps words to their task labels.
Our goal is to intervene such that L ( D ( h ))
=
L ( D ( h (cid:48) )) , but A ( D ( h )) (cid:54) =
A ( D ( h (cid:48) )) .
If this is the case, it implies that we have successfully identiﬁed where the task-relevant information that D uses is encoded, and how it is being used.
4.1 Intervention Methods
We consider two methods for modifying h π ( d ) k , and compare them.
Ablation A common modiﬁcation method is trying to remove the information [Ravfogel et al., 2020] by ablating some neurons [Bau et al., 2019, Lakretz et al., 2019], meaning we set h π ( d ) k = 0 .
By that we aim to erase the information encoded in h π ( d ) k .
Translation For a word w ∈ V with attribute label z ∈ Z , we attempt to translate its representation (in the geometric sense) to produce a word with attribute label z (cid:48)
∈ Z, z (cid:54) =
z (cid:48) by taking a step in the direction of z (cid:48) , where bigger steps are applied to neurons that are marked as more important for the attribute.
Formally, we apply the following protocol: We calculate q ( z ) and q ( z (cid:48) ) as in eq.
We set [Equation]
where α ∈ R d is a log -scaled coefﬁcients vector in the range [0 , β ] , such that the coefﬁcient of the highest-ranked neuron is β and that of the lowest-ranked neuron is 0 , and β is a hyperparameter.
Note that the rest of the neurons—those not in Π( d ) [ k ] —remain unaffected.
Using this protocol, we give each neuron its own special treatment—an approach that was not applied before (as far we know).
This can be seen as a generalization of Gonen et al.
4.2 Experimental Setup
We handle the data the same way as in our probing experiments.
However, since we analyze the model’s predictions—which may be different from the original input—we do not have gold morphology labels anymore.
Thus, for morphologically analyzing the model’s predictions ( L and A ), we use spaCy [Honnibal et al., 2020].
Out of the languages we used in our probing experiments, in this section we are limited only to those that are supported by spaCy (English, Spanish and French).
We calculate q ( z ) based on the entire training set, and perform our interventions on the test set.
We compare the same 7 rankings we used in our probing experiments (§ 3.1).
4.3 Metrics
Error Rate For our intervention experiments, we ﬁrst measure the error rate of the language model.
We want error rate to be high, since high error rate means we modiﬁed parts of the representation that have been used by the model in its prediction.
Correct Lemma, Wrong Value (CLWV)
While inspecting predictions that are wrong after intervening ( M ( h (cid:48) ) (cid:54) =
w , where w is the true word), we categorize them by L ( M ( h (cid:48) )) and A ( M ( h (cid:48) )) .
If our intervention were successful, meaning we changed only the word’s speciﬁc attribute, and not other information, then we expect to see L ( M ( h ))
=
L ( M ( h (cid:48) )) and A ( M ( h )) (cid:54)
=
A ( M ( h (cid:48) )) .
For example, if the word “makes” becomes “made” when intervening for tense, then it counts as a L INEAR G AUSSIAN P ROBELESS
Englishtense 0 .
Spanishnumber 0 .
Spanishgender 0 . [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
CLWV, but if it becomes “make” or “prepared” it does not.
Thus, we deﬁne CLWV as the portion of those errors out of all predictions.
4.4 Results 4.4.1 Ablation is not Effective
Generally, across most languages, attributes and rankings we use, about 400 neurons from layerand 200–300 neurons from layers 7 and 12 can be ablated without any implications on the output, meaning error rate remains the same; an example is shown in Appendix A.5.
Moreover, when we do encounter more errors made by the model, our analysis does not reveal any pattern among those errors, i.e. CLWV is very low.
By qualitatively analyzing those errors we see that most predicted words are mostly simply common words, e.g., “and”, “if” in English.
After ablating 600–700 (80%–90%) neurons from the representations, we observe a lot of errors, but most of them are because the word is predicted as nonsensical punctuation.
One major concern is that in these experiments, in some conﬁgs ablating by a bottom-to-top ranking provides better results than by the top-to-bottom version of the same ranking.
In general, there are no distinct differences between the rankings.
Thus, from now on we focus on translation rather than ablation.
4.4.2 Translation is Effective
Across all translation experiments (Fig.
2a shows one example), CLWV increases until a certain saturation point, after which it remains constant or drops a little 4 .
This means that we reached neurons that are not relevant for the attribute, and modifying them can result in loss of other information—error rate grows while CLWV does not.
Thus, we are interested in the CLWV value at the saturation point (higher is better), and in the number of neurons modiﬁed at the saturation point (lower is better).
We report these values in a number of conﬁgs in Table 2b, and the rest of the conﬁgs in Appendix A.6.
Compared to ablation, translating a relatively small number of neurons results in a higher error rate, and these errors are closer to what we would expect.
For example, translating only 50 neurons in Spanish gender layer 2 results in 37% CLWV and 49% error rate, while ablating 50 neurons from the same conﬁg and ranking gives no CLWV errors and only 1% of error rate.
We further note that unlike in ablation experiments, here our rankings are inherently consistent, i.e. across all conﬁgs, all top-to-bottom rankings perform better than the rest, while random ranking’s error rate sometimes increases a little, and bottom-to-top rankings do not manage to affect the model’s output at all (Fig. 2a).
4.4. P ROBELESS is the Most Effective Ranking for Interventions
A clear trend from our results (Tables 2b, 2 and Fig. 2a) is that in most cases, P ROBELESS achieves higher CLWV values, and does so using a smaller number of neurons, than the other two rankings.
Furthermore, its error rate is signiﬁcantly higher than the other two.
This implies that P ROBELESS tends to select neurons that are being used by the model, more so than the other rankings.
However, while it does select neurons that are relevant for the attribute in question (CLWV is relatively high), it also tends to select neurons that are used by the model for other kinds of attributes (the difference between error rate and CLWV is relatively high).
Among L INEAR and G AUSSIAN , L INEAR seems to have the upper hand, with higher CLWV values in most conﬁgs.
This provides another evidence that the superiority of G AUSSIAN in the probing experiments may be due to the quality of its classiﬁer, and speciﬁcally its memorization ability, rather than the quality of the ranking it produces, as here only the ranking affects results.
Discussion and Conclusion
In this work, we compare different methods for ranking neurons according to their importance for a morphological attribute.
We show that to evaluate a ranking in a probing scenario, one should separate between the ranking itself and the quality of the classiﬁer that is using the ranking.
We also show the need to evaluate a ranking using causal methods, such as interventions, especially for applications such as controlling or debugging the model.
We propose a new ranking method that relies solely on the data, without training any auxiliary classiﬁer, and show that it is valid, and prefers neurons that are being used by the model, more so than other ranking methods.
We also propose a method for intervening within the model’s representations such that it transforms the output in a desired way.
5.1 Encoded Information and Used Information are not the Same
When modifying the selected neurons, in most conﬁgs, top-to-bottom P ROBELESS affects the output more than top-to-bottom G AUSSIAN and top-to-bottom L INEAR , and does so using a smaller number of neurons.
This is in contrast to our probing results, where P ROBELESS rarely overcame the other two, implying that encoded information and used information are not the same, and high probing accuracy does not necessarily entail that the information is actually important for the model.
This ﬁnding is in line with previous work [Elazar et al., 2021, Ravichander et al., 2021].
5.2 Individual Neurons are Important
In our intervention experiments, modifying too many neurons results in more errors that are not related to the true word.
This proves the importance of looking into individual neurons, especially when trying to intervene in the inner workings of the model.
For example, Gonen et al.
[2020] try to change the language of a word by intervening with the representation, using the same translation method we use, but with the same coefﬁcient for every neuron, and on the entire representation.
Our results imply that they may get better results by using our method.
Acknowledgments
We thank Lucas Torroba Hennigen for his helpful comments.
This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and by an Azrieli Foundation Early
Career Faculty Fellowship.
YB is supported by the Viterbi Fellowship in the Center for Computer
Engineering at the Technion. A Appendix A.1 Data Preparation
We remove any sentences that would have a sub-token length greater than 512 , the maximum allowed for M-BERT, the language model we use for generating representations.
As in Torroba Hennigen et al.
[2020], we remove attribute labels that are associated with less than 100 word types in any of the data splits.
This mostly removes function words, and we found it makes it harder for probes to use memorization for solving the task.
A.2 Clustering probing results
For each conﬁg out of the 156 we experimented with, we have results of 14 classiﬁer–ranking combinations, each of length 150, the max k (number of neurons) we used.
For clustering these results, we ﬁrst remove all combinations involving a bottom-to-top ranking, as these add a lot of noise to the clustering algorithm, making it focus on irrelevant signals.
Thus, our results matrix is of shape [156 , 8 , 150] .
We then reshape the matrix to shape [156 , 8 × 150] and run K-means over it with
K = 3 .
Projecting the K-means output with t-SNE gives us Fig. 1a.
A.3 Statistical Signiﬁcance Tests
Table 1 shows the results of our statistical signiﬁcance tests.
The three rows in each cell correspond to using 10, 50 and 150 neurons.
If there is an * in the [ i, j ] cell, is means that the p -value under the null hypothesis that probe j is better than probe i is lower than 0 .
05 , when using the matching number of neurons.
For example, we see that there is an * in the ﬁrst and second rows in the [0 , 3] cell, meaning we can conﬁdently reject the hypothesis that L INEAR by L INEAR is better than G AUSSIAN by G AUSSIAN when using 10 or 50 neurons, but we cannot do so for 150 neurons.
In fact, looking at the [3 , 0] cell shows us that when using 150 neurons, G AUSSIAN by G AUSSIAN is not better than
L INEAR by L INEAR .
While we do not show random and bottom-to-top rankings in Table 1 for clarity, we asserted that each classiﬁer is statistically signiﬁcant better when using a top-to-bottom ranking compared to a random ranking, and when using a random ranking compared to a bottom-to-top ranking. A.4 Probing Selectivity A selectivity example is provided in Fig.
In all conﬁgs, L INEAR is signiﬁcantly more selective than G AUSSIAN using any ranking.
A.5 Ablation Results
One ablation example is shown in Fig.
No matter the ranking, ∼ 400 neurons can be ablated with little impact on the output, and CLWV remains low. A.6 Translation results
The rest of the translation results (complementing Table 2b) is found in Table 2.
G by ttb G L by ttb G G by ttb L L by ttb L
G by ttb P
L by ttb P G by ttb G — * * * * * * * * * * * * * L by ttb G — * * * * * * G by ttb L * — * * * *
L by ttb L * * * * — * * * *
G by ttb P * * — *
L by ttb P * * * —
[Figure]
L INEAR
[Table]
AUSSIAN
[Table]