O N THE P ITFALLS OF A NALYZING I NDIVIDUAL N EURONS IN L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is encoded.
Among these, the common approach is to use an external probe to rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it.
We show two pitfalls in this methodology: 1.
It confounds distinct factors: probe quality and ranking quality.
We separate them and draw conclusions on each.
It focuses on encoded information, rather than information that is used by the model.
We show that these are not the same.
We compare two recent ranking methods and a simple one we introduce, and evaluate them with regard to both of these aspects.
I NTRODUCTION
Many studies attempt to interpret language models by predicting different linguistic properties from word representations, an approach called probing classiﬁers (Adi et al., 2017; Conneau et al., 2018, inter alia ).
A growing body of work focuses on individual neurons within the representation, attempting to show in which neurons some information is encoded, and whether it is localized (concentrated in a small set of neurons) or dispersed.
Such knowledge may allow us to control the model’s output (Bau et al., 2019), to reduce the number of parameters in the model (Voita et al., 2019; Sajjad et al., 2020), and to gain a general scientiﬁc knowledge of the model.
The common methodology is to train a probe to predict some linguistic attribute from a representation, and to use it, in different ways, to rank the neurons of the representation according to their importance for the attribute in question.
The same probe is then used to predict the attribute, but using only the k -highest ranked neurons from the obtained ranking, and the probe’s accuracy in this scenario is considered as a measure of the ranking’s quality (Dalvi et al., 2019; Torroba Hennigen et al., 2020; Durrani et al., 2020).
We see this framework as exhibiting Pitfall I : Two distinct factors are conﬂated—the probe’s classiﬁcation quality and the quality of the ranking it produces.
A good classiﬁer may provide good results even if its ranking is bad, and an optimal ranking may cause an average classiﬁer to provide better results than a good classiﬁer that is given a bad ranking.
Another shortcoming of the current methodology, which we mark as Pitfall II , is the focus on encoded information, regardless of whether it is actually used by the model in its language modeling task.
A few studies (Elazar et al., 2021; Feder et al., 2020) have considered this question, and shown that encoded information is not necessarily being used for language modeling, but these do not look at individual neurons.
We argue that in order to evaluate a ranking, one should also examine if, and how, the k -highest ranked neurons are used by the model for the attribute in question, meaning that modifying them would change the model’s prediction—but with respect to that attribute only.
This would allow some control over the model’s output, and grant us parameter-level explanations of the model’s decisions.
In this work, we analyze three neuron ranking methods.
Since the ranking space is too large ( 768! in
BERT’s case)
, these methods provide approximations to the problem and are non-optimal.
Two of these methods—L INEAR (Dalvi et al., 2019) and G AUSSIAN (Torroba Hennigen et al., 2020)—rely on an external probe to obtain a ranking: the ﬁrst makes use of the internal weights of a linear probe, while the second considers the performance of a decomposable generative probe.
The third is a simple ranking method we propose, P ROBELESS , which ranks neurons according to the difference in their values across labels, and thus can be derived directly from the data, with no probing involved.
We experiment with disentangling probe quality and ranking quality, by using a probe from one method with a ranking from another method, and comparing the different probe–ranking combinations.
We expose the problematic nature of the current methodology (Pitfall I), by showing that in some cases, a suitable probe which is given an intentionally bad ranking, or a random one, provides higher accuracy than another which is given its allegedly optimal ranking.
We ﬁnd that while the G AUSSIAN method generally provides higher accuracy, its probe’s selectivity (Hewitt & Liang, 2019) is lower, implying that it performs the probing task by memorizing, which improves probing quality but not necessarily ranking quality.
We further ﬁnd that G AUSSIAN provides the best ranking for small sets of neurons, while L INEAR provides a better ranking for large sets.
We then turn to analyzing which ranking selects neurons that are used by the model, by applying interventions on the representation: we modify subsets of neurons from each ranking and measure— using a novel metric we introduce—the effect on language modeling w.r.t to the property in question.
We highlight the need to focus on used information (Pitfall II): even though P ROBELESS does not excel in the probing scenario, it selects neurons that are used by the model, more so than the two probing-based rankings.
We ﬁnd that there is an overlap between encoded information and used information, but they are not the same, and argue that more attention should be given to the latter.
We primarily experiment with the M-BERT model (Devlin et al., 2019) on 9 languages andmorphological attributes, from the Universal Dependencies dataset (Zeman et al., 2020).
We also experiment with XLM-R (Conneau et al., 2020), and ﬁnd that most of our results are similar between the models, with a few differences which we discuss.
Our experiments reveal the following insights:
• We show the need to separate between probing quality and ranking quality, via cases where intentionally poor rankings provide better accuracy than good rankings, due to probing weaknesses.
• We present a new ranking method that is free of any probes, and tends to prefer neurons that are being used by the model, more so than existing probing-based rankings.
• We show that there is an overlap between encoded information and used information, but they are not the same.
N EURON RANKINGS AND DATA
We begin by introducing some notation.
Denote the word representation space as H ⊆ R d and an auxiliary task as a function F : H → Z , for some task labels Z
(e.g., part-of-speech labels).
Given a word representation h ∈ H and some subset of neurons S ⊆ { 1 , ..., d } , we use h S to denote the subvector of h in dimensions S .
For some auxiliary task F and k ∈ N , we search for an optimal subset
S ∗ such that | S ∗ | =
k and h S ∗ contains more information regarding F than any other subvector h S (cid:48) , | S (cid:48)
| =
k .
For the search task, we deﬁne neuron-ranking as a permutation Π( d ) on { 1 , ..., d } and consider the subset Π( d ) [ k ] =
{ Π( d ) 1 , ..., Π( d ) k } .
One may wish to ﬁnd an optimal ranking Π ∗ ( d ) such that ∀ k, Π ∗ ( d ) [ k ] is the optimal subset with respect to F .
However, ﬁnding an optimal ranking, or even an optimal subset, is NP-hard (Binshtok et al., 2007).
Thus, we focus on several methods to produce rankings, which provide approximations to the problem, and compare them.
2. R ANKINGS
The ranking methods we compare include two rankings obtained from prior probing-based neuronranking methods, and a novel ranking we propose, based on data statistics rather than probing.
L INEAR
The ﬁrst method, henceforth L INEAR (named linguistic correlation analysis in Dalvi et al.
2019), 2 trains a linear classiﬁer on the representations to learn the task F .
Then, it uses the trained classiﬁer’s weights to rank the neurons according to their importance for F .
Intuitively, neurons with a higher magnitude of absolute weights should be more important, or contain more relevant information, for solving the task.
Dalvi et al.
(2019) showed that their method identiﬁes important neurons through probing and ablation studies, and found that while the information is distributed across neurons, the distribution is not uniform, meaning it is skewed towards the top-ranked neurons.
In this work, we use a slightly modiﬁed version of the suggested approach (Appendix A.1).
G AUSSIAN
The second method, henceforth G AUSSIAN (Torroba Hennigen et al., 2020), trains a generative classiﬁer on the task F , based on the assumption that each dimension in { 1 , ..., d } is Gaussian-distributed.
Then, it makes use of the decomposability of the multivariate Gaussian distribution to greedily select the most informative neuron, according to the classiﬁer’s performance, at every iteration.
This way we obtain a full neuron ranking after training only once, while applying this greedy method to L INEAR would require retraining the probe d ! times, which is clearly infeasible.
Torroba Hennigen et al.
(2020) found that most of the tasks can be solved using a low number of neurons, but also noted that their classiﬁer is limited due to the Gaussian distribution assumption.
P ROBELESS
The third neuron-ranking method we experiment with is based purely on the representations, with no probing involved, making it free of probing limitations (Belinkov, 2021) that might affect ranking quality.
For every attribute label z ∈ Z , we calculate q ( z ) , the mean vector of all representations of words that possess the attribute and the value z .
Then, we calculate the element-wise difference between the mean vectors,
[Equation]
and obtain a ranking by arg-sorting r , i.e., the ﬁrst neuron in the ranking corresponds to the highest value in r .
For binary-labeled attributes, this is simply the difference in means.
In the general case,
P ROBELESS assigns high values to neurons that are most sensitive to a given attribute.
2. D ATA AND MODELS
Throughout our work, we follow the experimental setting of Torroba Hennigen et al.
(2020): we map the UD treebanks (Zeman et al., 2020) to the UniMorph schema (Kirov et al., 2018) using the mapping by McCarthy et al.
We select a subset of the languages used by Torroba Hennigen et al.
(2020): Arabic, Bulgarian, English, Finnish, French, Hindi, Russian, Spanish and Turkish, to keep linguistic diversity.
The tasks we experiment with are predictions of morphological attributes from these languages.
Full data details are provided in Torroba Hennigen et al.
and further data preparation steps are detailed in Appendix A.2.
We process each sentence in pre-trained M-BERT and XLM-R (unless stated otherwise, all results are with M-BERT), and take word representations from layers 2, 7 and 12 of each model, to see if there are different patterns in the beginning, middle and end of the models.
We end up with a total of 156 different conﬁgs (language × attribute × layer) to test for each model.
For words that are split during tokenization, we deﬁne their ﬁnal representation to be the average over their sub-token representations.
Thus, each word has one representation for each layer, of dimension d = 768 .
We do not mask any words throughout our work.
2. O VERLAPS
Before evaluating our rankings in different scenarios, we ﬁrst characterize them by looking at the 100-highest ranked neurons (out of 768) from different rankings, across different conﬁgs.
Some neurons are important for an attribute across languages Since we work with multilingual models, we expect to see overlap in the selected neurons for one attribute across different languages.
Fig.
1 shows that for P ROBELESS this is indeed the case, as some attributes share a large number of important neurons across languages.
For example, number in Spanish and number in French share 70 of their 100 most important neurons, where the expected number for overlap of two random selections of neurons is only 13 (Appendix A.3).
Compared to the other two rankings (Appendix A.4),
P ROBELESS is the most consistent across languages, while G AUSSIAN rarely shows consistency, which may be a weakness.
Some neurons are unanimously important By looking at the overlaps between important neurons selected by different rankings for the same conﬁg, we observe that for all conﬁgs, the overlap between all three rankings surpasses the expected number (which is ∼ 1 . 69 ; Appendix A.3), meaning there are neurons that are recognized by all three rankings as important.
We further see that in most cases, the greatest overlap is between L INEAR and P ROBELESS .
We ﬁnd it reasonable, as both of them aim to select neurons that separate classes the best—one by a classiﬁer and the other by data statistics—while G AUSSIAN takes a different approach, assuming a Gaussian distribution and selecting neurons only by performance.
Examples from 8 conﬁgs are shown in Fig.
There are more overlaps in XLM-R
Performing the same analysis across languages on XLM-R (Appendix A.4), the overlap size is at least as the expected one between all conﬁg pairs, and is usually greater.
It may imply that XLM-R’s representations can be pruned more easily than M-BERT’s, since some neurons encode multiple attributes, and there is greater redundancy among the others.
P ITFALL I: C LASSIFIERS VS . RANKINGS
We now turn to evaluating the rankings, and present the pitfalls in doing so.
Given some ranking Π( d ) , we would like to evaluate how well it sorts the neurons for the task F .
Our ﬁrst ranking-evaluation approach is the standard probing approach from previous work (Dalvi et al., 2019; Torroba Hennigen et al., 2020), where we expose the classiﬁer to a subvector of the representation and evaluate how well it predicts the task.
However, while previous work conﬂated rankings and classiﬁers—Pitfall I—we are more careful: we separate the two, and pair each ranking with two classiﬁers, meaning that at least one of them is completely unrelated to the ranking.
Formally, for an increasing k ∈ N , we train a classiﬁer f : H k
→ Z to predict the task label,
F ( h ) , solely from h Π( d )
[ k ] (the subvector of the representation h in the top k neurons in ranking Π ), ignoring the rest of the neurons.
The assumption is that the better f performs, the more task-relevant information is encoded in h Π( d ) [ k ] .
Yet, the behaviour of f itself might affect results and conclusions about the ranking.
Thus, we take classiﬁer capabilities into consideration when analyzing results, and also measure selectivity (§3.1.1).
3. E XPERIMENTAL SETUP
As classiﬁers, we experiment with both classiﬁers used by the ﬁrst two ranking methods (L INEAR and G AUSSIAN ).
We use the hyperparameters reported in Durrani et al.
and Torroba Hennigen et al.
(2020) for training the classiﬁers.
As rankings, we experiment with the 3 ranking methods described in §2.1.
For each, we use the original ranking it produces and its reversed version, referred to as top-to-bottom and bottom-to-top, respectively.
To those we add a random ranking baseline, resulting in 7 different rankings overall.
We compare all classiﬁer–ranking combinations for each k .
Since both of the ﬁrst two ranking methods are inherently tied to the classiﬁer that was used to generate them, and the third ranking is a classiﬁer-neutral ranking, it can be used for a fair comparison between the classiﬁers.
3.1. M ETRICS
Accuracy First, we measure the accuracy of the probe’s predictions.
Since we experiment with many different conﬁgs, we use the Wilcoxon signed-rank test (Wilcoxon, 1992) as a statistical signiﬁcance test to determine whether a certain combination of a classiﬁer and a ranking is statistically signiﬁcantly better than another combination.
Selectivity We also evaluate our probes by selectivity (Hewitt & Liang, 2019), deﬁned as the difference between the classiﬁer’s accuracy on the actual probing task and its accuracy on predicting random labels assigned to word types, called a control task.
Low selectivity implies that the probe can memorize the word-type–label pair, and so high accuracy in the probing task does not necessarily entail the presence of the linguistic attribute.
Thus, we prefer probes that are both accurate and selective.
3. R ESULTS
Across the 156 conﬁgs we experiment with, we observe three different accuracy patterns, demonstrated in Figs.
3a-3c.
In these ﬁgures, each color represents a combination of a classiﬁer and a ranking, where a solid line is used for the top-to-bottom version of the ranking and a dotted line is for the bottom-to-top version of it, and a dashed line is used for the random ranking.
Almost half of the conﬁgs follow the Standard pattern (Fig. 3a), in which all top-to-bottom rankings are always better than the random ranking, which is always better than all bottom-to-top rankings.
The other half consists of two surprising patterns, that demonstrate the inherent ﬂaws in this ranking-evaluation approach.
In the G>
L pattern (Fig. 3b), the G AUSSIAN classiﬁer performs exceptionally well, providing higher accuracy (after a certain point) using a random or even a bottom-to-top ranking, than the L INEAR classiﬁer using its top-to-bottom ranking.
In the L>G pattern (Fig. 3c), the G AUSSIAN classiﬁer fails quickly, and thus the L INEAR classiﬁer provides higher accuracy using a random or bottom-to-top ranking than the G AUSSIAN classiﬁer using its top-to-bottom ranking.
Fig.
3d shows a t-SNE (van der Maaten & Hinton, 2008) projection after performing K-means clustering on our 156 accuracy results, where each point represents accuracy results from one conﬁg (details on clustering procedure are in Appendix A.5).
It shows three clusters of conﬁgs, that correspond to the three distinct patterns.
On XLM-R we see very similar results, and most conﬁgs follow the same pattern in each model (Appendix A.7).
We now turn to analyze these results.
3.2. R ANKING METHODS ARE INHERENTLY CONSISTENT
In most conﬁgs, each classiﬁer provides better accuracy using a top-to-bottom ranking (solid lines) compared to the bottom-to-top version of the same ranking (same color, dotted line), and the random ranking (dashed lines) is in between.
This is also seen in our statistical signiﬁcance tests (Appendix A.6).
We conclude that even if they are not optimal, all ranking methods we consider generally rank task-informative neurons higher than non-informative ones.
3.2. W HICH CLASSIFIER IS BETTER ?
In the Standard and G>L patterns (Figs. 3a, 3b), G AUSSIAN achieves better accuracy than L INEAR when both of them use the same ranking (including top-to-bottom L INEAR ), especially when using small sets of neurons.
Our statistical signiﬁcance tests (Appendix A.6) show that G AUSSIAN performs signiﬁcantly better than L INEAR with 6 out of the 7 rankings we tried when using 10 neurons, with 5 rankings when using 50 neurons, and with 5 rankings when using 150 neurons.
We now turn to analyze what makes G AUSSIAN more successful, and show some exceptions.
G AUSSIAN is memorizing Across all conﬁgs, L INEAR provides higher selectivity than G AUSSIAN using any ranking, after a certain point (Appendix A.7).
This means that G AUSSIAN tends to memorize the word-type–label pair when solving the task.
While this is apparent in all conﬁgs, we note that speciﬁcally in the part-of-speech attribute there is a large portion of function words, i.e., closed set labels (e.g., pronouns, determiners), meaning that memorization can signiﬁcantly help solve the task.
Thus, most conﬁgs involving part of speech belong to the G>L pattern.
Since memorization is a trait of the classiﬁer and not of the ranking, this pattern demonstrates the problematic nature of the current ranking-evaluation approach, as the results are highly dependent on the probe.
L INEAR is more stable On the other hand, pattern L>G shows that there are certain conﬁgs where
G AUSSIAN is struggling to model the distribution, resulting in mediocre accuracy results—which
[Figure]
even start decreasing at some point—sometimes even below majority baseline, as seen in Fig.
3c.
This has also been mentioned in Torroba Hennigen et al.
(2020), where it was shown that in those conﬁgs, there are only a few (or no) dimensions that are informative for the attribute and are Gaussiandistributed.
Thus, the G AUSSIAN classiﬁer tries to model these distributions with the wrong tools, and fails.
Poor modeling then leads to wrong predictions and low accuracy.
In general, L INEAR behaves similarly across conﬁgs, making it more stable.
3.2. W HICH RANKING IS BETTER ?
When looking at rankings, we would like to compare performance of the same classiﬁer, using different rankings.
We would expect that each classiﬁer would perform best when using the ranking it has generated.
However, this is not always the case.
As we can see in all patterns in Fig.
3, for small sets of neurons, L INEAR actually achieves better accuracy when using G AUSSIAN ’s ranking (solid green) than its own ranking (solid orange).
As the number of neurons increases, at some point its accuracy with its own ranking becomes higher than with G AUSSIAN ’s ranking.
We suggest two explanations for this phenomenon: First, due to its greediness, the G AUSSIAN ranking is not guaranteed to provide the optimal subset.
For a subset of size 1, it goes over all possibilities, but as the size grows there are more subsets that are not taken into consideration in the algorithm, so it is more likely to miss the best sets.
Second, G AUSSIAN assumes the embedding distribution to be
Gaussian.
On dimensions which are not Gaussian-distributed, it makes a less accurate evaluation of the contribution of each neuron.
So, if a neuron is informative towards the attribute but is not
Gaussian-distributed, its addition to the selected neurons set is unlikely to improve performance, and thus it is not selected.
This is a problem with a performance-based selection criterion, where the selection of neurons depends on the performance of the probe.
To summarize, it seems that G AUSSIAN is good at selecting speciﬁc informative neurons, but misses the rest.
While L INEAR ’s ranking is not optimal (it is deﬁnitely worse then G AUSSIAN ’s on small sets), it does seem to be more stable on different sizes.
P ROBELESS provides decent performance (and is inherently consistent), but is usually behind the other two.
P ITFALL II: E NCODED INFORMATION VS . USED INFORMATION
The variance of results in our probing experiments can mostly be attributed to probing limitations (Hewitt & Liang,
Belinkov, 2021), and emphasizes the need to distinguish between two properties: the probe’s classiﬁcation quality, and the neuron-ranking quality.
To isolate the latter, and to shed light on which ranking prefers neurons that are actually used by the model for the attribute in question (which is ignored by previous work—Pitfall II), we take a second ranking-evaluation approach: we intervene by modifying the representation in the neurons selected by the ranking, and observe if, and how, our intervention affects the language model output.
This approach is more of a causal one, inspired by similar prior work (Giulianelli et al., 2018; Elazar et al., 2021;
Feder et al., 2020;
Lovering et al., 2021; Ravfogel et al., 2021).
We note that in this section, we use only the ranking itself, detaching it from any probes, thus removing classiﬁcation quality from ranking comparisons.
Formally, for a representation h ∈ H , ranking Π( d ) (corresponding to an attribute F ) and an increasing k ∈ N , we intervene by modifying h only in the Π( d ) [ k ] neurons, and observe the effect our intervention had on the model’s output—the word prediction (given the modiﬁed representation).
For vocabulary V , we divide the model to two components: E : V → H and D : H → V , such that for interventions in layer i , E is composed of all of the layers of the model up to (including) i , and
D is composed of all of the rest of the layers, including the classiﬁcation head.
After receiving a representation h = E ( w ) for word w ∈ V , we intervene and modify h to get a new representation h (cid:48) .
If D ( h ) (cid:54) = D ( h (cid:48) ) , then D is using the information we modiﬁed.
However, knowing that the information is being used is not enough; we would like to know to what purpose it is being used, and to verify that it only affects the speciﬁc attribute we are interested in.
Thus, we perform a ﬁner-grained analysis, and check if D ( h (cid:48) ) is similar, to some extent, to
D ( h ) .
For that, we deﬁne a lemmatizer L : V → V , which maps words to their lemmas, and an analyzer A : V → Z , which maps words to their task labels.
Our goal is to intervene such that
L ( D ( h )) =
L ( D ( h (cid:48) )) , but A ( D ( h )) (cid:54) =
A ( D ( h (cid:48) )) .
For example, if we intervene for tense, we would like the word “sleeps” to become “slept”.
If this is the case, it implies that we have successfully identiﬁed where the task-relevant information that D uses is encoded, and how it is being used.
4. I NTERVENTION METHODS
We consider two methods for modifying h π ( d ) k , and compare them.
Ablation A common modiﬁcation method is trying to remove the information by ablating some neurons (Morcos et al., 2018;
Bau et al., 2019; Lakretz et al., 2019), meaning we set h π ( d ) k = 0 .
By that we aim to erase the information encoded in h π ( d ) k .
Translation For a word w ∈ V with attribute
label z ∈ Z , we attempt to translate its representation (in the geometric sense) to produce a word with attribute label z (cid:48)
∈
Z, z (cid:54) =
z (cid:48) by taking a step in the direction of z (cid:48) , where bigger steps are applied to neurons that are marked as more important for the attribute.
Formally, we apply the following protocol:
We calculate q ( z ) and q ( z (cid:48) ) as in eq.
We set
[Equation]
where α ∈ R d is a log -scaled coefﬁcients vector in the range [0 , β ] , such that the coefﬁcient of the highest-ranked neuron is β and that of the lowest-ranked neuron is 0 , and β is a hyperparameter.
Note that the rest of the neurons—those not in Π( d ) [ k ] —remain unaffected.
Using this protocol, we give each neuron its own special treatment—an approach that was not applied before (as far we know).
This can be seen as a generalization of Gonen et al.
4. E XPERIMENTAL SETUP
We handle the data the same way as in our probing experiments.
However, since we analyze the model’s predictions—which may be different from the original input—we do not have gold morphology labels anymore.
Thus, for morphologically analyzing the model’s predictions ( L and
A ), we use spaCy (Honnibal et al., 2020).
Out of the languages we used in our probing experiments, in this section we use only those that are supported by spaCy (English, Spanish and French).
We calculate q ( z ) based on the entire training set, and perform our interventions on the test set.
We compare the same 7 rankings we used in our probing experiments (§ 3.1).
4. M ETRICS
Error rate For our intervention experiments, we ﬁrst measure the error rate of the language model.
We want error rate to be high, since high error rate means we modiﬁed parts of the representation that have been used by the model in its prediction.
Correct Lemma, Wrong Value (CLWV)
While inspecting predictions that are wrong after intervening ( D ( h (cid:48) ) (cid:54) =
w , where w is the true word), we categorize them by L ( D ( h (cid:48) )) and A ( D ( h (cid:48) )) .
If our intervention were successful, meaning we changed only the word’s speciﬁc attribute, and not other information, then we expect to see L ( D ( h ))
=
L ( D ( h (cid:48) )) and A ( D ( h )) (cid:54) =
A ( D ( h (cid:48) )) ; that is, correct lemma but wrong value (CLWV).
For example, if the word “makes” becomes “made” when intervening for tense, then it is considered as a correct type of error, but if it becomes “make” or “prepared” it does not.
Thus, we deﬁne CLWV as the portion of those errors out of all predictions.
4. R ESULTS 4.4. A BLATION IS NOT EFFECTIVE
Across most conﬁgs, about 400 neurons from layer 2 and 200–300 neurons from layers 7 and 12 can be ablated without any implications on the output, meaning error rate remains the same; an example is shown in Appendix A.8.
Moreover, when error rate does grow, CLWV is very low.
By qualitatively analyzing those errors we saw that most predicted words are common words, e.g., “and”, “if” in
English.
After ablating 600–700 (80%–90%) neurons from the representations, we observe a lot of errors, but most of them are because the word is predicted as nonsensical punctuation.
Another major concern is that in some conﬁgs, ablating by a bottom-to-top ranking provides better results than by the top-to-bottom version of the same ranking.
In general, there are no distinct differences between the rankings.
Thus, from here on we focus on translation rather than ablation.
4.4. T RANSLATION IS EFFECTIVE
Across all translation experiments (Fig.
4 shows one example, more are in Appendix A.9), CLWV increases until a certain saturation point, after which it remains constant or drops a little.
This means that we reached neurons that are not relevant for the attribute, and modifying them can result in loss of other information—error rate grows while CLWV does not.
Thus, we are interested in the CLWV value at the saturation point (higher is better), and in the number of neurons modiﬁed at the saturation point (lower is better).
We would also like the difference between the error rate and CLWV at the saturation point to be as small as possible.
All terms considered, we perform a sweep search on the values of β in the range [1 , 12] on a dev set.
We ﬁnd that low β values provide low CLWV, while high values provide higher CLWV but also widen the gap between error rate and CLWV.
We ﬁnd β
= 8 to be a balanced point, and thus report test results with β = 8 in three conﬁgs in Table 1, and the rest of the conﬁgs in Appendix A.9.
The results for XLM-R are given in Appendix A.10.
Compared to ablation, translating a relatively small number of neurons results in a higher error rate, and these errors are closer to what we would expect.
For example, translating only 50 neurons selected by P ROBELESS in Spanish gender layer 2 results in 37% CLWV and 49% error rate, while ablating 50 neurons from the same conﬁg and ranking gives 0% CLWV errors and only 1% of error rate.
We further note that unlike in ablation experiments, here our rankings are inherently consistent: across all conﬁgs, all top-to-bottom rankings perform better than the rest, while random rankings’ error rate sometimes increases a little, and bottom-to-top rankings do not manage to affect the model’s output at all (Fig. 4 is one example, more are in Appendix A.9).
[Table]
[Table] [Table]
P ROBELESS IS THE MOST EFFECTIVE RANKING FOR INTERVENTIONS
A clear trend from our M-BERT results (Table 1, Fig. 4 and Appendix A.9) is that in most cases,
P ROBELESS achieves higher CLWV values, and does so using a smaller number of neurons, than the other two rankings.
Furthermore, its error rate is signiﬁcantly higher than the other two.
This implies that P ROBELESS tends to select neurons that are being used by the model, more so than the other rankings.
However, while it does select neurons that are relevant for the attribute in question (CLWV is relatively high), it also tends to select neurons that are used by the model for other kinds of attributes (the difference between error rate and CLWV is relatively high).
Among L INEAR and
G AUSSIAN , L INEAR seems to have the upper hand, with higher CLWV values in most conﬁgs.
This provides another evidence that the superiority of G AUSSIAN in the probing experiments may be due to the quality of its classiﬁer, and speciﬁcally its memorization ability, rather than the quality of the ranking it produces, as here only the ranking affects results.
In XLM-R, it seems that L INEAR and
P ROBELESS both have the lead, with G AUSSIAN falling behind (Appendix A.10).
D ISCUSSION AND CONCLUSION
In this work, we show two pitfalls with the common approach for ranking neurons according to their importance for a morphological attribute, and compare different ranking methods that follow this approach.
We show that to evaluate a ranking in a probing scenario, one should separate between the ranking itself and the quality of the classiﬁer that is using the ranking—Pitfall I. While previous work concentrated on encoded information—Pitfall II—we show that it is not the same as information used by a model, by showing that G AUSSIAN is inferior in the interventions scenario, in contrast to our probing results.
This implies that high probing accuracy does not necessarily entail that the information is actually important for the model.
This conclusion is also present in prior work (Elazar et al., 2021; Feder et al., 2020; Ravfogel et al., 2021), but it has been largely neglected in studies of individual neurons via probes.
We propose a new ranking method that relies solely on the data, without training any auxiliary classiﬁer, and show that it is valid, and prefers neurons that are being used by the model, more so than other ranking methods.
We also propose a method for intervening within the model’s representations such that it transforms the output in a desired way.
In our intervention experiments, modifying too many neurons results in more errors that are not related to the true word.
This proves the importance of looking into individual neurons, especially when trying to intervene in the inner workings of the model.
For example, Gonen et al.
(2020) try to change the language of a word by intervening with the representation, using the same translation method we use, but with the same coefﬁcient for every neuron, and on the entire representation.
Our results imply that they may get better results by using our ﬁner-grained method.
Ethics statement Our work contributes to the effort of improving the interpretability of language models, and more generally of neural networks.
Better explanations and controls over a model’s outputs can ameliorate its fairness, for example in the case of gender bias: our intervention method can guide users on how to reduce such biases, by pointing to model components (neurons) responsible for gender and offering intervention methods to control model behavior w.r.t a particular property.
On the other hand, malicious actors could use such capability to increase discrimination.
Exposing the capabilities may also help develop defense mechanisms.
Reproducibility statement All our results are reproducible using the code repository we will release.
All experimental details, including hyperparameters, are reported in §3.1, 4.1 and 4.2.
As language models, we used the implementation of the transformers library (Wolf et al., 2020).
We performed our experiments on NVIDIA RTX 2080 Ti GPU.
All data preparation details are reported in §2.2 and Appendix A.2.
A A PPENDIX A. L INEAR METHOD MODIFICATION
In Dalvi et al.
(2019), after the probe has been trained, its weights are fed into a neuron ranking algorithm.
However, we observed that the original algorithm distributes the neurons equally among labels, meaning that each label would contribute the same number of neurons at each portion of the ranking, regardless of the amount of neurons that are actually important for this label.
For example, if for label A there are 10 important neurons and for label B there are only 2, then the ﬁrst 10 neurons in the ranking would consist of 5 neurons for A and 5 for B, meaning that 3 non-important neurons are ranked higher than 5 important ones.
Thus, we chose a different way to obtain the ranking: for each neuron, we compute the mean absolute value of the | Z | weights associated with it, and sort the neurons by this value, from highest to lowest.
In early experiments we found that this method empirically provides better results, and is more adapted to large label sets.
A. D ATA PREPARATION
We remove any sentences that would have a sub-token length greater than 512 , the maximum allowed for M-BERT, the language model we use for generating representations.
As in Torroba Hennigen et al.
(2020), we remove attribute labels that are associated with fewer than 100 word types in any of the data splits.
This mostly removes function words, and we found it makes it harder for probes to use memorization for solving the task.
The morphological attributes we experiment with include (in UniMorph annotations): Animacy, Aspect, Case, Deﬁniteness, Gender and Noun Class, Mood,
Number, Part of Speech, Person, Polarity, Possession, Tense and Voice.
A. E XPECTED OVERLAP BETWEEN RANDOM RANKINGS
Given i rankings, we calculate the expected size of overlap between the ﬁrst M neurons across all rankings:
For selecting M neurons from the range { 1 , ..., N } , let
C i ∈ R N × N × N be a matrix such that in
C i [ n, m, k ] we keep the number of possibilities to select m neurons from the range [ n ] such that exactly k different neurons are selected by all i rankings, where k ≤ m ≤ n and n, m, k > 0 .
For calculating C i [ n, m, k ] we ﬁrst select the k neurons from range [ n ] that are selected by all i rankings, thus (cid:0) nk (cid:1) possibilities.
Then, for selecting the rest of the neurons, each ranking has to select m − k neurons from the remaining n − k neurons, so there are (cid:0) n − k m − k (cid:1) i possibilities.
From these, we want to substract the number of possibilities in which there is at least one neuron that is selected by all rankings, which is (cid:80) m
− k j =1 C i [ n − k, m − k, j ] .
Concluding, we compute C i [ n, m, k ] by:
[Equation]
after initializing C [1 , 1 , 1] = 1 .
Then, we calculate the expected number of overlapping neurons by:
[Equation]
since C [ n,m,k ] ( nm ) i is the probability to have exactly k overlapping neurons.
We thus get E 2 (768 , 100) ≈ 13 .
02 and E 3 (768 , 100) ≈ 1 . 69 .
A. O VERLAPS
Fig.
5 shows overlaps between the 100 most important neurons chosen by L INEAR and G AUSSIAN for different conﬁgs.
Both of them provide less overlaps then P ROBELESS , with G AUSSIAN having almost no overlaps at all, showing its inconsistency across languages.
Fig.
6 presents the same analysis, but for XLM-R, with P ROBELESS ranking (equivalent to Fig.
1 for
M-BERT).
We see far more
[Figure]
A. C LUSTERING PROBING RESULTS
For each conﬁg out of the 156 we experimented with, we have results of 14 classiﬁer–ranking combinations, each of length 150, the max k (number of neurons) we used.
For clustering these results, we ﬁrst remove all combinations involving a bottom-to-top ranking, as these add a lot of noise to the clustering algorithm, making it focus on irrelevant signals.
Thus, our results matrix is of shape [156 , 8 , 150] .
We then reshape the matrix to shape [156 , 8 × 150] and run K-means over it with
K = 3 .
Projecting the K-means output with t-SNE gives us Figs. 3d and 7a.
A. S TATISTICAL SIGNIFICANCE TESTS
Table 2 shows the results of our statistical signiﬁcance tests.
The three rows in each cell correspond to using 10, 50 and 150 neurons.
If there is an * in the [ i, j ] cell, is means that the p -value under the null hypothesis that probe j is better than probe i is lower than 0 .
05 , when using the matching number of neurons.
For example, we see that there is an * in the ﬁrst and second rows in the [0 , 3] cell, meaning we can conﬁdently reject the hypothesis that L INEAR by L INEAR is better than G AUSSIAN by G AUSSIAN when using 10 or 50 neurons, but we cannot do so for 150 neurons.
In fact, looking at the [3 , 0] cell shows us that when using 150 neurons, G AUSSIAN by G AUSSIAN is not better than
L INEAR by L INEAR .
While we do not show random and bottom-to-top rankings in Table 2 for clarity, we asserted that each classiﬁer is statistically signiﬁcantly better when using a top-to-bottom ranking compared to a random ranking, and when using a random ranking compared to a bottom-to
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
A. P ROBING : XLM-R AND SELECTIVITY
Fig.
7 shows XLM-R probing results.
XLM-R provides very similar results to M-BERT, apparent in
Figs.
7a and 7b compared to Figs.
3d and 3c, respectively.
Selectivity examples from both models are provided in Fig.
In all conﬁgs, both in M-BERT and
XLM-R, L INEAR is signiﬁcantly more selective than G AUSSIAN using any ranking.
[Figure]
A. A BLATION RESULTS
One ablation example is shown in Fig.
No matter the ranking, ∼ 400 neurons can be ablated with little impact on the output, and CLWV remains low.
This behaviour is generally consistent across all conﬁgs we experimented with.
A. T RANSLATION RESULTS
All of M-BERT’s translation results (complementing Table 1) are found in Table 3, and examples from two conﬁgs are in Fig.
As described in §4.4.3, across most conﬁgs, P ROBELESS achieves higher CLWV at the saturation point, and gets there earlier (using less neurons), than the other two rankings—in contrast to probing results.
Among the probing-based rankings, L INEAR generally provides better results than G AUSSIAN .
We also note that there are certain attributes that seem harder to control for, e.g., English number and
French tense.
[Figure]
[Table]
Frenchgender 0 .
[Table]
A. XLM-R TRANSLATION RESULTS
XLM-R translation results (equivalent to Table 3 in M-BERT) are shown in Table 4.
The superiority of P ROBELESS is not so clear in XLM-R compared to M-BERT, with L INEAR providing good competition.
G AUSSIAN on the other hand, still falls behind.
We note that in XLM-R the CLWV values are somewhat lower compared to M-BERT.
A possible explanation to that could be the difference in tokenization between the models.