Investigating the Encoding of Words in BERT ’s Neurons Using Feature Textualization (2023)

Abstract: Pretrained language models (PLMs) form the basis of most state-of-the-art NLP technologies. Nevertheless, they are essentially black boxes: Humans do not have a clear understanding of what knowledge is encoded in different parts of the models, especially in individual neurons. A contrast is in computer vision, where feature visualization provides a decompositional interpretability technique for neurons of vision models. Activation maximization is used to synthesize inherently interpretable visual representations of the information encoded in individual neurons. Our work is inspired by this but presents a cautionary tale on the interpretability of single neurons, based on the first large-scale attempt to adapt activation maximization to NLP, and, more specifically, large PLMs. We propose feature textualization, a technique to produce dense representations of neurons in the PLM word embedding space. We apply feature textualization to the BERT model to investigate whether the knowledge encoded in individual neurons can be interpreted and symbolized. We find that the produced representations can provide insights about the knowledge encoded in individual neurons, but that individual neurons do not represent clear-cut symbolic units of language such as words. Additionally, we use feature textualization to investigate how many neurons are needed to encode words in BERT.

## introduction in recent years, research on explainable ai (xai) has seen an upsurge due to the black box nature of large neural models that are ubiquitously used in state-of-the-art systems. while being highly performant, due to their massive amount of parameters, it is inherently incomprehensible to humans which and how information is stored in these models. techniques in xai seek interpretations of large neural models and explanations for their behavior. lipton (2016) identifies post-hoc interpretability and transparency as crucial model properties that enable interpretations. post-hoc interpretations often do not reveal precisely how a model works, but they can provide useful information for end users. most local explanation methods fall into this category (see @xcite for a recent survey on post-hoc interpretations). model transparency on the other hand implies an understanding of how the model works. in this work we are primarily interested in model transparency at the level of decomposability, i.e., xai methods that provide intuitive interpretations for individual neurons. in theory, a perfectly transparent and decomposable global interpretation of a neural model could be achieved by providing faithful interpretations of all individual model neurons. in this work we attempt to find symbolizable and thus intuitive interpretations of individual language model neurons. to obtain interpretations of individual neurons, we build on feature visualization @xcite , an interpretability technique in computer vision that provides visualized, and thus interpretable, representations of individual neurons in vision models. in feature visualization, activation maximization @xcite is used to synthesize an input image that maximizes a predefined neuron's activation value. the task is phrased as an optimization problem and gradient ascent is used to iteratively optimize an artificial input with respect to the neuron's activation value. the underlying assumption is that the input image that maximally activates a neuron visualizes the kind of information that is encoded in that neuron. in computer vision, this technique has provided global interpretations of vision models, and has led to important insights: for instance, that certain neurons encode specific patterns such as stripes or textures in a picture @xcite . the openai microscope @xcite is a collection of visualizations of every significant layer and neuron of 13 important vision models, and an impressive demonstration of the relevance of feature visualization in vision models. in this work we introduce feature textualization, an adaptation of activation maximization applicable to language models. our method provides the -to our knowledge -first attempt at directly interpreting bert's neurons in a full white box manner without imposing structural constraints. our contributions with this work are three-fold: ## related work interpreting neurons in plms. many previous works have explored interpretation methods for nlp models through the investigation of individual neurons. in an attempt to find concept-level interpretations of individual neurons, multiple studies (e.g., @xcite , @xcite ) pass synthetic n-gram inputs to a model and try to pinpoint the concept that is encoded in the neuron under investigation by automatically extracting a theme across the most activating inputs. this methodology can be useful for neurons that encode multi-word concepts such as phrases @xcite . however, since the synthetic multi-word inputs are often ungrammatical, there is a risk of identifying a response to arbitrary behavior (like repetition) instead of concept specific behavior @xcite . avoiding this obstacle, @xcite use dataset samples as inputs to detect concepts encoded in individual neurons of bert. they fall pray to an interpretability illusion, as neurons initially seem to encode concepts within datasets, however across datasets these concepts are entirely unrelated. the authors conclude that the knowledge encoded in individual neurons is not reflected by any of the dataset-concepts. in @xcite , the neurons to be interpreted are not hand-picked in advance, but the behavior of all neurons is observed and meaningful neurons are chosen depending on their response to certain inputs. the authors compare the presence or absence of different concepts in the input with a binary activation mask on each neuron, i.e., is the neuron more or less activated than a threshold. they use correlations between concepts and neural activations to generate compositional explanations. @xcite use neuron activation values as prediction scores for concepts and determine concept knowledge within that neuron via the prediction accuracy. @xcite present an attempt at generating explanations of a language model's behavior using gpt-4 (openai, 2023), based on neuron activation patterns. unlike our work, they train an explainer model to generate explanations, effectively rendering the explanation process as a black-box method again: neuron activations are not interpreted directly, but based on a second model's prediction. to our knowledge, only two other attempts @xcite have previously been made to use activation maximization in the language domain. @xcite employ activation maximization to synthesize inputs to highly activate single neurons in a small joint vision and language model. unlike our work, they force synthesized inputs to correspond to onehot encodings, using the gumbel-softmax trick. this induces the strongest possible constraint and bias on the generation process of artificial inputs, as it forces inputs during optimization to converge to words or n-grams of words. this simplification makes it impossible to find an optimal input if it falls in between words or out of the embedding cone completely, a restriction that we lift with our work. bäuerle and wexler (2020) follows a comparable approach to @xcite . @xcite provides a comprehensive survey on interpretation attempts of individual neurons in nlp. ## experiments: single neurons this section describes our experiments of applying feature textualization to single neurons in bert, to determine whether we can find intuitive interpretations through symbolic units of language for these neurons. ## experiments: groups of neurons in the previous section, we report evidence suggesting that single neurons do not encode words. therefore, in this section, we use activation maximization to try to answer the question: how many neurons does it take to encode words? we present a set of experiments on optimizing an input for groups of neurons. ## discussion our experiments with feature textualization on single neurons indicate that the meaning of single neurons cannot be mapped to words. unregularized feature textualization resulted in optimized inputs that are dissimilar from all words, but provide maximally faithful interpretations of neurons. we provide a quantitative evaluation of these maximally faithful interpretations compared to wordlevel interpretations, as obtained in related work. our results suggest that word-level interpretations of neurons are not faithful and thus not suitable. we thus take our results as a strong contraindication that words could serve as good interpretations of individual neurons in bert. especially the dramatic difference in activation potential between optimized inputs and most activating words can be taken as evidence that individual neurons are not particularly sensitive to words and do not directly encode them. our work thus far only investigates the quality of word-level interpretations of llm neurons. in future work, we plan to investigate what high-level semantic knowledge is encoded within the neurons of an llm. we are specifically interested to explore whether individual or small groups of neurons, despite not being interpretable on the wordlevel, might still be symbolizable in terms of some other form of linguistic or language knowledge. our investigation of groups of neurons reveal interesting insights: first, activation maximization and consequently feature textualization does in principle work for language models. the fact that the optimized inputs in the (relative, @xmath0, feature textualization detects that word. we conjecture that whenever the method fails completely in the same condition, i.e., in the 33% of cases where the target word was not at all similar to the optimized input, the corresponding group of neurons simply did not encode the target word. a second important insight from our investigations on groups of neurons is that words are certainly not encoded within 10 or less neurons in bert, as evident by the catastrophic results of all experiments with @xmath1. in addition to our interpretation of results for single neurons, it is also possible that non-regularized optimal inputs are so dissimilar from words as they are adversarial inputs. due to the vastness of the 768-dimensional embedding space, it is possible that our procedure finds local maxima that lie in an uninterpretable part of the vector space. this problem is well known in feature visualization in computer vision: @xcite find that non-regularized feature visualization produces ad-versarial examples and thus the use of some regularization or prior is strongly encouraged. based on this, we conjecture that an important avenue for future work is the development of regularizers that pull the optimized inputs further towards the populated part of the word embedding space. we conducted initial experiments with such regularizers and found indications that they help to further increase the similarity of optimal inputs to words and thus increase their interpretability. we will investigate further experiments with regularizers in future work. the creation of optimal input vectors through activation maximization is associated with a nonneglectable computational cost, since the optimization process is executed for each neuron and group of neurons of interest. the obtained optimal inputs are also model specific. the current method does therefore not scale to a complete investigation of arbitrarily large llms. however, recent work by @xcite introduces a technique to interpret model parameters in the embedding space in a static way, i.e., with zero passes through the model. future work should experiment with this approach to investigate the knowledge represented in individual neurons of llms. until then, we make our code and optimized inputs for bert neurons publicly available, and encourage future work to re-use the published optimal inputs, rather than re-computing them. future work also needs to investigate how stable our findings are once the input length is increased, i.e., once words are contextualized. it would also be interesting to apply the method to other language models that are fine-tuned to a downstream task or larger in size. ## conclusion in our work, we introduce feature textualization, a decomposability interpretability method for nlp models, based on an adaptation of feature visualization. our work provides a thorough quantitative evaluation of the application of the method to bert which leads us to conclude that feature textualization is able to reveal the information encoded in groups of neurons in language models. we find evidence that while single neurons seem to not encode words, feature textualization is able to retrieve interpretable synthetic inputs that are similar to words for well-chosen groups of neurons. feature textualization can thus help to pin-point the exact location of encoded knowledge in the parameters of language models. we hope that our work also shed light on the difficulties found when trying to discretize continuous, dense content representations in language. while the visualization of dense representations in the visual domain is trivial, as arbitrary vectors can be plotted as images, this task is far from trivial in the language domain, where the units that are easiest to interpret are discrete words. we thus see our work as a step towards tackling the problem of making non-discrete representations quasi-interpretable via evaluating their proximity to the interpretable space on a large scale. we think that future work on feature textualization needs to extend such efforts to other units of language, both larger and smaller than words, or other parts of a neural network that go beyond the neuron level.