Neuron-level Interpretation of Deep NLP Models: A Survey (2022)

Abstract: The proliferation of Deep Neural Networks in various domains has seen an increased need for interpretability of these models. Preliminary work done along this line, and papers that surveyed such, are focused on high-level representation analysis. However, a recent branch of work has concentrated on interpretability at a more granular level of analyzing neurons within these models. In this paper, we survey the work done on neuron analysis including: i) methods to discover and understand neurons in a network; ii) evaluation methods; iii) major findings including cross architectural comparisons that neuron analysis has unraveled; iv) applications of neuron probing such as: controlling the model, domain adaptation, and so forth; and v) a discussion on open issues and future research directions.

## introduction models trained using deep neural networks (dnns) have constantly pushed the state-ofthe-art in various natural language processing (nlp) problems, for example, language modeling @xcite and machine translation @xcite to name a few. despite this remarkable revolution, the black-box nature of dnns has remained a major bottleneck in their large scale adaptability-especially in the applications where fairness, trust, accountability, reliability, and ethical decision-making are considered critically important metrics or at least as important as the model's performance @xcite . this opaqueness of dnns has spurred a new area of research to analyze and understand these models. a plethora of papers have been written in the past five years on interpreting deep nlp models and to answer one question in particular: what knowledge is learned within representations? we term this work as the representation analysis. representation analysis thrives on post-hoc decomposability, where we analyze the embeddings to uncover linguistic (and non-linguistic) concepts 1 that are captured as the network is trained towards an nlp task @xcite @xcite @xcite . a majority of the work on representation analysis has focused on a holistic view of the representations, namely, how much knowledge of a certain concept is learned within representations as a whole @xcite for a survey done on this line of work). recently, a more fine-grained neuron interpretation has started to gain attention. in addition to the holistic view of the representation, neuron analysis provides insight into a fundamental question: how is knowledge structured within these representations? in particular, it targets questions such as: answers to these questions entail potential benefits beyond understanding the inner workings of models, for example: i) controlling bias and manipulating system's behavior by identifying relevant neurons with respect to a prediction, ii) model distillation by removing less useful neurons, iii) efficient feature selection by selecting the most salient neurons and removing the redundant ones, and iv) neural architecture search by guiding the search with important neurons. the work on neuron analysis has explored various directions such as: proposing novel methods to discover concept neurons @xcite , analyzing and comparing architectures using neuron distributions @xcite @xcite , and enabling applications of neuron analysis @xcite . in this survey, we aim to provide a broad perspective of the field with an in-depth coverage of each of these directions. we propose a matrix of seven attributes to compare various neuron analysis methods. moreover, we discuss the open issues and promising future directions in this area. the survey is organized as follows: section 2 defines the terminologies and formally introduces neuron analysis. section 3 covers various neuron analysis methods and compares them using seven attributes. section 4 presents the techniques that have been used to evaluate the effectiveness of neuron analysis methods. section 5 discusses the findings of neuron analysis methods. lastly, section 6 showcases various applications of the presented methods and section 7 touches upon the open issues and future research directions. ## definitions in this section, we define the terminology used in the paper and the objective of neuron analysis more formally. neuron neural networks, such as rnns or transformer models consist of various components such as gates/cells, blocks, layers, attention heads, and so on. we use the term neuron (also called features, experts, and units in the literature) to refer to the output of a single dimension from any neural network component. for example, in the bert base model, the output of a layer block has 768 neurons and the output of an attention head has 64 neurons. moreover, we refer to individual neurons that learn a single concept as focused neurons, and a set of neurons that in combination represent a concept as group neurons. concept a concept represents a coherent fragment of knowledge, such as ''a class containing certain objects as elements, where the objects have certain properties'' @xcite . for example, a concept could be lexical (e.g., words ending with suffix ''ed''), morphological (e.g., gerund verbs), or semantic (e.g., names of cities). we loosely define a concept c as a group of words that are coherent with respect to a linguistic property. objective figure 1 presents an overview of various objectives in neuron analysis. formally, given a model m and a set of neurons n (which may consist of all the neurons in the network or a specific subset from particular components like a layer or an attention head) and a concept c, neuron analysis aims to achieve one of the following objectives: the former two aim to understand what concepts are encoded within the learned representation. the last objective analyzes how knowledge is distributed across neurons. each neuron n @xmath0. here, every element of the vector corresponds to a word. for phrase-or sentence-level concepts, an aggregation of neuron activations over words in the phrase/sentence is used. alternatively, [cls] token representation is also used for transformer models that are transfer learned towards a downstream nlp task. ## neuron analysis methods we have classified the work done on neuron analysis into 5 broader categories of methods, namely: i) visualizations, ii) corpus-based, iii) probing-based, iv) causation-based, and v) miscellaneous methods, based on a set of attributes we describe below: ## evaluation in this section, we survey the evaluation methods used to measure the correctness of the neuron analysis methods. due to the absence of interpretation benchmarks, it is difficult to precisely define ''correctness''. evaluation methods in interpretation mostly resonate with the underlying method to discovered salient neurons. for example, visualization methods often require qualitative evaluation via human in the loop, probing methods claim correctness of their rankings using classifier accuracy as a proxy. @xcite highlighted this discrepancy and suggested disentangling the analysis methodology from the evaluation framework-for example, by using a principally different evaluation method compared to the underlying neuron analysis method. in the following, we summarize various evaluation methods and their usage in the literature. ## findings work done on neuron interpretation in nlp is predominantly focused on questions such as: i) what concepts are learned within neurons? ii) how the knowledge is structured within representations? we now iterate through various findings the above-described neuron analysis methods unravelled. based on our main driving questions, we classify these into two broad categories: i) concept discovery and ii) architectural analysis. ## applications neuron analysis leads to various applications beyond interpretation of deep models. in this section, we present several applications of neuron analysis: i) controlling model's behavior, ii) model distillation and efficiency, iii) domain adaptation, and iv) generating compositional explanations. ## open issues and future directions in the following section, we discuss several open issues and limitations related to methods, evaluation, and datasets. moreover, we provide potential future directions vital to the progress of neuron and model interpretation. a few other studies evaluated the causal relation of the selected concept neurons via ablation or by clamping their activation values @xcite and observed the change in model's prediction. however, most of the studies do not take into account the causal relation as part of the method or the evaluation of their method. the causal relation with respect to concept neurons is important to understand their importance to overall prediction and it leads way towards practical applications such as debiasing, model distillation, and domain adaptation.