Exploratory Model Analysis Using Data-Driven Neuron Representations (2021)

Abstract: Probing classifiers have been extensively used to inspect whether a model component captures specific linguistic phenomena. This top-down approach is, however, costly when we have no probable hypothesis on the association between the target model component and phenomena. In this study, aiming to provide a flexible, exploratory analysis of a neural model at various levels ranging from individual neurons to the model as a whole, we present a bottom-up approach to inspect the target neural model by using neuron representations obtained from a massive corpus of text. We first feed massive amount of text to the target model and collect sentences that strongly activate each neuron. We then abstract the collected sentences to obtain neuron representations that help us interpret the corresponding neurons; we augment the sentences with linguistic annotations (e.g., part-of-speech tags) and various metadata (e.g., topic and sentiment), and apply pattern mining and clustering techniques to the augmented sentences. We demonstrate the utility of our method by inspecting the pre-trained BERT. Our exploratory analysis reveals that i) specific phrases and domains of text are captured by individual neurons in BERT, ii) a group of neurons simultaneously capture the same linguistic phenomena, and iii) deeper-level layers capture more specific linguistic phenomena.

## introduction deep neural networks (dnns) learn to induce internal feature representations or neurons foot_0 for a given input, which are optimized for the target task. the success of dnns in the field of natural language processing (nlp) is underpinned by this flexibility to induce internal vector representations of input. it is, however, difficult for humans to interpret their roles and properties, which hinders us from leveraging dnns in practical applications. researchers have therefore investigated what linguistic aspects of input text are captured by the internal representations of dnns optimized for nlp tasks ( § 2). most of the existing methods inspect a pre-specified model component (e.g., individual bert layers) in a top-down manner. a typical approach first takes aim at specific linguistic phenomena that would be captured by the target components, and then trains a probing classifier that predicts the chosen linguistic phenomena from the target components @xcite @xcite @xcite @xcite @xcite @xcite @xcite . although this top-down approach based on probing classifiers has provided thought-provoking insights into the target model components, it becomes costly when we want to inspect many combinations of model components and linguistic phenomena. this is because the probing requires us to train machine-learning classifiers for each combination of model component and linguistic phenomenon. although there are a few bottom-up approaches to inspect dnns by examining the response of the neurons towards (perturbed) inputs that represent the target linguistic phenomenon @xcite @xcite , these approaches are labor intensive since they require manual intervention. this study aims at efficiently inspecting neural nlp models at various levels of granularity ranging from individual neurons to the model as a whole, and presents a bottom-up approach to inspecting what kind of concrete linguistic phenomena each neuron of the model responds to, without presupposing the target linguistic phenomenon to be examined ( § 3). given a massive amount of text, we first feed individual sentences to the target model, and collect sentences to which each neuron strongly 518 responds ( § 3.1). 2 we next annotate the collected sentences with the linguistic annotations (e.g., part of speech tags) and various metadata (e.g., topic and sentiment). with the help of these annotations, we then apply text mining techniques such as frequent pattern mining to extract common patterns as the linguistic signatures foot_3 that exist repeatedly and intricately in the sentences collected for the target neurons ( § 3.2). we finally investigate relationships between multiple neurons by comparing and clustering continuous neuron representations induced from the collected sentences. we apply our method to the pre-trained bert (base-uncased) @xcite to demonstrate how much insight into bert the method can actually provide. our exploratory model analysis have confirmed that it is possible to identify, without any prior assumptions, a wide variety of specific linguistic phenomena to which each neuron responds ( § 4.2). furthermore, by comparing the linguistic phenomena and sentences corresponding to individual neurons, we revealed the existence of neurons that work cooperatively for the same purpose. we finally revealed the impact of optimizing bert to the target task (here, the pre-training task and sentiment classification) by comparing neurons of randomly initialized bert with neurons of the pre-trained and the fine-tuned bert ( § 4.3). the contribution of this study is threefold: we investigate concrete linguistic phenomena (e.g., skip n-grams) captured by the neurons of the target model, without any prior assumptions about the phenomena to be examined. ## related work in order to clarify what kind of linguistic phenomena are captured by neural nlp models, researchers have analyzed the internal representations of the models at various levels of granularity such as neurons @xcite @xcite @xcite @xcite , layers @xcite @xcite @xcite , attentions @xcite @xcite , and the model as a whole @xcite @xcite . in what follows, we start by reviewing probing methods that investigate the classifier performance of external tasks based on the target model components. we next describe some microscopic methods that focus on individual neurons. we then discuss a method of inspecting each neuron using text generated for that neuron. most of the recent methods adopt a top-down approach called probing, which takes target component (e.g., layer) as inputs, trains a classifier that predicts the linguistic phenomena of interest such as syntactic information @xcite @xcite , agreement information @xcite , and semantic knowledge @xcite , and evaluates the properties of the internal representation with reference to the accuracy of the classifier. to reduce the cost of training a classifier, @xcite indirectly predict the performance of probing classifiers by analyzing how the labeled data is represented in the vector space. some studies identify neurons which make a huge contribution to solving the desired task, by looking at the performance of the task when the activation of neurons is forcibly controlled @xcite @xcite . these probing methods are inductive in that they examine whether the target model component captures the target linguistic phenomenon. it is costly to inspect various pairs of model components and phenomena. our approach does not presuppose linguistic phenomena to be examined and gives information that differs from the above methods; we show concrete linguistic phenomena each neuron responds to strongly, in a form that can be easily understood by humans (e.g., skip n-grams) ( § 4.2). there are several studies that have investigated the roles of neurons -the finest components of models such as a cell state in a long short term memory -by observing interactions between their values and (perturbed) input sentences @xcite @xcite @xcite . however, these methods are tailored to analyze a few pre-defined particular phenomena such as sequence length, or need to perform the analysis per phenomenon. our method, on the other hand, allows for a wide range of analytical perspectives in that it uses a massive amount of raw text to collect sentences in which individual neurons show strong interest and leverages pattern mining to highlight linguistic phenomena contained in the collected sentences. as the most similar to our approach, @xcite proposed a gradient-based method that generates input text which strongly activates neurons, inspired by work in computer vision @xcite . their method differs from the probing methods in that it embodies the information captured by the neurons as text, which is similar to our study. however, their method can generate only text of pre-fixed length, and requires hyper-parameter tuning (e.g., annealing temperature) to generate natural text, making it costly to apply to massive neurons in the models. moreover, it is difficult for humans to interpret properties of the generated text just by observing them. in contrast, we propose an example-based method for collecting human-written sentences from a huge text corpora, determining which each neuron finds interesting, without any parameter tuning. by exploiting text mining techniques, we can reveal various linguistic phenomena as long as they appear in the source raw text. ## data-driven inspection of neurons this section describes our methodology for providing a deeper insight into a neural model at the finest level of granularity (i.e., neurons 1 ). our approach abstracts the sentences retrieved from a massive amount of text such as the bookcorpus @xcite . the whole process is as follows. ## experiments taking the pre-trained bert (base-uncased) @xcite as an example, we demonstrate that our methodology can perform similar analysis to that obtained by existing approaches. we also provide novel findings on how and what aspects of language each neuron captures. ## conclusions this study aims to provide a method of exploratory model analysis for a neural nlp model at various levels of granularity ranging from individual neurons to the model as a whole, and proposed a bottom-up methodology for revealing what kind of concrete linguistic phenomena each neuron of the model strongly responds to. we take advantage of large-scale text data and data mining techniques to extract linguistic signatures (common patterns) that characterize the individual neurons. taking bert (base-uncased) as an example, we first showed that specific phrases such as those related to united states are captured by individual neurons in bert (e.g., united states neuron in in the future, we plan to investigate models with different architectures. for example, our method can be used to compare the differences of encoder and decoder neurons in end2end models; there is a debate in the field of neural machine translation as to whether modifying the encoder or the decoder contributes more to domain adaptation @xcite . our method enables us to find non-apriori linguistic phenomena the neurons may capture, so that it is possible to assist in the construction of a novel training/evaluation dataset for the probingbased evaluation methods. in addition, our method provides information of a different nature, i.e., the concrete linguistic phenomena to which the internal representations respond, as opposed to numerical information such as accuracy of probing tasks. as mentioned in § 4.2, our method has the potential to perform with less time complexity by reducing the corpus size fed to the target model in step 1. therefore, we plan to study how to select a text corpus of such a small size that it reproduces the analysis results obtained when using a very large corpus as the one used in this study.