N euro X Library for Neuron Analysis of Deep NLP Models (2023)

Abstract: Neuron analysis provides insights into how knowledge is structured in representations and discovers the role of neurons in the network. In addition to developing an understanding of our models, neuron analysis enables various applications such as debiasing, domain adaptation and architectural search. We present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis of natural language processing models. It implements various interpretation methods under a unified API, and provides a framework for data processing and evaluation, thus making it easier for researchers and practitioners to perform neuron analysis. The Python toolkit is available at https://www.github.com/fdalvi/NeuroX.Demo Video available at: https://youtu.be/mLhs2YMx4u8

## introduction interpretation of deep learning models is an essential attribute of trustworthy ai. researchers have proposed a diverse set of methods to interpret models and answered questions such as: what linguistic phenomena are learned within representations, and what are the salient neurons in the network. for instance, a large body of work analyzed the concepts learned within representations of pre-trained models @xcite @xcite and showed the presence of core-linguistic knowledge in various parts of the network. several researchers have carried out this interpretation at a fine-grained level of neurons e.g. @xcite ; torroba @xcite ; @xcite highlighted salient neurons w.r.t any linguistic property in the model and lundberg and lee (2017); @xcite ; @xcite identified a set of neurons responsible for a given prediction. at a broader level, these methods can be given an annotated text corpus, neuron interpretation methods aim to provide a ranking of neurons in a model w.r.t to their importance to one or more annotated properties (for e.g. "noun" in this instance) categorized into representation analysis, neuron analysis and feature attribution methods respectively. @xcite provides a comprehensive survey of these methods. a number of toolkits have been proposed to facilitate the interpretation of deep learning models. for instance, diagnnose @xcite provides representation analysis and attribution methods. lit @xcite can be used to visualize attention and counterfactual explanations using feature attribution methods. captum @xcite integrates a large set of attribution methods under a consistent api. all these tools facilitate the interpretability of models. however, due to the diverse ways of interpreting models, they do not cover all sets of methods. specifically, most of the toolkits do not cover neuron interpretation methods that discover and rank neurons with respect to a concept. neuron interpretation analyzes and gives insight into how knowledge is structured within a representation. it discovers neurons with respect to a concept and provides a fine-grained interpretation of deep models. figure 1 provides a simplified high-level overview of how neuron interpretation methods operate. given a model, some text and annotations, these methods output a ranking of neurons with respect to their importance to one or more annotated concepts. the ability to interpret neurons enables applications such as debiasing models, controlling predictions of the models on the fly @xcite , neural architectural search @xcite and domain adaptation @xcite . to make neuron interpretation more accessible, we propose neurox, an open-source python toolkit to facilitate neuron interpretation of deep natural language processing (nlp) models. neurox consists of three major components: i) data processing, ii) interpretation and iii) analysis. the data processing implements various ways to generate and upload data for analysis, extract activations and save them efficiently. the interpretation module implements six interpretation methods belonging to two different classes of methods. the analysis module brings together qualitative and quantitative methods to evaluate and visualize the discovered neurons. figure 2 shows these components and how they interact with each other. we describe them in detail in the following sections. the toolkit itself is compatible with huggingface's transformers @xcite api and supports all transformer-based models. to the best of our knowledge, neurox is the first toolkit that enables the interpretation of deep nlp models at the neuron level. it serves as a backbone to rapidly test new interpretation techniques using a unified framework and enables comparison and consistent evaluation of these techniques. the toolkit is easy to install and run: pip install neurox with detailed documentation is available at https: //neurox.qcri.org/docs , including tutorials that showcase various capabilities of the toolkit to quickly get started with neuron interpretation. ## data processing the data module is responsible for preparing all the inputs for the interpretation and analysis modules, as well as filtering the datasets to probe and interpret specific phenomena. as shown in figure 2 , the required inputs to the toolkit are: i) a model and ii) a text corpus annotated towards the property of interest (e.g. data annotated towards toxic word spans in the hate-speech-detection task). the interpretation module can then extract a neuron ranking, highlighting the saliency of the neurons in the model that capture this phenomenon. if annotations are not available, an annotation helper module is made available in the toolkit that can annotate tokens based on arbitrary phenomena e.g. suffixation, lexical properties, or using pre-existing vocabularies. below we describe the various components of the data module in detail. ## interpretation module the central module in the neurox toolkit is the interpretation module, which provides implementations of several neuron and representation analysis methods. the method implementations follow a consistent api to make it easy for the user to switch between them. specifically, each method at least implements the following functions: the interpretation methods themselves may be able to probe multiple properties at the same time (multi-class probing), or only a single concept (binary probing). additionally, some interpretation methods can also perform representation-level analysis, i.e. probe an entire layer rather than individual neurons. ## analysis and evaluation the analysis module provides implementations of various evaluation and analysis methods. some of these methods provide quantitative results like accuracy scores, while others allow users to perform qualitative analysis on neurons. ## related work a number of toolkits have been made available to carry out the analysis and interpretation of neural network models. the what-if tool @xcite inspects machine learning models and provides users an insight into the trained model based on the predictions. seq2seq-vis @xcite enables the user to trace back the prediction decisions to the input in neural machine translation models. captum @xcite provides generic implementations of a number of gradient and perturbation-based attribution algorithms. lit @xcite implements various methods of counterfactual explanations, attribution methods and visualization of attention. diagnnose @xcite integrates representation analysis methods and attribution methods and finally, imodelsx @xcite aims to provide natural explanations for datasets, which can provide insights into the models that are trained with these datasets. while these tools cover a number of interpretation methods, none of them facilitate neuron-level interpretation of nlp models. the lm-debugger toolkit @xcite is an interactive debugger for transformer lms, which provides a fine-grained interpretation and a powerful framework for intervening in lm behavior. ecco @xcite is a visualization based library that implements saliency methods and ad-ditionally enables visualization of neurons of the network. similar to ecco, the neurox toolkit enables visualization of neurons of the network. in addition, we implement a wide range of neuron interpretation methods that can be accessed using a uniform api and provide various analysis and evaluation methods. our toolkit empowers researchers to focus on specific parts of the neuron interpretation research such as interpretation, comparison or evaluation without worrying about setting up the rest of the pipeline like data processing, embedding extraction, integration with various pre-trained models, and evaluation of the method. neurox powers other interpretation analysis frameworks such as conceptx @xcite and nxplain @xcite . the previous version of neurox @xcite only supported a specific machine translation library and one neuron interpretation method @xcite as a gui app. the current python toolkit is a redesigned version with a unified architecture. it includes multiple features like a data processing module, numerous neuron interpretation and evaluation methods, and seamless integration with popular toolkits such as hugging-face's transformers. ## conclusion and future work we presented neurox, an open-source toolkit to carry out neuron-level interpretation of representations learned in deep nlp models. it maintains implementations of several neuron analysis methods under a consistent api, and provides implementations for preparing the data, analyzing neurons and evaluating the methods. in the future, neu-rox plans to expand its extraction module to other frameworks like fairseq and opennmt-py. in ad-dition, we plan to integrate attribution based neuron saliency methods to add another class of interpretation methods to the toolkit.