Fine-grained Interpretation and Causation Analysis in Deep NLP Models (2021)

Abstract: Deep neural networks have constantly pushed the state-of-the-art performance in natural language processing and are considered as the de-facto modeling approach in solving complex NLP tasks such as machine translation, summarization and question-answering. Despite the proven efficacy of deep neural networks at-large, their opaqueness is a major cause of concern. In this tutorial, we will present research work on interpreting fine-grained components of a neural network model from two perspectives, i) fine-grained interpretation, and ii) causation analysis. The former is a class of methods to analyze neurons with respect to a desired language concept or a task. The latter studies the role of neurons and input features in explaining the decisions made by the model. We will also discuss how interpretation methods and causation analysis can connect towards better interpretability of model prediction. Finally, we will walk you through various toolkits that facilitate fine-grained interpretation and causation analysis of neural models.

## introduction deep neural networks have constantly pushed the state-of-the-art performance in natural language processing and are considered as the de facto modeling approach in solving most complex nlp tasks such as machine translation, summarization and question-answering. despite the benefits and the usefulness of deep neural networks at-large, their opaqueness is a major cause of concern. interpreting neural networks is considered important for increasing trust in ai systems, providing additional information to decision makers, and assisting ethical decision making @xcite . interpretation of neural network models is a broad area of research. significant work has analyzed network at representation-level @xcite @xcite , and at neuron-level @xcite @xcite . others have experimented with various behavioural studies to analyze models @xcite @xcite . moreover, a number of studies cover the importance of input features and neurons with respect to a prediction @xcite @xcite . the topic of interpretation of neural models has gained a lot of attention in a last couple of years. for example, it has been added as a regular track in major *cl conferences. there is an annual workshop, blackboxnlp, dedicated for this purpose. the acl 2020 and emnlp 2020 foot_0 featured tutorials on the topic @xcite . the acl tutorial focused on two subareas of interpretation which are the representation analysis and the behavioral studies. the emnlp tutorial is solely focused on behavioral studies i.e. assess a model's behavior using constructed examples. both of these tutorials serves as a great starting point for the new researchers in this area. the representation analysis, also called as structural analysis, is useful to understand how various core linguistic properties are learned in the model. however, the analysis suffers from a few limitations. it mainly focuses at interpreting full vector representations and does not study the role of finegrained components in the representation i.e. neurons. also the findings of representation analysis do not link with the cause of a prediction @xcite . while the behavioral analysis evaluates model predictions, it does not typically connect them with the influence of the input features and the internal components of a model @xcite . in this tutorial, we aim to present and discuss the research work on interpreting fine-grained components of a model from two perspectives, i) finegrained interpretation, ii) causation analysis. the former will introduce methods to analyze individual neurons and a group of neurons with respect to a desired language property or a task. the latter will bring up the role of neurons and input features in explaining decisions made by the model. we will cover important research questions such as i) how is knowledge distributed across the model components? ii) what knowledge learned within the model is used for specific predictions? iii) does the inhibition of specific knowledge in the model change predictions? iv) how do different modeling and optimization choices impact the underlying knowledge? recent work on interpreting neurons has shown that in-addition to gaining better understanding of the inner workings of neural networks, the neuronlevel interpretation has applications in model distillation @xcite , domain adaptation @xcite or efficient feature selection @xcite e.g., by removing unimportant neurons, facilitating architecture search, and mitigating model bias by identifying neurons responsible for sensitive attributes like gender, race or politeness @xcite . these recent works are not only enabling better understanding of these networks, but are also leading towards better, fairer and more environmental-friendly models, which are all important goals for the artificial intelligence community at large. ## description the tutorial is divided into two main parts: i) finegrained interpretation, and ii) causation analysis. the first part of the tutorial covers methods that align neurons to human interpretable concepts or study the most salient neurons in the network. we cluster these methods into four groups i) visualization methods @xcite , ii) corpus selection @xcite @xcite , iii) neuron probing @xcite @xcite and iv) unsupervised methods @xcite @xcite ). we will discuss evaluation methods that are used to measure the effectiveness of an interpretation method, such as accuracy, control tasks @xcite and ablation studies @xcite @xcite . moreover, we will cover various applications of these methods that go beyond interpretation such as efficient transfer learning @xcite , controlling system's behavior @xcite , generating explanations @xcite and domain adaptation @xcite . the second part, causation analysis, will focus on methods that seek to characterize the role of neurons and layers towards a specific prediction. more concretely, we will discuss gradient and perturbation-based attribution algorithms such as integrated gradients @xcite , layer conductance (dhamdhere et al., 2018b), saliency @xcite , shapley additive explanations(shap) (lundberg and lee, 2017) and showcase how they can help us to identify important neurons in different layers of a deep neural network. besides that we will also dive deep into more recent and advanced attribution algorithms that take feature or neuron interactions into account. more specifically, we will look into integrated hessians @xcite , shapely taylor index (dhamdhere et al., 2020) and archipelago @xcite . lastly, we will mention various open source toolkits and libraries that provide implementation of notable techniques in the area. a few examples of the toolkits are: captum @xcite , interpretml foot_1 , neurox @xcite , ecco foot_2 and diagnnose @xcite . we will walk-through how some of these tools can be used for fine-grained interpretation and causation analysis. throughout the tutorial, our goal will also be to critically evaluate where the strengths and weakness of each of the presented methods lie, and provide ideas and recommendations around future directions. ## prerequisites we assume basic knowledge of the deep learning and familiarity with the lstm-based and transformer-based pre-trained models such as elmo @xcite and bert @xcite . additionally, some familiarity with natural language processing tasks such as, named entity tagging, natural language inference, etc. would be useful but not mandatory. we do not expect participants to have familiarity with the research on the interpretation and analysis of deep models. familiarity with python, pytorch and transformers library @xcite would be useful to understand the practical part. bau et al. (2019); dalvi et al. (2019a); mu and andreas (2020b); suau et al. (2020) etc. • causation analysis: lundberg and lee (2017) provides an overview of various methods introduced in literature. for more details, see the following papers: voita et al. (2020); sundararajan et al. (2017); dhamdhere et al. (2018b); ribeiro et al. (2016); janizek et al. (2020) in addition to the above list, interested trainees may look at the papers mentioned in section 2. 6 instructor information (alphabetic order fahim dalvi, software engineer, qatar computing research institute, qatar email: faimaduddin@hbku.edu.qa website: https://fdalvi.github.io fahim dalvi is an experienced software engineer with a demonstrated history of working in the research industry and is currently employed at the qatar computing research institute. fahim's research is centered around the intersection of natural language processing and deep learning, and he has worked on wide variety of problems in these fields including machine translation, language modelling and explainability in deep neural networks. he also spends his time converting research into practical applications, with a focus on scalable web applications. fahim also spends some time every year mentoring and teaching deep learning at fall and summer schools. hassan sajjad, senior research scientist, qatar computing research institute, qatar email: hsajjad@hbku.edu.qa website: https://hsajjad.github.io hassan sajjad is a senior research scientist at the qatar computing research institute (qcri), hbku. his research interests include the interpretation of deep neural models, machine translation, domain adaptation, and natural language processing involving low-resource and morphologically-rich languages. his research work has been published in several prestigious venues such as cl, csl, iclr, acl, naacl and emnlp. his work in collaboration with mit and harvard on the interpretation of deep models has also been featured in several tech blogs including mit news. hassan co-organized blackboxnlp 2020, and the wmt 2019/2020 machine translation robustness task. he served as an area chair for the analysis and interpretability, nlp application, and machine translation tracks at various *cl conferences. in addition, hassan has been regularly teaching courses on deep learning internationally at various spring and summer schools. narine kokhlikyan, research scientist, facebook ai email: narine@fb.com website: https://www.linkedin.com/ in/narine-k-88916721/ narine is a research scientist focusing on model interpretability as part of pytorch team at facebook. her research interests include the understanding of deep neural network internals and their predictions across different applications such as natural language processing, computer vision and recommender systems. in the recent years she gave talks and presented tutorials on model interpretability at kdd 2020 and neurips 2019. before joining facebook narine worked on natural language processing, time series analysis and numerical optimizations. nadir durrani, research scientist, qatar computing research institute, qatar email: ndurrani@hbku.edu.qa website: http://alt.qcri.org/ ~ndurrani/ nadir durrani is a research scientist at the arabic language technologies group at qatar computing research institute. his research interests include interpretation of neural networks, neural and statistical machine translation (with focus on reordering, domain adaptation, transliteration, dialectal translation, pivoting, closely related and morphologically rich languages), eye-tracking for mt evaluation, spoken language translation and speech synthesis. his recent work focuses on analyzing contextualized representations with the focus of linguistic interpretation, manipulation, feature selection and model distillation. his work on analyzing deep neural networks has been published at venues like computational linguistics, *acl, aaai and iclr. nadir has been involved in co-organizing workshops such as simultaneous machine translation and wmt 2019/2020 machine translation robustness task. he regularly serves as program committee and has served as area chair at acl and aaai this year.