Interpretability and Analysis in Neural NLP (2020)

Abstract: While deep learning has transformed the natural language processing (NLP) field and impacted the larger computational linguistics community, the rise of neural networks is stained by their opaque nature: It is challenging to interpret the inner workings of neural network models, and explicate their behavior. Therefore, in the last few years, an increasingly large body of work has been devoted to the analysis and interpretation of neural network models in NLP. This body of work is so far lacking a common framework and methodology. Moreover, approaching the analysis of modern neural networks can be difficult for newcomers to the field. This tutorial aims to fill this gap and introduce the nascent field of interpretability and analysis of neural networks in NLP. The tutorial will cover the main lines of analysis work, such as structural analyses using probing classifiers, behavioral studies and test suites, and interactive visualizations. We will highlight not only the most commonly applied analysis methods, but also the specific limitations and shortcomings of current approaches, in order to inform participants where to focus future efforts.

## tutorial description deep learning has transformed the nlp field and impacted the larger computational linguistics community. neural networks have become the preferred modeling approach for various tasks, from language modeling, through morphological inflection and syntactic parsing, to machine translation, summarization, and reading comprehension. the rise of neural networks is, however, stained by their opaque nature. in contrast to earlier approaches that made use of manually crafted features, it is more challenging to interpret the inner workings of neural network models, and explicate their behavior. therefore, in the last few years, an increasingly large body of work has been devoted to the analysis and interpretation of neural network models in nlp. the topic has so far been represented in two dedicated workshops (blackbox 2018 and 2019) and was recently established as a track in the main *cl conferences. due to these recent developments, methods for the analysis and interpretability of neural networks in nlp are so far lacking a common framework and methodology. moreover, approaching the analysis of modern neural networks can be difficult for newcomers to the field, since it requires both a familiarity with recent work in neural nlp and with analysis methods which are not yet standardized. this tutorial aims to fill this gap and introduce the nascent field of interpretability and analysis of neural networks in nlp. the tutorial will cover the main lines of analysis work, mostly drawing on the recent tacl survey by @xcite . 1 in particular, we will devote a large portion to work aiming to find linguistic information that is captured by neural networks, such as probing classifiers @xcite @xcite , inter alia), controlled behavior studies on language modelling @xcite @xcite or inference tasks @xcite @xcite @xcite , psycholinguistic methods @xcite , layerwise analyses @xcite , among other methods @xcite @xcite . we will also present various interactive visualization methods such as neuron activations @xcite , attention mechanisms @xcite , and saliency measures @xcite @xcite , including a walkthrough on how to build a simple attention visualization. next, we will discuss the construction and use of challenge sets for fine-grained evaluation in the context of different tasks @xcite @xcite . finally, we will review work on generating adversarial examples in nlp, focusing on the challenges brought upon by the discrete nature of textual input @xcite @xcite . a detailed outline is provided in section 3. throughout the tutorial, we will highlight not only the most commonly applied analysis methods, but also the specific limitations and shortcomings of current approaches. by the end of the tutorial, participants will be better informed where to focus future research efforts. ## tutorial type this tutorial will cover cutting-edge research in interpretability and analysis of modern neural nlp models. the topic has not been previously covered in *cl tutorials. ## prerequisites we would assume acquaintance with core linguistic concepts and basic knowledge of machine learning and neural networks, such as covered in most introductory nlp courses. ## reading list in addition to the papers mentioned in this proposal, a comprehensive bibliography can be found in the following website: for trainees interested in reading important studies before the tutorial, we recommend the following: @xcite glass (2019 6 names and affiliations yonatan belinkov, postdoctoral fellow, harvard university and mit email: belinkov@seas.harvard.edu website: http://people.csail.mit. edu/belinkov yonatan belinkov is a postdoctoral fellow at the harvard school of engineering and applied sciences (seas) and the mit computer science and artificial intelligence laboratory (csail). his research interests are in interpretability and robustness of neural models of language. he has done previous work in machine translation, speech recognition, community question answering, and syntactic parsing. his research has been published at acl, emnlp, naacl, cl, tacl, iclr, and neurips. his phd dissertation at mit analyzed internal language representations in deep learning models. he co-organized or co-organizes blackboxnlp 2019, blackboxnlp 2020, and the wmt 2019 machine translation robustness task, and serves as an area chair for the analysis and interpretability track at acl and emnlp 2020. sebastian gehrmann, research scientist, google ai email: gehrmann@google.com website: http://sebastiangehrmann. com sebastian is research scientist at google ai. he received his phd in 2020 from harvard university. his research focuses on the development and evaluation of controllable and interpretable models for language generation. by applying methods from human-computer interaction and visualization to problems in nlp, he develops interactive interfaces that help with the interpretation and explanation of neural networks. his research has been published at acl, naacl, emnlp, chi, and ieee vis. he received an honorable mention at vast 2018 and was nominated for acl best demo 2019 for his work on interactive visualization tools. he co-organized inlg 2019 and served as an area chair in summarization for acl 2020. ellie pavlick, assistant professor of computer science, brown university email: ellie pavlick@brown.edu website: http://cs.brown.edu/people/ epavlick ellie pavlick is an assistant professor at brown university and a research scientist at google. she received her phd in 2017 with her thesis on modeling compositional lexical semantics. her current work focuses on computational models of semantics and pragmatics, with a focus on building cognitively-plausible representations. her recent work has focused on "probing" distributional models in order to better understand the linguistic phenomena that are and are not encoded "for free" via language modelling. her work has been published at acl, naacl, emnlp, tacl, *sem, and iclr, including two best paper awards at *sem 2016 and 2019. ellie co-organized the 2018 jsalt summer workshop on building and evaluating general-purpose sentence representations. she also served as area chair for acl's sentencelevel semantics track. the association for computational linguistics (volume 1: long papers), pages 2126-2136, melbourne, australia. association for computational linguistics.