diag NN ose: A Library for Neural Activation Analysis (2020)

Abstract: In this paper we introduce diagNNose, an open source library for analysing the activations of deep neural networks. diagNNose contains a wide array of interpretability techniques that provide fundamental insights into the inner workings of neural networks. We demonstrate the functionality of diagNNose with a case study on subject-verb agreement within language models. diagNNose is available at https://github.com/i-machine-think/diagnnose .

## introduction we introduce diagnnose, an open source library for analysing deep neural networks. the diagnnose library allows researchers to gain better insights into the internal representations of such networks, providing a broad set of tools of state-ofthe-art analysis techniques. the library supports a wide range of model types, with a main focus on nlp architectures based on lstms @xcite and transformers @xcite . open-source libraries have been quintessential in the progress and democratisation of nlp. popular packages include huggingface's transformers @xcite -allowing easy access to pretrained transformer models; jiant @xcite ) -focusing on multitask and transfer learning within nlp; captum @xcite ) -providing a range of feature attribution methods; and lit @xcite ) -a platform for visualising and understanding model behaviour. we contribute to the open-source community by incorporating several interpretability techniques that have not been present in these packages. recent years have seen a considerable interest in improving the understanding of how deep neural networks operate @xcite . the high-dimensional nature of these models makes it notoriously challenging to untangle their inner dynamics. this has given rise to a novel subfield within ai that focuses on interpretability, providing us a peak inside the black box. diagnnose aims to unify several of these techniques into one library, allowing interpretability research to be conducted in a more streamlined and accessible manner. in this paper we present both an overview of the library, as well as a case study on subjectverb agreement within language models. we first present a brief overview of interpretability within nlp and a background to the analysis techniques that are part of the library (section 2). we then provide an overview of diagnnose and expand briefly on its individual modules (section 3). we conclude with a case study on subject-verb agreement, demonstrating several of diagnnose's features in an experimental setup (section 4). ## background the increasing capacities of language models (and deep learning in general) have led to a rich field of research that aims to gain a better understanding of how these models operate. approaches in this research area are often interdisciplinary in nature, borrowing concepts from fields such as psycholinguistics, information theory, and game theory. diagnnose provides support for several influential analysis techniques, for which we provide a brief background here. ## case study: subject-verb agreement to demonstrate the functionality of diagnnose we will consider the subject-verb agreement corpora of @xcite on a set of language models. for our experiments we consider the following models: bert @xcite , roberta @xcite , distilroberta @xcite , and the lstm language model of @xcite .