Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools (2021)

Abstract: In the language domain, as in other domains, neural explainability takes an ever more important role, with feature attribution methods on the forefront. Many such methods require considerable computational resources and expert knowledge about implementation details and parameter choices. To facilitate research, we present Thermostat which consists of a large collection of model explanations and accompanying analysis tools. Thermostat allows easy access to over 200k explanations for the decisions of prominent state-of-the-art models spanning across different NLP tasks, generated with multiple explainers. The dataset took over 10k GPU hours (> one year) to compile; compute time that the community now saves. The accompanying software tools allow to analyse explanations instance-wise but also accumulatively on corpus level. Users can investigate and compare models, datasets and explainers without the need to orchestrate implementation details. Thermostat is fully open source, democratizes explainability research in the language domain, circumvents redundant computations and increases comparability and replicability.

## introduction deep neural networks are state-of-the-art in natural language processing (nlp) but due to their complexity they are commonly perceived as opaque @xcite . for this reason, explainability has seen heightened attention in recent years @xcite @xcite . a prominent class of explainability methods, referred to as feature attribution methods (in the following used interchangeably with explainers), attributes the output of a complex model to its input features. feature attribution methods arguably have become a cornerstone of explainability research in nlp: for example, @xcite ; @xcite al. (2020); chen et al. (2021); neely et al. (2021 ## thermostat thermostat is intended to be a continuous, collaborative project. as new models, datasets and explainers are published, we invite the community to extend thermostat. in what follows, we describe the current state which is published under https://github.com/ dfki-nlp/thermostat . ## related work to the best of our knowledge, the closest work to our contribution is the language interpretability tool (lit) by @xcite which offers a graphical interface for exploring saliency maps, counterfactual generation and the visualization of attention and embeddings. we also draw connections to @xcite and @xcite who developed interfaces for analyzing the attention mechanism and embeddings of transformer architectures. these again are visualization and analysis tools and as such complementary to the collection of explanations that is our primary contribution. thus, an interface that bridges thermo-stat and lit, for instance, is an interesting future direction. there exist libraries, such as captum @xcite or transformers-interpret @xcite , that facilitate the generation of neural explanations. these libraries do not, however, free the user of the computational burden, nor are they easily accessible to non-technical researchers. as noted in section 2, the datasets library @xcite functions as the backbone of thermostat. the novelties our work brings with it are (1) attributions from a variety of models that took over 10,000 gpu hours to compute in total, and (2) the support for explainability-specific visualizations and statistics like heatmap visualization and rank correlation between multiple explainers. finally, tangentially related to our work are datasets that supply explanations on top of texts and labels, usually collected from human annotators. e-snli @xcite probably is the most famous example in this line of work. the reader is referred to @xcite for a concise survey. in contrast to our work, the above mentioned papers present human ground truths instead of machine-generated explanations. ## conclusion we present thermostat, an easily accessible data hub containing a large collection of nlp model explanations from prominent and mostly expensive explainers. we demonstrate the ease of access, extensibility and maintainability. new datasets can be added easily. furthermore, we showcase an accompanying library and outline use cases. users can compare models and explainers across a variety of datasets. thermostat democratizes explainability re-search to a certain degree as it mitigates the computational (environmentally and financially) and implementational burden. machine-generated explanations become accessible to non-technical researchers. furthermore, comparability and replicability are increased. it becomes apparent when consulting the literature in section 1 that interpretation beyond classification (e.g. machine translation) is still an open problem @xcite . hence, we focus on these four text classification problems that are well-trodden paths.