Explaining Classes through Stable Word Attributions (2022)

Abstract: Input saliency methods have recently become a popular tool for explaining predictions of deep learning models in NLP. Nevertheless, there has been little work investigating methods for aggregating prediction-level explanations to the class level, nor has a framework for evaluating such class explanations been established. We explore explanations based on XLM-R and the Integrated Gradients input attribution method, and propose 1) the Stable Attribution Class Explanation method (SACX) to extract keyword lists of classes in text classification tasks, and 2) a framework for the systematic evaluation of the keyword lists. We find that explanations of individual predictions are prone to noise, but that stable explanations can be effectively identified through repeated training and explanation. We evaluate on web register data and show that the class explanations are linguistically meaningful and distinguishing of the classes.

## introduction in recent years, various approaches to explaining predictions of deep neural networks have been attracting interest in the fields of nlp and computer vision (see, @xcite ). several techniques have been suggested in this vein, including model attention visualization (see, e.g., @xcite ), and input attribution (or saliency) methods (see @xcite @xcite , which focus on explaining individual predictions. however, showing how a model perceives larger units such as entire classes in a text classification task would be crucial for gaining a global understanding of deep classifiers and salient word features. moreover, text classification models often struggle to truly generalize @xcite . for instance, @xcite show in repeated experiments with bert on a text inference task that, while consistent test set performance was achieved, the degree of generalization as measured on a related task varied significantly, due to randomized initializations of the decision layer and order of training examples. similarly, @xcite demonstrate that resampling of the data had a positive impact on feature stability of linear support vector machines. thus, various random aspects of the training process may affect the reliability of modeling results, beyond predictive performance on a test set, especially in deep language models. in this paper, we propose a method for explaining classes in a text classification task using deep language models based on input attributions estimated with the integrated gradients (ig) method @xcite . we focus specifically on ig as it provides a general framework for estimating feature importance in deep neural networks and has been shown to provide reliable saliency maps in text classification among other tasks. for a discussion on the merits of ig, cf. @xcite , and bastings and filippova (2020) on saliency vs. attention methods in general. our class explanation method works by aggregating attributions in two ways: across documents and across models. on the one hand, we classify documents and aggregate word attribution scores from them, in order to extract the overall most predictive word features of a particular class. on the other hand, we aggregate these attributions over multiple random train/validation data splits and instances of a classifier, in order to identify stable attributions that are consistently assigned across rounds. thus, we consider the level of a particular classifier configuration-i.e., the combination of language model, decision layer, hyperparameters, loss function, etc.-and strive to capture its perception of a corpus. our method explains a class in the form of a list of words ranked by the aggregated attribution scores, and filtered based on their stability across experiments. following corpus linguistics' long tradition of analyzing style and content of text classes, we refer to these attributions as keywords (see @xcite . this type of analysis is concerned with identifying the words that are most informative about the characteristics conveyed by a given text class. while keyword analysis is widely employed in corpus linguistics, quantitative measures have been used only for extraction and not as a framework for evaluation, which is rather done qualitatively (cf. @xcite . thus, as a contribution of this paper, we propose three lexical measures of keyword quality, which help us optimize and evaluate our method. we also study syntactic and semantic properties to nuance our understanding of keywords obtained with a deep classifier and ig. we test our method by training a set of classifiers on the corpus of online registers of english (core) @xcite . core is sampled from the searchable english-language web and aims to be representative of the distribution of registers (or genres) found online. recent work in both linguistics and nlp has, however, demonstrated challenges of categorizing language use on the web pertaining to its extreme variation within and across classes @xcite @xcite . therefore, explanation methods are especially needed in web register classification, in order to explore the robustness and linguistic motivation of blackbox models. we put forward our stable attribution class explanation method (sacx) 1 as a support in understanding classes and their modeling by deep language model classifiers, in text classification tasks where keywords provide a suitable means of explanation. it can assist model development and debugging by highlighting salient word features, at a more general level compared to attributions at the document and classifier instance level. ## evaluation setting we evaluate the keyword quality based on usefulness and relevance, which are established concepts in feature selection and evaluation in machine learning (e.g., @xcite @xcite . usefulness refers to the discriminative power of the features used in a task, e.g., as measured by how well they allow to discriminate the classes in a test set. relevance refers to the association of the features with the actual object of study, i.e., their generalizability beyond a test set. not all useful features are relevant-for instance, some useful features may inherit their usefulness from data idiosyncrasies, unrepresentative train/test splits and spurious statistics (see @xcite . in the case of keywords, useful keywords allow to discriminate the classes in the data, while relevant keywords reflect meaningful and linguistically motivated characteristics associated with the classes. we propose three measures for assessing usefulness of keywords based on lexical overlap, presented in section 4.1, which we use to optimize parameters of our explanation method and to compare against the baseline methods. in section 4.2, we present further analyses conducted to assess the relevance of keywords and to form a qualitative understanding of the differences in output of the methods. the results are presented in section 5. ## results after completing @xmath0, we first inspect the predictive performances of the trained classifiers and study the degree of stability of the attributions. then, we report the results of the optimization of our method against the usefulness-focused lexical measures, and qualitatively inspect the extracted top keywords. finally, we present the syntactic and semantic analyses focusing on keyword relevance. ## conclusion we have presented the stable attribution class explanation method (sacx) for explaining classes in text classification, based on ig input attributions from deep language model classifiers. sacx produces lists of keywords reflecting a classifier's perception of classes. however, input attributions are prone to noise, which we have shown can be effectively filtered, as we performed 100 rounds of training an xlm-r classifier and applying ig. we have demonstrated that these stable keywords are of good quality-both useful as features and meaningfully relevant of the text classes studied. we have proposed lexical measures for evaluating distinctiveness and corpus coverage of keywords, and we have compared our method against two baseline class explanation methods. we compared the methods based on syntactic and semantic properties of the keywords, and found sacx to distinguish itself in that it extracts more content and less function words-a property which is generally considered to be a hallmark of a superior keyword analysis method in corpus linguistics. in particular, sacx has the ability to focus on more specific, topical words in the form of proper nouns, when relevant for depicting the class (such as for narrative). we have shown that sacx produces keywords that are highly coherent and tend to cluster densely throughout semantic vector space, rather than being evenly dispersed such as the word features extracted from svms. we also demonstrated that proper nouns are a distinguishing feature of these dense clusters, further illustrating the coherence of sacx keywords. we speculate that the use of token embeddings, and the xlm-r model's ability to learn local and highly non-linear functional forms afforded by the significant number of parameters, may give rise to these keyword characteristics. in the future, we seek to explore the utility of the method in various settings, and further investigate the quality and nature of its class explanations. we will test it on further text classification tasks and types of models, as well as apply the approach to other languages and cross-lingual settings. in particular, understanding model behavior in zero-shot classification through stable explanations at the class level may provide a useful tool in detecting systematic biases. in the context of register identification, recent pursuits in this direction of multi-and cross-lingual modeling @xcite @xcite have been making good progress in terms of predictive performance, but interpretability tools such as ours could offer linguistic insight, e.g., into language-independent markers of the classes. moreover, as we have demonstrated that input attributions are highly prone to noise at the level of individual classifier instances, the type of filtering we have proposed can be used, not only to stabilize class-level explanations, but, more generally to generate stable saliency maps for particular text inputs based on multiple classifier instances. future work should explore this direction further, as the contextualized interpretation of individual text inputs can provide a useful complement to the keywordbased class explanations for understanding model behavior.