Class Explanations: the Role of Domain-Specific Content and Stop Words (2023)

Abstract: We address two understudied areas related to explainability for neural text models. First, class explanations . What features are descriptive across a class, rather than explaining single input instances? Second, the type of features that are used for providing explanations. Does the explanation involve the statistical pattern of word usage or the presence of domain-specific content words? Here, we present a method to extract both class explanations and strategies to differentiate between two types of explanations – domain-specific signals or statistical variations in frequencies of common words. We demonstrate our method using a case study in which we analyse transcripts of political debates in the Swedish Riksdag.

## introduction recent developments in nlp are often the result of ever more complex model architectures and an increasing number of model parameters. yet, if we want to rely on these models, we should be able to review the similarities and dissimilarities between the model and human judgement. explainability frameworks can do this by highlighting on what the model has learnt to base its decisions. are these coincidental statistical patterns or something that a human would use as an explanation? @xcite argue that explanations should ideally be both functionally-grounded (true to the underlying machine learning model) as well as human-grounded (useful to a human). in this article, we propose a new method for extracting class explanations from text classifiers. besides, we also show a new way to distinguish between two types of features that appear in those explanations, that is, between informative content words and subtle statistical differences in common words' frequencies. our method aggregates explanations for individual data points (here provided by lime @xcite ), followed by a sorting stage that separates the different kinds of features. our work is in part motivated by use cases of machine learning for texts in the social sciences. in this field, explainability methods are relevant both as checks to compare against human expert knowledge and as a tool for bias detection. as a case study, we use our method to explain the decisions of a binary classifier trained to identify if speeches in the swedish riksdag belong to either of the two main parties, the moderates (m) or the social democrats (s). we find that our method can separate class explainability features and that those data points whose explanations contain primarily domainspecific content words are more often classified correctly. ## literature review as a result of the extensive work on explainability methods, a complex typology of different approaches exists (see @xcite or @xcite for a survey). one important distinction is between global and local. on the one hand, global methods aim to explain some general behaviour of a model, such as class explanations, which summarise the model with respect to a certain class. on the other, local methods aim to explain why the model assigned a single data point to a particular class. between global and local methods, the latter receive the most attention @xcite . three popular methods are gradient-based approaches @xcite , shapley values @xcite , and lime. gradient-based approaches use the model's weights and take the gradient with regard to the input. as such, they measure the change in the outcome given some small change in the input. yet, they are only an accurate reflection of the model if that model is linear @xcite , which is not the case for most deep nlp architectures. on the other hand, while shapley values have many theoretical guarantees to make them a faithful interpretation (they represent the true contributions of the features (ethayarajh and jurafsky, 2021)), their implementations (e.g. via attention flows for transformer-based architectures @xcite )) tend to be computationally expensive, which is problematic in the current setting, where we focus on aggregating a substantial number of individual explanations. finally, lime has an advantage over gradient-based approaches as it is model agnostic. this means that lime attempts to explain a trained classifier independently of its architecture @xcite . ## method our algorithm for computing class explanations consists of four steps: post-hoc instance explanations extraction, aggregation, sorting, and a keyword-in-context search that extracts example texts. this framework is formalized in algorithm 1. it is similar to sp-lime, but rather than searching for data points that capture the most diversity of the important features, we propose to work directly with the feature importance and explore ways to summarize and sort these by relevance. the replication materials and full results are available online 1 . ## data the dataset used for the case-study consists of transcripts of debates in the swedish riksdag, sourced from riksdagens öppna data -anföranden foot_1 . we use a pre-processed version available from språkbanken foot_2 consisting of debates from 1993 to 2018. for our experiment, texts from the social democrat (s) and moderate (m) parties have been extracted, resulting in 104,842 s and 62,160 m data points (one data point is one speech that could be part of a longer debate). from these, 100 examples have been sampled for a small-scale human baseline check, where two annotators are asked to perform the classification task of determining the party label from the speech texts and were evaluated against the true label. since these are debates, references to the opponent are a strong but trivial predictor of party. references to people and political parties have been removed by targeting swedish political party names' stems (for a full list please refer to the linked code base) and words tagged as "people along political spectrum" in språkbanken's tags, based on swedish framenet @xcite . since the cleanup is based on a coarse rule for party name stems detection and the automatic tags from språkbanken, not all references have been removed. we have opted for blanking all certain cases, so that enough of the interfering signal is removed to make the classification task non-trivial, rather than applying a comprehensive and exhaustive search of all mentions, since that is not the main goal of this work. data points shorter than 50 words have been removed, as manual analysis shows these tend to be entirely procedural and do not carry political sentiment. this is in line with similar cleaning practices used for us congressional debates @xcite . the data is undersampled to balance the classes and split into: train (108,169), test (12,019) and validation (2,000) sets. the validation set is used for explainability methods. ## experiments to test our methodology we apply it to a bert classifier trained to predict the party label of a text @xcite . the classifier is fine-tuned from a pre-trained model for swedish data released by the national library of sweden/kblab and available through the huggingface library foot_3 . the model has a 50,325 word vocabulary and 512 maximum token length. longer inputs are truncated. as a baseline for investigating class differences and separability of the data we use a logistic regression classifier, as this provides easy access to class explanations by simply looking at the top and bottom scoring internal weights of the model. n-gram spans from 1 to 3 and a combination of all have been compared. the number of input features is 50,325 -the same as the pre-trained bert model. a small-scale human annotation check on 100 instances shows the two annotators perform with 58 and 56 percent accuracy respectively. a cohen's kappa of 0.4 indicates this is a hard classification task. in the interest of space, the sections below contain partial results. the full results are available online. ## comparison to sp-lime our method is comparable with sp-lime, which aggregates individual lime explanations. sp-lime consists of three similar steps: post-hoc instance explanations extraction, sorting and exam-rank 1 sp-lime example (true label s): är (is), det (the), som (as), den (the), vi (we), natomedlemskap (nato membership), att (to), du (you), samlingsregeringen (the coalition government), vi (we) rank 2 sp-lime example (true label m): frågorna (the questions), protektionistiska (protectionist), önskar (wish), det (the), och (and), herr (mr), oerhört (incredibly), handelsminister (minister of trade), tackar (thanks), de (the) ... rank 12 sp-lime example (true label m): medelinkomsttagare (middle income earner), avregleringar (deregulations), vänster (left), tvivelaktiga (questionable), skattesänkningar (tax cuts), då (then), och (and), man (one/third person singular), bostadsmarknaden (the housing market), stöd (support) ... rank 16 sp-lime example (true label s): borgarna (the bourgeois), oss (us), långtidsarbetslösa (long-term unemployed), klyftorna (the cleavages), det (the), sjuka (sick), rödgröna (red green) 6 , vi (we), låt (let), är (is) we apply sp-lime to the bert classifier and extract the top 20 text examples that the explain-ability approach considers most representative. these contain 9 s examples and 11 m examples. a selected set of instance explanations can be seen in ## conclusion and discussion we have developed a new algorithm for extracting class explanations, which takes the distinction between stop words and content words into account. it thereby provides an alternative to prior methods like sp-lime, which mixes explanations based on e.g. stop word frequency with the presence of certain domain-specific terms. our motivation comes from the idea of human-grounded explainability: a useful explanation for a human will focus on content rather than stop words, while still being true to the model. in our case study, we demonstrated this for speeches from the swedish parliament, with the task of explaining a binary classifier associating speeches to either of the two main parties. this is a difficult task, our human annotation experiment showed humans performing just better than random, potentially as they primarily looked for clues about policy. the machine learning models performed better, as they likely also managed to identify statistical speech patterns of speakers, which we saw in explanations where e.g. stop words inevitably appear. our algorithm can not only identify these, but also separate them from explanations containing domainspecific words, hinting at policy, motivated by the needs of social scientists. additionally, we find indications that domain-specific explanations cor-relate with model performance. patterns related to policy in our experiment may be more robust than learned speech patterns of stop words, which risks being influenced by single frequent individuals in the dataset, rather than capturing patterns common to a political party. future work will focus on systematic and extensive testing of the proposed methodology in order to evaluate it along the twelve properties proposed by @xcite . the focus should be on measuring the faithfulness to the underlying black box model, correctness, as well as a larger scale domain expert evaluation to measure how relevant and valid the explanations are (context and coherence properties). the generalisability will also be tested, by studying other domains and classification tasks.