Interpreting Text Classifiers by Learning Context-sensitive Influence of Words (2021)

Abstract: Many existing approaches for interpreting text classification models focus on providing importance scores for parts of the input text, such as words, but without a way to test or improve the interpretation method itself. This has the effect of compounding the problem of understanding or building trust in the model, with the interpretation method itself adding to the opacity of the model. Further, importance scores on individual examples are usually not enough to provide a sufficient picture of model behavior. To address these concerns, we propose MOXIE (MOdeling conteXt-sensitive InfluencE of words) with an aim to enable a richer interface for a user to interact with the model being interpreted and to produce testable predictions. In particular, we aim to make predictions for importance scores, counterfactuals and learned biases with MOXIE. In addition, with a global learning objective, MOXIE provides a clear path for testing and improving itself. We evaluate the reliability and efficiency of MOXIE on the task of sentiment analysis.

## introduction interpretability, while under-specified as a goal, is a crucial requirement for artificial intelligence (ai) agents @xcite . for text classification models, where much of the recent success has come from large and opaque neural network models @xcite @xcite , a popular approach to enable interpretability is to provide importance scores for parts of the input text, such as words, or phrases. given only these numbers, it is difficult for a user to understand or build trust in the model. going beyond individual examples, such as scalable and testable methods input text: he played a homosexual character model prediction: negative sentiment 1 question 1 (importance scores): which words had the most influence towards the prediction? is the word 'homosexual' among them? answer: the word 'homosexual' has the highest negative influence. question 2 (counterfactuals): if so, which words instead would have made the prediction positive? answer: if you replace the word 'homosexual' with the word 'straight', the model would have made a positive sentiment prediction. question 3 (biases): is there a general bias against the word 'homosexual' compared to the word 'straight'? answer: yes, there are a large number of contexts where the model predicts negatively with the word 'homosexual', but positively with the word 'straight'. here are some examples: to identify biases at a dataset level, are desired but currently missing. questions can be raised about whether the methods of interpretation themselves are trustworthy. recent analyses @xcite of such interpretation methods for computer vision tasks suggest that such skepticism is valid and important. a method which aims to elucidate a black-box's behavior should not create additional black boxes. measuring trustworthiness, or faithfulness foot_0 , of interpretation methods, is itself a challenging task @xcite . human evaluation is not only expensive, but as jacovi and goldberg (2020) note human-judgments of quality shouldn't be used to test the faithfulness of importance scores. what needs testing is whether these scores reflect what has been learned by the model being interpreted, and not whether they are plausible scores. we believe the aforementioned issues in existing methods that produce importance scores can be circumvented through the following changes. a global learning objective: several existing approaches rely on some heuristic to come up with importance scores, such as gradients @xcite , attentions @xcite , or locally valid classifiers @xcite (see @xcite for a broad survey). instead, we propose to identify a global learning objective which, when learned, enables prediction of importance scores, with the assumption that if the learning objective was learned perfectly, we would completely trust the predictions. this would provide a clear path for testing and improving the interpretation method itself. quick and automatic evaluation on a held-out test set allows progress using standard machine learning (ml) techniques. going beyond importance scores: importance scores, even when generated using a theoretically inspired framework @xcite , are generally hard to evaluate. further, the aim of the interpretation method shouldn't be producing importance scores alone, but to enable a user to explore and understand model behavior foot_1 , potentially over large datasets. in importance scores 'which parts of the input text were most influential for the prediction?' such importance scores, popular in existing approaches, can provide useful insights but are hard to evaluate. counterfactuals 'can it predict counterfactuals?' we define a good counterfactual as one with minimal changes to the input text while causing the model to change its decision. such predictions can be revealing but easy to test. they can provide insights into model behavior across a potentially large vocabulary of words. in this work, we consider counterfactuals obtained by replacing words in the input text with other words in the vocabulary. we limit to one replacement. biases 'is the model biased against certain words?' for example, we could ask if the model is biased against lgbtq words, such as the word 'homosexual' compared to the word 'straight'? one way to provide an answer to such a question is to evaluate a large number of contexts, replacing a word in the original context with the words'homosexual' and 'straight'. doing that however is prohibitive with large text classification models. if an interpretation method can do this in a reasonable time and accuracy, it enables a user access to model behavior across a large number of contexts. considering the preceding requirements, we propose moxie (modeling context-sensitive influ-ence of words) to enable a reliable interface for a user to query a neural network based text classification model beyond model predictions. in moxie, we aim to learn the context-sensitive influence of words (see figure 1 for the overall architecture). we show that learning this objective enables answers to the aforementioned questions (section 3.2). further, having a global learning objective provides an automatic way to test the interpretation method as a whole and improve it using the standard ml pipeline (section 3.3). we evaluate the reliability and efficiency of moxie on the task of sentiment analysis (section 4) foot_2 . ## related work word importance scores have been a popular area of research for interpreting text classifiers, including gradient based methods @xcite , using nearest neighbors @xcite , intrinsic model-provided scores such as attention @xcite , and scores learned through perturbations of the test example @xcite . there has also been effort to expand the scope to phrases @xcite , as well as provide hierarchical importance scores @xcite . however these methods tend to derive from an underlying heuristic applicable at the example level to get the importance scores. with text z y f a c context processor a m : combine a t token processor masked text token masked + z'' '… very , very slow .' '… very , @xmath0.' '… very , @xmath1.' 'slow' h r c (linear layer) y'' model being interpreted (teacher model, f) interpretation model (student model, g) minimize distance(z, z'') . . very very slow' to produce a representation z through module m and label scores y through a linear classification layer c. when presented with the same input but the word 'slow' masked, it produces outputs z and y respectively. we learn the difference in the two representations (z -z ) as a proxy for the context-sensitive influence of the word 'slow' in the student model (g). this is done by processing the masked context and the token masked through arbitrarily complex modules a c and a t which produce fixed length representations z c and z t respectively. the combine module (a m ) takes these as input to produce the output r. we learn by minimizing the mean square error between z and @xmath2. keeping the combine module shallow allows the processing of a large number of tokens for a given context and vice versa in a reasonable time. please see section 3.1 for details on the architecture and section 3.2 for how this architecture enables answers to the motivating questions. while it's not inherently problematic to use such heuristics, it makes it hard to improve upon the method, as we need to rely on indirect measures to evaluate the method. further, recent work has shown that the skepticism in the existing methods is valid and important @xcite . in this work, we use a global learning objective which allows us to make predictions of importance scores. apart from word importance scores, explanation by example style method have been studied @xcite . like word importance based methods, however, these methods don't provide a clear recipe for further analysis of the model. in this work, we aim to produce testable predictions such as counterfactuals and potential biases. measuring faithfulness of an interpretation model can be hard. @xcite suggest that human evaluation shouldn't be used. in this work, we circumvent the hard problem of evaluating the faithfulness of an interpretation method by making it output predictions which can be tested by the model being interpreted. ## moxie the overall architecture employed to learn moxie is shown in figure 1 . we introduce the notation and describe the architecture in detail in section 3.1. in section 3.2, we discuss how moxie provides answers to the motivating questions. ## experiments we aim to answer the following questions: q1 how well does the student model approximate the teacher model? (section 4.2) q2 how does moxie compare with methods which access test example neighborhoods to generate importance scores? (section 4.3) q3 can moxie reliably produce counterfactuals? (section 4.4) q4 can moxie predict potential biases against certain words? (section 4.5) we use the task of binary sentiment classification on the stanford sentiment treebank-2 (sst-2) dataset @xcite for training and evaluation. in section 4.1.2, we provide text preprocessing details. we evaluate the student model accuracy against the teacher model (q1) across four models: bert-base-cased @xcite , roberta-base @xcite , xlmrbase @xcite , roberta-large @xcite . for the rest of the evaluation, we use roberta-base as the teacher model. we use the hugging face transformers library v3.0.2 @xcite for our experiments. ## conclusion in summary, we have shown that moxie provides a novel framework for interpreting text classifiers and a method to draw quick insights about the model on large datasets. moxie can make efficient, testable and reliable predictions beyond importance score, such as counterfactuals and potential biases. further, with a global learning objective, it provides a clear path for improving itself using the standard ml pipeline. finally, the principles and the evaluation methodology should help the interpretability research overcome the problem of testing the faithfulness of interpretation methods. as future work, we identify improving the accuracy of the student model. further analysis of the nature of counterfactuals selected by the student model could lead to useful insights towards improving the interpretation method. finally, identifying other learning objectives which enable testable predictions would be useful and challenging.