“Honey, Tell Me What’s Wrong”, Global Explanation of Textual Discriminative Models through Cooperative Generation (2023)

Abstract: The ubiquity of complex machine learning has raised the importance of model-agnostic explanation algorithms. These methods create artificial instances by slightly perturbing real instances, capturing shifts in model decisions. However, such methods rely on initial data and only provide explanations of the decision for these. To tackle these problems, we propose Therapy, the first global and model-agnostic explanation method adapted to text which requires no input dataset. Therapy generates texts following the distribution learned by a classifier through cooperative generation. Because it does not rely on initial samples, it allows to generate explanations even when data is absent (e.g., for confidentiality reasons). Moreover, conversely to existing methods that combine multiple local explanations into a global one, Therapy offers a global overview of the model behavior on the input space. Our experiments show that although using no input data to generate samples, Therapy provides insightful information about features used by the classifier that is competitive with the ones from methods relying on input samples and outperforms them when input samples are not specific to the studied model.

## introduction the emergence of machine learning models has led to their adoption in domains spanning from mere recommendations to critical areas such as healthcare @xcite and law @xcite . these already complex models keep becoming larger, emphasizing their black-box denomination. this lack of transparency however slows their adoption in various areas since we witness a notable rise of deployed models suffering from bias. for example, some chatbots biased toward religious @xcite * equal contribution. and gender @xcite minorities have been released and explaining their inner mechanisms is still an ongoing problem. among the methods proposed to tackle these problems, model-agnostic approaches are favored since applicable to any machine learning model. among these, local explanations have obtained strong success by maintaining a good trade-off between accuracy and transparency. these explanations are generated in the proximity of a target instance by tampering this input to create neighbors and study how the model reacts to these changes. this allows them to highlight which features are important for the model and to provide explanations on the decision for this input (e.g., the most important words for each class). according to a recent study @xcite , @xcite , while being the first model-agnostic local explanation method is still the most widely used. however, local explanations have three main flaws when trying to explain a model. first, it obviously requires to have inputs to explain, which might not be possible due to confidentiality or privacy reasons @xcite . second, selecting inputs that are representative of the model or the downstream data distribution is difficult. finally, it will explain the decision for this input and for this input only. this only provides very local information on the model behavior, which represents only a very small piece of the input domain of the model. therefore, lime and other local explanation methods have proposed to aggregate the information from multiple samples to provide global explanations. however, these explanations are strongly tied to the input samples and only provide cues about the samples' neighborhood. these methods thus require samples that cover as much of the space as possible. to relax this sample dependency and generate global explanations of the model, we propose therapy, a method that leverages cooperative genera-tion @xcite @xcite to generate texts following the distribution of a classifier. the distribution of the resulting samples can then be used to study which features are important for the model, providing global information on its behavior. in this paper, we first introduce the related work in section 2 and cooperative text generation in section 3. we then present therapy in section 4 and the experiments conducted to compare its performance to standard explanation methods in section 5. ## related work generating explanations for textual data is challenging since it requires considering both the text semantics and task domains. moreover, it is frequent that models are already deployed and further evaluations are required (e.g., fairness, bias detection) but the training data is not accessible. this may be caused by data privacy, security, or simply because the dataset is too large to be analyzed. thus, to fulfil this objective, researchers have focused on post-hoc explanations @xcite . following the categorization by bodria et al. @xcite , we distinguish between example-based and feature-attribution explanations. ## method in this paper, we introduce therapy, a global and model-agnostic explanation method that does not require input data. in place of these input data, therapy employs an lm guided by the model to explain. this cooperation generates texts that are representative of the classes learned by the studied discriminative model. to do so, therapy extracts the most important words for the classifier by employing it to steer an lm through cooperative generation. texts generated using cooperative generation follow the distribution p(x) * d(c | x). their distribution can thus be used to study the classifier d: words with high frequencies are likely to be important for the classifier. a logistic regression is then learned on tf-idf representations of generated samples and the weights associated with each term are returned as the explanation. an illustration of the method is proposed in figure 1 . because p(x) is the same for every class, by using tf-idf on the whole corpus (i.e., samples from every class), words that are frequent because of p(x) or in multiple classes will be filtered out. hence, the logistic regression model learned on the tf-idf score of each feature allows therapy to study their relative importance and to extract the most important ones for each class. the method thus offers the level of explainability of n-grams based on logistic regression models to any classifier. indeed, since any type of (auto-regressive) lm can be guided during decoding by any classifier using mcts, the proposed approach is totally model-agnostic. we call this approach therapy because its functioning is similar to that of a therapist. this therapist (the lm) queries its patient (the classifier) to understand its behavior and eventually discover pathologic behaviors (some biases). in essence, the method is similar to using lime jointly with a masked lm to generate neighbors when the number of replaced tokens grows a lot but with two benefits. first, the method does not rely on input examples but creates samples out of nothing using the lm. this is useful for cases where the data cannot be shared because it contains confidential information (amin-nejad et al., 2020). moreover, rather than exploring the neighborhood of these examples (and so conditioning the explanations on these examples' context), the domain of the exploration is defined by the domain of the lm, which is significantly broader. besides, either a general lm can be used to study the model behavior on generic data or an lm specific to the downstream domain to make sure it works well on this specific type of data. second, the method does not generate before classifying the text but employs the classifier during the generation. hence, instead of "randomly" generating texts and hoping for important features to appear, we explicitly query the model for stereotypic features by maximizing d(c | x). this makes the method more efficient and reduces the probability of generating rare features that are not important for the model while reducing the odds of generating "in the middle" texts containing features from various classes that are misleading. besides, our method directly relies on the distribution learned by the studied model to guide the generation, unlike methods like polyjuice and gyc, which, in addi-tion to requiring input data, count on a distribution learned by the lm to bias the generation towards the desired property (using control codes). finally, therapy is distinctive from methods analyzing the frequency of input terms in the training data such as sensitivity analysis since it does not require access to (training) data and directly exploits the distribution effectively learned by the model, whereas nothing guarantees that a model is actually using the terms extracted from training data to make a prediction. furthermore, our method differs from existing example-based and feature attribution methods since to the best of our knowledge, there exists no global and model-agnostic explanation methods that do not require any input data. ## experiments in this section, we first give technical details on the experiments conducted to evaluate therapy (section 5.1). we then evaluate therapy through three experiments. the first one (section 5.2), measures the spearman correlation of the explanations and the weights of a glass box and studies the influence of the number of generated texts on the quality of the explanation returned by the linear model. we then compare the capacity of the method to correctly identify the most important words of the glass box to the one of lime and shap using precision/recall curves in section 5.3. finally, we test whether the terms returned by the different approaches are sufficient to modify the prediction of the classifier in section 5.4. the code of therapy and our experiments will be made available upon acceptance. ## conclusion usual explainability methods heavily rely on input data, which is not necessarily available and might not contain model biases or important features. we propose therapy, a method that leverages cooperative textual generation to create synthetic data that follow the studied model distribution. thus, the search is driven by a pre-trained lm rather than input samples. the pre-trained lm allows a broader exploration than being restricted to input data neighborhood, relaxing most of the constraints and a priori induced by examples-driven methods. in the extreme case where extremely representative data (such as the test set of a given dataset) of important features of the model is available, therapy lacks a bit behind state-of-the-art shap while being competitive. however, when considering more realistic cases where we do not explicitly give the important features to the explainer or do not have any available data, its performances are very good whereas the other methods are collapsing when even applicable. comparisons with a generate-then-classify baseline highlight the benefits of the cooperative generation when the lm does not generate texts that are representative of a single specific class by itself. therefore, therapy is a useful tool to explore the model behavior on a large domain when collecting data that exactly match the downstream distribution is not feasible. finally, we opposed the proposed approach to lime and shap to highlight the interest of generating representative texts using cooperative generation when input data is lacking. however, an interesting avenue of research would be to use these established explainability methods on cooperatively generated texts, replacing the proposed logistic regression on the tf-idf representation. this potential combination might allow to leverage their performance while alleviating the input data dependency. a.3 ag_news, "world" class samples: • the world's most popular and sought after luxury home in dubai, the al khaleel international airport (ali), is a city that is renowned as the hub of the middle east and north africa region. it is also the capital city of abu dhabi, united arab emirates with a population over 2 billion people and a gdp per person.al-khaliel airport is an important gateway into europe to the north east and west of europe, asia, and australia • in the last month, the us government has been accused of using a "secret" military intelligence agency in iraq and afghanistan for spying on its allies, including pakistan, turkey, iran, russia, syria, sudan, lebanon, libya, egypt, nigeria. . . and even saudi arabia!according to a new report published today, the pentagon is now investigating the alleged use by the us military intelligence community (iuc), which was created by president bush after the 9/11 terror attack that killed over top-words: people, man, country, city, party, killed, family, agree, wrong, general, children, sex, president, police, working, military, dead, missing, woman, days a.4 ag_news, "sport" class samples: top-words: time, game, back, season, play, didn, team, guy, field, night, games, left, 12, title, won, saturday, playing, great, day, wasn a.5 ag_news, "business" class samples: top-words: ve, ll, idea, phone, internet, make, system, video, online, life, understand, version, pc, found, 13, thing, computer, lot, hard, issue, people, work, information, future