SHAP -Based Explanation Methods: A Review for NLP Interpretability (2022)

Abstract: Model explanations are crucial for the transparent, safe, and trustworthy deployment of machine learning models. The SHapley Additive exPlanations (SHAP) framework is considered by many to be a gold standard for local explanations thanks to its solid theoretical background and general applicability. In the years following its publication, several variants appeared in the literature—presenting adaptations in the core assumptions and target applications. In this work, we review all relevant SHAP-based interpretability approaches available to date and provide instructive examples as well as recommendations regarding their applicability to NLP use cases.

## introduction several methods have been proposed to address the issue of opacity in modern machine learning models. most notoriously, explanations are fundamental for deep neural networks (dnns) @xcite @xcite as these automatically learn millions of parameters and behave like black-boxes. lundberg and @xcite proposes shapley additive explanations (shap), a unified local-interpretability framework with a rigorous theoretical foundation on the game-theoretic concept of shapley values @xcite . shap is nowadays considered a core contribution to the field of explainable artificial intelligence (xai). following its publication, a variety of explainability approaches based on shap's methodology has populated the literature and this trend continues to grow. some present a new version of shap tailored to a certain type of input data-e.g. graphs @xcite ) and text @xcite )-or to specific models such as random forests @xcite . others, instead, modify shap's underlying assumptionse.g. features independence-to increase the original framework's flexibility for cases in which they are too strict or overly simplistic @xcite . in this work, we (1) identify five broad research directions inspired by shap, (2) review available shap-based (or shapley-value-based) approaches as members of such categories, and (3) investigate their applicability in the domain of natural language processing (nlp). our work reviews 41 methods with a particular focus on their core assumptions, input require-ments, explanation form, and available implementations. furthermore, we provide nlp researchers with use-case-based recommendations and instructive examples. ## background for the sake of clarity, we provide a gentle introduction to shapley values and the methods for their estimation, most notably shap. all concepts will be explained informally, resorting to formalities when necessary. ## search and selection criteria as the popularity of shap increases, also the number of approaches based on it or directly on shapley values has been on the rise. in fact, ∼ 3, 200 of the ∼ 6, 900 papers citing lundberg and lee (2017) are from 2021, an exponential increase when compared to previous years (1563, 567, and 118) foot_1 . besides the papers already known to us, we manually screened all works citing shap with at least 15 citations 2 . this systematical search, based on the assumption that shap-based approaches should at least reference lundberg and lee (2017), helped us uncover several relevant contributions and mitigate the selection bias induced by our previous knowledge. the threshold of 15 citations was introduced to speed up our manual search and to filter out works that have not received the research community's attention. to account for temporal bias-i.e. that publications accumulate citations over time-we lowered the threshold to 10 for papers published in the most recent years (2021 and 2022) 2 . we only consider and review papers that contributed new shap-based approaches and exclude those-like @xcite and ( antwarg et al., 2019 )-utilizing shap (almost) off-theshelf. similarly, we exclude works such as wang et al. (2020) and huber et al. (2022) utilizing shapley values for purposes not directly connected with explainability. ## existing reviews previous reviews like @xcite , vilone and longo (2020), and @xcite present extensive overviews of explainability methods, but only briefly mention shap and a few of its derivates. others-such as @xcite , @xcite , and kumar et al. (2020)-review some shapley-based methods in detail (between 5 and 9) but do not construct a comprehensive review. our work, in contrast, significantly extends this range and covers more than 40 approaches. ## review: shap-based approaches several works proposed methods based on shap, or more generally on shapley values, following the contribution from @xcite . while the changes and variations introduced have been at times criticized for not being as rigorous as shap in following its core assumptions @xcite , shap-based methods continue to increase in both quantity and popularity. our review categorizes shap-based approaches available to date based on how they differ from and how they improve on the original shap framework. we identify five broad categories in the existing literature, each one of them describing a different research direction pursued by its members: in some cases, approaches are used complementary for applications dealing with multimodal inputs @xcite . clearly, these categories are not designed to be exclusive. therefore, an approach can fall in more than one if it differs from shap in multiple aspects. ## relevance for nlp research large and complex neural nlp models-such as bert @xcite and gpt-3 @xcite -are used extensively in research and industry. the trend is justified by the strong correlation between models' size and their performance @xcite . naturally, increasing model complexity causes a higher demand for nlp explainability. in this section, we match this demand to the reviewed shap-based methods and provide researchers with use-casebased recommendations. ## criticisms the usage of shapley values for generating model explanations has also been criticized. for instance, @xcite shows that using shapley values for feature importance leads to mathematical inconsistencies which can only be mitigated by introducing further complexity like causality assumptions. moreover, the authors argue that shapley values do not represent an intuitive solution to the human-centric goals of model explanations and thus are only suitable in a limited range of settings. @xcite , on the other hand, criticize some shapley-value-based methods. in fact, while a strong case for utilizing shapley values can be made thanks to their uniqueness result in satisfying certain properties (see 2.1), often methods employing them operate under different assumptions and hence the uniqueness results loses validity in their context. @xcite argues that existing shap-based literature focuses on the axiomatic foundation of shapley values and their efficient estimation but neglects the uncertainty of the explanations produced. the authors illustrate how small differences in the underlying game formulation can lead to sudden leaps in shapley values and can attribute a positive contribution to features that do not play any role in the machine learning model. this work has reviewed a total of 41 approaches and has organized them based on the introduced categories. as expected, given the overlapping nature of the classification, the majority of existing methods fall into multiple categories and have therefore each made distinct contributions to the field. while most of them are not directly applicable to nlp settings, we identified a few that can be beneficial for current practitioners. furthermore, we have compiled a list of recommendations for each nlp use case. we also observe a severe limitation of shap-based methods in terms of applicability to sequence-to-sequence nlp tasks. we hope our work provides nlp/xai practitioners and newcomers with a comprehensive overview of shap-based approaches, with references to stimulate further investigation and future advances in academic and industrial research.