G rammar SHAP : An Efficient Model-Agnostic and Structure-Aware NLP Explainer (2022)

Abstract: Interpreting NLP models is fundamental for their development as it can shed light on hidden properties and unexpected behaviors. However, while transformer architectures exploit contextual information to enhance their predictive capabilities, most of the available methods to explain such predictions only provide importance scores at the word level. This work addresses the lack of feature attribution approaches that also take into account the sentence structure. We extend the SHAP framework by proposing GrammarSHAP—a model-agnostic explainer leveraging the sentence’s constituency parsing to generate hierarchical importance scores.

## introduction deep learning models have raised the bar in terms of performance in a variety of natural language processing (nlp) tasks @xcite . however, also model complexity has been steadily increasing, which in turn hinders the interpretability of their predictions. this is particularly true for transformer architectures, currently established as the state of the art in various applications but at the same time containing billions of parameters @xcite . local explanations have become a popular tool to understand and interpret models' decisions @xcite . thesebesides increasing the public's trust in machine learning systems-can uncover unwanted behaviors such as unintended bias @xcite . feature attribution explanations are the most commonly used and can highlight parts of the input text that are relevant for the obtained outcome @xcite . almost all available methods, however, can only attribute a relevance score to single words. this is highly unintuitive as natural language in human communication can be very articulated and contextdependent. indeed, a word's neighborhood can drastically alter its intended message and sentiment. our work focuses on generating explanations that account for the language structure. more specifically, we build hierarchical explanations that attribute relevance scores to sentence constituents at multiple levels. in contrast to previous work addressing the same issue @xcite , we build our approach as an extension of shap (lundberg and lee, 2017)-a local explainability framework renowned for its solid theoretical background. our contribution can be summarized as follows: ## related work several local explainability techniques exist to interpret predictions produced by nlp models @xcite . among them, features attribution (or feature relevance) approaches quantify each input component's contribution to the model's output, i.e. how each feature affects the observed prediction. methods in this category are available in a large variety: gradient-based @xcite , neuralnetwork specific e.g. lrp @xcite and deeplift @xcite , and modelagnostic e.g. lime @xcite . shap (lundberg and lee, 2017)-particularly relevant for our methodology-is by many considered to be a gold standard thanks to its solid theoretical background and broad applicability. this framework builds a unified view of methods like lime, lrp, and deeplift and the game-theoretic concept of shapley values @xcite . more recent works address the limitations of word-level relevance scores by focusing on phraselevel and hierarchical explanations. the proposed approaches analyze and quantify words' interactions through exhaustive search @xcite , combining their contextual decomposition scores @xcite , or via measuring shap interaction values along a predefined tree structure @xcite . chen and jordan (2020) combines a linguistic parse tree with banzhaf values @xcite to capture meaningful interactions in text inputs. @xcite , instead, propose to detect directly feature interaction without resorting to external structures. they propose a hierarchical explainability method that, in a topdown fashion, breaks down text components in shorter phrases and words based on the weakest detected interactions. ## methodology we extend the shap framework (lundberg and lee, 2017) by proposing a model-agnostic explainer that considers the text's structural dependencies to generate importance scores at multiple levels. in particular, we couple a constituency parsing layer to hierarchically select multi-word tokens with a custom version of kernelshap ## limitations and future work grammarshap meaningfully extends the shap framework by providing efficient hierarchical explanations that reflect the sentence structure. however, limitations of our methodology and experi-figure 3 : comparison of three explanation methods for grouped features relevance (5th level). distilbert predicted the sample's sentiment as negative with a 79.5% confidence. figure 4 : comparison of three explanation methods for grouped features relevance (5th level). distilbert predicted the sample's sentiment as negative with a 81.8% confidence. mentation need to be acknowledged and motivate our future work. regarding the explanation quality, our evaluation process is based on the introduced methodological improvements and on a qualitative analysis of the produced explanations. although evaluation metrics for explanations are complex to define and have not been standardized yet, our comparison would considerably benefit from the usage of quantitative diagnostic properties @xcite and word-level level metrics @xcite . in terms of execution time, our method is still reasonable considering the granularity of contributions that it can detect. however, the necessity for further improvements in terms of efficiency becomes apparent when producing real-time explanations on the large scale. ## conclusion in this work we proposed grammarshap: a modelagnostic explainer for text data that accounts for the sentence structure and the existing grammatical relationships between the text tokens. our approach leverages constituency parsing to extend the shap framework by providing hierarchical explanations that go beyond word-level attribution scores. our qualitative analysis of the produced explanation yields promising results as grammarshap appears to identify more fine-grained contribution in structured text than its existing shap counterparts. at the same time, the usage of masking tokens instead of a background dataset considerably speeds up its execution in comparison with kernal-shap. these properties make grammarshap also suitable for long texts, especially if they contain sentences carrying different types of sentiment. as a first priority for our future work, we will focus on the quantitative evaluation the produced explanation.