S yntax S hap: Syntax-aware Explainability Method for Text Generation (2024)

Abstract: To harness the power of large language models in safety-critical domains, we need to ensure the explainability of their predictions. However, despite the significant attention to model interpretability, there remains an unexplored domain in explaining sequence-to-sequence tasks using methods tailored for textual data. This paper introduces *SyntaxShap*, a local, model-agnostic explainability method for text generation that takes into consideration the syntax in the text data. The presented work extends Shapley values to account for parsing-based syntactic dependencies. Taking a game theoric approach, SyntaxShap only considers coalitions constraint by the dependency tree. We adopt a model-based evaluation to compare SyntaxShap and its weighted form to state-of-the-art explainability methods adapted to text generation tasks, using diverse metrics including faithfulness, coherency, and semantic alignment of the explanations to the model. We show that our syntax-aware method produces explanations that help build more faithful and coherent explanations for predictions by autoregressive models. Confronted with the misalignment of human and AI model reasoning, this paper also highlights the need for cautious evaluation strategies in explainable AI.

## introduction language model (lm) interpretability has become very important with the popularity of generative ai. despite the great results achieved by the most recent lms, there is still a large range of tasks where the models fail, e.g., capturing negations @xcite . therefore, it is crucial to get a better understanding of the lm reasoning and develop faithful explainability methods. as many lms have little transparency and their use is restricted to api calls, model-agnostic explainability methods have become the most practical techniques for gaining better insights into lms. the shapley additive explanations (shap) framework is popular for generating local explanations thanks to its solid theoretical background and general applicability @xcite . however, regarding sequence-to-sequence tasks such as next token generation, the usage of shap-based methods has not been explored in depth @xcite . we address this gap and develop a coalition-based explainability method inspired by shapley values for text generation explanation. our explainability method (in figure 1 ) considers syntactic word dependencies @xcite . the syntax is important as next-word predictions in autoregressive (ar) lms underlie implicit incremental syntactic inferences, i.e., lms implicitly capture dependencies in text data @xcite . in this paper, we investigate if dependency parsing trees can be used in the explainability process as syntactic relational graphs and help shed light on the influence of words on the model's prediction given their syntactic role in the sentence. we evaluate the explanations on diverse metrics. first, we adapt fidelity, one of the most popular model-based evaluation metrics in xai (explainable ai), to the text generation task and introduce two new metrics to test whether the generated explanations are faithful to the underlying model. second, we introduce two qualitative evaluation metrics that capture the explanation quality with regard to human expectations, i.e., the coherency of explanations and their semantic alignment. our evaluation procedure compares our method syn-taxshap to state-of-the-art explainability methods. explanations produced by our method of the next token generation by two popular ar models respect the sentence syntactic structure while being faithful and more coherent compared to state-ofthe-art shap-based methods that do not explicitly consider the word dependency for text generation figure 1 : given an input sentence, an autoregressive language model (ar lm) predicts the next token. the syntax of the sentence is extracted using dependency parsing (spacy @xcite ). to measure the importance of the word husband for the model to predict the next token wife, our method (1) extracts multiple coalitions of words following specific paths in the dependency tree, (2) analyze the contribution of adding husband to each coalition in the change of probability to predict the next token wife, and (3) average those contributions to compute its final syntaxshap value. tasks. our semantic alignment analysis also proves that faithful explanations do not necessarily meet human expectations in terms of token importance, highlighting a clear disagreement between our human mental model and ai models. to summarize, our contributions are (1) syn-taxshap, a new shap-based explainability method that incorporates dependency tree information, (2) quantitative metrics that address lm's stochasticity and qualitative metrics to account for human semantic expectations, and (3) an evaluation of the explanation quality on two ar lms. our work opens multiple research directions for future work. ## evaluation this section describes our model-based evaluation procedure that encompasses both quantitative and qualitative analysis of the explanations. while previous works only focus on the faithfulness of explanations to assess their quality, we also propose to consider human qualitative expectations. ## experiments we evaluate syntaxshap and syntaxshap-w on various criteria such as faithfulness in section subsection 5.2, the coherency in section subsection 5.3, and the semantic alignment of their explanations in section subsection 5.4. ## discussion addressing stochasticity the traditional faithfulness metrics like fidelity, aopc @xcite or log-odds @xcite scores take a deterministic approach to evaluate explanations computed on stochastic predictions. this paper evaluated ar lms that adopt top-k sampling to randomly select a token among the k tokens with the highest probability. to account for this stochasticity, we proposed additional evaluation metrics, div@k and acc@k, that consider not only the final prediction but the top-k predictions, balancing the model's probabilistic nature. nevertheless, further methods that address the stochastic nature of the models should be designed in future research. ## conclusion we proposed syntaxshap -a local, model-agnostic syntax-aware explanability method. our method is specifically tailored for text data and meant to explain text generation tasks by ar lms, whose interpretability in that context has not yet been addressed. syntaxshap is the first shap-based method to incorporate the syntax of input sentences by constraining the construction of word coalitions on the dependency trees. our experimental results demonstrate that syntaxshap and its weighted variant can improve explanations in many aspects: they generate more faithful and coherent explanations than the standard model-agnostic explainability methods in nlp for advanced ar lms like mistral 7b. the semantic alignment analysis also shows that faithfulness and human intelligibility are distinct evaluation criteria that do not necessarily align. this study addresses a pressing and significant issue regarding the explainability of ar models, contributing to an ongoing dialogue in the research community.