T oken SHAP : Interpreting Large Language Models with M onte C arlo Shapley Value Estimation (2024)

Abstract: As large language models (LLMs) become increasingly prevalent in critical applications, the need for interpretable AI has grown. We introduce TokenSHAP, a novel method for interpreting LLMs by attributing importance to individual tokens or substrings within input prompts. This approach adapts Shapley values from cooperative game theory to natural language processing, offering a rigorous framework for understanding how different parts of an input contribute to a model’s response. TokenSHAP leverages Monte Carlo sampling for computational efficiency, providing interpretable, quantitative measures of token importance. We demonstrate its efficacy across diverse prompts and LLM architectures, showing consistent improvements over existing baselines in alignment with human judgments, faithfulness to model behavior, and consistency. Our method’s ability to capture nuanced interactions between tokens provides valuable insights into LLM behavior, enhancing model transparency, improving prompt engineering, and aiding in the development of more reliable AI systems. TokenSHAP represents a significant step towards the necessary interpretability for responsible AI deployment, contributing to the broader goal of creating more transparent, accountable, and trustworthy AI systems. Open Source code https://github.com/ronigold/TokenSHAP

## introduction large language models (llms) have greatly advanced natural language processing, delivering near or at human-level performance on many tasks. however, their "black box" nature poses interpretability challenges, crucial for applications in fields like healthcare and legal analysis, where understanding ai decision-making is vital. this paper introduces tokenshap, a method enhancing llm interpretability by adapting shapley values from game theory. tokenshap treats input tokens as players, assessing their contribution to model outputs. this allows for a deeper understanding of how llms process information, crucial for improving model transparency and reliability. we propose a monte carlo sampling method for practical shapley value estimation, accommodating the variable lengths and contextual nature of language inputs. our evaluations across different prompts and models confirm tokenshap's versatility and effectiveness in revealing llm decisionmaking processes. this breakthrough aids the development of more accountable ai systems, ensuring their responsible use as they become more integrated into critical applications. ## conclusion tokenshap offers a significant advancement in the interpretability of large language models (llms) by adapting shapley values to natural language processing and employing monte carlo estimation for feasibility. this approach overcomes the challenges of variable input lengths and contextual dependencies, offering a scalable solution for complex language models. key achievements include: our method's capacity to capture detailed token interactions enhances model transparency and aids in debugging, bias mitigation, and regulatory compliance, which is essential as llms are increasingly deployed in critical domains. future research will explore sophisticated value functions, the stability of shapley values across models, and the extension of tokenshap to conversational ai. developing interactive tools based on tokenshap could also enhance its accessibility and practical utility for practitioners. tokenshap represents a vital step towards making ai systems not only powerful but also transparent and accountable, ensuring their responsible development and deployment in transformative applications.