On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLM s (2025)

Abstract: Recent advancements in NLP systems, particularly with the introduction of LLMs, have led to widespread adoption of these systems by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and scientific research. This surge in usage has led to an explosion in NLP model interpretability and analysis research, accompanied by numerous technical surveys. Yet, these surveys often overlook the needs and perspectives of explanation stakeholders. In this paper, we address three fundamental questions: Why do we need interpretability, what are we interpreting, and how? By exploring these questions, we examine existing interpretability paradigms, their properties, and their relevance to different stakeholders. We further explore the practical implications of these paradigms by analyzing trends from the past decade across multiple research fields. To this end, we retrieved thousands of papers and employed an LLM to characterize them. Our analysis reveals significant disparities between NLP developers and non-developer users, as well as between research fields, underscoring the diverse needs of stakeholders. For example, explanations of internal model components are rarely used outside the NLP field. We hope this paper informs the future design, development, and application of methods that align with the objectives and requirements of various stakeholders.

## introduction recent advancements in natural language processing (nlp), particularly with the introduction of large language models (llms), have dramatically enhanced model performance. these models are now capable of executing a wide array of tasks and have been adopted across various domains and research fields @xcite @xcite . their applications extend beyond the nlp community, and they are widely used by the general public @xcite 0 100 200 300 400 500 600 700 78% 84% 79% 74% 77% 73% 22% 16% 21% 26% 23% 27% # nlp interpretability papers field nlp outside '16 '17 '18 '19 '20 '21 '22 '23 '24 0 25 50 75 100 125 150 175 outside nlp humanities neuroscience psychology education economics natural sci social sci medicine shamszare, 2023; @xcite . however, these black-box models are complex and opaque @xcite @xcite . while performance has advanced, this comes at the cost of understanding their underlying mechanisms @xcite @xcite . the ability to explain decisions is particularly crucial, given that nlp models, especially llms, significantly influence individual decision-making @xcite , society @xcite , the job market @xcite , and academic research @xcite . moreover, model interpretability and analysis are utilized for scientific insights and discoveries @xcite @xcite @xcite . unsurprisingly, research on model interpretability and analysis has become one of the most prolific areas within the nlp community and beyond, yielding thousands of publications in recent years, as illustrated in figure 1 . consequently, many tech-nical nlp model interpretability and analysis surveys have emerged, reviewing hundreds of methods @xcite @xcite @xcite @xcite . in this paper, we aim to bridge a gap in the existing literature and discuss model interpretability from the stakeholders' perspective. our goals are to broaden the nlp community's point of view on the application of interpretability methods in various fields and to promote the design and development of methods that align with the objectives, expectations, and requirements of various stakeholders. we will explore three key questions: why do we need interpretability ( §2), what are we interpreting ( §3), and how are we interpreting ( §4)? this approach allows us to examine common interpretability paradigms ( we start by presenting four perspectives on interpretability and their relevant stakeholders in §2. next, in §3, we address a pressing issue in the literature: the lack of consensus on the definition of interpretability. we examine various definitions within and outside the nlp community and propose a broad definition: extracting insights into a mechanism of the nlp system and communicating them to the stakeholders in understandable terms. in §4, we propose six properties of interpretability methods and discuss the relevance of each property to different stakeholders. for example, the scope property distinguishes between local and global explanations. if the stakeholder is a physician, a local explanation that clarifies the prediction for an individual patient is preferred. conversely, a global explanation is more suitable for a scientist, as it facilitates understanding broader phenomena. we survey in appendix §b seven prevalent interpretability paradigms, explain which properties characterize each (see following that, in §5 we aim to understand how the paradigms and their properties are reflected in practice by analyzing trends over the years and across different research fields. to this end, we retrieved over 14,000 papers using the semantic scholar api and employed an llm to select only relevant ones, resulting in 2,000 papers. furthermore, we utilized the llm to annotate papers with their interpretability paradigm and properties. 1 importantly, the llm annotation is in strong agreement (over 90%) with human expert annotation. to the best of our knowledge, this is the first successful application of an llm for such a task. below, we summarise our main findings: 1. within the nlp community, interpretability paradigm trends have remained stable over the decade. however, the introduction of llms in the past two years has prompted a notable shift. 2. outside the nlp community, our main claim gains support: different stakeholders have varying needs, reflected in significant differences between research fields in terms of both the paradigms and their properties. 3. comparing nlp developers to non-developers reveals that the latter group is less interested in understanding internal model components. 4. non-developers opt for popular methods not originally developed within the nlp community, such as shap and lime, likely due to their user-friendly and easy-to-apply software. 5. llms have shifted the trends in interpretability research: not only has the number of published papers doubled, but there has also been a substantial increase in the use of natural language explanations. these explanations are utilized in nearly half of the papers outside the nlp field. we hope this first-of-its-kind paper, which reviews nlp interpretability through the stakeholders' perspective and rigorously analyzes trends within and outside the nlp field, will pave the way for improved design, development, and application of these essential methods. to further this aim, we outline in §6 practical steps that nlp researchers can undertake to promote the adoption of interpretability methods in other disciplines. ## why do we need interpretability? understanding why interpretability is necessary provides a solid framework for discussing, assessing, and enhancing interpretability methods, ensuring they meet practical objectives and expectations. so, when and why do we need interpretability? we gather ideas from other surveys @xcite @xcite and propose a decomposition of the need for interpretability into four perspectives (see figure 2 ): algorithmic, business, scientific, and social. the four perspectives define the objective and use case of the interpretability method. clearly, there can be overlaps between the different perspectives, particularly with the algorithmic one. for example, using interpretability to build a better model (algorithmic perspective) might coincide with making it fairer (social perspective) or one that promotes more informed business decisions (business perspective). similarly, promoting social values through interpretability (social perspective) can build customer trust (business perspective). besides the objectives of the interpretability method, another key consideration is the stakeholders-the audience to whom the explanation is aimed and communicated. accordingly, when designing the interpretability method, we should consider not only the objective (and the usage) of the explanation but also the stakeholders, their level of expertise, and their familiarity with nlp models. by identifying different stakeholders' specific requirements and concerns, we can foster practical interpretability methods that align with their expectations @xcite . we next discuss the four perspectives and the main stakeholders (in bold): legal compliance includes cases where interpretability is explicitly the regulator's requirement, such as the gdpr's "right to explanation" (goodman and flaxman, 2017) and the algorithmic accountability act proposed in the us (maccarthy, 2019), or cases where interpretability is instrumental in ensuring the business adheres to legal standards, thereby reducing the risk of legal penalties. for instance, when nlp models are used to process credit applications @xcite @xcite , they must comply with the equal credit opportunity act (ecoa), which prohibits discrimination. finally, interpretability enhances transparency, fostering trust and goodwill. when end-users understand how decisions are made, they are more likely to trust ai systems, improving business reputation. for example, facebook's 'why am i seeing this ad' tool is designed specifically to provide more transparency and build trust @xcite . the link between interpretability and trust is welldocumented @xcite @xcite . ## properties and categorization in this section, we propose and briefly describe properties that answer the what and how questions derived from our interpretability definitions. in ## trends in model interpretability in this section, we analyze trends over the last decade in papers that propose or employ an interpretability method in the nlp field or fields outside of nlp. the analysis covers trends in interpretability method paradigms and their properties. ## conclusions and recommendations in this half-position-half-survey paper, we reviewed hundreds of works on nlp model interpretability and analysis from the past decade. unlike other surveys, we examined interpretability methods, paradigms, and properties from the stakeholders' perspective. additionally, we conducted a first-of-its-kind large-scale trend analysis by exploring the usage of interpretability methods within the nlp community and in research fields outside of it. our analysis reveals substantial diversity between research fields, particularly between nlp developers and non-developer stakeholders. to bridge these gaps and promote the adoption of nlp interpretability methods in other fields, we recommend the following steps for nlp researchers: clearly define the stakeholders and applications of your work. researchers should explicitly state in the introduction who the stakeholders of their method are, the needs it addresses, its core properties, and its potential applications within and outside the nlp community. articulating these aspects helps position the research within a broader context and ensures relevant audiences can effectively engage with the method. additionally, demonstrating applications of interpretability methods in other fields can enhance their visibility and adoption. publishing nlp research in interdisciplinary venues @xcite fosters cross-domain collaboration and broadens the impact beyond nlp. develop user-friendly code and write detailed guides for non-technical users. researchers outside the nlp community sometimes utilize specific methods due to specific needs (e.g., probing in neuroscience is used for aligning representations with brain activity). yet, many utilize methods for the wrong reason: extensive familiarity with popular methods in non-nlp domains and with well-documented code in common data science libraries (e.g., shap, lime, and scikit-learn). to encourage the adoption of nlp interpretability methods beyond our community, researchers should prioritize developing user-friendly code accompanied by detailed guides for non-technical users. additionally, the code should generate attractive and easy-to-understand visualizations. making the methods more accessible can help integrate them into other scientific and industrial domains. expand the reach of prevalent nlp interpretability paradigms. two paradigms have gained traction in nlp, particularly with the rise of llms: natural language explanations and mechanistic interpretability. we found that natural language explanation methods are also extremely prevalent in non-nlp fields. we believe this rapid adoption is concerning, as their reliability remains a topic of ongoing debate in research. our community should investigate the faithfulness of these methods @xcite @xcite and determine whether they can replace traditional, extensively researched methods. conversely, while mechanistic interpretability research is trending within the nlp community, explanations of internal model components are rarely used in other fields. our community should explore whether and how mechanistic interpretability can be adapted more broadly @xcite . we need more concept-level, self-explaining, and causal-based methods. in appendix §a.1.2, we highlight the potential of high-level concept explanations, particularly for non-expert stakeholders such as end-users, given the challenges of explaining lengthy raw textual inputs. even though they can improve the accessibility of model insights @xcite , concept-level methods remain largely underutilized, accounting for only 2% of the papers, as shown in figure 6 . stakeholders using nlp models for decisionmaking require faithful explanations @xcite . in appendix §a.2.4, we highlight the important role of causality in fostering faithfulness. yet, finally, building on the seminal calls of xai researchers @xcite , we believe in self-explaining methods as a promising path toward the "holy grail" of nlp: achieving intrinsic interpretability while minimizing performance degradation. yet, as the llm era presents new research opportunities. despite the expectation that non-developers would benefit from concept-level, self-explaining, and causal-based methods, their adoption remains limited. we believe this is mainly due to the lack of research and development within the nlp community. this gap restricts the broader applicability of nlp models, particularly in domains where transparency and interpretability are essential. the increasing capabilities of llms provide an unprecedented opportunity to develop novel concept-level, self-explaining, and causal-based interpretability methods. indeed, many of the works discussed in this paper demonstrate such potential (e.g., @xcite and @xcite ). by expanding research in these directions, the nlp community can contribute to developing models that are more reliable, explainable, and accessible to a broader range of stakeholders.