From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP (2024)

Abstract: Interpretability and analysis (IA) research is a growing subfield within NLP with the goal of developing a deeper understanding of the behavior or inner workings of NLP systems and methods. Despite growing interest in the subfield, a criticism of this work is that it lacks actionable insights and therefore has little impact on NLP. In this paper, we seek to quantify the impact of IA research on the broader field of NLP. We approach this with a mixed-methods analysis of: (1) a citation graph of 185K+ papers built from all papers published at ACL and EMNLP conferences from 2018 to 2023, and their references and citations, and (2) a survey of 138 members of the NLP community. Our quantitative results show that IA work is well-cited outside of IA, and central in the NLP citation graph. Through qualitative analysis of survey responses and manual annotation of 556 papers, we find that NLP researchers build on findings from IA work and perceive it as important for progress in NLP, multiple subfields, and rely on its findings and terminology for their own work. Many novel methods are proposed based on IA findings and highly influenced by them, but highly influential non-IA work cites IA findings without being driven by them. We end by summarizing what is missing in IA work today and provide a call to action, to pave the way for a more impactful future of IA research.

## introduction the rapid progress made in the development of large language models (llms, @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ) has had a profound impact on the field of natural language processing (nlp) @xcite . while these models demonstrate unprecedented performance figure 1 : interpretability and analysis (ia) is an increasingly popular subfield of nlp: (top) number of ia papers in acl/emnlp in comparison to other tracks that have existed since 2020. the number of ia papers has grown considerably, from 90 papers in 2020 to 160 papers in 2023 (a growth rate of 77.8%). this is the highest growth rate among these tracks. (bottom) citations to ia papers compared to other highly cited tracks. and novel capabilities @xcite , and are rapidly finding their way into real-world applications @xcite @xcite , they are also largely treated as black boxes, which does not satisfy other expectations for successful machine learning deployment, such as trust, accountability, and explainability @xcite . in nlp research, these factors have motivated a large body of work on interpretability and analysis (ia), which aims to understand the inner workings of llms and explain their predictions @xcite @xcite . many researchers in this area believe that better understanding llms is imperative to improve their efficiency, robustness, and trustworthiness, towards successful, safe deployment. ia research has thus witnessed rapid growth in recent years and is now one of the biggest research areas (in terms of number of publications and citations) at major nlp conferences (see figure 1 ). despite the rapid growth of ia research (see also figure 9 ), a criticism of this work is that it often lacks actionable insights, especially for how to improve models, and therefore has little impact on how new nlp models are designed and built @xcite . this criticism raises questions about whether its current form is the right path towards progress in nlp. in this work, we tackle these questions with a systematic, mixed-methods study of the impact of ia research on nlp in the past and the present, and use our findings to inform a vision for the future of ia. more specifically, we ask: how does interpretability and analysis research influence nlp researchers in what they choose to work on, what they cite, and how they think about nlp altogether? we perform a bibliometric analysis of 185,384 publications based on the two major nlp conferences, acl and emnlp, between 2018 and 2023, and solicit opinions from 138 members of the nlp community via a survey. in addition to quantitative results, we perform qualitative analysis of survey responses and 556 papers. this approach gives us a holistic view of the impact of ia research on nlp. our analysis reveals that (1) nlp researchers build on findings from ia work in their research, regardless of whether they work on ia themselves or not ( §4), (2) nlp researchers and practitioners perceive ia work to be important for progress in nlp, multiple subfields, and their own work, for various reasons ( §5), and (3) many novel non-ia methods are proposed based on ia findings and highly influenced by them, for various areas, even though highly influential non-ia work is not driven by ia findings despite citing them ( §6). while our findings show that ia work presents insightful observations, there are still opportunities for greater impact on the rest of nlp. thus, based on survey responses, we identify the key ingredients that are missing in ia research todayunification; actionable recommendations; humancentered, interdisciplinary work; and standardized, robust methods -and close with a call to action and recommendations ( §7). we hope our work paves the way towards a more impactful future for ia research as the field continues to grow. ## methodology we start by discussing what we consider as ia research and our approach for measuring impact. ## citation graph and community survey here, we describe the construction of our citation graph for bibliometric analysis, and the design of our survey of the community. ## researchers build on findings from ia research in their work we begin by analyzing whether researchers use contributions of ia research in their work. we approach this by analyzing citational use, as well as survey-reported use beyond citations. ia papers are cited more often than other tracks when comparing papers from different tracks, global counts of citations can be misleading, as a small number of papers can account for most of the citations in a field @xcite . to account for this, we compare citations based on the citation success index (csi; @xcite metric. given two groups of papers a and b, the csi score computes the probability that a random paper from a is more cited than a random paper of b. this score is not subject to biases from the skewness of the citation distribution, and it is clearly interpretable; e.g., if we draw random ia and machine translation papers from emnlp or acl in 2023, there is a 57.1% chance that the ia paper is more cited than the machine translation paper. figure 3 shows that csi scores for the ia track are often favorable (@xmath0. in 2023, only the ethics and the large language models tracks had favorable csi scores when compared to ia. this shows that ia papers have higher citational impact than other tracks. ia papers are well cited outside of ia while high csi scores tell us that ia papers are cited well, they do not tell us where these citations are coming from, i.e., are ia papers mostly cited by other ia papers or by papers outside of ia? to evaluate the impact of ia work outside of ia, we compare citations within the same track, which we call intra-track citations, to extra-track citations, i.e., citations from outside the track. figure 4 shows that most citations to ia papers are predicted to be extra-track citations. the proportion of references to ia papers differs considerably by citing track, with papers about efficient methods, machine learning, and large language models citing ia research more frequently than others (see figure 11 ). while the ia track does not stand out in terms of its extra-track citations compared to other tracks (see figure 12 ), these results still demonstrate that the citational impact of ia research extends well beyond the ia track itself. ia papers are central in nlp next, we assess whether ia papers are impacting nlp as a whole rather than just specific tracks. we quantify this with the betweenness centrality (bc) metric, a measure of interdisciplinarity @xcite @xcite . bc quantifies the extent to which a node in the graph acts as a bridge along the shortest path between two other nodes @xcite ; nodes with higher bc are considered more important as more information passes through them. 5 therefore, we interpret papers with a high bc as important papers that are essential for the connectivity of the citation network. we compute the bc for every paper in emnlp and acl since the ia track started (2020), and find that the median bc of ia papers is higher than most other tracks, at 1.23 × 10 -7 . notably, ia ranks as the second most central track overall, following the large language models track, which has a median bc of 1.95 × 10 -7 . these results (shown in figure 10 ) provide further evidence that ia work plays a central role in the acl/emnlp citation network. ia influences the work of nlp researchers for a complementary view of impact beyond citations, we survey nlp community members on how often they use concepts from ia in their day-to-day work, and more broadly, how ia influences their work. as figure 5 shows, the median rating for use of ia concepts by respondents who work on ia is often, while even the median respondent who doesn't work on ia uses concepts from ia sometimes. in both groups of respondents, there are people who always use ia concepts in their day-to-day work. beyond this, ia work influences respondents in different ways: it provides respondents with research ideas (91% of respondents who work on ia; 60% of respondents who don't), changes mental models of model capabilities and limitations (77%; 65%), and helps ground explanations of respondents' results (64%; 59%). notably, only 9 (6.5%) respondents state that ia does not affect their work. these results complement our citation-based findings by providing further evidence that ia work impacts both ia and non-ia researchers and their research. 5 we provide further discussion of bc in appendix a.1. ## a closer look at influential papers so far we have discussed findings about ia as a whole, either by considering the role of ia papers in the acl/emnlp citation graph or the perception of ia work within the community. in this section, we zoom in on specific influential papers sourced from both our survey and citation graph. we seek to answer: what are these papers about? what kind of work are they impacting, and how? to this end, we inductively obtain the themes of a total of 585 papers, through qualitative coding of their titles and abstracts by two authors @xcite . the 585 papers include: (1) all papers mentioned more than once as having influenced survey respondents' work (@xmath1; (2) highly-cited ia papers from our citation graph (@xmath2; (3) highly-cited non-ia papers from our citation graph (@xmath3; (4) non-ia papers that cite and are highly influenced by the top-10 most-cited ia papers (@xmath4. the resulting themes are mostly descriptive, including topics (e.g., in-context learning, training dynamics) and contribution types (e.g., novel method, analysis). percentage agreement on our coded themes is above 90% for each subset of papers. see appendix c for more details. our analysis reveals that beyond background citations, ia work influences the development of many novel models and metrics outside of ia work, and affects work in domains such as question answering (qa), reasoning, and bias. what are influential ia papers about? of the papers that survey respondents submitted as examples of work that has directly influenced their own work, representation analysis appears in over a third of the papers, novel methods for interpretability (e.g., causality, interventions, steering, neuron/activation analysis, etc.) are proposed in nearly a quarter of them, and probing also appears in 24% of these papers. in contrast, the top-50 most cited ia papers are more often about the analysis component of ia (40%). novel methods (for analysis, evaluation, linguistics, probing) are proposed in 26% of papers, and evaluation is a main contribution of 32%. as expected, the most cited non-ia papers in our citation graph mostly consist of highly influential datasets, models, and methods, e.g., hotpotqa, @xcite @xcite . more top themes are shown with the percentage of papers in we also find evidence that many ia papers create novel metaphors to understand models -e.g., seeing feed-forward layers as key-value memories @xcite , or reading from and writing to the "residual stream" @xcite , and many analysis papers highlight the limits of models. as survey respondents cited these very reasons for why they perceive ia work as important, these themes corroborate why these papers would be particularly influential. in addition, many of the qualities that survey respondents feel are currently lacking in ia research (see §7) appear in these papers, such as moving beyond toy models @xcite , and providing actionable methods @xcite . why are influential ia papers cited? as citations can have a variety of reasons @xcite , we examine three types of citational intentbackground, methods and results citations (see figure 13 in appendix d). overall, we find that influential ia papers are cited most often as background citations, then as methods citations, and least frequently when comparing results. in comparison, highly cited papers that are not about ia tend to be cited most frequently for methods. this is expected, as many of these papers are about popular datasets and models, as described above. what are the citing papers about? despite the large number of background citations, however, there is plenty of work-including non-ia workthat is highly influenced (according to semantic scholar) by ia research. for a closer look at what these citing papers do, we analyze all 456 papers with highly influential citations to one of the top 10 most-cited ia papers, and annotate their themes based on titles and abstracts. unsurprisingly, many of the papers have themes in common with what they cite, e.g., papers that analyze multilingual models are frequently cited by papers on cross-lingual transfer. we thus focus on the difference in themes between citing papers and cited papers, and find that over 33% of non-ia papers that are highly influenced by ia work propose novel methods, e.g., many novel icl methods cite analysis work on demonstrations @xcite and similarly, many novel methods for bias mitigation cite datasets for stereotype evaluation such as @xcite and @xcite . these provide concrete counterexamples to the claim that ia work does not influence modeling improvements. is ia work impacting highly cited non-ia work? looking at the highly-cited non-ia papers, we find that these too tend to cite ia work frequently. 22 out of the top 50 most cited non-ia papers are even highly influenced by some ia work, but 28 are not highly influenced by any ia work. these results show that while highly influential non-ia work does acknowledge ia findings, it is likely not driven by them. ## main takeaways and discussion we end by discussing our main findings and recommendations on how to move ia research forward. ## related work the increasing number of ia publications during the last few years has resulted in several survey or position papers that critically discuss existing work, identify common patterns, and provide suggestions for how to go forward. @xcite critically questions common motivations behind interpretability and the lack of definitions in the field. following their recommendation, we provide a definition of what we consider interpretability and analysis research in §2. @xcite summarize trends in early ia work and discuss recommendations for how to overcome the limitations of ia research. similar to our work, they recommend that future work should think about better ways to evaluate ia research and findings. @xcite survey and synthesize ia work on bertology, a subfield of ia work that focuses on encoder-only language models. @xcite survey a large number of papers that study the internals of language models (transparency), and discuss key challenges in the field. like us, they also argue for better ways of evaluating ia methods, as well as more actionability and grounding in real-world applications. more recently, @xcite discuss two prominent trends in interpretability research (posthoc explanations and intrinsic interpretability) and argue that interpretability ("the study of explaining models in understandable terms to humans") needs a new paradigm centered around faithfulness. several other works study citational patterns and trends within the broader nlp community. mohammad (2020) uses citations to measure the impact of nlp publications indexed by the acl anthology. like us, they compare how well papers from different areas within nlp are cited, and use citation statistics to draw conclusions about the impact of different subfields within nlp. @xcite use citations as an indicator for how widely the community is reading. they demonstrate a recency bias in citation behavior with a study of temporal citation trends, i.e., a majority of cited papers fall within a five year time period before publication of the citing work. @xcite analyze the influence between nlp and other fields over the years. also using semantic scholar, they rely on citations to conclude that nlp has become more insular over time. similarly, @xcite find low levels of extra-disciplinary citation when analyzing how nlp and ml researchers dis-cuss democracy. more specific to ia, @xcite uses semantic scholar to curate a large number of papers focusing on explainability, studying citation trends in the field based on this collection. another set of related papers surveys the nlp community for their perceptions and opinions, a method we also use. @xcite , for example, focus on paradigm shifts and study factors that shape nlp as a field. they conduct interviews with nlp researchers and experts and gather their opinions on critical trends and patterns that emerge in the field. @xcite also focus on paradigm shifts and impact, but from a diachronic perspective. they provide a novel framework to study the evolution of research topics within a field to establish what drives research in nlp across time, and they find that tasks and methods have a bigger impact on the field than metrics do. lastly, there are several papers in the scientometrics literature that study and compare the impact of research using the same metrics as we do: chacon et al. (2020) apply the citation success index to compare sub-fields in physics, and @xcite propose the use of betweenness centrality as a measure of the interdisciplinarity of journals. ## conclusion we contribute a mixed-methods analysis of the impact of interpretability and analysis research on nlp. by analyzing a citation graph of 185k+ papers built from all papers published at acl and emnlp from 2018 to 2023, surveying 138 respondents from the nlp community, and manually annotating 556 papers, we found that ia work is wellcited in other subfields of nlp, central to the nlp citation graph, and highly influential to many novel methods. nlp researchers and practitioners perceive ia work as important for progress in nlp, multiple subfields (especially reasoning and fairness), and for their own work. in sum, even though highly influential models, methods and datasets are not driven by ia findings, ia work still has a great impact on nlp in the past and the present. we conclude with a call to action based on what is missing in the subfield, to pave the way for ia work to be even more impactful in the future.