Mechanistic? (2024)

Abstract: The rise of the term “mechanistic interpretability” has accompanied increasing interest in understanding neural models—particularly language models. However, this jargon has also led to a fair amount of confusion. So, what does it mean to be mechanistic? We describe four uses of the term in interpretability research. The most narrow technical definition requires a claim of causality, while a broader technical definition allows for any exploration of a model’s internals. However, the term also has a narrow cultural definition describing a cultural movement. To understand this semantic drift, we present a history of the NLP interpretability community and the formation of the separate, parallel mechanistic interpretability community. Finally, we discuss the broad cultural definition—encompassing the entire field of interpretability—and why the traditional NLP interpretability community has come to embrace it. We argue that the polysemy of “mechanistic” is the product of a critical divide within the interpretability community.

## introduction the field of mechanistic interpretability is growing dramatically, constantly motivating new workshops, forums, and guides. and yet, many are unsure what the term mechanistic interpretability entails. researchers, whether experienced or new to the field, often ask what makes some interpretability research "mechanistic" @xcite @xcite . because both fields study language models (lms), the distinction between traditional nlp interpretability (nlpi) and mechanistic interpretability (mi) is unclear. in fact, when work is labelled as mechanistic interpretability research, the label may refer to: ## conclusion whatever terminological confusion and ideological tension they have brought to the interpretability field, the mi community is also responsible for its newfound popularity. the interest, energy, and opportunities mi brings to the field cannot be understated, nor should they be taken for granted. nlpi and mi researchers alike are motivated by social responsibility, intellectual curiosity, and the possibility of improving our tools. however, many mi researchers are also members of the alignment community concerned about catastrophic ai risk, where the value of mi is questioned @xcite @xcite . there may come a time when alignment community consensus turns away from mi. though many current mi researchers may leave-and some generous resources could disappear-others are likely to continue pursuing our shared objectives. our communities have too much in common: scientific curiosity and a belief that we should understand the tools we use. we will all continue striving for that objective as long as there are opaque models to understand. why not, therefore, also aim to connect? yonatan belinkov. 2018. on internal language representations in deep learning: an analysis of machine translation and speech recognition. ph.d. thesis, massachusetts institute of technology. yonatan belinkov. 2022. probing classifiers: promises, shortcomings, and advances. computational linguistics, 48(1):207-219. yonatan belinkov. 2023a. "excited to see important work from @andyzou_jiaming, @danhendrycks..., on interpreting & controlling language models at representation level, to improve fairness & safety of lms. unfortunately it fails to engage with a large body of work on these topics from the past 5 years.". tweet. accessed: 2024-08-28. yonatan belinkov. 2023b. "we are interested! #black-boxnlp has been the largest #nlproc workshop for several years now. and we have an interpretability track in all main #nlproc confs! please submit your work to be reviewed in such venues...even if you disagree with other approaches to interpretability, i think engagement through common conferences would help the community grow.". tweet. accessed: 2024-08-16. yonatan belinkov, nadir durrani, fahim dalvi, hassan sajjad, and james glass. 2017a. what do neural machine translation models learn about morphology? in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers). association for computational linguistics. yonatan belinkov, sophie hao, jaap jumelet, najoung kim, arya mccarthy, and hosein mohebbi. 2023. panel discussion on "mechanistic interpretability" at the 6th blackboxnlp workshop at emnlp 2023. video recording from 8:05. yonatan belinkov, lluís màrquez, hassan sajjad, nadir durrani, fahim dalvi, and james glass. 2017b. evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pages 1-10, taipei, taiwan. asian federation of natural language processing. nora belrose. 2024. "the @aieleuther interpretability team is releasing a set of top-k sparse autoencoders for every layer of llama 3 8b: https://huggingface.co/eleutherai/ sae-llama-3-8b-32x . we are working on an automated pipeline to explain the sae features, and will start training saes for the 70b model shortly.". tweet. accessed: 2024-08-27. nathan beniach. 2024. "is mechanic [sic] interpretability a sexier way of saying interpretability?". tweet. accessed: 2024-08-10. leonard bereska and efstratios gavves. 2024. mechanistic interpretability for ai safety -a review. transactions on machine learning research. adrien bibal, rémi cardon, david alfter, rodrigo wilkens, xiaoou wang, thomas françois, and patrick watrin. 2022. is attention explanation? an introduction to the debate. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3889-3900, dublin, ireland. association for computational linguistics. tolga bolukbasi, adam pearce, ann yuan, andy coenen, emily reif, fernanda viégas, and martin wattenberg. 2021. an interpretability illusion for bert. arxiv: 2104.07143. dan braun, jordan taylor, nicholas goldowsky-dill, and lee sharkey. 2024. identifying functionally important features with end-to-end sparse dictionary learning. in icml 2024 workshop on mechanistic interpretability. neel nanda. 2024a. "a bunch of people who don't seem to identify as ea at all use the term to describe their work nowadays. i'm happy the field is growing outside of the ea bubble!". tweet. accessed: 2024-08-19. neel nanda. 2024b. an extremely opinionated annotated list of my favourite mechanistic interpretability papers v2. blogpost. accessed: 2024-08-19. neel nanda, lawrence chan, tom lieberum, jess smith, and jacob steinhardt. 2023. progress measures for grokking via mechanistic interpretability. in the eleventh international conference on learning representations. vlad niculae, andre martins, mathieu blondel, and claire cardie. 2018. sparsemap: differentiable sparse structured inference. in proceedings of the 35th international conference on machine learning, pages 3799-3808. pmlr. issn: 2640-3498. nostalgebraist. 2020. interpreting gpt: the logit lens. lesswrong blogpost. chris olah. 2015. neural networks, types, and functional programming -colah's blog. blogpost. chris olah. 2024a. "i introduced the term to distinguish the work i was doing on circuits from a lot of other work that was going on circa 2018, notably saliency maps...i was motivated by many of my colleagues at google brain being deeply skeptical of things like saliency maps. when i started the ope-nai interpretability team, i used it to distinguish our goal: understand how the weights of a neural network map to algorithms...since then, i think it's become an umbrella term for a variety of other things.". tweet. accessed: 2024-08-10. chris olah. 2024b. "the motivating moment for me was that i went on a walk with a senior colleague shortly before i left google brain, and they matter of factly told me that "all interpretability is bullshit" because they'd been so turned off by saliency maps...i wanted a way to get people who were skeptical in this way to realize that i was talking about something pretty different than saliency maps and be willing to give it a second look.". tweet. accessed: 2024-08-10.