C ausal G ym: Benchmarking causal interpretability methods on linguistic tasks (2024)

Abstract: Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM behavior. To help bring these strands of research closer together, we introduce CausalGym. We adapt and expand the SyntaxGym suite of tasks to benchmark the ability of interpretability methods to causally affect model behaviour. To illustrate how CausalGym can be used, we study the pythia models (14M–6.9B) and assess the causal efficacy of a wide range of interpretability methods, including linear probing and distributed alignment search (DAS). We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena in pythia-1b: negative polarity item licensing and filler–gap dependencies. Our analysis shows that the mechanism implementing both of these tasks is learned in discrete stages, not gradually.

## introduction language models have found increasing use as tools for psycholinguistic investigation-to model word surprisal @xcite @xcite , graded grammaticality judgements @xcite , and, broadly, human language processing @xcite @xcite . to benchmark the linguistic competence of lms, computational psycholinguists have created targeted syntactic evaluation benchmarks, which feature minimally-different pairs of sentences differing in grammaticality; success is measured by whether lms assign higher probability to the grammatical sentence in each pair @xcite . despite the increasing use of lms as models of human linguistic competence and how much easier it is to experiment on them than human brains, we do not understand the mechanisms underlying model behaviour-lms remain largely uninterpretable. the linear representation hypothesis claims that 'concepts' form linear subspaces in the representations of neural models. an increasing body of experimental evidence from models trained on language and other tasks supports this idea @xcite @xcite . per this hypothesis, information about high-level linguistic alternations can be localised to linear subspaces of lm activations. methods for finding such features, and even modifying activations in feature subspaces to causally influence model behaviour, have proliferated, including probing @xcite , distributed alignment search @xcite , and difference-in-means @xcite . psycholinguistics and interpretability have complementary needs: thus far, psycholinguists have evaluated lms on extensive benchmarks but neglected understanding their internal mechanisms, whereas interpretability methods have only been evaluated on one-off datasets and sorely need more diverse benchmarks. thus, we introduce causalgym (figure 1 ). we adapt linguistic tasks from syntaxgym @xcite to benchmark interpretability methods on their ability to find linear features in lms that, when subject to intervention, causally influence linguistic behaviours. we study the pythia family of models @xcite , finding that das is the most efficacious method. however, our investigation corroborates earlier findings that das is powerful enough to make the model produce arbitrary input-output mappings @xcite . to address this, we adapt the notion of control tasks from the probing literature @xcite , finding that adjusting for performance on the arbitrary mapping task reduces the gap between das and other methods. we further investigate how lms learn two difficult linguistic behaviours during training: fillergap extraction and negative polarity item licensing. we find that the causal mechanisms require multi-step movement of information, and that they emerge in discrete stages (not gradually) early in training. ## benchmark to create causalgym, we converted the core test suites in @xcite into templates for generating large numbers of spanaligned minimal pairs, a process we describe below along with our evaluation setup. ## methods we briefly describe our choice of f * and the featurefinding methods that we benchmark in this paper. ## experiments we perform all experiments on the pythia model series @xcite , which includes 10 models ranging from 14 million to 12 billion parameters, all trained on the same data in the same order. this model series allows us to study how feature representations change with scale and training data size in a controlled manner-all models were trained on the same data in the same order, and checkpoints are provided. ## case studies in this section, we use causalgym to study how lms learn negative polarity item (npi) licensing and wh-extraction from prepositional phrases over the course of training using checkpoints of pythia-1b. we first describe the tasks. ## conclusion we introduced causalgym, a multi-task benchmark of linguistic behaviours for measuring the causal efficacy of interpretability methods. we showed the impressive performance of distributed alignment search, but also adapted a notion of control tasks to causal evaluation to enable fairer comparison of methods. finally, we studied how causal effect propagates in training on two linguistic tasks: npi licensing and filler-gap dependency tracking. in recent years, much effort has been devoted towards developing causally-grounded methods for understanding neural networks. a probe achieving high classification accuracy provides no guarantee that the model actually distinguishes those classes in downstream computations; evaluating probe directions for causal effect is an intuitive test for whether they reflect features that the model uses downstream. overall, while methods may come and go, we believe the causal evaluation paradigm will continue to be useful for the field. a major motivation for releasing causalgym is to encourage computational psycholinguists to move beyond studying the input-output behaviours of lms. our case studies in §6 are a basic example of the analysis that new methods permit. ultimately, understanding how lms learn linguistic behaviours may offer insights into fundamental properties of language (cf. @xcite . we hope that causalgym will encourage comprehensive evaluation of new interpretability methods and spur adoption of the interventional paradigm in computational psycholinguistics.