Paper: Towards Meta-Models for Automated Interpretability (ICLR-2024)

Reviewer comment (Novelty):
Though this paper focuses on an interesting topic, it is less than 9 pages and the amount of work is not enough.

Author response:
We believe research should be judged by the value of its contribution, not by quantity of work or number of pages. Could you please let us know if there are specific weaknesses in our experiments, or follow-up work that would be appropriate for us to do?

While it is true that our results do not require much page-space to present (especially as we deferred many details to the appendices for conciseness), they provide evidence that meta-models can potentially be used on a variety of problems of previous interest.

It is also true that, as other reviewers have noted, we included to little detail at times - e.g. we failed to include example RASP programs that can be reconstructed by our meta-model. We aim to address these weaknesses in our revisions.

Reviewer comment (Experiments):
Dependence on Large Amounts of Training Data: Their method relies heavily on having a significant amount of training data, which might not always be feasible for every application.

Author response:
It is true that reliance on training data is a limitation of the meta-model approach. While a meta-model may not need a large number of datapoints in an absolute sense, the training data may be expensive to obtain if each datapoint involves training a large model. Scaling to massive modern language or vision models is an important question, but out of scope for this work. Ways in which we might overcome the scaling challenge in future work include:
1. Pretraining, e.g. masked weight prediction or contrastive learning. This might allow finetuning on new tasks with fewer examples.
2. Experiments on generalization capabilities across scale. If a meta-model can (partially) generalize from small to large networks, then we may be able to reduce the number of large networks in the training data.
3. Applying meta-models to subsections of a target network. Depending on the target task, a single network might yield many datapoints, for example if individual circuits within a network can be mapped to specific functions.

We have edited the paper to discuss these points (see Section 5).

Reviewer comment (Experiments):
Model Size: As mentioned in this paper, the models they worked with are relatively small, with an average of only 3,000 parameters. This contrasts with much larger models often used in deep learning, which can have millions or even billions of parameters. It is much better to see the performance of larger models.

Author response:
This is an inaccurate representation of our experiments. Only the Tracr models had an average of 3,000 parameters. In the backdoors detection setting, models had around 70,000 parameters. In the hyperparameter-prediction setting, models had an average of 560,000 parameters, with the largest models up to 2,000,000 parameters.

Reviewer comment (Experiments):
The baseline in this paper is not enough to show the performance of their methods, adding more baselines will be better.

Author response:
It would be useful if you could clarify which baselines are missing. We compare against leading methods ([1] for hyperparameter prediction, [2,3] for backdoor detection), and achieve are state-of-the-art results. Do you have recommendations for how to proceed?

Note: we are running experiments to compare our hyperparameter prediction performance against [4] and plan to add them to the revision.

[1] https://arxiv.org/abs/2002.05688

[2] https://arxiv.org/abs/1910.03137

[3] https://arxiv.org/abs/1906.10842

[4] https://arxiv.org/abs/2110.152880

Reviewer comment (Experiments):
I hope this method can be tested on a larger model and prove to be effective.

Author response:
This is an inaccurate representation of our experiments. Only the Tracr models had an average of 3,000 parameters. In the backdoors detection setting, models had around 70,000 parameters. In the hyperparameter-prediction setting, models had an average of 560,000 parameters, with the largest models up to 2,000,000 parameters.

Reviewer comment (Theory):
**Black-box model to explain a black-box model**: I think this approach is fundamentally limited because you are now using one model you don't understand to try to explain another model that you don't understand. What happens if the training data of the meta-model has backdoors in it for example? That is, you make it so that the back-door model gives interpretations that are benign for a model that is actually problematic. This might seem far-fetched, but I think one of the key limitations of this approach is that we are left to just 'trust' the output of the meta-model. But as we know, transformer-based models can easily learn spurious signals, and other problematic behavior, so it is unclear how this approach fundamentally solves the interpretability-problem at hand. Perhaps if we could always convert trained models to their tracr equivalent then we would have more confidence in this approach. However, to me, it is a non-starter that one cannot ascertain the reliability of the results of the meta-model.

Author response:
You write:
> Black-box model to explain a black-box model: I think this approach is fundamentally limited because you are now using one model you don't understand to try to explain another model that you don't understand. 

We greatly appreciated this comment. We’ve added a section in the Section 4 more precisely discussing this limitation of “a black-box interpreting a black-box”.

Mechanistic interpretability can be viewed as consisting of two problems:
1. Proposing a (mechanistic) explanation for some behavior.
2. Validating the correctness of an explanation.

Currently, automated methods exist for validating (e.g. [1]) but not proposing explanations. 

Our paper only addresses problem 1. We agree this is a limitation - without means of verifying an explanation, meta-models can only provide limited assurance. Applying meta-models to verification is beyond the scope of this work.

Quoting more:
> What happens if the training data of the meta-model has backdoors in it for example? That is, you make it so that the back-door model gives interpretations that are benign for a model that is actually problematic. This might seem far-fetched, but I think one of the key limitations of this approach is that we are left to just 'trust' the output of the meta-model. But as we know, transformer-based models can easily learn spurious signals, and other problematic behavior, so it is unclear how this approach fundamentally solves the interpretability-problem at hand. Perhaps if we could always convert trained models to their tracr equivalent then we would have more confidence in this approach. However, to me, it is a non-starter that one cannot ascertain the reliability of the results of the meta-model.

While meta-models can certainly be backdoored, we emphasize our point above – that meta-models (at present) only propose an explanation, rather than verify it. If proposed explanations can be validated, then the performance of the meta-model need only be good on average rather than good in the worst case. While robustly validating mechanistic explanations is out of scope for our paper, some candidates include testing the base models’ output vs the reconstructed RASP program, or (in a different setting) using causal scrubbing [1].

When the meta-model’s output cannot be verified, we caution against completely trusting predictions. We hope future work will make meta-model predictions more reliable and robust.

Reviewer comment (Experiments):
**What does a mechanistic interpretation mean in this setting**: For the tracr program reconstruction, I think I get it. Here it seems like the tracr program itself constitute the explanation of the model weights. In the backdoor setting it is less clear. How does the backdoor classification task demonstrate interpretability? Specifically, how do I know how the meta-model is able to detect which model has a backdoor? Here I think you would actually want to make it input specific. Specifically, I think only a certain subsets of inputs trigger the wrong outputs for models with backdoors. It is the model's behavior on these inputs that we are most interested in. I don't quite see how the current setup helps us to do this.

Author response:
You write:
> What does a mechanistic interpretation mean in this setting: For the tracr program reconstruction, I think I get it. Here it seems like the tracr program itself constitute the explanation of the model weights. In the backdoor setting it is less clear. How does the backdoor classification task demonstrate interpretability?

Good point! We agree that in the backdoor setting, the meta-model is doing interpretability but not mechanistic interpretability. That is, it is surfacing information about the behavior of the model (the backdoor) but not how that behavior is implemented mechanistically. In general, meta-models can report any type of information to the user.

We have edited the paper to make this clear (Section 3.2).

Note: in a sense, the meta-model itself is doing mechanistic interpretability, since it examines the base model weights to locate the backdoor. However, it doesn’t surface that information in a way that is useful for humans.

> Specifically, how do I know how the meta-model is able to detect which model has a backdoor? Here I think you would actually want to make it input specific. Specifically, I think only a certain subsets of inputs trigger the wrong outputs for models with backdoors. It is the model's behavior on these inputs that we are most interested in. I don't quite see how the current setup helps us to do this.

The meta-model determines which base models are poisoned by their weights alone, with extremely high reliability in the domains we examined. We suspect it does so by examining the weights and identifying circuits that strongly alter the classification decision based solely on simple features – but we have no evidence at present for that claim. We agree that using a meta-model in this way to determine how backdoor circuits are implemented in base models would be worthwhile.

There are already an abundance of methods (https://arxiv.org/pdf/1811.03728.pdf, https://arxiv.org/pdf/1811.00636.pdf, https://proceedings.neurips.cc/paper_files/paper/2022/file/76917808731dae9e6d62c2a7a6afb542-Paper-Conference.pdf) that examine model activations to determine whether an example is poisoned or not.

Reviewer comment (Experiments):
Can you show examples in the appendix of settings where you compile a program with tracr, and then reverse-engineer it with the meta-model. It would be helpful to compare the output of the meta-model and the original tracr program.

Author response:
> Can you show examples in the appendix of settings where you compile a program with tracr, and then reverse-engineer it with the meta-model. It would be helpful to compare the output of the meta-model and the original tracr program.

Thank you for the suggestion. We will add a section to the Appendix with examples summarizing these results once we finish updating the Tracr experiments.

Reviewer comment (Experiments):
How do you envision that this meta-modeling approach will actually be used in practice, on real models, in the future? I struggle to see how one could ever train a meta model for a realistic setting, e.g., a vision transformer trained on ImageNet. Would I need to train thousands of imagenet models first to fit the metamodel?

Author response:
> How do you envision that this meta-modeling approach will actually be used in practice, on real models, in the future? I struggle to see how one could ever train a meta model for a realistic setting, e.g., a vision transformer trained on ImageNet. Would I need to train thousands of imagenet models first to fit the metamodel?

This is an important question. How to scale meta-models is an open research question, but it seems doable to us. Ways in which we might overcome the scaling challenge in future work include:
1. Pretraining, e.g. masked weight prediction or contrastive learning. This might allow finetuning on new tasks with fewer examples.
2. Experiments on generalization capabilities across scale. If a meta-model can (partially) generalize from small to large networks, then we may be able to reduce the number of large networks in the training data.
3. Applying meta-models to subsections of a target network. Depending on the target task, a single network might yield many datapoints, for example if individual circuits within a network can be mapped to specific functions. This also means the meta-model could be smaller than the base model it helps to analyze.

We’ve added a section called “Future Work”, covering some ways in which we expect that meta-models might be useful. Some speculative examples include: highlighting backdoored neurons, producing simplified interpretations of neural network circuits, and identifying attention heads that perform a particular function. 

[1] https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN