Paper: Metanetwork: A novel approach to interpreting ANNs (ICLR-2024)

Reviewer comment (Novelty):
I could not understand the novelty of this work compared to existing works, such as TASK2VEC or MODEL2VEC mentioned in the paper, which also try to embed tasks or models.

Author response:
The novelty of our work is that our approach does not need access to the dataset while TASK2VEC or MODEL2VEC uses the dataset to characterize the model.

Reviewer comment (Evaluation):
Even if modality or dataset-id could be predicted by the proposed autoencoder, I'm not sure what we can say about interpreting ANNs.

Author response:
We added a conceptual figure to Introduction, which illustrates the comparison between mechanistic interpretability approaches and our approach. What we really like to propose in our work is a new approach to interpreting NNs by embedding them in a lower-dimensional space and quantitatively characterizing them in relation to other NNs. In order to make this happen, the lower-dimensional space of the models has to reflect the functional similarities of models, and we demonstrate this by successfully predicting certain properties of the task that an unseen model is trained on.

Reviewer comment (Experiments):
In Section 4.4, it seems to me that there is nothing to train with a k-NN classifier when the meta-encoder is frozen. What do you mean by training here?

Author response:
Yes, you’re right that there is no parameter to update when the meta-encoder is frozen. The choice of the word “train” is indeed confusing so we changed it to “fit” in the modified version.

Reviewer comment (Novelty):
(This is not a question, just a comment.) There are several works that deal directly with DNN weights as input, such as [1], which could be used to analyze trained DNN models more efficiently. [1] Navon et al., Equivariant architectures for learning in deep weight spaces, ICML'23.

Author response:
We were not aware of this work though it seems to give us some good insights on how better to embed weight matrices. Thank you so much for sharing.

Reviewer comment (Presentation):
The methodology section is hard to follow and very confusing. It would be helpful to have a Figure in the beginning (introduction already), to explain the intuition behind the idea and to clearly explain the role of each component (model to interpret, meta-encoder, meta-autoencoder, Knn classifier, ...).

Author response:
We added conceptual figures to Introduction, which illustrate the comparison between mechanistic interpretability approaches and our approach. In Fig. 2, “unseen model” is the model to interpret, “meta-encoder” is what maps “unseen model” to the embedding space, and KNN classifier predicts the ability of “unseen model” by relating it to “known models”.

Reviewer comment (Experiments):
I was surprised that models are only trained for 1 epoch. Some explanation would be helpful, as this is uncommon.

Author response:
We train the models for 1 epoch because the dataset is big for the task (1 million 16x16 patches). 1 epoch turns out to be sufficient for the model to generalize well to the dataset. We believe that this is a similar situation to LLM pertaining.

Reviewer comment (Experiments):
Poor experiment section: There is no comparison with other approaches. The relation of the tasks (predicting modality, predicting dataset) to interpretability is unclear.

Author response:
We think we don’t need to compare our approach to others in our result section because the metanetwork is tested in a new problem setting where we have to infer the ability of a model without access to the dataset.

Modality and dataset ID are both properties of the task. We think that successfully predicting these is one way of interpreting NNs, especially when the original task is not known.

Reviewer comment (Experiments):
Are the original models (to be interpreted) trained on the full datasets? The dataset splitting by class label (see sec 4.2) is only done to train the meta network, right?

Author response:
When we predict the modality of the task, the models to be interpreted are trained on the full datasets. But when we predict the dataset-id of the task, the models to be interpreted are trained on the datasets that are used to train the metanetwork. Please note that we trained the metanetwork without using the task information, and we only used it in the test phase.

Reviewer comment (Evaluation):
The paper claims that encoding the model's weights improves interpretability, but it doesn't explain how it achieves this. It doesn't clarify how the model makes decisions or whether it reveals any general biases the model has learned. The only thing I can see from here is how close two models are.

Author response:
We added a figure in our modified version, which illustrates how the metanetwork infers a property of the task an unseen model can solve. As for general biases models have learned, we showed that the model embeddings are separated according to their task property, but we have to admit that we are yet to pinpoint what exactly it is that makes them separate in that way, and figuring this out should be included in our future work.

Reviewer comment (Theory):
Although the authors claim that the proposed approach can be used with any model, it appears that this may not be the case. It's unclear which layer to encode, and if we attempt to encode multiple layers, we would need different encoders due to varying input sizes. Even with the option of ReLU in section 5.3, it may be impossible for large networks. Representing an entire large network with a single-layer network doesn't seem feasible.

Author response:
You are right on the point that it is quite challenging to apply our proposed methodology (unit-wise meta-encoder and inter-unit meta-encoder and so on) to any architecture of NN models. Although we didn’t empirically show this in this paper, we think we can get to embed a far wider range of NNs by first converting them to the same format of three-layer NNs as we discussed in section 5.3. But as you mentioned in the feedback, it might be difficult to be applied to “any” architecture (that uses other activation functions or has recurrent structure for example). We might have to come up with a way to solve this issue in future work.

Reviewer comment (Writing):
Can the authors provide more detailed clarification for this part: "Therefore, if we infer the training dataset of the model using the feature visualization technique, without access to the dataset, we can obtain a low-dimensional vector representation of the model, using an approach similar to MeLA."

Author response:
What we mean by this sentence is that it has been shown that some feature visualization techniques (As discussed in [Olah et al. 2017](https://distill.pub/2017/feature-visualization/)) can elicit what kind of feature is captured in a particular subnetwork of a given model, without accessing the original dataset. Also, there are some work on generating a model given a dataset ([Wu et al. 2018](https://arxiv.org/abs/1807.09912); [Ha et al. 2016](https://arxiv.org/abs/1609.09106) and so on). If we combine these two together, we should be able to reconstruct the original model without using the original dataset and gain a lower dimensional representation of the model as a by-product.

Reviewer comment (Experiments):
Why is it necessary to use two different meta encoders? Can we use only one of them, such as either the "unit-wise" meta-autoencoder or the "inter-unit" meta-autoencoder only?

Author response:
We need these two types of meta-autoencoders to facilitate ability-related encoding. 
Firstly, in our experiment setting the order of the output units does not have any functional meaning because they are in an intermediate layer in the original autoencoder. Therefore, we can take a single fully connected layer as a set of filters. Although there are some studies on the autoencoding of sets, there is no standard method. In this research, we considered a set as an array by sorting it. For example, if the original weight matrix is sorted by some key, it is possible to autoencode the entire weight end to end. However, the weight space has a very large number of dimensions, making it difficult to find an appropriate key. Therefore, we first autoencode the elements of the weight matrix, embed them in a lower dimension for feature extraction, and then sorted them using these features to create a sorting key. Thus, we first trained a unit-wise meta-encoder (for filters), sorted the obtained representations along the primary dimension, concatenated them, and then used an inter-unit meta-encoder to obtain a low-dimensional representation of the set of filter representations.