Paper: NPEFF: Non-Negative Per-Example Fisher Factorization (ICLR-2024)

Reviewer comment (Presentation):
While the advantages of the method are well explained, it would be good to add a separate limitations section for transparency. Could you elaborate in your comments on what you see as the biggest limitations.

Author response:
We've added a limitations/future work paragraph to the end of the conclusion. The two main limitations
we see in our current implementation of NPEFF are supporting models with a large output space (e.g.
having many classes or text generation) and allowing for control over component tunings.

For the former, we can sample from the output distribution to create an approximation of the PEFs.
This would require things like determining the number of samples needed per example. Alternatively if
canonical input/output pairs are provided, we can use something like the empirical Fisher (as per Kunstner et al. 2019)
instead.

Although the unsupervised nature of component recovery using NPEFF can be considered one of its advantages, it
can also be a limitation when trying to use NPEFF in some applications. For example, being able to create a
component tuned for a particular user-defined concept would be useful when using something like our
perturbation procedure to make user-guided changes to a model. One potential method for this is to create a component whose
coefficients would be forced to be non-zero for a selected set of examples and zero for all others.

Reviewer comment (Writing):
I have found the toy model difficult to understand. I think it would be good to provide an intuitive explanation in the main text on the programmatic search.

Author response:
There was general confusion amongst the reviewers about this. We have updated Section 3.2 to more clearly describe
what the "concepts" were (values of Boolean intermediate variables in the RASP program) and the tunings of the
recovered components.

Reviewer comment (Novelty):
Could your example about fixing flawed heuristics have applications to the unlearning literature?

Author response:
Possibly. One limitation of NPEFF, however, is the inability to directly create a component tuned for a
human-defined concept. This could make it challenging to obtain a component tuned for something that you
want to unlearn. As was done in the flawed heuristics experiments, the components set expansion procedure
could help in creating components tuned to user-defined concepts/information, but there would be no guarantees
in actually recovering such a component. However, it is possible that modifications could be made to the
NPEFF decomposition to procedure to better support this. For example, one could create a component whose
coefficients would be forced to be non-zero for a selected set of examples and zero for all others.

Reviewer comment (Theory):
As pointed out by Kunstner et al. 2019, the fisher information matrix is an overloaded object which may or may not refer to the empirical fisher. Since you switch from the true fisher to an approximation which Kunstner et al. 2019, please check section 3.2 of their paper that you followed the correct terminology.

Author response:
For PEFs (per-example Fishers), there is no expectation over the model inputs $x$, so there is no conceptual difference
between a "statistics Fisher" and "statistics empirical Fisher". As for the role of
the expectation over classes "y", we make no use of any label provided with the example and perform
an expectation over the set of classes.

Reviewer comment (Novelty):
Interpretability has been studied for a long time, so the novelty is rather low.

Author response:
While interpretability has been studied for a long time, we argue that it is a broad enough field
that there still exists room for significant novelty. In particular, unsupervised concept discovery
methods like NPEFF are highly novel. We were only able to find two existing unsupervised concept discovery
methods: Ghorbani et al. (2019) and Yeh et al. (2020). These both operate by mapping activations to concepts.
For models without a fixed dimension activation space, these methods need some way to get fixed-dimensional (set of) activations
for example. Neither previous work experimented on Transformers, but you could use token-level activations at a particular layer.
Since these are not example-level activations, they can leave out important information that is spread out across multiple sequence
positions. In constrast, NPEFF provides a full example-level decomposition of the model's processing.

In our baselines, we compared NPEFF to Ghorbani et al. (2019) by using activations that come right before the classification
head. Although this allowed us to get an example level decomposition for the baseline, the activations in the transformer body
cannot be analyzed this way. Yeh et al. (2020) experimented with performing a sequence/image-position level decomposition of
activations. However, they only experimented with convolutional models. Some of their methods rely on the properties of the
receptive fields of intermediate activations in convolutional models, so their method would be harder to adapt to transformers.

Reviewer comment (Experiments):
The results are only presented for small Bert/Resnet models. It would be better with results for LLMs.

Author response:
Our current method of computing PEFs uses an expectation over the set of possible model outputs. For
classification models with few classes, we compute this expectation exactly, and we ignore classes
assigned low probabilities for classification models with many classes. For autoregressive language models, however,
we would need to approximate the PEF by doing something like taking several samples from the model's output.
If model inputs are paired with ground-truth outputs, then we could use something like empirical Fisher instead of the
true Fisher. We leave exploration of this to future work.

In terms of applying NPEFF to large models in general (e.g. xlm-roberta-xl and xlm-roberta-xxl), NPEFF doesn't have
any methodological issues. We can include experiments at that scale, but we do not have the time to run them during
the rebuttal period.

Operating on sparse representations of PEF matrices allows NPEFF to better scale to large models.
It is also possible to run NPEFF on PEFs computed using only a subset of the model variables. For example, one might
run NPEFF using only the parameters from a single transformer layer. While we do not explore that in this work,
we have explored this a bit and found the components to have interpretable tunings.

Reviewer comment (Experiments):
It is hard to know if the proposed method works well. Interpretability is very subjective, and a few examples (which could be cherry picked) are not very convincing. The authors only compare against a single baseline.

Author response:
Succinctly evaluating interpretability methods is tricky due to the generally subjective nature of interpretability.
We've added to the supplemental material a pdf (mnli_to_snli.pdf) of all of the component top examples for the NLI experiment from Section 3.1.1. While not ideal, these can be used to verify that most components have an interpretable tuning and that the examples included in the paper were not cherry-picked.

Reviewer comment (Experiments):
Could you compare against more baselines?

Author response:
In our baselines, we compared NPEFF to Ghorbani et al. (2019) by using activations that come right before the classification
head. Although this allowed us to get an example level decomposition for the baseline, the activations in the transformer body
cannot be analyzed this way. Yeh et al. (2020) experimented with performing a sequence/image-position level decomposition of
activations. However, they only experimented with convolutional models. Some of their methods rely on the properties of the
receptive fields of intermediate activations in convolutional models, so their method would be harder to adapt to transformers.

Reviewer comment (Evaluation):
Could you do some kind of human evaluation? E.g. give a sample of 10 people two interpretability models and ask which one they prefer.

Author response:
While human evaluation might be useful for verifying that components have interpretable tunings, we think that using it to compare
NPEFF to another interpretability model would have many pitfalls. The goal of an unsupervised concept discovery method like NPEFF is to uncover components that accurately reflect the processing done by a model. An interpretability method that produces
overly simple components might win out in a human evaluation over a model that more accurately reflects the processing done
by the model. Such comparisons could be of value if the goal of an interpetability method was to get the model to "show
its work" to an end user, but that was not a goal of NPEFF.

Reviewer comment (Experiments):
Could you give results on LLMs?

Author response:
Our current method of computing PEFs uses an expectation over the set of possible model outputs. For
classification models with few classes, we compute this expectation exactly, and we ignore classes
assigned low probabilities for classification models with many classes. For autoregressive language models, however,
we would need to approximate the PEF by doing something like taking several samples from the model's output.
If model inputs are paired with ground-truth outputs, then we could use something like empirical Fisher instead of the
true Fisher. We leave exploration of this to future work.

In terms of applying NPEFF to large models in general (e.g. xlm-roberta-xl and xlm-roberta-xxl), NPEFF doesn't have
any methodological issues. We can include experiments at that scale, but we do not have the time to run them during
the rebuttal period.

Operating on sparse representations of PEF matrices allows NPEFF to better scale to large models.
It is also possible to run NPEFF on PEFs computed using only a subset of the model variables. For example, one might
run NPEFF using only the parameters from a single transformer layer. While we do not explore that in this work,
we have explored this a bit and found the components to have interpretable tunings.

Reviewer comment (Evaluation):
The experiment on fixing flawed heuristics achieves only a slight improvement of 0.5% accuracy, by improving predictions for 4/48 components associated with incorrect heuristics.

Author response:
We agree that 0.5% is a small boost, but a small boost is to be expected when correcting a single incorrect heuristic
that is used only on a relatively small subset of examples. The change was significantly bigger when we look at
model's predictions for the component top examples. The four components that provided a boost of 0.5% or greater to
the total accuracy had accuracies of 27%, 24%, 58%, and 41% on their top 128 examples using the base model. Their
respective perturbed models reached accuracies of 88%, 72%, 95%, and 77%, respectively, on the same examples.

We also note that these improvements are for correcting
a single heuristic at a time. We found that perturbating multiple components at once could boost total accuracy
further, but we did not explore this beyond some preliminary experiments.

Reviewer comment (Evaluation):
Unclear how the perturbations experiment in Section 3.1.2 demonstrates that NPEFF's discovered parameter space directions are important as stated at the end of the section. Would be nice to see more potential application of NPEFF with strong results.

Author response:
We've added some text in the last paragraph of that section to help explain this. The goal of the perturbation
experiments was to show that the direction in parameter space associated to a component (its pseudo-Fisher) is of particular importance
to the model's processing of the component's top examples. The perturbation experiment tests this by constructing
a perturbation derived from a components pseudo-Fisher and seeing how much the predictions for each example change
as measured by their KL-divergences from their original predictions. Taking the ratio of the average KL-divergence for a component's top
examples to the average KL-divergence for the data set as a whole measures how much more/less the predictions of the top
examples changed compared to the rest of the examples. However, the model's predictions for some examples are simply more
sensivitive to perturbations in general, which is measured by the norm of their PEFs. Hence, if component top examples tended
to have PEFs with larger norms than average, then looking at the KL-divergence ratios alone would be misleading.
We therefore also looked at the ratios of the average PEF norm for the component's top examples to average PEF norm for the
rest of the examples. As the KL-ratios tended to be significantly higher than the PER-norm ratios, the component-derived
perturbations affected the component's top examples more than can be explained by their general sensitivity to perturbations.

Reviewer comment (Evaluation):
In section 2.2's preprocessing section, you state that raw PEFs gave components tuned to outlier examples. Could NPEFF with raw PEFs be useful for some form of OOD detection or prediction confidence measure?

Author response:
Potentially. A simple alternative would be just looking at the norm of a PEF for something like OOD detection.
NPEFF could provide an advantage over this by having components match up with specific reasons why examples
are OOD or hard to make good predictions on. For example, such a component from a digit identification task
could be tuned to ambiguous 4/9 and 5/6 digits.
Knowing that an example had a high coefficient for such a component could be useful for providing a reason why
one might not want to place much confidence on the model's prediction for a particular example. We haven't done much
exploration with NPEFF on raw PEFs, but it is a potential future research detection.

Reviewer comment (Evaluation):
Could you give more information on how Section 3.1.2 perturbation experiment demonstrates the claims at the end of this section?

Author response:
We've added some text in the last paragraph of that section to help explain this. The goal of the perturbation
experiments was to show that the direction in parameter space associated to a component (its pseudo-Fisher) is of particular importance
to the model's processing of the component's top examples. The perturbation experiment tests this by constructing
a perturbation derived from a components pseudo-Fisher and seeing how much the predictions for each example change
as measured by their KL-divergences from their original predictions. Taking the ratio of the average KL-divergence for a component's top
examples to the average KL-divergence for the data set as a whole measures how much more/less the predictions of the top
examples changed compared to the rest of the examples. However, the model's predictions for some examples are simply more
sensivitive to perturbations in general, which is measured by the norm of their PEFs. Hence, if component top examples tended
to have PEFs with larger norms than average, then looking at the KL-divergence ratios alone would be misleading.
We therefore also looked at the ratios of the average PEF norm for the component's top examples to average PEF norm for the
rest of the examples. As the KL-ratios tended to be significantly higher than the PER-norm ratios, the component-derived
perturbations affected the component's top examples more than can be explained by their general sensitivity to perturbations.

Reviewer comment (Evaluation):
Figure 3: it seems interesting how in QQP and NLI, the PEF norm ratios take a wider range of values than the KL ones, while in ImageNet the opposite is true. Is this meaningful in any way?

Author response:
We think this might be due to the difference in the number of classes between the tasks. The text tasks have 2 or 3 classes, and the
model can use some heuristics very confidently to make predictions. This leads to these component top examples having very small
PEF norms. In contrast, ImageNet is a fine-grained image classification task with a 1000 classes. Most strategies used by the
model wouldn't be able to confidently make predictions for a single class, so component top examples will tend to have norms
more in line with average examples. This leads to PEF norm ratios taking on a far larger range for QQP and NLI compared to ImageNet.

Reviewer comment (Evaluation):
Could you give more details on what constitutes a "concept" and what components resulted in Section 3.2? For instance, did each component tend to focus on one particular concept and components were quite diverse from each other (as likely desirable), or do many components tend to identify with many different concepts so that components are relatively similar to each other?

Author response:
We have updated Section 3.2 to more clearly describe what the "concepts" were and the tunings of the
recovered components. Essentially, concepts corresponded to values of Boolean intermediate variables in the RASP program.
Since the RASP program was human written and readable, these meanings of these concepts can be ascertained by looking
at the associated variable in the RASP code.

We found the set of component tunings to be fairly diverse. We also found that many components were tuned to the logical "and"
of multiple concepts, which can be thought of as more fine-grained concepts. Compared to the logical "or" of components mentioned
in the latter part of your question, this does not make component tuning relatively similar to each other.