Paper: Don't trust your eyes: on the (un)reliability of feature visualizations (ICLR-2024)

Reviewer comment (Experiments):
A visualization of late layer features does not need to be processed similarly to natural images at least on early layers.

Author response:
**Point 1** *“a visualization of late layer features does not need to be processed similarly to natural images at least on early layers”*:

We fully agree that differences in early layers wouldn’t be problematic. As an example, it would be totally fine for an “airplane” visualization to have different color statistics or low-level features compared to natural airplane images. It would only be problematic if the “airplane” visualization lacks mid- and high-level features that belong to airplanes. Our Figure 6 didn’t really allow a reader to assess which layers correspond to low-, mid-, and high-level features. In the light of your comment, we have therefore updated Figure 6 with annotations that show not only the name of the layer (previously we used just an uninformative index) but also an annotation copied from https://distill.pub/2017/feature-visualization/appendix/ that describes layers as responding to “textures”, “object parts”, “sophisticated concepts” etc.
Given this annotation, it becomes clear that feature visualizations are processed differently from natural images not only in low-level layers but also in layers corresponding to object parts, complex and sophisticated concepts - precisely the kinds of features that the visualizations were designed to ‘explain’ and visualize. If a feature visualization wants to succeed in explaining how natural input is processed, it needs to better capture those features. We have added the following to Section 3: *“While it would be fine for a feature visualization to show different low-level features compared to natural images, any visualization that seeks to explain how natural input is processed should capture similarities in mid- and high-level layers that are described by Olah et al. (2017) as responding to ‘object parts’ and ‘complex/sophisticated concepts’.”*

Reviewer comment (Experiments):
Do you think this work should also raise concerns against using let's say 5 most highly activating natural images to describe a neuron, as this is a common alternative to using feature visualizations?

Author response:
**Point 2** *“Do you think this work should also raise concerns against using let's say 5 most highly activating natural images to describe a neuron”:*

That’s an interesting question! Our paper covers three perspectives and we thus have three answers to this question.

From the *adversarial perspective*, we could easily use our method from Section 2.2 to build a network where the top k natural images don’t correspond to what the unit is usually selective for; this would simply mean setting the bias parameter b to a lower value such that it would only suppress activations up to, say, the 95th percentile of natural input. A recent paper specifically looked into manipulating the top-k activating images; we cited it in the related work section as “A complementary approach is proposed by Nanfack et al. (2023), which changes highly activating dataset samples” (https://arxiv.org/pdf/2306.07397.pdf).

From the *empirical perspective*, highly activating natural images would pass the sanity check since highly activating natural images are, by definition, natural images that highly activate a unit, and they would thus be processed like other natural images.

From the *theoretical perspective*, our impossibility results can be extended to many variations on using the argmax for feature visualization, including using the most activating dataset samples as explanations. Essentially, as long as the feature visualization method does not narrow down the function space too much, our results will apply. It is easy to see that two simple (e.g., piecewise linear) functions could have the same 5 (or 10, etc.) local (arg)maxima and yet behave very differently even quite near these local maxima, and hence our theorems could be extended.

Thus, in summary, highly activating natural images would pass our sanity check, but it would still be possible to construct networks that show misleading highly activating natural images, which is a case that’s covered by the theory. Since this is a question we feel might interest readers as well, we have added a dedicated and detailed explanation to Appendix A.3 (“Relationship to highly activating natural samples”), essentially covering the points we describe above. Thanks again for the question!

Reviewer comment (Experiments):
The NN model interpretation method for dense prediction tasks，like class activation map is widely used in weakly learning community. It will be more convincing if the author can provide some visualization results using their proposed method to see how will the final target object saliency map changes.

Author response:
We’re not sure we fully understand the question. Saliency maps ask “which pixels provide evidence for a specific class given this particular image”. In contrast, feature visualizations ask “how does the maximally activating input for a unit look like” (without explaining any specific input image). Both areas are interesting but the methods are different. In particular, in this paper we focus on feature visualizations and provide ample samples (Figures 1,2,3,5). Our methods neither aim nor attempt to fool saliency maps; this has already been done extensively by prior work (e.g. Adebayo et al. 2018, Nie et al. 2018). Could you let us know whether this clarifies the matter, or elaborate if it doesn’t?

Reviewer comment (Writing):
While feature visualization is one of the major types of interpretability methods in computer vision, it seems misleading to talk about "widespread use", as it is still a relatively minor method compared to truly widespread attribution methods (as feature visualization already suffers from other more practical limitations).

Author response:
1. *Contextualizing / describing “widespread use” of feature visualizations:*

We agree that contextualization is important. It is correct that, given the size of the overall ML community, feature visualizations are not a generally widespread tool. What we meant to say is that feature visualizations are one of the standard tools within the mechanistic interpretability community (which we think resonates with your review stating that feature visualizations are a “central tool to generate insights about the features encoded by the units of computer vision systems”). We identified a number of sentences where we stated “widely used / widespread use”  and contextualized to make this more clear. For instance, we now write “Despite its widespread use within the mechanistic interpretability community” / “Feature visualizations based on activation maximization are a widely used tool within the mechanistic interpretability community”.

Reviewer comment (Presentation):
While the proofs of concept are informative and something to keep in mind, they are extreme cases unless one is using the method for auditing external models (which is far from the common scenario). The scope of the proof of concept might be clear in the conclusion but not as much in the rest of the paper.

Author response:
2. *Contextualizing / The scope of the proof of concept might be clear in the conclusion but not as much in the rest of the paper.*

It seems that the right way to contextualize the proof of concept is to emphasize that it requires access to the model. We have thus revised the introduction summary bullet point 1. to state “Thus, feature visualizations can be deceived if one has access to the model” and the conclusion of Section 2 which now reads “This establishes that feature visualizations can be arbitrarily manipulated if one has access to the model” to make this more clear.

Reviewer comment (Evaluation):
No sanity check is perfect, especially if it becomes a metric to optimize for. Do the authors have intuition about the limitation of the sanity check, i.e., how could the sanity check be fooled?

Author response:
4. *No sanity check is perfect, especially if it becomes a metric to optimize for. Do the authors have intuition about the limitation of the sanity check, i.e., how could the sanity check be fooled?*

That’s an interesting point to think about! We’ve added the following example of a limitation to the limitations section (App. G):
“No sanity check is perfect, and like any sanity check, ours is just a necessary but not a sufficient condition for a reliable feature visualization. For instance, if the training data contains spurious correlations (all cows are on grass); a unit in the network is selective for grass and not for cows but the feature visualization shows realistic-looking cows on grass (rather than just grass), then the visualizations would pass the sanity check without raising a warning about the spurious correlation present in the visualization. We would love to see more research on sanity checks - to the best of our knowledge we provide the first one which hopefully serves as a motivation for research on both better visualizations and more sanity checks.”

Reviewer comment (Experiments):
While the experiments proposed are indeed not model-specific, half seem to be done using an Inception and the other half a ResNet50, is there a reason for this inconsistency?

Author response:
5. *While the experiments proposed are indeed not model-specific, half seem to be done using an Inception and the other half a ResNet50, is there a reason for this inconsistency?*

Yes! We were worried that readers and reviewers might think the experiments/methods are somewhat model-specific or do not generalize across architectures if we just use a single network. We therefore made sure to include two different model families. We realize that’s a bit confusing without explanation and have thus revised the intro to Section 2.2 to include the following: “We designed this method to show that fooling can easily be achieved in different ways and across different architectures. In order to demonstrate that different model families can be fooled, we here consider a different architecture (ResNet-50 instead of Inception-V1) and a randomly selected intermediate layer instead of the last layer (but note that the approach is not specific to a certain architecture or layer).”

Reviewer comment (Experiments):
Regarding the sanity check I would be very curious to see the results from other baseline to further explore the causes that could explain the difference.

Author response:
6. *Regarding the sanity check I would be very curious to see the results from other baseline to further explore the causes that could explain the difference.*

We’re working on this and will update the response once we have results. [EDIT: please see https://openreview.net/forum?id=OZWHYyfPwY&noteId=H0vbRC6wgf below for our update and Appendix E for the results]

Reviewer comment (Experiments):
Is it simply a question of the ood-ness of the feature visualization as pointed out by Figure 3? In that case, I would be curious to see as extra baselines such as natural images vs natural images of the same class from another dataset (e.g., [1]) and natural images vs natural images of different classes from another dataset.

Author response:
*Regarding the sanity check I would be very curious to see the results from other baseline to further explore the causes that could explain the difference.*:

We love the suggestion of using out-of-distribution datasets like NINCO to better understand whether feature visualizations fail the sanity check since they might be out-of-distribution. We have now included an analysis on two commonly used and complementary datasets in a dedicated section (Appendix E):

1. **ImageNet-V2**, which has the same classes as original ImageNet and only a small distribution shift as evidenced by a roughly 10-15\% accuracy drop of standard models;
2. **NINCO** as susggested by you, a dedicated out-of-distribution detection dataset that specifically contains no classes that are part of original ImageNet. 

Appendix E contains results and a brief discussion on those new baselines. Interestingly, we find that the similarity between same-class images of those OOD datasets (including NINCO, which was designed to be a strongly out-of-distribution!) is still a lot higher than the natural-vs-feature-visualization similarity on ImageNet.

Reviewer comment (Theory):
Not putting this in weakness, but I am not sure what to make of the theoretical section. Even though feature visualizations have been shown, at times ([4-5-6-7]), to offer insights about the network's unit activation, the main takeaway of this section, for now, is to caution that there is no guarantee that this is a general trend or that the insights are not only at a coarse level, which is important to keep in mind but it does not invalidate the empirical results previously cited. That being said, if one of the conclusions of this section is that alternatives to the current paradigm of activation maximization should be seriously considered by the community, it would be insightful if the authors could suggest possible alternatives that would overcome some of the limitations pointed out in this work.

Author response:
8. *It would be insightful if the authors could suggest possible alternatives that would overcome some of the limitations.*

We agree with you that finding a new paradigm that overcomes the reported issues will be an exciting result. While we would have loved to provide a more reliable feature visualization method for general black-box neural networks, our theory proves that this cannot be guaranteed to be reliable: irrespective of the nuances of different feature visualization methods based on activation maximization, much more knowledge about the ‘black-box’ is necessary to make reliable statements — a pretty strong result, backed up by extensive theory, that we believe will have a substantial impact on XAI since it challenges the concept of “black-box interpretability”. We believe that future work could instead explore constructing networks that incorporate more constraints and assumptions (i.e., networks that are far from being black boxes) and thus enable more reliable predictions.

Reviewer comment (Writing):
With regard to softening the claims (and with respect to the previous point), one concrete suggestion is, in the conclusion: ": Given that our theory proves that visualizing black-box systems via feature visualizations based on activation maximization (i.e., the current dominant paradigm) is impossible -> "Given that our theory proves that there is no guarantee that ... is possible".

Author response:
9. *With regard to softening the claims (and w.r.t the previous point), one concrete suggestion is, in the conclusion …*

Thanks for the suggestion. We rephrased the conclusion as follows: “Given that our theory proves that visualizing black-box systems via feature visualizations based on activation maximization (i.e., the current dominant paradigm) cannot be guaranteed to be reliable without making strong assumptions about the system, …”.