Paper: Feature Accentuation: Explaining 'what' features respond to in natural images (ICLR-2024)

Reviewer comment (Evaluation):
My main concern is I'm not sure what the utility of feature accentuation is. A good chunk of the experiments focuses on natural-looking images, but I'm slightly less concerned about that. It's certainly a desirable property -- to look more natural and less like hallucinations. However, I'm not sure how this helps us, concretely: Can this help us improve accuracy? Diagnose mistakes? (This is suggested in several of the figures) Suggest a path for fixing the model? (Maybe by pruning "wrong" nodes?) There's a preview of this in figures 9 and 10, but the utility isn't studied in the paper. [1] for example has several approaches for defining and quantifying interpretability/utility of a method.

Author response:
**“My main concern is I'm not sure what the utility of feature accentuation is.”**

Feature Accentuation represents a novel tool that, as you rightly pointed out, yields feature visualizations ranging from more to less naturalistic. In recent years, general feature visualizations have significantly impacted the literature, enabling the understanding of internal mechanisms within neural networks (such as specific filters and invariances)[1,2,3], detection of biases [4], enhanced comprehension of complex concepts [6], and visualization of internal states of intricate models[7].

We have specifically chosen to illustrate the utility of our method with a use-case presented in Figure 9. This example demonstrates how Feature Accentuation can provide valuable insights in explaining failure cases. For instance, in the first image, it might seem perplexing that an image of a barber shop is classified as a moving truck. Our approach sheds light on such cases revealing that the model mistakenly associates the white sheet on the wall with the trailer of a truck.

We posit that Feature Accentuation serves as a superset of traditional feature visualizations, offering the potential to accelerate all those use cases simultaneously. By providing a way to explore a diverse set of explanations ranging from a natural image to purely abstract feature visualization, our method contributes to the broader landscape of interpretability tools.

Reviewer comment (Evaluation):
Along the lines of the above, I'm not sure how to use the information these visualizations provide. For example, in figure 9, for "bow" vs. "chainsaw", I can certainly see that the "chainsaw" visualization is more chainsaw-like, but couldn't I just pull a random other word and visualize that? For example, I could feature accentuate "matchsticks" or "juggler fire torches", and I'm not sure how any of those visualizations would help me diagnose mistakes in the original model. Another way to put this would be: If I feature accentuate something clearly unrelated, eg.., "snail" for the "bow" image, it seems like this method would find *some way to insert a snail, but what does that tell me exactly, if I can insert any concept into the image? (One possibility is that natural looking insertions are "valid" explanations, and grotesque abstract art is unrelated? This would need some more fleshing out though, but that would be one way). On a side note, the main utility in figures 9 and 10 is ironically that it highlights some part of the original image, like saliency maps. (I'm not sure how important it is to be able to modify the image, per the above). A study would probably need to show that the image modification is helpful too.

Author response:
**“couldn't I just pull a random other word and visualize that? For example, I could feature accentuate "matchsticks" or "juggler fire torches", and I'm not sure how any of those visualizations would help me diagnose mistakes in the original model.”**

This is an excellent point, and something that concerned us as we were developing the method. Indeed, if we can accentuate arbitrary features, it’s possible the technique could mislead the user, ‘revealing’ how features are expressed in an image when in fact there is nothing in the image that excites the feature. We demonstrate this phenomenon empirically in Figure 6. Given this concern, we demonstrate in section 2.4 how feature accentuations can be combined with attribution masks as a means of filtering, revealing exaggerated versions of the original image only in those image regions that excited the feature in the original. In this way, feature accentuation can help intuit the many different ways natural images excite a feature, without suggesting to the user that everything excites it.

Reviewer comment (Presentation):
Raster plots in figure 8 are poor quality; hard to see detail.

Author response:
Thank you for your feedback; we have incorporated it into the manuscript. As our paper contains a lot of images we were constrained by the file size limit on our submission, and subsequently compressed the PDF too aggressively. Our undated manuscript is higher resolution, and the camera-ready version will feature the highest possible quality.

Reviewer comment (Presentation):
Figure 8 B, C: lambda should probably be 1.0 not 10.0.

Author response:
We actually meant for this to be 10.0, as we wanted to measure the effects setting lambda to extreme values.

Reviewer comment (Novelty):
How novel is the frequency domain parameterisation? Citations to other literature should be provided.

Author response:
While the frequency domain parameterization is not a novel concept and has been discussed in [1,2], we have, nonetheless, integrated and tested a more recent parametrization (NeurIPS 2023) that proves effective on newer models [3]. Additionally, we have included several missing citations.

Reviewer comment (Evaluation):
What does it mean to achieve higher correlations in circuit similarity than natural images themselves? In a positive view, this could mean that the feature accentuation technique is settling onto good *prototypes* that provide a coherent illustration of the core concept of the label, and thereby reduces variance from natural depictions of the concept. A less positive view could be that this means the technique is finding local minima that will fail to generalize.

Author response:
Our interpretation aligns with your ‘positive view’, which you articulate quite well (perhaps better than we articulate ourselves in the paper)! With regards to generalization, we are confident this result will generalize to new examples given how it was calculated. Each natural image in our experiment can be paired one-to-one with an accentuation of that image towards its class logit. These accentuations are conditioned in no way on any other images in the experiment. Within a given class, we get the average pair-wise correlation for each natural image, and similarly the average correlation of each accentuation to the natural images (excluding the correlations between accentuations and their natural image pair, which would be trivially high). We find the accentuations correlate higher on average, which means accentuating an image towards its class tends to make it follow a circuit closer to other, randomly selected images of the class.

Reviewer comment (Reproducibility):
Figure 8: What are the smooth curves, and how were their hyperparameters chosen? The underlying data are very noisy, so the choice of smoothing and its associated uncertainty should be reported.

Author response:
The curves depicted in these figures are a spline interpolation (degree 2), across the underlying data-points (class-wise average correlations) averaged into 10 bins. We took this approach so as to convey the raw data and the general trend simultaneously. A difficulty with plotting this data stems from the fact that there is a large amount of variance in the correlation measure across nearby layers of different architectural type. We can filter out this variance in Figure 8.a and isolate the effect we are interested in by plotting the difference between the correlation measure for accentuations and natural images in each layer. We have added this version of the plot – which requires no smoothing – to appendix A, as well as additional clarification on the figures and design of the path coherence experiment.

Reviewer comment (Experiments):
All results in the main paper are quantitative and are shown only on a few selected examples.

Author response:
**“All results in the main paper are quantitative (assume means qualitative)”**

It’s true that a significant portion of our results focuses on qualitative outcomes, as in some sense feature accentuation is an intrinsically qualitative explanation method; it is designed to produce visualizations a user can qualitatively evaluate to gain intuition for what excites a given feature about an image. Attribution methods and feature visualizations, the standard approaches in this area, are also qualitative in this respect. That said, we respectfully disagree with the notion that our paper lacks quantitative depth. We would like to draw attention to an entire section dedicated to quantitative results (Section 3, circuit analysis, Figure 8 a, b, and c) that we feel compelled to highlight.
In this specific section, we argue, akin to the findings presented in [1], that feature accentuations for maximizing an activation should do so in a ‘natural’ way, i.e. by a similar path through the network as the natural images that excite the neuron. Quantitatively, we demonstrate that our method – which does not require additional models that enforce a natural-image prior – achieves excellent results when compared to state-of-the-art feature visualizations on this metric (Figure 8.a). Furthermore, we argue this analysis can be extended to address whether feature accentuations are providing veridical, local explanations for their seed image. That is, a feature accentuation is not a good explanation for why a neuron is excited by an image if the accentuation takes a distinct path through the network. Figure 8.c shows that increasing regularization strength keeps the path through the entire network closer to the seed image. The quantitative evidence presented in Section 3, coupled with our method's independence from additional models, make this work an advancement on the original feature visualizations. We believe this unique aspect of our methodology not only adds depth to the interpretability toolkit but also addresses concerns raised by previous studies.

Reviewer comment (Experiments):
The framework is sensitive to hyperparameters such as learning rates and regularization weights, as noted in Appendix C-D, which makes it challenging to adapt to different models.

Author response:
**“The framework is sensitive to hyperparameters such as learning rates and regularization weights, as noted in Appendix C-D, which makes it challenging to adapt to different models.”**

Indeed, our framework includes a regularization term designed to keep accentuations close to the seed image while performing activation maximization for a target feature. We posit that this hyperparameter doesn't necessarily require tuning, but rather allows the user to control the extent to which the target feature is exaggerated, a desirable property for VCEs [2]. Additionally, we have added a small experiment to the appendix (section F) that further explores the setting of regularization strength in different contexts. We find that while the appropriate level of regularization must be identified across models, the same regularization strength works well across images/features within a model layer. Further, we present a method for finding the appropriate amount of regularization that does not require visual inspection by the user, thereby automating the selection of this hyperparameter. 
Regarding other hyperparameters, while we do spend a considerable portion of the paper exploring the hyperparameter space, we ultimately prescribe settings that work well in the general case. With regards to the learning rate in particular, our phrasing in appendix C was too strong (it has been updated), a more accurate statement is that the upper limit for the learning rate in feature accentuation is lower than for feature visualization. As long as the learning rate is sufficiently low, it will work well in a variety of contexts. To demonstrate this, we have added Appendix F, which shows feature accentuations for a range of models. The only hyperparameter that is adjusted in this experiment is the regularization strength, which is set once per model, but not adjusted across different images. All other hyperparameters are fixed across all examples. The regularization layer, which of course can not be identical across models, is simply set to the ReLU layer closest to ¼ depth through each model.

Reviewer comment (Evaluation):
Multiple qualitative examples are shown but the analysis is lacking. For example, how to interpret the results from Figure 12 in the Appendix?

Author response:
“how to interpret the results from Figure 12 in the Appendix?”
An interpretation of ‘super-natural images’ like those depicted in Figure 12, can be found in Section 3; they are images generated by feature accentuation that take a prototypical path through the network for their corresponding class. We put Figure 12 in the appendix to limit the number of images in the main paper (as there are a lot already). Still, we presumed the reader would be curious to see what these ‘super-natural’ images actually looked like. We have clarified the figure caption.