Paper: SPADE: Sparsity-Guided Debugging for Deep Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
The SPADE algorithm would require a massive amount of computation time when applied to datasets like the ImageNet validation set, taking 41 minutes per example. This would translate into several weeks to process all examples, and would necessitate re-running the inference algorithm for each example or storing a different set of weights for every example. This is impractical and greatly limits applicability in real-world settings.

Author response:
* **Slow computation time/massive resource requirements to apply at scale.** We thank the reviewer for raising this concern, which was also raised by others. To address this, we note that the bottleneck is the sparsification cost. Therefore, we ran additional experiments with a more efficient solver, to provide speed-accuracy tradeoffs. The results are described in the general response above, but in a nutshell, *using the fastest pruning strategy drops the per-example requirements to 70 seconds on ResNet 50, at a small fidelity drop that still improves substantially over the baselines.*

Reviewer comment (Experiments):
The usage of post-hoc custom sparsity solvers requires ad-hoc thresholds. The authors attempt to mitigate this by using 100 calibration samples to maximize the average input pixel AUC score for the saliency method when ground truth is known, but this approach has three major issues: (i) it uses additional information to tune the sparsity ratio, making the comparison with [1] unfair, as [1] balances sparsity and accuracy rather than optimizing a saliency method; (ii) it uses all 100 examples to compute sparsity ratios, thus employing global information and undermining the per-example sparsity argument; (iii) it adds further complexity to an already computationally intensive method.

Author response:
* **Feasibility and fairness of calibration.**  We agree that the calibration can be infeasible. In that case, we propose a fixed sparsity schedule independent of interpretability method, that increases linearly with layer depth, and demonstrate that such a schedule is also quite effective (Appendix D/Table D.14). We also believe that our calibration does not unfairly leak information from the test data, as we use different data samples (also drawn from the ImageNet validation set) and different Trojan patches (emoji) for calibration.

Reviewer comment (Experiments):
The comparison between SPADE and [1] may be inappropriate, as [1] aims to create highly sparse networks while maintaining accuracy, focusing only on the last linear layer, and enabling inspection of individual neurons — a design choice with different goals than SPADE.

Author response:
* **Comparison with Wong et al.** We agree that our approach differs from Wong et al, in that we use per-example sparsity, whereas they train a single sparse FC layer for use in all examples, and for all interpretation methods. However, our approach is quite original (something that we see as a strength), and so Wong et al. was the closest approach available in the literature, thus, the best competitive baseline. We updated Section 4.1 to highlight the issues that you have raised.

Reviewer comment (Novelty):
The per-example sparsity rationale itself is not novel, as [2] recently proposed a data-driven, Bernoulli-based per-example sparsity method in Concept Bottleneck Models (CBMs). This method allows the model to select features for the last linear layer, bypasses limitations of [1], is highly efficient and generalizable, and requires training only a single linear layer, potentially mitigating many issues faced by SPADE.

Author response:
* **Novelty.** We feel that our method is novel in the sense that it uses *sample-specific, dynamic* sparsity to improve model interpretations -  so much so that we had trouble finding baselines to compare against, except for Wong et al, which as you note is an imperfect comparison. Regarding Panousis et al., we first note that ICCV was in October of this year, while the deadline for ICLR was in September; thus this is a concurrent work. Nevertheless, note that the method described in this paper creates sparse interpretable networks with a specific architecture (concept bottleneck), which is substantially different from what our method does - namely, provide a way to create better interpretations for *dense* models with a standard architecture.

Reviewer comment (Evaluation):
The computational effort required for preprocessing by SPADE is relatively high, making it difficult to use in practical situations.

Author response:
As all reviewers noted that the computational requirements of SPADE can limit its practical use, we amended the paper to propose other variants of SPADE that provide different speed/accuracy tradeoffs. In particular, for a small accuracy drop, SPADE preprocessing can be done in 70 seconds per example.

Reviewer comment (Experiments):
I could not fully understand the details of the human study.

Author response:
The image patches are contiguous regions of the image that, according to a low-resolution saliency map method (applied without SPADE) are the most important evidence for a specific possible predicted class. In our human evaluation, we used misclassified examples and produced image patches for the correct and predicted class, filtering only for those input images for which these do not intersect. The idea is that if the class visualization is human interpretable, the evaluator would be able to use the visual similarities between the class neuron visualization and the image patch to find the part of the image that resembles what the neuron is ‘looking for’.

Reviewer comment (Reproducibility):
Is the re-calibration of batchnorm mentioned in page 4 also applied to the model after SPADE preprocessing?

Author response:
The re-calibration of the batchnorm was not applied to the model after SPADE preprocessing, as the model trace produced by SPADE is not useful for inference.  The similarity result for which the batchnorm is applied was only presented as a sanity check that the top-1 prediction of the trace is still close to the network prediction.

Reviewer comment (Presentation):
Figure 2 (left) implies that image rotation is used as data augmentation, which is not the case according to Section 3.2.

Author response:
Thank you for  observing that we do not use rotations; we corrected this in the figure.

Reviewer comment (Writing):
In page 9, I could not understand the meaning of "the image patches were always generated from the dense model". What are "image patches" in this context?

Author response:
The image patches are contiguous regions of the image that, according to a low-resolution saliency map method (applied without SPADE) are the most important evidence for a specific possible predicted class. In our human evaluation, we used misclassified examples and produced image patches for the correct and predicted class, filtering only for those input images for which these do not intersect. The idea is that if the class visualization is human interpretable, the evaluator would be able to use the visual similarities between the class neuron visualization and the image patch to find the part of the image that resembles what the neuron is ‘looking for’.

Reviewer comment (Writing):
In page 9: "there were were" -> "there were"

Author response:
Thank you for the correction, we will fix the typo.

Reviewer comment (Experiments):
**Small-scale ImageNet experiments?** — Please clarify if I misunderstood but it appears that the main result (i.e., saliency map "accuracy" of SPADE vs. Dense vs. Sparse FC on ResNet50/ImageNet) is only calculated for **140 test samples out of the available 50,000** in the ImageNet-1K validation set. It seems bold to claim AUC improvements and Pointing Game score gains when evaluation is done on 0.0028 of the actual validation set, especially when it is unclear how/why these 140 chosen samples are to be representative of the wider dataset.

Author response:
* **Small-scale ImageNet experiments.** The 140 test samples were chosen at random from the ImageNet validation dataset, subject to the following conditions: that the number of samples for each Trojan patch (emoji) was the same, and that only samples where the predicted label was changed by the Trojan were selected. The small number is due to the fact that each sample required custom pruning with SPADE, which takes ~40 minutes for each saliency method. We do believe that while the relatively small sample size makes the evaluation more noisy, it does present a useful comparison of the methods, especially as the results are quite consistent across datasets and saliency methods.

   However, due to your and others’ feedback, we will add to our paper experiments on a modified version of SPADE, that uses a more efficient pruner to create the saliency maps. Using this version of the method, **we were able to run SPADE + LRP on 21121 samples from the ImageNet validation set (all that met our other criteria) in 120 GPU- hours on GeForce RTX 3090 GPUs.**

Reviewer comment (Experiments):
**Non-negligible cost for constrained optimisation of sparse network** — As described in Section 3, SPADE relies on activation matching for every layer and takes 41 minutes for single example preprocessing. This method becomes prohibitively costly with increasing cost proportional to dataset size and the number of network layers, rendering it impractical for large-scale deep neural network debugging.

Author response:
* **Non-negligible cost for constrained optimisation of sparse network.** In the original work, we conceived of SPADE as a method for human debugging, for which perhaps a few dozen examples would be examined, and so we believed the per-example cost to be acceptable. However, as described in the previous answer, SPADE can also be used with more efficient sparsity solvers at a small accuracy cost (but still an improvement over the baselines), **reducing the cost to 70 seconds per example for a ResNet50**. We provide these tradeoffs below, and also in Appendix I of the revision.

Reviewer comment (Novelty):
**Limited novelty** — Using sparsity to factorise / disentangle concepts is a known direction in literature. SPADE adopts this perspective, then leverages an existing OBC sparsity solver to obtain sparse subnetworks w.r.t. every example (and its augmentations), then applies off-the-shelf interpretability techniques to generate various interpretation maps. SPADE does not seem to present novel insights, methods or findings.

Author response:
* **Limited novelty.** We respectfully disagree with this assertion. Other works (with the exception of Wong et al (2021), which we address separately), construct sparse networks and demonstrate that they are somewhat more interpretable than dense networks, at a cost of some degradation in the performance of the network on the desired task, e.g. image classification. In particular, these networks are intended for inference, and replace the dense network entirely (in cases where the sparse network is obtained from a dense one, rather than trained from scratch). Conversely, the sparsification in SPADE is used as an additional step during the computation of an example interpretation, and does not produce a replacement network for inference or any other purpose. The sparsification in SPADE removes network connections that are least relevant to the particular example being studied. This preprocessing reduces noise in interpretability methods and increases their accuracy and usefulness on that example, but is not intended to produce networks useful for classification; nor do we propose using SPADE during regular inference. Note that this means that there is no inference performance tradeoff (in time or quality)  to using SPADE, as SPADE is used solely for creating interpretations, and regular inference is not affected.

   The only other work to our knowledge that uses sparsity in this way -  solely for the creation of better interpretations - is the work of  Wong et al, which replaces the final layer with a sparse one. However, unlike that work, our method introduces the creation of example-specific traces through the network. This, to our knowledge, is completely novel, and we believe to be the core driver of the improved saliency map accuracy results.

Reviewer comment (Evaluation):
**Fairness and fidelity of input saliency maps (not neuron visualisations)** — SPADE requires sample-wise sparsification before interpreting each different image sample, meaning that the saliency map is generated with respect to a different subnetwork for each image. Sparse networks A and B could exhibit drastically different performance (accuracy and interpretability) on examples A and B, it therefore seems unfair to interpret different examples using different subnetworks. SPADE saliency map visualisations can only be matched / replicated by using the exact same sparse subnetworks for every example and is hence costly to reproduce. It is furthermore unclear to me why SPADE saliency maps would be representative of those of the original dense model.

Author response:
* **Fairness and fidelity of input saliency maps.** We respectfully disagree with large parts of this point as well. We consider the preprocessing with SPADE to be a denoising step that is part of the saliency map creation, and so would argue that the saliency maps are still created with respect to the original, dense network. As network fidelity is always a concern with saliency maps, we validate this in the human study by using SPADE in the process of creating neuron visualizations and verifying that these are still relevant to the original network. We also performed an additional experiment (please see general comment above), which demonstrates that preprocessing with SPADE allows saliency methods to identify pixels that have a greater effect on the confidence of the dense network, providing additional evidence of our method’s applicability to the original, dense network.

  We do, however, note the reviewer’s excellent observation that combining interpretability methods with SPADE can lead to some randomness in the resulting saliency map, due to a small amount of randomness in the pruning algorithm. This can be overcome if necessary by randomly generating and then fixing the transforms used to create sample batches used for pruning.

Reviewer comment (Experiments):
Does the sparsity solver find highly similar or dissimilar sparse subnetworks for different image examples? In other words, are the same set of neurons consistently retained for different images?

Author response:
1. We added an appendix to the paper (Appendix J) that considers similarities between the traces of different inputs. In short, there is about 20-60% overlap between the traces of two images, with more similar images having greater overlap, and more overlap in the earlier layers.

Reviewer comment (Evaluation):
Are there any specific patterns or properties of retained neurons in the sparse subnetworks? Do they have larger activation magnitudes, or perhaps do they belong to earlier / later or wider / narrower layers?

Author response:
1. Generally, the masks created by SPADE favor neurons with higher magnitudes, although only those that are relevant to the example in question. The sparsities of the different layers are fixed for each task/architecture.  We found that SPADE works best when the deeper (later) layers are pruned to a higher degree than the earlier layers, in fact, we show in Appendix D that SPADE also works well when a simple linearly increasing (with layer depth) sparsity schedule is used to set the desired layer sparsities.

Reviewer comment (Experiments):
Selecting an appropriate sparsity ratio for each layer — In Section 3.2, the choice to use cross-validation with ground-truth information to select the optimal "sparsity ratio" may lead to an overly optimistic representation of SPADE's performance, since in practice users would not have such ground truth available. This should be acknowledged as a limitation, and the paper should examine SPADE's sensitivity to different sparsity ratio values (e.g., if the wrong ratio is used, is it now useless?).

Author response:
* **Sparsity ratio tuning.**  We cross-validate by using a different set of emoji (and images) for tuning than we use for the main experiment. This is, of course, an imperfect proxy for the case where we are not looking for emoji trojans at all. Therefore, we also ran experiments where we used fixed sparsity ratios that are linearly interpolated by layer depth, from 0 in the initial layer to 99% in the final layer. We show in Table D13 that our method is also effective using these ratios.

Reviewer comment (Experiments):
Computational cost — Given that SPADE is computationally expensive, guidance is needed for how users should choose individual samples to explain. Is there a more principled strategy than explaining all misclassified or "surprising" examples?

Author response:
* **Computational cost.** We acknowledge that the computational cost of using SPADE can be high when using the method  on many examples, although we believe it is reasonable in cases where examples are created for human review, and so perhaps a few dozen are needed. Based on your and other reviewers’ feedback, *we added a faster version of SPADE that requires 70 seconds per sample.* Please see the general response to the reviewers above, and appendix I of the revision for details.

Reviewer comment (Evaluation):
Improvement over baselines is not that strong — Baseline saliency and visualization methods establish a surprisingly strong baseline, with differences between SPADE and baselines being relatively small in Table 1 and Figure 4. The paper should elaborate on why this is the case — for example, whether existing explanations are well-suited for the Trojan task but less appropriate in settings with subtler issues.

Author response:
* **Strength of results.** We would push back on the assertion that the results are not that strong. Preprocessing with SPADE improves the attribution map AUC for all methods tried. The gains are certainly larger where the accuracy gap is higher, and smaller where the accuracy is already very high, and so there is not much room for improvement. Concretely, the prior work of Wong et al., ICML 2021 improved attribution AUC on average by 0.5%, whereas SPADE improves results by 4.2-5.5% on average (depending on the variant used). 

   We are not aware of any research that shows that the Trojan emoji evaluation favors any specific methods, and so have no reason to believe that gains from SPADE would be greater in ‘real-world’ scenarios. Specifically, we emphasize that SPADE *largely closes the gap* between different saliency attribution methods, allowing the field practitioner to choose one that is best suited for their task on other criteria.

Reviewer comment (Experiments):
The paper should provide greater clarity about some experimental design details in the main text.

Author response:
Thank you for your very concrete and helpful suggestions on improving the text. We will update the draft to better clarify these points.

Reviewer comment (Experiments):
Clarify motivation for the Trojan experiment set-up — Based on [1], in real-life scenarios to find unknown "backdoors" or Trojans, we do not have access to individual examples with the Trojan present. This would make a data-dependent workflow, which requires explaining a model on an example with a Trojan, incompatible with such setups. The choice to use this Trojan set-up should be clarified in terms of its motivation for evaluating a data-dependent explanation method.

Author response:
* **Clarify motivation for the Trojan experiment setup.** Our motivation for using them in this paper is that, to our knowledge, Trojan patches are the best available way to evaluate saliency map accuracy, which was our motivation for using them. We agree that our method cannot be used to find backdoors that are not present in the test set; we will add this limitation to the paper.

Reviewer comment (Experiments):
Questions about saliency map experiments (Section 4.1) — In this set-up, is "accuracy" of saliency maps calculated only for images with Trojans (i.e., are the 140 examples in Table 1 all Trojan images)? How exactly is each saliency map calculated — is it the gradient of the neuron corresponding to the predicted class or the true (Trojan) class?

Author response:
* **Saliency map experiments.** We calculate accuracy only for test samples with Trojan examples, as those are the ones for which there is a ground truth to compare to. As the Trojan patches are not 100% effective for changing the label, we only use examples in which the predicted label (of the original network) is changed due to the presence of the Trojan. We always compute gradients (or any other saliency metric) with respect to the predicted (Trojan) class for the experiment, as that is the one for which the ground truth is available. However, SPADE can be used to compute saliency for any possible class as well.

Reviewer comment (Writing):
Language nit — The term "debugging": "Interpreting a model's predictions on specific examples" is more commonly referred to as a "local explanation" (see Section 4.2 of [1]), whereas "debugging" often implies the end user is a developer aiming to improve the model (as in [2]). While calling the method "sparsity-guided debugging" is fine, alternatives like "sparsity-guided interpretability" or "sparsity-guided explanations" could be considered.

Author response:
Thank you for these as well, especially for your thoughtful note on terminology. Our choice of the term ‘debugging’ stems from its use in Wong et al. (ICML 21). If the reviewer feels strongly about this, then we will consider changing our use of the term throughout the paper.

Reviewer comment (Writing):
Language nit — The term "preprocessing" is used to describe SPADE's model pruning step, but this term is typically reserved for data preprocessing before model training. Since this step is a post-hoc interpretation method that learns a sparse model, "pruning" may be a clearer term.

Author response:
We chose the term ‘preprocessing’ to emphasize the fact that SPADE is part of the interpretation process, rather than the model training process. We avoid using the term ‘pruning’ for the same reason (as it is conventionally used to mean creating a sparse network for inference).

Reviewer comment (Theory):
Section 3.1 — Provide a more formal definition of "sparse" as used in the paper. It appears to refer to L0 sparsity (majority of weights are 0) rather than L2 (weights having very small values). Since Section 3.2 mentions an L2 sparsity constraint, intuition should be given as to why L2 was chosen over another norm.

Author response:
To answer your questions, we mean sparsity in the L0 sense, and will clarify this; we use the L2 criterion in the sense that even as we impose the L0 sparsity we minimize the L2 change in that layer’s outputs on our input data.

Reviewer comment (Theory):
Section 3.1 — Provide further intuition or evidence for the "thinning" hypothesis. It is unclear why sparsity discourages 'multifacetism'; intuitively, if neurons are multifaceted at every layer, then even a sparse combination will still be multifaceted.

Author response:
Finally, we provide an explanation of the Multifacetism resolution in the general response above.

Reviewer comment (Evaluation):
Section 3.2 — Clarify in the draft what the 96.5% "agreement percentage" measures. Does it refer to the agreement of the single final class prediction?

Author response:
The 96.5% is the agreement of the final class prediction (used only as a sanity check, as the SPADE-preprocessed models are not used for inference).