Paper: DAME: A Distillation Based Approach For Model-agnostic Local Explainability (ICLR-2024)

Reviewer comment (Evaluation):
The main weakness of this paper is in the evaluation, because we do not know the reasoning behind DNN decisions — i.e., we do not know what a good explanation of its decision is — that we carefully develop methods for that purpose. In that sense, a subjective human evaluation of the quality of the explanation (b) is not actually informative of the quality of the explanation.

Author response:
***subjective evaluation not informative***: Thank you for the pointer on the evaluation. 
* We agree that there is no single way to explain away the decisions made by models.  Hence, we evaluate in a multi-pronged way, using fidelity measure, IoU, and the human subjective study.
* The motivation for the human evaluation is that, for the samples in which the black-box model is highly confident, how do the humans evaluate saliency maps generated by different XAI methods under the assumption that the black-box model, on these samples with high classification confidence, has attended to the most salient parts of the objects. 
* There are prominent studies and regulations that highlight the importance of explanations being human comprehensible as they are the end users [2,3,4,5]. 
* The collective evaluation with multiple metrics highlighted the advantage of DAME.

Reviewer comment (Evaluation):
The decisions of a DNN do not necessarily rely on the same features humans rely on (the opposite has previously been shown [1-2]). Hence an explanation that accurately depicts that the DNN does not use human-like features will be wrongly penalized by IoU metrics (a).

Author response:
***DNN not use human-like features .. penalized by IoU***: The IoU based evaluation is based on the assumption that, the classifier relies on the target object region for condidently right classified samples. To validate:
* We mask away the regions of the target object annotations, and these masked images leads black box accuracy going down from  $92.3$\% to $18.3$\%.
* However, masking away the regions outside the object class and keeping the object annotations leads to $76.5$\% accuracy. Although a drop from $92.3$\%, it validates that the annotations as primary information used by ViT model to classify the objects.
* As pointed by the reviewer, the misclassification may be due to the short-cut learning. To probe this, we computed the IoU on the mis-classified samples while explaining them using ground truth labels and obtained IoU of $16.3$\% ($31.4$\% on correct samples) using DAME. It indicates that the model has failed to focus on the object regions. The drop in IoU for other XAI methods are:

| method | mean IoU | IoU drop from correct samples |
|------------|----------------|----------------------------------------------|
| RISE       | 19.7 (14.2)    | 11.4                                         |
| LIME       | 16.6 (13.2)    | 10.2                                         |
| DAME       | 16.2 (13.9)    | 15.2                                         |

Reviewer comment (Experiments):
The motivation of the paper comes from the claim that linear models are inapt to accurately approximate non-linear models locally. An instantiation of the proposed framework with linear models is missing to make the claim more concrete.

Author response:
***proposed framework with linear models is missing***: We used a linear student network in DAME and observe that the model fails to generate meaningful explanations. The fidelity measure at $30$\% feature deletion results in only $24.3$\% drop in accuracy as opposed to $56.2$\% for the non-linear model.

Reviewer comment (Evaluation):
I was wondering if the authors have thought about running standard attribution methods on the original and student models as a sanity check that they do seem to have similar decisions for similar reasons?

Author response:
***similar decisions for similar reasons?***: We removed the top-k features (based on XAI output) gradually from the image and pass the masked images through the DAME student model  as well as the black-box model to get the  scores. The DAME scores show high similarity with the black box scores with $R^2$ of $0.819$ on VOC dataset, confirming their agreement.

Reviewer comment (Experiments):
The biggest bottleneck to using this approach would be having to train a whole new model to understand the behavior of the model for one single input sample.

Author response:
Thank you for pointing this out. Because of unavailability of gradient access to black box in gradient-free explainability, all post-hoc methods (LIME, DAME, and RISE) rely on generating  local perturbations from the  input and approximating the corresponding black box responses. In this setting, DAME takes about $50$ sec. of computation time (single A-6000 NVidia GPU on an Intel server) to train on a single sample. The reference numbers for LIME: 25 sec., and RISE: 40 sec. for the same settings. Hence, DAME is only about $25$\% more expensive compared to baseline method of RISE, while yielding performance gains on various tasks and evaluation metrics listed in the paper and the responses.

Reviewer comment (Evaluation):
Results from RISE are often quite competitive in tables 1 and 2. Smooth Grad is also quite competitive.

Author response:
SmoothGrad requires internal access to the model in the form of gradients to generate explanations, which is highly restrictive for various black-box models. The proposed approach is gradient-free post-hoc XAI setting, with only input-output access (as described in the paper, various recent models like GPT are released without any internal access to the model architecture). Hence, gradient based XAI methods merely correspond to upper-bound for gradient free methods. Further, as seen in Table I of the main draft, gradient methods like SmoothGrad perform poorly on Transformer based models like ViT.

With regard to comparison with RISE, the  proposed  DAME framework,
   * Improves in IoU metric over the RISE on all the audio based evaluations.
   * Improves over the RISE/LIME on image based evaluations using the fidelity based measures, for example, deletion (please refer to Section A.12 and Figure 19 in revised draft). 

Also The XAI for object/event classification should be able to shed some light on mis-classifications by the black box, apart from explaining the correctly predicted samples. To examine this, we explored generating saliency based explanations for the mis-classified samples in VOC dataset and compared them with the ground truth annotations. We found that the mean IoU, using different XAI methods, elicit a drop for all XAI methods, with the largest drop for DAME approach as shown in below Table. It indicates that the black box model may have focused on the contextual regions (outside the target class bounding box) for these samples.
| method | mean IoU | IoU drop from correct samples |
|------------|----------------|----------------------------------------------|
| RISE       | 19.7 (14.2)    |     11.4                                         |
| LIME       | 16.6 (13.2)    |     10.2                                         |
| DAME       | 16.2 (13.9)    |     15.2                                         |
**Table**: IoU for misclassified samples.
   * Subjective tests indicated improved preference for the explanations provided by DAME over the other methods.

Reviewer comment (Experiments):
This approach is akin to a gradient-based approach in the guise of gradient-free. If one was to distill the whole black-box into another model (not just in the sample’s neighborhood) and then apply any gradient-based method, I believe that that would be much simpler since one won’t have to train a smaller model to get an explanation for each sample and I believe it would perform competitively seeing the numbers in tables 1 and 2. So, I have doubts about why the authors have taken this round-about route. It is at least worth it to compare this with works that use distillation to understand models.

Author response:
Thank you for raising it, which also warrants a discussion in the main draft about the motivation for DAME. We argue why the global distillation is not a feasible choice for post-hoc XAI methods.
  * Global distillation requires access to the training data of the original teacher (black-box) model.  As DAME is proposed for a post-hoc XAI setting, the requirement of access to the supervised training data might pose a severe limitation to the applicability of the XAI (for example, an XAI for an LLM might need a very training corpus to generate a suitable student model). 
   * Simulating the classification bias and spurious correlation behavior of the larger black-box with a simpler student model globally might be infeasible, which may cause a divergent explanation using the student model compared to the explanations generated for the teacher model.

Reviewer comment (Reproducibility):
In the audio experiment results for task 3, cough data says that it was manually annotated. Are there going to be any plans to release this to enable discussion/reproducibility?

Author response:
The annotations are already publicly available from the publication "On the importance of different cough phases for COVID-19 detection''. Interspeech 2022 with the public link available in that paper. For the camera ready submission, we will also ensure that all our results are reproducible by providing the source codes for DAME through  open-source platforms.

Reviewer comment (Experiments):
Have the authors considered using this method on well-known spurious feature detection image datasets like Decoy-MNIST and ISIC?

Author response:
Thank you for raising the interesting question. Studying the effect of bias, spurious correlation in datasets/models on XAI method performance is an interesting area of research. 
We plan to include this as a future scope of extension of this work.

Reviewer comment (Experiments):
If one was to distill the whole black-box into another model (not just in the sample’s neighborhood) and then apply any gradient-based method, I believe that that would be much simpler since one won’t have to train a smaller model to get an explanation for each sample and I believe it would perform competitively seeing the numbers in tables 1 and 2. So, I have doubts about why the authors have taken this round-about route. It is at least worth it to compare this with works that use distillation to understand models. I would like to get the authors thoughts on these points.

Author response:
Thank you for raising it, which also warrants a discussion in the main draft about the motivation for DAME. We argue why the global distillation is not a feasible choice for post-hoc XAI methods.
  * Global distillation requires access to the training data of the original teacher (black-box) model.  As DAME is proposed for a post-hoc XAI setting, the requirement of access to the supervised training data might pose a severe limitation to the applicability of the XAI (for example, an XAI for an LLM might need a very training corpus to generate a suitable student model). 
   * Simulating the classification bias and spurious correlation behavior of the larger black-box with a simpler student model globally might be infeasible, which may cause a divergent explanation using the student model compared to the explanations generated for the teacher model.

Reviewer comment (Reproducibility):
Can the authors clarify if I have incorrectly interpreted the audio experiments setup or results? Are there going to be any plans to release manually annotated cough data to enable discussion/reproducibility?

Author response:
The annotations are already publicly available from the publication "On the importance of different cough phases for COVID-19 detection''. Interspeech 2022 with the public link available in that paper. For the camera ready submission, we will also ensure that all our results are reproducible by providing the source codes for DAME through  open-source platforms.

Reviewer comment (Experiments):
Does the architecture of the map generation and student network affect the performance? Does it need to be shallow or deeper? What are the design considerations for these networks?

Author response:
We experimented with different architectures of the student network. Using a two layer fully connected network (FCN) following a 2 layer convolutional network (CNN) as student network gives significantly better performance than a other architecture chices as shown in Table 2.2 below.
| **DAME variants**                               | **ResNet-101** | **Vit base-16** |
|-------------------------------------------------|----------------|-----------------|
| student network: 2 CNN + 1 FCN layers           | 31.8           | 30.3            |
| student network: 2 FCN layers                   | 32.2           | 30.9            |
| student network (paper) : 2 CNN + 2 FCN layers  | 33.3           | 31.4            |
**Table 2.2:** IoU values obtained using different student networks with DAME.

Regarding depth, a student network which is  deeper, increases the time overhead to generate the explanations for each input sample. As we attempt a local approximation only, a shallow network may be sufficient which also provides significant savings in computation time.

Reviewer comment (Experiments):
How does the proposed method compare to just distilling a smaller model from the black-box model and then using the distilled network to generate saliency maps (and use these as explanations for the black-box model)? This should be a baseline.

Author response:
Thank you for raising it, which also warrants a discussion in the main draft about the motivation for DAME. We argue why the global distillation is not a feasible choice for post-hoc XAI methods.
  * Global distillation requires access to the training data of the original teacher (black-box) model.  As DAME is proposed for a post-hoc XAI setting, the requirement of access to the supervised training data might pose a severe limitation to the applicability of the XAI (for example, an XAI for an LLM might need a very training corpus to generate a suitable student model). 
   * Simulating the classification bias and spurious correlation behavior of the larger black-box with a simpler student model globally might be infeasible, which may cause a divergent explanation using the student model compared to the explanations generated for the teacher model.

Using a local student model for the sample under consideration and its perturbations is experimented and reported in Table 2.3 below.

Reviewer comment (Experiments):
What is the need for a map-generation network in the framework? Can't we distill the black-box model through a student network exposed to the perturbations?

Author response:
Thank you for pointing this out. We used the student model alone (without the map generation model) to distill directly on perturbations. This locally trained model is then used to generate explanations using LIME/RISE. They prove to be inferior to the ones proposed in the paper (DAME). Also the key observation is, performing explanation and distillation jointly, as performed in the DAME, is superior to the two steps performed in isolation (distillation followed by explanation). The results are in Table 2.3 below.

| **Baselines**                       | ResNet-101 | Vit base-16 |
|-------------------------------------|------------|-------------|
| Baseline 1: student network + RISE  | 31.2       | 30.5        |
| Baseline 2: student network + LIME  | 28.5       | 25.9        |
| mask-generation + student (DAME)    | 33.3       | 31.4        |
**Table 2.3**: Student model with and without mask generation network.

Reviewer comment (Experiments):
The authors should add the above two setups as baselines.

Author response:
Yes, we will add them in the final draft.

Reviewer comment (Experiments):
The authors are encouraged to consider benchmarks like CUB and AwA2 (and other benchmarks where concepts are annotated), which contain annotations of salient parts of the image; this helps them compare against some gold standards.

Author response:
Thank you for pointing us to these datasets. We are experimenting with them, and will include the results in the final draft.

Reviewer comment (Reproducibility):
Based on the 3 loss functions that need to be handled, it seems likely that the method may not work out of the box (like LIME) and users will probably need to customize/tune hyper-parameters etc. to get the explanations right.

Author response:
Thank you for the question. Although it is true that the loss function has three components, we found that tuning them for a modality works well for multiple applications with that modality. For example, the hyperparameters tuned for ImageNet was used as is for VOC dataset, and hyperparameters tuned for ESC-10 based audio classification were used as is for the COSWARA application.

Moreover, we would also like to add that the KL-div loss component contributes minimally as can be seen from the Table below. Hence, the MSE loss along with a L1 loss component (to avoid singularity) itself may constitute a  reasonable choice for the loss function in DAME.
| **DAME variants**    | **ResNet-101** | **Vit base-16** |
|----------------------|----------------|-----------------|
| loss: MSE+L1         | 32.4           | 30.8            |
| loss: MSE+L1+KL-div. | 33.3           | 31.4            |

Reviewer comment (Reproducibility):
How is local vicinity and distance between the given sample and perturbations defined in DAME - is this same as LIME?

Author response:
We used the same measure of distance between a sample and its perturbations, the L2 distance between them **in the input space**. On the contrary, LIME uses the distance as the L2 distance between them **in the binary space** ($M_{x^{(k)}}$ space, as described in Section 4.2 in the draft). 
The distance measure in input space results in higher sensitivity because masking-off larger regions and smaller segments may not be similar.

Reviewer comment (Evaluation):
In case of DAME, can the authors comment on local invariance of explanations (do similar inputs yield similar explanations)?

Author response:
Yes, we have verified this experimentally that similar inputs indeed generate similar explanations.
We have also performed an attribution experiment between the student and teacher model. 
We removed the top-k features (based on XAI output) gradually (k increasing from 0 to 100\%) from the image and pass the masked images through the DAME student model  as well as the blackbox model to get the  scores. The DAME scores show high similarity with the black box scores. An average coefficient of determination ($R^2$) of $0.819$ was obtained for the samples in VOC dataset, confirming the agreement between DAME student and the black box model.

Reviewer comment (Experiments):
Would DAME be impacted by correlated featured?

Author response:
Examining sensitivity of XAI methods with respect to correlated features is an important area of research. While a detailed quantitative evaluation was not possible within the short rebuttal period, we have resorted to a qualitative evaluation. An example image of a cat with correlated features (the shadow associated with it) and the explanation generated by DAME is shown in Figure 21 of the updated draft.  
The shadow has similar shape as the cat, leading to a correlated feature in the input. However, the DAME generated explanation is not influenced by the presence of the shadow.

Reviewer comment (Theory):
Instead of using a masking approach to generate perturbations (e.g. LIME), if we have a realistic distribution of perturbed images (e.g. MeLIME), can the DAME pipeline still be used?

Author response:
We thank the reviewer for pointing us to this resource.
The proposed approach is indeed agnostic to the choice of masking or perturbation generation approach.
MeLIME proposes an effective perturbation strategy in general, that improves the explanations provided by different methods like LIME. Because this perturbation strategy essentially focuses on generating samples that better capture the representation of local vicinity of the input, it constitutes a promising experiment to explore before the camera ready version of the submission.