Paper: Path Choice Matters for Clear Attributions in Path Methods (ICLR-2024)

Reviewer comment (Writing):
There are some minor typos and latex error for the manuscript.

Author response:
Thank you for the feedback from the reviewer. We have double check the content of the paper and have rectified the typos and other mistakes.

Reviewer comment (Theory):
Brownian motion is the erratic motion of particles suspended in a medium due to continuous collisions. Assumption 1 assume the additive process as the Brownian motion. Please explain why is this valid and the intuition behind.

Author response:
We mainly elaborate on the rationality of Brownian motion assumption in the SAMP method from four aspects.
- **Intuition behind Assumption 1:** We believe that the allocation of attributions should be a random variable in each step **without any constraints**, and there should be independence among allocations. Without special knowledge about the internal structure of the model, we give the assumption of **isotropy in each allocation**, i.e., each allocation follows an independent and identical Gaussian distribution.
- **The relationship with Proposition 1:** It is noteworthy that **Proposition 1** has significant differences from **Assumption 1**. Assumption 1 is the **joint distribution** of the isotropic Gaussian assumption under the situation where **allocations have no constraints**, while the real allocation process is **under the condition $\sum_{i=1}^d a_i = C$**, so **Proposition 1** explores the **conditional distribution** among attributions under this condition.
- **The universality of this assumption:**. Brownian motion (also known as Wiener process) originally refers to the erratic motion of particles suspended in a medium due to continuous collisions, which is a concept in physics. Thereafter, its mathematical model has been widely applied in various fields such as queueing theory, insurance, and computer science. For example, recent popular **diffusion models**[2] make a **Brownian motion assumption** on the forward process for image inputs and achieve excellent image generation performance. In this work, we use the Brownian motion assumption as a tool to **reduce computational complexity** and experimentally verify the effectiveness of this assumption.
- **The consistency of experimental results:** Based on the Brownian motion assumption, we derived **Proposition 1**, and **Eq(6)** of Proposition 1 demonstrates that $\mathbb{E}(u_k|u_d=C) =  kC / d$, implying that a randomly selected path tends to result in **linear changes** (increase or decrease) in output scores. This phenomenon was also observed in our experiments, as shown in **Figure 1**. We discovered that most other methods (except SAMP) exhibited a trend of **linear change** in their Deletion/Insertion curves, which aligns with our theoretical assumptions and further validates the rationale of this assumption.

Reviewer comment (Theory):
The manipulation path is pre-defined. How is the path defined? Could these paths sufficiently cover most scenarios for the search of near-optimal path?

Author response:
Thanks for the suggestions. We have added the following supplementary information related to path concepts:
- **About Path definition:** Our definition of path primarily follows that in **Integrated Gradients (IG)**[1], as also detailed in **Section 3.1** of our paper. In brief, a path refers to a integral path along which the model output changes. For instance, in the case of deletion direction, the starting point is the original image, and the ending point is the black/noisy image.
- **Rationality of Manipulation Path:** We mainly explain the rationality from two aspects:
  - **Relationship between Manipulation Path and Shapley Values:** Shapley values[3] are a classic axiomatic explanation method. Recent work[4] demonstrates how to apply Shapley values to generate explanations in machine learning problems, where a typical scoring function is **removing features with baseline values**, which involves removing partial data features and replacing them with baseline values. This is fundamentally **consistent** with the definition of the Manipulation Path in **Definition 2**.
  - **Relationship between Manipulation Path and Some Classic Paths:**
    - The classic path method **IG**[1] chooses a straight line connecting the starting and ending points, which can be seen as a special case of the Manipulation Path with infinitesimal constraints of a very small upper bound $\eta$ and the update step as $s=d$.
    - Additionally, the **Insertion/Deletion metric** proposed in RISE[5] also utilizes an operation involving replacing partial pixels with baseline values at each step, which is fundamentally **consistent** with the definition of the Manipulation Path.

Reviewer comment (Writing):
What is the full name for IG?

Author response:
We apologize for the confusion about this terminology. 
IG's full name is **Integrated Gradients**[1]. We have clarified the terminology of this method in the **introduction** section of the revised version of the paper.

Reviewer comment (Writing):
Should "Attributions" be changed to "Attribution" in the title of this paper?

Author response:
Thanks for the reviewer's suggestion. We have corrected "Attributions" to "Attribution" in the revised version.

Reviewer comment (Evaluation):
Since the proposed method is attributed to the pixel level, I hope the author can discuss the advantages of this method compared to the method attributed to features, because it seems that the explanation attributed to the region level is more convenient for human understanding.

Author response:
Thanks for the insightful questions from the reviewer. Our responses are as follows:
- **About the level of Path method explanation:** We agree with the reviewer that feature/region-level explanations are often more understandable. However, we also want to emphasize that the path method is an explanation technique that is **independent of the explanation level**, and can theoretically be applied at both the pixel level and the feature level. For instance, one can change the path $\gamma(\phi)$ in **Eq(1)** from the pixel space to the feature space.
  - When conducting literature survey, we discovered that some studies (e.g., **XRAI**[4]) have explored how to apply the path method to the feature/region level explanation. Therefore, our study did not focus on the level of explanation, but rather, it primarily investigated the method itself.
- **About future directions:** Thanks for the reviewer's suggestions. We also believe that explanations at various levels can enhance the diversity of explanations. In our future research plans, we are inspired to integrate existing regional explanation techniques (e.g., XRAI[4]) with the proposed SAMP and provide it to other researchers in the form of an open-source code library.

Reviewer comment (Experiments):
In addition to natural image datasets such as Imagenet, I suggest that the author can try to apply this method to natural language interpretation or medical images. Because a single word in natural language has strong semantics and is easy for humans to understand, it is most critical for medical images to have small areas. This can better reflect the practical application value of this article.

Author response:
We find that this is an issue that all three reviewers are concerned about.
But we have to admit that verifying its effectiveness on NLP within a limited time poses a challenge since the SAMP method was not originally intended for such tasks.
Moreover, our expertise in medical imaging is somewhat limited, leaving us unsure of the specific requirements for image interpretation in this field. It would be greatly appreciated if the reviewer could recommend relevant literature on this topic.
We will consider these 2 points in our future work.
Regarding the reviewer's inquiry regarding the applicability of SAMP in real-world scenarios, we have carried out supplementary testing on two more intricate situations, as detailed below:
- **Multi-category attribution:** To further explore the potential applications of SAMP, we conducted an experiment detailed in **Appendix A.5.7** involving the splicing of images from the **MNIST** dataset into **2x2 grids**, each containing at least **two digits**. These images were then classified using a CNN before being subjected to attribution via the SAMP method for various categories. Our findings indicate that SAMP can accurately locate only the specified category of data within a given image, even when presented with a mixture of different digit classes. This suggests that SAMP may be **applicable to multi-category** classification problems.
- **Fine-grained attribution:** We conducted experiments on the **CUB-200-2011** fine-grained bird classification dataset to further evaluate SAMP's performance on complex data. Details of this experiment can be found in **Appendix A.5.8**. Here, we randomly selected 5 images from the dataset and cross-attributed their respective categories, producing a total of **25 attribution visualizations** displayed in a **5x5 grid** format. Our observations revealed that attribution results for the correct categories were more pronounced, supporting the effectiveness of SAMP for fine-grained data.

Reviewer comment (Theory):
The author mentions the case of model agnosticism. If we consider that the internal gradient of the model is accessible, that is, a white-box model, can the method proposed in this article achieve better attribution results? Or why doesn't it work with white box? I hope the author can discuss relevant content.

Author response:
Thanks for raising this question about white-box. 
We apologize for not being clear about this in the original paper.
- **About the nature of the proposed method:** The SAMP method proposed in this paper belongs to **Path Method**, which is related to related works, such as **IG**[1], **BlurIG**[2], and **GuidedIG**[3]. This category of methods **do not** rely on internal variables or gradients of the model, making them distinct from the white-box methods mentioned by the reviewer.
- **About the advantages of path methods:** There are two key benefits to not considering model gradients. 
  - **Firstly**, for large and complex models, accounting for all internal variables would significantly increase computational overhead. 
  - **Secondly**, focusing solely on the input and output characteristics of the model allows for **implementation invariance** in explanation methods, meaning that for different model implementations, as long as the external input and output characteristics remain the same, the generated explanations will be consistent. This property has been highlighted in the previous research[1] (i.e., Integrated gradients).
- **About compatibility with white-box methods:** To be honest, as our motivation was **not to consider** variables within the model (as outlined in the "About the advantages of path methods" above), our method does **not natively support** incorporating gradients within the model. However, the reviewer's suggestion has given us food for thought. We can explore taking gradients within the model as an additional input in future work and investigate whether it can enhance the performance of post-hoc interpretable methods.

Reviewer comment (Evaluation):
Figure 8 states that "as η decreases, the attribution visualization becomes fine-grained." However, at η=10, the visualization appears overly dark, making it challenging to understand. Similarly, Table 3 suggests that “since the main role of IC is to ensure rigorousness, its effect in improving performance is not significant”. Have the authors conducted a statistical test to substantiate this claim?

Author response:
Thanks for the insightful suggestions on details. We have carefully considered them and implemented appropriate changes to the paper. The detailed responses are as follows:
- a.Sorry for the confusion caused by the description. 
  - **Firstly**, $\eta$ does not solely refer to the denominator. "as η decreases, the attribution visualization becomes fine-grained" actually means that the finest granularity is achieved when $\eta = 1/200$, not $\eta = 1/10$. We will clarify this in our revised version.
  - **Secondly**, in the description "its effect in improving performance is not significant", the "performance" refers to the Deletion/Insertion metrics. We apologize for not being explicit about this. Since Deletion/Insertion[2] is fundamentally an heuristic metric, the strictness of the interpretation method is not directly associated with it. Therefore, we observed this phenomenon experimentally (**Table 3** and the "+IC" row in **Table 2**). Thank Inspired by the insightful feedback from the reviewer, we will continue to delve into the underlying causes in future works.
- b.Thanks for the suggestion on **Section 4.4.2**, we have now clarified the specific meaning of "B+W" in the revised paper. Regarding **Figure 7**, we apologize for the confusion in our description. We found that the impact of different baselines on the explanations is not significant compared to different methods (as shown in **Figure 4**). However, visually, different baselines have little impact on the **contour information** in the explanation results, but they do significantly affect the **overall intensity** of the explanation results (e.g., brightness), which leads to significant differences in appearance. This point has already been added in the revised version.
- c.We apologize for the confusion caused by certain terms in the original paper. In **Figure 4**, the SAMP actually refers to the results of **adding IC and MS (i.e., SAMP++)**. We have updated **Figure 4** accordingly. At the same time, we have also included the **original SAMP** algorithm without IC and MS for comparison. As can be seen from the comparison, the visualization results of SAMP++ are **broadly similar** to SAMP, but the results of SAMP++ are more fine-grained (for instance, the subject is more separated from the background).

Reviewer comment (Experiments):
How do we understand the impacts of baselines on two paths? To be concrete, in Appendix A.5.3, “the setting of Deletion/Insertion is consistent with ‘B+G’”. How are they related?

Author response:
Sorry for the lack of clarity in this section. we have revised the original paper in **Appendix A.5.3**. Specifically, for the Deletion/Insertion metrics, we refer to the **open-source code**[1] from RISE[2]. When calculating this metric, RISE utilizes a **black image** (i.e., **B**) as the baseline for the Deletion metric and a **Gaussian blurred image** (i.e., **G**) as the baseline for the Insertion metric.
Hence, based on the results in **Table 6**, if we also adopt the **B+G baseline** to generate explanations, which maintains consistency with the metric calculation setup in RISE, the Deletion/Insertion metric would be the best.

### Reference:

[1] https://github.com/eclique/RISE  
[2] Petsiuk V, Das A, Saenko K. Rise: Randomized input sampling for explanation of black-box models, BMVC 2018