Paper: Toward $\textbf{F}$aithfulness-guided $\textbf{E}$nsemble $\textbf{I}$nterpretation of Neural Network (ICLR-2024)

Reviewer comment (Evaluation):
Lack of in-depth discussion of the visualization result. It seems that EP can also obtain good visualization and locate the Submarine better than FEI. It would be better to discuss the impact of 'hot' and 'cold' areas in heat map.

Author response:
From the visualizations we presented, certain conclusions can be drawn regarding hot and cold areas. For instance, when the subject is an animal, the model tends to focus more on the animal itself, whereas with objects, only parts of the object may be sufficient for the decision. However, it's important to note that we provided a limited number of visualizations, and generalizing these phenomena might be challenging. Therefore, we refrain from making extensive discussions.

Additionally, we aim to avoid comparing visualizations with segmentation because it assume that the model perceives the image from a segmentation point of view, which might not be true. As you pointed out, EP may appear closer to segmentation for the seaship compared to our method. However, closer resemblance to segmentation does not necessarily imply a better visualization.

Reviewer comment (Writing):
Section 3 is difficult to understand, at least partially due to notation. For example, the LHS of (2) seems to be a scalar based on notation (dot products on the RHS), but from context it seems that the authors are using the \cdot to mean element-wise multiplication (Hadamard product). There are other things that are confusing like having $\alpha_f(p)$ and then $\alpha_p$, where $p$ presumably means different things in each context - it's an argument, but then it's also used as the fractile. Another example; in (3), $\tilde x$ and $l_{faith}$ should be notated to depend on $f$.

Author response:
We apologize for any confusion caused by the notation. The dot product in equation (2) should be interpreted as a Hadamard product, and $\alpha_p$ should be written as $\alpha(p)$ in equation (3). The symbol 'p' represents the pixel index of the mapping consistently across these notations. $\bar{x}$ is a typo and should be corrected to $\tilde{x}$. We also change notation for to clarify that $loss_{faith}(f)$ and $\tilde{x}(f)$ corresponding to different fractiles

Reviewer comment (Evaluation):
The relation between faithfulness metrics and faithfulness optimization is unclear.

Author response:
The details of faithfulness metric is described in the faithfulness section in section 2.2. The faithfulness metric can be categorized into two types: deletion metric and preservation metric. The deletion metric assesses the quality of the explanation by systematically removing highly attributed pixels and evaluating the decrease in probability output for the desired category . On the other hand, the preservation metric retains highly attributed pixels while removing others and measures the increase in probability confidence. 

Since we don't know how many pixels to operate on, the usual practice is to start with the entire input set to blank and compute the area under the curve (AUC) as $Q_D$, and then proceed from blank to the entire image to compute $Q_P$.

To convert this metric into an optimization problem, a common practice is to use the Hadamard product. For the Preservation metric, we multiply the attribution map with the input and the complement of the map with a reference image. For the Deletion metric, we do the opposite.

However, this approximation lacks knowledge regarding the number of pixels to be operated on. Therefore, we introduce fractiles to approximate a specific point on the curve. Multiple such points are required to interpolate the entire process. Hence, we propose the adoption of an ensemble method involving multiple fractiles.

Reviewer comment (Evaluation):
The results seem mixed at best; there is improvement in Q_P, but overall Q_D is not improved and their defense mechanisms seem to be worse than baseline (this section was difficult to interpret, but based on the text it seems that lower is better).

Author response:
The quantitative evaluation metric we chose is commonly used. Recent studies typically limit their comparisons to methods within specific categories, like those exclusively gradient-based or perturbation-based. In comparison, our approach conduct a comprehensive comparison across all categories. Gradient-based methods commonly exhibit superior deletion score ($Q_D$) but a terrible visualization quality and preservation score ($Q_P$). Our method, even with a slightly lower performance than the top-performing gradient-based method (Integrated Gradient) in $Q_D$, demonstrates significantly improved preservation scores and visualizations.

Moreover, we also demonstrate the power of gradient clipping in both the defense evaluation and a qualitative visualization.

Reviewer comment (Theory):
In 3.2, you define a consistency constraint $\alpha_1 \le \alpha_2$ and then state that optimizing under these constraints is challenging. Have you considered parameterizing $\alpha$ as a cumulative sum? e.g. $\alpha_i = \sum_{j\le i} \delta_j$ In this case, the constraints becomes $\delta_j > 0$ and $\alpha_N=1$, which is easier to handle in an SGD framework.

Author response:
Optimizing the attribution maps as a cumulative sum of $\delta_i$ could offer better consistency guarantees. However, If we  search for all $\delta_i$ at the same time, it is nontrivial to set$ \alpha_N < 1$, so this approach still requires optimizing each \delta_i separately. In this formulation, there is a loss of flexibility and a potential increase in noise. For instance, if $\delta_1=1$, the sum would be fixed at 1, regardless of later $\delta_i$ leading to the accumulation of noise from different $\delta_i$, This issue can be alleviated by increasing the optimization time, but it becomes less efficient. Our cascading optimization procedure, while providing a weaker consistency guarantee, presents a higher chance of eliminating more noise since each map is generated independently.

Reviewer comment (Writing):
What is $\bar x$ in (5)? Is it the same as $\tilde x$?

Author response:
$\bar{x}$ is a typo and should be corrected to $\tilde{x}$.

Reviewer comment (Reproducibility):
Are the gradients clipped at every single layer? If not, how do you define the decomposition of the network?

Author response:
Gradient Clipping is applied to the activation layer in the feature extraction part of the neural network as described in end of section 3.3