Paper: On the Generalization of Gradient-based Neural Network Interpretations (ICLR-2024)

Reviewer comment (Theory):
Regarding the definition of \( f^* \) in Eq.(5): there is no guarantee that you only have one such optimal classifier. In the case where Eq.(5) returns a set of \( f^* \), doesn't the definition of loss in Eq.(6) become problematic? While all the \( f^* \) will yield the same testing performance, they do not necessarily produce the same output for the interpretation method. In other words, \( \mathrm{Loss}_I(f,x) \) is also a function of \( f^* \), denoted as \( \mathrm{Loss}_I(f^*,f,x) \). This differs significantly from studying standard generalization error. Therefore, the generalization error defined in Eq.(6) can vary for the same \( f \) and \( x \).

Author response:
**Re:** We note that if the minimizer to the population risk function is not unique, we could define  $f^*(x)=E_{f\sim \mathrm{Unif}(F^*)} [f(x)]$ as the expectation under the uniform distribution on the set $F^*$ of optimal population solutions, and our generalization guarantees for Theorems 1,2 will hold for this choice of $f^*$.

Furthermore, as we discussed in the general response to comment 2, $I(f^*,x)$ in our definition of interpretation loss can be changed to $E_{S,A}[I(f_{A(S)},x)]$ where we take the expected value of  the interpretation map over the randomness of training set $S$ of size $n$ and a stochastic learning algorithm $A$. In that case, we have a unique expected value which can also be used for the analysis. Please note that the same Theorems 1,2 will also hold for the choice of reference map $I(f^*,x) = E_{S,A}[I(f_{A(S)},x)]$.

Reviewer comment (Theory):
Let's assume there is only one such \( f^* \). According to Theorem 1, the generalization bound implies that interpretation demands a larger training set compared to the standard classification problem. An essential mystery in the success of deep learning is that overparameterized neural networks can generalize well without needing more data than the number of parameters. If Simple Gradient Method and Integrated Gradient prove to be unreliable with the same amount of data, does this imply that they are ineffective for interpreting deep neural networks?

Author response:
**Re:** This question is precisely the motivation of our generalization study, which highlights the importance of the generalization error (or variance) of an interpretation map with respect to the randomness of training data in the selection of an explanation method and its hyperparameters. For example, if SmoothGrad has a significantly lower generalization error in a learning setting, should a practitioner prefer SmoothGrad over SimpleGrad? As we argued in our general response to comment 1, we believe the answer to this question is yes and our work aims to better understand the generalizability of gradient-based maps.

Reviewer comment (Evaluation):
Post-hoc explanations have been criticized for some time now in the community due to their reliability in explaining deep network presentations [1,2,3] (some of the references analyze the saliency map techniques evaluated in this work). This means that even if I had the **exact** same saliency map as the ground truth f*, methods like integrated gradients and simple gradients are simply unreliable in explaining what the deep network is doing.

Author response:
**Re**: We would like to clarify that our generalization analysis does not focus on finding the interpretation map under ground-truth solution $f^*$. As argued in the general response to the previous comment, we only attempt to measure and control the variance of the interpretation map with respect to the randomness of training data used for training the neural network. Also, our analysis reveals the application of Gaussian smoothing in SmoothGrad to control the variance of the interpretation map.

Reviewer comment (Evaluation):
The following is unclear "On test set data, we plot the normalized Spearman correlation of network interpretations against softmax predictions." What exactly is the equation for this computation? Is the softmax predictions the argmax of the softmax or the entire k-dimensional softmax scores? What are the two quantities among which the correlation is computed?

Author response:
**Re:** In Figure 1.b, we plot the Spearman rank correlation between the two interpretation maps of the networks $f_{S_1},f_{S_2}$ in the y-axis versus the (Pearson) correlation coefficient between the argmax (i.e. prediction) of the softmax layer (i.e. the fraction of samples with the same argmax prediction by $f_{S_1},f_{S_2}$) in the x-axis. We have clarified this point in the revision.